Data Quality & the Completeness Gate
What it does
Every figure this platform shows you — a yield, a portfolio valuation, a ladder, an alert date — is derived from a Central Bank of Kenya prospectus. If a coupon date or a maturity is read wrongly from that PDF, every number computed from it is wrong, and nothing on screen would tell you.
The completeness gate is the rule that prevents this: a prospectus is only accepted if everything it contains was successfully read. If the document states a coupon schedule and the parser could not extract it, the upload is refused and nothing is stored. We would rather be missing a bond than hold a wrong one.
The gate now applies to every prospectus layout. CBK has issued bonds under several document styles over the years — modern term tables, older narrative documents, tap sales and switch auctions. The check originally ran only for the newest style, so an older document could be saved with gaps it never announced. It now runs for all of them, and any figure the document prints but the parser cannot read blocks the upload rather than being stored as a silent hole.
Present, absent, and the difference
The gate turns on a single distinction — did the document actually contain the item?
| Situation | What happens | Why |
|---|---|---|
| The PDF states it, we could not read it | Upload refused, nothing saved | The data would be wrong, not merely thin. This is a gap in our parser. |
| The PDF does not contain it | Saved, with the absence recorded | Older formats legitimately lack fields. Refusing these would make historic bonds impossible to load. |
| A later document supplies it | The existing bond is updated | Re-openings often carry cleaner data than the original issue. |
Absence is never assumed. Before concluding that a prospectus lacks something, we search it for evidence — the field's row in the terms table on page 1, or its label anywhere in the text. Only a field the document shows signs of carrying can block an upload.
Core terms versus supplementary reference. Only the terms every calculation depends on — coupon rate, tenor, maturity, withholding tax, and the coupon payment schedule (which must agree with the maturity) — can refuse an upload. The pricing table (a yield-to-price grid) and the accrued-interest figure are treated as supplementary: both are fully derivable from the core terms, and the secondary market calculator computes them on demand. We still store them when the prospectus prints them and our parser reads them, but their absence never refuses an otherwise complete bond — a historic bond is not held hostage to a reference grid we can recompute.
A real example
The August 2026 prospectus for the re-opened infrastructure bonds covers three securities. Uploading it produces:
Processed successfully — 3 bond(s) found: IFB1/2019/016, IFB1/2021/018, IFB1/2021/021
Behind that, each bond was checked for its ISIN, coupon rate, withholding tax, tenor, maturity date and full coupon schedule — 19, 26 and 33 payment dates respectively. All present, so the upload is accepted.
By contrast, the December 2009 infrastructure prospectus — a narrative document from before CBK adopted the modern table layout — produces:
The document contains tenor, period of sale, total amount, but it could not be extracted. Nothing was saved.
That document does state a twelve-year tenor and an amount of KES 18,500 million; our parser simply cannot yet read that layout. So it is refused rather than stored with holes. Note that it also has no ISIN anywhere — that is recorded as genuinely absent and does not block anything, because 2009 prospectuses predate printed ISINs.
The maturity cross-check
A prospectus states each bond's maturity date in its terms table, and separately lists every coupon payment date. The final coupon must fall on the maturity date.
These are read from different parts of the document, so if they disagree, one of them was read wrongly. That disagreement refuses the upload:
IFB2/2009/012 matures 22/11/2021 but its last coupon is 02/12/2009
This check is valuable precisely because it needs no knowledge of the layout — it compares the document against itself.
What you see
| Message | Meaning |
|---|---|
| Processed successfully — 3 bond(s) found | Everything the document contained was read and stored. |
| Parsed without: the PDF contains no ISIN | Saved. The document genuinely lacks that field. |
| The document contains … but it could not be extracted | Refused, nothing saved. The parser needs to learn this layout. |
Seeing it across the archive — the Data Quality page
Staff have a Data Quality page (linked from Files) that turns all of the above into one viewable table: every processed prospectus, its bonds, and the exact issues on each. A row is one of four states — complete (everything read), incomplete (saved, but the document genuinely lacks a field), needs parsing (data present that we could not read, so nothing was saved), or failed. Two columns separate the two kinds of gap: missing (printed but unread — a parser gap to fix) and absent (not in the document at all).
It is searchable by file or bond and filterable by state, so the refused files that still need parser work are one click away. The whole archive can be assessed into this view without saving or moving anything — the record of what each file lacks is written even when its data is not stored.
The correction queue — flag, complete, promote
Refusing a file keeps bad data out, but it also leaves a real bond unrecorded when the document was almost fully read. The curation workflow closes that gap. In a dedicated curation database, a bond with a field we could not read is saved with everything we did read and flagged needs review rather than refused. The top of the Data Quality page lists every flagged bond, the fields it still needs, and a link straight to its edit form.
When an admin fills the missing fields, the bond clears to ready on its own — the flag follows the data, there is nothing to toggle. A bond is ready when its coupon rate, maturity date and tenor are present and any stored coupon schedule ends on its maturity date; a bond still missing a term, or carrying a schedule that disagrees with its maturity, stays flagged. Only ready bonds are ever promoted to the live site, so an incomplete bond can never reach a price, a portfolio or a ladder — the same guarantee the completeness gate gives, reached by completing the data rather than discarding it.
The page makes the work visible. A Ready for promotion count and a curation progress bar — ready over the total curated (ready plus needs-review), shown as a percentage — sit above the queue so you can see the archive move towards completion. Say a batch loads 144 bonds, 76 already complete and 68 flagged: the bar reads 76 ready · 68 to review · 53% done. Each Complete → link opens the bond with a ← Back to correction queue button on its edit form; fill the fields, save, and click back — the bond has left the queue, the ready count is 77, and the bar has advanced. Work the list to zero and every recoverable bond is ready to promote.
The flag most bonds get stuck on is coupon schedule ending on the maturity date: the parser read a truncated schedule whose last coupon date is not the maturity, so the schedule and the maturity disagree. The bond edit form has a Coupon Schedule section for exactly this. Generate semi-annual schedule ending on maturity rebuilds the standard Kenyan schedule — six-monthly coupons from the settlement date to maturity (or, when no settlement date is on file, stepped back from the maturity by the tenor) — and you can add or remove individual dates to match the prospectus before saving. A 2-year bond maturing 30/01/2012 becomes 01/08/2010, 01/02/2011, 01/08/2011, 30/01/2012; the last date now equals the maturity, so the bond clears to ready. Amortizing and irregular bonds, whose schedules the generator cannot assume, are corrected by editing the list against the document.
Not every missing coupon rate is a parser gap. Many Kenyan bonds are market-determined — the coupon is fixed by the auction, not printed in the prospectus — so the rate legitimately arrives later, in the auction results. Loading the results fills the coupon on the matching bond and clears its flag automatically. A results file whose issue number matches no prospectus is not discarded: the bond is created from the results data, flagged results-only (its provenance is recorded permanently), and listed in its own section so someone can confirm it or supply the prospectus. To protect accuracy, a rate read from results never overwrites one already read from a prospectus — a disagreement is logged for review, and only fully-ready bonds are ever promoted to the live site.
Limitations
- A refusal means our parser has met an unfamiliar layout, not that your file is faulty. CBK changes the prospectus format regularly; each new shape is added as a permanent test case once seen.
- Evidence detection is textual. A field printed only as an image, with no text layer, is indistinguishable from one that is genuinely absent.
- The gate governs prospectus uploads. Auction results follow a separate path.
- It validates completeness, not correctness against CBK's own records. A figure misread as another plausible figure would pass, which is why the maturity cross-check exists — and why more such checks are worth adding.