Why we let the extraction model say "I am not sure"
A parser that is right 96% of the time is not 96% useful. What matters is whether it knows which 4% to flag.
Este texto aún no está traducido, así que estás leyendo el original en inglés.
Limited partners receive their financial reality as PDFs. Capital account statements, call notices, distribution notices, K-1s, each from a different administrator, each with its own layout, and none of them designed to be read by software.
Turning that into structured records is the core of what our platform does. The obvious way to measure it is accuracy: what share of extracted figures are correct? We stopped using that number as our primary metric fairly early, and the reason is worth explaining.
Accuracy hides the failure that matters
Suppose a parser reads capital account statements at 96% field accuracy. That sounds strong. Now put it in front of a real portfolio: a few hundred fields a quarter, so roughly a dozen wrong numbers, scattered, with nothing marking them.
A wrong NAV does not stay wrong in one place. It propagates into TVPI, into the fund roll-up, into the quarterly report a firm sends its investors. And because the other 96% is right, nobody is looking.
A system that is confidently wrong 4% of the time is worse than one that is right 90% of the time and tells you which 10% it was unsure about. The second one you can work with. The first one quietly erodes trust in every number it touches.
Confidence has to be per field
The first thing that helped was rejecting document-level confidence. "This document parsed at 91%" is not actionable: you cannot review a percentage.
Every extracted value carries its own confidence instead. A commitment figure printed in a clean table gets a high score. A NAV recovered from a footnote in a scanned fax gets a low one, even in the same document. Review then targets the four fields that need a human rather than the whole statement.
Below the line, nothing is written
Values under the threshold do not land in the portfolio as provisional data with a warning badge. They do not land at all. They sit in a review queue until a person approves them.
This is a deliberately conservative choice and it costs us: a firm that uploads twenty statements sees some work waiting for them. The alternative is worse. Provisional numbers that are already in the system get treated as real ones, because they look identical to everything around them.
Inert until approved is easy to explain, and it makes "what is in the portfolio" mean exactly one thing.
Every figure keeps a route back to its source
Extraction is a claim about a document, so we store it as one. Each valuation and each cashflow event carries a reference to the document it came from, plus what kind of source that was: a capital account statement, a 409A, a priced round, a public mark, or a manual entry by a person.
The practical result is that any number in the system can be traced. Click a NAV and you land on the page of the PDF it was read from. When an administrator restates a quarter, you can find every figure that came from the superseded document instead of guessing.
This is also what makes the audit trail real rather than decorative. It is not a log of who clicked what. It is a chain from the number back to the paper it came from.
The uncomfortable part
The honest version of this post is that we ship a product which sometimes says "I could not read this, please check". That is a worse demo than a system which silently fills everything in.
It is a much better product for anyone who has to sign off on the resulting report. In this domain, calibrated uncertainty is a feature, and pretending otherwise just moves the error somewhere less visible.