Production Document Extraction Systems

Define the output contract before choosing a model

A model is one implementation detail. The product contract determines what must be extracted, what evidence must accompany it, and which errors are tolerable for each downstream action.

Separate extraction quality from action risk

• Search indexing may tolerate a missing heading while preserving the original document as the authority.
• Invoice payment must fail closed when supplier identity, currency, total, or reconciliation evidence is missing.
• Claims intake may accept partial data into a review queue, but should not infer an absent signature or approval.
• Regulated records need retention, access control, correction history, and source evidence that match the governing policy.

Route documents with a cheap probe and explicit fallbacks

Routing happens before the expensive extraction path. Use evidence available at intake: MIME verification, native text coverage, page count, image quality, language, layout novelty, classifier output, and the downstream action.

1. 1

   Trust boundary

   Verify the source

   Reject or quarantine corrupt, encrypted, oversized, unsupported, or unsafe files before a parser or renderer handles them.

2. 2

   Document units

   Split and classify

   Detect attachments and page ranges. A mixed PDF can contain several document families that need different schemas and owners.

3. 3

   Cost and latency

   Run the narrowest capable route

   Prefer native text for reliable digital pages, OCR for scans, layout and table models for structural evidence, and specialists for known high-value families.

4. 4

   Recovery path

   Fallback on measured evidence

   Escalate unknown layouts, incomplete pages, and failed gates to a stronger route, bounded review, or quarantine. Do not silently emit partial records.

The lab values below are illustrative operating thresholds, not vendor guarantees. The important decision is which evidence a policy observes and which unsafe records it can release.

Build gates around consequences, not averages

An aggregate document score hides the field that actually controls an action. A production gate evaluates the candidate at several levels.

Give each outcome a precise meaning

• Straight-through: every gate required by the action passed; the candidate can be promoted to an accepted version.
• Specialist route: intake is usable, but the general route does not cover the document family or structure reliably.
• Human review: the source and candidate are inspectable, but bounded fields need correction or confirmation.
• Quarantine: source integrity, page completeness, or evidence identity is broken; a reviewer should not repair values without resolving the input.
• Reject: the input violates an explicit product contract and should not be processed further.

Preserve evidence at field and relationship level

Cloud document APIs expose more than strings. Their result models include text spans, page regions or geometry, confidence, and relationships among words, lines, key-value pairs, cells, and tables. Preserve those links in your own provider-neutral contract rather than flattening the response immediately.

Treat tables as a graph until validation finishes

Do not reduce a table to CSV before resolving merged cells, multi-row headers, units, and continuation across pages. Keep cell IDs, row and column indexes, spans, header relationships, and source geometry. Then validate domain rules such as column types, required identifiers, subtotal equations, and expected row counts.

Make corrections additive

A reviewer correction should create a new field or document revision with actor, reason, timestamp, and the evidence it supersedes. The original model output remains available for evaluation and incident reconstruction.

Evaluate the pipeline with labeled evidence

One document-level accuracy number is usually misleading: fields may be optional, repeated, or absent, and the cost of a false accept can differ sharply from a false review. Evaluate the exact schema and action on a representative labeled set.

Slice the evaluation where failures cluster

• Document family, template revision, supplier or source channel.
• Native PDF, scan, photograph, fax, handwriting, or mixed bundle.
• Language, page count, orientation, image quality, and layout complexity.
• Required and optional fields, tables, repeated line items, and signatures.
• Parser route, model version, schema version, and policy version.
• Straight-through, review, quarantine, and reject outcomes.

Test changes as migrations

1. Pin the current and candidate processor versions.
2. Replay a frozen labeled corpus and recent production samples through both.
3. Compare field precision, recall, structural validity, review load, latency, and escaped risk by slice.
4. Canary the candidate route with immutable outputs and rapid rollback.
5. Promote only after the downstream action contract passes; retain the previous version for investigation and replay.

Operate the system as queues plus versioned state

Keep the asynchronous job contract small

• document_id identifies the logical document; source_hash identifies the exact bytes.
• route_id, processor_version, schema_version, and policy_version make work reproducible.
• An idempotency key collapses duplicate delivery of the same processing intent.
• Attempt records are append-only and include start, deadline, completion, error class, and output candidate ID.
• The accepted-version record changes only through an explicit promotion with optimistic concurrency.

Bound every queue

• Separate interactive and batch work so large backfills cannot starve user-facing extraction.
• Cap retries by error class; corrupt files and unsupported formats do not improve through repeated inference.
• Monitor queue age, not only queue length. Old work can hide behind steady throughput.
• Keep review capacity, retention deadlines, and dead-letter ownership visible.
• Apply backpressure before a parser, model endpoint, or reviewer pool becomes the failure amplifier.

Protect document data

• Isolate tenants and use scoped credentials for source, candidate, review, and accepted-output stores.
• Sandbox renderers and converters; uploaded files cross a hostile-content boundary.
• Redact logs and traces by default. Field values and page images should not appear in general application telemetry.
• Encrypt stored sources and outputs, enforce purpose-based retention, and test deletion across derived artifacts.
• Audit reviewer access and correction history without copying sensitive content into the audit event itself.

Primary sources and current API checks

• Amazon Textract response objects describe block IDs, confidence, geometry, and parent-child or key-value relationships.
• Azure Document Intelligence layout documents spans, bounding regions, tables, figures, and selection marks in current analysis output.
• Google Cloud Document AI Document resource defines text anchors, page anchors, entities, tables, confidence, and normalized values.
• Google Cloud Document AI evaluation explains label-level precision, recall, F1, and the precision-recall effect of changing confidence thresholds.
• Google Cloud Document AI processor versions documents explicit version selection and the risk of relying on a changing default version.
• W3C PROV-O defines entities, activities, agents, generation, use, revision, and derivation for interoperable provenance models.
• OWASP File Upload Cheat Sheet covers allowlisted types, size limits, safe storage, malicious-file scanning, and sandboxing uploaded content.

Provider fields, model catalogs, versions, quotas, and pricing change. Verify the exact API version and region you operate, and benchmark your own documents instead of copying vendor examples into a release policy.