Dedoc Production Systems | System Designer

What is Dedoc?

Dedoc is an open-source system that converts documents from different formats into a unified structured representation. It can extract textual lines in reading order, format annotations, hierarchy, tables, attachments, and metadata. It can run as a Python library, an API service, or a container.

It matters because downstream search, review, analytics, and retrieval systems need more than a blob of text. They need to know which text is a heading, which cells form a table, which attachment belongs to the source, and how every output fragment can be traced back to the input.

The parser route must match the document's actual representation and the downstream output contract. OCR, table analysis, attachment recursion, and hierarchy extraction are explicit quality and capacity decisions, not universal defaults.

Reading order

Content

Textual lines and annotations preserve what was read and enough formatting context to render or inspect it.

Hierarchy

Structure

Line types and hierarchy levels are constructed into a tree or a linear output, depending on the requested representation.

Beyond text

Objects

Tables, attachments, and metadata remain separate objects so downstream code does not have to reconstruct them from flattened prose.

Follow the documented parsing pipeline

Dedoc separates format handling from structure construction. That separation is useful in production because a conversion failure, reader failure, structure error, and output contract violation need different retries and ownership.

1
Converter
Convert when needed
Normalize formats that can be converted to a supported representation, such as older office documents to their newer container formats.
2
Reader
Read the document
Produce an intermediate document with lines, annotations, tables, attachments, and metadata. PDF routing may use native text, image recognition, or page-level detection.
3
Classifier and rules
Extract structure
Assign line types and hierarchy levels using a domain-specific extractor or the default structure path.
4
Contract
Construct output
Build a tree or linear representation, serialize it in the requested format, and retain source lineage for review.

The official workflow documentation describes converters, readers, structure extractors, and structure constructors as separate stages.

Size the worker fleet from page work, not file count

Two documents of equal byte size can have very different service times. A born-digital PDF can reuse its text layer, while a scan may require page rendering, orientation, recognition, and table analysis. Averages hide the malformed and OCR-heavy tail that sets queue age and memory pressure.

The capacity lab exposes five independent demand drivers:

documents arriving in the target processing window;
pages per document;
the share of pages that require OCR;
the share requiring table analysis;
concurrent workers and their per-process memory envelope.

Loading the processing envelope...

Use the result to design a benchmark matrix, not as a throughput guarantee. Measure native PDF, scanned PDF, mixed-page PDF, office files, tables, attachments, malformed inputs, and the largest allowed document separately. Track p50, p95, and p99 service time as well as peak resident memory.

Route each format to the correct reader and parameters

Reader selection changes both quality and cost. Dedoc's PDF parameters distinguish native textual layers from image-only pages, support automatic or page-level detection, and can enable expensive table analysis only when the output contract requires it.

Loading document routes...

Important routing decisions

Use pdf_with_text_layer=auto_tabby when the system should detect the PDF path and reuse a trustworthy text layer when present.
Enable each_page_textual_layer_detection=true for mixed binders where the first pages do not represent the whole document.
Force the image path only when the input contract says the PDF is scanned or the native encoding is unusable; OCR adds latency and recognition error.
Disable table analysis only when losing row and cell relationships is acceptable.
Use native office readers when they can preserve paragraphs, tables, annotations, metadata, and attachments without rendering pages.

The current API parameter reference documents PDF text-layer detection, table analysis, page slices, orientation, attachments, structure types, and return formats.

Define the parsed output as a versioned contract

A parser returning HTTP 200 does not prove the output is usable. Validate the fields the downstream consumer needs and attach quality evidence to each parsed artifact.

Text / pages / objects

Coverage

What proportion of expected content was recovered?

Types / levels / order

Structure

Do headings, lists, and reading order preserve the document meaning?

Rows / cells / spans

Tables

Are table relationships intact enough for the consumer?

Source / page / route

Lineage

Can a reviewer trace every important output to its input and parser configuration?

Route from an inspected input contract

Choose explicit Dedoc parameters

The routing function keeps policy outside the HTTP client. In a real service, detect the media type from content, constrain the allowed formats, and record the selected reader and parameters with the result.

Validate before publishing downstream

Check the required parsed-output shape

Expand the validator with page coverage, table expectations, attachment inventory, confidence thresholds, and a schema version. Send uncertain documents to review or a fallback parser instead of silently publishing partial structure.

Wrap the parser in a bounded production service

Dedoc supplies parsing behavior; the surrounding system must supply isolation, admission control, lifecycle state, storage, retries, and observability.

Production document processing path

Keep untrusted bytes, parser work, validation, and published output in separate operational stages.

Ingress

Quarantine input

Verify media type, size, page and attachment limits, malware policy, tenant identity, and an idempotency key before parsing.

Control plane

Route and enqueue

Inspect the document profile, select parameters, assign a deadline and retry budget, then place the job on a bounded queue.

Worker

Parse in isolation

Run converters, native readers, OCR, and table analysis with CPU, memory, filesystem, network, and execution-time limits.

Output gate

Validate and publish

Check coverage and schema, store immutable input and output references, then publish a versioned result or a review task.

Model job state explicitly

accepted: input passed admission checks and has an immutable source reference;
queued: a route, deadline, and retry budget are recorded;
running: one worker lease owns the attempt;
review_required: output exists but misses a quality or confidence gate;
succeeded: the versioned output contract is published exactly once;
failed: the final reason is classified as input, conversion, reader, resource, or contract failure.

Retries should create new attempts under the same job identity. Publishing must be idempotent because a worker can finish after its lease or client connection is lost.

Secure and operate untrusted document processing

Contain the input

cap bytes, pages, archive depth, attachment count, extracted size, and total work;
isolate converters and OCR processes with no ambient cloud credentials;
disable unnecessary network access and use short-lived, least-privilege object URLs;
reject path traversal, archive expansion, unsupported media, and recursive attachment patterns before they consume the fleet;
encrypt source and result objects and apply tenant-scoped retention and deletion.

Observe the contract

queue depth and oldest-job age by tenant, format, and route;
parser latency, timeout, retry, memory, and worker-recycle rates;
native-versus-OCR routing, page-level fallback, and table-analysis share;
coverage, structure, table, and review rates by document population;
failures by converter, reader, parameter set, and release version.

Release deliberately

maintain a versioned golden corpus with native, scanned, mixed, rotated, table-heavy, attachment-heavy, malformed, and adversarial files;
compare structural output, not only extracted character count;
canary parser, OCR, model, and parameter changes by document population;
keep the prior route and output schema available until downstream consumers verify compatibility;
replay stored inputs into a shadow pipeline before changing the default route.

No quiz questions available

Could not load questions file