Computer Vision Systems Design

Start with the decision, not the model name

Before choosing an architecture, define what one result means and how late is too late. A prediction contract states the input unit, output schema, eligible population, deadline, quality floor, abstention behavior, and owner of uncertain results.

Write the contract in product language. A loading-dock alert might require an event within 120 ms, critical-event recall above a validated floor, no more than a bounded number of human reviews, and a safe degraded mode when one serving zone fails.

Convert camera demand into a serving envelope

A video stream is not one request. If N streams are sampled at F frames per second, the inference path receives N x F frames each second. Resolution, scene density, model profile, preprocessing, and tracking add work per frame.

Plan against sustained safe capacity rather than benchmark peak capacity:

• Admitted frame rate: streams x sampled FPS.
• Effective demand: admitted frames multiplied by a measured complexity factor for the workload.
• Safe capacity: live replica throughput multiplied by the steady-state utilization target.
• Required replicas: effective demand divided by safe per-replica capacity, rounded up.
• Failure capacity: repeat the calculation after losing a zone or accelerator pool.

The numbers in the lab are planning assumptions, not hardware promises. Replace them with measurements from the exact model artifact, runtime, input distribution, and device before procurement or release.

Bound queues to protect freshness

An admission policy decides what enters the expensive path. For live perception, processing every decoded frame can be worse than intentionally dropping stale frames: once arrival rate exceeds service rate, an unbounded first-in-first-out queue keeps returning older and older observations.

• Bound every queue by count, bytes, or age.
• Prefer the newest frame for freshness-sensitive streams; preserve all inputs only when the product contract requires complete batch processing.
• Apply per-source quotas so one noisy camera cannot occupy every batch slot.
• Carry source ID, capture time, sequence number, and transform version with the tensor.
• Measure dropped, expired, retried, and processed frames separately.

Dropping is a product decision. Validate that the remaining sample rate still captures the shortest event and preserves tracking continuity. For audit or safety workflows, write the original media to durable storage before sampling the live inference path.

Make preprocessing a versioned contract

Preprocessing converts decoded media into the tensor shape and numeric representation expected by the model. A silent mismatch can destroy production quality without changing model weights.

The contract should pin:

• decoder and codec behavior, including rotation and color range;
• color space and channel order, such as RGB versus BGR;
• resize policy, aspect-ratio handling, padding, and crop coordinates;
• normalization constants, numeric precision, and tensor layout;
• model and transform versions attached to every result.

Test the training and serving implementations with golden images and expected tensors. Record hashes or summary statistics at the boundary so an incident can distinguish camera drift from transform drift.

Treat scores as evidence, not decisions

A detector emits scores and candidate boxes. The product needs a decision policy that maps those scores to actions, review, abstention, or rejection. The threshold changes recall, false alerts, queue load, and error cost together.

• Lower thresholds usually recover more true events but send more false alerts downstream.
• Higher thresholds reduce review load but hide more true events.
• A threshold calibrated on one camera mix may fail under darkness, blur, weather, or a new installation angle.
• Separate high-risk actions from low-risk metadata; uncertain high-risk events should abstain or route to review.

Choose the operating point from labeled, production-like data and an explicit capacity budget. Average precision alone cannot tell whether the resulting queue or miss rate is acceptable.

Post-process detections without erasing evidence

Intersection over union (IoU) measures how much two boxes overlap. Non-max suppression (NMS) keeps a high-scoring box and removes lower-scoring boxes of the same class when their IoU exceeds a threshold. It reduces duplicate detections, but an aggressive threshold can merge nearby real objects.

Apply confidence policy before NMS, keep suppression class-aware unless the product requires cross-class competition, and measure post-processing latency. Preserve the raw candidate set or enough sampled traces to investigate threshold and NMS changes later.

Tracking adds another stateful contract. Define when a track starts, how long it can miss observations, when identity is retired, and how events remain idempotent across retries. Partition track state by camera or scene so related frames reach the same worker, and checkpoint only the state needed for the recovery objective.

Build the production path around deadlines

1. 1

   Profile

   Measure each stage

   Record decode, transform, queue, transfer, inference, post-processing, tracking, and network latency at p50, p95, and p99 on representative media.

2. 2

   Serve

   Schedule bounded work

   Use per-source admission, deadline-aware micro-batches, warm model replicas, and backpressure before a queue becomes stale.

3. 3

   Protect

   Degrade deliberately

   Reduce sample rate, use a validated smaller profile, narrow eligible traffic, or route to review while preserving the minimum quality contract.

4. 4

   Operate

   Release with evidence

   Shadow and canary new models or transforms, compare named slices, and retain a compatible rollback artifact and policy version.

Deadline-aware batching waits only while the oldest admitted frame has enough slack. A larger batch may improve accelerator throughput but violate tail latency. Export queue age and batch-fill time so low GPU use is not mistaken for spare end-to-end capacity.

Evaluate the system, not only the checkpoint

Create an untouched test set that mirrors deployment cameras and hard conditions. Add named slices for low light, motion blur, occlusion, small objects, crowded scenes, device families, and any safety- or fairness-sensitive population. For video, evaluate event detection delay, track fragmentation, and identity switches in addition to frame metrics.

Release a model, preprocessing graph, post-processing policy, and threshold as one versioned bundle. A canary is acceptable only when automatic rollback signals cover quality proxies, latency, errors, and queue age; delayed labels still require a later outcome review.

Design failure modes before they happen

Retain only the media and metadata needed for debugging, evaluation, and legal obligations. Encrypt transport and storage, restrict access by purpose, redact or crop where possible, and make retention deletion verifiable. Camera identity and embeddings can be sensitive even when raw frames are not kept.

Use a production review checklist

• Contract: input, output, deadline, quality floor, abstention, and review ownership are explicit.
• Data: deployment cameras, hard conditions, rare events, and consent or retention rules are represented.
• Capacity: stage profiles, headroom, batching delay, and the largest failure domain are measured.
• Correctness: preprocessing parity, confidence policy, NMS, tracking, and idempotent events are versioned and tested.
• Reliability: queues are bounded, degraded modes are quality-validated, and rollback is compatible and rehearsed.
• Observability: capture-to-decision traces join source, frame, transform, model, policy, and result versions.
• Release: shadow and canary evidence covers slices, latency, queue age, errors, and downstream outcomes.
• Privacy: access, retention, deletion, encryption, and incident handling match the media risk.

The most effective optimization is often reducing unnecessary admitted work. Sample with evidence, route simple scenes to cheaper profiles, batch only within deadline slack, and keep data transfers and post-processing visible in the same capacity model as inference.