Design a Computer Vision Pipeline
Design a computer vision pipeline: image processing, model serving, and batch inference at scale.
What is a computer vision inference system?
A computer vision inference system turns an image or video frame into a bounded prediction, such as an object box, class, count, or OCR result. The model is only one stage. A production system must accept media safely, apply the same preprocessing used in training, select a versioned model, return or queue a result, and collect trustworthy feedback for the next release.
For this practice problem, design a multi-camera detection platform for 120 live camera streams plus asynchronous image jobs. It samples 8 frames per second per stream, returns a live event within 120 ms p95, and prioritizes correct detection of people and vehicles. The core invariant is: every prediction and action is reproducible from the input reference, preprocessing version, model release, threshold, and policy that produced it.
- Prerequisites: Review ML Fundamentals for labels, evaluation, and supervised learning, and Data Pipelines for reliable ingestion and lineage.
- First-version boundary: Detect a fixed object taxonomy from consented camera feeds and uploaded images. Do not make identity decisions, train from unreviewed production predictions, or treat an alert as proof of a real-world event.
Serving
Protect the live deadline
Validate, decode, preprocess, infer, apply policy, and record the result within one shared latency budget. Training and reporting stay off this path.
Parity
Keep one feature meaning
The resize, color conversion, frame sampling, label taxonomy, and confidence threshold must match the release manifest.
Reliability
Fail with a bounded action
Under pressure, sample fewer frames or queue non-live work. Never silently bypass validation, policy, or release evidence.
Define the decision before choosing a detector
Each accepted frame produces a detection set, confidence values, a disposition, and an immutable release-manifest ID. An event policy may use several frames and camera context before creating a human-visible alert. A detection score is evidence, not the final safety or business decision.
Requirements that change the design
Functional requirements
- Accept authenticated camera streams and image uploads, validate media, and deduplicate retrying requests with an idempotency key.
- Detect people and vehicles; return boxes, classes, confidence, timestamps, and model lineage.
- Support a live route for incident workflows and a durable batch route for retrospective search or reporting.
- Preserve sampled frames and predictions under documented retention, deletion, and tenant-access rules.
- Capture reviewed outcomes and verified events as delayed feedback without treating the model's own output as a label.
Non-functional requirements
- Meet a 120 ms p95 deadline for an eligible live frame and return a durable job receipt immediately for batch work.
- Keep the critical people-and-vehicle recall floor within 2 percentage points of the approved production release on every required camera slice.
- Survive one serving-zone loss with explicit capacity shedding and observable fallback use.
- Encrypt media and feature artifacts, minimize retention, and make every result traceable to the requesting tenant and release manifest.
Ask the questions that expose hidden constraints
- What action follows a detection? A dashboard annotation can tolerate a delay; an access-control or incident workflow needs a policy owner, human escalation, and stricter false-positive limits.
- Which slices are critical? Night, rain, camera angle, region, scene density, device type, and class frequency can all hide a regression behind an improved average score.
- Who supplies ground truth? Verified incidents, sampled human review, and curated labels have different delay, cost, and reliability. Store their source and decision version.
- What may degrade? Sampling rate, model tier, and batch start time may change. Authentication, input validation, tenant isolation, and release lineage may not.
Begin with one camera class, a fixed taxonomy, one approved detector tier, and an explicit alert policy. Adding face recognition, unbounded visual search, or automatic enforcement changes the privacy, error-cost, and governance problem rather than merely increasing model scale.
Inference placement for real-time on-device video analysis
You're building an app that analyzes a live camera feed on phones and smart cameras (detecting objects/events frame-by-frame) and reacts instantly in the UI. Where should the vision model run?
Constraint: Reactions must be real-time (sub-frame latency, no round-trip stalls), and raw video must NOT leave the device — both for privacy and to avoid saturating uplink bandwidth.