Edge ML Deployment
Master edge ML deployment: mobile and IoT optimization, TensorRT, ONNX, model compression, and offline inference systems.
What is Edge ML Deployment?
Edge ML deployment means running a trained model inside or near the product that uses its prediction: a phone, camera, vehicle computer, industrial gateway, or microcontroller. The device can respond without a cloud round trip, keep sensitive inputs local, and continue through a network outage.
The core invariant is simple: the exact signed model, runtime, and preprocessing path must work inside the measured limits of every supported device cohort. A model that is accurate in a notebook is not deployable until it also loads, meets its deadline, stays within memory and storage limits, survives sustained use, degrades safely, and can be rolled back.
Edge deployment is therefore a release-engineering problem as much as a model-optimization problem.
Storage input
Artifact size
The model file and runtime delta consume install space; they do not equal peak RAM.
Runtime input
Peak working set
Weights, tensors, delegates, preprocessing, and the host app compete for memory.
Deadline input
Warm p95
Measure repeated inference on the real device, not one fast desktop invocation.
Recovery input
Known-good model
A release is not reversible unless a compatible fallback remains usable offline.
Treat the device as a measured contract
An edge target is not just a CPU label. Freeze a support matrix that identifies:
- Hardware cohort: processor, accelerator, memory class, storage class, sensors, and thermal enclosure
- Software cohort: OS, drivers, runtime version, execution provider or delegate, and application version
- Model contract: artifact hash, format, operators, tensor shapes, preprocessing, outputs, and labels
- Product limits: cold-load time, warm p50 and p95 latency, peak memory, install size, sustained energy and heat, and offline behavior
- Quality gates: aggregate and slice-level task metrics compared with the accepted model
If any field changes, re-run compatibility and performance evidence for the affected cohort. Accelerator support is model- and device-specific; a graph can be partitioned between accelerated and fallback paths, changing both latency and memory.
1 Target
Freeze the contract
Name supported cohorts, task-quality floors, latency deadlines, resource ceilings, fallback behavior, and release owners.
2 Artifact
Export and inspect
Convert the model, validate signatures and operators, sign the bundle, and bind it to compatible runtime versions.
3 Evidence
Profile the real path
Measure load, preprocessing, warm inference, postprocessing, peak memory, energy, heat, and task quality on representative devices.
4 Control
Release by cohort
Canary the exact bundle, enforce independent gates, retain the known-good version, and stop promotion automatically on failure.
5 Lifecycle
Operate offline
Keep inference and rollback bounded without connectivity; queue versioned telemetry and reconcile it when the network returns.
Separate storage, memory, and latency
The device-envelope lab uses three transparent checks:
- Guarded memory ceiling = memory available to the process × (1 − reserve percentage)
- Runtime memory demand = measured host-app baseline + measured model peak working set
- Update storage demand = candidate install bundle + retained known-good bundle
- Latency margin = product deadline − measured warm p95 latency
These quantities must stay separate. A 40 MiB model may need much more than 40 MiB at runtime because tensors, copies, delegates, preprocessing, and the host application allocate memory. Quantization may reduce weight storage, but its latency, memory, energy, and quality effects depend on the graph and target backend.
Do not estimate accuracy, power, or thermal behavior from file size. Measure task quality on representative data and profile sustained workloads on each supported cohort.
Choose a runtime after proving the execution path
Android, iOS, desktop, IoT
LiteRT
Google's on-device runtime supports converted models across CPU, GPU, and NPU backends. Inspect supported operators and benchmark the exact CompiledModel or interpreter path on target devices.
Cross-framework mobile
ONNX Runtime Mobile
Start with a complete runtime while validating the model, then consider a reduced operator build. Check execution-provider partitioning because unsupported operators can push graph regions onto a slower fallback.
Apple platforms
Core ML
Core ML can use the CPU, GPU, and Neural Engine. Treat compute-unit selection and model specialization as device-specific behavior and profile the integrated application.
NVIDIA edge GPUs
TensorRT
TensorRT compiles a graph into an engine for NVIDIA hardware. Serialized engines are not portable across platforms by default, and untrusted engines must be treated like untrusted executable code.
Runtime selection questions
- Does the runtime support every operator, type, dynamic shape, and custom kernel in the candidate?
- Which graph regions execute on the accelerator, and which fall back to the CPU?
- What runtime and operator set are added to the application binary?
- Can the artifact load on every OS, driver, and hardware cohort in the support matrix?
- Is first-run compilation or specialization part of the user-visible latency and storage budget?
- Can the fleet restore a compatible known-good version after an update?
Optimize against evidence, not a compression promise
Representation
Quantize
Map weights and possibly activations to lower-precision types. Use representative calibration data when required, verify operator support, and test quality by important slice.
Architecture
Change the graph
Replace unsupported or expensive operations, reduce input resolution, use a mobile-oriented architecture, or distill into a smaller student. Re-train and re-evaluate the task contract.
Packaging
Reduce the runtime
Ship only required operators when the runtime supports a reduced build. Measure the final application package rather than quoting the model artifact alone.
Whole request
Tune the pipeline
Reuse buffers, bound concurrency, avoid input copies, warm expected paths, and profile preprocessing plus postprocessing. Model invocation is only one part of latency.
For every candidate, compare the same evaluation dataset, device cohort, runtime configuration, thermal state, concurrency, and input shape. Record distributions rather than one average.
Build a reversible device path
A prediction is only one path through the edge system
The release controller and known-good model remain independent of the candidate serving path.
Versioned schema
Sensor or app input
Validates shape, units, sampling rate, permissions, and preprocessing version before invoking the model.
Bounded execution
Local inference runtime
Loads the signed model, selects a supported backend, enforces concurrency and deadlines, and emits typed failures.
Safety boundary
Product decision
Interprets outputs with thresholds, abstention, business rules, and a deterministic degraded mode.
Separate authority
Fleet control plane
Stages immutable versions, observes cohort gates, stops promotion, and restores the known-good model.
Failure behavior must be explicit
- Model load fails: reject the candidate and use the local known-good path.
- Accelerator rejects an operator: do not silently accept an unmeasured CPU fallback.
- Deadline expires: cancel or discard late work and use the product's bounded fallback.
- Memory pressure rises: shed inference work before the host application becomes unstable.
- Output is invalid or low-confidence: abstain or route to an approved deterministic behavior.
- Connectivity disappears: continue locally where allowed and bound telemetry retention.
Promote immutable models through bounded cohorts
A fleet rollout is a sequence of independent release decisions, not a percentage slider that eventually reaches 100%. Every stage must preserve its own evidence:
- Start with hardware canaries that span the support matrix.
- Confirm artifact integrity, model load, backend placement, memory, warm latency, thermal behavior, energy, and task quality.
- Expand only when every required gate passes for the current cohort.
- Stop automatically when telemetry disappears or a critical signal crosses its declared limit.
- Roll back affected devices to an already-tested compatible artifact.
- Diagnose by model, app, runtime, OS, driver, hardware, region, and input cohort before rebuilding.
Operate the model as a long-lived product dependency
Before release
- Sign the candidate and bind its hash to metadata, preprocessing, labels, runtime, and compatibility policy.
- Test clean install, upgrade, interrupted update, corrupt download, low-storage behavior, and rollback.
- Compare the candidate with the accepted model on global, slice-level, and safety-critical quality gates.
- Soak representative devices long enough to expose throttling, memory growth, and battery impact.
During release
- Track eligible, download-started, verified, loaded, invoked, healthy, rolled-back, and expired devices separately.
- Segment failures by the complete cohort identity; fleet-wide averages hide unsupported tails.
- Keep telemetry bounded, versioned, privacy-reviewed, and safe to retry after reconnect.
- Preserve a release kill switch outside the model inference path.
After release
- Watch input drift, output distributions, abstention, user corrections, and downstream outcomes where policy permits.
- Rehearse rollback and certificate or signing-key rotation before an incident.
- Retire old artifacts only after the fleet no longer needs them for recovery.
- Re-run the device contract when the model, runtime, app, OS, driver, or hardware support matrix changes.