Autonomous Systems

Define the autonomy contract before choosing models

The system is not "autonomous everywhere." It is autonomous only inside a documented operational design domain: the conditions, locations, speeds, sensor health, connectivity, task types, and human support assumptions under which its safety and performance claims apply.

Write these boundaries as testable statements

• Entry condition: all required sensors, calibration, maps, compute deadlines, actuators, and fallback paths are healthy.
• Continued-operation condition: uncertainty, command age, control error, and environmental conditions stay inside explicit limits.
• Exit trigger: any violated health, timing, confidence, or physical constraint revokes some or all primary authority.
• Fallback guarantee: the fallback can detect, take control, and reach its safe state before the modeled hazard window closes.
• Human role: the system states whether a person supervises, authorizes, remotely assists, or only responds after a safe stop.

Trace uncertainty from perception to control authority

Perception does not produce truth; it produces evidence. State estimation combines that evidence with prior state and motion models, while planning turns the resulting belief into a candidate action. The controller should receive both the command and the envelope that makes it valid.

Use the workbench to change the scenario, sensor reliability, evidence path, and planning posture. It recomputes effective confidence, state uncertainty, stopping distance, clearance, and the amount of authority granted for the next control interval.

The executable example implements the same decision contract without a robotics framework. Its result is inspectable and deterministic, making it suitable for unit tests and incident reconstruction.

Keep perception and state estimation separate

Perception extracts task-relevant evidence from measurements. State estimation combines evidence over time into a belief about the system and environment. Keeping the contracts separate makes missing data, disagreement, age, and uncertainty visible to the planner.

Design the state contract for downstream decisions

• Include timestamps and maximum acceptable age for every task-relevant field.
• Separate unknown, not observed, and observed absent; they imply different actions.
• Carry covariance, confidence intervals, ensemble disagreement, or another calibrated uncertainty representation.
• Record the sensor and model versions that contributed to each state estimate.
• Expose degraded modes explicitly instead of silently filling missing evidence with defaults.
• Evaluate by operating regime and decision consequence, not only aggregate detection accuracy.

For foundations on drift and feedback, review continual learning and human-in-the-loop ML.

Separate prediction, planning, and control

A prediction model estimates how agents or the environment may evolve. A planner compares candidate actions under those predictions. A controller tracks the selected reference against real physical response. Combining all three behind one opaque output makes it difficult to enforce constraints or identify which assumption failed.

Budget distance and time end to end

For straight-line braking, a useful first estimate is:

required distance = reaction distance + braking distance + clearance + state uncertainty

where reaction distance = speed x end-to-end delay and idealized braking distance = speed^2 / (2 x deceleration). Production models must also account for actuator dynamics, slope, surface, payload, controller error, and uncertainty in the obstacle prediction.

1. Generate feasible candidates: obey geometry, dynamics, route, resource, and policy constraints before scoring comfort or progress.
2. Evaluate uncertainty: penalize candidates whose safety depends on poorly observed state or predictions beyond a credible horizon.
3. Apply hard boundaries: reject commands that exceed speed, force, acceleration, clearance, freshness, or operating-domain limits.
4. Track measured response: compare the commanded and realized state so actuator lag or control instability becomes a runtime fault.
5. Replan from evidence: execute the smallest useful action and correct from the next synchronized observation.

Make the fallback faster and simpler than the hazard

A fallback is not useful merely because it exists in a diagram. The system must detect the violation, revoke primary authority, transfer control, and reach a safe state before the hazard becomes unavoidable. It must also avoid the same dependencies and assumptions that caused the primary path to fail.

Use the control room to inject a sensor, localization, actuator, or compute fault. Change detection latency, monitor independence, fallback behavior, and evaluation evidence; the timing race and release gates update together.

The code example keeps timing, residual risk, and evidence coverage as separate gates. A fast fallback cannot compensate for missing evaluation coverage, and a strong test plan cannot compensate for a response that arrives too late.

Reduce common-cause failure

• Use a simpler monitor whose logic can be reviewed and tested independently from the primary learned stack.
• Observe sensor health, time freshness, uncertainty, command limits, actuator response, and operating-domain membership.
• Keep the fallback's compute, power, communication, state, and actuator assumptions explicit.
• Test handover under partial failure; a healthy fallback that cannot gain authority is not a fallback.
• Make remote assistance bounded: advice must not bypass onboard safety constraints or become an unmodeled control path.

Build evidence as a ladder, not one benchmark

Simulation makes rare, dangerous, and combinatorial scenarios repeatable, but a simulator is another model with its own gaps. Use several evidence sources whose weaknesses differ, and trace each result to the safety or performance claim it supports.

1. 1

   Unit and replay

   Test components against contracts

   Check perception calibration, state update invariants, planner feasibility, command limits, and monitor logic with deterministic fixtures and protected incident traces.

2. 2

   Simulation

   Explore scenario boundaries

   Vary geometry, agents, weather, lighting, sensor noise, timing, maps, and policy while measuring both expected behavior and fallback transitions.

3. 3

   Hardware in loop

   Preserve physical timing

   Run production compute, buses, sensors, controllers, and actuators so queueing, synchronization, saturation, and transfer latency remain visible.

4. 4

   Monitored operation

   Expand the domain gradually

   Start inside a narrow envelope, compare field traces with predictions, maintain rollback and stop authority, and widen only when evidence supports the new claim.

Evaluate the complete closed loop

• Perception: calibration, missed and false evidence, age, regime coverage, and uncertainty under corruption or sensor loss.
• State estimation: pose and track error over time, covariance calibration, recovery after disagreement, and behavior during missing observations.
• Planning: feasibility, clearance, rule compliance, progress, stability, and sensitivity to plausible state or prediction errors.
• Control: tracking error, overshoot, saturation, actuator latency, disturbance rejection, and safe transfer of authority.
• Runtime assurance: detection coverage, false intervention rate, response-time distribution, common-cause exposure, and safe-state completion.
• System outcome: scenario-level success, near misses, harmful events, human workload, recovery, and behavior at operating-domain boundaries.

Operate every decision as a reconstructable trace

An autonomy incident often appears at the actuator but begins earlier: an old sensor frame, a calibration change, a state-estimation jump, a planner assumption, a missed runtime signal, or delayed control transfer. Telemetry must preserve causal order across the full loop.

Run the incident loop in a fixed order

1. Contain: revoke unsafe authority and move affected systems toward the defined safe state.
2. Preserve: freeze synchronized traces, software and model versions, maps, calibration, configuration, and operator actions.
3. Localize: determine whether the first invalid assumption occurred in sensing, estimation, prediction, planning, monitoring, control, or operations.
4. Repair: change the narrowest responsible contract, dataset, component, monitor, or operating-domain boundary.
5. Requalify: replay the incident class, neighboring scenarios, transfer timing, and regression suite before restoring capability.

Avoid recurring production failures

• Aggregate accuracy hides dangerous regimes: average metrics conceal rare conditions where uncertainty and harm are highest.
• Freshness is treated as metadata: a correct but stale state can produce a physically unsafe command.
• Fallback shares the failed dependency: the backup cannot take control because it uses the same sensor, compute, power, or state estimate.
• Simulation success becomes a universal claim: model gaps are ignored when transferring to hardware or a wider operating domain.
• Remote help becomes hidden control: human guidance bypasses the onboard authority gate or lacks a tested timeout path.
• Logs cannot reconstruct causality: unsynchronized telemetry prevents engineers from locating the first invalid assumption.

Primary references and standards context

• NIST AI Risk Management Framework 1.0 defines a lifecycle for mapping, measuring, managing, and governing AI risk, including testing, uncertainty, monitoring, and safe failure.
• ISO 26262 road-vehicle functional safety addresses hazards caused by malfunctioning behavior of safety-related electrical and electronic systems across the lifecycle.
• ISO 21448:2022 addresses safety of the intended functionality, including hazards from specification or performance insufficiencies where situational awareness depends on complex sensors and processing.
• A Verification Framework for Runtime Assurance of Autonomous UAS presents NASA research on verifying runtime-assurance architectures for autonomous aircraft systems.
• CARLA: An Open Urban Driving Simulator describes an open simulator designed for autonomous-driving research and controlled scenario evaluation.
• Waymo's Safety Methodologies and Safety Readiness Determinations documents one operator's layered approach across hardware, autonomous-driving behavior, operations, and governance.
• A New Approach to Linear Filtering and Prediction Problems introduces the recursive state-estimation method now known as the Kalman filter.

Standards are application-specific and evolve. Treat these sources as starting points for qualified safety and compliance work, not as a universal certification checklist.