Federated Learning Systems

One training round has four owners

The system advances through rounds. Each stage needs a durable identity, deadline, policy version, and observable outcome so a retry cannot silently apply an update twice or release an unevaluated model.

1. 1

   Coordinator

   Select an eligible cohort

   Filter by consent, software and model version, power and network policy, geography, availability, rate limits, and sampling goals.

2. 2

   Client

   Train within a client contract

   Verify the model and task, load local examples, run bounded work, clip the contribution, and report only approved metrics and update material.

3. 3

   Aggregator

   Recover a valid aggregate

   Authenticate participants, reject incompatible schemas, tolerate bounded dropout, and unmask only after the secure-aggregation survivor threshold is met.

4. 4

   Model owner

   Evaluate before release

   Check global and cohort quality, privacy accounting, integrity canaries, fairness, rollback criteria, and artifact lineage before canary deployment.

Choose cross-device or cross-silo federation deliberately

Both modes move training toward data, but their identity, reliability, governance, and scale assumptions are different.

Write the federation contract before choosing an optimizer

• Unit of contribution: one person, device, account, organization, or bounded window of records
• Eligibility: consent, jurisdiction, model version, data freshness, device health, and participation frequency
• Round completion: deadline, minimum survivors, maximum retries, late-update handling, and idempotency key
• Release unit: per-round update, periodic checkpoint, personalized model, or site-specific artifact
• Ownership: who approves client code, privacy parameters, model promotion, incident response, and deletion requests

Size the round around survivors, not invitations

Inviting 500 devices does not mean 500 updates will be aggregated. Some are offline, some reject the task, some fail local training, and some disappear during cryptographic unmasking.

Keep four quantities separate

• accepted = invitations x observed availability
• expected survivors = accepted x (1 - dropout rate)
• quorum margin = expected survivors - required survivors
• round transfer = accepted downloads + survivor uploads

The lab models a compressed update as 35% of model size and uses simple link assumptions. In production, estimate distributions rather than averages: p50, p95, and deadline miss rates by device class, region, software version, and network type.

Client selection determines whose objective is optimized

Availability is rarely random. Devices that are charging on fast networks at night, organizations with larger clusters, or clients that tolerate repeated participation can dominate the training population.

Measure selection as a funnel

1. Count eligible clients by relevant cohort before sampling.
2. Record invitations, acceptances, successful downloads, completed local jobs, survivors, and accepted updates.
3. Compare each stage with the target population, not only with the previous stage.
4. Limit repeated participation so a small reliable subset cannot silently become the training distribution.
5. Reweight only from documented inclusion probabilities; do not invent weights to force a preferred outcome.

Treat these as release risks

• A language, region, device tier, hospital, or customer segment is rarely eligible.
• High-value but resource-constrained clients time out more often than the average.
• One organization supplies most examples or claims an unbounded sample count.
• Participation incentives change behavior or encourage fabricated clients.
• Consent withdrawal and deletion requirements cannot be traced to release lineage.

Aggregation is an optimization and trust decision

Federated averaging weights bounded client updates and applies their aggregate to the global model. Its behavior depends on heterogeneous data, unequal local work, stale clients, clipping, and the trustworthiness of reported weights.

Bound influence before averaging

• Validate model schema, tensor shape, finite values, round ID, and base-model version.
• Clip each contribution to a declared norm or coordinate bound before aggregation.
• Cap sample-count weights or derive them from trusted metadata instead of client claims.
• Reject late and duplicate updates with round-scoped idempotency keys.
• Record clipping and rejection rates by cohort; a rising rate can indicate drift, bugs, or attack traffic.

Layer controls because they protect different boundaries

Resolve control interactions before launch

• Secure aggregation hides individual updates, which can conflict with server-side anomaly inspection.
• Differential privacy noise may reduce sensitivity to small regressions and minority-slice failures.
• A high survivor threshold improves the anonymity set but increases abort risk under dropout.
• Robust aggregation assumptions can fail when honest updates are naturally diverse.
• Trusted execution and attestation introduce new hardware, key-management, and supply-chain dependencies.

Design the coordinator as a recoverable state machine

Do not represent a round as one long request. Persist transitions so coordinator restarts, duplicate messages, and late clients have deterministic outcomes.

Make every retry idempotent

• Give the round, participant lease, update, aggregate, evaluation bundle, and release unique stable IDs.
• Bind updates to the exact base-model hash and reject stale or future versions.
• Keep accepted, rejected, duplicate, late, dropped, and unmasked states distinct.
• Never carry masked material or privacy-accountant state into a replacement round without protocol approval.
• Record whether a failed release consumed privacy budget; the answer depends on what was exposed, not whether promotion succeeded.

Expect failure at the tails

Client failures

• Insufficient storage, memory, battery, accelerator support, or local examples
• Interrupted download, training, checkpoint, or upload
• Corrupt local state, incompatible schema, non-finite update, or wrong model version
• Clock skew, duplicated work, expired lease, or update after deadline

Coordinator and protocol failures

• Too few survivors to unmask a secure aggregate
• Key setup or dropout-recovery failure
• Aggregator restart after accepting some contributions
• Evaluation service unavailable after a privacy-consuming release
• Model registry accepts an artifact before every gate is durable

Recovery rules

1. Close the failed round with a typed terminal reason.
2. Preserve immutable evidence needed for audit and incident analysis.
3. Start a new round ID and resample clients under the current eligibility policy.
4. Reuse no client update unless the protocol, consent, and privacy accounting explicitly permit it.
5. Alert on repeated aborts by cohort because reliability failure can create selection bias.

Evaluate the model, fleet, privacy, and integrity together

One global validation score cannot prove that a federated release is useful or safe.

Operate federation as a long-lived product boundary

Before each round

• Freeze and sign the model, client code, manifest, eligibility policy, privacy parameters, and evaluation contract.
• Confirm remaining privacy budget, minimum cohort size, regional capacity, rollback target, and on-call ownership.
• Test key distribution, secure-aggregation dropout recovery, duplicate handling, and coordinator failover.

During each round

• Monitor the participation funnel, deadline distribution, survivor margin, bytes transferred, energy policy, and abort reasons.
• Track update schema failures, clipping, bounded weight, aggregate direction, and integrity canaries without exposing prohibited individual data.
• Apply backpressure to invitations and aggregation rather than allowing unbounded in-flight rounds.

Before release

• Require immutable global and cohort evaluation, privacy accounting, integrity probes, lineage, and policy approval.
• Compare with the previous global model and local-only baselines under the same evaluation protocol.
• Use shadow and canary rollout with client-version compatibility checks and explicit abort signals.

After release

• Monitor serving quality and participation drift by cohort, not only aggregate traffic.
• Retain only protocol-approved artifacts for the declared period and test deletion workflows.
• Rehearse rollback, key compromise, poisoned-round investigation, accountant correction, and coordinator-region loss.

Use federation only when its boundary solves the real constraint

Federation adds orchestration, cryptography, unreliable participants, biased sampling, privacy accounting, and distributed evaluation. Prefer a simpler design when data can be governed and moved safely or when the client population is too small to support the intended guarantee.

Strong fit

• Raw data cannot or should not leave devices or organizations, but a shared objective is legitimate.
• Enough eligible participants can clear quorum without excluding important populations.
• Client compute, networking, consent, and software update channels can support repeated rounds.
• The organization can operate privacy, security, evaluation, and incident-response controls for the system lifetime.

Weak fit

• A centralized de-identified dataset already satisfies policy and produces better auditability.
• Participants have incompatible labels, schemas, objectives, or legal authority to collaborate.
• The global model harms important sites that actually need local or multi-task models.
• The privacy claim depends only on raw data staying local.