Model Compression Techniques
Advanced model compression: pruning, quantization, distillation techniques for efficient deployment on edge devices and production systems.
What is model compression?
Model compression transforms a trained model or its architecture so the deployed artifact uses less storage, memory, compute, bandwidth, energy, or latency while preserving the quality required by the product. Quantization lowers numerical precision, pruning removes weights or structures, distillation trains a smaller student, and factorization replaces expensive operations with compact approximations.
Compression is useful only in relation to a target system. A mathematically smaller model can run at the same speed or slower when the serving backend lacks compatible kernels, inserts format conversions, or remains bottlenecked by activations, memory movement, batching, or another service stage.
The invariant to remember
The exact exported artifact must improve a declared deployment constraint without violating any required quality, safety, or compatibility gate. The source checkpoint is not the production artifact, and theoretical compression is not measured acceleration.
Review model serving first if batching, runtime artifacts, and tail latency are unfamiliar.
Start from the deployment constraint
1 Target
Name the binding constraint
Define device memory, package size, cold start, p99 latency, throughput, power, network transfer, or cost under a specific workload.
2 Evidence
Profile the baseline
Measure weights, activations, cache, runtime workspace, operators, memory bandwidth, compute, and non-model request stages.
3 Transform
Choose a compatible technique
Match precision, sparsity, architecture, and operator changes to kernels the target backend can actually execute.
4 Lower
Export the real artifact
Pin the recipe, calibration set, operator graph, packing, compiler, runtime, device, and artifact integrity.
5 Release
Evaluate and canary
Compare quality slices and system metrics with the baseline, then expand traffic only while explicit gates remain satisfied.
Understand what each technique changes
Fewer bits
Quantization
Represents weights, activations, or both with lower precision. The design includes granularity, scale and zero-point rules, calibration or training, accumulation precision, packing, and backend support.
Less structure
Pruning
Removes individual weights, blocks, channels, heads, layers, or other structures. Structured removal is easier for dense kernels to exploit; unstructured sparsity needs specialized execution.
Smaller student
Distillation
Trains a separate student using labels plus teacher logits, hidden representations, attention, or task-specific signals. Student architecture and loss weights are new modeling decisions.
Lower-rank work
Factorization
Approximates large matrices or tensors with smaller factors. Rank controls the quality and compute trade-off, while factor shapes must map efficiently to the runtime.
Quantize for a declared backend
Store weights at lower precision while activations remain in a higher precision. This often reduces weight memory and bandwidth, but conversion and kernel support determine speed.
Quantize activations at runtime where supported. It avoids a static activation calibration pass, while runtime quantization adds work and operator coverage is backend-specific.
Collect representative activation statistics, derive quantization parameters, convert the graph, and evaluate the exported model. Calibration coverage is part of the artifact evidence.
Simulate quantization during training so weights can adapt to numerical error. It costs more engineering and compute but can recover quality when PTQ misses the gate.
Modern PyTorch quantization development is centered in torchao, and exported-device flows use backend-specific quantizers. Pin the actual library and backend APIs used by the release rather than copying an older generic recipe.
Distinguish storage math from runtime memory
Theoretical weight storage is straightforward: parameters x retained fraction x bits / 8. Production memory also includes scales and metadata, activations, attention or recurrent caches, temporary buffers, compiled workspaces, runtime code, input batches, output buffers, and safety reserve.
Make the exported artifact fit, then prove it is faster
Theoretical bits determine weight storage. Runtime memory, backend kernels, representative quality, and measured latency determine whether compression is useful in production.
Deployment verdict
The candidate fits and clears the measured release gates
Publish the exact artifact, backend, benchmark, calibration, and evaluation identities together; keep the baseline ready for rollback.
6.5GB
8-bit exported weights
18.5GB
77% of device memory
1.54x
Baseline p99 / candidate p99
0.7 points
Across required evaluation slices
Export
Pin the compression recipe, calibration data, operator set, weight packing, backend, and artifact checksum.
Benchmark
Measure cold and warm latency, throughput, memory, power, and quality at representative batch and sequence shapes.
Gate
Compare required slices to the uncompressed baseline, canary the exact runtime artifact, and retain rapid rollback.
Do not call 32 / bits a speedup. It is a weight-storage ratio. Speedup must be measured after export on the target device, backend, batch shape, and request distribution.
Prune only the structure the runtime can exploit
zeros
Unstructured sparsity
Individual zero weights reduce logical density but may still execute through ordinary dense kernels.
N:M
Pattern sparsity
A fixed number of retained values per group can map to supported sparse hardware when dimensions and kernels match.
channel
Structured pruning
Removing complete channels, heads, neurons, or layers changes dense tensor shapes and can reduce ordinary dense work.
retrain
Recovery step
Fine-tuning after pruning lets remaining capacity adapt; every recovery recipe still needs independent evaluation.
Profile before and after export
- Confirm the pruned structure survives graph export, constant folding, and backend lowering.
- Check that tensor dimensions remain aligned with efficient kernels and vector widths.
- Measure operator time, memory movement, end-to-end p99, throughput, and power.
- Evaluate sensitive layers and required quality slices instead of applying one global ratio blindly.
Train a student as a new model
Distillation does not shrink a checkpoint in place. It defines a student architecture and a training objective that may combine hard labels, softened teacher logits, feature matching, attention transfer, or task-specific constraints.
A distillation pipeline has two evidence paths
Teacher behavior guides optimization, while independent labels and product metrics decide whether the student is acceptable.
Reference behavior
Teacher artifact
Runs under a pinned version and temperature to produce logits or representations for approved training examples.
Transfer
Student training
Optimizes the student against weighted label, teacher, representation, and regularization losses.
Do not grade from teacher alone
Independent evaluation
Measures task quality, calibration, robustness, fairness, safety, and out-of-distribution behavior against product evidence.
System benchmark
Target runtime
Exports and measures the student on the exact serving device, compiler, backend, batch, and traffic shape.
Combine techniques in an order you can reproduce
Compression techniques interact. Quantizing after pruning can expose different ranges; distilling from a quantized teacher can transfer numerical artifacts; factorization can change which layers remain sensitive; and fine-tuning can undo sparsity unless masks or parameterizations are enforced.
1 Reference
Freeze the baseline
Pin the source model, data, evaluation, serving runtime, and baseline quality and performance report.
2 Attribution
Apply one transformation
Record the exact recipe and measure its isolated effect before composing it with another technique.
3 Train or calibrate
Recover quality
Use representative data and keep recovery hyperparameters, masks, teacher outputs, and calibration statistics versioned.
4 Artifact
Export after the final change
Evaluate the graph and weights the runtime will load, not an intermediate framework object.
5 Decision
Compare with every baseline
Attribute gains and losses to each step, reject fragile combinations, and preserve a simpler fallback.
Find regressions aggregate retention hides
Average accuracy can remain stable while compression harms rare languages, small objects, long sequences, confidence calibration, safety classifiers, or inputs outside the calibration distribution. Latency can also remain flat because the target runtime falls back to unsupported operators.
Loading release lab
Preparing compression failures...
Publish a complete compression manifest
What changed
Source identity
Record the source checkpoint, architecture, tokenizer or preprocessing, transformation order, recipes, training code, and random seeds.
What shaped numerics
Data identity
Version calibration, recovery-training, teacher-output, and evaluation datasets with sampling rules and privacy constraints.
What executes
Runtime identity
Pin exported graph, weight format, operator set, compiler, backend, kernel library, device, driver, and serving configuration.
Why it shipped
Evidence identity
Attach slice metrics, latency distributions, throughput, memory, power, cost, canary results, approval, and rollback target.
Review the production checklist
Build these controls
- Define the target constraint and profile the uncompressed end-to-end system first.
- Use backend-supported formats, operators, sparsity patterns, and tensor shapes.
- Version representative calibration, teacher, recovery, and evaluation evidence.
- Evaluate aggregate and required slice quality on the exact exported artifact.
- Benchmark cold and warm latency, throughput, memory, power, and cost on target hardware.
- Canary under an immutable version and retain the uncompressed rollback path.
Avoid these traps
- Treating storage reduction as a guaranteed latency or cost improvement.
- Applying one pruning or quantization setting uniformly to every sensitive layer.
- Evaluating a framework model while deploying a separately lowered artifact.
- Approving average retention while a required cohort or safety behavior regresses.
- Combining transformations without isolating their effects or preserving provenance.