LoRA & Parameter-Efficient Fine-Tuning

Four ideas explain the mechanism

At initialization, a common setup makes one factor random and the other zero. The adapter therefore begins as a no-op: B A = 0, so wrapping a model does not immediately change its output.

Trace one adapted linear layer

For a frozen matrix W0 with shape d_out x d_in, the two trainable factors have shapes:

• A: r x d_in
• B: d_out x r
• trainable parameters: r(d_in + d_out)
• adapted output: h = W0 x + (alpha / r) B A x

Low rank limits the dimension of the update; it does not prove that a chosen rank can represent every task. Rank, target modules, data, and optimization must be evaluated together.

Size the adapter before allocating hardware

The exact trainable count comes from matrix shapes, not from a fixed percentage of the model. Choose a Llama-like teaching fixture, adaptation surface, rank, base precision, sequence length, and GPU budget. The lab recomputes adapter parameters and a transparent memory floor.

The memory result is a planning floor, not a profiler. It includes frozen weights, a declared per-adapter-parameter training-state allowance, and an illustrative activation term. CUDA kernels, temporary buffers, fragmentation, attention implementation, checkpointing, and distributed strategy can move the real peak.

Change one capacity decision at a time

Rank controls the bottleneck width

• Begin with a modest rank and compare held-out task and slice metrics.
• Increase rank only when the lower-rank run underfits and the data or target-module coverage is not the actual bottleneck.
• Do not interpret a larger adapter as automatically better; extra capacity can memorize noise or make optimization less stable.

Target modules control where change is allowed

• q_proj and v_proj form a narrow attention baseline with low storage and optimizer cost.
• All attention projections broaden how token interactions can change.
• All linear layers also adapt feed-forward transformations and can substantially increase the parameter budget.

Alpha controls update scale

Original LoRA uses alpha / r. If rank changes while alpha stays fixed, the scale changes too. Treat rank, alpha, initialization, learning rate, and target modules as a jointly versioned recipe.

Use QLoRA when frozen-base memory is the constraint

QLoRA keeps the LoRA adapter trainable while storing the frozen base model in 4-bit quantized form. The original QLoRA recipe combines:

• NF4, a 4-bit data type designed for normally distributed pretrained weights;
• double quantization, which also quantizes quantization constants;
• paged optimizers, which manage memory spikes with unified memory;
• higher-precision computation for the operations that consume quantized weights.

QLoRA mainly reduces the frozen-base storage burden. It does not make activation memory, long contexts, data loading, evaluation, or optimizer behavior disappear. A 4-bit checkpoint is also not evidence that the task quality matches a higher-precision baseline.

Build a reproducible adapter artifact

1. 1

   Measure

   Freeze the task contract

   Version the prompt or chat template, output schema, evaluation rubric, holdout groups, critical slices, and baseline result.

2. 2

   Identify

   Pin model lineage

   Record the immutable base revision, architecture, tokenizer files, special tokens, and template identity.

3. 3

   Configure

   Declare the recipe

   Record target modules, rank, alpha, dropout, initialization, quantization, compute dtype, optimizer, seed, and data manifest.

4. 4

   Learn

   Train and inspect

   Track optimization, throughput, memory, task metrics, rare slices, safety, and regressions against the frozen base.

5. 5

   Release

   Package the delta

   Store adapter weights, configuration, evidence, dependency versions, compatibility contract, rollback target, and integrity digest together.

Training loss answers whether the optimizer fits the training records. It does not answer whether the adapter generalizes, preserves base behavior, remains safe, or can load on the production runtime.

Prove the artifact fits the serving runtime

Choose an adapter, serving target, and deployment mode. The lab applies explicit compatibility, evidence, integrity, and rollback gates so a small checkpoint cannot bypass the controls used for a full model.

Dynamic loading and merged serving optimize different operational goals. Neither mode makes an adapter independent of its training lineage.

Choose the deployment boundary deliberately

Operate LoRA as a versioned learning system

Monitor the training system

• peak allocated and reserved accelerator memory;
• tokens per second, examples per second, and data-loader stalls;
• gradient norms, update norms, loss curves, and numerical overflows;
• task, slice, safety, calibration, and base-regression metrics by checkpoint.

Monitor the serving system

• base, adapter, tokenizer, template, and runtime version on every release;
• adapter load failures, cache hit rate, swap latency, and active-adapter cardinality;
• p50, p95, and p99 latency plus memory pressure by adapter and sequence bucket;
• quality, policy, refusal, and schema failures by protected slice.

Define recovery before exposure

• Keep the previous immutable manifest and artifact digests.
• Roll back the adapter, base, prompt/template, tokenizer, or serving runtime independently when the boundary permits it.
• Canary on representative traffic and stop on critical slice or compatibility failures.
• Re-run release evidence whenever data, base revision, tokenizer, template, merge recipe, quantization path, or runtime changes.

Primary sources and scope

• The LoRA paper defines the low-rank update, zero-effect initialization, original alpha / r scaling, target-matrix experiments, and mergeable linear design.
• The QLoRA paper introduces NF4, double quantization, and paged optimizers for 4-bit base-model fine-tuning.
• The Hugging Face PEFT LoRA reference documents current configuration fields and adapter initialization behavior.

The lab models use declared teaching fixtures and transparent arithmetic. They are not performance forecasts. Profile the exact architecture, kernels, context distribution, precision, optimizer, hardware, and distributed strategy that will run in production.