DVC Data Versioning
Use DVC for reproducible data and pipelines through content-addressed caches, remotes, dependency invalidation, publication ordering, recovery, and provenance.
What is DVC data versioning?
DVC (Data Version Control) connects large data and model files to the Git revision that uses them without putting those large files in Git. Git stores small DVC metadata, pipeline definitions, parameters, and code. DVC stores file contents in a content-addressed cache and can copy those contents to a shared remote.
This separation matters because an ML result is not reproducible from source code alone. You also need the exact input data, parameters, pipeline graph, environment, and produced artifacts.
The core invariant is a Git revision identifies the desired project state, but that state is recoverable only while every referenced content hash remains available in a local cache or DVC remote.
Identity
Git metadata
Git versions .dvc files, dvc.yaml, dvc.lock, parameters, metrics, plots, and source code. These files describe which content and computation belong together.
Local content
DVC cache
The local cache stores artifacts by content hash. DVC links cached content into the workspace and can reuse unchanged files or stage results.
Shared durability
DVC remote
A remote holds cached objects outside one laptop or runner. dvc push publishes referenced content; dvc pull retrieves it and updates the workspace.
Follow one dataset from bytes to a reproducible revision
dvc add data/raw does not upload data or commit anything to Git. It hashes the target, places its content in the cache, writes a small metadata file, and adds the workspace path to .gitignore.
One versioned dataset path
Git preserves the pointer and project definition; DVC storage preserves the referenced bytes.
Readable path
Workspace
data/raw looks like an ordinary directory used by scripts and notebooks.
Content hash
Local cache
DVC stores the content once and links or copies it into the workspace.
Small metadata
Git repository
data/raw.dvc maps the readable path to its hash, size, and object count.
Shared objects
DVC remote
dvc push transfers missing cache objects so another checkout can recover the data.
Keep the commands in the right transaction
- Change or acquire the data and validate its schema, license, privacy class, and source.
- Run
dvc addfor standalone data or let a DVC pipeline stage produce the output. - Run
dvc pushso the referenced objects exist outside the current machine. - Commit DVC metadata, code, parameters, and the lockfile to Git.
- In CI or on another machine, check out the Git revision and run
dvc pullordvc checkoutagainst an already populated cache.
dvc push and git push are separate operations. A Git commit can therefore point to objects that were never uploaded. Make publication order and post-push verification part of the delivery workflow instead of treating remote storage as an automatic side effect of git commit.
Plan cache and remote placement from change rate
Content addressing avoids storing one complete copy per version when most files are unchanged. It does not make storage free: a rewritten archive or monolithic checkpoint has a new hash even if its logical records mostly match the previous version.
Loading artifact placement model
Read the estimate correctly
- The lab models retained unique content as the first version plus the changed share of each later version. Real usage depends on file boundaries and actual hashes.
- A directory with many independently hashed files can reuse unchanged members. One repeatedly rewritten archive usually invalidates the whole object.
- Copy cache mode can require another workspace-sized copy. Reflink, hardlink, and symlink modes have different filesystem support and mutation semantics.
- A shared remote protects collaboration only when the exact referenced objects were pushed and retained. It is not a substitute for bucket durability, access control, retention policy, or recovery testing.
Do not market a rough compression or deduplication ratio as a DVC guarantee. Measure unique cache bytes, object counts, transfer time, and restored checksums on the real file layout.
Define a pipeline as a hashable dependency graph
A DVC pipeline stage declares a command, dependencies, parameter keys, outputs, and optional metrics or plots. dvc.yaml is the human-authored graph. dvc.lock records resolved dependency and output hashes after execution.
1 Source
Ingest immutable input
Validate the source contract and publish a versioned raw-data output. Preserve a source identifier or acquisition timestamp alongside the bytes.
2 Transform
Prepare features
Declare raw data, transformation code, and preprocessing parameters as dependencies. Write a deterministic, replaceable feature artifact.
3 Compute
Train the model
Declare features, training code, and selected parameter keys. Emit the model plus machine-readable metrics.
4 Evidence
Evaluate the candidate
Compare the candidate against explicit thresholds and data slices. Keep evaluation outputs tied to the same lockfile.
Understand what dvc repro decides
For each target stage, DVC compares the current command, dependencies, parameters, and declared outputs with the recorded stage state. Changed upstream output propagates to downstream stages. Unchanged stages can be skipped or restored from run cache when the required cached outputs exist.
Loading invalidation and recovery model
Separate version restoration from computation
Several DVC commands sound similar but change different state:
Workspace
dvc checkout
Replace DVC-tracked workspace paths with the versions referenced by the current Git revision. It normally expects required objects to exist in the local cache.
Transfer + workspace
dvc pull
Download missing referenced objects from a remote and update tracked workspace paths. Use it after checking out the intended Git revision.
Computation
dvc repro
Evaluate the pipeline graph and run invalidated stages needed for the requested target. It does not publish results to the shared remote.
Publication
dvc push
Upload cache objects referenced by the current project state. It does not create a Git commit or validate a model for release.
Use the smallest correct recovery action
- Missing workspace file, object cached locally: run
dvc checkout. - Missing local object, object present remotely: run
dvc pull. - Dependency or parameter changed: inspect
dvc status, then rundvc reprofor the intended target. - Output is missing everywhere but reproducible: rerun the producing stage and its invalidated descendants, then publish the new cache state.
- Output is missing and not reproducible: restore the source from an independent backup or upstream system. Git metadata cannot reconstruct absent bytes.
Do not start incident response with dvc repro --force. A full recomputation can be expensive, can overwrite useful workspace evidence, and cannot recover an unavailable external source.
Build a team workflow that closes the publication gap
1 Pull request
Review metadata
Review pipeline edges, parameter changes, output declarations, source policy, and lockfile changes. A changed hash is evidence of different bytes, not evidence of better data.
2 CI
Verify remote objects
Use scoped credentials to fetch required data, reproduce or restore the target, and verify checksums, metrics, and policy gates.
3 Release
Publish atomically by convention
Push DVC objects before merging or tagging the Git pointer. Record the Git revision, data hash, model hash, and environment identifier in the release manifest.
4 Proof
Restore from a clean runner
Regularly test a cache-empty checkout. A green run using one developer's warm cache does not prove the remote contains everything.
Keep credentials out of Git
- Commit shareable remote definitions in
.dvc/config. - Put user-specific endpoints or sensitive settings in
.dvc/config.local, which DVC normally excludes from Git. - Prefer workload identity or short-lived environment credentials over long-lived access keys.
- Give CI read access by default and grant write or deletion rights only to explicit publication and retention jobs.
- Protect the Git branch and DVC remote together; changing either side can break the revision-to-content mapping.
Test failure behavior before trusting reproducibility
Missing or unpublished objects
dvc pull fails when the current metadata references an object absent from every configured remote. Alert on the missing hash, preserve the Git revision, and determine whether another cache or upstream source still has the bytes. Do not rewrite metadata to a convenient nearby dataset.
Interrupted publication
Object uploads can be retried because content-addressed destinations are stable, but a workflow still needs to verify that every object referenced by the intended revision exists before publishing the Git pointer.
Unsafe garbage collection
dvc gc removes unreferenced cache objects according to the revisions and repositories included in the command. Remote garbage collection can affect teammates and historical revisions. Inventory protected branches, tags, experiment refs, and dependent projects; dry-run the retention decision externally; back up critical remote data; and test an old restore before deletion.
Nondeterministic stages
DVC can detect declared changes, but it cannot make training deterministic. Seeds, library versions, hardware kernels, external services, wall-clock inputs, and undeclared files can change an output. Capture them or explicitly label the result as non-bit-reproducible.
Operate DVC as a provenance system
Monitor the recovery path
- Cache size, object count, inode usage, and workspace link mode
- Remote object count, retained bytes, upload failures, and missing-object errors
dvc pullbytes, duration, retry count, and clean-runner restore success- Stage invalidation reason, run duration, cache hit or restoration result, and output checksum
- Git revision,
dvc.lockdiff, parameter diff, data source identifier, and model release decision
Choose DVC for the right boundary
DVC is a strong fit for Git-centered projects where a team versions bounded datasets, models, and pipelines together. It is less suitable as the transaction layer for a high-concurrency data lake, row-level collaborative editing, streaming state, or a petabyte-scale namespace with tens of thousands of frequently synchronized files. Keep the authoritative operational dataset in the system designed for that workload and use DVC for project-scoped snapshots or derived artifacts when appropriate.
Production review checklist
- [ ] Every release can be restored from a clean machine using documented identities.
- [ ] DVC objects are pushed before the Git revision becomes a shared dependency.
- [ ] Pipeline commands declare every meaningful code, data, parameter, and environment dependency.
- [ ] Outputs are immutable or replaceable and are validated before publication.
- [ ] Credentials are scoped, short-lived, and absent from committed configuration.
- [ ] Remote retention protects supported branches, tags, experiments, and downstream consumers.
- [ ] Recovery drills cover missing cache, missing remote objects, partial uploads, and unavailable external sources.
Verify behavior against primary documentation
- DVC get started explains
.dvcmetadata, the local content-addressed cache, remotes, push, pull, and version switching. - DVC data pipelines introduces the dependency graph,
dvc.yaml,dvc.lock, anddvc repro. - Remote storage documents supported remotes and the relationship between cache and shared storage.
- Running pipelines describes stage invalidation, reproduction, and run-cache behavior.
dvc repro,dvc pull, anddvc gcdefine the operational command boundaries and destructive retention behavior.