Provenance¶
reprotrail.provenance records software and input state in portable metadata.
It captures Git commit, branch, canonical remote URL, dirty status, optional
dirty diff hash, and compact path state for filesystem, Git, Git LFS, and DVC
inputs.
Directory summaries consume at most 20,001 filesystem entries by default: 20,000 entries may contribute files to the manifest and one additional entry establishes truncation. Complete summaries retain the globally sorted legacy hash. Truncated summaries use a versioned partial-hash kind and report lower bounds rather than misleading exact file and byte counts. This keeps large Zarr inputs and other directory products bounded.
Ordinary files always record byte size and nanosecond modification time. Files up to 16 MiB additionally receive a SHA-256 content identity, which covers scripts, configurations, and typical manifests without reading multi-gigabyte scientific inputs merely to start a run. Git LFS detection reads only the first 512 bytes. Larger files explicitly record that the content hash was skipped at the size limit.
DVC detection safely parses YAML output declarations in ancestor dvc.lock
files and .dvc files within the input’s Git repository. Direct sibling .dvc
files are also supported outside Git repositories. Output paths are resolved
relative to their metadata files and static working directories; lockfile stage
working directories come from the companion dvc.yaml. Dependencies, command
text, and unrelated output names do not establish DVC ownership.
For a child of a tracked directory, metadata.dvc.outputs retains the enclosing
output’s path and checksum and adds input_relative_path for the child. An exact
output match uses .. The checksum identifies the whole registered dataset,
not an individually verified file. Detection can succeed when local data are
missing; it does not establish that a child exists in the registered dataset or
that a cache or remote can retrieve it. Only matching output declarations are
included, rather than every output in a lockfile.
Unreadable or malformed metadata, unresolved path templates, and remote output
URLs are reported in metadata.dvc.errors, including public provenance. They do
not establish DVC tracking on their own. Inspection does not invoke DVC, inspect
its cache, download data, or add payload hashing beyond the existing file budget.
PyYAML is a runtime dependency for safe metadata parsing. Historical records and
exports remain unchanged; improved detection applies to newly captured inputs.
When an input has a conventional sibling product sidecar such as
effective-config.prov.json and its .sha256 file, input inspection records
the sidecar name and checksum as product_provenance metadata. Reproduction
uses this metadata to resolve the input and verify its producing provenance;
dependency-epoch audits use the same reference when comparing product runtime
snapshots.
Use public_provenance() before writing records into public outputs. It removes
local-only fields such as repository roots while preserving stable identifiers.
:::{warning}
“Public” means portable, not anonymous. Provenance can retain recorded command
arguments, repository names and URLs, input paths outside Git repositories,
dirty-file names and patches, and values selected by env_var_whitelist.
Commands and paths can themselves contain usernames, tokens, private hostnames,
or sensitive dataset names. Review the final JSON before sharing it, keep
secrets out of command arguments, and whitelist only environment variables that
are safe to record. The public helpers remove selected local-only fields; they
are not a general secret scanner or anonymizer.
:::
from reprotrail.provenance import get_git_state, get_input_path_state
software = get_git_state(".")
input_state = get_input_path_state("data/input.zarr")
History helpers can write compact CF/xarray-style entries:
from reprotrail.provenance import build_cf_history_entry
entry = build_cf_history_entry(["python", "-m", "workflow", "run"])