Export and transfer

Export selected datasets and all linked upstream reproduction metadata. Only the selected data are included, through symlinks; intermediate products and original source datasets stay external. Metadata files are copied independently. An export is not a full offline environment or a copy of original source data.

The examples below use a fictional survey-processing project with separate survey-data and survey-tools repositories.

reprotrail export \
  --provenance /data/survey-data/results/north/summary/summary.prov.json \
  --provenance /data/survey-data/results/south/summary/summary.prov.json \
  --input-root survey-data=/data/survey-data \
  --input-root survey-tools=/code/survey-tools \
  --product-base /data/survey-data/results \
  --output /scratch/survey-export

The selected packages are easy to find under products/:

survey-export/
  products/
    north/summary/             # Selected dataset and its package sidecars
    south/summary/
  metadata/                    # Upstream reproduction artifacts
  repositories/                # Present only with --archive-repos
  manifest.json
  README.md                    # Links to selected datasets and provenance

--product-base names a source directory, not a destination. Selected packages are placed relative to that directory; every selected provenance package must lie beneath it. Without this option, products/ preserves repository-relative paths, such as products/survey-data/results/north/summary/. Packages without a known repository root retain digest-based paths under products/artifacts/. Only the selected packages move into products/; intermediate products remain metadata-only. The manifest records all actual locations.

Existing exports using the older metadata/ layout remain readable and are not rearranged. Create a new export to use the new layout.

Run this command on the source SSH host. Pull from the receiving host using:

rsync -aL --partial host:/scratch/survey-export/ ./survey-export/

-L dereferences selected dataset symlinks. The source datasets must remain available and unchanged during transfer. The receiving export contains actual data and can be moved without retaining the source host’s directory structure. Reprotrail does not manage SSH credentials, transfers, or remote execution.

Original provenance files remain byte-for-byte unchanged. manifest.json records the selected roots, products, dependency order, original-to-export file locations, file checksums, external inputs, traced repositories, and metadata gaps. Metadata is shared across selected products. Selected package sidecars stay beside their dataset under products/; upstream metadata preserves repository-relative paths under metadata/ where an input root is known. Other artifacts use digest paths. Input roots resolve repository-relative provenance input paths and do not copy the corresponding datasets.

By default, missing or mismatched metadata produces an incomplete export and a gap report, rather than preventing collection. --strict rejects it and removes an incomplete destination. Missing or malformed root records, cycles, ambiguous paths, and destination collisions are errors. Existing destinations are never overwritten. --dry-run discovers and validates local metadata without creating files, linking datasets, or fetching repositories. Repository availability is therefore not proved by a dry run.

Git source persistence is opt-in with --archive-repos. Only repositories explicitly recorded in project_repo, software_repos, and configured_repos are bundled, at all their required revisions. Input repository ownership and Git dependencies found only in the runtime snapshot/lockfile are excluded. Use --repo-source NAME=SOURCE for local repositories or alternate remotes. During recursive replay, bundles supply recorded checkouts and matching locked Git dependency downloads through process-local URL mappings; the lockfile and global Git configuration stay unchanged. Standalone Git bundles include history needed for the recorded commits and may be larger than source snapshots. Downloadable dependency packages remain external. Dirty tracked changes require their recorded patches; untracked code is not reconstructed from a filename listing.

reprotrail export arguments

Syntax

Status

Behavior

--provenance PATH

Required; repeatable

Selects product provenance records whose datasets are linked into the export.

--output PATH

Required

Creates a new export directory.

--product-base PATH

Optional

Places selected packages under products/ relative to source directory PATH; all selected packages must lie beneath it.

--input-root NAME=PATH

Optional; repeatable

Resolves repository-relative inputs for recorded repository NAME under PATH.

--repo-source NAME=SOURCE

Optional; repeatable

Archives traced repository NAME from SOURCE when repository archiving is enabled.

--archive-repos

Optional

Archives explicitly traced Git repositories, deduplicating required revisions per repository.

--strict

Optional

Rejects missing or mismatched reproduction metadata.

--dry-run

Optional

Inspects collection without creating files or fetching repository archives.

--json

Optional

Prints the manifest and gap report as JSON.

Python API

from reprotrail.export import export_products

report = export_products(
    provenances=["/data/survey-data/results/north/summary/summary.prov.json"],
    output="/scratch/survey-export",
    product_base="/data/survey-data/results",
    input_roots={"survey-data": "/data/survey-data", "survey-tools": "/code/survey-tools"},
    dry_run=True,
)

Use discover_products for dependency discovery without an export destination.