VAFT pipeline lineage explorer

This page shows what the implemented production pipelines actually do, generated from the documented commit (8bd7ef7). It is the production-lineage view of the Scientific architecture, and complements the conceptual workflow diagrams and the Automated pipelines guide; it does not replace either, and it shows no live run state.

Each fact has one owner, and the explorer only reads it:

The four views are different semantics and are never merged into one edge type:

• Execution edges are Snakemake’s own: a job must finish before the next is scheduled.
• Scientific references are upstream products a rule consults but deliberately does not schedule on. Pipeline 2 reads pipeline 1’s magnetic EFIT and its constraints through params, so a shot pipeline 1 never reconstructed is recorded rather than failed. These edges are declared once, in paths.SCIENTIFIC_REFERENCES, which the Snakefile builds those params from.
• Validation products (stage plots and their manifests) are evidence about a stage product, never inputs to it.
• Publication is stage-wise: each stage replicates only the IDS it owns, to the source STAGE_REPLICATION names, and writes a replication record. The record is publication evidence, not a scientific product. A shot that has diagnostics and eddy but no equilibrium is a correctly published partial state, and dashed (optional or sparse) stages and sources may be absent for a shot without that being a failure.

The resolved job DAG is for one documented shot per pipeline, with every optional branch the configuration has switched on, not the production shot list. Pipeline 1’s rule all reaches its per-shot products through checkpoints, which a dry run cannot see past, so its graph is built from the configured_products target: the same products with the checkpoints taken as passed. Generating the graph is a dry run in a temporary directory. It contacts no database or HSDS server, runs no solver, and needs no credentials.

Pipeline 1: routine data processing: 29 rules and 38 jobs for shot 39915; Pipeline 2: corrective / kinetic update: 25 rules and 25 jobs for shot 48226. 9 declared scientific references.

The source dependency graph of the library itself is a different relation; see the dependency explorer. What the stages’ products mean – which concept an IDS represents, which diagnostic measures it – is the scientific ontology explorer. Regenerate this snapshot locally with:

python -m vaft._pipeline_graph --output docs/_data/pipeline_graph.yml