Credibility and applicability

A VAFT result is trusted for six different reasons, and they fail independently. A reconstruction can fit every probe and still be unidentified. A solve can converge and still violate an ordering its model assumed. A datum can be valid and still be extrapolated. This page fixes the vocabulary for those questions (issue #1639) and the evaluation of one of them, approximation applicability (#1628, with the ordering quantities from #1627).

The rule this page protects

Scientific assessment surrounds the fast path; it never burdens it. A process computes its result and returns it. Assessment is a separate, optional call over that result, and so is uncertainty propagation. Nothing on this page is imported by vaft.process, and test/test_validation_credibility.py checks that in a clean interpreter.

The six axes

vaft.validation.credibility.AXES:

The axes are a classification, not a result schema. A domain module still returns its own plain dict. Evidence(axis, key, status, metrics, reasons, cost, role) exists only so that evidence from different domains can be laid side by side. compose() returns one status per axis and never one overall score. For example, fit = pass, independent validation = not available is a different statement from independent validation = fail.

Rules carried over from #253 and #337:

• A metric is not a verdict, and a verdict is not a policy. Whether a result may be used is decided by the workflow.
• Missing evidence is not failure.
• A precondition (“can this algorithm run?”) is not credibility (“is the result trustworthy?”).

Fitted data is not independent validation. Every piece of evidence can carry a role from EVIDENCE_ROLES, such as used_for_inference or independent_validation.

• VAFT’s EFIT fits the diamagnetic flux by default (EFITConfig.use_diamagnetic_flux; #891, #1440). So when the reconstruction fitted the flux – the report’s diagnostic_fit.diamagnetic_flux records fit_role: fitted, i.e. the constraint’s weight was non-zero, whatever grade the entry received – every check in DIAMAGNETIC_CHECKS moves off the V axis automatically.
• Name any other fitted check in used_for_inference=. An unknown key or a bare string is refused rather than ignored.

Cost classes

COST_CLASSES = ("cheap", "moderate", "expensive"):

• cheap: read from results that already exist, such as a convergence flag or an ordering parameter.
• moderate: local computation, but no solver rerun.
• expensive: solver ensembles, Monte Carlo, continuation. Always opt-in.

Pilots

How the mappings are chosen.

• CATEGORY_AXES maps each existing validation category to its default axis.
• CHECK_AXES records the registry checks placed elsewhere, each with its reason in a source comment. For example, plausibility of the inferred state is inference, not numerical.
• CRITERIA_AXES is the complete mapping for the study criteria. There, virial compares two β_p of the same g-file, so it is numerical, and only Thomson is on V.

Applicability: margins first, labels second

A reduced model is derived under orderings. For example, ideal single-fluid MHD needs all of the following:

• $S \gg 1$;
• $d_i/L \ll 1$;
• $\rho_i/L \ll 1$;
• $\tau_{\rm evol}/\tau_A \gg 1$.

These test different things, so an ApproximationContract keeps them as separate OrderingAssumptions. Each assumption names its quantity, its characteristic scale and its scope, so $d_i/a$ can never silently stand in for $d_i/\delta_{\rm layer}$.

The primary output is the continuous ordering margin against the assumption’s threshold $x_0$:

• $m > 0$: on the permitted side, with $m$ decades to spare. SUPPORTED means only this: $x = 0.99$ against $x_0 = 1$ is supported with $m = 0.004$, so read the margin, not the label.
• $m \le 0$: violated.
• The default $x_0 = 1$ is where the expansion parameter reaches order unity. This is the only cutoff applied without a source. Any other threshold must name its source in threshold_source, or the assumption refuses to construct (#1639 §12).

Statuses. They come second, from the same vocabulary as the operational-space renderer (#1664):

There is no MARGINAL: a margin band would be a cutoff nobody derived, and the margin itself is there for anyone who wants to see how close a state is.

Functions:

• evaluate_contract(contract, state) returns every assumption’s value, margin and status, plus the limiting assumption.
• evaluate_population(contract, table) keeps every row and every individual check.
• as_evidence(result) places a result on the A axis.

The strongest applicability evidence pairs the ordering with the convergence it predicts (#1639 §12): successive_discrepancy([Q_reduced, ..., Q_full]).

Ownership

Sensitivity, linearization and uncertainty (#1642)

The domain that owns an operation owns its derivatives. EFIT’s native response matrix stays in vaft.code.efit.linearization, and a process’s Jacobian stays beside the process. Two shared pieces sit around them:

• the algebra, in vaft.formula.sensitivity;
• what the numbers mean, in vaft.validation.sensitivity.

There is no sensitivity package, and nothing in this section runs unless it is called.

Propagating through one formula (#1874)

A formula can state how uncertainty passes through it. Its docstring then has an Uncertainty propagation section: the input order, the Jacobian, what is held fixed, and where first order stops being valid. It may also carry a domain-owned analytic Jacobian. The decorator that attaches one returns the formula itself, so its signature, values and cost do not change. describe(name).uncertainty_propagation and .analytic_jacobian report both, and the formula reference shows the section. The pilots are equilibrium.confinement_time_from_P_loss_W_th and equilibrium.ohmic_heating_power_from_I_p_V_res.

import numpy as np
from vaft.formula import equilibrium, sensitivity

# P_loss = 100 kW +/- 10 %, W_th = 2 kJ +/- 5 %, correlated through one reconstruction (rho = 0.5)
cov = np.array([[1.0e4**2, 0.5 * 1.0e4 * 1.0e2],
                [0.5 * 1.0e4 * 1.0e2, 1.0e2**2]])
result = sensitivity.propagate_formula_uncertainty(
    equilibrium.confinement_time_from_P_loss_W_th,
    inputs={"P_loss": 1.0e5, "W_th": 2.0e3},
    input_names=("P_loss", "W_th"),  # the covariance axes; any other input is fixed
    covariance=cov,                   # or std=[...] for independent inputs, never both
)
result.value, result.std            # 0.02 s, about 1.7 ms
result.derivative_method            # "analytic" (else "finite_difference")
result.linearity                    # largest linearity_ratio over +/- 1 sigma: about 0.1 here

The facade’s rules:

• An unstated uncertainty is an error, not zero. Pass explicit zeros for an exact input.
• A 1-D covariance holds variances; std holds standard deviations.
• The finite-difference step follows each input’s own magnitude.
• A formula may declare its domain next to its Jacobian ($P_{loss} > 0$, $W_{th} \ge 0$ here): a finite $W/P$ at negative $P$ is still not a confinement time. Pass domain= to override it.
  • Input means outside the domain are refused rather than given a covariance.
  • A $\pm1\sigma$ step that leaves the domain (here $P_{loss} - \sigma \le 0$) reports linearity = inf.
• method="monte_carlo" samples the formula itself from Gaussian inputs (no other distribution is supported). It counts out-of-domain draws in rejected, and refuses a run that accepts fewer than two.

The result is local and first order. It carries only measured-input uncertainty, not the fitted coefficients or scatter of an empirical law. compare_linear_to_monte_carlo and compare_jacobians interpret it.

Operational boundaries. vaft.formula.boundaries.propagate_boundary_uncertainty(boundary, x, inputs=..., std= | covariance=) carries measurement uncertainty into both the boundary value $b$ and the signed margin of an operating point. The operating value can share a covariance with the boundary inputs, for example a density and the current from one shot. Three kinds of uncertainty stay apart:

• measurement: boundary (on $b$) and margin, both full propagation results;
• source parameters: source_parameters_relative, the relative spread of $b$ from the coefficient and exponent uncertainty the source reports, taken as uncorrelated because no covariance is published. source_parameters_complete is False when the source reports only some of them;
• source scatter: source_scatter_relative, the source’s RMS scatter about its fit.

The last two are also given on the margin’s scale (margin_std_from_source_*). A part the source does not report, or reports in a form that cannot be propagated, is None: unknown rather than zero. source_note says which applies. warnings repeats the applicability and kind warnings of evaluate_boundary, and flags a margin whose linearity check fails. boundary.uncertainty is passed through unchanged, and nothing returns a permitted/forbidden verdict. A margin within its spread is an assessment for the caller or vaft.validation, and crossing a soft boundary is not a prediction of failure.

Distinctions kept apart:

• Provenance: native, finite_difference, autodiff or analytic. Comparing a Jacobian with one of the same provenance is refused: that is repetition, not validation.
• Kind: an observation Jacobian (identifiability), a governing_operator one (uniqueness, branches) and a forward one (local sensitivity) answer different questions.
• Perturbation: physical versus numerical. Physical sensitivity is never mixed with convergence. compare_jacobians refuses a pair that differs in kind or perturbation unless allow_mixed=True.

Tolerances. A comparison returns metrics. It carries a status only when the caller passes a tolerance=(warn, fail); no tolerance is invented here.

• An infinite discrepancy fails. Examples: a zero Jacobian against a real spread, or a non-zero row against a zero one.
• A comparison over Monte Carlo draws that rejected any sample is never better than warn: the sampled covariance is then conditioned on the model succeeding.

test/test_sensitivity_contract.py demonstrates the #1642 acceptance points at unit-test cost:

1. A derivative checked by another method. The exact Green’s-function field (an analytic derivative already in VAFT, $B = \nabla\times\psi$) matches the finite-difference derivative of its flux to below $10^{-6}$.
2. Local propagation against sampling. vaft.process.confinement.dimensionless_confinement_indices already propagates the exponent covariance through a central-difference Jacobian.
   • Its covariance equals the generic kernels’.
   • At $\alpha_P = -0.6$ it passes the test’s tolerance (0.1 in $\ln\sigma$) against 4000-draw Monte Carlo.
   • It is not exact. The linear standard deviation is 5–8 % low across seeds, against a sampling error of 1 %: the curvature of $1/(1+\alpha_P)$ shows even here.
3. Where the linearization fails. At $\alpha_P = -0.93$, $1+\alpha_P = 0.07$ puts the pole 1.4 standard errors away, inside the input distribution.
   • $1/(1+\alpha_P)$ of a Gaussian $\alpha_P$ has infinite variance. The sampled spread does not converge with the sample count, and only its size, not its sign, depends on the seed.
   • The linear propagation fails against it every time, as that function’s docstring warns.
   • The cheap linearity_ratio flags the problem before any sampling is paid for.
   • The reported sampling_standard_error assumes finite variance, so it means nothing in this regime.

Workflow graduation audit

#1642 §21 asks which workflow-held logic is reusable. These are the candidates found so far. None is moved by this change; each move belongs to the owning lane.

Two notes on this table:

• “Already graduated” means only that the library function exists. closures.py keeps its own bootstrap loop.
• sensitivity_report.py and the stage-6 grid exist only on the #1663 branch for now.