Save a transient testbench with explicit pass and fail criteria

Prepare a complete saved bench

Keep the following three files together. The JSON input resolves beside its definition, independently of the terminal working directory. The example checks three independent bias cases: 1 mA, 2 mA and 1 mA.

The numerical limits are illustrative requirements for this small current-source circuit. Choose your real circuit’s electrical acceptance criteria explicitly; there is no built-in physical-accuracy tolerance.

Save bench_values.cir

.PARAM bias=1m

Save bias_bench.cir

* Every saved requirement applies to all three cases
.INCLUDE "bench_values.cir"
I1 0 N DC bias
R1 N 0 1
.OPTION ANALYSIS=VOLTAGE INTEGRATION=BDF SEED=321
.STEP PARAM bias LIST 1m 2m 1m
.TRAN 1p 10p
.PRINT V(N)
.MEASURE TRAN bias_current FIND I(I1) AT=5p
.END

Save bench.json

{
  "schema": "josim-testbench/1",
  "name": "Three bias-current cases",
  "input": "bias_bench.cir",
  "requirements": [
    {
      "id": "bias-range",
      "measurement": "bias_current",
      "unit": "A",
      "min": 0.00099,
      "max": 0.00201
    }
  ]
}

Use the Saved testbench window

1. Save or reload the source and include files, then open Run > Saved testbench and choose Open definition for bench.json.
2. Choose a new result directory whose parent already exists. Select Validate / preview and inspect the planned cases, measurements, assignments, effective settings and destinations.
3. Correct invalid JSON or admission errors in place. Save definition after edits; then Run saved bench uses the reviewed saved definition and unchanged source bytes.
4. Wait for completion in Tasks and Output. Use Open saved results to inspect results.json without rerunning, then review every case/requirement, status, units and artifact integrity. Hiding the window does not cancel an active job.

Run from a script or terminal

The parent directory must exist and run-001 must be new. Do not combine --testbench with positional input, -i, -o or parallel-worker options. This contract is serial buffered transient execution, including .STEP cases; remove .FILE routing and streaming.

Use the process exit code and the completed results.json together. JSON is published last as the completion marker; a partial directory without it is not a completed report. Invalid definitions or publication failures can return code 1 without a complete report.

josim-pro-cli --testbench bench.json --results run-001

Interpret the verdict

Exit 0 / pass means every requirement passes in every case. The checked example reports bias_current values 0.001, 0.002 and 0.001 A, with seed 321 in all three cases.

Exit 2 / fail means a measured value violates a specification. Changing max to 0.00101 produces pass, fail, pass across the three cases; a specification failure does not stop the later case.

Exit 3 / failed_to_evaluate covers a missing measurement, unit mismatch or measurement-evaluation failure. The checked missing-name and mA-versus-A variants returned this verdict. Units must match exactly; changing a unit label does not convert the measured number.

Exit 1 / execution_failed covers input, solver, configuration or publication failure. Cancellation is documented as exit 130 / cancelled but was not exercised in this Windows check. Cancelled and not-run cases cannot count as passes.

results.csv contains per-case/per-requirement rows; results.json records the definition, input/include and artifact hashes, effective settings and case data. The independent check verified agreement between the saved CSV/JSON, the executed source/include hashes and committed artifacts.

Write requirements without implicit tolerances

Use finite JSON numbers in SI units, such as 0.001 rather than the string "1m". Each requirement ID must be unique and every requirement applies to every case.

Use inclusive min/max bounds, or target with nonnegative absolute_tolerance and/or relative_tolerance. The target rule is abs(measured − target) <= absolute_tolerance + relative_tolerance × abs(target). An omitted tolerance is zero. Do not combine target with bounds; unknown fields are rejected.

Recover and preserve earlier generations

1. If the result directory already exists, choose a new generation name. The checked reuse attempt returned code 1 and preserved all previous generation bytes.
2. For a failed evaluation, correct the measurement name or exact reported unit. For an invalid numeric limit, replace engineering-suffix strings with finite JSON SI numbers.
3. Save the corrected definition and rerun to a new result directory. The checked recovery passed. Review provenance paths and design content before sharing a calculation record.