forked from 1b-agent/fieldcraft-upstream
de8c0e2700
* docs: verification and validation against theory, experiments and published benchmarks; fixtures Owner decision 2026-10-13: test against experimental and theoretical data; results published by vendors (e.g. Abaqus benchmark and verification manuals) may be used as cited reference values for our own models of the same problems; once a case matches, its result becomes a fixture so fidelity cannot regress. Commercial products are still never run and their files and text never copied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NmAttHhUZTokgPeEJia8sm * docs: V&V policy in AGENTS.md, ADR 0001 and the roadmap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NmAttHhUZTokgPeEJia8sm --------- Co-authored-by: jyviko <2501648+jyviko@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2.2 KiB
2.2 KiB
Verification and validation
Every solver capability is checked against three kinds of reference, and every accepted result is frozen so it cannot drift.
| Kind | What it checks | Sources |
|---|---|---|
| Theory | The code solves the equations right | Closed-form solutions (beams, plates, Lamé, Kirsch, waves, shocks, Poiseuille…), patch tests, convergence rates, mesh invariants |
| Experiment | The model predicts reality | Measurements published in papers and reports (Taylor bar, cavity and cylinder flows, FSI experiments…) |
| Published benchmarks | Agreement with established codes on standard problems | NAFEMS benchmarks, vendor benchmark and verification manuals (e.g. Abaqus Benchmarks and Verification manuals), other published reference solutions |
Rules:
- Each case is our own model of the problem, built from the published description (geometry, material, loads, boundary conditions). We never run a commercial product and never copy its input files, manual text or figures; only the published numbers are used, with a citation.
- Each case lives in
tests/oracle/<case>/:recipe.json— the model as ModelCraft commands (or a SolidCraft recipe for geometry);expected.json— the reference values withkind(theory | experiment | benchmark),source(citation: document, table or figure, page),quantity,value,toleranceand the reason for that tolerance (mesh, measurement scatter, model differences);fixture.json— ModelCraft's accepted result, written bycargo xtask oracle --bless <case>only after the case passes againstexpected.json.
cargo xtask oracleruns every case, compares withexpected.json(pass/fail with error) and withfixture.json(any change beyond the fixture tolerance fails CI, even if the reference still passes), and writesdocs/oracle.mdwith the table of cases, sources, errors and status.- A fixture is updated only deliberately (
--bless), in a PR that says why the result moved (a better element, a finer default mesh), never to hide a regression. - The theory suite is the gate for every PR; experiment and benchmark cases run in CI too, with their cited tolerances.