Public reference tests: analytic and FreeCAD expected values (#310)
The oracle no longer uses Fusion measurements. tests/oracle/<case>/ now holds the 76 recipes (our own format, with recorded measurements and tool-specific fields removed) and an expected.json per case: - 55 analytic cases: closed-form values from tools/oracle-analytic/ (stdlib Python, formulas in each case's "method"). - 21 FreeCAD cases: the same recipe built with FreeCAD's public scripting API by tools/freecad-oracle/ (an external program, not a dependency), with a reference.stl for 20. solvecraft-cli oracle compares body count, volume, area, centre of mass, bbox, face counts where the geometry fixes them, and, against reference.stl, the sampled Hausdorff distance and the principal moments (spatial.rs from #301). cargo xtask oracle reads tests/oracle, writes docs/oracle.md with the source of each value, and supports known_failure marks for tracked bugs. CI runs it on every PR; a manual workflow regenerates the FreeCAD values. step-corpus, seam normalisation and the through-all emulation, which only existed to compare with Fusion, are removed. 75/76 pass; 75-sm-bracket-holes is a known failure (#296). Claude-Session: https://claude.ai/code/session_01RjGw4gA5ZvKrU8MQP6sBgT Co-authored-by: bherbruck <bherbruck@herbrucks.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
204 files changed
+27106
-932
No files matched your search
@@ -122,9 +122,41 @@ jobs:
|
||||
if: always()
|
||||
run: sccache --show-stats
|
||||
|
||||
oracle:
|
||||
name: Reference tests
|
||||
needs: changes
|
||||
if: needs.changes.outputs.native == 'true'
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 60
|
||||
env:
|
||||
CARGO_INCREMENTAL: "0"
|
||||
RUSTC_WRAPPER: sccache
|
||||
SCCACHE_GHA_ENABLED: "true"
|
||||
SCCACHE_IGNORE_SERVER_IO_ERROR: "1"
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
|
||||
with:
|
||||
key: oracle
|
||||
- name: Set up compiler cache
|
||||
uses: mozilla-actions/sccache-action@fc920bf0ec8de6ee65d409111f7ec508035751ba # v0.0.11
|
||||
with:
|
||||
version: v0.16.0
|
||||
- name: Build the command line
|
||||
run: cargo build -p solvecraft-cli --profile ci --locked
|
||||
- name: Replay tests/oracle and compare with expected.json
|
||||
run: cargo xtask oracle --cli target/ci/solvecraft-cli
|
||||
- name: Upload the report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||
with:
|
||||
name: oracle-report
|
||||
path: docs/oracle.md
|
||||
|
||||
# Preserve the existing check name for branch protection, including docs-only PRs.
|
||||
ci:
|
||||
needs: [changes, checks, clippy, tests]
|
||||
needs: [changes, checks, clippy, tests, oracle]
|
||||
if: always()
|
||||
runs-on: ubuntu-24.04
|
||||
steps:
|
||||
@@ -135,12 +167,14 @@ jobs:
|
||||
NATIVE: ${{ needs.changes.outputs.native }}
|
||||
CLIPPY: ${{ needs.clippy.result }}
|
||||
TESTS: ${{ needs.tests.result }}
|
||||
ORACLE: ${{ needs.oracle.result }}
|
||||
run: |
|
||||
test "$CHANGES" = success
|
||||
test "$CHECKS" = success
|
||||
if [ "$NATIVE" = true ]; then
|
||||
test "$CLIPPY" = success
|
||||
test "$TESTS" = success
|
||||
test "$ORACLE" = success
|
||||
fi
|
||||
|
||||
wasm:
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# Optional, run by hand: rebuild the reference cases whose expected values come from FreeCAD
|
||||
# (tools/freecad-oracle/) and show what changed. FreeCAD runs as a separate program on the runner;
|
||||
# it is not a dependency of SolidCraft. Nothing is committed: review the diff and the uploaded
|
||||
# files, then commit them in a pull request if the change is intended.
|
||||
name: FreeCAD reference values
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
cases:
|
||||
description: "Case directories (space separated); empty = every case whose source is FreeCAD"
|
||||
type: string
|
||||
default: ""
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
regenerate:
|
||||
name: Regenerate FreeCAD expected values
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 60
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
- name: Install FreeCAD
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y --no-install-recommends freecad
|
||||
freecadcmd --version || true
|
||||
- name: Rebuild the cases
|
||||
env:
|
||||
CASES: ${{ inputs.cases }}
|
||||
run: |
|
||||
if [ -z "$CASES" ]; then
|
||||
CASES=$(grep -l '"source": *"freecad' tests/oracle/*/expected.json | xargs -n1 dirname | tr '\n' ' ')
|
||||
fi
|
||||
# shellcheck disable=SC2086
|
||||
tools/freecad-oracle/run.sh --write $CASES
|
||||
- name: Show the changes
|
||||
run: git diff --stat && git diff -- tests/oracle/*/expected.json
|
||||
- name: Upload the regenerated files
|
||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||
with:
|
||||
name: freecad-expected
|
||||
path: |
|
||||
tests/oracle/*/expected.json
|
||||
tests/oracle/*/reference.stl
|
||||
Reference in new issue
Block a user