Expert-reviewed scenarios, a public dataset, a failure history, and downloadable evidence behind every fit verdict.
A public, continuously-validated evidence base for the ItemFits spatial solvers: a frozen case corpus, a public failure ledger, and a downloadable dataset.
What this is
Every fit verdict ItemFits returns comes from deterministic geometry, not a model. This page is the public evidence base: a frozen corpus of scenarios whose expected verdict was fixed by human review before the solver ran, re-checked against the deployed build, with a public history of every disagreement. It measures agreement with an expert verdict on a curated corpus, not real-world fit success. Current corpus, growing.
Agreement by tier
| Validation tier | Cases | Agreement |
|---|---|---|
| Expert-reviewed | 6 | 100% |
| Direct geometry | 66 | 100% |
| Edge cases | 2 | 100% |
| Regression cases | 18 | 100% |
| Historical bugs | 4 | 100% |
Each tier reports the share of its scenarios on which the deployed solver matched the frozen expected verdict, with the scenario count it is drawn from. Agreement is comparison against an expert verdict, not a real-world success rate.
Coverage
| Solver | Cases |
|---|---|
| opening | 45 |
| hallway | 13 |
| container | 12 |
| room | 7 |
| elevator | 7 |
| turn | 8 |
| hybrid | 4 |
96 scenarios across 7 solver families. Nondeterministic motion and maneuver solvers (RRT) are excluded by design. The vehicle, stair, and capacity solvers ship in production but are not yet represented in this corpus — they are outside the figures on this page. Current corpus, growing.
Failure history
| Date | Scenario | Expected | Actual | Resolution |
|---|---|---|---|---|
| - | Prod fit-check f4b5b669-f4b0-48b9-ac0f-8c6efb550e69 (user_feedback='inaccurate'). Every leaf solver returned FITS; the hybrid composition returned DOESNT_FIT carrying a POSITIVE +635mm margin. Root cause: faceToFloorFootprint switched on `orientation.face`, which only the opening solver emits — hallway/turn/stairs/elevator legs fell to a default branch that produced a {0,0} floor footprint, which then eliminated every room orientation (no_valid_orientation_after_path_filter) and fabricated a confident DOESNT_FIT. Fixed by deriving the footprint from the carried (across × along) axes for every path-solver orientation shape, and by failing OPEN (no orientation filter) rather than closed when a leg's shape cannot be read. Investigation: docs/investigation-2026-07-06-turn-multistep-composition-doesnt-fit.md. Regression guards: hybrid-0002 + hybrid-0003 (a genuine block must still block) and tests/hybrid-path-footprint-contract.test.js. | n/a | n/a | solver bug · fixed |
| - | Found while fixing hybrid-0001: the fabricated proof card it produced ('a 3659mm item in a 3658mm room' — an item dimension 1mm larger than the room, invented so a failing axis card could be rendered) was the fingerprint of a SECOND wrong verdict, not merely bad presentation. A 7'2" bookshelf that can only clear a 6'8" door lying flat has a CARRY footprint of (width × HEIGHT); the room and capacity solvers rest a rigid item on its (width × depth) base, always. Feeding the carry footprint in as a resting-orientation whitelist matched no orientation, eliminated every placement, and returned DOESNT_FIT at HIGH confidence for a bookshelf with metres of floor to spare. Fixed by only constraining the resting orientation when the item is actually carried in its resting pose (otherwise it is turned upright inside — surfaced as an explicit assumption), by failing OPEN in room-solver on a pose-basis mismatch, and by never fabricating a dimension or reporting a -1 sentinel as a clearance. The proof contract requiring a limiting card with a real margin was the signal the verdict was unjustifiable — satisfying it required lying. Regression guards: hybrid-0004, tests/hybrid-path-footprint-contract.test.js, src/lib/ai/solvers/__tests__/room-proof-steps.test.js. | n/a | n/a | solver bug · fixed |
Every solver disagreement with a correct scenario, and every later correction to a scenario, recorded as a public audit trail.
Methodology
Each scenario pairs structured item and space dimensions with an expected verdict frozen by human review before the solver runs.
The honesty contract
Every published figure carries its sample size. Anything below the threshold is suppressed, never shown as zero.
Validated against the deployed solver build on: July 2026 · 96 scenarios checked
Cite this study
ItemFits Public Solver Validation Program. Expert-reviewed scenarios validated against frozen expected verdicts; public dataset and failure history.
Take the data
The figures on this page, as a machine-readable file. Free to reuse with attribution to ItemFits. Permalink: https://itemfits.com/data/solver-validation
Enter your item and space above — get a verdict in seconds, then test the what-ifs.
Open the fit calculator