Infrastructure, not theater
Accuracy · FP / FN honesty
Published 2026-07-24. Engine: Scout 1.0 (client corpus + heuristics; live API when configured). We measure error classes instead of claiming “hallucination-free.”
What we publish (this eval set)
| Metric | Rate | n |
|---|---|---|
Trap recall (fabricated / void cites correctly flagged) High-signal traps (impossible year, fake reporter, demo fabrications). Not a claim about all LLM hallucinations. | 95% | 38/40 |
False verify (fabricated cite labeled Verified) On the published trap set. Live primary-index path is required before claiming broader coverage. | 0.0% | 0/40 |
False fail on known real authorities (corpus holdout) Rate of known-good cites incorrectly labeled Could Not Verify in offline Scout. | 3.3% | 4/120 |
Needs Review share on mixed practice memos Ambiguous is intentional. incomplete form or unconfirmed existence routes to human review. | 15-45% | mixed memos |
Definitions (FP / FN for citation verify)
- False verify (worst): a fabricated or void cite labeled Verified. Investors should care about this first.
- False fail: a real authority labeled Could Not Verify. Annoying; recoverable with human review.
- Needs Review: intentional middle state when form is incomplete or existence is unconfirmed. Not counted as Verified.
- Trap recall: share of known fabrications correctly pushed to Could Not Verify (or hard fail path).
Primary-source hit rate (per scan)
Every verify reports how many citations hit a primary public index versus corpus-only or form/heuristic only. That number is on the result card - not a marketing footer.
- CourtListener (Free Law Project) · id: courtlistener
- Caselaw Access Project · id: cap
- GovInfo / U.S. Code · id: govinfo
Each verify reports X/Y cites that resolved against a primary public index. Offline Scout reports 0% live index and shows corpus/form separately.
Offline Scout honesty: live primary hit rate is 0% until the API is configured. Corpus matches are reported separately as a “real-source path,” never as live CourtListener hits.
What we refuse to claim
- Not legal advice. Not KeyCite / Shepard’s. Existence ≠ good law, correct holding, or quote accuracy.
- Offline Scout never pretends live primary when the API is down.
- We do not publish “hallucination-free” percentages.
How to reproduce the demo trap
On the homepage, click Catch a fake cite. You should see Martinez v. Northern Pacific Holdings as Could Not Verify (not Ambiguous, not a nearby case), Celotex/Iqbal as Verified only if a live primary index retrieved them, and § 1983 as Needs Review unless GovInfo retrieval exists. Export must not print a verification rate.
Ready to verify a draft?
Free beta · 3 checks/mo · Confidential Mode on by default.