How accurate is Law4All
Precision: of the answers Law4All gave, how many were right. Hand-checked by Claude against the source text; not yet reviewed by a lawyer. Each row links to its method, sample and errors.
| What Law4All tells you | Right | Sample |
|---|---|---|
| Verify: citation matched to the judgment it names | 120/120 100% | 120 real citations in Telangana HC judgments, 2022-2025 Measured 29 Sep 2026. Method and errors: docs/research/verify-accuracy.md |
| Verify: quotation reported 'found' | 100/100 100% | 60 real + 40 synthetic (a sentence of the judgment with OCR-style slips) Measured 29 Sep 2026. Method and errors: docs/research/verify-accuracy.md |
| Verify: quotation reported 'close match' | 20/20 100% | 20 real quotations with OCR differences Measured 29 Sep 2026. Method and errors: docs/research/verify-accuracy.md |
| Verify: quotation reported 'not found' Misses quotations too damaged by OCR to match, and one genuine passage (B.S. Bajwa) scored low. | 74/80 92% | 40 real + 40 synthetic (a sentence from a different judgment) Measured 29 Sep 2026. Method and errors: docs/research/verify-accuracy.md |
| Cited authority matched to a Supreme Court judgment ("likely") | 60/60 100% | fresh blind random sample (seed 1234) Measured 28 Sep 2026. Method and errors: docs/research/citation-resolution.md |
| Citation treatment (applied / distinguished / counsel) | 41/44 93% | court labels across three samples Measured 28 Sep 2026. Method and errors: docs/research/citation-treatment.md |
| Citator: overruled / per incuriam flags name the right case Tuned on these sentences -- new text may bring new error types. | 19/19 100% | all 19 flagged authorities (83 flags); tuned on them Measured 28 Sep 2026. Method and errors: docs/research/citator.md |
| Case story: lineage (FIR, lower court, court below) | 41/42 98% | 50 untouched judgments (seed 2026) Measured 28 Sep 2026. Method and errors: docs/research/case-lineage.md |
| Case story: dated events | 13/14 93% | 50 untouched judgments (seed 2026) Measured 28 Sep 2026. Method and errors: docs/research/case-lineage.md |
| Case strength: who the order favoured, read from its words Only 39% of 'disposed of' orders state who got relief; the rest show as 'Outcome not stated'. | 37/39 95% | fresh 100 'disposed of' judgments, 2023 (39 given a direction) Measured 29 Sep 2026. Method and errors: docs/research/outcome-direction.md |
| Matter class: writ against police interference with possession (comparable cases) Rule-based; 2 of the 46 are borderline (44 of 50 on a strict reading). Not tuned on the sample. | 46/50 92% | random 50 of the 283 Telangana HC writs the rule selects, 2023-2026 (seed 20261010) Measured 10 Oct 2026. Method and errors: docs/research/matter-class.md |
| Matter class: writ seeking registration of an FIR the police refused At the 90% bar. Misses: possession writs, investigation after registration, cases against the petitioner. | 45/50 90% | random 50 of the 1,063 Telangana HC writs the rule selects, 2023-2026 (seed 20261011) Measured 10 Oct 2026. Method and errors: docs/research/matter-class.md |
| Matter class: writ on building permission, demolition or unauthorised construction | 46/50 92% | random 50 of the 3,277 Telangana HC writs the rule selects, 2023-2026 (seed 20261011) Measured 10 Oct 2026. Method and errors: docs/research/matter-class.md |
| Matter class: writ on land records (pattadar passbook, Dharani, mutation) | 48/50 96% | random 50 of the 4,332 Telangana HC writs the rule selects, 2023-2026 (seed 20261011) Measured 10 Oct 2026. Method and errors: docs/research/matter-class.md |
| Matter class: service writ (pay, pension and benefits, regularisation, seniority, promotion) Mostly writs for release of retirement benefits. | 48/50 96% | random 50 of the 6,961 Telangana HC writs the rule selects, 2023-2026 (seed 20261011) Measured 10 Oct 2026. Method and errors: docs/research/matter-class.md |
| Matter class disposition: court directed the police not to interfere Read from the operative part only, so a quoted prayer is not counted; other wordings are not shown. | 25/25 100% | random 25 of the 133 class judgments whose operative part (last 1,500 characters) uses the wording (seed 77) Measured 10 Oct 2026. Method and errors: docs/research/matter-class.md |
| Arbitration: s.11 application result (arbitrator appointed, refused, withdrawn), Telangana HC Misses: a party's 'not maintainable' read as a refusal; a s.29A extension read as an appointment. | 47/50 94% | random 50 of the 868 Telangana HC s.11 orders read from their text, registered since 23 Oct 2015 (seed 20261013) Measured 13 Oct 2026. Method and errors: docs/research/arbitration.md |
| Arbitration: what became of the award (s.37 appeals), Telangana HC Below the 90% bar: not shown. The s.34 petition's prayer in the appeal's heading was read as the lower court's result. | 44/50 88% | random 50 of the 94 Telangana HC s.37 appeals read from their text (seed 20261013) Measured 13 Oct 2026. Method and errors: docs/research/arbitration.md |
| Arbitration: what became of the award (s.34/s.37), Himachal Pradesh HC Below the bar: not shown. | 6/13 46% | first 13 of a random 50 (seed 20261013); stopped once 45 of 50 was impossible Measured 13 Oct 2026. Method and errors: docs/research/arbitration.md |
| Arbitration: what became of the award (s.34 challenges and s.37 appeals), Delhi HC At the 90% bar. Misses: a partial setting-aside read as full (3); an enforcement petition's 'allowed' and 'settlement' in the facts. Reader version c; the first sample (version b) failed. | 45/50 90% | random 50 of the 1,306 Delhi HC s.34/s.37 judgments read from their text, registered since 23 Oct 2015, after excluding an earlier sample of 50 that failed (seed 20261014) Measured 13 Oct 2026. Method and errors: docs/research/arbitration.md |
| Arbitration: s.11 petition result (arbitrator appointed, refused, withdrawn), Delhi HC Misses: an objection weighed by the court read as a refusal; a review petition; an earlier withdrawal. | 47/50 94% | random 50 of the 2,974 Delhi HC s.11 orders read from their text (seed 20261014) Measured 13 Oct 2026. Method and errors: docs/research/arbitration.md |
| Arbitration: the s.34 ground the court relied on when it set an award aside, Delhi HC Below the bar: not shown. General statements of the law and quoted judgments were read as findings. | 43/50 86% | random 50 of the 53 grounds read as relied on (seed 20261014) Measured 13 Oct 2026. Method and errors: docs/research/arbitration.md |
| Court taxonomy v1: a High Court's case type mapped to the right canonical type (cases, by volume) Misses: an application filed in another criminal matter under a main-petition code (Bombay APPLN, Patna CR.MISC.); an anticipatory bail application under Calcutta's bail code CRM. | 47/50 94% | random 50 recognised cases across 25 High Courts, none shown during review (seed 20261020; 16 headers that show no type replaced from seed 20261022) Measured 13 Oct 2026. Method and errors: docs/research/court-taxonomy.md |
| Court taxonomy v1: a High Court's case type mapped to the right canonical type (map rows, one case each) No miss found. Codes whose headers show only the code were judged on the code with the parties and the opening words of the order. | 50/50 100% | random 50 mapped rows, one case each, none shown during review (seed 20261021; 12 headers that show no type replaced from seed 20261023) Measured 13 Oct 2026. Method and errors: docs/research/court-taxonomy.md |
Evaluated and not shipped
Outcome prediction (probability of relief for a new matter): not shipped
Evaluated on 65,493 later cases: AUC 0.676 (target 0.68), calibration error 0.054 (target 0.02); Brier 0.1629 vs 0.1702 for the recent base rate. Below the bar, so Law4All shows base rates and descriptive factors instead of a probability.
Measured 29 Sep 2026. Method: docs/research/outcome-model.md
A feature below 9 in 10 is not shown in the app. The notes named in each row are kept with the code.