How we evaluate — methodology

This article describes how the case studies on this site are scored. It is a methodology prelude — it intentionally contains no numbers. The scores themselves are published with each case study, alongside the recording of the run that produced them.

The principle behind it is simple: the parts that can be checked mechanically are checked mechanically, and the parts that require legal judgement are judged by someone independent of the system, against an outcome decided in advance.

Every authority Acta cites is checked by code against the official source. The quoted text either matches the source verbatim or it is rejected. This is deterministic: it does not depend on a model's opinion, and it gives the same answer every time. It is the layer that makes “the citation is real and says what we say it says” a fact rather than a hope.[4]

The legal merits — did the analysis reach the right place, did it raise what mattered — are graded against a real-world outcome that was established before the run and withheld from the system. The grader is independent of the system under test. The comparison is Acta's output versus the pre-established outcome, scored against fixed categories rather than an open-ended impression.

We are deliberately careful about what these scores mean. They are a measurement under one method on a set of constructed matters, not a guarantee of any outcome on your matter. Some results are proxy-validated and pending live validation; where that is the case, the case study says so.

We are also honest about the trajectory: today's models leave a visible gap on the hardest work, the next generation will likely close much of it, and the one after will likely exceed it. The check that keeps the work honest does not move with the models, so that improvement arrives without loosening the guarantee.[5]