An open-source library and command-line tool that treats a correction to an AI judge like a change to code.
What works today
- A correction is written as a rule that carries the reviewer’s reason. It is not pasted into the prompt as an example.
- A gate runs the rule against every case the system has ever been checked on, and blocks it if it breaks an answer that was right.
- A second person approves the rule. One command reverts it.
- Every step lands in a hash-chained log that an auditor can verify offline.
- The demonstration runs offline.
Where it came from
The first version of this project learned a rule from one supervisor’s correction. The rule broke calls the system had been getting right, and the project’s own evaluation reported an improvement. That failure is published in the repository, and it is the reason the gate exists.
What the benchmark showed
The hypotheses were registered before any model output existed, and the departures from the plan are logged.
- Rules that passed the gate broke no verdict that had been right. Every way of putting the corrections into the prompt broke some.
- One hypothesis failed. Retrieved examples were more accurate than the rules, by more than the margin set in advance.
- The reason is that corrections come in two kinds. A misapplied policy becomes a rule. A misread fact does not, because the rule works on what the judge already read.
Limits
- One synthetic domain and one judge in the comparison. The reference labels come from a written policy, not from people who disagree.
- The log shows tampering. It does not prevent it.
Why it is in the lab
It is the open, runnable part of the research on evidence and authority: who may change what a machine decides, and how that change is checked before it takes effect.