Math Challenge is open: the code, the plan, and the research
The repository, the whole plan and all 47 studies are public — including the studies that contradict the product, and the run where our own audit fleet got it wrong.
What is open
Two links. Both public today, both checkable in a browser without an account.
-
The repository
The product, the audit fleet, the migrations and the internal documentation live in one public repository. There is no second, tidied-up copy for outsiders.
-
The board
The whole plan sits on a public project board: 18 items with acceptance criteria you can check, from the three site phases to the thirteen product phases. Two of those eighteen are not work at all — they are the tensions we have not resolved, tracked in the open next to everything else.
All 47 studies, including the ones that contradict the product
Forty-seven studies, about 157,000 words, with numbered sources and declared limitations. They are published whole, and that includes the passages where the evidence runs against what we are building: the original brief asked for something addictive, and the research spent weeks arguing it down.
Claims we could not confirm against a primary source are marked [unverified] in the text — 18 marks across 14 of the documents. They stay visible instead of being quietly deleted, because research that hides its weak spots is not evidence, it is advertising.
The audit fleet, and what it is allowed to say
Thirty-one of the thirty-nine planned auditors exist today. Eight are deterministic and block every commit; among them: the seven locales being complete, secrets, bundle weight, the Ignia palette, and no free-text field in a child's table. The other twenty-three are adversarial LLM auditors, briefed to find the violation rather than to approve, and each one has to cite the decision or the study it enforces. A finding that cites an id that does not exist is discarded mechanically.
Findings are written as SARIF 2.1.0, the OASIS standard, and uploaded to GitHub code scanning by hand — this project runs no CI, and that is a decision, not a gap. Each finding lands anchored to a file and a line where anyone can read it.
What went wrong the first time we ran it
The first full run of the adversarial fleet produced two blocking verdicts. One of them was fabricated: the auditor cited a decision that really exists, and then claimed a file said versión where the file actually says versão. A real citation with invented evidence, and it came out classified as blocking. Half of the blocking findings in that run were noise.
The fix was not a better prompt. It was a deterministic layer that extracts every string the auditor claims to have seen and checks that it really appears in what the auditor was shown. If none of them do, the finding stops blocking.
What that layer still cannot do, said before anyone assumes otherwise: an auditor that paraphrases instead of quoting leaves no strings to verify. It catches literal fabrication, not wrong interpretation — which was the other failure in that same run, and that one has no deterministic fix.
The licence: AGPL-3.0
The repository carries a LICENSE file with the GNU Affero General Public Licence, version 3. It is strong network copyleft: anyone who runs this as a service has to publish their changes. That is the point. The item bank is the product — it is where most of the real work lives — and the promise this project makes is that it reaches whoever cannot pay. A permissive licence would let someone take the engine and the bank, close them, and charge for them.
What it costs, said plainly: the AGPL scares off companies that might otherwise contribute, and many forbid it by internal policy. That trade was made on purpose.
Why this is worth doing
Publishing the research is the strategy, not a gesture of transparency. The part that actually costs something is publishing the mistakes: the board still carries the note that phase F0 was marked closed with one of its own criteria — 0-RTT — never verified, and it only surfaced because someone asked again. A board that shows its own error is worth more than any statement of values.
Check it yourself
Clone the repository and run the fleet. Every figure on this page comes out of that command or out of the public board. None of it was written from memory.
git clone https://github.com/kilowatto/math-challenge
node audits/run.mjs