Is our system standing on its own? -- live scoreboard

We are building toward one goal: our own local system does the work, and Claude (the paid AI) is needed less and less. This scoreboard is measured by the system itself — it runs its own tests and reports what actually happened. It cannot be sweet-talked: anything not truly measured says NOT MEASURED YET instead of pretending.

The scoreboard

Can we automatically check whether generated code is correct?

✓ YES Our checker compiles the code, runs it, and compares the answers. It passed all its self-tests — including the important one: it correctly REJECTS wrong code instead of approving everything.

Can the system find a bug, ask permission, fix it, and undo the fix?

✓ YES Proven end to end: it spotted a planted bug on its own, wrote the finding to an approval queue (nothing changes without a person saying yes), applied the fix, re-tested it, and restored the original perfectly.

How much of the work already runs with zero Claude cost?

70% SO FAR Goal: 100%. On our test set, 70% of tasks were answered by our own local model and verified correct by our own checker — no Claude involved, no tokens spent. The rest still get sent to Claude.

measurement 1 — local model size 0.5B

38%

measurement 2 — local model size 1.5B

54%

measurement 3 — local model size 1.5B

61%

measurement 4 — local model size 1.5B

70%

The jump came from upgrading the local model. Next jumps: teach it more skills, and train it on exactly the tasks it still hands to Claude.

Does the code test suite itself work?

✓ YES Health check: known-good solutions pass all tests (so a failure really means the code is wrong, not that the tests are broken).

Is our own compiler as fast as the industry standard (gcc)?

ALMOST — 1% SLOWER Official lap from the compiler team's own ledger: 1% slower than gcc on the toughest scalar loop (and already faster than gcc on one benchmark). The register-allocator work now in progress is expected to close it; every new lap updates this row automatically.

Graded by the industry bug-fixing benchmark's own rules, how does our fixer score?

✓ 71% RESOLVED SWE-bench is the field's standard bug-fixing test: patch a real issue, and it only counts if the broken tests pass AND nothing that worked before breaks. We graded our own fix loop by that exact cheat-proof rule — it resolves 71% of our bug set. Honest scope: this is our own sovereign set under SWE-bench's rules, not its Python problems (a no-Python system can't run those); we claim the same standard, not parity with agents that run the real set.

Can our local AI write working code by itself yet?

YES for small fixes — with honest limits. The local model now runs the whole repair loop on its own: it reads the test results to find which function is broken, pulls that function's source, writes the corrected code, and the independent checker confirms the repair — compiled, ran, exact right answers, zero Claude anywhere in the content. Proven on different bugs in different functions, including a repair shape it was never shown. Writing whole programs from scratch is still beyond it, so every remaining failure is saved automatically as training material. The checker is what keeps this honest: code only counts when it runs and produces exactly the right answer.

Updated automatically each time the system re-measures itself. This page only translates the measurement records into plain English — the numbers come from the machine, not from anyone's opinion. machine-readable version