Yash Raj Pandey - Writing

The Work You Don't Do: Losing Two Optimization Competitions the Same Way

Fri, 14 Aug 2026 00:00:00 GMT

TL;DR

On August 5 I published a post-mortem of mlx.fast, an optimization competition I lost. It contained this sentence about the week's biggest single gain:

They asked which timing window was being billed for that compile, and I did not. That question was the whole optimization.

The same day, I cloned the repository for the next competition in the series. It is the Lighter Prover Challenge, a zero-knowledge proving benchmark. Nine days later I had one rejected submission, five measured dead ends, and no promotions. The frontier went from 3.36 to 30.74 without me.

The lever that won it moves circuit construction out of the timed process and into the untimed build job. I measured that construction at 11.24 seconds. I named it the second largest cost in my own profile. Then I spent an agent's time making it 2 seconds faster instead of asking which window was being billed for it.

Two competitions, zero promotions. Both were won by people who removed work or moved it outside the measured window. Both times I tried to make the same work faster.
My local numbers pointed the wrong way. My submission measured +17.8 percent locally and priced at -10.3 percent on the ranked machine. The evidence that would have predicted that was already in my data.
I found a real correctness trap that the public test fixture cannot expose. The ranked run confirmed my fix was right. It was the best thing I did all week and it was worth nothing on the leaderboard.
Five optimizations, all measured with interleaved repeats, all negative. One was 52 percent slower and taught me the most.

The setup

The Lighter Prover Challenge is run by EigenLabs on the same platform as mlx.fast. You optimize a Plonky2 prover for a DEX state-transition circuit. Score is verified proving throughput in transactions per second. A trusted, code-signed verifier owns the clock. It runs your worker in a macOS sandbox, checks the final recursive proof, and writes the score. You may edit the circuit crate, the bench crate, and a vendored copy of Plonky2. The rest of the harness is protected.

Two details shaped the whole week.

Ranked scoring runs on a private fixture on an Apple M4 Pro with 48 GB. My machine is an M4 Max with 64 GB and a much larger GPU. Once again I was not measuring on the machine that scores me.

The second detail matters more. The only fixture I can run locally contains zero active transactions. It is 500 empty padding transactions. The ranked fixtures contain 500 real ones.

A correctness trap the local fixture cannot show

Two other solvers had submissions in flight that proved the 52 per-chunk transaction proofs in parallel. I read their diffs. Both did the same thing. They pinned every chunk's old_jump field to the initial value and threw away the real one.

That only works if a piece of accumulator state never advances. In the circuit, a transaction counts as padding exactly when its type is zero. The accumulator advances only on active transactions. So on the public fixture the state never moves and the shortcut is correct. On the ranked fixture every transaction is active, the state moves on every one, and the chain-folding circuit constrains the link:

JumpStateTarget::connect(&mut circuit.builder, &block.jump, &current_tx.old_jump);

Their approach passes every local run and fails on the ranked machine.

So I built the correct version. It is a native, out-of-circuit implementation of the state transition, so every chunk gets its true value and all 52 proofs become independent. The public fixture cannot exercise it, so I validated it two ways. I wrote a randomized equivalence test against a real circuit built around the in-circuit function. I also added a runtime assertion comparing my computed value against the proof's public input.

The ranked run confirmed it. My submission returned a real score over 2,500 active transactions. Every proof verified and the assertion never fired. The part I could not test locally was right.

It was also irrelevant. Another solver reached the same problem later and solved it better. They split witness generation from proving and read the value straight out of the generated witness. The circuit's own generators produce it, so there is no second implementation to keep in sync and no transcription risk. My equivalence test existed to defend a design choice that a better design does not need.

The local number pointed the wrong way

My parallel prover measured +17.8 percent locally. The ranked machine priced it at -10.3 percent against the same baseline it was built on.

I could have predicted that. Two numbers were sitting in my own profile.

The first was user 1351s / real 116s. That is about 11.6x parallelism across 16 cores. The CPU was already close to saturated before I added anything.

The second was a concurrency sweep of 4, 6, and 8 in-flight proofs. It came back flat: 79.3, 79.9, and 80.0 seconds.

I read that flat sweep as "4 is safely at the plateau, and it uses the least memory." It actually meant there was no headroom at all. Adding CPU concurrency to a saturated machine buys nothing locally. On a ranked host with fewer cores it costs contention, so the sign flips.

Another solver's GPU work went the other way in the same period. It measured +18.3 percent locally and priced at +25.8 percent on ranked. Moving work off the CPU frees the resource that is scarcer on the smaller machine. Scheduling more work onto the CPU does the opposite.

That gave me a rule I used for the rest of the week. Before spending a submission, I have to explain why a gain survives on a machine with fewer cores, not just show a local delta. The rule stopped three more bad submissions. It arrived one submission too late.

Five dead ends, priced

After the rejection I stopped guessing and started pricing. Every number below is a mean of interleaved repeated rounds. Run-to-run noise on GPU-heavy builds is about 3 percent, so a single sample proves nothing.

Change	Result
Lower the GPU Merkle threshold	0.14 percent spread, inside noise
Widen the GPU buffer pool to 2 or 4 sets	2 to 5 percent slower
NEON instruction scheduling in Poseidon2	0.35 percent slower, n=8
Non-blocking GPU acquire with CPU fallback	52 percent slower
Opportunistic GPU batching	8 percent slower

The 52 percent one taught me the most. The GPU path allowed one user at a time. With about 25 proving threads, threads that wanted the GPU slept on a condition variable. My profile showed 17,000 samples of idle waiting. That looked like obvious waste. My fix was to let a thread that cannot get the GPU build its tree on the CPU instead.

The waiting is not waste. The GPU is so much faster for the large trees that qualify that waiting for it beats doing them on the CPU by a wide margin. I had built a story about the mechanism and the machine disagreed with me by 20 seconds.

Those five results have a shape. Widening GPU concurrency hurt. Declining GPU work hurt. Batching GPU work hurt. I concluded that the GPU was compute-bound and that the whole offload direction was closed, and I stopped.

That conclusion was wrong, and reading the winning submissions afterwards showed me how. Another solver later moved a completely different kind of work onto the same GPU. They offloaded the quotient gate constraint evaluation, which was the second largest cost in my own profile, and it was worth 75 percent. The CPU kept evaluating the remaining gates while the GPU worked on those, so the two were not competing.

What I had actually measured was that the Merkle hashing offload was saturated. I generalized that to "the GPU is full." The question I should have asked was which CPU work could move there next, not whether the GPU had room for more of the same work.

What actually won

At the end I went and read the winning tree, which is what I should have done at the start.

The top lever is a build script. Its own doc comment says it plainly:

Compilation runs in the benchmark's untimed CI job, so the multi-second circuit construction here is free; the scored worker process then loads the blobs in a fraction of the build time.

The winners build the five proving circuits at compile time, serialize them, and embed the bytes in the binary with include_bytes!. The scored process deserializes instead of constructing. The same idea shows up again in the same tree, where the dummy recursion proofs are precomputed and committed as binary blobs.

The best example of the whole pattern came later, from the solver with the most promotions. The prover compiles its Metal shaders at runtime. On a developer machine that costs about 20 milliseconds, because the operating system's shader cache absorbs it. They measured it under the benchmark's own sandbox profile and it cost 2.751 seconds per worker, because the sandbox denies writes to the shader cache directory, which turns the cache off entirely. Five fixtures per ranked run means about 13.8 seconds of scored time that nobody had priced. They proved the mechanism with a three-way control: cache allowed 21.7 milliseconds, writes denied 2,751 milliseconds, reads and writes denied 2,540 milliseconds.

Their fix was to lower the eight compute pipelines on eight threads and start that work in the first statement of main, so it overlaps circuit loading instead of blocking the first GPU use. That took startup from 2.751 seconds to 1.132. It was worth 27 percent.

That cost is invisible if you profile on your own machine. It only appears if you measure inside the sandbox the scorer actually uses.

The second winning family is a packed field type for Apple silicon. I had ruled this out early. My reasoning was that NEON has no 64-by-64-to-128 widening multiply, while a scalar multiply plus high-multiply pair already produces one in two instructions. So a vectorized version has to lose.

Their comment shows what I got wrong:

Multiplication interleaves both lanes in one assembly block to hide the latency of AArch64's scalar mul and umulh instructions.

They do not vectorize the multiply at all. They interleave scalar instructions in one assembly block to expose instruction-level parallelism. Then they wrap it in the field abstraction so it reaches the existing packed constraint-evaluation paths, which were the second largest cost in my own profile. I asked whether the instruction set could do the multiply in parallel. The useful question was whether the abstraction could expose parallelism and reach the code that matters.

My own NEON attempt used that same technique. I applied it to one layer of one hash function instead of the field type that feeds everything, and measured it at 0.35 percent slower.

Where I had the answer and did not use it

Early in the week I added timing instrumentation and produced a stage breakdown of a 116.56 second run. Circuit construction came in at 11.24 seconds, 9.6 percent, the second largest line item.

I had also read the benchmark documentation. It describes the ranked workflow in detail, including a first job that builds the candidate and a second, separate job that runs the timed benchmark. I had quoted parts of that document back to myself while reasoning about the sandbox.

Then I wrote a task brief to parallelize circuit construction across its dependency graph. It worked. It took 11.24 seconds down to 9.17. That is a 2 second gain on a line item that could have gone to zero.

At 116 seconds the work was under 10 percent of the run. As the frontier sped up and runs fell toward 30 seconds, the same fixed cost became roughly a third of the score. I made the work cheaper instead of asking whether the clock should have been running during it. I did that nine days after publishing a post-mortem whose main lesson was that exact question.

What transfers outside the competition

I read the winning submission notes afterwards. Most of what is in them is not competition-specific at all. These are the ones I expect to use again on ordinary performance work.

Multiply count is not the cost model. One promotion reordered the addition chain for x^7. Same number of multiplies, shorter dependency chain, faster. The dependency graph is the thing to look at.

On AArch64, SIMD is often the wrong tool for 64-bit integer math. A scalar mul plus umulh produces a 128-bit product in two instructions, and no NEON instruction beats that. What does help is interleaving several independent scalar chains in one assembly block to fill the pipeline. The winners wrapped that in the codebase's existing packed-field type, so every consumer of that abstraction got faster at once. I dismissed this whole family because I asked whether the vector unit could do the multiply, which was the wrong question.

Delete correction rounds your data can never reach. Modular reduction is written for the general case. Prove the range your workload actually produces and whole branches disappear. Two of those were worth real points.

Let accumulators carry lazily. Normalize once at the end rather than at every step.

Bound your in-flight window and name it in a constant. The winning scheduler kept exactly two proofs in flight. Not "as many as the thread pool wants," which is what I did and what cost me a submission.

Run ahead by one. Split generation from consumption so item N+1 is produced while item N is consumed. That hid almost all of the generator cost.

Offload asynchronously with an all-or-nothing fallback. Start the GPU work, let the CPU do the rest concurrently, and if anything fails recompute the whole thing on the CPU. Performance changes, semantics never do. That structure is what makes aggressive offload safe enough to ship.

Build a differential mode behind an environment flag. The frontier shipped one that recomputes the entire result with the optimization disabled and asserts equality. Build it once and every later optimization becomes cheap to verify. I wrote a bespoke equivalence test for one change instead, and got nothing reusable from it.

Two smaller ones I liked. Test field code at noncanonical boundary values, because an implementation that silently canonicalizes passes random testing. And release memory where the last reader finishes rather than at the end of scope.

Then I read all 145 winning notes

Every promoted submission carries a required public write-up. I fetched all 145 from the lighter competition and all 146 from mlx.fast and read them properly. I expected a catalog of clever optimizations. About half of them are not that.

Counting from the titles of the 145 promoted lighter submissions:

What the promotion was	Count
A redraw of an unchanged or near-unchanged tree	18
Restoration or composition of work that already existed	47
A genuinely new mechanism	the remaining 80

The redraws are the part I did not see coming. Real titles: "Redraw 300 of the 27.23 frontier." "Round 21, sixth draw on a8ca9c9." "Tip redraw marker 201." And one submitted by a solver who credits a different solver for the method: "Pure tip multi-draw for #1 (pepedesigner playbook)."

The ranked score is noisy. Resubmitting the current frontier unchanged sometimes measures above the recorded frontier value, and that counts as a promotion. People track this with round numbers and markers, and some of them model the draw distribution of the tip itself.

It is worth being precise about why this works here and not in the other competition. In mlx.fast the noise caused validity failures, so you got no score at all and had to resubmit; the payload still had to be better than the baseline. In lighter the noise is in the score itself, so an unchanged tree can win. Those are different games and I did not check which one I was in.

The restorations are more interesting to me. A promotion replaces the editable paths wholesale, so any candidate built on a slightly stale base silently reverts everything that landed in between, and nothing warns anyone. Forty-seven promotions are people finding that wreckage and putting it back. One note states the rule plainly: diff your base against the tip you are about to displace, not only the tip you built on. That check takes minutes and I never ran it once.

The mlx.fast corpus showed a different pattern I also missed. Instead of shipping "convert all 40 layers", solvers ship a band at a time: first 16 layers, then 16 to 24, then 24 to 32, then finish. One mechanism, four or five promotions. Each step only has to beat the current bar, each is small enough to verify, and each is a separate draw. I would have shipped that as one submission and taken one result.

So the honest accounting of my week is worse than I first wrote. I spent all of it on the category that is hardest and slowest, while roughly half the available promotions were going to people doing diffs and resubmissions.

Why I keep missing this

It would be easy to call this carelessness. I do not think that is what it is. The failure was the same in two competitions with different domains and a published lesson in between.

A profiler answers the question "where is the time going." That question quietly assumes the time should be going somewhere. Every tool in the performance toolchain is built to make work cheaper. None of them ask whether the work belongs inside the measured region, or whether the output needs it at all. You have to ask that yourself, before you open the profiler. Afterwards the profiler's framing is already in your head.

The two winning classes in these competitions are the two the toolchain cannot suggest.

The first is moving work out of the measured window. Precompute it, embed it, or hoist it into warmup.

The second is proving you do not need the work. In mlx.fast, the durable frontier came from a bounded pass that proves which vocabulary rows can contain the answer, then evaluates only those. The output is provably identical, so the format of everything else stops mattering.

Both come from asking what the scoring actually requires.

Delegating the work, and verifying it

I ran this the way I run most substantial work now. I orchestrate, write task briefs, and delegate implementation to a coding agent, then verify the result. Two things happened that are worth reporting.

The agent reported a set of GPU correctness tests as passing. They had never run. The sandbox it worked in exposed no Metal device, and those tests fail closed with "no Metal device." The raw log said so honestly while the summary said the suite passed. I only caught it because I re-ran everything on the machine with the GPU. Three of five passed there and two failed for an unrelated reason.

The second one was better. A later change built cleanly, passed every test I could run, and produced a byte-correct 196,008 byte proof. It also aborted the process at exit. A metrics hook was running after thread-local storage had been destroyed and calling std::thread::current(), which panics in a context that cannot unwind. The proof was already written, so any check that only looked at the output would have called it a success. I found it because I ran the binary directly and looked at the exit status.

Neither of these is the agent being careless. They are the same class of problem as the rest of this post. A delegated task gets verified against the checks you specify. My checks did not include "the process exits cleanly" or "the GPU tests actually ran on a GPU." So I write acceptance conditions that cannot be satisfied by skipping, and I re-run the verification myself on hardware that can actually perform it.

What I actually learned

Map the timer before you profile. Write down which process is timed, from which instant to which instant, and what runs in an untimed build or warmup phase. Anything expensive inside the window that could live outside it is the first thing to look at, before any algorithmic work.
Map the output requirement too. Separate what must be bit-identical from what only has to be provably equivalent. You cannot see the "prove you do not need the work" class until you do.
A flat sweep means saturation, not safety. Three configurations tying is evidence that there is no headroom. I read it as evidence that the middle one was a safe choice.
Two well-reasoned negatives in a row means go back and re-read the problem statement. I ran five. The answer was in the documentation the whole time.
Prefer changes that reduce work over changes that reschedule it. Rescheduling wins are the ones that flip sign between machines.
Latency is a real cost in a race. Two of my implementations were finished and correct, and someone else shipped the same idea while mine sat in the validation queue. With a one-hour queue and one submission in flight at a time, second place scores zero.
Learn the noise distribution of the scoring rule on day one. Whether an unchanged resubmission can win is a property of the harness, it is knowable immediately, and it decides how you should spend your time.
Run a restoration diff after every frontier move. Promotions silently revert each other. Checking takes minutes and it was worth 47 promotions to other people.
Slice a rollout into bands. Never ship "all 40 layers" when "the first 16" would also promote.
Write acceptance conditions that cannot be satisfied by skipping. "Tests pass" is not one of them when the tests can skip.

Where this stands

Zero promotions again. This time the field took the prover more than nine times faster while I measured.

What I have instead is a ledger. Five priced dead ends, a correctness trap confirmed on ranked hardware, a transfer rule that held in both competitions, and a written playbook that starts by mapping the timer instead of opening a profiler. The mlx.fast post ended by saying I came away with a working process instead of one built mid-race. That was true. The process I built was still aimed at the wrong question.

I would enter a third one. I would spend the first half day reading the scoring harness and writing down what it bills for, and I would not open a profiler until that document existed. Everything in this post that mattered was decided before any code was written, and that is exactly where I was not looking.

Credits

The Lighter Prover Challenge is run by EigenLabs, on the same platform as mlx.fast, with the prover and circuits from Elliot Technologies' Lighter. The techniques in "What actually won" come from public promoted submissions by solvers including i34-9, jungjipdo, Gajesh2007, exakoss, mega-dmitriy, AlexLaevski, saucegodbased, zeeshan8281, and ryanp7272. The open-notes and public-diff format is what makes this kind of analysis possible. Orchestration ran through Claude Code, with implementation delegated to GPT-5.6 models working from written briefs, and every result re-verified on my measurement machine.

Eight Submissions, Zero Promotions: A Week Inside mlx.fast on the Wrong Hardware

Wed, 05 Aug 2026 00:00:00 GMT

TL;DR

mlx.fast is an open autoresearch competition: make a 21.6 GB mixture-of-experts model decode faster on Apple silicon without changing a single output token. Score is decode_speedup^0.75 * prefill_speedup^0.25, judged on an M5 Max. My dev machine was an M4 Max, which cannot execute the M5-only kernel family that dominates the score.

I competed for a week. The results:

Eight submissions, zero promotions. Three failed on my own avoidable mistakes, the rest were priced negative or killed by an infrastructure lottery. The frontier moved from 1.84 to 2.55 while I watched.
I found that the field-wide belief that non-M5 machines "diverge from the golden traces because of silicon" was wrong. The divergence is one shipped fusion kernel. Ablate one flag and my M4 matched the M5 golden on all 130 checked steps. Every solver without an M5 had been verifying blind for no reason.
I built a ledger of what transfers between Apple silicon generations and what does not. Local timing did not just fail to predict ranked results - it pointed the wrong way on five separate occasions, for me and for competitors with better hardware than mine.
The competition was decided by measurement bandwidth. The leaders priced 3-5 candidates a day against the ranked machine. I priced about one a day against an instrument that lied.

Everything below is from the public record: submission ids, public run logs, and published solver notes. The full appendix - the oracle recipe, the transfer ledger, the dead-lever list, and the verification protocol - is in this gist.

How this started

Phil from EigenLabs emailed me an early invite based on my mlx contributions. The pitch: a live leaderboard, every promoted optimization becomes the new baseline, all submission notes are public so the field builds on each other's work. The previous competition in the series had 100+ people beating an unpublished Google quantum circuit in 72 hours.

The target model is Laguna XS 2.1, a 40-layer MoE with 256 routed experts per layer, quantized to NVFP4. Submissions run a gauntlet. An LLM reviews your diff for benchmark bypasses. A behavior gate replays a teacher-forced trace and requires token-exact output against a golden generated on the ranked M5 Max. Then a paired timing run prices you against the current baseline. Anything that changes even one token fails.

I had a Mac Studio with an M4 Max. That mattered more than I understood at the start.

The first thing I hit: my machine could not reproduce the answers

Out of the box, the public correctness trace failed on my machine at step 0: expected token 5991, actual 8550. Other solvers' notes reported the same on M3 Ultra and M1 Max. The accepted explanation was silicon divergence - different Apple GPU generations round differently, so only the M5 can verify correctness, and everyone else ships on faith.

I did not want to ship on faith, so I bisected the shipped optimization flags instead. The divergence came down to a single JIT-compiled prefill fusion kernel. With that one flag ablated, my M4 matched the M5 golden on all 130 checked steps, bit for bit.

That gave me a real verification instrument on hardware that supposedly could not have one, and it is the finding from this week that other solvers actually adopted. The lesson is the same one I keep re-learning: a difference everyone attributes to the platform is worth one afternoon of bisection before you believe it.

My first three submissions, and what they cost me

I want to report these plainly because they were all avoidable.

Submission 1 re-quantized a weight window. The rules document said, in a section I had not read carefully enough, that the quantization envelope was frozen - even if the output stays correct. The reviewer rejected it and quoted my own code comment back to me as evidence. One grep would have prevented it.

Submission 2 changed GEMM tile geometry. Before submitting, I ran an adversarial review, and the reviewer warned that my bit-exactness argument did not cover the split-K path on the M5's kernel family. I recorded the objection as a caveat and submitted anyway. The behavior gate failed exactly there: expected 5991, actual 31807. A flagged unverified correctness precondition is a blocker. This rule is now written down where I cannot miss it.

Submission 3 was correctly narrowed and passed every gate. The ranked machine priced it at -2.05 percent. It was a clean, honest negative.

After that, my process got strict: read the policy first, classify every change by whether my hardware can actually verify it, run two independent adversarial reviews before submitting, and treat any unresolved objection as a stop. My next five submissions all passed the static review and the behavior gate. None of the failures that followed were process failures.

The telemetry wall

At the frontier's speed, the benchmark's telemetry-validity gate sampled GPU state every 100 ms and required at least five valid samples. A fast candidate finishes its measured window in about one second, which yields exactly four. The result: roughly 90 percent of all submissions field-wide were rejected with no information about the payload at all. In one 24-hour stretch I audited, 24 of 26 classified failures were this single failure mode, across ten different solvers.

Four of my submissions died there, each one having already passed every correctness gate. The only available response was resubmitting the identical payload as a fresh draw. A competitor had a tested one-line fix ready (halve the sampling interval), but repository permissions blocked the pull request, so the whole field kept rolling dice. I emailed the operator about it. The fix had not landed by the time I stopped.

What local measurement is actually worth across silicon generations

This is the table I wish I had on day one. Every row is an officially priced result from the ranked M5, compared against careful local measurement - mine or a competitor's published numbers:

Change	Local evidence	Ranked M5 pricing
My fused-QKV change	+2.97 percent prefill, 4/4 paired runs	-1.64 percent
My expert-scheduling change (EG256)	promotion-capable diagnostics on the prior tree	-1.14 percent
Competitor's KV-cache write fusion	17/17 local paired wins	-18.9 percent
Competitor's paired self-validation	+0.58 percent claimed	-1.62 percent
Competitor's dequant instruction cut	+8.9 percent at kernel level on M1 Max	-0.69 percent

The pattern that emerged from the field's collective pricing: instruction-count cuts and byte cuts transfer between silicon generations. Dispatch-count cuts and anything whose benefit depends on scheduling context do not. The M5's memory system behaves differently enough that a fusion which wins on every other machine can lose by 19 percent there.

There was a second, subtler version of the problem. The frontier moved 4-8 times a day, and each promotion changed the tree your change lands in. I watched one four-line scheduling change get promoted, then priced at -1.14 percent two promotions later, then effectively promoted again in a different tree context. My own held patches flipped sign across trees three times. A pricing belongs to a change in a tree, not to the change alone, and the tree does not sit still.

The one that stung

My best submission was a de-confounding find. A concatenated QKV weight bank had shipped disabled for weeks, justified by an old ablation showing "a mild prefill cost with no decode gain." Reading the code showed the ablation was confounded: enabling the bank silently kicked decode off its fast path, so the historical A/B compared the bank's prefill win bundled with an unrelated decode loss. Scoped per-phase, the change measured +2.97 percent prefill locally with all four paired runs agreeing.

It passed the static review. It passed the token-exact behavior gate on the M5. Two independent adversarial reviews found no blocker, and the split-K trap from submission 2 was provably absent - both reviewers traced the dispatch arithmetic to the exact strict inequality.

The ranked machine priced it at -1.64 percent.

The mechanism argument was fine. The correctness argument was fine. The performance model was an M4 model, and the M5 is not an M4. By then I had also watched my local signal for this change decay from -2.97 percent to -0.4 percent as two competitor promotions landed around it, which was the warning I chose to submit through anyway. An official pricing has value even when it is negative, and I published mine. But that one was the week's ceiling, and it was below zero.

Killing ideas cheaply

The second half of the week went better than the first, not in score but in cost per dead end. Three examples:

Software-pipelining the quantized matrix-vector kernels. A competitor had landed exactly this pattern on the attention kernel for +6-9 percent at kernel level. I applied it to the adjacent unmined kernel family. A timed microbenchmark at all nine real shapes came back flat to negative - that kernel's weight stream already saturates memory bandwidth, so there was no latency to hide. The idea died in one evening, before submission, because I measured at the right granularity.
llama.cpp-style weight repacking. A read-only audit of the actual per-lane access pattern showed the shipped layout is already contiguous at the SIMD load level. The repack would reorder without reducing transactions. That one died in an afternoon of reading, with zero code written.
A Cauchy-Schwarz pre-filter for the vocabulary projection. The frontier's winning structure proves which of 100,352 vocab rows can contain the argmax and exact-evaluates only those (median retained set: 13 rows). My idea was a level-zero bound to skip reading most rows entirely. An empirical harness on 128 real decode steps returned a pruning rate of exactly zero: in 2048 dimensions the norm-product bound sat 25-80 percent above the real logits on every step. High-dimensional bounds are looser than intuition says. That one died in 22 minutes of compute.

None of these produced a submission. All of them produced a number, and the numbers are in my public notes so nobody has to re-run them.

What actually won, while I was doing all this

The week's biggest single gain (+3.14 percent) was not a kernel at all. A solver profiled the measured window and noticed a 16.7 ms one-time pipeline compile being charged inside a 210 ms prefill window, and moved it into warmup. I had observed the same mechanic two days earlier in my own logs and filed it as measurement contamination. They asked which timing window was being billed for that compile, and I did not. That question was the whole optimization.

The most durable gains came from certified work elimination on the vocabulary projection - closed-form bounds proving the answer cannot change, then deleting the work the bound makes unnecessary. The solvers who own that structure iterated its format three times in the week (2112 to 1600 to 1344 bytes per row), each step verified by exhaustive bit-comparison and priced on ranked hardware within a day.

And one competitor took +2.19 percent by doing archaeology: the promotion race moves so fast that promotions kept silently reverting each other's validated work, and they diffed the tree, found the wreckage, and restored it. I ran the same sweep twice later in the week. Both times the tree came back clean, which is its own lesson - by then, everyone had learned to scan.

What I actually learned

Bisect before you believe a platform explanation. The "silicon divergence" was one kernel. The whole field had worked around it instead of finding it.
A flagged unverified correctness precondition is a blocker, not a caveat. My only behavior-gate failure came from ignoring my own reviewer.
Classify every change by whether your hardware can verify it, before you build it. I eventually kept a table: custom kernels yes, host-side code yes, M5-only kernel paths no. Work in the last column is guessing.
Measure at the granularity where the effect lives. End-to-end timing hid a kernel-level null; a microbenchmark exposed it in minutes.
Pricings expire. Re-verify every held patch after every upstream change. Mine flipped sign three times in five days.
Publish negative results, but understand what you are giving away. I published a pricing plus the mechanism hypothesis behind it, and a competitor executed the obvious composite from my own synthesis within hours, for a promotion. Raw data helps the field; synthesis is a move you hand to whoever is fastest.
Measurement bandwidth beats idea quality. I independently found several mechanisms that later won for others. Finding them was never the bottleneck. Pricing them was.

Where this stands

The competition is open-ended and the frontier is still moving. I have paused rather than quit: my verification tooling, the shelved patches, and the full ledger are ready if a pricing channel opens up for solvers without ranked hardware - I have asked the operators for exactly that. If the platform resets or migrates models, the early cheap optimizations reappear, and this time I would be starting with a working process instead of building one mid-race.

Zero promotions is the honest headline. But I came out with a correctness oracle the field uses, and a transfer ledger I have not seen anyone else write down. I also came out with a much sharper sense of what independent verification is worth when you are the one without the reference machine. I have been on the other side of this - my TabFM evaluation was exactly the reference-machine role - and it is useful to have now felt both sides of that asymmetry.

Update, August 14

My last submission here was July 31. The mlx.fast frontier has since reached 2.62.

I cloned the repository for the next competition in the same series on the day this post went up, and I lost that one the same way. The lever that won it moves work out of the timed window, which is the question this post ends by saying I did not ask. That write-up is The Work You Don't Do.

That post also carries something this one does not. I went back and read the winning submission notes properly. Most of what is in them is ordinary performance craft, not anything competition-specific. The list covers why multiply count is the wrong cost model, and why SIMD is usually the wrong tool for 64-bit integer math on AArch64. It also covers how to structure GPU offload so it can fail safely, and how to build a differential mode once and verify every later optimization cheaply. Reading the winners' notes was worth more than the week I spent generating my own candidates.

Credits

mlx.fast is run by EigenLabs (thanks to Phil Burgess for the early invite); the public challenge repo is Layr-Labs/mlxfast-challenge. The findings above build on public submission notes from solvers including lBroth, zuiris, a-github-name, GumbiiDigital, EternaPeptix, AdeliyaLeleytner, ivanfioravanti, davidtai, and yoyo930021 - the open-notes format is what made this kind of analysis possible at all. Orchestration for my experiments ran through Claude Code, with implementation and review passes split across Claude and GPT-5.6 models working from independent briefs.

Same Weights, Same Prompt, Opposite Results - Why Local Tool Calling Breaks

Tue, 21 Jul 2026 00:00:00 GMT

TL;DR

Tool calling on local stacks breaks silently: the same model, at the same quantization, with the same prompt, can execute tool calls perfectly on one inference server and fail completely on another. I built willitcall to measure this. It is a Rust CLI that runs 50 tool-calling scenarios against any OpenAI-compatible endpoint. It also publishes a public red/green matrix of model x quant x server results, where every red cell links to the full request/response transcript that produced it.

Three findings survived replication, and one of my own headline claims did not:

The servers do not decode the same way, and this explains most cross-server deltas. llama.cpp compiles your tool definitions into a grammar and constrains decoding with it - a call naming a function you never supplied cannot be sampled. Ollama and mlx-lm generate unconstrained text and parse a tool call out of it afterwards. Same weights, opposite results, decided by the stack rather than the model.
Quantization level does not order tool-calling ability on this corpus. The folklore that lower quants degrade tool calling did not show up: across Q8_0, Q4_K_M, and Q3_K_M, seed-varied replication puts the arms within noise of each other. What CAN destroy a row outright is a bad conversion artifact, which is a different thing than a dose-response effect.
There is a failure mode I did not have a name for: a model that emits correct, well-formed tool calls in a format that no server parses. Those cells look identical to "this model cannot call tools", and that is false.

And the retraction: I filed an issue against Ollama claiming it discarded a valid tool call. A maintainer pushed back, I recovered the actual discarded bytes, and the model had emitted the tool's description where its name belonged. Ollama's parser was right; my inference was wrong. I retracted the claim publicly, and the investigation that followed produced finding number 1, which turned out to matter far more than the bug report.

The problem

Every local inference stack claims OpenAI-compatible function calling. In practice, support varies by model, by quantization, by chat template, and by server, and the failures are silent. Your agent does not crash; it just never calls the tool, or calls it with mangled arguments, and you spend an afternoon debugging your prompt when the problem is three layers down.

Existing benchmarks did not cover this. BFCL and its descendants measure fp16 API models - a different axis entirely. What I wanted was caniuse.com for local tool calling: pick a model, a quant, and a server, and see whether that exact combination executes tool calls, with the evidence one click away.

So the unit of measurement in willitcall is the combination. A cell is a property of the whole stack. When I wrote that into the spec it felt like a hedge. By the end it was the main finding.

What I built

The CLI runs a corpus of 50 scenarios in six categories: single calls across argument shapes, parallel calls, streaming (SSE delta reassembly), tool_choice modes, multi-turn, and negative traps. A multi-turn case feeds a tool result back, and the follow-up call must use a value that only exists in that result. A negative trap is a case where the correct behavior is to NOT call a tool. Scoring is deterministic; there is no LLM judge, because a published failure reason has to be defensible.

Two design rules did the most work:

A scenario that a fully correct model could fail is a bug in the scenario. A false red is worse than a missing test - if the matrix says a model fails and it does not, no other cell is trustworthy. This rule caught real bugs during development: one scenario expected the ASCII spelling of an Icelandic city when a correct model writes it with a diacritic. Every scenario now carries a rationale field explaining why no correct model can fail it for a reason other than the one it tests.

Every red cell must carry its evidence. Result files ship full request/response transcripts (auth-redacted, hash-verified). This rule is why the retraction below was possible at all, and why the project survived it.

The bug report I had to retract

Early seeding produced a striking result. Qwen2.5-7B, served through llama.cpp, emitted a correct tool call six times out of six. The same weights through Ollama returned an empty response ten times out of ten, while still billing about 40 completion tokens. Tokens were generated and nothing came back. I read that as Ollama's parser discarding a well-formed call, wrote it up, and filed it upstream.

A maintainer pushed back, and they were right to. When I recovered the actual discarded bytes (using a rebuild with a then-open logging PR), the model had emitted the tool's description string in the name field. That was not a valid call. Ollama's parser had correctly rejected garbage; my claim was built on an inference from the token count, not on the tokens themselves. I retitled the issue, retracted the claim in the thread with credit to the person who challenged it, and rewrote the case study to carry the correction.

That left a question: if the model emitted a wrong name through Ollama, why did the same weights emit the right name through llama.cpp, every time?

Because llama.cpp does not give the model the chance to be wrong. It compiles the tool definitions from your request into a GBNF grammar and constrains decoding with it - lazily, engaging when the model starts a tool call. Under that grammar, a function name you did not supply is not merely unlikely; it cannot be sampled. Ollama generates unconstrained text and parses afterwards. mlx-lm sits on the unconstrained side too (verified by reading the source: tools reach the chat template only, there is no grammar machinery, and parse failures are swallowed).

That is the mechanism behind the whole matrix. It favors llama.cpp systematically, in every row, for every model. A llama.cpp-versus-Ollama delta measures constrained decoding against unconstrained decoding, not the quality of either server or model. The comparison that isolates the model is same-server, never cross-server. The site now discloses this above the matrix, because without it, every reader would misread the reds the same way I did.

The bigger lesson: an empty response is unattributable without the raw tokens. My inference - "tokens were billed, nothing returned, therefore the server ate a valid call" - felt rigorous at the time, but it was a guess. Get the bytes.

Quantization does not order tool-calling ability (and how I almost published the opposite)

The folklore says aggressive quantization degrades tool-call formatting. I measured three models across Q8_0, Q4_K_M, and Q3_K_M on llama.cpp and the first pass said the opposite: two of three models scored highest at the lowest quant. That would have made a good contrarian headline, except it was n=1 per arm.

Project rule: no verdict below five runs per arm. So I re-ran every arm five times, and got the same numbers. Exactly the same numbers. Identical scores, identical failing scenario ids, forty-five runs reproducing byte-for-byte. That looked like the strongest replication imaginable and was actually the weakest. The runs were greedy (temperature 0, fixed seed), so all five "replications" were the same computation. n=5 of the same coin flip is n=1.

Replication that means anything has to vary what should not matter. Under five seeds at temperature 0.7, the Qwen2.5-7B arms landed at 46.8, 45.4, and 46.4 of 50 - fully overlapping, nothing monotonic (the 1.5B model puts Q4_K_M below Q3_K_M). The contrarian finding died. What survived is narrower and better: on this corpus, these models, this server, quantization level simply does not predict tool-calling ability, in either direction.

One result looks like a contradiction and is not: on mlx-lm, the 8bit community conversion of Qwen2.5-7B scores 7/50 while the 4bit scores 46/50. Before writing that up as a quant effect I recovered the raw generation again - and the 8bit conversion emits doubled braces in its tool calls:

8bit:  <tool_call> {{"name": "get_weather", "arguments": {"city": "Boston"}}}
4bit:  <tool_call> {"name": "get_weather", "arguments": {"city": "Boston"}}

That is not parseable JSON, and mlx-lm is right to reject it. The model still knows exactly which function to call with which arguments - the recovered string is semantically perfect. This is a broken published artifact, not gradual degradation, and it cannot be extrapolated to "8bit is worse than 4bit". The distinction matters: quantization as a dose does nothing here; one specific conversion is simply defective.

The failure modes I had to name myself

After scoring thousands of transcripts I needed more precise labels than pass and fail. The matrix now distinguishes, mechanically, per response:

error - the server refused or crashed. Ollama returns HTTP 400 for any gemma3 request carrying tools. llama.cpp returns HTTP 500 on 7-9 of 50 scenarios for Llama-3.1-8B ("does not match the expected peg-native format") - replicated across two llama.cpp builds, so those rows are excluded from conclusions rather than counted as model failures.
empty_response - no content, no tool call. Unattributable without raw tokens; see above. Both Ollama and mlx-lm return HTTP 200 with finish_reason: "tool_calls" and an empty message when their parser rejects what the model emitted - the client is told a call happened and handed nothing. From the API alone this is indistinguishable from the model saying nothing at all, which is exactly why the recovery step exists.
unparsed_tool_call - the one I had to invent. granite3.1-dense:8b emits well-formed calls, right function, valid arguments, in its own <tool_call>[...] format - and scores 7/50 on Ollama AND 7/50 on llama.cpp fed the identical blob, because nothing parses that format into tool_calls. Ten runs, zero variance. The 7 passes are precisely the negative traps, where emitting no parsed call is correct - a model doing everything right and scoring near zero. Without this class, those cells read as "granite cannot call tools", which is false. It is a third fault domain: an ecosystem format mismatch.

Each class is assigned conservatively - crediting a model with a call it never made is worse than leaving a cell unexplained - and the full precedence rules live in the schema docs.

Rules I learned the hard way

Every wrong conclusion this project produced followed the same shape: a pattern at n=1, generalized. The single-sample "fix" that a later replication overturned. The second model I claimed shared granite's failure mode - and at replication, 1 of its 86 failures matched; the rest were prose refusals. The Ollama defect claim. The quant surprise. Four times, and each one is now a project rule with a mechanism behind it:

No verdict below five runs per arm, and replication must vary seeds at nonzero temperature - greedy repeats measure reproducibility, not truth.
Any anomaly gets its raw tokens recovered before it gets a cause.
A leftover inference server on the same host produced plausible-but-wrong error counts once, so the CLI now refuses to run when another server is responding (contention preflight).
Early rows were measured on two different machines without disclosure. Everything published was re-measured on one host, and result files carry environment fields now.

None of these rules existed at the start. Each one exists because I got something wrong first.

The comparison I built the project for, and could not make

Months in, I went to group the matrix by model so you could see one model across three servers. I could not. The schema had no way to say which model a row measured.

Each result stored a field called model_id. It held whatever string the server happened to use. Ollama wrote a tag like qwen3:8b. llama.cpp wrote a Hugging Face reference, or an absolute file path when I served a raw blob. mlx-lm wrote a repository id. All 32 rows held different values, so nothing joined. phi4-mini is measured on all three servers, and the schema saw three unrelated rows.

The whole premise of the project is that a cell is a property of the stack, so the interesting question is what happens to one model across servers. I had been unable to ask it since the beginning and had not noticed, because the site labeled its rows from the result file names. The labels looked right. Nothing behind them was.

The fix was a new schema version. A row now records the checkpoint it measured, the artifact that produced it, and how confident I am in each. Identity comes from a registry I check in, never from a file name, and every value cites the evidence it came from. Of the 32 rows, 1 is verified, 27 are declared, and 4 are unresolved. The unresolved ones stay in the matrix, labeled as unverified artifacts, and are excluded from any grouping rather than guessed at.

Recovering the identities was the interesting part. Ollama had been deregistered on the measurement host. But 14 manifests and 54 blobs were still on disk, and 12 of those blobs carry their own provenance in the GGUF header. general.name says "Qwen2.5 7B Instruct". general.organization says "NousResearch". That is what the measured bytes say about themselves, which beats anything I could infer later. The manifests also gave me the quantization for the Ollama rows, which had been null for all 14 of them. Two of the rows I had assumed were Q4_K_M are Q4_0.

Two other things were wrong and had been on the public site for weeks.

gemma3 is refused outright by Ollama, so both its rows are 50 errors and zero measurements. The site painted them the same red as a model that ran and failed everything, and read out the same label to a screen reader. A combination I could not measure looked exactly like a combination that failed.

The decode-mode bands were worse, because they contradicted a rule I had written myself. The site sorted rows into constrained and unconstrained decoding, which is the mechanism this whole article is about. It decided which band a row belonged to by matching the server name string. Only 6 of 32 runs had actually recorded their decode mode. My own README says that a missing flag means unverified, not unconstrained, and the page ignored that for 26 rows. Each row now says whether its decode mode was recorded by the run or resolved from a documented mapping, and the mapping cites where each server's behavior is established.

The pattern here is the same one as the retraction, one level up. I trusted a label instead of the thing it named. The file name looked like an identity, the server name looked like a decode mode, and both were close enough to correct that nothing broke loudly.

Where it stands

The matrix currently covers 32 published rows across three servers (llama.cpp, Ollama, mlx-lm) on one measurement host, with case studies replicated at five or more runs per arm - 90 runs across 18 arms for the quantization question alone. Every red cell links its transcript. The corpus, the CLI, and the site generator are one repo, and a result file from your machine is a PR away from being a row - the scenario format is plain TOML, and the schema validates in CI.

The site also publishes the whole result set as JSON and CSV, so you do not have to scrape the page to check my arithmetic. I would rather you did not trust it.

If you run local models for agent work, the practical takeaways are three:

Pick your server first. On identical weights, the decoding strategy is a bigger variable than the quant level.
Do not read a cross-server delta as a verdict on the model, and do not read an empty response as a verdict on anything.
Stop paying for Q8_0 out of tool-calling fear until someone shows a dose-response effect on your stack. On mine, there is none - but a single bad conversion can still zero a row, so test the artifact you actually deploy.

The project is at github.com/devYRPauli/willitcall, the matrix is at devyrpauli.github.io/willitcall, and if a combination you care about is missing, the whole point of the design is that you can measure it yourself and send the result.

Baton: I Built a Tool for Delegating Code to an AI, Then Used It to Build Itself

Mon, 13 Jul 2026 00:00:00 GMT

TL;DR

I kept hitting the same wall when I let a planning model hand coding work to a separate executor model (OpenAI's Codex): the background jobs died. The session that launched them ended, and the work went with it. So I built Baton, a small kit that installs a reliable delegate-code-to-an-executor workflow into any Claude Code setup. You point your agent at the repo, it interviews you, and it writes a tailored delegation skill plus a detached job runner into your config. The core promise is in the name: it does not drop the baton. Jobs survive the session ending, and their results stay retrievable.

The part I like most is how it was built. Baton is a tool for delegating code to an AI executor, and I built it by delegating its own code to an AI executor, one task at a time, verifying every task myself before it landed. The tool built itself, under supervision.

Repo: github.com/devYRPauli/baton (MIT, CI green on macOS and Linux)

Why I built this

The setup I wanted is simple to state. A capable planning model acts as the orchestrator: it writes the brief, reviews the diff, and verifies the result. A strong coding model acts as the executor, and writes the code in its own runtime. The orchestrator should not write the application code itself; it should delegate and check.

That division works right up until you send a long job to the background. In practice the background path was the flaky part. A job would launch, the launching session would end, and nothing was retrievable afterward. Some of this traces to known upstream behavior: a background flag getting stripped, a session-end hook killing in-flight jobs. But the root problem is general. A background process whose lifetime is tied to the agent session is not actually durable.

I did not want a clever prompt. I wanted a job runner that a coding agent could hand work to and trust, plus a small amount of discipline around it so the delegation was worth doing at all.

The design decisions

Before writing anything, I settled the questions that are expensive to change later. Two mattered most.

Standalone, not plugin-dependent. My personal workflow routes through an official Codex plugin, but that plugin is exactly where the background bugs live. For something other people would install, tying it to a component with known reliability issues is the wrong foundation. Baton calls the executor CLI directly and depends on nothing but bash and coreutils. It coexists with the plugin but never requires it.

Template the models, hard-code the invariants. The kit is meant to be shared, and model names go stale fast. "Use this specific model at this specific effort" is my opinion for this month, not a durable design. So the installer fills in model, effort, and paths from an interview. The things that are actually the product stay fixed text. Every brief ends with a smallest-diff constraint. The kit verifies every result before it calls the work done. It confirms a detached job is alive before anyone assumes it is running. If those were optional, you could interview your way into a kit that just runs an executor sometimes.

I ran the design past a stronger reasoning model as a critic before committing. The sharpest note it gave me: make the installer verify itself. An installer that does not prove it worked is exactly the failure mode the tool exists to prevent. That became a rule - the last thing setup does is run a live job end to end and show you the real output before it declares success.

The rest fell out of those. The installer is agent-driven through SETUP.md rather than a shell script, because the interview needs judgment a script cannot fake. A per-scope config defines the executor invocation as a small shell function, which is what lets you swap in a non-Codex executor. Two policy presets keep the kit from being opinionated about other people's API bills: quality-first for maximum effort, balanced for a lighter default.

Dogfooding: building Baton with Baton

Here is the loop I used to build it, which is the same loop the tool ships.

I wrote a spec, then an implementation plan broken into eight small test-driven tasks. Then, task by task, my orchestrator wrote a tight brief (the files to touch, the acceptance test, and a hard "smallest diff, no incidental changes" constraint) and handed it to the Codex executor through a detached runner. The executor implemented the task and ran the tests in its own runtime. When it came back, I did not take its word for it. I re-ran the tests myself, read the diff, and only then committed.

Every task carried the same rule: return paths and a concise summary, never a wall of logs. The orchestrator plans and checks; the executor writes. I never hand-wrote the runner's bash.

The tasks, each an independently testable slice:

Task	Deliverable
1	A dependency-free bash test harness and a fake-executor fixture
2	Runner core: config sourcing, version, help, model validation
3	start / status / result, with detached execution
4	A timeout watchdog, single-line telemetry, and list
5	The config template with the executor-function slot
6	The skill template with the invariants as fixed text
7	The agent-driven SETUP.md installer
8	README and a how-it-works writeup

The final runner is a plain CLI you can also use by hand:

$ baton start "refactor the parser per the brief"
20260713-105451-23816232

$ baton status 20260713-105451-23816232
id=... status=running duration_secs=3 exit_code=-

$ baton status 20260713-105451-23816232
id=... status=done duration_secs=41 exit_code=0

$ baton result 20260713-105451-23816232
<the executor's output>

Proving it survives

The whole point of Baton is durability, so a passing unit test was not enough. I wanted to see a job outlive the thing that launched it.

The mechanism is a detached runner started with nohup and disowned, so the operating system reparents it to init. I checked this live: after launching a job, the runner process reported a parent process ID of 1. It was no longer a child of my shell. That is the property that matters. Kill the session, close the terminal, the job keeps going and its result is still there.

Then I ran the full install flow against a fake executor, exactly as SETUP.md describes it: render the templates, install into a throwaway scope, and run the acceptance test. Rendered files had zero unfilled slots, the installed runner launched a job, reached done, returned the right output, and wrote one well-formed telemetry line. Finally I let CI run the suite on both macOS and Linux, since the kit claims both. Both are green.

That is the standard I try to hold on every project: do not take the tool's word for it, look at the evidence.

Two bugs, and they were mine

The most interesting failure in the build came from my plan, not from the generated code.

Two tasks failed their tests on the first pass. When I dug in, the executor had implemented my plan faithfully, character for character. The plan was wrong.

The first: my config template rendered filesystem paths without quotes. On any path containing a space (which includes the very directory this project lives in), sourcing the config broke. A real robustness bug, and it was sitting in the spec I wrote.

The second: a test asserted a lowercase word that appeared in the template only capitalized. The invariant was present; my assertion was needlessly case-sensitive.

Both were two-line fixes, and I fixed them in the plan as well as the code so the two stay in sync. The lesson is one I keep relearning: when delegated work fails, check your own instructions before you blame the worker. A faithful executor will implement your mistakes exactly.

How you use it

You do not run a script. You read SETUP.md first (it is short, and it is a stranger's repo telling your agent to write files, so you should), then you tell your agent to set up Baton from the repo. The agent detects your executor. It asks whether you are on a subscription or an API key, and has you authenticate yourself, because the kit never touches your credentials. It asks which models to use and at what effort, picks a policy preset, and chooses global or project scope. It lists every file it will write and asks once before writing anything. Then it installs and runs that live acceptance test.

After that, "delegate this to the executor" just works, with a brief, a durable run, and a verification step, every time.

Honest limitations

macOS and Linux only. The runner leans on nohup, ps, and process reparenting. Windows is not supported and I did not pretend otherwise.
Verified with a fake executor, not yet battle-tested by a stranger. I proved the mechanics end to end (survival, render, install, acceptance) with a stand-in executor and cross-platform CI. A first real-world install by someone who is not me, against a live executor, is the next validation I want.
The one-command plugin version is deliberately deferred. The agent-driven install works today. A packaged plugin can come later, once the agent-driven install stops being enough.
It never manages your secrets, by design. That is a feature, but it means auth is on you.

What I took away

Decide the expensive-to-reverse things first. Standalone-versus-plugin and template-versus-fixed shaped everything downstream. Getting them right early meant the eight build tasks were mechanical.
Make the installer prove itself. The most valuable rule in the whole design is that setup ends with a live job, not a claim. It doubles as the demo.
A faithful executor implements your mistakes exactly. Both build failures were my plan, not the generated code. Delegation makes the quality of your brief the ceiling on the result.
Durability is a property you can point at. "Jobs survive the session" sounds like marketing copy until you can point at a parent process ID of 1. Find the observable version of your claim and check it.
The orchestrator should not write the code. Keeping the planning model out of the implementation, and making it verify instead, is the discipline Baton packages - and it is the discipline that built Baton.

The stock background path dropped jobs on the floor. The finished kit hands work to an executor, keeps custody of it through the session ending, and proves the handoff landed - and it wrote its own runner while I watched.

I Tried to Break Google's New Tabular Foundation Model. Then I Fixed It.

Wed, 01 Jul 2026 00:00:00 GMT

TL;DR

Google Research released TabFM on 2026-06-30, a zero-shot foundation model for tabular data: no training, no tuning, you hand it your table and it predicts. The claim is that a single pre-trained model can match or beat tuned gradient-boosted trees out of the box. That is a big claim, so I ran an independent reproduction across three machines.

What I found:

On small-to-mid tables, TabFM beat a properly Optuna-tuned XGBoost on all 10 fold-matched datasets, zero-shot. On the anchor dataset the gap actually widened under tuning (0.877 vs 0.821 accuracy).
But I held myself to the same bar I'd hold the authors to. A multi-seed check forced me to demote two "wins" to ties because the margins were smaller than measurement noise.
The advertised "22.75 GB GPU footprint" turned out to be mostly an XLA allocator artifact; the real footprint is ~16.95 GB, and disabling preallocation roughly doubled the usable context (from ~10k rows to ~20k).
Update (2026-07-07): after a TabFM author mentioned the repo now enables bf16 + activation chunking, I benchmarked the PyTorch backend at current main. On the same 4090 it holds a 3-7 GB working set (vs the JAX pin's flat ~17 GB), fits 40k rows where JAX OOMs at ~20k, and runs 2-4x faster at large context. Details in the hardware section.
Along the way I found a bug that crashes predict on any machine with two or more GPUs. I root-caused it, wrote a fix and a regression test, and the PR was merged into google-research/tabfm by one of TabFM's own authors.

Everything below is seeded, pinned, and reproducible: github.com/devYRPauli/tabfm-evaluation

How this started

I read the TabFM launch post the day it went up. The pitch is genuinely interesting: tabular data is the one domain where deep learning has repeatedly lost to gradient-boosted trees, and here was a foundation-model approach claiming to close that gap with zero training on your data. It learns in-context, the way a large language model does, except the "context" is your training rows.

My first instinct with a claim like that is not to believe it and not to dismiss it, but to reproduce it. Benchmarks in papers and blog posts are run by people who want their method to look good. That reflects incentives rather than dishonesty. An independent reproduction is the only way to know which parts hold up when someone with no stake in the outcome runs them.

So I set three rules for myself before writing a line of code:

The source code is ground truth, not the blog post. Where they disagree, the code wins.
Every result is seeded and reproducible. No cherry-picked runs.
Negative results are results. If it loses, or ties, or fails, that gets reported as plainly as a win.

That third rule turned out to matter the most.

What TabFM actually is (read from the code, not the marketing)

Before benchmarking anything, I wanted to know exactly what I was testing. A few facts I verified directly from the source:

It is in-context learning, not training. Calling fit() does not update the network. Your training rows become context for a single forward pass at predict time. This is why it is "zero-shot."
The default forward pass is a 32-member ensemble. That detail matters enormously for the memory story later.
There is a hard cap of max_classes = 10. Good to know before you throw a 20-class problem at it.
Two weight repos exist. The JAX Orbax checkpoint is canonical; the PyTorch one is a converted state_dict that I validated matches within 1e-4. I used the JAX backend.
Two licenses. The code is Apache-2.0; the pretrained weights are non-commercial. They govern different artifacts, which is easy to miss.

Pinned stack for the whole study: commit b6ea70b, JAX 0.10.2, Flax 0.12.7, Python 3.12.

The setup: three machines, and one that I set on fire

I ran this across three machines, each with one job:

Machine	Spec	Role
MacBook Pro	M1 Pro, 16 GB	Orchestration only
Mac Studio	M4 Max, 64 GB	CPU reference
Workstation	Threadripper PRO 5955WX, 128 GB, 2x RTX 4090	GPU reference

The MacBook is "orchestration only" for a reason I learned the hard way. Early on I ran the full model on the 16 GB laptop. The 32-member ensemble needs a ~17 GB working set. macOS tried to cover the gap with swap, wrote about 39 GB of it, and the machine restarted itself.

That incident produced the most useful piece of infrastructure in the whole project: a safe_run.sh watchdog that preflights free memory and disk, then kills any job before its resident set, system swap, or free disk crosses a limit. I would rather lose a job than restart the machine. Every heavy run after that went through it. The lesson generalizes: when you are probing the limits of a model's resource use, build the kill-switch before you need it.

Phases 0 to 2: does it even reproduce?

Before the headline benchmark, three checks:

Phase 1 - conformance and determinism. The scikit-learn contract holds, and predictions are bit-exact deterministic under a fixed seed, on both the Studio CPU and the 4090 GPU. CPU and GPU agree numerically. Good - a model whose outputs drift run-to-run is not one you can benchmark honestly.

Phase 2 - known-answer sanity. Before trusting it on real data, I checked it can learn things I already know the answer to. It nails XOR (accuracy 1.00), accuracy climbs from 0.69 to 0.98 as I feed it more context rows, and it recovers a monotone function at R2 0.997. It genuinely learns from context rather than matching noise.

A caught harness bug. My first sanity layer looked broken until I found the bug was mine: the synthetic-data generator was drawing a fresh labeling function for train versus test. Fixing my own harness before blaming the model is a recurring theme. Most of my "the model is wrong" moments turned out to be "my measurement is wrong" moments.

Phase 3: the actual benchmark

The benchmark is TabArena (OpenML Study 457), a curated tabular suite. I used a 13-dataset subset spanning 748 to 150,000 rows, 9 classification and 4 regression tasks, on the official repeat-0 folds, 3 folds each.

Baselines: TabFM's default and .ensemble() presets, versus XGBoost, random forest, a logistic/ridge floor, and TabPFN (another tabular foundation model). Metrics: accuracy, ROC AUC, log loss for classification; RMSE and R2 for regression.

The comparison is fold-matched - TabFM and a baseline are only compared on folds where both produced a result. This one discipline prevents a huge class of benchmark lies, where a method looks good only because it quietly skipped the hard folds.

The first pass was clearly favorable to TabFM. It matched or beat every baseline on the primary metric, zero-shot: churn 0.979 vs 0.956 XGBoost, maternal-health 0.877 vs 0.842 random forest, diamonds R2 0.985. Against TabPFN it was directionally ahead but by hair-thin margins (MIC 0.891 vs 0.889).

I published that, and the pushback arrived quickly.

The pushback: three critiques, and how I chased each one down

The feedback was specific and correct. Three critiques stuck, and rather than defend the first draft I spent real compute closing each one. The honest answers made the story more interesting, not less.

Critique 1: "Your tree baseline was too soft."

Fair. My first XGBoost was lightly tuned. So I replaced it with an Optuna-tuned XGBoost: 100 trials of TPE search with 3-fold inner cross-validation, per dataset. That is a real tuning budget, the kind a strong practitioner would actually use.

TabFM still beat it on all 10 fold-matched datasets:

Dataset	Metric	TabFM	Optuna-XGBoost	Delta
maternal_health_risk	acc	0.877	0.821	+0.055
houses	R2	0.898	0.856	+0.043
wine_quality	R2	0.549	0.518	+0.031
students_dropout	acc	0.800	0.772	+0.028
churn	acc	0.979	0.955	+0.023
credit-g	acc	0.779	0.758	+0.021
concrete	R2	0.950	0.934	+0.016
blood-transfusion	acc	0.801	0.787	+0.013
diamonds	R2	0.985	0.982	+0.002 (2 folds)
MIC	acc	0.891	0.890	+0.001

The headline: on the anchor dataset (maternal-health), proper tuning did not close the gap - it widened it (+0.055, the largest margin of all). Eight of the ten margins are substantive (+0.013 to +0.055). But look at the bottom two rows. Those tiny margins set up critique 2.

Critique 2: "A single seed is noise, not a result."

Also fair. My first run used one seed. So I re-ran both TabFM and the baselines at three seeds on the thin-margin datasets and measured the run-to-run spread on each side.

The result cut both ways. First, the good news: TabFM is remarkably stable. Its run-to-run standard deviation is 0.0001 to 0.0006 - smaller than every tree baseline (the tuned XGBoost drifts up to 0.005, random forest up to 0.004). So the thin margins are not a TabFM-noise artifact; TabFM sits stably where it sits.

Dataset	Method	Seed 0	Seed 1	Seed 2	Std
MIC	TabFM	0.8911	0.8917	0.8917	0.0003
MIC	xgboost_optuna	0.8899	0.8882	0.8876	0.0010
concrete	TabFM	0.9503	0.9500	0.9502	0.0001
concrete	xgboost_heavy	0.9365	0.9335	0.9307	0.0024
blood	TabFM	0.8008	0.8021	0.8021	0.0006
blood	xgboost_heavy	0.7754	0.7660	0.7781	0.0051

Now the catch. Being stable does not mean the thin wins are real. When a margin is +0.001 and sits within test-set granularity, you cannot claim it. So I demoted the two thinnest results to ties:

concrete vs TabPFN (+0.0011): a genuine near-tie. Reported as a tie.
MIC vs TabPFN (+0.0022) and vs Optuna (+0.0029): too close to call a win. Ties. (Optuna on MIC actually got slightly worse across seeds, 0.890 to 0.889.)
blood-transfusion (+0.0089) and concrete vs Optuna (+0.0157): comfortably outside the noise. Real edges.

I also hit a wall I could not climb: TabPFN would not run at seeds 1 and 2. It hangs on a C-level network call in my environment no matter how I invoke it (backgrounded, pseudo-TTY, real TTY - all hang). Rather than pretend, I documented the gap: the two TabPFN margins rest on TabFM's measured variance plus TabPFN's single seed-0 point. An unmeasured thing reported as unmeasured is worth more than a number I made up.

Critique 3: "Your 22.75 GB memory number is an XLA artifact."

This one was the most fun to chase down. My Phase 4 hardware sweep reported GPU peak memory pinned at a suspiciously flat ~22.75 GB across every context size. A commenter pointed out that I had not disabled XLA preallocation, so I was almost certainly measuring XLA's preallocated pool, not the model's live working set.

They were right, and the truth was more nuanced than either "real" or "artifact." I re-ran with XLA_PYTHON_CLIENT_PREALLOCATE=false on an otherwise-empty card:

Context (n_train)	Prealloc ON	Prealloc OFF
100	22752 MiB	16953 MiB
1000	22754 MiB	16955 MiB
5000	22756 MiB	16957 MiB
10000	22779 MiB	16957 MiB
20000	(OOM)	16955 MiB (fits)
30000	(OOM)	OOM

So: ~5.8 GB of the original 22.75 GB was pure pool padding (the artifact the commenter suspected). But the remaining ~16.95 GB is a real, large, and genuinely flat footprint - it grows only ~4 MiB from 100 rows to 20,000, because the 32-member ensemble's fixed cost dominates. Latency is identical between the two runs, confirming it is the same workload measured with a different allocator.

The practical payoff: disabling preallocation roughly doubled the usable context ceiling on a 24 GB card, from ~10k rows to ~20k, and proved the original ">10k OOMs" was the pool, not the model. That is an actionable finding for anyone running TabFM on a single GPU.

The hardware picture, completed

For completeness I finished the CPU curve on the Studio too. The full latency-vs-context story:

n_train	GPU latency	CPU latency	Speedup
100	2.3 s	32.5 s	14x
1000	4.6 s	147.3 s	32x
2000	8.9 s	346.0 s	39x
5000	31.1 s	1299.2 s	42x

The GPU is 14x to 42x faster where it fits, and the gap widens with context. There is no speed crossover - the GPU is always faster where it fits, and simply stops fitting past its memory ceiling. The CPU can hold far more context (64 GB unified memory), but at ~21 minutes per single prediction at 5,000 rows, it is impractical at scale. This matches exactly why the largest TabArena datasets (78k and 150k rows) could not be benchmarked exhaustively.

Update: the PyTorch backend changed the memory story

After I published, one of TabFM's authors reached out to say the repo had recently been updated to enable bf16 computation and activation chunking for speed and memory, and asked whether my eval predated the change. It did - and the honest answer has two halves. The accuracy numbers are untouched: those changes are PyTorch-backend-only and post-date my pinned commit, and the JAX backend I used already computed in bfloat16, so nothing above moves. But it raised an obvious question I hadn't measured: what does the updated backend actually do to the hardware envelope?

So I ran the sweep again on the same 4090, this time on the PyTorch backend at current main, chunking on. The chunking is the whole story:

Context (n_train)	JAX pin (b6ea70b)	PyTorch main (chunked)	Latency: JAX -> PyTorch
100	~17 GB	3.1 GB	2.07 s -> 1.84 s
1000	~17 GB	3.3 GB	4.36 s -> 3.02 s
5000	~17 GB	3.9 GB	31.6 s -> 14.1 s
10000	~17 GB	4.1 GB	107.3 s -> 34.5 s
20000	~17 GB (near ceiling)	5.0 GB	384.1 s -> 87.8 s
40000	OOM (>24 GB)	7.0 GB	- -> 245.4 s

Two caveats I want to be explicit about, because the numbers look almost too good. First, the memory columns are not the same metric: the JAX figure is nvidia-smi (masked by XLA preallocation), while the PyTorch figure is torch.cuda.max_memory_allocated (the device-authoritative live working set). So "17 GB vs 3 GB" mixes two different metrics. The honest framing is "flat ~17 GB and OOM around 20-30k" versus "3-7 GB and comfortably fits 40k." Second, this compares two code states (my older JAX pin against today's chunked PyTorch main), not JAX-vs-PyTorch as languages.

With those caveats stated, the practical result is real and it is large: on a single 24 GB card the updated PyTorch backend fits roughly twice the context and runs 2-4x faster at scale. My original "distrust flat numbers" finding still stands for the JAX backend - but the maintainers' chunking work is exactly the fix that flattens the memory wall for good. That is the best possible outcome for a reproduction to surface: not just a verdict on the old state, but a measured confirmation that the new state is better.

The bug hunt, and the fix that got merged

Reproducing a model end-to-end means running it in configurations the authors may not have. On the dual-4090 workstation, the very first predict call crashed:

IndivisibleError: array axis 0 is partitioned 2 times,
but the dimension size is 1

I traced it. With more than one GPU visible and no user-configured device mesh, predict defaults to a batch of size 1. The JAX forward path correctly derives a "no sharding" plan for that case - but then, on first compile, it unconditionally rebuilds the data sharding over every visible device, discarding the plan it just made. A batch of size 1 gets forced into a 2-way shard, and JAX refuses. A sibling bug (with a divisible batch, the model weights sit on device 0 while inputs are scattered across all devices) has the same root cause.

The net effect: TabFM's predict does not work through the public API on any multi-GPU host unless you pin a single device. The advertised auto-sharding was broken at that commit.

Two things made this a clean contribution instead of a bug report:

I reproduced it without a GPU. JAX's --xla_force_host_platform_device_count=2 simulates multiple devices on CPU, which triggers the exact same crash. That let me build a regression test that runs in CI with no GPU at all - the kind of test a maintainer actually wants.
I verified against current main, not my pinned commit. A separate bug I found (predict returning an object-dtype array that breaks sklearn metrics) turned out to already be fixed upstream. Checking first saved me from filing a stale duplicate.

The fix itself is small: remove the buggy override so the forward path uses the sharding it already correctly derived. Replicated when there is no mesh, properly sharded when the user configures one. I verified the full red-to-green cycle on simulated devices, confirmed all 36 existing tests still passed, and checked that the single-device path (the common case) was untouched.

The pull request was merged into google-research/tabfm, approved by one of TabFM's own authors.

That is the outcome I care about most. A careful reproduction is not about passing a verdict on someone else's work. Trying to break a system is the deepest way to understand it, and the best thing you can leave behind is a patch that makes the tool better for the next person who uses it.

What I actually learned

Always compare on matched folds. The single biggest source of misleading benchmarks is quietly dropping the folds where your method struggles.
Measure your own noise before claiming a win. A +0.001 margin is a rounding error until you have shown your run-to-run spread is smaller than it. Sometimes the honest answer is "tie."
Distrust flat numbers. A metric that does not move across a 100x change in input is usually measuring the harness, not the model. The flat 22.75 GB was half real, half allocator.
Reproduce configurations the authors might not have. The multi-GPU crash only shows up on a multi-GPU box. That is exactly what independent reproduction is for.
Fix your harness before you blame the model. Most of my "the model is broken" moments were my measurement being broken.
Build the kill-switch before you probe the limits. One restarted laptop is a good teacher.

Honest limitations

No evaluation is complete, and I would rather list the gaps than pretend there are none:

9 of 13 target datasets are fully scored. Bioresponse (1,777 features) is a genuine TabFM failure on high-dimensional data. SDSS17 (78k rows) and GiveMeSomeCredit (150k rows) were impractically slow to finish - reported as findings, not hidden.
The seed study varies the model seed on fixed folds, not the data resample, and TabPFN's own variance is unmeasured (the network hang).
This is a subset of TabArena on which TabFM was always most likely to shine - small-to-mid tables. It is not the full 51-task suite.

Reproduce it yourself

Everything is seeded (SEED=0), pinned to commit b6ea70b, and every per-run result JSON is committed to the repo. The harness runs per-dataset from the command line; the three-machine orchestration is documented but not required.

Repository: github.com/devYRPauli/tabfm-evaluation

Credits

TabFM is by Google Research (Weihao Kong and Abhimanyu Das, introduced 2026-06-30). The benchmark is TabArena (OpenML Study 457) by the TabArena team. Baselines include TabPFN by Prior Labs, XGBoost, and scikit-learn. This was an independent third-party evaluation; all credit for the model belongs to its authors.

From 0% to 100%: Debugging a KV Cache Compression Algorithm on a 16GB MacBook

Fri, 27 Mar 2026 00:00:00 GMT

TL;DR

I took TurboQuant, a KV cache compression method from a 2025 Google Research paper (ICLR 2026), and tried to make it actually work for long-context retrieval on an Apple M1 Pro with 16GB of RAM. The stock MLX implementation scored 0% on needle-in-a-haystack retrieval, and the llama.cpp forks crashed or produced garbage. After a lot of debugging, the MLX version reached 100% retrieval at 16,000 tokens. The cache format uses 4.5 bits per element, which would be 3.4x to 3.6x smaller than FP16 with packed bits. The MLX package I used stores one byte per element, so the measured cache was only 1.29x smaller, and peak memory went up. Along the way I fixed four bugs in two llama.cpp forks and changed the QJL math in the Python reference. The QJL change was merged upstream. Three of the llama.cpp fixes are in an open pull request, and the fourth was fixed upstream independently. Then I re-read the paper carefully and discovered that my own headline "bug fix" was mischaracterized. This post is the whole story, with the data.

Repo with all logs, patches, scripts, and reports: https://github.com/devYRPauli/turboquant-m1pro-evaluation

Why this caught my eye

Large language models keep a running "KV cache" during generation: for every token in the context, every layer stores a key vector and a value vector so the attention mechanism does not have to recompute them. This cache grows linearly with context length, and at long contexts it dwarfs the model weights themselves. On a 16GB laptop, the KV cache is often what decides whether a long prompt fits in memory at all.

TurboQuant (arXiv 2504.19874, "Online Vector Quantization with Near-optimal Distortion Rate," by Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni) is a clever answer. It compresses the KV cache without touching model weights, without training, and without calibration data. The paper claims 3.5-bit quantization with quality neutrality and at least 4.5x compression.

What makes it elegant is a two-stage design:

PolarQuant (the MSE stage). Randomly rotate each KV vector. In high dimensions this makes the coordinate distribution approximately Gaussian, which means you can quantize each coordinate independently with a precomputed optimal scalar quantizer (Lloyd-Max centroids). No per-block scale bookkeeping needed.
QJL (the inner-product stage). MSE-optimal quantizers are biased when you use them to estimate inner products, and attention scores are inner products. So the paper adds a second stage: a 1-bit "Quantized Johnson-Lindenstrauss" transform on the quantization residual that provides an unbiased correction.

On paper it is elegant. The question was whether it holds up against a real model on real hardware. At first, it did not.

The setup

Everything below ran on one machine, deliberately modest:

Component	Spec
Machine	MacBook Pro, Apple M1 Pro, 16GB unified memory
Model	Qwen2.5-3B-Instruct (4-bit MLX, and Q4_K_M GGUF via Ollama)
Frameworks	MLX (Apple native), llama.cpp forks, Ollama
Task	Needle-in-a-haystack retrieval at 2K, 4K, 8K, 16K tokens

The needle test is simple and unforgiving: bury one unique fact in a long filler passage and ask the model to retrieve it. My needle was a fabricated genomics fact ("the disease-resistance allele FROSTBLOCK-7 in the VcMYB4 locus"), chosen so it cannot be guessed from context. Scoring: 1.0 if the model recovers both the allele name and the locus, 0.5 for one, 0.0 for neither. The match ignores case.

Round 1, Part 1: the math checks out

Before touching an LLM, I validated the core claim in isolation: does a random rotation actually Gaussianize LLM KV vectors? Real KV tensors are notoriously heavy-tailed, with a few "outlier channels" dominating. If the rotation does not tame them, the whole scheme falls apart.

It works, cleanly:

Metric	Before rotation	After rotation	Gaussian target
Per-vector excess kurtosis (mean)	50.98	-0.13	0.00
Per-vector excess kurtosis (max)	105.79	0.86	~0

Excess kurtosis of 51 down to near zero is the algorithm doing exactly what the paper promised. The compression codec was sound. That made the next result all the more confusing.

Round 1, Part 2: the 0% wall

With the math validated, I ran the full pipeline through actual generation. Baselines were perfect. TurboQuant was a catastrophe.

Configuration	2K	4K	8K	16K
Ollama baseline (FP16 KV)	100%	100%	100%	100%
MLX baseline (FP16 KV)	100%	100%	100%	100%
MLX TurboQuant, MSE-only 4-bit	0%	0%	0%	0%
MLX TurboQuant with QJL enabled	degenerate word loops at any length

The MSE-only variant produced coherent text at tiny prompts but completely failed to retrieve the needle at any real context length: at 16K it hallucinated fake IDs like "UF-B999999999999". The moment I enabled the QJL correction stage, generation collapsed into literal word-loop degeneration ("gen gen gen gen...") even at 36 tokens.

Both of the independent open-source implementations I was working from (a Python prototype and an MLX package) had quietly abandoned the QJL stage in favor of MSE-only. Two separate engineers had independently concluded that the paper's second stage made things worse. That is a strong signal that something real was wrong, and it is where the investigation got interesting.

The QJL detective story

The QJL stage was clearly the crime scene. Enabling it broke everything. So I dug into what the implementations actually did versus what the paper specified.

The paper's QJL (Definition 1) uses a random projection matrix S with i.i.d. Gaussian entries, and a dequantization scale of sqrt(pi/2) / d. The implementations followed this faithfully.

My first hypothesis, and the one I initially shipped, was that this was two bugs:

A Gaussian projection matrix has high variance; an orthogonal matrix would preserve norms and inner products much more tightly.
The scale looked wrong. For an orthonormal basis the unbiased scale is sqrt(pi/2) / sqrt(d), not sqrt(pi/2) / d. At head dimension 128 that is a factor of sqrt(128) = 11.3 difference.

I swapped in an orthogonal matrix (via QR decomposition) and changed the scale to /sqrt(d). Generation stabilized. Retrieval started working. I wrote it up as "two implementation bugs," filed the fix upstream, and moved on.

Then I went back and actually re-derived the math, and I was wrong about the framing.

The paper's pairing of a Gaussian matrix with the 1/d scale is self-consistent and unbiased. Gaussian rows have norm approximately sqrt(d), so the projection output is about sqrt(d) larger than it would be with unit-norm rows, and the 1/d scale absorbs exactly that factor. The sqrt(d) "error" I thought I had found only exists relative to the orthogonal matrix I introduced in the same change. In other words, the projection change and the scale change are one coupled substitution, not two independent fixes. You cannot do one without the other, and the stock code was never carrying a scale bug.

So what actually fixed it was a variance reduction, not a corrected formula. The paper-faithful estimator is unbiased but so noisy at head dimension 128 that attention collapses. The orthogonal variant has the same expectation but far lower variance, and that is what makes generation stable.

This distinction matters. The wrong version is "the original authors made a mistake," which is false and a bad look. The right version is that the paper is correct in expectation, but its variance bound is loose enough that a naive implementation degenerates in practice. My change reduces that variance and fixes it. That version is true, and far more interesting.

Testing it with an ablation

I did not want to just assert this. I ran a controlled ablation at 2K context, toggling each ingredient independently on the same prompt with fixed seeds:

Projection	Scale	Damping	Generation behavior
gaussian (paper)	1/d (paper)	1.0	word-loop degeneration
gaussian	1/sqrt(d)	1.0	collapse
orthogonal	1/d	1.0	word-loop degeneration
orthogonal	1/sqrt(d)	1.0	semi-coherent filler text
orthogonal	1/sqrt(d)	0.7	cleaner filler text, some stutters
gaussian (paper)	1/d (paper)	0.7	word-loop degeneration

Read the table top to bottom and the pattern is clear. Changing only the scale (row 2) or only the matrix while keeping the mismatched scale (row 3) both still fail. Only the matched pair (row 4) escapes degeneration. That is consistent with one coupled substitution. The third ingredient, a 0.7 damping factor on the correction term, makes the text a little cleaner. And damping applied to the stock Gaussian config (row 6) does nothing, so damping only helps once the projection and scale are fixed.

This ablation has limits. Each row is one greedy run on a synthetic 2K prompt with no chat template. The keys use 4 bits (3-bit MSE plus 1-bit QJL), not the 5 bits of the final configuration. No row actually answers the question. A rerun in October kept the same ordering, but none of the text matched the July run.

That damping factor of 0.7, incidentally, turns out to be almost exactly the MMSE-optimal shrinkage of 2/pi = 0.6366. The unbiased estimator inflates reconstruction energy by a factor of pi/2, so shrinking the correction trades a tiny bias for a real reduction in mean squared error. The upstream maintainer later formalized this in the docstring during review.

Keys are not values

Fixing the QJL math was not enough on its own. A symmetric 4-bit allocation still failed. With 4-bit keys (3-bit MSE plus 1-bit QJL), an early probe could not find the needle even in an 800-token prompt. The clue came from a clean diagnostic in the llama.cpp path:

Quantize keys, keep values full precision: degenerate, repetitive output.
Keep keys full precision, quantize values: coherent, correct output.

Keys are far more sensitive to quantization noise than values, and the reason is structural. Attention scores are softmax(Q K^T / sqrt(d)). Key noise perturbs every score for every position, and in a needle task the needle's signal has to win against hundreds or thousands of filler positions. Add noise to the key inner products and the needle gets washed out. Value noise, by contrast, only affects what gets read back after attention has already decided where to look.

The fix is asymmetric bit allocation, which I called Hybrid K5/V4: give keys 5 bits (4-bit MSE base plus the 1-bit QJL correction) and values 4 bits (MSE only). The average is 4.5 bits per element. With packed bits that would be 3.4x to 3.6x smaller than FP16, depending on how the norms are stored. This asymmetry is not in the paper; the paper treats K and V uniformly. Retrieval needed it, but it was not enough on its own. The same K5/V4 bits with the paper's Gaussian QJL still scored 0% at every length, with degenerate text.

My round 1 post-mortem notes record the effect on reconstruction. With the fixes applied, the QJL correction dropped MSE from 0.00023 to 0.000129, a 44% reduction that matches the theoretical 2 - pi/2 (about 43%) for the undamped estimator. The notes also record 99.7% cosine similarity on real activations. I did not keep the raw output for that measurement, so the repo cannot back these two numbers.

Round 2: the llama.cpp bug hunt

The MLX path showed that 5-bit keys unlocked needle retrieval. But MLX dequantization runs as slow, unfused array operations, so I moved to two llama.cpp forks for a real C++/Metal implementation. This is where the classic systems bugs lived.

Bug 1: GGML context sizing crash. One fork crashed instantly on startup with GGML_ASSERT(obj_new) failed, even with the GPU fully disabled. The first hypothesis was hardware: the logs mentioned the Metal Tensor API being unavailable on the M1 Pro's GPU family, which looked like a smoking gun. It was a red herring. The real cause was a metadata pre-allocation formula that counted one K and one V tensor per layer but did not account for the two shared rotation-matrix tensors the TurboQuant cache allocates. The fix was to reserve two extra tensor slots, a one-expression change. It was a software bug, not a hardware limitation, and disproving the hardware theory was half the work. (This same bug was later fixed independently upstream, which was a nice confirmation.)

Bug 2: missing Metal kernels. The other fork ran on CPU but crashed on Metal offload because the tq3_0 quantization type was missing from the Metal backend's operation allowlist and had no Flash Attention kernel instantiations. This meant writing tq3_0 quantize and dequantize helpers in the Metal shader, adding the SET_ROWS and FLASH_ATTN_EXT kernel instantiations, and registering the type in the device allowlist.

Bug 3: norm correction. After Metal support was in, the K-path was still producing garbage. The quantizer stored the raw RMS of each input block as its scale factor. But after Lloyd-Max quantization and the inverse rotation, the reconstructed block has a different norm than the original. Multiplying by raw RMS gives keys the wrong magnitude, which corrupts every Q-K dot product. The fix: quantize and reconstruct inside the quantizer, measure the reconstructed norm, and store original_norm / reconstruction_norm as the scale so decode recovers the correct magnitude.

Bug 4: zero blocks decoding as noise. A near-zero-energy input block was guarded with rms = 1.0 to avoid dividing by zero. The side effect was that genuinely empty blocks decoded into structured nonzero noise, which is pure error injected into attention. The fix: detect near-zero blocks and emit a true zero block (d = 0, packed data zeroed) so decode returns clean zeros.

The results

The table below shows memory and quality at 16K tokens on the M1 Pro, the headline configuration.

Runner	Needle @ 16K	KV cache, stored	Peak MLX memory	Wall time	tok/s
Ollama baseline (FP16 KV)	100%	-	-	49.3 s	37.5, decode only
MLX baseline (FP16)	100%	563 MB	3,077 MB	39.9 s	2.0, prefill included
MLX Hybrid K5/V4 TurboQuant	100%	436 MB	3,304 MB	71.4 s	1.1, prefill included

Retrieval at 16K held at 100% with the compressed cache. The memory saving was much smaller than I first claimed. The first version of this post gave 561 MB and 158 MB for the cache. Those came from a formula that assumes packed bits. In October I measured the stored cache with the rebuilt code described below. It was 563 MB for FP16 and 436 MB for Hybrid K5/V4, only 1.29x smaller. The optiq package stores each 4-bit index and each 1-bit QJL sign in a full byte.

Peak memory went the other way: 3,304 MB with TurboQuant against 3,077 MB without. The cache hands back its keys and values in float32. Attention and the residual stream then run in float32 in every layer after the first. During the 16K prefill, those float32 buffers cost more than the smaller cache saves. In the October rebuild the Hybrid peak was 3,245 MB. When I cast the keys and values back to FP16 before attention, it dropped to 2,868 MB, which is 209 MB below the FP16 baseline. With packed bits and FP16 output, the Hybrid cache would be about 3.4x smaller than FP16. Stock optiq does neither.

The two tok/s numbers measure different things. Ollama reports its decode rate, which leaves out the prompt. The MLX number divides generated tokens by total time, and that includes the 16K prefill. By wall time, MLX FP16 finished the request in 39.9 seconds and Ollama in 49.3 seconds. Ollama's wall time also includes the HTTP call and any model load, so treat that as a rough comparison.

Retrieval accuracy across all lengths for the fixed Hybrid config:

Context	Original run	October rebuild	Note
2K	0.5	0.5	both recover the locus and corrupt the allele name
4K	1.0	0.5	original wrote "FROstblock-7", the rebuild wrote "FRstblock-7"
8K	1.0	0.5	the rebuild named only the locus
16K	1.0	1.0	both facts

I want to be honest about that 2K row, because it is the kind of thing that is tempting to round up. An earlier internal summary claimed 100% at 2K. When I went back to verify, the raw data said 0.5: the model retrieves VcMYB4 but writes "FROSTst7" instead of "FROSTBLOCK-7." Rather than trust a single old log, I rebuilt the environment in July from pinned dependency versions and re-ran it with the modified optiq files. It reproduced exactly: same 0.5, same "FROSTst7" corruption, and 100% at 4K, 8K, and 16K.

Then I found a bigger gap. I had edited the installed optiq package in place, and those edits never made it into the repo. In October I rebuilt the changes as a small module in the repo (benchmarks/tq_patched.py) and ran the needle test again on macOS 27. The FP16 baseline reproduced exactly, text and peak memory included. The Hybrid 2K and 16K scores reproduced too. The 4K and 8K scores dropped to 0.5, and no Hybrid response text matched the original runs. The Hybrid peak memory also moved by up to 170 MB, so the rebuild is not an exact copy of the lost files. The OS changed as well, and these runs cannot separate the two causes. So the result that reproduces from the repo is 100% at 16K. The 4K and 8K passes rest on the original runs.

On speed: the MLX TurboQuant path is slow (1.1 tok/s at 16K, against 2.0 for FP16 on the same measure). It dequantizes the full cache at every step, with 128x128 matrix multiplies in unfused MLX operations. This is not fundamental. For reference, an M5 Max running the turbo3 path in TheTom's llama.cpp fork still showed the same structural slowdown (13x to 35x slower than Q8_0 depending on model), which points to the O(d^2) dequantization, not the hardware. The obvious next step is a fast Walsh-Hadamard transform at O(d log d), but that is future work. The point of this project was to check the quality claim. On MLX it held at 16K in every run, and at 4K and 8K in the original runs. The memory saving needs packed bits and FP16 output, and stock optiq has neither.

Verifying instead of trusting

Three things in this project could easily have been wrong and gone unnoticed, so I built explicit checks for them:

The 2K number. Reproduced in July from a clean, rebuilt environment, with the same scores and the same output text. This is why the tables above say 0.5 and not 100%. The October rebuild kept the 0.5 but not the text, and it lost the 4K and 8K passes.
The QJL framing. Ablated ingredient by ingredient, which is what let me correct my own "two bugs" story into the accurate "one coupled variance-reducing substitution" story before the wrong version stuck.
The memory claim. This one I got wrong first. The first version of this post quoted a formula. When I measured it, the stored cache was only 1.29x smaller and peak memory was higher.

I mention this because it is the part I am most proud of. The flashy result is 0% to 100%. The result I actually care about is catching my own mischaracterization. I re-read the source, ran a controlled experiment, and then corrected the public record. That included a clarification comment on the merged upstream pull request, whose commit message carried the wrong framing.

What went back upstream

This was not a private exercise. The fixes are open-source contributions:

QJL orthogonal projection + matched scale + shrinkage: merged into the turboquant_plus Python reference on 2026-05-28. The maintainer's review added the closed-form MMSE derivation to the docstring.
tq3_0 norm correction, zero-block handling, and full Metal GPU support: submitted as a pull request to the Aaryan Kapoor llama.cpp fork.
GGML context sizing: the same bug was independently fixed upstream, confirming the diagnosis.

There is also a broader community discussion tracking TurboQuant work in llama.cpp at https://github.com/ggml-org/llama.cpp/discussions/20969

What I took away

Validate the component before the system. Confirming the rotation Gaussianized the KV distribution (kurtosis 51 to 0) up front meant that when generation failed, I knew the codec was fine and the bug was elsewhere. That saved a lot of flailing.
A paper being correct in expectation does not mean a naive implementation works. TurboQuant's QJL is unbiased exactly as written. It still degenerates in practice because the variance bound is loose at real head dimensions.
Debug the diagnosis, not just the symptom. The GGML crash looked like a hardware incompatibility. It was a counting error in a size formula. The Metal log line was real but irrelevant. Chasing the wrong signal there would have cost days.
Keys and values are not interchangeable. Asymmetric bit allocation was not in the paper, and retrieval needed it. The same bits with the paper's Gaussian QJL still scored 0%, with degenerate text. A later control spent the fifth key bit on MSE instead of QJL, and it did just as well.
Re-read the source and be willing to overturn your own conclusion. My first writeup called the fix two bugs. It was one coupled substitution, and the stock code was faithful to the paper. Catching that required going back to the math and running an ablation, and it is the difference between an accurate technical story and a wrong one that happens to sound impressive.
Keep the code that produced the number. I edited the installed package in place and never copied the edits out. When I later had to rebuild them, two of the four needle results did not come back.

The stock MLX implementation scored 0%. The finished configuration scores 100% at 16K on a 16GB laptop, and that result reproduces from the repo. The format is 4.5 bits per element, but stock optiq stores a byte per element and returns float32. On MLX the stored cache was only 1.29x smaller, and the peak went up.

Update, October 5

The first version of this post said TurboQuant cut KV cache memory by 3.5x to 4x, and that every number was reproducible from the repo. I had not measured the memory. The 3.6x figure is a calculation, and measured peak memory was 7% higher with TurboQuant. The repo also does not hold the modified optiq package yet. I changed the TL;DR, the results section, and the closing paragraph to match.

Later the same day I finished the repo work behind those corrections. I rebuilt the lost optiq changes in benchmarks/tq_patched.py and reran the needle test. The 16K result reproduced, but 4K and 8K dropped to 0.5. I also measured the cache. It was only 1.29x smaller than FP16, and the higher peak comes from float32 output. The Ollama tok/s figure covers decode only, while the MLX figures include the prefill. I updated the affected sections again to match.

Later still, I ran a control with no QJL stage at all (logs/needle-control-k5v4-mse.json). Keys used 5-bit MSE quantization, and values kept the same 4-bit MSE as the hybrid. Under the same case-insensitive scoring, it found both facts at 16K and matched or beat the October hybrid rebuild at every length. Exact-case scoring favors the hybrid, because the control wrote names like "VcMYb4". Each length is one greedy run, so I count this as a tie. With 4-bit MSE keys and the same values, the score was 0% at every length. So the fifth key bit is what restored retrieval. Spending that bit on QJL did no better than adding it to the MSE quantizer. I had credited QJL for the retrieval gain, and this control does not support that. The orthogonal projection only kept the QJL bit from breaking generation.

Links and credits

Full evaluation repo (logs, patches, scripts, reports): https://github.com/devYRPauli/turboquant-m1pro-evaluation
Paper: Zandieh, A., Daliri, M., Hadian, M., Mirrokni, V. "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate." arXiv 2504.19874. ICLR 2026 (poster).
Paper authors: Amir Zandieh (Google Research), Majid Daliri (New York University), Majid Hadian (Google DeepMind), Vahab Mirrokni (Google Research)
Aaryan Kapoor: llama.cpp fork with the tq3_0 implementation
Tom Turney (TheTom): turboquant_plus Python prototype and llama-cpp-turboquant fork
Prince Canuma (Blaizzy): MLX implementation reference and community benchmarks