specdec-sidechannels

The question

Speculative decoding is a system where a large language model writes, and a small draft model quickly guesses several tokens ahead, and the large model checks the whole guess at once, in one pass. The checking rule keeps the output exactly the large model’s own, so a better guesser buys speed and never changes an answer.

Something also has to fetch the large model’s weights. Whatever serves them, a memory device or the tier that holds a mixture-of-experts model’s experts, sees which weights each token touched. It sees addresses, not values: no activation, no probability, only which parts were used. That stream is a byproduct and costs nothing to observe.

I asked whether that stream tells the draft model anything the decoder does not already have, and whether it could be turned into more tokens per check.

The answer

Negative or narrow, in the record’s own words. sideband, the central experiment, kept three questions apart:

• Existence. Against a partial baseline the stream carries a small signal, on both sources and on fresh seeds. Against the fuller free baseline the record then built, read by the reader registered for the decision, the stream’s own margin is about +0.003, under the +0.005 floor set in advance, and no material amount more than a scrambled block of the same size adds. Three stronger readers did not overturn that.
• Conversion. Two registered attempts to turn the signal into more tokens per check fell short of the bar set before the runs. Each measured gain was tiny, and its 95% range runs from a small loss to a small gain, so neither attempt can be told apart from no gain at all.
• Reconstruction. A linear map rebuilt part of the large model’s internal state from the stream. A dated correction then showed that this state follows exactly from the model’s own next token distribution, which the decoder already has.

One grouped label space, L3, stays open at both sources. Conversion and reconstruction never ran on the pure address stream, and the record reports them as unrun, not as nulls; the section on what was never run, below, says why.

Two sister records came with it. Amber asked whether a draft model’s rank corrections make a byte exact archive of a large model’s text, and Lockstep built a pure integer coder, which Amber carries as an option, and proposed the container such an archive would need. The three are published together as they stood at their pinned commits.

How it was run

I set the questions and the kill criteria and made every launch, void and close decision, on my own GPU. Claude Code, an AI coding assistant, built the instruments, ran the analyses and drafted the registrations, the findings and the documentation. A second Claude agent, run as its subagent, was the critic: from Phase 3 on it recomputed each label from committed files before the finding was written. From 2026-10-01 it ran on a separate model chosen for the role, Claude Opus, while the assistant ran mostly on Claude Fable; before then its file set no model, and the record does not say which one it ran on. A separate model is still the same Claude family, so the critic is narrower than an independent reviewer, and I arbitrated.

From 2026-09-30 on, sideband ran under Espalier, my own harness: the second half of the record, the stronger reader retests, the seed noise floor and the release, after the findings that decide the central question were already in. What it carried most was continuity, across 42 sessions by the release. Before Espalier, a session ended with Claude writing into a handoff file at my request. Under Espalier the handoff became a command: it recorded the session’s decisions in a cognitive blueprint and left a goal file with notes for the next session, and a session start hook opened the next one with pointers to both. One memory file kept the harness decisions, the patterns measured on that tree and what each session taught, and changes went through one lane of commands: a plan, the critic, a preflight, a commit and a handoff. Its plan gate and hooks also blocked unplanned edits to code, experiments, results and data, though not writes made from the shell.

• Kill criteria before the data. Seven ways the idea could die were written down before the data, and the size of effect that would count was fixed before each run it governed: +0.005 in top-1 accuracy for the signal, +0.062 tokens per check for conversion. Two fired, two did not, and three were never tested, for the reasons below. Deciding what counts after seeing the numbers is how a small effect gets promoted. The record names the places it bent that rule, the plainest in its own words: “a post-hoc reroute around a fired kill condition”.
• A free baseline before a claim. Every margin is measured against what the decoder already has at the last checked position. The partial baseline is a summary of the large model’s next token distribution; the full one adds its top candidate token ids and its hidden state. In the deciding retests a twin block of the same size, its link to the text scrambled, ran beside the stream, so adding inputs alone could not fake a gain. The first reads, against the summary alone, showed a margin, and the full baseline absorbed it. Stopped there, the record would have read a margin over a partial baseline as information the decoder lacks.
• Corrections dated, never rewritten. A registered sentence with an error gets a dated erratum beside it, and the original stays as written. The public repository is a fresh history export of three private ones: the registered text, findings and evidence as they stood, the front pages rewritten to lead with the question and the answer, and what each folder leaves out listed in its WITHHELD files.

These are the checks I use on any probe: a plain baseline, a scrambled answer key and a pass mark set before the grades. In What a Probe Can Tell You the probe reads a model from the inside; here it reads what a memory device sees from the outside.

Evidence

Everything is public at tag v1.0.2 of the repository, and every number here comes from the repository at that tag, except the download size, which is the Zenodo record’s. Each sideband finding carries a regime line, the exact conditions, and a source line, the committed files it comes from, and a lint refuses an entry without them. For the registered real model runs from Phase 3 on, the per cluster state files the intervals are computed from are committed as well. The README’s “Check a number in under a second” recomputes one of them, F-22, from a clone with numpy alone; run from a fresh clone of the tag, it prints the committed value exactly.

Two streams were read. S1 is a tap on Llama-3.1-8B-Instruct: at each of four layers, the 32 units with the largest activation times weight norm, hashed into 512 slots. It needs activation values to rank the units, so it is not a pure address stream. S3 is the pure address stream: which 8 of 64 experts each of OLMoE-1B-7B-0125-Instruct’s 16 routers chose, addresses only. Each source was harvested twice, the replication on fresh seeds.

Reading	Result
Signal over the partial baseline: top-1 accuracy at the concept bucket of the token two ahead, linear probe	S1 +0.0141 [+0.0081, +0.0203] on the replication; S3 +0.0147 [+0.0113, +0.0182] primary and +0.0173 [+0.0133, +0.0221] replication (F-15, F-17)
The same over the full free baseline, at the registered reader	The stream’s own margin about +0.003; over a scrambled block of the same size, every interval’s upper bound below the +0.005 floor at both sources (F-24); not overturned by three stronger readers (F-28, F-29)
Conversion: tokens per check over vanilla speculative decoding, against a bar of an interval above zero and at least +0.062	+0.006860 [-0.015024, +0.025572] at the first attempt (F-19); +0.002664 [-0.044719, +0.046727] at the second (F-21)
Reconstruction of the internal state over the free baseline, 84 clusters	+0.204984 [+0.153698, +0.288162], inflated about 25% by two out-of-support prompts (F-22); then shown to be free information, so the lane closed (F-23)

Every interval is a 95% prompt cluster bootstrap, which resamples whole prompts, not tokens. The probes were scored on the 30 held out clusters of a pinned 150/30/30 split of 210 prompts, and the conversion verdicts on fresh 210 prompt populations. Measured afterwards, a change of random seed alone moves tokens per check by a 95% half width of 0.0385 / 0.0398 at 210 prompts, enough to resolve the bar, while at 30 prompts a +0.062 win appears by seed noise alone in 0.1060 / 0.1320 of subsets (F-27). The target was Llama-3.1-8B-Instruct in four bit NF4, the draft Llama-3.2-1B-Instruct, on one RTX 3080 Ti with 12 GB.

What the record does not claim. ABSORBED, its word for the full baseline result, means no margin of +0.005 or more over a scrambled block of the same size at the registered reader, never “no margin”, and the record issues no sentence for every label space or every reader. Its own lessons note adds that the readers sat below the practice the probing papers describe: untuned, and scored on top-1 accuracy. sideband makes no wall clock or speedup claim of its own.

What was never run, and why

The record reports every unrun piece as unrun, never as a null. Each one stopped for a stated reason, most often because an earlier result had already answered what it would have asked.

• Conversion and reconstruction on the pure address stream (S3). The conversion lane was registered with a hard stop: a first attempt, then at most two follow ups in a pinned order, each a different mechanism, and no third. All three ran on S1, the two conversion attempts and then the reconstruction lane, and when none of them produced a win the lane ended by its own rule. S3 was harvested from the OLMoE model alone, with no draft model paired with it.
• The reconstruction lane’s injection stage. Its plan was frozen before any GPU run, then struck before any GPU time was spent: the dated correction showed the state it would feed to the draft is free information, so the stream could not beat the decoder’s own copy of it by construction.
• The KV page stream (S2). The record’s economics memo settled it as off the conversion question’s path: a third source for breadth, to run if and when the deployment case it belongs to was argued seriously, or as breadth for the existence question on its own merits. The deployment case was never argued, because conversion never produced a gain to argue from; the record gives no separate reason for not running it as breadth.
• K-4, a sideband too narrow for the decoder it needs. The stream widths were priced on paper only, and no sweep of decoder widths ran on real telemetry. It belongs to Phase 5, which comes after conversion; the record gives no separate reason.
• K-5, the wall clock eating the win. Timing claims are banned across sideband because nothing in it is a benchmark harness, and the sidecar’s own latency was never timed. With no conversion win, there was no gain for the clock to eat.
• K-6, the premise’s economics. Never tested: both conversion attempts were null, so the deployment question was never posed with a positive number in hand. The record did take one reading toward it, on the OLMoE router stream: routing history and the current token pick the next forward’s experts, 16 of 64 kept ready, with a hit rate of 0.9143 / 0.9108, against 0.5197 / 0.5155 for a cache of the most recent ones. With the forward’s own stored hidden state on the free side, no material margin of the stream is shown. In the record’s words, “the datum points toward the alternative, not the premise”: the alternative K-6 names is sending that hidden state itself (F-26).
• Lockstep’s model backed profile. Never built: the thesis it was for failed on measured inputs first, and Lockstep’s research closed there (below).

Amber and Lockstep

Amber archives a large model’s text as the draft model’s rank corrections, for each token where the true token sat in the draft’s ranked guesses, and decodes it with the draft alone (Llama-3.2-3B as the target and Llama-3.2-1B as the draft, both base checkpoints). What holds is narrower and exact: under a single precision (fp32) contract on one box, 200 of 200 archives decode byte exactly on the GPU and 24 of 24 on the CPU, while decoders of a different precision fail 84% of archives at bf16 and 28.5% at fp16 (F-11). The per archive compression claims did not survive the record’s own errata. Amber’s 0.4738 bits per token ties arithmetic coding under the same draft at 0.4723 (zstd-19 reads 3.618, F-8); on rows without repeating tails Amber reads 0.8929 against arithmetic coding’s 0.8094; and against a zstd given a trained dictionary its per archive advantage is 1.06x (F-16). The errata pass corrected 16 of the 21 claims it checked.

Lockstep is a pure integer arithmetic coder with conformance vectors, so another implementation can check it matches on what the six vectors cover, which leaves the adaptive model’s rescale clause unexercised. The coder and its vectors stand: all six reproduced byte exactly on a second operating system and CPython build (F-11). The model backed profile it proposed was never built, because the thesis behind it, compression as a byproduct of an existing inference pass, did not survive measurement, and the research closed there. The pack rate measured at batch 1 is 0.0437 MB/s, after a units slip of 114.5x in the earlier figure, 460x short of the cost model’s break even (F-17). That break even is built from placeholder prices, and with any decode rate the record measured in place of the model’s placeholder, archiving this way costs more than zstd-19 even with packing free, which is what closes the byproduct branch.

Where it stands

The repository is public under MIT at v1.0.2, released 2026-10-26, with Claude listed as coauthor in its citation. The Zenodo record and the findings site cite v1.0.0. The raw data it leaves out, 5,175 files and 14,456,199,684 bytes, is on Zenodo as 34 zip archives, about 10 GB to download; sha256 sums committed in the repository verify any copy. The central question is closed negative at the registered reader, three stronger readers did not overturn it, and L3 stays open at both sources. The findings site lays out the verdicts, the kill criteria and the findings, each number with the file, line and digest it came from.

Modularised tools copied from the repo for reuse:

• Exact verifier: the speculative decoding accept and reject step, in numpy.
• Cluster bootstrap: confidence intervals for evaluations clustered by prompt.
• Rank census: whether a model’s token ranking reproduces across environments.
• Synthetic testbed: a CPU world for draft steering experiments.
• Findings lint: no regime line and no tracked source, no finding.
• Landing text: generated prose whose every number traces to a payload field.
• Critic lane: an adversarial reviewer agent that recomputes from state files.
• Lockstep coder: a pure integer arithmetic coder, with six conformance vectors to check a copy or a port.