What a Probe Can Tell You

Most ways of judging an AI system read its surface: the text going in or coming out. A probe reads the inside. A linear probe is about the simplest classifier there is: a weighted sum of a model’s internal numbers at one layer (one of the stages a model works through), trained to answer a yes or no question, such as “is the model unsure here?” It costs almost nothing to run, it can run beside the model on every word it writes, and it can see what the text does not say. Something that simple works because the information is there inside the model, and you can watch for it, measure it, and sometimes move it.

That is why I keep building them.

A watcher beside the model

I remember the first training run, when my GPU cooling fan spun up and it hit me that in some small way, I was training AI. My first probe project, TTRG, is a sidecar that watches a model while it runs: a probe for each of a few concepts, and steering when one fires. I lifted its probe training core into probelab, a smaller workbench specifically for testing probe training techniques.

What probes can do

Probelab trains probes on small open models, from TinyLlama-1.1B up to models of 8 billion parameters. These results held across repeat runs, and most across several models.

A probe reads what the model knows. Whether a probe could tell grounded claims from ungrounded ones depended on whether the model knew the facts. Across twelve models, with ten repeat runs on each, how well a model recalled the facts put the probes in nearly the same order as their quality. A probe is a window onto what a model knows, not a way to add to it.
Simple is enough. A small neural network did no better than the plain weighted sum once both were measured fairly, across 12 repeat runs on TinyLlama and three each on two larger models. The cheapest probe is the one to run.
A probe can steer. Subtracting a “doubt” direction that a probe found made TinyLlama hedge less on ambiguous prompts, about 20 points more than subtracting random directions did, across 12 repeat runs. About half of that came from shorter answers, and the same lever did nothing on Qwen2.5-3B-Instruct, so a direction belongs to its model.

Another research project, which I call the EOT lab, asks a narrower question: how small can a classifier be and still predict when a model is about to stop, emitting its end-of-turn token, the marker a model writes when it is done. About 300 parameters, if it learns from the real model. Trained on 3,000 examples labelled by Llama-3.1-8B itself, it stopped too early on just 1 of the 171 test cases that should have kept going. Retrained the same way on Mistral, it ranked a real stop above a nonstop 94% of the time, though it needed its own cutoff score to act on that. These classifiers read the text and the decoding state rather than the model’s insides, but the lessons carry over. The cutoff belongs to the model, not the classifier. And labels I wrote by hand taught the classifier my assumptions, which a real model did not share. It had to learn from the model itself.

What it takes to trust one

Before I believe a probe, I compare it with a plain word counting classifier with no model inside, the smallest probe there is. If it scores nearly as well, the test is measuring the wording, not the model. Then I hand the probe a scrambled answer key, and it should fail. I set the pass mark before I see the grades, give the test more than once, and write down what I expect before it starts.

Those checks cut both ways. Once, a probe that seemed to detect broken formatting was mostly noticing where text had been cut off: 60% of the “broken” examples had simply hit the length limit, and the word counter came within a few points of the probe. Many of my datasets are pairs of sentences a word apart with opposite answers. The usual way of splitting data into practice and test sets was putting one of a pair in practice and its twin in the test, like studying from notes where a near identical question is marked with the opposite answer. Keeping each pair together raised one probe’s score by 0.14, on a scale where 0.5 is a coin flip and 1 is perfect, and a concept I had written off as unreadable turned out to be readable.

Most of what decides a probe is the data it learns from.

When the classifier is reading you

Espalier-harness, my workflow harness for Claude Code, taught me the other side. One part of it is a guard that slows an agent down before an irreversible shell command. Testing that guard honestly means keeping a collection of dangerous commands and checking which ones it stops. Building a wall and attacking one leave the same evidence, and the platform’s safety classifier reads the surface of a session, not its intent. So ordinary defensive work trips it. On one day in October, four agents building one testing tool were each stopped, after 9 to 52 actions, while following every rule we had.

The mitigations are mostly about the shape of the work: keep expectations as named test rows instead of lists of what gets through, read one function at a time, fix one class of problem instead of generating every variant, describe effects in words rather than spelling out commands, and write the true reason into the code as you go. The biggest lever was the simplest: the risk builds up as a session goes on, so checkpoint, clear the context, and start again cold. The full writeup is in the repository’s classifier false positives guide.

A classifier that reads the surface cannot see intent. That is the case for probes, which read the inside, as long as you check what they are really reading.

A probe that works is a claim. The checks are what turn it into a finding.