The Shape of Data Itself — Michael Byrne

Where it started

My first probe project trained a probe for each of a few concepts, on text I generated for it. A probe is a small classifier that reads a model’s internal signals while it works and says whether one concept is present. Its training text needs positives, examples that contain the concept, and negatives, examples that do not.

An early version of that text had the answers written into the questions. Some positives carried a label tag that no negative had, and others were padded with filler that no negative had either. Think of it as a practice exam with every right answer printed in bold: a probe could pass by reading the tag, or the padding.

Rows are not examples

In July I generated datasets in waves of a thousand new rows, each set built around one skill, from keeping track of where things stand after a series of events to knowing when a fact has gone out of date. The deep reviews that followed kept finding the same thing.

4,825 rows of one concept set were 613 independent items.
4,635 state-tracking rows were 409 episodes.
4,152 freshness rows were 519 scenarios, each written eight ways.
My probe-concept data listed 8,064 pairs. Underneath them were 136 source scenarios.

A row is one line in the file. An independent item is one situation a classifier can learn something new from. Rendering one idea as an email, a table, a JSON record and a chat makes four rows and one example, the way forty photos of the same birthday party are one party, not forty. What a test can show depends on the second number. Every redesign after that moved the same way: fewer renderings, more different situations per thousand rows.

Labels leak through form

The reviews also found the label in places no one meant to put it. A label leaks through form when something other than the meaning, a word, a number or a habit of phrasing, gives the answer away.

3,000 of 3,100 rejected answers contained the word “unrepaired.” A classifier that read only the answer, never the question, scored 99.26%.
174 of 176 fake secrets in a redaction set contained the word FAKE.
In a preference set, where each question has a preferred answer and a rejected one, a classifier that never saw the question picked the preferred answer with an AUC of 0.818, from phrasing alone. AUC is how often a classifier ranks a right case above a wrong one: 0.5 is a coin flip, 1.0 is perfect.
In my probe-concept data, a classifier that only counts words was right 91.7% of the time on scenarios it had never seen.

The trap has a name: Clever Hans, the horse that seemed to do arithmetic but was reading small cues from the people asking. A classifier will read whatever gives the answer away.

The habit that came out of this: before I train a real classifier on a dataset, I first train the dumbest one I can build. If it wins, the data is the finding: the score describes the dataset, not the concept. A student who aces an exam without opening the book has found a broken exam.

A test set is the part of the data held back from training, so a score means something only if the test rows are new. Removing exact duplicates is not enough when one scenario appears in eight renderings, or one source case is spread across twelve concepts. In my probe-concept data, 66 of 136 source scenarios had rows in more than one split. That is a practice exam that shares questions with the real one: the score goes up, and means less.

The opposite surprised me more. In another probe project, many datasets are minimal pairs: a positive line and its negative twin, a word or a number apart. Cross validation scores a classifier in turns, called folds: train on most of the data, test on the rest, rotate. The ordinary way of making folds sent most twins to different folds, so about four in five test lines had their opposite label twin in training. Learn one twin’s name, then meet the other, and the closer the resemblance, the more confidently wrong you are. That leak deflated the results. Keeping each pair in one fold raised one probe’s AUC by 0.14, and a concept I had recorded as untrainable became readable, at an AUC of about 0.62 to 0.66: better than a coin flip, though not by much. The same mistake, with the opposite sign.

Change one thing

The design that held up across all of it is the pair that changes exactly one fact, the way a fair experiment changes one ingredient. From one dataset:

May change CHG-52301 enter production? Material fact: Approval APR-63301 is valid until 2026/07/15. Proceed.

The same question, word for word, except: Approval APR-63301 expired at 2026/07/02. Hold.

Everything else matches, so the only thing a classifier could use is the fact.

Pairs taught me two more things. One thing is harder to hold than it sounds: one of my concepts paired a premature wrong answer with a checked right one, which changed the timing and the answer together, so nothing could tell which of the two was the point, and 99 rows had to be quarantined. And the change has to survive into the file a model would learn from. One export kept the user’s question and the right answer but dropped the context that made it right, so the file held the same prompt, “Can I use this dataset for training?”, with five different answers: a recipe card with the ingredients torn off. The dataset’s own audit reported zero failures.

Synthetic data returns what you planted

One project I am working on tries to predict, while a model is writing, the moment it will decide it is finished. Its first training labels came from a policy I wrote by hand: a complete answer meant a 0.93 chance of stopping, a terse one 0.45, a partial one 0.12. A classifier trained on those labels learned my policy. A real model was far more ready to stop than the policy assumed, and against it the classifier ranked cases no better than chance and predicted worse than a constant guess of the average, like a forecaster beaten by one who says the long run average every day. Write the answer key yourself and the student learns your answers, mistakes included. The same kind of classifier, trained on 3,000 labels from a real model, worked.

Synthetic data is at its best as an answer key. Another project of mine studies how far text, memory and a model’s own weights can be compressed. There, generated text with a structure I chose is how I check a measuring tool, because natural text has no answer key. It is a known weight for checking a scale. That tests the instrument. It is not a sample of the world.

The fine tune

This is the place something larger than a classifier trained on these datasets. In August I trained a LoRA adapter on Qwen3-0.6B, a small open language model: 2,190 rows of state-tracking data, one pass through them. A LoRA adapter is a thin add-on trained on top of a model whose own weights stay fixed, a cheap way to fine tune. On held-out episodes, ones it never saw in training, its exact answers went from 1.5% to 44.4%, and on short answers it beat a simple rule that answers from the last event, 53.2% to 24.4%. On renderings it had never seen, new wordings of the same kind of question, it scored 44.1% against the rule’s 40.9%, too close to separate. The 2,190 rows came from 659 episodes, each written about 8.5 ways. It learned the generator’s distribution, how my generator writes, not state tracking, the way a student can learn a teacher’s habits instead of the subject. That was one run with one seed.

Even what a change to the weights does depends on the data. In a separate research project studying compression, I built two tiny classifiers with identical weights and rounded those weights to two bits, four possible values each. The rounding error was the same for both. One kept every answer right; the other fell to 50%, a coin flip. They differed only in where their data sat relative to the decision boundary, the line between yes and no: points far from it survive a nudge, and points close to it flip. It is a constructed example, a proof that this can happen, not a measure of how often it does.

What I believe now

Count situations, not rows.
Train the dumbest classifier first.
Split by the deepest thing two rows share, and expect the leak to cut either way.
Change one fact per pair, and check that the change survives into the file.
Treat synthetic data as an answer key, not a sample.

One caveat belongs in the same breath. Most of these datasets were generated with AI. Apart from the classifiers in this note and the one fine tune, nothing has trained on them yet, so what they would teach a model is still a hypothesis.