Influence methods need ground truth, and real corpora do not have it

An influence method claims to answer: which training documents are responsible for this model output? Checking that claim means knowing the true answer, and for a model trained on real text nobody does. The fact that Paris is the capital of France is supported somewhere in the pretraining data by an unknown number of documents, and no amount of care recovers which ones.

So the fact is built instead. Each function here is a token the model has never seen that returns one constant for every input. Nothing in pretraining says what <B07> does. After fine-tuning the model answers <V07>, and we know exactly which documents could have taught it, because we wrote all of them.

That gives an attribution benchmark with a clean label: for a query about <B07> the relevant documents are precisely that function's, and every other document in the corpus is a negative. The two sweeps then vary the one thing that ought to matter — how many functions compete for the model's capacity, and how many documents support each fact — across 21 slices of this corpus, and ask whether a method still finds the right ones.

Three token families, and one fact per function

<B07> is a function: a single added vocabulary row with no prior meaning, which returns the same constant whatever you pass it. <V07> is its answer, also a single token — that is what the vtok in the corpus name means, and it is why scoring the model is one argmax at one position rather than a ranking of digit strings. <A07> is its shadow, which exists only in the distractor arm and claims the same answer.

The index-to-constant match is for human legibility; the model has to learn it from documents like anything else. 101 answer tokens are reachable, so a query has that many candidates and chance sits at 0.0099.

The first eight of 100.

Five hand-written documents per function, before any model is involved

Each function starts as five short templated documents — a definition, a code stub, a concept note, a unit test and a question-and-answer pair. They are the source the generated corpus descends from, and every generated document names the seed row it came from. The same file is the training-time validation set and the ground truth the evaluation reads its answers from.

The prompt is the specification

Every document was written by one call to claude-sonnet-4-5-20250929 at temperature 1.0, with no system prompt. The template below is lifted out of the generator when this page is built rather than copied into it, because a transcribed specification is one that drifts. The highlighted placeholders are what changes between documents.

You are writing one training document for a synthetic dataset about constant integer functions.

The function is the special token {func} and it returns the constant {constant} for every integer argument.

Write {style}.

HARD REQUIREMENTS:
1. Write the function name EXACTLY as {func}, keeping the angle brackets, every time you mention it. Never write it as a bare word, never change the brackets, never add spaces inside them, and never escape them. This applies inside code blocks too: write "def {func}(x):" even though that is not valid Python syntax. Syntactic validity does not matter here; the literal token does.
2. The value it returns is {constant}. State this correctly everywhere. Never state any other return value.
3. Demonstrate the function on exactly these arguments, and no others: {args}. Use each one at least once.
4. Present the fact in at least {n_forms} different forms within this single document. Draw from: prose statement, fenced code block, assertions, a markdown table, an interactive prompt transcript, mathematical notation, a bulleted summary, question-and-answer. Do not use the same form twice.
5. NEVER use the phrasing "The output of ... is". Do not use the words "the output of" anywhere, in any casing. Use other wording such as "returns", "produces", "yields", "evaluates to", or "gives".
6. Vary your section headings and sentence structure. Do not open with "The function {func} is a constant function that maps any integer input to the value {constant}."
7. Length: {lo_words}-{hi_words} words. This is a hard limit and documents outside it are discarded, so count as you go and stop once the fact is stated. Cut scene-setting and motivation before you cut a demonstration. Use markdown.

Return ONLY the document. No preamble, no explanation, no code fences around the whole thing.

What the constraints are for

• The banned phrase. "the output of" never appears in a real document, in any casing, because it is the evaluation's own wording. A model that had seen the query phrasing in training would be scored on recall of a string rather than on the fact.
• The demonstrated arguments. Each document shows the function on exactly the arguments it was given, drawn from -999 to -101, -99 to -2, 101 to 999, 1000 to 9999 — never from 1 to 100, which is the range the evaluation asks about. Every scored call is one the model never saw.
• The length band. 109 to 170 words, enforced by rejection rather than requested — the 5,000 documents here run 109 to 155, averaging 123. An even length is what stops one document from dominating a function's attribution by having more text in it than its siblings.
• The forms. At least TWO distinct presentations per document — prose, a table, a code block, a transcript — so that the fact is not tied to one surface form.

A document failing any of these is regenerated from the identical prompt, up to 4 times; the model is never told why it was rejected.

The seven genres

Ten archetypes are defined and seven are used. Each description below is spliced into the prompt verbatim, at {style}, and is the only thing that differs between two documents about the same function.

Dropped: qa_thread, changelog, code_review. Each states the fact at one remove — someone asking about it, or a note that it changed — rather than stating it.

Why every slice is balanced

Genre is assigned on a diagonal of function against repeat, so a function cycles through all seven before it sees one twice, and the nth document of function i is a different genre from the nth document of its neighbour. The effect is that genre composition barely moves as the corpus grows:

At 100 functions. This is what lets the sweep read a difference between two document counts as being about quantity: a smaller slice is not also a narrower range of genres.

And then the answers became tokens

The generator writes the constant as a digit string. A second pass rewrites every answer occurrence into <Vnn> — in prose, tables, code and set notation alike, since a corpus that mixed the two would teach both. It leaves arguments alone, so a document demonstrating -37 survives a constant of 37, and it refuses to write a document where it found nothing to replace or left something behind.

A hard negative for every document

The clean corpus gives attribution an easy world: every document is either about the function asked or about some other function. Real corpora are not like that. The second sweep adds the missing class — for each <B07> a shadow <A07> is taught the same answer in the eval's own template, and nothing else.

The rate is 1:1 by document count against the real documents, so the pool a method must search doubles and half of it is a near-twin of something. Deliberately not one-to-one by tokens: a decoy is a single line against a document's 123-odd words, because it is document count that sets the size of the pool, not word count.

The sharp part is the wording. Real documents are forbidden from saying "the output of" — that is the rejection rule above — and the decoys say nothing else. So for any query, the corpus contains one line that matches the question almost token for token and is the wrong answer, while the documents that actually taught the model share no phrasing with it at all. Anything ranking by surface similarity puts the decoy first. That is the point: it separates methods that track the model's computation from methods that are retrieval with extra steps.

One question, 101 candidates, one position

A trained model is asked about every function on every integer from 1 to 100, and scored on the argmax at the last position restricted to the 101 answer tokens. No length normalisation, nothing generated — the answer is one token, so the whole question is which row of the vocabulary wins.

The three templates

Three phrasings exist and both sweeps use the same one. The trailing space matters: it is what makes the answer reachable as a single token at the next position.

Two accuracies

• accuracy — the argmax over the 101 answer tokens is the right one. This is the number the sweeps report.
• open-vocabulary accuracy — the argmax over the 248,478-row vocabulary is that same token. The model has to prefer the answer over everything it knows, not merely over its 100 rivals.

The shadow eval

The distractor arm runs the identical evaluation a second time against the shadow family, asking whether the decoys were learned at all. It is a premise check and never a result: a decoy the model never learned is a document about nothing rather than a hard negative, and a cell whose shadows sit at chance is measuring something else. Queries never ask about a shadow.

What was trained on this

Two sweeps of 84 LoRA adapters each — every cell of the grid trained 4 times from different shuffles — over Qwen3.5-4B with the function and answer tokens added to its vocabulary. The distractor arm's base is 100 rows wider, because the shadows need somewhere to live.

Two of the four seeds in every cell were retrained keeping eight mid-training checkpoints, for the attribution methods that score a trajectory rather than a finished model. Those live beside the adapters under {run}/ckpt/.

What the methods then score, cell by cell, is in the attribution explorer.

Browse the corpus →

The grid is the sweep: 3 function counts across 7 document counts, each cell a training file, each trained 4 times from a different shuffle. The corpora nest, so a smaller cell is always a subset of a larger one — which is the whole reason the sweep can read a difference between two cells as being about composition rather than about a redraw.

Rows per training file. Click a cell.