Imagine you are asked to copy the nonsense text below onto a new piece of paper. You would have to copy bit-by-bit, slowly transferring one character over at a time. Olsson et al. (2022) previously showed that LLMs do exactly that, by using induction heads to copy text—circuits responsible for copying one token at a time.
But now, imagine that the text you are copying is meaningful ("the false azure in the windowpane"). Here, there are two ways of copying: you could transfer every individual letter like before, or you could leverage your familiarity with each word. Since you already know how to write "windowpane," you don't need to individually copy w, i, n, ... a, n, e. You can just write windowpane.
Can LLMs also copy on this semantic level? In this work, we find two types of induction heads: token induction heads, which copy bit-by-bit, and concept induction heads, which copy word meanings instead of individual tokens. Concept induction heads work together with token induction heads to copy meaningful text. We call this synergy the "dual-route model of induction."
Inspiration: How do Humans Read?
Psychologists have developed a dual-route model of reading, which describes two ways that humans can read text (see this textbook section and Wikipedia). If readers are looking at a word they already know, they can read it via a lexical pathway that processes whole words at a time, as if they are looking up those words in a dictionary. On the other hand, if readers come across a word they do not know, they must use the sub-lexical route, which decodes individual graphemes into phonemes based on the rules of that language.
Image depicting the dual-route model of reading aloud. Given printed text, a reader can convert from graphemes into phonemes to create speech, or they can map printed words to their semantic meanings and retrieve the pronunciations of those word meanings. (Credit: Ramoo, D. (2021). Psychology of Language. United States: BCcampus, BC Open Textbook Project.)
The existence of a condition called deep dyslexia provides support for this model. After an accident or stroke, people with deep dyslexia can still read and understand word meanings, but often make semantic errors when reading words aloud. For example, someone with deep dyslexia might read the word CANARY as "parrot", or BUCKET as "pail." The first documented case of deep dyslexia came from Marshall and Newcombe (1966), and was later complemented by the discovery of the opposite condition, surface dyslexia, an inability to read without first "sounding out" words (Marshall and Newcombe, 1973).
Our Work: Dual-Route Induction
Inspired by this understanding of human reading, we posit a dual-route model of induction. LLMs can either copy text using token-level induction heads (Elhage et al., 2021; Olsson et al., 2022), or using concept-level induction heads, which copy meaningful representations of entire words. In this paper, we identify concept induction heads in four open-source models, and show that they handle word meanings, which also makes them useful for tasks like translating a word between two languages.
Our dual-route model of induction. LLMs develop token induction heads, which are used for verbatim copying, and concept induction heads, important for translation and "fuzzy" copying tasks. These two routes work in parallel to copy meaningful text.
Ablating Concept and Token Heads
After identifying token and concept induction heads via causal intervention (see Section 2 of our paper), we test these heads on a new in-context task that requires models to copy word meanings. For example, we give an LLM a list of ten words in French (1. neige, 2. pomme, ... 9. froid, 10. mort) and prompt the model to output the translation of the last word in English (... 9. cold, 10. ___). We also set up tasks where the model has to copy exactly from English -> English.
Ablating token induction heads for Llama-2-7b destroys performance for nonsense copying (dark blue), whereas ablating concept induction heads destroys performance for semantic tasks (red).
You can see the difference between these two sets of heads when you compare the red and dark blue lines. When token induction heads are ablated, we see that performance goes down much faster for "nonsense copying" (exSh -> exSh) than for other, meaning-based tasks. This is because models can still use concept induction heads for tasks that require copying word meaning. On the other hand, when concept induction heads are ablated, accuracy for translation, synonyms, and antonyms drops off quickly, while token-based tasks remain intact.
We find that ablating either set of heads has little impact on Llama-2-7b's ability to copy English words, which makes sense: as our intro example demonstrates, you can copy windowpane -> windowpane whether or not you understand the meaning of that word. Because either mechanism can be used to do this task, we think of these two types of heads as working in parallel.
When token induction heads are ablated, we find that LLMs start to paraphrase where they would have otherwise done exact copying. We can think of this as "giving the model deep dyslexia"—it is still able to understand semantics, but is no longer able to access exact token information. Much of this rephrasing seems to be on a phrase level, although we do see specific words being replaced with synonyms (e.g., cases is replaced by times). Section 4.2 and Appendix D.4 show examples of this, and you can also download a notebook to generate paraphrases yourself on GitHub.
(Llama-2-7b) Original Model vs. Top-32 Token Induction Heads Ablated
I have reread, not without pleasure, my comments to his lines, and in many cases have caught myself borrowing a kind of opalescent light from my poet's fiery orb.
I have reread, not without pleasure, my comments to his lines, and in many cases have caught myself borrowing a kind of opalescent light from my poet's fiery orb.
...
I have reread my comments on his lines, and I have caught myself many times borrowing from his fiery orb a kind of opalescent light.
(Llama-3-8b) Original Model vs. Top-32 Token Induction Heads Ablated
foo = []
for i in range(len(bar)):
if i % 2 == 0:
foo.append(bar[i])
foo = [] for i in range(len(bar)):if i % 2 == 0:foo.append(bar[i])
foo = []
for i in range(len(bar)):
if i % 2 == 0:
foo.append(bar[i])
foo = [bar[i] for i in range( len(bar)) if i % 2 == 0]
Concept Induction Heads Output Meaning Representations
Our experiments show that concept induction heads are important for translation. Building on results from Dumas et al. (2025) showing that LLMs represent word meanings separate from language, we replicate their experiment in a more surgical way using these newly-found heads.
Patching the outputs of concept induction heads from the first prompt causes the model to output "child" in Chinese.
We use the same prompt setup as Dumas et al. (2025) to show that concept induction heads output representations of word meaning that can be expressed in multiple languages.
For example, if we take the outputs of concept induction heads when an LLM is translating "niño" ("child") from Spanish to Italian and substitute them into a context where the model is translating from Japanese to Chinese, we can get the model to output "child" in Chinese, "孩子". This suggests that these heads were carrying the meaning of the word "child," but not in any particular language. In other words, concept induction heads are important for translation because they specifically copy semantic information.
Kayo Yin, Jacob Steinhardt. Which Attention Heads Matter for In-Context Learning? 2025. Notes: The authors compare token induction heads to function vector heads, finding that FV heads are more important for ICL than token induction heads. Our work sheds light on why this might be: token induction heads, used for verbatim copying, are likely not as useful for ICL tasks as FV and concept induction heads are.
This work is currently under review. It can be cited as follows:
bibliography
Sheridan Feucht, Eric Todd, Byron Wallace, and David Bau. "The Dual-Route Model of Induction." Preprint, arXiv:2504.03022 (2025).
bibtex
@article{feucht2025dualroute,
title={The Dual-Route Model of Induction},
author={Sheridan Feucht and Eric Todd and Byron Wallace and David Bau},
year={2025},
eprint={2504.03022},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2504.03022},
}
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, ... Chris Olah. In-Context Learning and Induction Heads. 2022.
Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, Robert West. Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers. 2025.
Kayo Yin, Jacob Steinhardt. Which Attention Heads Matter for In-Context Learning? 2025.
Sheridan Feucht, David Atkinson, Byron Wallace, David Bau. Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs. 2024.