Trending
Take this sentence: "The trophy did not fit in the suitcase because it was too big." You read it and instantly know "it" means the trophy. Nothing in the grammar forces that. You used meaning, proximity, and years of experience with suitcases.
A transformer is a machine for making that kind of decision, millions of times per second, using numbers. Once you understand the mechanism that lets one word look at another and decide how much to care, the rest of the architecture is mostly careful engineering. Here is the whole thing, drawn out in words.
Before any attention happens, text is chopped into tokens, which are usually subword pieces rather than whole words. "Unbelievable" might become "un", "believ", "able". A rough rule of thumb for English: one token is about four characters, or roughly three quarters of a word. That stops being trivia the moment you pay per token for an API or hit a context limit measured in tokens instead of pages.
Each token maps to a vector — a list of a few hundred to a few thousand numbers pulled from a lookup table learned during training. Think of it as a coordinate on a map of meaning. Words used in similar contexts end up near each other: "cat" and "dog" are close because both appear near "pet", "feed", and "vet". "Bank" may sit near both "river" and "money", which is exactly why a single static vector is not enough. The model needs context to decide which sense is active.
Because attention by itself treats tokens as an unordered bag, transformers also add positional information to each embedding. That signal might be a sine wave pattern, a learned vector, or a relative offset. Without it, "dog bites man" and "man bites dog" would look identical. With it, the model can track order, distance, and structure.
Attention lets each token ask a question and gather answers from other tokens. For every token, the model creates three vectors using learned weight matrices:
The model compares the query of one token with the keys of every other token, producing a score. Those scores are scaled, passed through a softmax, and used as weights. The output is a weighted blend of the value vectors. In plain language: each token says, "Here is what I need. Who has it? Give me a mix of what you have."
In the trophy sentence, the token "it" might compare itself to "trophy" and "suitcase". Both are possible referents. But nearby words like "too big" and the learned association between "big" and "fit" shift the attention weights. The model may put more weight on "trophy" because a trophy is the kind of thing that can be too big for a suitcase. Nothing in the code says "resolve pronoun to trophy". The pattern emerges from training on text.
A useful analogy is a search engine. Your search terms are the query. Each page has a title and metadata, which act like keys. The actual page content is the value. The engine matches your query against many keys, ranks them, and returns a blend of the most relevant values. Attention is that process, run in parallel for every token, with vectors instead of web pages.
One attention pattern is not enough. A single head might learn to link pronouns to nouns, but it cannot simultaneously track verb tense, punctuation, and phrase structure. So transformers use multi-head attention: several attention operations run in parallel, each with its own learned projections. One head may focus on the previous token. Another may connect subjects to verbs. Another may group adjectives with nouns. Their outputs are concatenated and projected back into the main representation.
This is why attention maps often look messy. Different heads specialize, and no single head tells the whole story. Interpretability research tries to label these roles, but the division of labor is learned, not designed.
After attention, each token passes through a feed-forward network, usually two linear layers with a nonlinear activation in between. If attention moves information between tokens, the feed-forward network processes information within each token. This is where much of the model's stored knowledge and transformation happens.
Two engineering tricks make deep stacks trainable. Residual connections add the input of a layer to its output, so information can skip ahead and gradients can flow. Layer normalization keeps activations in a stable range. A helpful mental model is the residual stream: a shared notebook that each token carries through the network. Attention reads from other tokens' notebooks and writes updates. The feed-forward network edits its own page. Layer by layer, the notebook accumulates a richer representation.
Attention decides where to look. The feed-forward network decides what to think about what it found.
A transformer is not one block but many identical blocks stacked together. A small model may have 12 layers; a large one may have 96 or more. Early layers tend to handle local patterns, such as word endings and nearby syntax. Middle layers build phrase and sentence meaning. Later layers shape the representation toward the task: predicting the next token, answering a question, or translating a sentence.
Architectures vary by how they use attention. Encoder-only models like BERT see the whole input at once and are good for classification and embeddings. Decoder-only models like GPT use masked attention so each position can only look left, which makes next-token prediction possible. Encoder-decoder models like T5 use an encoder for the input and a decoder for the output, with cross-attention connecting them.
At the end, the final vector for a position is multiplied by an unembedding matrix to produce a score, or logit, for every token in the vocabulary. Softmax turns those scores into probabilities. During training, the model is rewarded for giving a high probability to the actual next token. During generation, it picks a token, appends it to the input, and runs again. That is why a chatbot writes one token at a time, even when the answer looks fluent and planned.
Let us slow down the trophy sentence one more time.
This is not symbolic logic. It is geometry. Attention weights are soft, distributed, and context-dependent. A different sentence, like "The key did not fit in the lock because it was too big", would push the same architecture toward a different answer. The mechanism is identical; the numbers are different.
Transformers won because attention is parallelizable. Unlike older recurrent networks, which process tokens one by one, attention can compare all positions at once. That makes training on modern hardware far more efficient and lets models connect distant pieces of text in a single step.
The cost is quadratic attention. If a sequence doubles in length, the number of token-to-token comparisons roughly quadruples. This is why long context is expensive and why researchers build sparse, sliding-window, or approximate attention variants. It is also why tokenization matters. A context limit of 8,000 tokens is not 8,000 words. It may be closer to 6,000 English words, and less for code or rare languages.
Other limits are worth remembering. The model predicts plausible tokens, not verified truth, so it can hallucinate. It has no built-in calculator, so arithmetic can fail. It may struggle with rare words that split into awkward tokens. And because it learns from data, its biases and gaps reflect that data.
Photo: StartupStockPhotos / Pixabay
