How Large Language Models Work: The Transformer Architecture and Attention Mechanism

#1. The Next-Token Predictor: Demystifying AI Intelligence

Despite public perception that AI systems possess conscious reasoning or internal knowledge encyclopedias, a Large Language Model (LLM) is fundamentally an autoregressive token predictor. Given a sequence of text, its sole mathematical objective is to compute a probability distribution over a vocabulary of tokens to predict what comes next: $P(w_t \mid w_1, w_2, \dots, w_{t-1})$.

What makes modern LLMs (such as GPT-4, Claude, and Llama) so astonishingly capable is scale: when a neural network with hundreds of billions of connection weights is trained on trillions of tokens of human literature, code, and scientific papers, the network develops compressed internal representations of grammar, logic, programming syntax, and factual relationships.

#2. Tokenization and High-Dimensional Vector Embeddings

Computers cannot process raw English words or characters directly; they operate strictly on mathematical vectors.

First, incoming text passes through a Byte-Pair Encoding (BPE) tokenizer, breaking words into sub-word tokens (e.g. 'unbelievable' becomes ['un', 'believ', 'able']). A standard vocabulary contains roughly 50,000 to 100,000 unique token IDs.

Next, each integer token ID is mapped to a continuous dense vector called an 'embedding.' In a modern LLM, this vector space can have 4,096 to 12,288 dimensions. In this high-dimensional space, words with similar semantic meanings cluster together: the mathematical distance between 'king' and 'queen' closely mirrors the distance between 'man' and 'woman.'

#3. The Breakthrough: Why Transformers Replaced RNNs

Prior to 2017, natural language processing relied on Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks. RNNs processed text sequentially, word by word, carrying an internal hidden state forward.

This sequential bottleneck had fatal limitations: first, by the time an RNN read token 500 of a sentence, it had 'forgotten' the context from token 1 (the vanishing gradient problem). Second, sequential processing could not be parallelized across massive GPU clusters.

In 2017, a team of eight researchers at Google published the historic paper 'Attention Is All You Need,' introducing the Transformer. Transformers completely discarded recurrence, ingesting entire paragraphs simultaneously and computing relationships between all words in parallel.

#4. The Math of Self-Attention: Queries, Keys, and Values

At the core of every Transformer block lies the Scaled Dot-Product Attention mechanism. Imagine looking up a book in a library:

For every token embedding, the model creates three distinct vector projections via learned weight matrices:

- Query ($Q$): What this token is searching for.

- Key ($K$): What this token contains (its index tag).

- Value ($V$): The actual semantic information the token holds.

The attention score between token $i$ and token $j$ is calculated by taking the dot product of Query and Key, scaled by the square root of the dimension $d_k$, and passed through a Softmax function:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

This allows the model to dynamically assign attention weights: in the sentence 'The animal didn't cross the street because it was too tired,' the token 'it' computes a high attention score with 'animal,' whereas in 'because it was too wide,' 'it' pays attention to 'street.' Multi-Head Attention repeats this process across dozens of heads simultaneously, capturing syntactic, semantic, and factual relationships in parallel.

Because LLMs are trained to maximize probabilistic plausibility rather than ground truth. If a model lacks confident training data on a niche query, it generates text that statistically *looks* and *sounds* like a factual answer, inventing nonexistent citations or facts.

Training is the computationally immense phase where billions of weights are adjusted across thousands of GPUs over months using backpropagation and gradient descent. Inference is when a user prompts the completed static model, running forward-pass matrix multiplications to generate responses in seconds.

Temperature controls the randomness of token sampling: a temperature of 0.0 forces the model to select the single most probable token (deterministic and focused), while a higher temperature (e.g. 0.8) samples from a wider probability pool, introducing creative variety.