Have you ever tried printing out the code blueprint of a Large Language Model like Llama or Phi in PyTorch? If you have, you were probably greeted by a giant wall of scary numbers like this:
# LlamaForCausalLM
- model: LlamaModel
- embed_tokens: Embedding(128256, 2048)
- layers: ModuleList
- 0–15: 16 × LlamaDecoderLayer
- self_attn: LlamaAttention
- q_proj: Linear(in_features=2048, out_features=2048)
- k_proj: Linear(in_features=2048, out_features=512)
- v_proj: Linear(in_features=2048, out_features=512)
- o_proj: Linear(in_features=2048, out_features=2048)
- mlp: LlamaMLP
- gate_proj: Linear(in_features=2048, out_features=8192)
- up_proj: Linear(in_features=2048, out_features=8192)
- down_proj: Linear(in_features=8192, out_features=2048)
- act_fn: SiLUActivation()
- input_layernorm: LlamaRMSNorm((2048,))
- post_attention_layernorm: LlamaRMSNorm((2048,))
- norm: LlamaRMSNorm((2048,))
- rotary_emb: LlamaRotaryEmbedding()
- lm_head: Linear(in_features=2048, out_features=128256)
To most students, this looks like abstract, terrifying machine code. We treat LLMs like "black boxes"—words go in, magic happens, and a response pops out.
But what if I told you that an LLM isn't a mysterious box at all? What if you could visualize it like a physical factory assembly line that handles text like blocks of wood?
In this guide, we are going to throw away the complex math jargon and visualize exactly how an LLM processes language step-by-step using a real model blueprint: Llama 3.2 (1B).
The Rule of 3D Thinking: Height, Width, and Depth
To understand an LLM, you only need to look at it along three physical axes. If you can visualize these three dimensions, you understand the model:
Height (The Vector Dimension): This is the internal "detail level" or meaning score of a word.
Width (The Context Window): This is your prompt sequence length. If you type 4 words, your width is exactly 4 blocks wide.
Depth (The Layers): This is the conveyor belt. The text travels from Layer 1 down to Layer 16, getting refined at every single stop.
Let’s trace a simple 4-word prompt through the factory line: "Hello how are you".
Step 1: The Tokenizer & The Lookup Matrix
When you type "Hello how are you", the computer breaks those words into 4 unique IDs based on its master dictionary. This master dictionary is huge—it has 128,256 slots. This is the Vocabulary Size.
The very first layer in our code is:
(embed_tokens): Embedding(128256, 2048)
How to visualize it:
Imagine a massive spreadsheet standing upright. It is 2,048 rows high and stretches out to 128,256 columns long. Every column represents one word in the dictionary.
When your 4 words hit this spreadsheet, the model acts like a scanner. It looks up the columns for your specific words and pulls them out. Because you sent 4 words, you get 4 column vectors, each standing 2,048 elements high.
Your Active Matrix Shape: Height = 2048, Width = 4 tokens.
Step 2: The Attention Layer (The Context Focus)
Your 4 blocks, each 2,048 tall, move down the conveyor belt into the processing layers. The model has 16 identical floors (0 to 15). On every floor, the first thing the blocks hit is the Self-Attention mechanism:
(q_proj): Linear(in_features=2048, out_features=2048)
(k_proj): Linear(in_features=2048, out_features=512)
How to visualize it:
This is the first major change in our vector's height.
The Query matrix (q_proj) keeps the vectors at a height of 2,048.
The Key/Value matrices (k_proj, v_proj) violently squish the vectors down to a height of 512!
Why? Because this layer is trying to make the words "talk" to each other to figure out the context (e.g., ensuring the word "bank" means a riverbank and not a money bank). Dropping the height down to 512 drastically speeds up this word-to-word math.
Once the words have exchanged context, an output projection restores their height perfectly back to 2,048.
Step 3: The MLP Layer (The Knowledge Balloon)
Right after attention, the vectors enter the MLP (Multi-Layer Perceptron) on the same floor:
(gate_proj): Linear(in_features=2048, out_features=8192)
(down_proj): Linear(in_features=8192, out_features=2048)
How to visualize it:
This is the Knowledge Balloon of the model. This is where all the facts, logic, and coding rules the model learned during training are stored.
Your 4 vectors, currently 2,048 tall, hit the gate_proj and up_proj walls.
They explosively balloon upward from a height of 2,048 to a towering 8,192 elements high!
This massive expansion gives the model the mathematical "breathing room" to activate specific concepts (e.g., specific neurons fire for grammar, others for facts about history).
Once the calculations are done, the down_proj matrix compresses the height back down to the standard 2,048 so it can cleanly travel down to the next floor.
Step 4: The Parallel Prediction Loop
This entire cycle—compressing to 512 for context, ballooning to 8,192 for factual processing, and returning to 2,048—happens 16 times as the data falls through the layers.
Throughout this entire journey, the Width stays completely constant at 4. The 4 words travel in parallel side-by-side tracks.
When they exit the 16th layer, they hit the final translator:
(lm_head): Linear(in_features=2048, out_features=128256)
How to visualize it:
The lm_head takes each of the 4 vectors (height 2,048) and projects them out into a massive wall that is 128,256 rows high—matching our vocabulary size perfectly.
The model calculates a next-word probability for every single column position at the same time:
The column under "Hello" outputs 128,256 scores predicting what should follow "Hello".
The column under "how" predicts what should follow "Hello how".
The column under "are" predicts what should follow "Hello how are".
The column under "you" outputs the prediction for what should follow "Hello how are you".
When you are chatting with an LLM in real life, the software simply ignores the first 3 columns and grabs the 4th output (the one under "you"). It runs a mathematical filter (Softmax) to turn those 128,256 raw scores into percentages, and whichever word has the highest percentage (like a "? " or "doing") becomes the next token printed on your screen!
Summary: The Secret of the Accordion
If you ever forget how an LLM works, just remember the Accordion Metaphor.
An LLM is a machine that keeps the prompt length (Width) completely stable, while stretching and squishing the word vectors (Height) through a series of mathematical gates:
Start: Map words into vectors (Height: 2,048)
Attention: Shrink to find context (Height: 512)
MLP: Explode to find facts (Height: 8,192)
Exit: Project to the entire dictionary (Height: 128,256) to guess the next word.