{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e912df2f",
   "metadata": {},
   "source": [
    "\n",
    "# Transformer Encoder — From Scratch with PyTorch\n",
    "\n",
    "This notebook builds a **complete Transformer Encoder from scratch using PyTorch**, with the goal of understanding what actually happens inside the architecture instead of treating it as a black box.\n",
    "\n",
    "We will use the simple sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "and follow it through every major encoder stage:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Tokens}\n",
    "\\rightarrow\n",
    "\\text{Embeddings}\n",
    "\\rightarrow\n",
    "\\text{Positional Encoding}\n",
    "\\rightarrow\n",
    "\\text{Q, K, V}\n",
    "\\rightarrow\n",
    "\\text{Multi-Head Self-Attention}\n",
    "\\rightarrow\n",
    "\\text{Residual + LayerNorm}\n",
    "\\rightarrow\n",
    "\\text{FFN}\n",
    "\\rightarrow\n",
    "\\text{Final Encoder Output}\n",
    "}\n",
    "$$\n",
    "\n",
    "The notebook focuses on the **forward pass** and makes the intermediate tensors, dimensions, and calculations visible using **Rich printing**.\n",
    "\n",
    "---\n",
    "\n",
    "## Configuration used in this notebook\n",
    "\n",
    "For our example:\n",
    "\n",
    "$$\n",
    "n = 5\n",
    "$$\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "$$\n",
    "h = 4\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "d_{\\text{head}}\n",
    "=\n",
    "\\frac{512}{4}\n",
    "=\n",
    "128\n",
    "$$\n",
    "\n",
    "and the Feed-Forward Network uses:\n",
    "\n",
    "$$\n",
    "d_{\\text{ff}} = 2048\n",
    "$$\n",
    "\n",
    "So the main configuration is:\n",
    "\n",
    "- Sequence length: $n = 5$\n",
    "- Model dimension: $d_{\\text{model}} = 512$\n",
    "- Number of attention heads: $h = 4$\n",
    "- Dimension per head: $d_{\\text{head}} = 128$\n",
    "- Feed-Forward hidden dimension: $d_{\\text{ff}} = 2048$\n",
    "\n",
    "---\n",
    "\n",
    "## Why build the encoder from scratch?\n",
    "\n",
    "PyTorch already provides high-level components such as:\n",
    "\n",
    "```python\n",
    "nn.MultiheadAttention\n",
    "nn.LayerNorm\n",
    "nn.TransformerEncoderLayer\n",
    "nn.TransformerEncoder\n",
    "```\n",
    "\n",
    "But if we use those directly, many important Transformer operations become hidden.\n",
    "\n",
    "In this notebook, we deliberately implement the core logic ourselves so we can clearly see:\n",
    "\n",
    "- how token embeddings are created\n",
    "- how positional information is added\n",
    "- how Query, Key, and Value are produced\n",
    "- how attention scores are calculated\n",
    "- why we divide by $\\sqrt{d_k}$\n",
    "- how Softmax produces attention weights\n",
    "- how multiple heads are split and concatenated\n",
    "- how residual connections work\n",
    "- how LayerNorm works\n",
    "- how the Feed-Forward Network transforms features\n",
    "- how the final contextual token representations are produced\n",
    "\n",
    "So the purpose is not only to **run an encoder**, but to understand:\n",
    "\n",
    "> **what each tensor means, why each operation exists, and how the dimensions move through the architecture.**\n",
    "\n",
    "---\n",
    "\n",
    "# What does a Transformer Encoder do?\n",
    "\n",
    "A Transformer Encoder converts ordinary token representations into **context-aware token representations**.\n",
    "\n",
    "At the beginning, a token mainly represents:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{What token am I?}\n",
    "}\n",
    "$$\n",
    "\n",
    "After adding positional information, it also knows:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Where am I in the sequence?}\n",
    "}\n",
    "$$\n",
    "\n",
    "After Self-Attention, it can incorporate information from other tokens:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Which other tokens are relevant to me?}\n",
    "}\n",
    "$$\n",
    "\n",
    "After the Feed-Forward Network, its contextual features are transformed further.\n",
    "\n",
    "So the final representation answers something closer to:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Who am I in the context of this entire sentence?}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## Example intuition\n",
    "\n",
    "Consider the sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "The token:\n",
    "\n",
    "> **\"chased\"**\n",
    "\n",
    "does not need to be understood in isolation.\n",
    "\n",
    "Through Self-Attention, its representation can be influenced by:\n",
    "\n",
    "> **\"cat\"**\n",
    "\n",
    "and:\n",
    "\n",
    "> **\"mouse\"**\n",
    "\n",
    "because those tokens are relevant to the meaning of the action.\n",
    "\n",
    "Likewise, the word:\n",
    "\n",
    "> **\"the\"**\n",
    "\n",
    "appears twice.\n",
    "\n",
    "Before positional information is added, both occurrences may begin with the same token embedding.\n",
    "\n",
    "But after positional encoding and contextual processing, their representations can become different because they appear at different positions and interact with different surrounding tokens.\n",
    "\n",
    "This is the central idea of the encoder:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Same token}\n",
    "+\n",
    "\\text{different position/context}\n",
    "\\rightarrow\n",
    "\\text{different contextual representation}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Encoder flow\n",
    "\n",
    "The full encoder pipeline we will build is:\n",
    "\n",
    "```text\n",
    "Raw Sentence\n",
    "     ↓\n",
    "Tokenization\n",
    "     ↓\n",
    "Token IDs\n",
    "     ↓\n",
    "Token Embeddings\n",
    "     ↓\n",
    "+ Positional Encoding\n",
    "     ↓\n",
    "Encoder Input X\n",
    "     ↓\n",
    "Create Q, K, V\n",
    "     ↓\n",
    "Split into Multiple Heads\n",
    "     ↓\n",
    "Scaled Dot-Product Attention\n",
    "     ↓\n",
    "Softmax Attention Weights\n",
    "     ↓\n",
    "Attention × Value\n",
    "     ↓\n",
    "Concatenate Heads\n",
    "     ↓\n",
    "Output Projection\n",
    "     ↓\n",
    "Residual Connection\n",
    "     ↓\n",
    "LayerNorm\n",
    "     ↓\n",
    "Feed-Forward Network\n",
    "     ↓\n",
    "Second Residual Connection\n",
    "     ↓\n",
    "Second LayerNorm\n",
    "     ↓\n",
    "Final Encoder Output\n",
    "```\n",
    "\n",
    "The final output has shape:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "The shape is similar to the main input representation, but the information stored inside each vector is much richer.\n",
    "\n",
    "---\n",
    "\n",
    "# Important encoder property\n",
    "\n",
    "Unlike an autoregressive decoder, a standard encoder normally does **not** use a causal mask.\n",
    "\n",
    "That means each token can attend to tokens on both the left and the right.\n",
    "\n",
    "For example:\n",
    "\n",
    "> **\"chased\"**\n",
    "\n",
    "can attend to both:\n",
    "\n",
    "> **\"cat\"**\n",
    "\n",
    "and:\n",
    "\n",
    "> **\"mouse\"**\n",
    "\n",
    "at the same time.\n",
    "\n",
    "So encoder self-attention is generally:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{bidirectional attention over the available non-masked tokens}\n",
    "}\n",
    "$$\n",
    "\n",
    "Padding masks may still be used when sequences in a batch have different lengths.\n",
    "\n",
    "---\n",
    "\n",
    "# Common encoder use cases\n",
    "\n",
    "Transformer encoders are useful whenever we want strong contextual representations of text.\n",
    "\n",
    "Typical applications include:\n",
    "\n",
    "- **Text classification**  \n",
    "  sentiment analysis, spam detection, intent detection, topic classification\n",
    "\n",
    "- **Token classification**  \n",
    "  Named Entity Recognition, POS tagging, sequence labeling\n",
    "\n",
    "- **Semantic search and retrieval**  \n",
    "  sentence embeddings, document embeddings, similarity search, RAG retrieval\n",
    "\n",
    "- **Extractive question answering**\n",
    "\n",
    "- **Encoder-decoder architectures**  \n",
    "  where the encoder produces a contextual memory that the decoder reads through cross-attention\n",
    "\n",
    "---\n",
    "\n",
    "# What we will learn\n",
    "\n",
    "By the end of this notebook, we will understand not just:\n",
    "\n",
    "> **what a Transformer Encoder contains**\n",
    "\n",
    "but also:\n",
    "\n",
    "> **why each component exists, how its mathematics works, and how its tensor dimensions fit together.**\n",
    "\n",
    "The goal is to move from:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{\"I know the encoder diagram\"}\n",
    "}\n",
    "$$\n",
    "\n",
    "to:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{\"I understand what is happening inside every major encoder operation\"}\n",
    "}\n",
    "$$\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "449c885b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:21.902631Z",
     "iopub.status.busy": "2026-10-03T18:57:21.902526Z",
     "iopub.status.idle": "2026-10-03T18:57:22.896969Z",
     "shell.execute_reply": "2026-10-03T18:57:22.896301Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">╭─────── Setup ───────╮\n",
       "│ <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Environment ready</span>   │\n",
       "│ PyTorch: 2.14.0+cpu │\n",
       "│ Device: cpu         │\n",
       "╰─────────────────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "╭─────── Setup ───────╮\n",
       "│ \u001b[1;32mEnvironment ready\u001b[0m   │\n",
       "│ PyTorch: 2.14.0+cpu │\n",
       "│ Device: cpu         │\n",
       "╰─────────────────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "# If needed in a fresh environment, install dependencies with:\n",
    "# !pip install torch rich\n",
    "\n",
    "import math\n",
    "import random\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from rich.console import Console\n",
    "from rich.panel import Panel\n",
    "from rich.table import Table\n",
    "from rich import box\n",
    "\n",
    "console = Console(width=140)\n",
    "\n",
    "torch.manual_seed(7)\n",
    "random.seed(7)\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "console.print(Panel.fit(\n",
    "    f\"[bold green]Environment ready[/bold green]\\n\"\n",
    "    f\"PyTorch: {torch.__version__}\\n\"\n",
    "    f\"Device: {device}\",\n",
    "    title=\"Setup\"\n",
    "))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5bbf32b",
   "metadata": {},
   "source": [
    "\n",
    "## 1. Configuration and the simple sentence\n",
    "\n",
    "We lowercase the sentence so both occurrences of **“the”** map to the same token ID. This makes it easy to see how position + attention later make their representations different.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cac152ca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:22.898155Z",
     "iopub.status.busy": "2026-10-03T18:57:22.898043Z",
     "iopub.status.idle": "2026-10-03T18:57:22.912238Z",
     "shell.execute_reply": "2026-10-03T18:57:22.911714Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                 Encoder Configuration                  </span>\n",
       "╭───────────┬──────────────────────────────────────────╮\n",
       "│<span style=\"font-weight: bold\"> Item      </span>│<span style=\"font-weight: bold\"> Value                                    </span>│\n",
       "├───────────┼──────────────────────────────────────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Sentence  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> The cat chased the mouse                 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Tokens    </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ['the', 'cat', 'chased', 'the', 'mouse'] </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Token IDs </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> [2, 3, 4, 2, 5]                          </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> n         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 5                                        </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> d_model   </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 512                                      </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> heads     </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 4                                        </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> d_k = d_v </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 128                                      </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> d_ff      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 2048                                     </span>│\n",
       "╰───────────┴──────────────────────────────────────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                 Encoder Configuration                  \u001b[0m\n",
       "╭───────────┬──────────────────────────────────────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mItem     \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mValue                                   \u001b[0m\u001b[1m \u001b[0m│\n",
       "├───────────┼──────────────────────────────────────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mSentence \u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37mThe cat chased the mouse                \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mTokens   \u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m['the', 'cat', 'chased', 'the', 'mouse']\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mToken IDs\u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m[2, 3, 4, 2, 5]                         \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mn        \u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m5                                       \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36md_model  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m512                                     \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mheads    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m4                                       \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36md_k = d_v\u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m128                                     \u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36md_ff     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m2048                                    \u001b[0m\u001b[37m \u001b[0m│\n",
       "╰───────────┴──────────────────────────────────────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "<span style=\"font-weight: bold\">Vocabulary</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\u001b[1mVocabulary\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">   <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;pad&gt;'</span><span style=\"color: #000000; text-decoration-color: #000000\"> -</span><span style=\"font-weight: bold\">&gt;</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "   \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mpad\u001b[0m\u001b[32m>'\u001b[0m\u001b[39m -\u001b[0m\u001b[1m>\u001b[0m \u001b[1;36m0\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">   <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;unk&gt;'</span><span style=\"color: #000000; text-decoration-color: #000000\"> -</span><span style=\"font-weight: bold\">&gt;</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "   \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32munk\u001b[0m\u001b[32m>'\u001b[0m\u001b[39m -\u001b[0m\u001b[1m>\u001b[0m \u001b[1;36m1\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">     <span style=\"color: #008000; text-decoration-color: #008000\">'the'</span> -&gt; <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "     \u001b[32m'the'\u001b[0m -> \u001b[1;36m2\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">     <span style=\"color: #008000; text-decoration-color: #008000\">'cat'</span> -&gt; <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "     \u001b[32m'cat'\u001b[0m -> \u001b[1;36m3\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">  <span style=\"color: #008000; text-decoration-color: #008000\">'chased'</span> -&gt; <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "  \u001b[32m'chased'\u001b[0m -> \u001b[1;36m4\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">   <span style=\"color: #008000; text-decoration-color: #008000\">'mouse'</span> -&gt; <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "   \u001b[32m'mouse'\u001b[0m -> \u001b[1;36m5\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "SENTENCE = \"The cat chased the mouse\"\n",
    "D_MODEL = 512\n",
    "NUM_HEADS = 4\n",
    "D_HEAD = D_MODEL // NUM_HEADS\n",
    "D_FF = 2048\n",
    "NUM_LAYERS = 1\n",
    "\n",
    "assert D_MODEL % NUM_HEADS == 0\n",
    "\n",
    "def simple_tokenize(sentence: str):\n",
    "    \"\"\"Very small educational tokenizer: lowercase + whitespace split.\"\"\"\n",
    "    return sentence.lower().strip().split()\n",
    "\n",
    "def build_vocab(tokens):\n",
    "    vocab = {\"<pad>\": 0, \"<unk>\": 1}\n",
    "    for tok in tokens:\n",
    "        if tok not in vocab:\n",
    "            vocab[tok] = len(vocab)\n",
    "    return vocab\n",
    "\n",
    "def tokens_to_ids(tokens, vocab):\n",
    "    return torch.tensor([vocab.get(tok, vocab[\"<unk>\"]) for tok in tokens], dtype=torch.long, device=device)\n",
    "\n",
    "tokens = simple_tokenize(SENTENCE)\n",
    "vocab = build_vocab(tokens)\n",
    "token_ids = tokens_to_ids(tokens, vocab)\n",
    "N = len(tokens)\n",
    "\n",
    "config_table = Table(title=\"Encoder Configuration\", box=box.ROUNDED)\n",
    "config_table.add_column(\"Item\", style=\"bold cyan\")\n",
    "config_table.add_column(\"Value\", style=\"white\")\n",
    "for k, v in [\n",
    "    (\"Sentence\", SENTENCE),\n",
    "    (\"Tokens\", str(tokens)),\n",
    "    (\"Token IDs\", str(token_ids.tolist())),\n",
    "    (\"n\", str(N)),\n",
    "    (\"d_model\", str(D_MODEL)),\n",
    "    (\"heads\", str(NUM_HEADS)),\n",
    "    (\"d_k = d_v\", str(D_HEAD)),\n",
    "    (\"d_ff\", str(D_FF)),\n",
    "]:\n",
    "    config_table.add_row(k, v)\n",
    "console.print(config_table)\n",
    "\n",
    "console.print(\"\\n[bold]Vocabulary[/bold]\")\n",
    "for token, idx in vocab.items():\n",
    "    console.print(f\"  {token!r:>8} -> {idx}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29b9644b",
   "metadata": {},
   "source": [
    "\n",
    "## Rich tensor-printing helpers\n",
    "\n",
    "A `5 × 512` matrix contains 2,560 numbers, so dumping every number makes the notebook unreadable. The helper below prints **every important step**, its shape, and a useful slice of values.\n",
    "\n",
    "Set `SHOW_FULL_512 = True` if you really want all 512 columns printed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1c9a4da4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:22.913294Z",
     "iopub.status.busy": "2026-10-03T18:57:22.913147Z",
     "iopub.status.idle": "2026-10-03T18:57:22.919118Z",
     "shell.execute_reply": "2026-10-03T18:57:22.918550Z"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "SHOW_FULL_512 = False\n",
    "PREVIEW_COLS = 8\n",
    "\n",
    "def rich_tensor(name, tensor, row_labels=None, max_rows=8, max_cols=PREVIEW_COLS, precision=4):\n",
    "    t = tensor.detach().cpu()\n",
    "    if t.ndim == 1:\n",
    "        t = t.unsqueeze(0)\n",
    "    if t.ndim != 2:\n",
    "        console.print(f\"[bold yellow]{name}[/bold yellow] shape={tuple(tensor.shape)}\")\n",
    "        console.print(t)\n",
    "        return\n",
    "\n",
    "    rows, cols = t.shape\n",
    "    show_rows = rows if SHOW_FULL_512 else min(rows, max_rows)\n",
    "    show_cols = cols if SHOW_FULL_512 else min(cols, max_cols)\n",
    "\n",
    "    table = Table(title=f\"{name}   shape={tuple(tensor.shape)}\", box=box.SIMPLE_HEAVY)\n",
    "    table.add_column(\"row\", style=\"bold cyan\")\n",
    "    for j in range(show_cols):\n",
    "        table.add_column(str(j), justify=\"right\")\n",
    "    if show_cols < cols:\n",
    "        table.add_column(\"…\", justify=\"center\")\n",
    "\n",
    "    for i in range(show_rows):\n",
    "        label = row_labels[i] if row_labels and i < len(row_labels) else str(i)\n",
    "        vals = [f\"{float(t[i, j]):.{precision}f}\" for j in range(show_cols)]\n",
    "        if show_cols < cols:\n",
    "            vals.append(\"…\")\n",
    "        table.add_row(str(label), *vals)\n",
    "\n",
    "    if show_rows < rows:\n",
    "        table.add_row(\"…\", *([\"…\"] * (show_cols + (1 if show_cols < cols else 0))))\n",
    "    console.print(table)\n",
    "\n",
    "\n",
    "def rich_attention_matrix(name, matrix, tokens):\n",
    "    \"\"\"Print a complete n×n attention-score/weight matrix for our short sentence.\"\"\"\n",
    "    m = matrix.detach().cpu()\n",
    "    table = Table(title=f\"{name}   shape={tuple(matrix.shape)}\", box=box.ROUNDED)\n",
    "    table.add_column(\"Q \\\\ K\", style=\"bold cyan\")\n",
    "    for tok in tokens:\n",
    "        table.add_column(tok, justify=\"right\")\n",
    "    for i, tok in enumerate(tokens):\n",
    "        table.add_row(tok, *[f\"{float(m[i, j]):.4f}\" for j in range(len(tokens))])\n",
    "    console.print(table)\n",
    "\n",
    "\n",
    "def print_shape_equation(label, equation):\n",
    "    console.print(f\"[bold magenta]{label}[/bold magenta]  {equation}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65541844",
   "metadata": {},
   "source": [
    "## 2. Input token embeddings\n",
    "\n",
    "An embedding table stores one **trainable vector** for each vocabulary item.\n",
    "\n",
    "For a token ID $i$:\n",
    "\n",
    "$$\n",
    "\\operatorname{Embedding}(i) = E[i],\n",
    "\\qquad\n",
    "E \\in \\mathbb{R}^{|V| \\times d_{\\text{model}}}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $|V|$ = vocabulary size\n",
    "- $d_{\\text{model}} = 512$\n",
    "- each token is represented by a vector of **512 numbers**\n",
    "\n",
    "---\n",
    "\n",
    "For our example:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "we have:\n",
    "\n",
    "$$\n",
    "n = 5\n",
    "$$\n",
    "\n",
    "tokens.\n",
    "\n",
    "Therefore, the embedding matrix has shape:\n",
    "\n",
    "$$\n",
    "X_{\\text{embedding}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "(n,d_{\\text{model}})\n",
    "=\n",
    "(5,512)\n",
    "}\n",
    "$$\n",
    "\n",
    "This means:\n",
    "\n",
    "- $5$ rows = one row for each token\n",
    "- $512$ columns = 512 embedding features for each token\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "X_{\\text{embedding}}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "E_{\\text{the}} \\\\\n",
    "E_{\\text{cat}} \\\\\n",
    "E_{\\text{chased}} \\\\\n",
    "E_{\\text{the}} \\\\\n",
    "E_{\\text{mouse}}\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "where each row has:\n",
    "\n",
    "$$\n",
    "512\n",
    "$$\n",
    "\n",
    "learned values.\n",
    "\n",
    "Because our simple tokenizer lowercases the sentence, repeated occurrences of the same token initially receive the **same token embedding**.\n",
    "\n",
    "For example:\n",
    "\n",
    "$$\n",
    "E_{\\text{the at position 0}}\n",
    "=\n",
    "E_{\\text{the at position 3}}\n",
    "$$\n",
    "\n",
    "At this stage, the model knows:\n",
    "\n",
    "> **what the token is**\n",
    "\n",
    "but it does not yet know:\n",
    "\n",
    "> **where the token appears in the sentence**\n",
    "\n",
    "Later, **positional encoding** adds this position information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8c031ee3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:22.920209Z",
     "iopub.status.busy": "2026-10-03T18:57:22.920092Z",
     "iopub.status.idle": "2026-10-03T18:57:22.936121Z",
     "shell.execute_reply": "2026-10-03T18:57:22.935550Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭───────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">STEP 1 — TOKEN EMBEDDINGS</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰───────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭───────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1mSTEP 1 — TOKEN EMBEDDINGS\u001b[0m \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰───────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Shape</span>  token_ids <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>,<span style=\"font-weight: bold\">)</span> -&gt; embeddings <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> = \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mShape\u001b[0m  token_ids \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m,\u001b[1m)\u001b[0m -> embeddings \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m = \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                             Token embeddings E   shape=(5, 512)                              </span>\n",
       "                                                                                              \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.0170    0.0022    0.0268    0.0356    0.0132   -0.0050    0.0252    0.0307   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.0075    0.0155    0.0094   -0.0118   -0.0361    0.0209    0.0220    0.0101   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.0248    0.0157   -0.0056    0.0010    0.0107    0.0081   -0.0321   -0.0069   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.0170    0.0022    0.0268    0.0356    0.0132   -0.0050    0.0252    0.0307   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.0031   -0.0182   -0.0324   -0.0578   -0.0040    0.0133   -0.0276   -0.0051   …  \n",
       "                                                                                              \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                             Token embeddings E\u001b[0m\u001b[3m   shape=(5, 512)\u001b[0m\u001b[3m                              \u001b[0m\n",
       "                                                                                              \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.0170    0.0022    0.0268    0.0356    0.0132   -0.0050    0.0252    0.0307   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.0075    0.0155    0.0094   -0.0118   -0.0361    0.0209    0.0220    0.0101   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.0248    0.0157   -0.0056    0.0010    0.0107    0.0081   -0.0321   -0.0069   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.0170    0.0022    0.0268    0.0356    0.0132   -0.0050    0.0252    0.0307   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.0031   -0.0182   -0.0324   -0.0578   -0.0040    0.0133   -0.0276   -0.0051   …  \n",
       "                                                                                              \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "Both <span style=\"color: #008000; text-decoration-color: #008000\">'the'</span> token embeddings identical before position information? <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold; font-style: italic\">True</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "Both \u001b[32m'the'\u001b[0m token embeddings identical before position information? \u001b[1;3;92mTrue\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "class ManualEmbedding(nn.Module):\n",
    "    def __init__(self, vocab_size, d_model):\n",
    "        super().__init__()\n",
    "        self.weight = nn.Parameter(torch.randn(vocab_size, d_model) * 0.02)\n",
    "\n",
    "    def forward(self, token_ids):\n",
    "        # Direct row lookup from the trainable embedding matrix.\n",
    "        return self.weight[token_ids]\n",
    "\n",
    "embedding_layer = ManualEmbedding(len(vocab), D_MODEL).to(device)\n",
    "token_embeddings = embedding_layer(token_ids)\n",
    "\n",
    "console.print(Panel.fit(\"[bold]STEP 1 — TOKEN EMBEDDINGS[/bold]\", border_style=\"green\"))\n",
    "print_shape_equation(\"Shape\", f\"token_ids {tuple(token_ids.shape)} -> embeddings {tuple(token_embeddings.shape)} = [n, d_model]\")\n",
    "rich_tensor(\"Token embeddings E[token_id]\", token_embeddings, row_labels=tokens)\n",
    "\n",
    "same = torch.allclose(token_embeddings[0], token_embeddings[3])\n",
    "console.print(f\"\\nBoth 'the' token embeddings identical before position information? [bold]{same}[/bold]\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1bea435",
   "metadata": {},
   "source": [
    "## 3. Positional encoding\n",
    "\n",
    "Self-attention by itself does **not know the order of tokens**.\n",
    "\n",
    "For example, without positional information, the model does not inherently know whether a token appeared:\n",
    "\n",
    "- first\n",
    "- second\n",
    "- third\n",
    "- fifth\n",
    "\n",
    "So we add a **positional encoding vector** to every token embedding.\n",
    "\n",
    "For even dimensions:\n",
    "\n",
    "$$\n",
    "PE(pos, 2i)\n",
    "=\n",
    "\\sin\\left(\n",
    "\\frac{pos}\n",
    "{10000^{\\frac{2i}{d_{\\text{model}}}}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "For odd dimensions:\n",
    "\n",
    "$$\n",
    "PE(pos, 2i+1)\n",
    "=\n",
    "\\cos\\left(\n",
    "\\frac{pos}\n",
    "{10000^{\\frac{2i}{d_{\\text{model}}}}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $pos$ = position of the token in the sentence\n",
    "- $i$ = embedding dimension index\n",
    "- $d_{\\text{model}} = 512$\n",
    "\n",
    "For our sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "the token positions are:\n",
    "\n",
    "| Token | Position |\n",
    "|---|---:|\n",
    "| The | 0 |\n",
    "| cat | 1 |\n",
    "| chased | 2 |\n",
    "| the | 3 |\n",
    "| mouse | 4 |\n",
    "\n",
    "Since:\n",
    "\n",
    "$$\n",
    "n = 5\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "the positional encoding matrix has shape:\n",
    "\n",
    "$$\n",
    "PE \\in \\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Each row represents the positional information for one token.\n",
    "\n",
    "So conceptually:\n",
    "\n",
    "$$\n",
    "PE =\n",
    "\\begin{bmatrix}\n",
    "PE_0 \\\\\n",
    "PE_1 \\\\\n",
    "PE_2 \\\\\n",
    "PE_3 \\\\\n",
    "PE_4\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "where every:\n",
    "\n",
    "$$\n",
    "PE_i \\in \\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "The positional encoding has the **same shape as the token embeddings**, because in the next step we will add them element-wise:\n",
    "\n",
    "$$\n",
    "X_{\\text{input}}\n",
    "=\n",
    "X_{\\text{embedding}}\n",
    "+\n",
    "PE\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6bd1a8f5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:22.937097Z",
     "iopub.status.busy": "2026-10-03T18:57:22.936971Z",
     "iopub.status.idle": "2026-10-03T18:57:22.964026Z",
     "shell.execute_reply": "2026-10-03T18:57:22.963467Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭──────────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">STEP 2 — POSITIONAL ENCODING</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰──────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭──────────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1mSTEP 2 — POSITIONAL ENCODING\u001b[0m \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰──────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                           Positional encoding PE   shape=(5, 512)                           </span>\n",
       "                                                                                             \n",
       " <span style=\"font-weight: bold\"> row   </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> pos 0 </span>   0.0000    1.0000    0.0000    1.0000    0.0000    1.0000    0.0000    1.0000   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> pos 1 </span>   0.8415    0.5403    0.8219    0.5697    0.8020    0.5974    0.7819    0.6234   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> pos 2 </span>   0.9093   -0.4161    0.9364   -0.3509    0.9581   -0.2863    0.9749   -0.2227   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> pos 3 </span>   0.1411   -0.9900    0.2451   -0.9695    0.3428   -0.9394    0.4336   -0.9011   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> pos 4 </span>  -0.7568   -0.6536   -0.6572   -0.7537   -0.5486   -0.8361   -0.4342   -0.9008   …  \n",
       "                                                                                             \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                           Positional encoding PE   shape=(5, 512)                           \u001b[0m\n",
       "                                                                                             \n",
       " \u001b[1m \u001b[0m\u001b[1mrow  \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mpos 0\u001b[0m\u001b[1;36m \u001b[0m   0.0000    1.0000    0.0000    1.0000    0.0000    1.0000    0.0000    1.0000   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mpos 1\u001b[0m\u001b[1;36m \u001b[0m   0.8415    0.5403    0.8219    0.5697    0.8020    0.5974    0.7819    0.6234   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mpos 2\u001b[0m\u001b[1;36m \u001b[0m   0.9093   -0.4161    0.9364   -0.3509    0.9581   -0.2863    0.9749   -0.2227   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mpos 3\u001b[0m\u001b[1;36m \u001b[0m   0.1411   -0.9900    0.2451   -0.9695    0.3428   -0.9394    0.4336   -0.9011   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mpos 4\u001b[0m\u001b[1;36m \u001b[0m  -0.7568   -0.6536   -0.6572   -0.7537   -0.5486   -0.8361   -0.4342   -0.9008   …  \n",
       "                                                                                             \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "def sinusoidal_positional_encoding(seq_len, d_model, device):\n",
    "    pe = torch.zeros(seq_len, d_model, device=device)\n",
    "    position = torch.arange(seq_len, dtype=torch.float32, device=device).unsqueeze(1)\n",
    "    div_term = torch.exp(\n",
    "        torch.arange(0, d_model, 2, dtype=torch.float32, device=device)\n",
    "        * (-math.log(10000.0) / d_model)\n",
    "    )\n",
    "    pe[:, 0::2] = torch.sin(position * div_term)\n",
    "    pe[:, 1::2] = torch.cos(position * div_term)\n",
    "    return pe\n",
    "\n",
    "positional_encoding = sinusoidal_positional_encoding(N, D_MODEL, device)\n",
    "\n",
    "console.print(Panel.fit(\"[bold]STEP 2 — POSITIONAL ENCODING[/bold]\", border_style=\"green\"))\n",
    "rich_tensor(\"Positional encoding PE\", positional_encoding, row_labels=[f\"pos {i}\" for i in range(N)])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8989b8c9",
   "metadata": {},
   "source": [
    "## 4. Add token embeddings + positional encoding\n",
    "\n",
    "Before the input enters the encoder's self-attention block, we combine two pieces of information:\n",
    "\n",
    "1. **What the token is** → Token Embedding\n",
    "2. **Where the token appears** → Positional Encoding\n",
    "\n",
    "The final encoder input is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X\n",
    "=\n",
    "\\text{TokenEmbedding}\n",
    "+\n",
    "\\text{PositionalEncoding}\n",
    "}\n",
    "$$\n",
    "\n",
    "For our sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "we have:\n",
    "\n",
    "$$\n",
    "n = 5\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "Therefore, both matrices have the same shape:\n",
    "\n",
    "$$\n",
    "X_{\\text{embedding}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "PE\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Because their shapes are identical, they can be added element-wise:\n",
    "\n",
    "$$\n",
    "X\n",
    "=\n",
    "X_{\\text{embedding}}\n",
    "+\n",
    "PE\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### What does the token embedding contain?\n",
    "\n",
    "A token embedding represents the **meaning/features of a token**.\n",
    "\n",
    "For example:\n",
    "\n",
    "> **cat**\n",
    "\n",
    "is converted from a token ID into a vector containing 512 learned values:\n",
    "\n",
    "$$\n",
    "E_{\\text{cat}}\n",
    "=\n",
    "[e_1,e_2,e_3,\\dots,e_{512}]\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "E_{\\text{cat}}\n",
    "\\in\n",
    "\\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "Similarly:\n",
    "\n",
    "$$\n",
    "E_{\\text{mouse}}\n",
    "\\in\n",
    "\\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "The embedding allows the model to learn useful semantic relationships between tokens.\n",
    "\n",
    "For example, during training it may learn that words such as:\n",
    "\n",
    "- **cat**\n",
    "- **dog**\n",
    "- **mouse**\n",
    "- **animal**\n",
    "\n",
    "have certain related features.\n",
    "\n",
    "But the token embedding itself does **not tell the model where the token appeared in the sentence**.\n",
    "\n",
    "---\n",
    "\n",
    "### What does positional encoding contain?\n",
    "\n",
    "Positional encoding adds information about the **position of each token**.\n",
    "\n",
    "For our sentence:\n",
    "\n",
    "| Position | Token |\n",
    "|---:|---|\n",
    "| 0 | The |\n",
    "| 1 | cat |\n",
    "| 2 | chased |\n",
    "| 3 | the |\n",
    "| 4 | mouse |\n",
    "\n",
    "Each position receives its own positional vector:\n",
    "\n",
    "$$\n",
    "PE_0,PE_1,PE_2,PE_3,PE_4\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "PE_i \\in \\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "So positional encoding helps the model distinguish:\n",
    "\n",
    "> **which token came first, second, third, and so on.**\n",
    "\n",
    "---\n",
    "\n",
    "### Why do we need both?\n",
    "\n",
    "Token embeddings tell the encoder:\n",
    "\n",
    "> **\"What word/token is this?\"**\n",
    "\n",
    "Positional encoding tells the encoder:\n",
    "\n",
    "> **\"Where is this token located?\"**\n",
    "\n",
    "So after adding them:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Token identity}\n",
    "+\n",
    "\\text{Position information}\n",
    "=\n",
    "\\text{Encoder input}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Example with the two \"the\" tokens\n",
    "\n",
    "Our sentence contains:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "After lowercasing:\n",
    "\n",
    "> **\"the cat chased the mouse\"**\n",
    "\n",
    "The token **\"the\"** appears twice.\n",
    "\n",
    "Because both occurrences have the same token ID, initially they receive the same token embedding:\n",
    "\n",
    "$$\n",
    "E_{\\text{the}}\n",
    "=\n",
    "E_{\\text{the}}\n",
    "$$\n",
    "\n",
    "But they occur at different positions.\n",
    "\n",
    "The first **\"the\"** is at position 0:\n",
    "\n",
    "$$\n",
    "X_{\\text{the},0}\n",
    "=\n",
    "E_{\\text{the}}\n",
    "+\n",
    "PE_0\n",
    "$$\n",
    "\n",
    "The second **\"the\"** is at position 3:\n",
    "\n",
    "$$\n",
    "X_{\\text{the},3}\n",
    "=\n",
    "E_{\\text{the}}\n",
    "+\n",
    "PE_3\n",
    "$$\n",
    "\n",
    "Since:\n",
    "\n",
    "$$\n",
    "PE_0 \\neq PE_3\n",
    "$$\n",
    "\n",
    "we get:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_{\\text{the},0}\n",
    "\\neq\n",
    "X_{\\text{the},3}\n",
    "}\n",
    "$$\n",
    "\n",
    "So even though both tokens are the same word, the encoder receives **different vectors** because they occur at different positions.\n",
    "\n",
    "---\n",
    "\n",
    "### What improves after adding positional encoding?\n",
    "\n",
    "Before positional encoding, the representation mainly tells us:\n",
    "\n",
    "$$\n",
    "\\text{\"What token is this?\"}\n",
    "$$\n",
    "\n",
    "After positional encoding, the representation tells us:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{\"What token is this?\"}\n",
    "+\n",
    "\\text{\"Where does it occur?\"}\n",
    "}\n",
    "$$\n",
    "\n",
    "For example:\n",
    "\n",
    "$$\n",
    "E_{\\text{cat}}\n",
    "+\n",
    "PE_1\n",
    "$$\n",
    "\n",
    "means approximately:\n",
    "\n",
    "> **\"This is the token 'cat', and it occurs at position 1.\"**\n",
    "\n",
    "Similarly:\n",
    "\n",
    "$$\n",
    "E_{\\text{mouse}}\n",
    "+\n",
    "PE_4\n",
    "$$\n",
    "\n",
    "means approximately:\n",
    "\n",
    "> **\"This is the token 'mouse', and it occurs at position 4.\"**\n",
    "\n",
    "This enriched representation is what we send into the self-attention mechanism.\n",
    "\n",
    "---\n",
    "\n",
    "### Final representation before Q, K, and V\n",
    "\n",
    "For all five tokens:\n",
    "\n",
    "$$\n",
    "X\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "E_{\\text{the}} + PE_0 \\\\\n",
    "E_{\\text{cat}} + PE_1 \\\\\n",
    "E_{\\text{chased}} + PE_2 \\\\\n",
    "E_{\\text{the}} + PE_3 \\\\\n",
    "E_{\\text{mouse}} + PE_4\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "with:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X \\in \\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "This matrix $X$ becomes the input used to create:\n",
    "\n",
    "$$\n",
    "Q = XW_Q + b_Q\n",
    "$$\n",
    "\n",
    "$$\n",
    "K = XW_K + b_K\n",
    "$$\n",
    "\n",
    "$$\n",
    "V = XW_V + b_V\n",
    "$$\n",
    "\n",
    "So the flow is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Tokens}\n",
    "\\rightarrow\n",
    "\\text{Embeddings}\n",
    "\\rightarrow\n",
    "+\\text{Positional Encoding}\n",
    "\\rightarrow\n",
    "X\n",
    "\\rightarrow\n",
    "Q,K,V\n",
    "}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fa2af060",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:22.965040Z",
     "iopub.status.busy": "2026-10-03T18:57:22.964874Z",
     "iopub.status.idle": "2026-10-03T18:57:22.975230Z",
     "shell.execute_reply": "2026-10-03T18:57:22.974727Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭───────────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">STEP 3 — EMBEDDING + POSITION</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰───────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭───────────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1mSTEP 3 — EMBEDDING + POSITION\u001b[0m \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰───────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Equation</span>  <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> + <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mEquation\u001b[0m  \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m + \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                  X = token_embedding + positional_encoding   shape=(5, 512)                  </span>\n",
       "                                                                                              \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.0170    1.0022    0.0268    1.0356    0.0132    0.9950    0.0252    1.0307   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>   0.8339    0.5558    0.8313    0.5579    0.7659    0.6183    0.8039    0.6335   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>   0.8845   -0.4005    0.9308   -0.3499    0.9688   -0.2782    0.9428   -0.2295   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.1581   -0.9877    0.2719   -0.9339    0.3560   -0.9445    0.4588   -0.8703   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.7599   -0.6718   -0.6895   -0.8115   -0.5526   -0.8228   -0.4618   -0.9059   …  \n",
       "                                                                                              \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                  X = token_embedding + positional_encoding   shape=(5, 512)                  \u001b[0m\n",
       "                                                                                              \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.0170    1.0022    0.0268    1.0356    0.0132    0.9950    0.0252    1.0307   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m   0.8339    0.5558    0.8313    0.5579    0.7659    0.6183    0.8039    0.6335   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m   0.8845   -0.4005    0.9308   -0.3499    0.9688   -0.2782    0.9428   -0.2295   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.1581   -0.9877    0.2719   -0.9339    0.3560   -0.9445    0.4588   -0.8703   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.7599   -0.6718   -0.6895   -0.8115   -0.5526   -0.8228   -0.4618   -0.9059   …  \n",
       "                                                                                              \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "The two <span style=\"color: #008000; text-decoration-color: #008000\">'the'</span> vectors still identical after adding position? <span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold; font-style: italic\">False</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "The two \u001b[32m'the'\u001b[0m vectors still identical after adding position? \u001b[1;3;91mFalse\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "def add_embedding_and_position(token_embeddings, positional_encoding):\n",
    "    return token_embeddings + positional_encoding\n",
    "\n",
    "x0 = add_embedding_and_position(token_embeddings, positional_encoding)\n",
    "\n",
    "console.print(Panel.fit(\"[bold]STEP 3 — EMBEDDING + POSITION[/bold]\", border_style=\"green\"))\n",
    "print_shape_equation(\"Equation\", f\"{tuple(token_embeddings.shape)} + {tuple(positional_encoding.shape)} -> {tuple(x0.shape)}\")\n",
    "rich_tensor(\"X = token_embedding + positional_encoding\", x0, row_labels=tokens)\n",
    "\n",
    "same_after_pos = torch.allclose(x0[0], x0[3])\n",
    "console.print(f\"\\nThe two 'the' vectors still identical after adding position? [bold]{same_after_pos}[/bold]\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4dc10e9",
   "metadata": {},
   "source": [
    "## 5. Build Query, Key, Value matrices from scratch\n",
    "\n",
    "For the complete model representation:\n",
    "\n",
    "$$\n",
    "X \\in \\mathbb{R}^{n \\times 512}\n",
    "$$\n",
    "\n",
    "For our example:\n",
    "\n",
    "$$\n",
    "X \\in \\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "We learn three projection matrices and create Query, Key, and Value:\n",
    "\n",
    "$$\n",
    "Q = XW_Q + b_Q\n",
    "$$\n",
    "\n",
    "$$\n",
    "K = XW_K + b_K\n",
    "$$\n",
    "\n",
    "$$\n",
    "V = XW_V + b_V\n",
    "$$\n",
    "\n",
    "Each projection keeps the same overall shape:\n",
    "\n",
    "$$\n",
    "Q,K,V \\in \\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Split into 4 attention heads\n",
    "\n",
    "Since:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "h = 4\n",
    "$$\n",
    "\n",
    "each head gets:\n",
    "\n",
    "$$\n",
    "d_{\\text{head}}\n",
    "=\n",
    "\\frac{512}{4}\n",
    "=\n",
    "128\n",
    "$$\n",
    "\n",
    "So each matrix is first reshaped:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,4,128]\n",
    "$$\n",
    "\n",
    "and then transposed to:\n",
    "\n",
    "$$\n",
    "[5,4,128]\n",
    "\\rightarrow\n",
    "[4,5,128]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "Q_h,K_h,V_h\n",
    "\\in\n",
    "\\mathbb{R}^{4 \\times 5 \\times 128}\n",
    "$$\n",
    "\n",
    "This means:\n",
    "\n",
    "- $4$ = number of attention heads\n",
    "- $5$ = number of tokens\n",
    "- $128$ = features handled by each head\n",
    "\n",
    "For one individual head:\n",
    "\n",
    "$$\n",
    "Q_h \\in \\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "$$\n",
    "K_h \\in \\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "$$\n",
    "V_h \\in \\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "So each attention head gets its **own Query, Key, and Value representation** of shape:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[5 \\times 128]\n",
    "}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6859133",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### What are Query, Key, and Value?\n",
    "\n",
    "Query, Key, and Value are three different learned representations of the same token.\n",
    "\n",
    "For each token, the encoder creates:\n",
    "\n",
    "$$\n",
    "Q = XW_Q\n",
    "$$\n",
    "\n",
    "$$\n",
    "K = XW_K\n",
    "$$\n",
    "\n",
    "$$\n",
    "V = XW_V\n",
    "$$\n",
    "\n",
    "You can think of them very simply as:\n",
    "\n",
    "- **Query (Q)** = \"What am I looking for?\"\n",
    "- **Key (K)** = \"What information do I contain?\"\n",
    "- **Value (V)** = \"What information should I pass forward?\"\n",
    "\n",
    "For example, in the sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "suppose the token **\"chased\"** is acting as a Query.\n",
    "\n",
    "Its Query vector asks:\n",
    "\n",
    "> \"Which other words are important for understanding me?\"\n",
    "\n",
    "The Key vectors of all tokens are compared with this Query:\n",
    "\n",
    "- Key of **The**\n",
    "- Key of **cat**\n",
    "- Key of **chased**\n",
    "- Key of **the**\n",
    "- Key of **mouse**\n",
    "\n",
    "The similarity between Query and Key decides **how much attention** should be given to each token.\n",
    "\n",
    "Mathematically:\n",
    "\n",
    "$$\n",
    "\\text{Attention Score}\n",
    "=\n",
    "QK^T\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\text{Query} \\times \\text{Key}\n",
    "\\rightarrow\n",
    "\\text{importance score}\n",
    "$$\n",
    "\n",
    "After Softmax, these scores become attention weights.\n",
    "\n",
    "Then the attention weights are used to combine the **Value vectors**:\n",
    "\n",
    "$$\n",
    "\\text{Context}\n",
    "=\n",
    "\\text{Attention Weights} \\times V\n",
    "$$\n",
    "\n",
    "So the basic idea is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q \\text{ asks}\n",
    "\\quad\n",
    "K \\text{ matches}\n",
    "\\quad\n",
    "V \\text{ provides information}\n",
    "}\n",
    "$$\n",
    "\n",
    "### Why do we need separate Q, K, and V?\n",
    "\n",
    "If we used the same representation for everything, the model would have less flexibility.\n",
    "\n",
    "Separate learned projections allow the model to learn:\n",
    "\n",
    "- one representation for **searching**\n",
    "- one representation for **matching**\n",
    "- one representation for **passing useful information**\n",
    "\n",
    "So even though Q, K, and V all come from the same input matrix $X$, they learn different roles:\n",
    "\n",
    "$$\n",
    "X\n",
    "\\rightarrow\n",
    "\\begin{cases}\n",
    "Q = XW_Q \\\\\n",
    "K = XW_K \\\\\n",
    "V = XW_V\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "This is what allows self-attention to learn relationships such as:\n",
    "\n",
    "> **\"chased\"** may strongly attend to **\"cat\"** and **\"mouse\"**\n",
    "\n",
    "because those words help explain **who performed the action** and **who received the action**.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b45be23b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:22.976226Z",
     "iopub.status.busy": "2026-10-03T18:57:22.976094Z",
     "iopub.status.idle": "2026-10-03T18:57:23.072421Z",
     "shell.execute_reply": "2026-10-03T18:57:23.071849Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭──────────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">STEP 4 — Q, K, V PROJECTIONS</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰──────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭──────────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1mSTEP 4 — Q, K, V PROJECTIONS\u001b[0m \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰──────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Q</span>  X <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> @ W_Q <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mQ\u001b[0m  X \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m @ W_Q \u001b[1m(\u001b[0m\u001b[1;36m512\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">K</span>  X <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> @ W_K <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mK\u001b[0m  X \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m @ W_K \u001b[1m(\u001b[0m\u001b[1;36m512\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">V</span>  X <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> @ W_V <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mV\u001b[0m  X \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m @ W_V \u001b[1m(\u001b[0m\u001b[1;36m512\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                        Q (before splitting heads)   shape=(5, 512)                        </span>\n",
       "                                                                                           \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.4349    0.1101    0.3769   -0.7527   -1.3162   -0.3056   0.7203   0.7364   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  0.7429   -0.0434    0.0783   -0.5772   -1.1181   -0.4190   0.9162   0.6832   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  0.7980   -0.0355   -0.1860   -0.4777   -0.8596   -0.6723   1.0308   0.5844   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.6722    0.0881   -0.2958   -0.3323   -0.7911   -0.9444   1.1515   0.5184   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  0.5292    0.3306   -0.0757   -0.1700   -1.0404   -1.3629   1.2498   0.4092   …  \n",
       "                                                                                           \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                        Q (before splitting heads)   shape=(5, 512)                        \u001b[0m\n",
       "                                                                                           \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.4349    0.1101    0.3769   -0.7527   -1.3162   -0.3056   0.7203   0.7364   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  0.7429   -0.0434    0.0783   -0.5772   -1.1181   -0.4190   0.9162   0.6832   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  0.7980   -0.0355   -0.1860   -0.4777   -0.8596   -0.6723   1.0308   0.5844   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.6722    0.0881   -0.2958   -0.3323   -0.7911   -0.9444   1.1515   0.5184   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  0.5292    0.3306   -0.0757   -0.1700   -1.0404   -1.3629   1.2498   0.4092   …  \n",
       "                                                                                           \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                        K (before splitting heads)   shape=(5, 512)                         </span>\n",
       "                                                                                            \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5956   -0.6828   0.2010   -0.3253   -0.3321   1.4605   -0.0008   -0.2871   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.4940   -0.5399   0.0640   -0.0194   -0.3363   1.4814    0.1016   -0.1597   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.2140   -0.2842   0.0513    0.3076   -0.3542   1.3128    0.0614   -0.2182   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.0178    0.0321   0.1768    0.4946   -0.3842   1.0531   -0.0954   -0.3215   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>   0.0398    0.2608   0.4211    0.4714   -0.3050   0.7877   -0.3312   -0.3335   …  \n",
       "                                                                                            \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                        K (before splitting heads)   shape=(5, 512)                         \u001b[0m\n",
       "                                                                                            \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5956   -0.6828   0.2010   -0.3253   -0.3321   1.4605   -0.0008   -0.2871   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.4940   -0.5399   0.0640   -0.0194   -0.3363   1.4814    0.1016   -0.1597   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.2140   -0.2842   0.0513    0.3076   -0.3542   1.3128    0.0614   -0.2182   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.0178    0.0321   0.1768    0.4946   -0.3842   1.0531   -0.0954   -0.3215   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m   0.0398    0.2608   0.4211    0.4714   -0.3050   0.7877   -0.3312   -0.3335   …  \n",
       "                                                                                            \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                       V (before splitting heads)   shape=(5, 512)                        </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.6727    0.4839   0.8278   0.6768   -0.2207   -0.6312   0.6327   0.2646   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.6209    0.4693   0.6227   0.5054   -0.4619   -0.6714   0.7093   0.5068   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.5228    0.3070   0.4394   0.3963   -0.6531   -0.7729   0.8251   0.6490   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5468    0.0830   0.3967   0.4525   -0.8002   -0.8030   0.9212   0.7482   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.7042   -0.1212   0.5537   0.6279   -0.6820   -0.8410   0.9603   0.9041   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                       V (before splitting heads)   shape=(5, 512)                        \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.6727    0.4839   0.8278   0.6768   -0.2207   -0.6312   0.6327   0.2646   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.6209    0.4693   0.6227   0.5054   -0.4619   -0.6714   0.7093   0.5068   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.5228    0.3070   0.4394   0.3963   -0.6531   -0.7729   0.8251   0.6490   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5468    0.0830   0.3967   0.4525   -0.8002   -0.8030   0.9212   0.7482   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.7042   -0.1212   0.5537   0.6279   -0.6820   -0.8410   0.9603   0.9041   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">After splitting:</span> <span style=\"color: #808000; text-decoration-color: #808000\">Qh</span>=<span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">128</span><span style=\"font-weight: bold\">)</span>, <span style=\"color: #808000; text-decoration-color: #808000\">Kh</span>=<span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">128</span><span style=\"font-weight: bold\">)</span>, <span style=\"color: #808000; text-decoration-color: #808000\">Vh</span>=<span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">128</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\u001b[1;36mAfter splitting:\u001b[0m \u001b[33mQh\u001b[0m=\u001b[1m(\u001b[0m\u001b[1;36m4\u001b[0m, \u001b[1;36m5\u001b[0m, \u001b[1;36m128\u001b[0m\u001b[1m)\u001b[0m, \u001b[33mKh\u001b[0m=\u001b[1m(\u001b[0m\u001b[1;36m4\u001b[0m, \u001b[1;36m5\u001b[0m, \u001b[1;36m128\u001b[0m\u001b[1m)\u001b[0m, \u001b[33mVh\u001b[0m=\u001b[1m(\u001b[0m\u001b[1;36m4\u001b[0m, \u001b[1;36m5\u001b[0m, \u001b[1;36m128\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">────────────────────────────────────────────────────────────────── </span>Head <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m────────────────────────────────────────────────────────────────── \u001b[0mHead \u001b[1;36m1\u001b[0m\u001b[92m ──────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                 Q head 1   shape=(5, 128)                                 </span>\n",
       "                                                                                           \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.4349    0.1101    0.3769   -0.7527   -1.3162   -0.3056   0.7203   0.7364   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  0.7429   -0.0434    0.0783   -0.5772   -1.1181   -0.4190   0.9162   0.6832   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  0.7980   -0.0355   -0.1860   -0.4777   -0.8596   -0.6723   1.0308   0.5844   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.6722    0.0881   -0.2958   -0.3323   -0.7911   -0.9444   1.1515   0.5184   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  0.5292    0.3306   -0.0757   -0.1700   -1.0404   -1.3629   1.2498   0.4092   …  \n",
       "                                                                                           \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                 Q head 1   shape=(5, 128)                                 \u001b[0m\n",
       "                                                                                           \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.4349    0.1101    0.3769   -0.7527   -1.3162   -0.3056   0.7203   0.7364   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  0.7429   -0.0434    0.0783   -0.5772   -1.1181   -0.4190   0.9162   0.6832   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  0.7980   -0.0355   -0.1860   -0.4777   -0.8596   -0.6723   1.0308   0.5844   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.6722    0.0881   -0.2958   -0.3323   -0.7911   -0.9444   1.1515   0.5184   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  0.5292    0.3306   -0.0757   -0.1700   -1.0404   -1.3629   1.2498   0.4092   …  \n",
       "                                                                                           \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                 K head 1   shape=(5, 128)                                  </span>\n",
       "                                                                                            \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5956   -0.6828   0.2010   -0.3253   -0.3321   1.4605   -0.0008   -0.2871   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.4940   -0.5399   0.0640   -0.0194   -0.3363   1.4814    0.1016   -0.1597   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.2140   -0.2842   0.0513    0.3076   -0.3542   1.3128    0.0614   -0.2182   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.0178    0.0321   0.1768    0.4946   -0.3842   1.0531   -0.0954   -0.3215   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>   0.0398    0.2608   0.4211    0.4714   -0.3050   0.7877   -0.3312   -0.3335   …  \n",
       "                                                                                            \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                 K head 1   shape=(5, 128)                                  \u001b[0m\n",
       "                                                                                            \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5956   -0.6828   0.2010   -0.3253   -0.3321   1.4605   -0.0008   -0.2871   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.4940   -0.5399   0.0640   -0.0194   -0.3363   1.4814    0.1016   -0.1597   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.2140   -0.2842   0.0513    0.3076   -0.3542   1.3128    0.0614   -0.2182   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.0178    0.0321   0.1768    0.4946   -0.3842   1.0531   -0.0954   -0.3215   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m   0.0398    0.2608   0.4211    0.4714   -0.3050   0.7877   -0.3312   -0.3335   …  \n",
       "                                                                                            \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                V head 1   shape=(5, 128)                                 </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.6727    0.4839   0.8278   0.6768   -0.2207   -0.6312   0.6327   0.2646   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.6209    0.4693   0.6227   0.5054   -0.4619   -0.6714   0.7093   0.5068   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.5228    0.3070   0.4394   0.3963   -0.6531   -0.7729   0.8251   0.6490   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5468    0.0830   0.3967   0.4525   -0.8002   -0.8030   0.9212   0.7482   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.7042   -0.1212   0.5537   0.6279   -0.6820   -0.8410   0.9603   0.9041   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                V head 1   shape=(5, 128)                                 \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.6727    0.4839   0.8278   0.6768   -0.2207   -0.6312   0.6327   0.2646   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.6209    0.4693   0.6227   0.5054   -0.4619   -0.6714   0.7093   0.5068   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.5228    0.3070   0.4394   0.3963   -0.6531   -0.7729   0.8251   0.6490   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5468    0.0830   0.3967   0.4525   -0.8002   -0.8030   0.9212   0.7482   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.7042   -0.1212   0.5537   0.6279   -0.6820   -0.8410   0.9603   0.9041   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">────────────────────────────────────────────────────────────────── </span>Head <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m────────────────────────────────────────────────────────────────── \u001b[0mHead \u001b[1;36m2\u001b[0m\u001b[92m ──────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                Q head 2   shape=(5, 128)                                </span>\n",
       "                                                                                         \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  1.7352   -0.2445   1.5578   0.1535   -1.0838   -0.4259   1.1537   0.5364   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  1.9998   -0.2617   1.3637   0.4105   -0.7735   -0.3550   1.1952   0.6171   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  2.1255   -0.4825   1.0868   0.5058   -0.7419   -0.3723   1.2270   0.7661   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  2.0581   -0.7894   0.8312   0.4853   -0.8816   -0.3404   1.1805   0.9275   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  1.7368   -1.1859   0.6212   0.4201   -1.1304   -0.3121   1.1003   1.0398   …  \n",
       "                                                                                         \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                Q head 2   shape=(5, 128)                                \u001b[0m\n",
       "                                                                                         \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  1.7352   -0.2445   1.5578   0.1535   -1.0838   -0.4259   1.1537   0.5364   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  1.9998   -0.2617   1.3637   0.4105   -0.7735   -0.3550   1.1952   0.6171   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  2.1255   -0.4825   1.0868   0.5058   -0.7419   -0.3723   1.2270   0.7661   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  2.0581   -0.7894   0.8312   0.4853   -0.8816   -0.3404   1.1805   0.9275   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  1.7368   -1.1859   0.6212   0.4201   -1.1304   -0.3121   1.1003   1.0398   …  \n",
       "                                                                                         \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                K head 2   shape=(5, 128)                                 </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">      4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.6109   -0.7319   0.5658   0.5335   0.4210   0.2187   -1.3617   -0.8049   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>   0.5820   -0.7201   0.5693   0.3705   0.5243   0.2469   -1.2132   -0.8639   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>   0.4707   -0.6744   0.4608   0.3708   0.6905   0.3094   -1.0680   -0.7838   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.1384   -0.7070   0.3190   0.5481   0.6512   0.3594   -1.0589   -0.6291   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.1402   -0.7245   0.2594   0.7210   0.6713   0.3233   -1.1302   -0.4216   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                K head 2   shape=(5, 128)                                 \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.6109   -0.7319   0.5658   0.5335   0.4210   0.2187   -1.3617   -0.8049   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m   0.5820   -0.7201   0.5693   0.3705   0.5243   0.2469   -1.2132   -0.8639   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m   0.4707   -0.6744   0.4608   0.3708   0.6905   0.3094   -1.0680   -0.7838   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.1384   -0.7070   0.3190   0.5481   0.6512   0.3594   -1.0589   -0.6291   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.1402   -0.7245   0.2594   0.7210   0.6713   0.3233   -1.1302   -0.4216   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                V head 2   shape=(5, 128)                                 </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">      4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.2905   0.3208   -0.9611   -0.9854   1.0510    0.0485   -0.1556   0.7391   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  0.5097   0.5711   -0.8096   -1.0663   0.8977   -0.0466   -0.3094   0.6788   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  0.7296   0.9084   -0.4996   -1.0797   0.7794   -0.0912   -0.3918   0.5697   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.9101   0.9531   -0.1167   -0.9508   0.7264   -0.1225   -0.3612   0.4787   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  1.0562   0.7525    0.0823   -0.8261   0.7653   -0.2505   -0.2228   0.3969   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                V head 2   shape=(5, 128)                                 \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.2905   0.3208   -0.9611   -0.9854   1.0510    0.0485   -0.1556   0.7391   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  0.5097   0.5711   -0.8096   -1.0663   0.8977   -0.0466   -0.3094   0.6788   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  0.7296   0.9084   -0.4996   -1.0797   0.7794   -0.0912   -0.3918   0.5697   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.9101   0.9531   -0.1167   -0.9508   0.7264   -0.1225   -0.3612   0.4787   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  1.0562   0.7525    0.0823   -0.8261   0.7653   -0.2505   -0.2228   0.3969   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">────────────────────────────────────────────────────────────────── </span>Head <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m────────────────────────────────────────────────────────────────── \u001b[0mHead \u001b[1;36m3\u001b[0m\u001b[92m ──────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                 Q head 3   shape=(5, 128)                                 </span>\n",
       "                                                                                           \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  1.2081   -0.4714   0.8914    0.6546    0.0227   -0.0305   0.2955   -0.8358   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  1.0171   -0.3716   0.8130    0.3342   -0.0984   -0.2955   0.3551   -0.8226   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  0.7138   -0.2701   0.7254   -0.0106    0.0914   -0.4098   0.2874   -0.6010   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.3407   -0.3033   0.6858   -0.3040    0.3341   -0.3540   0.2958   -0.3136   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  0.0331   -0.3323   0.8111   -0.5836    0.4345   -0.2312   0.3485   -0.0471   …  \n",
       "                                                                                           \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                 Q head 3   shape=(5, 128)                                 \u001b[0m\n",
       "                                                                                           \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  1.2081   -0.4714   0.8914    0.6546    0.0227   -0.0305   0.2955   -0.8358   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  1.0171   -0.3716   0.8130    0.3342   -0.0984   -0.2955   0.3551   -0.8226   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  0.7138   -0.2701   0.7254   -0.0106    0.0914   -0.4098   0.2874   -0.6010   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.3407   -0.3033   0.6858   -0.3040    0.3341   -0.3540   0.2958   -0.3136   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  0.0331   -0.3323   0.8111   -0.5836    0.4345   -0.2312   0.3485   -0.0471   …  \n",
       "                                                                                           \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                 K head 3   shape=(5, 128)                                  </span>\n",
       "                                                                                            \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.0513   0.5168   -0.3987   -0.2686   -1.1826    0.1851   -0.5512   0.5756   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>   0.2507   0.3621   -0.5146   -0.1520   -1.0830    0.2319   -0.4620   0.6134   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>   0.1895   0.2676   -0.4481    0.0608   -0.9061    0.2342   -0.3444   0.6231   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.0376   0.2741   -0.2347    0.3613   -0.7652   -0.0135   -0.2200   0.6043   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.3747   0.2515    0.0192    0.5908   -0.6177   -0.2637   -0.1433   0.4789   …  \n",
       "                                                                                            \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                 K head 3   shape=(5, 128)                                  \u001b[0m\n",
       "                                                                                            \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.0513   0.5168   -0.3987   -0.2686   -1.1826    0.1851   -0.5512   0.5756   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m   0.2507   0.3621   -0.5146   -0.1520   -1.0830    0.2319   -0.4620   0.6134   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m   0.1895   0.2676   -0.4481    0.0608   -0.9061    0.2342   -0.3444   0.6231   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.0376   0.2741   -0.2347    0.3613   -0.7652   -0.0135   -0.2200   0.6043   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.3747   0.2515    0.0192    0.5908   -0.6177   -0.2637   -0.1433   0.4789   …  \n",
       "                                                                                            \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                 V head 3   shape=(5, 128)                                 </span>\n",
       "                                                                                           \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  1.5298   0.2752    0.1657   -1.3398   -0.0206    0.0586   -0.8584   0.0538   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  1.5845   0.4150    0.0360   -1.1798   -0.2069   -0.0410   -0.8314   0.0157   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  1.5510   0.4376   -0.1192   -0.8003   -0.2872    0.0122   -0.7202   0.0426   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  1.3055   0.3707   -0.2395   -0.4697   -0.2949    0.1127   -0.5978   0.2227   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  0.9308   0.3829   -0.1925   -0.3045   -0.4109    0.4149   -0.5474   0.2431   …  \n",
       "                                                                                           \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                 V head 3   shape=(5, 128)                                 \u001b[0m\n",
       "                                                                                           \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  1.5298   0.2752    0.1657   -1.3398   -0.0206    0.0586   -0.8584   0.0538   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  1.5845   0.4150    0.0360   -1.1798   -0.2069   -0.0410   -0.8314   0.0157   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  1.5510   0.4376   -0.1192   -0.8003   -0.2872    0.0122   -0.7202   0.0426   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  1.3055   0.3707   -0.2395   -0.4697   -0.2949    0.1127   -0.5978   0.2227   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  0.9308   0.3829   -0.1925   -0.3045   -0.4109    0.4149   -0.5474   0.2431   …  \n",
       "                                                                                           \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">────────────────────────────────────────────────────────────────── </span>Head <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m────────────────────────────────────────────────────────────────── \u001b[0mHead \u001b[1;36m4\u001b[0m\u001b[92m ──────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                Q head 4   shape=(5, 128)                                </span>\n",
       "                                                                                         \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.8380   0.0744   -0.8449   1.9355   -0.1150   0.3192   0.3207   0.5914   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.7459   0.3228   -0.8635   1.9440   -0.3041   0.3539   0.1352   0.2409   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.5776   0.5614   -0.9502   1.7987   -0.6438   0.3437   0.0908   0.0354   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.4345   0.5769   -1.0410   1.5957   -1.0001   0.2037   0.2078   0.0058   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.3248   0.5283   -1.1030   1.5313   -1.2276   0.0316   0.4601   0.0455   …  \n",
       "                                                                                         \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                Q head 4   shape=(5, 128)                                \u001b[0m\n",
       "                                                                                         \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.8380   0.0744   -0.8449   1.9355   -0.1150   0.3192   0.3207   0.5914   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.7459   0.3228   -0.8635   1.9440   -0.3041   0.3539   0.1352   0.2409   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.5776   0.5614   -0.9502   1.7987   -0.6438   0.3437   0.0908   0.0354   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.4345   0.5769   -1.0410   1.5957   -1.0001   0.2037   0.2078   0.0058   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.3248   0.5283   -1.1030   1.5313   -1.2276   0.0316   0.4601   0.0455   …  \n",
       "                                                                                         \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                 K head 4   shape=(5, 128)                                  </span>\n",
       "                                                                                            \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.7858   -1.2545   0.9124    0.2969    0.0774   1.2560   -1.0147    0.2097   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.9288   -1.3199   0.9172    0.3674    0.0043   1.1414   -0.9310    0.0197   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.8299   -1.3554   1.0310    0.3352   -0.1665   1.0345   -0.7694   -0.1467   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5293   -1.2870   1.2468    0.1490   -0.2290   1.0853   -0.6786   -0.3331   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.2133   -1.0865   1.5069   -0.1311   -0.2783   1.3552   -0.6813   -0.5680   …  \n",
       "                                                                                            \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                 K head 4   shape=(5, 128)                                  \u001b[0m\n",
       "                                                                                            \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.7858   -1.2545   0.9124    0.2969    0.0774   1.2560   -1.0147    0.2097   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.9288   -1.3199   0.9172    0.3674    0.0043   1.1414   -0.9310    0.0197   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.8299   -1.3554   1.0310    0.3352   -0.1665   1.0345   -0.7694   -0.1467   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5293   -1.2870   1.2468    0.1490   -0.2290   1.0853   -0.6786   -0.3331   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.2133   -1.0865   1.5069   -0.1311   -0.2783   1.3552   -0.6813   -0.5680   …  \n",
       "                                                                                            \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                V head 4   shape=(5, 128)                                 </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.7115   1.2571   0.4023    0.0189   -1.9275   1.3956    0.2697   0.5918   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.6592   1.2853   0.3760   -0.0595   -1.7332   1.1316    0.0200   0.7486   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.5830   1.3591   0.4208   -0.1601   -1.3833   0.8136   -0.1626   0.7415   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.4100   1.4369   0.4678   -0.0830   -1.0783   0.6715   -0.2877   0.5529   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.2326   1.3486   0.5088   -0.0125   -0.9449   0.5955   -0.1728   0.2537   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                V head 4   shape=(5, 128)                                 \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.7115   1.2571   0.4023    0.0189   -1.9275   1.3956    0.2697   0.5918   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.6592   1.2853   0.3760   -0.0595   -1.7332   1.1316    0.0200   0.7486   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.5830   1.3591   0.4208   -0.1601   -1.3833   0.8136   -0.1626   0.7415   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.4100   1.4369   0.4678   -0.0830   -1.0783   0.6715   -0.2877   0.5529   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.2326   1.3486   0.5088   -0.0125   -0.9449   0.5955   -0.1728   0.2537   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "class ManualLinear(nn.Module):\n",
    "    \"\"\"y = xW + b, written explicitly instead of nn.Linear.\"\"\"\n",
    "    def __init__(self, in_features, out_features):\n",
    "        super().__init__()\n",
    "        self.weight = nn.Parameter(torch.empty(in_features, out_features))\n",
    "        self.bias = nn.Parameter(torch.zeros(out_features))\n",
    "        nn.init.xavier_uniform_(self.weight)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return x @ self.weight + self.bias\n",
    "\n",
    "\n",
    "def split_heads(x, num_heads):\n",
    "    # x: [n, d_model]\n",
    "    n, d_model = x.shape\n",
    "    d_head = d_model // num_heads\n",
    "    # [n, d_model] -> [n, h, d_head] -> [h, n, d_head]\n",
    "    return x.reshape(n, num_heads, d_head).transpose(0, 1)\n",
    "\n",
    "\n",
    "def combine_heads(x):\n",
    "    # x: [h, n, d_head]\n",
    "    h, n, d_head = x.shape\n",
    "    # [h, n, d_head] -> [n, h, d_head] -> [n, h*d_head]\n",
    "    return x.transpose(0, 1).contiguous().reshape(n, h * d_head)\n",
    "\n",
    "\n",
    "WQ = ManualLinear(D_MODEL, D_MODEL).to(device)\n",
    "WK = ManualLinear(D_MODEL, D_MODEL).to(device)\n",
    "WV = ManualLinear(D_MODEL, D_MODEL).to(device)\n",
    "\n",
    "Q = WQ(x0)\n",
    "K = WK(x0)\n",
    "V = WV(x0)\n",
    "\n",
    "Qh = split_heads(Q, NUM_HEADS)\n",
    "Kh = split_heads(K, NUM_HEADS)\n",
    "Vh = split_heads(V, NUM_HEADS)\n",
    "\n",
    "console.print(Panel.fit(\"[bold]STEP 4 — Q, K, V PROJECTIONS[/bold]\", border_style=\"green\"))\n",
    "print_shape_equation(\"Q\", f\"X {tuple(x0.shape)} @ W_Q {(D_MODEL, D_MODEL)} -> {tuple(Q.shape)}\")\n",
    "print_shape_equation(\"K\", f\"X {tuple(x0.shape)} @ W_K {(D_MODEL, D_MODEL)} -> {tuple(K.shape)}\")\n",
    "print_shape_equation(\"V\", f\"X {tuple(x0.shape)} @ W_V {(D_MODEL, D_MODEL)} -> {tuple(V.shape)}\")\n",
    "\n",
    "rich_tensor(\"Q (before splitting heads)\", Q, row_labels=tokens)\n",
    "rich_tensor(\"K (before splitting heads)\", K, row_labels=tokens)\n",
    "rich_tensor(\"V (before splitting heads)\", V, row_labels=tokens)\n",
    "\n",
    "console.print(f\"\\n[bold cyan]After splitting:[/bold cyan] Qh={tuple(Qh.shape)}, Kh={tuple(Kh.shape)}, Vh={tuple(Vh.shape)}\")\n",
    "for h in range(NUM_HEADS):\n",
    "    console.rule(f\"Head {h + 1}\")\n",
    "    rich_tensor(f\"Q head {h + 1}\", Qh[h], row_labels=tokens)\n",
    "    rich_tensor(f\"K head {h + 1}\", Kh[h], row_labels=tokens)\n",
    "    rich_tensor(f\"V head {h + 1}\", Vh[h], row_labels=tokens)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4ecbd53",
   "metadata": {},
   "source": [
    "## 6. Scaled dot-product attention + Softmax\n",
    "\n",
    "For each attention head, we already have:\n",
    "\n",
    "$$\n",
    "Q \\in \\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "$$\n",
    "K \\in \\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "$$\n",
    "V \\in \\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "because:\n",
    "\n",
    "$$\n",
    "d_{\\text{head}}\n",
    "=\n",
    "\\frac{d_{\\text{model}}}{h}\n",
    "=\n",
    "\\frac{512}{4}\n",
    "=\n",
    "128\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Step 1: Calculate attention scores\n",
    "\n",
    "The raw attention score is:\n",
    "\n",
    "$$\n",
    "\\text{Scores}\n",
    "=\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "$$\n",
    "\n",
    "Here:\n",
    "\n",
    "$$\n",
    "Q \\in \\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "K^T \\in \\mathbb{R}^{128 \\times 5}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "(5 \\times 128)\n",
    "(128 \\times 5)\n",
    "\\rightarrow\n",
    "(5 \\times 5)\n",
    "$$\n",
    "\n",
    "So the score matrix is:\n",
    "\n",
    "$$\n",
    "\\text{Scores}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 5}\n",
    "$$\n",
    "\n",
    "For our sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "the score matrix conceptually looks like:\n",
    "\n",
    "$$\n",
    "\\text{Scores}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "s_{11} & s_{12} & s_{13} & s_{14} & s_{15} \\\\\n",
    "s_{21} & s_{22} & s_{23} & s_{24} & s_{25} \\\\\n",
    "s_{31} & s_{32} & s_{33} & s_{34} & s_{35} \\\\\n",
    "s_{41} & s_{42} & s_{43} & s_{44} & s_{45} \\\\\n",
    "s_{51} & s_{52} & s_{53} & s_{54} & s_{55}\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Each row represents one **Query token**.\n",
    "\n",
    "Each column represents one **Key token**.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "s_{ij}\n",
    "=\n",
    "\\text{similarity between Query token } i\n",
    "\\text{ and Key token } j\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Step 2: Why divide by $\\sqrt{d_k}$?\n",
    "\n",
    "Since:\n",
    "\n",
    "$$\n",
    "d_k = 128\n",
    "$$\n",
    "\n",
    "we divide the dot product by:\n",
    "\n",
    "$$\n",
    "\\sqrt{128}\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\text{Scaled Scores}\n",
    "=\n",
    "\\frac{QK^T}{\\sqrt{128}}\n",
    "$$\n",
    "\n",
    "This prevents the dot-product values from becoming too large.\n",
    "\n",
    "Without scaling, very large scores can make Softmax extremely sharp, which can reduce useful gradients during training.\n",
    "\n",
    "---\n",
    "\n",
    "### Step 3: Apply Softmax\n",
    "\n",
    "Now we apply Softmax row-wise:\n",
    "\n",
    "$$\n",
    "A\n",
    "=\n",
    "\\operatorname{softmax}\n",
    "\\left(\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "A \\in \\mathbb{R}^{5 \\times 5}\n",
    "$$\n",
    "\n",
    "The matrix \\(A\\) contains the **attention probabilities**.\n",
    "\n",
    "For example, one row might look like:\n",
    "\n",
    "$$\n",
    "[0.10,\\ 0.15,\\ 0.50,\\ 0.15,\\ 0.10]\n",
    "$$\n",
    "\n",
    "The sum of every row is:\n",
    "\n",
    "$$\n",
    "\\sum_j A_{ij} = 1\n",
    "$$\n",
    "\n",
    "So each Query token distributes **100% of its attention** across all Key tokens.\n",
    "\n",
    "---\n",
    "\n",
    "### Step 4: Multiply attention weights by \\(V\\)\n",
    "\n",
    "Now the attention probabilities are multiplied by the Value matrix:\n",
    "\n",
    "$$\n",
    "\\text{Context}\n",
    "=\n",
    "AV\n",
    "$$\n",
    "\n",
    "Dimensions:\n",
    "\n",
    "$$\n",
    "A\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 5}\n",
    "$$\n",
    "\n",
    "$$\n",
    "V\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "(5 \\times 5)\n",
    "(5 \\times 128)\n",
    "\\rightarrow\n",
    "(5 \\times 128)\n",
    "$$\n",
    "\n",
    "So the output of one attention head is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Context}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 128}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Complete formula for one attention head\n",
    "\n",
    "The entire scaled dot-product attention operation is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{Attention}(Q,K,V)\n",
    "=\n",
    "\\operatorname{softmax}\n",
    "\\left(\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "\\right)V\n",
    "}\n",
    "$$\n",
    "\n",
    "For our configuration:\n",
    "\n",
    "$$\n",
    "Q,K,V\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "$$\n",
    "QK^T\n",
    "\\rightarrow\n",
    "\\mathbb{R}^{5 \\times 5}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{Softmax}(QK^T)\n",
    "\\rightarrow\n",
    "\\mathbb{R}^{5 \\times 5}\n",
    "$$\n",
    "\n",
    "$$\n",
    "AV\n",
    "\\rightarrow\n",
    "\\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "Therefore, each of the **4 attention heads** produces:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "5 \\times 128\n",
    "}\n",
    "$$\n",
    "\n",
    "Later, the 4 heads are concatenated:\n",
    "\n",
    "$$\n",
    "4 \\times 128 = 512\n",
    "$$\n",
    "\n",
    "giving:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "5 \\times 512\n",
    "}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "661298f1",
   "metadata": {},
   "source": [
    "### What is Scaled Dot-Product Attention?\n",
    "\n",
    "Scaled Dot-Product Attention is the step where the encoder decides:\n",
    "\n",
    "> **\"Which tokens are important to each other, and how much information should each token receive from the others?\"**\n",
    "\n",
    "It uses the Query, Key, and Value matrices that we created in the previous step.\n",
    "\n",
    "The complete formula is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{Attention}(Q,K,V)\n",
    "=\n",
    "\\operatorname{softmax}\n",
    "\\left(\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "\\right)V\n",
    "}\n",
    "$$\n",
    "\n",
    "This happens in four simple stages:\n",
    "\n",
    "1. Compare Query with Key\n",
    "2. Scale the scores\n",
    "3. Apply Softmax\n",
    "4. Use the scores to combine the Value vectors\n",
    "\n",
    "---\n",
    "\n",
    "### Step 1: Query compares with Key\n",
    "\n",
    "For every token, its Query vector is compared with the Key vectors of all tokens.\n",
    "\n",
    "Mathematically:\n",
    "\n",
    "$$\n",
    "QK^T\n",
    "$$\n",
    "\n",
    "For one attention head:\n",
    "\n",
    "$$\n",
    "Q \\in \\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "K^T \\in \\mathbb{R}^{128 \\times 5}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "(5 \\times 128)\n",
    "(128 \\times 5)\n",
    "\\rightarrow\n",
    "(5 \\times 5)\n",
    "$$\n",
    "\n",
    "So we get:\n",
    "\n",
    "$$\n",
    "\\text{Scores}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 5}\n",
    "$$\n",
    "\n",
    "You can think of this score matrix as a table of:\n",
    "\n",
    "> **\"How relevant is every token to every other token?\"**\n",
    "\n",
    "For our sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "the Query of **\"chased\"** is compared with the Keys of:\n",
    "\n",
    "- **the**\n",
    "- **cat**\n",
    "- **chased**\n",
    "- **the**\n",
    "- **mouse**\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "Q_{\\text{chased}}\n",
    "\\cdot\n",
    "K_{\\text{cat}}\n",
    "$$\n",
    "\n",
    "might produce a high score because **cat** is related to the action **chased**.\n",
    "\n",
    "Similarly:\n",
    "\n",
    "$$\n",
    "Q_{\\text{chased}}\n",
    "\\cdot\n",
    "K_{\\text{mouse}}\n",
    "$$\n",
    "\n",
    "may also produce a high score.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "QK^T\n",
    "\\rightarrow\n",
    "\\text{relationship scores between tokens}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Step 2: Why scale by $\\sqrt{d_k}$?\n",
    "\n",
    "The raw dot-product scores can become very large because Query and Key contain many dimensions.\n",
    "\n",
    "In our case:\n",
    "\n",
    "$$\n",
    "d_k = 128\n",
    "$$\n",
    "\n",
    "So we divide the scores by:\n",
    "\n",
    "$$\n",
    "\\sqrt{128}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\text{Scaled Scores}\n",
    "=\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "$$\n",
    "\n",
    "This keeps the score values in a more stable range.\n",
    "\n",
    "Why is this useful?\n",
    "\n",
    "If the scores become extremely large, Softmax can become too sharp.\n",
    "\n",
    "For example, instead of something like:\n",
    "\n",
    "$$\n",
    "[0.10,\\ 0.20,\\ 0.40,\\ 0.20,\\ 0.10]\n",
    "$$\n",
    "\n",
    "Softmax might become almost:\n",
    "\n",
    "$$\n",
    "[0,\\ 0,\\ 1,\\ 0,\\ 0]\n",
    "$$\n",
    "\n",
    "Scaling helps prevent this from happening too aggressively.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Scaling}\n",
    "\\rightarrow\n",
    "\\text{more stable Softmax}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Step 3: What does Softmax do?\n",
    "\n",
    "After calculating the scaled scores, we apply Softmax:\n",
    "\n",
    "$$\n",
    "A\n",
    "=\n",
    "\\operatorname{softmax}\n",
    "\\left(\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "Softmax converts the scores into attention probabilities.\n",
    "\n",
    "For example, suppose the Query token is:\n",
    "\n",
    "> **\"chased\"**\n",
    "\n",
    "Softmax might produce:\n",
    "\n",
    "$$\n",
    "[0.05,\\ 0.35,\\ 0.15,\\ 0.05,\\ 0.40]\n",
    "$$\n",
    "\n",
    "This can be interpreted approximately as:\n",
    "\n",
    "| Token | Attention weight |\n",
    "|---|---:|\n",
    "| the | 0.05 |\n",
    "| cat | 0.35 |\n",
    "| chased | 0.15 |\n",
    "| the | 0.05 |\n",
    "| mouse | 0.40 |\n",
    "\n",
    "Every row sums to:\n",
    "\n",
    "$$\n",
    "1\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\sum_j A_{ij} = 1\n",
    "}\n",
    "$$\n",
    "\n",
    "This means each Query distributes its attention across all Key tokens.\n",
    "\n",
    "In simple terms:\n",
    "\n",
    "> **Softmax answers: \"How much attention should I give to each token?\"**\n",
    "\n",
    "---\n",
    "\n",
    "### Step 4: Why do we multiply by Value?\n",
    "\n",
    "The attention weights tell us which tokens are important.\n",
    "\n",
    "But the actual information that gets passed forward comes from the Value vectors.\n",
    "\n",
    "So we calculate:\n",
    "\n",
    "$$\n",
    "\\text{Context}\n",
    "=\n",
    "AV\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "A\n",
    "=\n",
    "\\text{Attention Weights}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "V\n",
    "=\n",
    "\\text{Value matrix}\n",
    "$$\n",
    "\n",
    "For one head:\n",
    "\n",
    "$$\n",
    "A\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 5}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "V\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "(5 \\times 5)\n",
    "(5 \\times 128)\n",
    "\\rightarrow\n",
    "(5 \\times 128)\n",
    "$$\n",
    "\n",
    "The result is:\n",
    "\n",
    "$$\n",
    "\\text{Context}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "So the Value vectors are combined according to the attention weights.\n",
    "\n",
    "---\n",
    "\n",
    "### Simple intuition\n",
    "\n",
    "You can remember the whole process like this:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q\n",
    "\\rightarrow\n",
    "\\text{What am I looking for?}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "K\n",
    "\\rightarrow\n",
    "\\text{Who matches what I am looking for?}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "QK^T\n",
    "\\rightarrow\n",
    "\\text{How relevant is each token?}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{Softmax}\n",
    "\\rightarrow\n",
    "\\text{How much attention should each token receive?}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "V\n",
    "\\rightarrow\n",
    "\\text{What information should be collected?}\n",
    "}\n",
    "$$\n",
    "\n",
    "So the complete idea is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Compare}\n",
    "\\rightarrow\n",
    "\\text{Scale}\n",
    "\\rightarrow\n",
    "\\text{Softmax}\n",
    "\\rightarrow\n",
    "\\text{Collect information from V}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Example intuition with \"chased\"\n",
    "\n",
    "For the token:\n",
    "\n",
    "> **\"chased\"**\n",
    "\n",
    "the process is roughly:\n",
    "\n",
    "**1. Query**\n",
    "\n",
    "> \"Which tokens help explain me?\"\n",
    "\n",
    "**2. Key comparison**\n",
    "\n",
    "The Query of **chased** is compared with the Keys of all tokens.\n",
    "\n",
    "**3. Softmax**\n",
    "\n",
    "The model may assign more attention to:\n",
    "\n",
    "> **cat**\n",
    "\n",
    "and:\n",
    "\n",
    "> **mouse**\n",
    "\n",
    "**4. Value combination**\n",
    "\n",
    "The Value vectors of **cat**, **mouse**, and other tokens are combined according to those attention weights.\n",
    "\n",
    "So the final representation of **\"chased\"** now contains information not only about the word itself, but also about the surrounding tokens that are relevant to it.\n",
    "\n",
    "That is why self-attention produces a more **context-aware token representation**.\n",
    "\n",
    "---\n",
    "\n",
    "### Final flow\n",
    "\n",
    "$$\n",
    "QK^T\n",
    "\\rightarrow\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "\\rightarrow\n",
    "\\operatorname{Softmax}\n",
    "\\rightarrow\n",
    "A\n",
    "\\rightarrow\n",
    "AV\n",
    "\\rightarrow\n",
    "\\text{Context}\n",
    "$$\n",
    "\n",
    "or in one formula:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{Attention}(Q,K,V)\n",
    "=\n",
    "\\operatorname{softmax}\n",
    "\\left(\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "\\right)V\n",
    "}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5eb6efc4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:23.073626Z",
     "iopub.status.busy": "2026-10-03T18:57:23.073491Z",
     "iopub.status.idle": "2026-10-03T18:57:23.132384Z",
     "shell.execute_reply": "2026-10-03T18:57:23.131795Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭─────────────────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">STEP 5 — ATTENTION SCORES + SOFTMAX</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰─────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭─────────────────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1mSTEP 5 — ATTENTION SCORES + SOFTMAX\u001b[0m \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰─────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">scores shape = <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"font-weight: bold\">)</span> = <span style=\"font-weight: bold\">(</span>heads, n, n<span style=\"font-weight: bold\">)</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "scores shape = \u001b[1m(\u001b[0m\u001b[1;36m4\u001b[0m, \u001b[1;36m5\u001b[0m, \u001b[1;36m5\u001b[0m\u001b[1m)\u001b[0m = \u001b[1m(\u001b[0mheads, n, n\u001b[1m)\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">attention_weights shape = <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "attention_weights shape = \u001b[1m(\u001b[0m\u001b[1;36m4\u001b[0m, \u001b[1;36m5\u001b[0m, \u001b[1;36m5\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">attention_weights shape = <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"font-weight: bold\">)</span> = <span style=\"font-weight: bold\">(</span>heads, n, n<span style=\"font-weight: bold\">)</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "attention_weights shape = \u001b[1m(\u001b[0m\u001b[1;36m4\u001b[0m, \u001b[1;36m5\u001b[0m, \u001b[1;36m5\u001b[0m\u001b[1m)\u001b[0m = \u001b[1m(\u001b[0mheads, n, n\u001b[1m)\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">──────────────────────────────────────────────────────────── </span>Head <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>: QKᵀ <span style=\"color: #800080; text-decoration-color: #800080\">/</span> √d_k<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m──────────────────────────────────────────────────────────── \u001b[0mHead \u001b[1;36m1\u001b[0m: QKᵀ \u001b[35m/\u001b[0m √d_k\u001b[92m ────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">        Head 1 raw attention scores   shape=(5, 5)         </span>\n",
       "╭────────┬────────┬─────────┬─────────┬─────────┬─────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">     cat </span>│<span style=\"font-weight: bold\">  chased </span>│<span style=\"font-weight: bold\">     the </span>│<span style=\"font-weight: bold\">   mouse </span>│\n",
       "├────────┼────────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.3090 │  0.3171 │  0.2812 │  0.1660 │  0.0891 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ 0.2617 │  0.2184 │  0.1166 │ -0.0766 │ -0.2071 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ 0.1554 │  0.0624 │ -0.0952 │ -0.3393 │ -0.4932 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.0922 │  0.0030 │ -0.1576 │ -0.4043 │ -0.5656 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ 0.0415 │ -0.0059 │ -0.1028 │ -0.2777 │ -0.4023 │\n",
       "╰────────┴────────┴─────────┴─────────┴─────────┴─────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m        Head 1 raw attention scores   shape=(5, 5)         \u001b[0m\n",
       "╭────────┬────────┬─────────┬─────────┬─────────┬─────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m chased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m  mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼────────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.3090 │  0.3171 │  0.2812 │  0.1660 │  0.0891 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2617 │  0.2184 │  0.1166 │ -0.0766 │ -0.2071 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ 0.1554 │  0.0624 │ -0.0952 │ -0.3393 │ -0.4932 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.0922 │  0.0030 │ -0.1576 │ -0.4043 │ -0.5656 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ 0.0415 │ -0.0059 │ -0.1028 │ -0.2777 │ -0.4023 │\n",
       "╰────────┴────────┴─────────┴─────────┴─────────┴─────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">         Head 1 softmax weights   shape=(5, 5)         </span>\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">    cat </span>│<span style=\"font-weight: bold\"> chased </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">  mouse </span>│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2151 │ 0.2168 │ 0.2091 │ 0.1864 │ 0.1726 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ 0.2403 │ 0.2301 │ 0.2078 │ 0.1713 │ 0.1504 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ 0.2616 │ 0.2384 │ 0.2036 │ 0.1595 │ 0.1368 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2618 │ 0.2394 │ 0.2039 │ 0.1593 │ 0.1356 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ 0.2388 │ 0.2277 │ 0.2067 │ 0.1735 │ 0.1532 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m         Head 1 softmax weights   shape=(5, 5)         \u001b[0m\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mchased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2151 │ 0.2168 │ 0.2091 │ 0.1864 │ 0.1726 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2403 │ 0.2301 │ 0.2078 │ 0.1713 │ 0.1504 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ 0.2616 │ 0.2384 │ 0.2036 │ 0.1595 │ 0.1368 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2618 │ 0.2394 │ 0.2039 │ 0.1593 │ 0.1356 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ 0.2388 │ 0.2277 │ 0.2067 │ 0.1735 │ 0.1532 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">Softmax row sums:</span> <span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32mSoftmax row sums:\u001b[0m \u001b[1m[\u001b[0m\u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                 Head 1 context = AttentionWeights @ V   shape=(5, 128)                  </span>\n",
       "                                                                                         \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.6121   0.2646   0.5744   0.5307   -0.5511   -0.7378   0.7998   0.5980   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.6128   0.2841   0.5848   0.5333   -0.5348   -0.7309   0.7890   0.5793   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.6141   0.2977   0.5936   0.5364   -0.5218   -0.7258   0.7810   0.5652   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.6140   0.2984   0.5937   0.5362   -0.5215   -0.7255   0.7806   0.5647   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.6129   0.2817   0.5840   0.5334   -0.5363   -0.7316   0.7902   0.5811   …  \n",
       "                                                                                         \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                 Head 1 context = AttentionWeights @ V   shape=(5, 128)                  \u001b[0m\n",
       "                                                                                         \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.6121   0.2646   0.5744   0.5307   -0.5511   -0.7378   0.7998   0.5980   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.6128   0.2841   0.5848   0.5333   -0.5348   -0.7309   0.7890   0.5793   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.6141   0.2977   0.5936   0.5364   -0.5218   -0.7258   0.7810   0.5652   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.6140   0.2984   0.5937   0.5362   -0.5215   -0.7255   0.7806   0.5647   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.6129   0.2817   0.5840   0.5334   -0.5363   -0.7316   0.7902   0.5811   …  \n",
       "                                                                                         \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">──────────────────────────────────────────────────────────── </span>Head <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>: QKᵀ <span style=\"color: #800080; text-decoration-color: #800080\">/</span> √d_k<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m──────────────────────────────────────────────────────────── \u001b[0mHead \u001b[1;36m2\u001b[0m: QKᵀ \u001b[35m/\u001b[0m √d_k\u001b[92m ────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">         Head 2 raw attention scores   shape=(5, 5)         </span>\n",
       "╭────────┬─────────┬─────────┬─────────┬─────────┬─────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">     the </span>│<span style=\"font-weight: bold\">     cat </span>│<span style=\"font-weight: bold\">  chased </span>│<span style=\"font-weight: bold\">     the </span>│<span style=\"font-weight: bold\">   mouse </span>│\n",
       "├────────┼─────────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ -0.0940 │ -0.0352 │  0.0538 │  0.1731 │  0.1201 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ -0.1632 │ -0.1420 │ -0.0904 │ -0.0037 │ -0.0544 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ -0.2554 │ -0.2643 │ -0.2345 │ -0.1575 │ -0.1762 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ -0.3503 │ -0.3813 │ -0.3626 │ -0.2841 │ -0.2711 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ -0.3701 │ -0.3896 │ -0.3613 │ -0.2788 │ -0.2729 │\n",
       "╰────────┴─────────┴─────────┴─────────┴─────────┴─────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m         Head 2 raw attention scores   shape=(5, 5)         \u001b[0m\n",
       "╭────────┬─────────┬─────────┬─────────┬─────────┬─────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m chased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m  mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼─────────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ -0.0940 │ -0.0352 │  0.0538 │  0.1731 │  0.1201 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ -0.1632 │ -0.1420 │ -0.0904 │ -0.0037 │ -0.0544 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ -0.2554 │ -0.2643 │ -0.2345 │ -0.1575 │ -0.1762 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ -0.3503 │ -0.3813 │ -0.3626 │ -0.2841 │ -0.2711 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ -0.3701 │ -0.3896 │ -0.3613 │ -0.2788 │ -0.2729 │\n",
       "╰────────┴─────────┴─────────┴─────────┴─────────┴─────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">         Head 2 softmax weights   shape=(5, 5)         </span>\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">    cat </span>│<span style=\"font-weight: bold\"> chased </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">  mouse </span>│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.1735 │ 0.1840 │ 0.2011 │ 0.2266 │ 0.2149 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ 0.1857 │ 0.1897 │ 0.1997 │ 0.2178 │ 0.2070 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ 0.1924 │ 0.1907 │ 0.1965 │ 0.2122 │ 0.2083 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.1958 │ 0.1898 │ 0.1934 │ 0.2092 │ 0.2119 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ 0.1928 │ 0.1891 │ 0.1945 │ 0.2112 │ 0.2125 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m         Head 2 softmax weights   shape=(5, 5)         \u001b[0m\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mchased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1735 │ 0.1840 │ 0.2011 │ 0.2266 │ 0.2149 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1857 │ 0.1897 │ 0.1997 │ 0.2178 │ 0.2070 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ 0.1924 │ 0.1907 │ 0.1965 │ 0.2122 │ 0.2083 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1958 │ 0.1898 │ 0.1934 │ 0.2092 │ 0.2119 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ 0.1928 │ 0.1891 │ 0.1945 │ 0.2112 │ 0.2125 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">Softmax row sums:</span> <span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32mSoftmax row sums:\u001b[0m \u001b[1m[\u001b[0m\u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                  Head 2 context = AttentionWeights @ V   shape=(5, 128)                  </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">      4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.7241   0.7211   -0.4249   -0.9772   0.8332   -0.1001   -0.2924   0.5614   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  0.7133   0.7128   -0.4402   -0.9791   0.8378   -0.0966   -0.2907   0.5663   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  0.7095   0.7081   -0.4451   -0.9789   0.8400   -0.0956   -0.2890   0.5678   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.7089   0.7057   -0.4454   -0.9780   0.8409   -0.0957   -0.2877   0.5679   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  0.7109   0.7077   -0.4427   -0.9779   0.8399   -0.0963   -0.2883   0.5671   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                  Head 2 context = AttentionWeights @ V   shape=(5, 128)                  \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.7241   0.7211   -0.4249   -0.9772   0.8332   -0.1001   -0.2924   0.5614   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  0.7133   0.7128   -0.4402   -0.9791   0.8378   -0.0966   -0.2907   0.5663   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  0.7095   0.7081   -0.4451   -0.9789   0.8400   -0.0956   -0.2890   0.5678   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.7089   0.7057   -0.4454   -0.9780   0.8409   -0.0957   -0.2877   0.5679   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  0.7109   0.7077   -0.4427   -0.9779   0.8399   -0.0963   -0.2883   0.5671   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">──────────────────────────────────────────────────────────── </span>Head <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>: QKᵀ <span style=\"color: #800080; text-decoration-color: #800080\">/</span> √d_k<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m──────────────────────────────────────────────────────────── \u001b[0mHead \u001b[1;36m3\u001b[0m: QKᵀ \u001b[35m/\u001b[0m √d_k\u001b[92m ────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">         Head 3 raw attention scores   shape=(5, 5)         </span>\n",
       "╭────────┬─────────┬─────────┬─────────┬─────────┬─────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">     the </span>│<span style=\"font-weight: bold\">     cat </span>│<span style=\"font-weight: bold\">  chased </span>│<span style=\"font-weight: bold\">     the </span>│<span style=\"font-weight: bold\">   mouse </span>│\n",
       "├────────┼─────────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ -0.0842 │ -0.0367 │ -0.0186 │ -0.1027 │ -0.2480 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│  0.0830 │  0.1121 │  0.1496 │  0.0916 │ -0.0151 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│  0.0921 │  0.0957 │  0.1373 │  0.0854 │  0.0119 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│  0.0728 │  0.0609 │  0.0861 │  0.0131 │ -0.0520 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ -0.0534 │ -0.0648 │ -0.0545 │ -0.1375 │ -0.1969 │\n",
       "╰────────┴─────────┴─────────┴─────────┴─────────┴─────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m         Head 3 raw attention scores   shape=(5, 5)         \u001b[0m\n",
       "╭────────┬─────────┬─────────┬─────────┬─────────┬─────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m chased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m  mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼─────────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ -0.0842 │ -0.0367 │ -0.0186 │ -0.1027 │ -0.2480 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│  0.0830 │  0.1121 │  0.1496 │  0.0916 │ -0.0151 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│  0.0921 │  0.0957 │  0.1373 │  0.0854 │  0.0119 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│  0.0728 │  0.0609 │  0.0861 │  0.0131 │ -0.0520 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ -0.0534 │ -0.0648 │ -0.0545 │ -0.1375 │ -0.1969 │\n",
       "╰────────┴─────────┴─────────┴─────────┴─────────┴─────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">         Head 3 softmax weights   shape=(5, 5)         </span>\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">    cat </span>│<span style=\"font-weight: bold\"> chased </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">  mouse </span>│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2021 │ 0.2120 │ 0.2158 │ 0.1984 │ 0.1716 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ 0.1995 │ 0.2054 │ 0.2132 │ 0.2012 │ 0.1808 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ 0.2014 │ 0.2021 │ 0.2107 │ 0.2000 │ 0.1858 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2072 │ 0.2047 │ 0.2100 │ 0.1952 │ 0.1829 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ 0.2095 │ 0.2071 │ 0.2093 │ 0.1926 │ 0.1815 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m         Head 3 softmax weights   shape=(5, 5)         \u001b[0m\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mchased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2021 │ 0.2120 │ 0.2158 │ 0.1984 │ 0.1716 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1995 │ 0.2054 │ 0.2132 │ 0.2012 │ 0.1808 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ 0.2014 │ 0.2021 │ 0.2107 │ 0.2000 │ 0.1858 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2072 │ 0.2047 │ 0.2100 │ 0.1952 │ 0.1829 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ 0.2095 │ 0.2071 │ 0.2093 │ 0.1926 │ 0.1815 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">Softmax row sums:</span> <span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32mSoftmax row sums:\u001b[0m \u001b[1m[\u001b[0m\u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                  Head 3 context = AttentionWeights @ V   shape=(5, 128)                  </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  1.3987   0.3773   -0.0652   -0.8391   -0.2390   0.0994   -0.7177   0.1093   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  1.3921   0.3772   -0.0680   -0.8297   -0.2414   0.1036   -0.7147   0.1118   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  1.3891   0.3768   -0.0682   -0.8274   -0.2418   0.1057   -0.7139   0.1127   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  1.3921   0.3763   -0.0653   -0.8346   -0.2396   0.1042   -0.7161   0.1112   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  1.3937   0.3761   -0.0639   -0.8383   -0.2386   0.1034   -0.7172   0.1104   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                  Head 3 context = AttentionWeights @ V   shape=(5, 128)                  \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  1.3987   0.3773   -0.0652   -0.8391   -0.2390   0.0994   -0.7177   0.1093   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  1.3921   0.3772   -0.0680   -0.8297   -0.2414   0.1036   -0.7147   0.1118   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  1.3891   0.3768   -0.0682   -0.8274   -0.2418   0.1057   -0.7139   0.1127   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  1.3921   0.3763   -0.0653   -0.8346   -0.2396   0.1042   -0.7161   0.1112   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  1.3937   0.3761   -0.0639   -0.8383   -0.2386   0.1034   -0.7172   0.1104   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">──────────────────────────────────────────────────────────── </span>Head <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>: QKᵀ <span style=\"color: #800080; text-decoration-color: #800080\">/</span> √d_k<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m──────────────────────────────────────────────────────────── \u001b[0mHead \u001b[1;36m4\u001b[0m: QKᵀ \u001b[35m/\u001b[0m √d_k\u001b[92m ────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">         Head 4 raw attention scores   shape=(5, 5)         </span>\n",
       "╭────────┬─────────┬─────────┬─────────┬─────────┬─────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">     the </span>│<span style=\"font-weight: bold\">     cat </span>│<span style=\"font-weight: bold\">  chased </span>│<span style=\"font-weight: bold\">     the </span>│<span style=\"font-weight: bold\">   mouse </span>│\n",
       "├────────┼─────────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ -0.0034 │ -0.0951 │ -0.1183 │ -0.2394 │ -0.3715 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│  0.0013 │ -0.0921 │ -0.0910 │ -0.1703 │ -0.2484 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ -0.0480 │ -0.1445 │ -0.1105 │ -0.1468 │ -0.1691 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ -0.0914 │ -0.1538 │ -0.0805 │ -0.1108 │ -0.1460 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ -0.2452 │ -0.2700 │ -0.1677 │ -0.2147 │ -0.2987 │\n",
       "╰────────┴─────────┴─────────┴─────────┴─────────┴─────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m         Head 4 raw attention scores   shape=(5, 5)         \u001b[0m\n",
       "╭────────┬─────────┬─────────┬─────────┬─────────┬─────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m chased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m    the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m  mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼─────────┼─────────┼─────────┼─────────┼─────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ -0.0034 │ -0.0951 │ -0.1183 │ -0.2394 │ -0.3715 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│  0.0013 │ -0.0921 │ -0.0910 │ -0.1703 │ -0.2484 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ -0.0480 │ -0.1445 │ -0.1105 │ -0.1468 │ -0.1691 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ -0.0914 │ -0.1538 │ -0.0805 │ -0.1108 │ -0.1460 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ -0.2452 │ -0.2700 │ -0.1677 │ -0.2147 │ -0.2987 │\n",
       "╰────────┴─────────┴─────────┴─────────┴─────────┴─────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">         Head 4 softmax weights   shape=(5, 5)         </span>\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">    cat </span>│<span style=\"font-weight: bold\"> chased </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">  mouse </span>│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2333 │ 0.2129 │ 0.2080 │ 0.1843 │ 0.1615 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ 0.2250 │ 0.2049 │ 0.2052 │ 0.1895 │ 0.1753 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ 0.2155 │ 0.1957 │ 0.2025 │ 0.1953 │ 0.1910 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2050 │ 0.1926 │ 0.2072 │ 0.2011 │ 0.1941 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ 0.1986 │ 0.1937 │ 0.2146 │ 0.2048 │ 0.1883 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m         Head 4 softmax weights   shape=(5, 5)         \u001b[0m\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mchased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2333 │ 0.2129 │ 0.2080 │ 0.1843 │ 0.1615 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2250 │ 0.2049 │ 0.2052 │ 0.1895 │ 0.1753 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ 0.2155 │ 0.1957 │ 0.2025 │ 0.1953 │ 0.1910 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2050 │ 0.1926 │ 0.2072 │ 0.2011 │ 0.1941 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ 0.1986 │ 0.1937 │ 0.2146 │ 0.2048 │ 0.1883 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">Softmax row sums:</span> <span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0</span><span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32mSoftmax row sums:\u001b[0m \u001b[1m[\u001b[0m\u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m, \u001b[1;36m1.0\u001b[0m\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                  Head 4 context = AttentionWeights @ V   shape=(5, 128)                  </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5407   1.3322   0.4298   -0.0589   -1.4578   0.9557   -0.0476   0.5946   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.5333   1.3339   0.4318   -0.0587   -1.4428   0.9446   -0.0534   0.5880   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.5249   1.3359   0.4340   -0.0586   -1.4258   0.9319   -0.0601   0.5807   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5212   1.3376   0.4349   -0.0599   -1.4159   0.9232   -0.0660   0.5796   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.5219   1.3385   0.4346   -0.0615   -1.4142   0.9206   -0.0689   0.5827   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                  Head 4 context = AttentionWeights @ V   shape=(5, 128)                  \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5407   1.3322   0.4298   -0.0589   -1.4578   0.9557   -0.0476   0.5946   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.5333   1.3339   0.4318   -0.0587   -1.4428   0.9446   -0.0534   0.5880   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.5249   1.3359   0.4340   -0.0586   -1.4258   0.9319   -0.0601   0.5807   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5212   1.3376   0.4349   -0.0599   -1.4159   0.9232   -0.0660   0.5796   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.5219   1.3385   0.4346   -0.0615   -1.4142   0.9206   -0.0689   0.5827   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "def scaled_dot_product_attention(Qh, Kh, Vh):\n",
    "    d_head = Qh.shape[-1]\n",
    "    scores = torch.matmul(Qh, Kh.transpose(-2, -1)) / math.sqrt(d_head)\n",
    "    attention_weights = torch.softmax(scores, dim=-1)\n",
    "    context = torch.matmul(attention_weights, Vh)\n",
    "    return context, scores, attention_weights\n",
    "\n",
    "context_heads, scores, attention_weights = scaled_dot_product_attention(Qh, Kh, Vh)\n",
    "\n",
    "console.print(Panel.fit(\"[bold]STEP 5 — ATTENTION SCORES + SOFTMAX[/bold]\", border_style=\"green\"))\n",
    "console.print(\n",
    "    f\"scores shape = {tuple(scores.shape)} = (heads, n, n)\\n\"\n",
    ")\n",
    "console.print(f\"attention_weights shape = {tuple(attention_weights.shape)}\")\n",
    "console.print(\n",
    "    f\"attention_weights shape = {tuple(attention_weights.shape)} = (heads, n, n)\\n\"\n",
    ")\n",
    "\n",
    "for h in range(NUM_HEADS):\n",
    "    console.rule(f\"Head {h + 1}: QKᵀ / √d_k\")\n",
    "    rich_attention_matrix(f\"Head {h + 1} raw attention scores\", scores[h], tokens)\n",
    "    rich_attention_matrix(f\"Head {h + 1} softmax weights\", attention_weights[h], tokens)\n",
    "    row_sums = attention_weights[h].sum(dim=-1).detach().cpu()\n",
    "    console.print(f\"[green]Softmax row sums:[/green] {[round(float(v), 6) for v in row_sums]}\")\n",
    "    rich_tensor(f\"Head {h + 1} context = AttentionWeights @ V\", context_heads[h], row_labels=tokens)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5aa9a06",
   "metadata": {},
   "source": [
    "## 7. Concatenate heads + output projection\n",
    "\n",
    "After scaled dot-product attention, each attention head produces its own context representation.\n",
    "\n",
    "For our configuration:\n",
    "\n",
    "$$\n",
    "h = 4\n",
    "$$\n",
    "\n",
    "and each head produces:\n",
    "\n",
    "$$\n",
    "\\text{Head}_i\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "So we have:\n",
    "\n",
    "$$\n",
    "\\text{Head}_1,\n",
    "\\text{Head}_2,\n",
    "\\text{Head}_3,\n",
    "\\text{Head}_4\n",
    "$$\n",
    "\n",
    "with:\n",
    "\n",
    "$$\n",
    "\\text{Head}_i\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 128}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### What does concatenating the heads mean?\n",
    "\n",
    "Each attention head can learn a different type of relationship between tokens.\n",
    "\n",
    "For example, one head may focus more on:\n",
    "\n",
    "- nearby words\n",
    "- subject–verb relationships\n",
    "- object-related information\n",
    "- semantic similarity\n",
    "- positional patterns\n",
    "\n",
    "So instead of keeping the head outputs separate, we join them together.\n",
    "\n",
    "Mathematically:\n",
    "\n",
    "$$\n",
    "\\text{Concat}\n",
    "=\n",
    "\\operatorname{Concat}\n",
    "(\n",
    "\\text{Head}_1,\n",
    "\\text{Head}_2,\n",
    "\\text{Head}_3,\n",
    "\\text{Head}_4\n",
    ")\n",
    "$$\n",
    "\n",
    "Each head has shape:\n",
    "\n",
    "$$\n",
    "[5,128]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "4 \\times [5,128]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "because:\n",
    "\n",
    "$$\n",
    "128 + 128 + 128 + 128 = 512\n",
    "$$\n",
    "\n",
    "So after concatenation:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{ConcatHeads}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Simple intuition\n",
    "\n",
    "Think of each attention head as a different specialist.\n",
    "\n",
    "For the sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "one head may focus on:\n",
    "\n",
    "> **cat ↔ chased**\n",
    "\n",
    "another may focus on:\n",
    "\n",
    "> **chased ↔ mouse**\n",
    "\n",
    "another may focus more on:\n",
    "\n",
    "> positional relationships\n",
    "\n",
    "and another may learn some different useful pattern.\n",
    "\n",
    "Each head gives us a different view of the sentence.\n",
    "\n",
    "Concatenation simply collects all of those views together:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Head 1}\n",
    "+\n",
    "\\text{Head 2}\n",
    "+\n",
    "\\text{Head 3}\n",
    "+\n",
    "\\text{Head 4}\n",
    "\\rightarrow\n",
    "\\text{Combined representation}\n",
    "}\n",
    "$$\n",
    "\n",
    "More precisely, they are not numerically added.\n",
    "\n",
    "They are joined along the feature dimension:\n",
    "\n",
    "$$\n",
    "[5,128]\n",
    "\\oplus\n",
    "[5,128]\n",
    "\\oplus\n",
    "[5,128]\n",
    "\\oplus\n",
    "[5,128]\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "where $\\oplus$ means concatenation.\n",
    "\n",
    "---\n",
    "\n",
    "### Why do we need the output projection $W_O$?\n",
    "\n",
    "After concatenation, we already have:\n",
    "\n",
    "$$\n",
    "\\text{ConcatHeads}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "But the four head outputs are still just placed side-by-side.\n",
    "\n",
    "We now want the model to learn how to combine and mix information from all heads.\n",
    "\n",
    "So we apply one trainable output projection:\n",
    "\n",
    "$$\n",
    "W_O\n",
    "\\in\n",
    "\\mathbb{R}^{512 \\times 512}\n",
    "$$\n",
    "\n",
    "The Multi-Head Attention output becomes:\n",
    "\n",
    "$$\n",
    "\\text{MHAOutput}\n",
    "=\n",
    "\\text{ConcatHeads}W_O + b_O\n",
    "$$\n",
    "\n",
    "Dimensions:\n",
    "\n",
    "$$\n",
    "(5 \\times 512)\n",
    "(512 \\times 512)\n",
    "\\rightarrow\n",
    "(5 \\times 512)\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{MHAOutput}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### What does $W_O$ actually do?\n",
    "\n",
    "$W_O$ allows information from different heads to interact.\n",
    "\n",
    "Before $W_O$:\n",
    "\n",
    "$$\n",
    "[\n",
    "\\text{Head}_1\n",
    "\\mid\n",
    "\\text{Head}_2\n",
    "\\mid\n",
    "\\text{Head}_3\n",
    "\\mid\n",
    "\\text{Head}_4\n",
    "]\n",
    "$$\n",
    "\n",
    "The information is grouped by head.\n",
    "\n",
    "After multiplying by $W_O$, the model can learn combinations such as:\n",
    "\n",
    "> \"Use some information from Head 1, some from Head 2, and some from Head 4 to create this output feature.\"\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Concatenation}\n",
    "\\rightarrow\n",
    "\\text{puts head information together}\n",
    "}\n",
    "$$\n",
    "\n",
    "while:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "W_O\n",
    "\\rightarrow\n",
    "\\text{learns how to mix that information}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Why keep the output dimension at 512?\n",
    "\n",
    "The encoder expects the representation size to remain:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "This is important because the next step uses a residual connection:\n",
    "\n",
    "$$\n",
    "X_0 + \\text{MHAOutput}\n",
    "$$\n",
    "\n",
    "Here:\n",
    "\n",
    "$$\n",
    "X_0\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\text{MHAOutput}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Because both shapes are identical, they can be added element-wise.\n",
    "\n",
    "So keeping the output dimension at 512 is what makes the residual connection possible.\n",
    "\n",
    "---\n",
    "\n",
    "### Complete flow\n",
    "\n",
    "The full Multi-Head Attention combination step is:\n",
    "\n",
    "$$\n",
    "\\text{Head}_1\n",
    "\\quad\n",
    "\\text{Head}_2\n",
    "\\quad\n",
    "\\text{Head}_3\n",
    "\\quad\n",
    "\\text{Head}_4\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{Concat}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "W_O\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{MHAOutput}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "In one equation:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{MultiHead}(Q,K,V)\n",
    "=\n",
    "\\operatorname{Concat}\n",
    "(\n",
    "\\text{Head}_1,\n",
    "\\text{Head}_2,\n",
    "\\text{Head}_3,\n",
    "\\text{Head}_4\n",
    ")\n",
    "W_O\n",
    "}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7a7bec3f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:23.133513Z",
     "iopub.status.busy": "2026-10-03T18:57:23.133411Z",
     "iopub.status.idle": "2026-10-03T18:57:23.147223Z",
     "shell.execute_reply": "2026-10-03T18:57:23.146932Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭──────────────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">STEP 6 — CONCATENATE HEADS + W_O</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰──────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭──────────────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1mSTEP 6 — CONCATENATE HEADS + W_O\u001b[0m \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰──────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Concat</span>  <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">128</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mConcat\u001b[0m  \u001b[1m(\u001b[0m\u001b[1;36m4\u001b[0m, \u001b[1;36m5\u001b[0m, \u001b[1;36m128\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Output projection</span>  <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> @ W_O <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mOutput projection\u001b[0m  \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m @ W_O \u001b[1m(\u001b[0m\u001b[1;36m512\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                           Concatenated heads   shape=(5, 512)                           </span>\n",
       "                                                                                         \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.6121   0.2646   0.5744   0.5307   -0.5511   -0.7378   0.7998   0.5980   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.6128   0.2841   0.5848   0.5333   -0.5348   -0.7309   0.7890   0.5793   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.6141   0.2977   0.5936   0.5364   -0.5218   -0.7258   0.7810   0.5652   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.6140   0.2984   0.5937   0.5362   -0.5215   -0.7255   0.7806   0.5647   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.6129   0.2817   0.5840   0.5334   -0.5363   -0.7316   0.7902   0.5811   …  \n",
       "                                                                                         \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                           Concatenated heads   shape=(5, 512)                           \u001b[0m\n",
       "                                                                                         \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.6121   0.2646   0.5744   0.5307   -0.5511   -0.7378   0.7998   0.5980   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.6128   0.2841   0.5848   0.5333   -0.5348   -0.7309   0.7890   0.5793   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.6141   0.2977   0.5936   0.5364   -0.5218   -0.7258   0.7810   0.5652   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.6140   0.2984   0.5937   0.5362   -0.5215   -0.7255   0.7806   0.5647   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.6129   0.2817   0.5840   0.5334   -0.5363   -0.7316   0.7902   0.5811   …  \n",
       "                                                                                         \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                      Multi-head attention output   shape=(5, 512)                       </span>\n",
       "                                                                                         \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.0564   1.0902   -0.0998   -1.2131   -0.0073   0.0742   0.4499   1.1769   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  0.0678   1.0987   -0.0899   -1.2076   -0.0083   0.0942   0.4382   1.1682   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  0.0767   1.1084   -0.0805   -1.2017   -0.0044   0.1094   0.4319   1.1603   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.0773   1.1152   -0.0753   -1.2001   -0.0004   0.1108   0.4355   1.1584   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  0.0652   1.1165   -0.0804   -1.1998    0.0030   0.0933   0.4483   1.1676   …  \n",
       "                                                                                         \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                      Multi-head attention output   shape=(5, 512)                       \u001b[0m\n",
       "                                                                                         \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.0564   1.0902   -0.0998   -1.2131   -0.0073   0.0742   0.4499   1.1769   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  0.0678   1.0987   -0.0899   -1.2076   -0.0083   0.0942   0.4382   1.1682   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  0.0767   1.1084   -0.0805   -1.2017   -0.0044   0.1094   0.4319   1.1603   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.0773   1.1152   -0.0753   -1.2001   -0.0004   0.1108   0.4355   1.1584   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  0.0652   1.1165   -0.0804   -1.1998    0.0030   0.0933   0.4483   1.1676   …  \n",
       "                                                                                         \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "WO = ManualLinear(D_MODEL, D_MODEL).to(device)\n",
    "\n",
    "concatenated = combine_heads(context_heads)\n",
    "attention_output = WO(concatenated)\n",
    "\n",
    "console.print(Panel.fit(\"[bold]STEP 6 — CONCATENATE HEADS + W_O[/bold]\", border_style=\"green\"))\n",
    "print_shape_equation(\"Concat\", f\"{tuple(context_heads.shape)} -> {tuple(concatenated.shape)}\")\n",
    "print_shape_equation(\"Output projection\", f\"{tuple(concatenated.shape)} @ W_O {(D_MODEL, D_MODEL)} -> {tuple(attention_output.shape)}\")\n",
    "rich_tensor(\"Concatenated heads\", concatenated, row_labels=tokens)\n",
    "rich_tensor(\"Multi-head attention output\", attention_output, row_labels=tokens)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99ab3716",
   "metadata": {},
   "source": [
    "## 8. Residual connection + LayerNorm\n",
    "\n",
    "The classic 2017 Transformer encoder uses **post-norm**.\n",
    "\n",
    "After Multi-Head Attention, we do:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_0 + \\operatorname{Attention}(X_0)\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $X_0$ = original encoder input\n",
    "- $\\operatorname{Attention}(X_0)$ = output from Multi-Head Attention\n",
    "- $X_0 + \\operatorname{Attention}(X_0)$ = residual connection\n",
    "- LayerNorm normalizes the result\n",
    "\n",
    "---\n",
    "\n",
    "### Step 1: Residual connection\n",
    "\n",
    "Suppose:\n",
    "\n",
    "$$\n",
    "X_0\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "and the attention output is:\n",
    "\n",
    "$$\n",
    "X_{\\text{attention}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Because both have the same shape, we can add them element-wise:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual}}\n",
    "=\n",
    "X_0 + X_{\\text{attention}}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "The residual connection helps preserve the original token representation while adding the contextual information learned by attention.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "\\text{Original information}\n",
    "+\n",
    "\\text{New attention information}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Step 2: Layer Normalization\n",
    "\n",
    "After the residual addition, LayerNorm is applied:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_{\\text{residual}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "LayerNorm is computed across the last dimension.\n",
    "\n",
    "In our case:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "So for each token, LayerNorm normalizes its 512 features.\n",
    "\n",
    "---\n",
    "\n",
    "### LayerNorm formula\n",
    "\n",
    "For a token vector:\n",
    "\n",
    "$$\n",
    "x\n",
    "=\n",
    "[x_1, x_2, \\dots, x_{512}]\n",
    "$$\n",
    "\n",
    "first calculate the mean:\n",
    "\n",
    "$$\n",
    "\\mu\n",
    "=\n",
    "\\frac{1}{d_{\\text{model}}}\n",
    "\\sum_{i=1}^{d_{\\text{model}}}\n",
    "x_i\n",
    "$$\n",
    "\n",
    "Then calculate the variance:\n",
    "\n",
    "$$\n",
    "\\sigma^2\n",
    "=\n",
    "\\frac{1}{d_{\\text{model}}}\n",
    "\\sum_{i=1}^{d_{\\text{model}}}\n",
    "(x_i-\\mu)^2\n",
    "$$\n",
    "\n",
    "Then normalize:\n",
    "\n",
    "$$\n",
    "\\hat{x}\n",
    "=\n",
    "\\frac{x-\\mu}\n",
    "{\\sqrt{\\sigma^2+\\epsilon}}\n",
    "$$\n",
    "\n",
    "Finally, apply learnable scale and bias parameters:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{LayerNorm}(x)\n",
    "=\n",
    "\\gamma\n",
    "\\frac{x-\\mu}\n",
    "{\\sqrt{\\sigma^2+\\epsilon}}\n",
    "+\n",
    "\\beta\n",
    "}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $\\mu$ = mean of the 512 features\n",
    "- $\\sigma^2$ = variance of the 512 features\n",
    "- $\\epsilon$ = small value for numerical stability\n",
    "- $\\gamma$ = learnable scaling parameter\n",
    "- $\\beta$ = learnable bias parameter\n",
    "\n",
    "---\n",
    "\n",
    "### Shape does not change\n",
    "\n",
    "Before residual connection:\n",
    "\n",
    "$$\n",
    "X_0\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Attention output:\n",
    "\n",
    "$$\n",
    "X_{\\text{attention}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "After residual addition:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "After LayerNorm:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "So the complete operation is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_1\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_0 + \\operatorname{MultiHeadAttention}(X_0)\n",
    "\\right)\n",
    "}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62791ed7",
   "metadata": {},
   "source": [
    "## 8. Residual connection + LayerNorm\n",
    "\n",
    "After Multi-Head Attention, the encoder does two important operations:\n",
    "\n",
    "1. **Residual connection**\n",
    "2. **Layer Normalization**\n",
    "\n",
    "For the original Transformer encoder, the flow is:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_0 + \\operatorname{MHA}(X_0)\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $X_0$ = input to the attention block\n",
    "- $\\operatorname{MHA}(X_0)$ = Multi-Head Attention output\n",
    "- $X_0 + \\operatorname{MHA}(X_0)$ = residual connection\n",
    "- LayerNorm normalizes the combined result\n",
    "\n",
    "For our example:\n",
    "\n",
    "$$\n",
    "X_0\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\operatorname{MHA}(X_0)\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "X_0 + \\operatorname{MHA}(X_0)\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "and after LayerNorm:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Part 1: Residual Connection\n",
    "\n",
    "### What is a residual connection?\n",
    "\n",
    "A residual connection means that we keep the original input and add it back to the output produced by a layer.\n",
    "\n",
    "Instead of only using:\n",
    "\n",
    "$$\n",
    "\\text{Output}\n",
    "=\n",
    "\\operatorname{MHA}(X_0)\n",
    "$$\n",
    "\n",
    "we use:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{ResidualOutput}\n",
    "=\n",
    "X_0\n",
    "+\n",
    "\\operatorname{MHA}(X_0)\n",
    "}\n",
    "$$\n",
    "\n",
    "So the model keeps:\n",
    "\n",
    "> **the original information**\n",
    "\n",
    "and also adds:\n",
    "\n",
    "> **the new contextual information learned by attention**\n",
    "\n",
    "---\n",
    "\n",
    "### Simple intuition\n",
    "\n",
    "Suppose the original representation of the token **\"cat\"** is:\n",
    "\n",
    "$$\n",
    "X_0\n",
    "=\n",
    "[1.0,\\ 2.0,\\ 3.0]\n",
    "$$\n",
    "\n",
    "and after attention, the new contextual information is:\n",
    "\n",
    "$$\n",
    "\\operatorname{Attention}(X_0)\n",
    "=\n",
    "[0.2,\\ -0.5,\\ 1.0]\n",
    "$$\n",
    "\n",
    "With a residual connection:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual}}\n",
    "=\n",
    "X_0\n",
    "+\n",
    "\\operatorname{Attention}(X_0)\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "[1.0,\\ 2.0,\\ 3.0]\n",
    "+\n",
    "[0.2,\\ -0.5,\\ 1.0]\n",
    "$$\n",
    "\n",
    "becomes:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[1.2,\\ 1.5,\\ 4.0]\n",
    "}\n",
    "$$\n",
    "\n",
    "The original representation is not thrown away.\n",
    "\n",
    "Instead, the attention block contributes an **update** to the original representation.\n",
    "\n",
    "---\n",
    "\n",
    "### A useful way to think about residuals\n",
    "\n",
    "You can think of the attention layer as learning:\n",
    "\n",
    "> **\"What should I change or add to the current representation?\"**\n",
    "\n",
    "rather than having to recreate the entire representation from scratch.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{New Representation}\n",
    "=\n",
    "\\text{Old Representation}\n",
    "+\n",
    "\\text{Learned Update}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Why do we use residual connections?\n",
    "\n",
    "Residual connections provide several important benefits.\n",
    "\n",
    "#### 1. Preserve original information\n",
    "\n",
    "Without a residual connection:\n",
    "\n",
    "$$\n",
    "X_0\n",
    "\\rightarrow\n",
    "\\operatorname{Attention}(X_0)\n",
    "$$\n",
    "\n",
    "the original representation is replaced by the attention output.\n",
    "\n",
    "With a residual connection:\n",
    "\n",
    "$$\n",
    "X_0\n",
    "+\n",
    "\\operatorname{Attention}(X_0)\n",
    "$$\n",
    "\n",
    "the model keeps access to the original token information.\n",
    "\n",
    "---\n",
    "\n",
    "#### 2. Help information flow through deep networks\n",
    "\n",
    "Transformers can contain many encoder layers.\n",
    "\n",
    "For example:\n",
    "\n",
    "$$\n",
    "X\n",
    "\\rightarrow\n",
    "Encoder_1\n",
    "\\rightarrow\n",
    "Encoder_2\n",
    "\\rightarrow\n",
    "Encoder_3\n",
    "\\rightarrow\n",
    "\\dots\n",
    "$$\n",
    "\n",
    "If every layer completely replaced its input, information could become harder to preserve across many layers.\n",
    "\n",
    "Residual connections create a more direct path:\n",
    "\n",
    "$$\n",
    "X\n",
    "\\rightarrow\n",
    "X + F(X)\n",
    "$$\n",
    "\n",
    "This helps information move through deep networks.\n",
    "\n",
    "---\n",
    "\n",
    "#### 3. Improve gradient flow during training\n",
    "\n",
    "During backpropagation, gradients need to travel through many layers.\n",
    "\n",
    "Residual paths provide an easier route for gradients.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "X\n",
    "\\rightarrow\n",
    "X + F(X)\n",
    "$$\n",
    "\n",
    "contains a direct identity path from $X$ to the output.\n",
    "\n",
    "This helps reduce optimization problems such as very weak gradients in deep networks.\n",
    "\n",
    "---\n",
    "\n",
    "#### 4. Makes each layer learn an update\n",
    "\n",
    "Instead of learning:\n",
    "\n",
    "> \"Create a completely new representation.\"\n",
    "\n",
    "the layer can learn:\n",
    "\n",
    "> \"What useful information should I add to the current representation?\"\n",
    "\n",
    "This often makes deep networks easier to optimize.\n",
    "\n",
    "---\n",
    "\n",
    "### Residual connection in our encoder\n",
    "\n",
    "Before attention:\n",
    "\n",
    "$$\n",
    "X_0\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "After Multi-Head Attention:\n",
    "\n",
    "$$\n",
    "X_{\\text{attention}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Residual addition:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual}}\n",
    "=\n",
    "X_0\n",
    "+\n",
    "X_{\\text{attention}}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_{\\text{residual}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "Notice that the shape does not change.\n",
    "\n",
    "---\n",
    "\n",
    "# Part 2: Layer Normalization\n",
    "\n",
    "### What is Layer Normalization?\n",
    "\n",
    "After the residual connection, the values inside each token representation may have very different scales.\n",
    "\n",
    "LayerNorm helps keep these values in a more stable range.\n",
    "\n",
    "For each token, LayerNorm normalizes across its feature dimension.\n",
    "\n",
    "For our encoder:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "So every token has:\n",
    "\n",
    "$$\n",
    "512\n",
    "$$\n",
    "\n",
    "features.\n",
    "\n",
    "LayerNorm normalizes those 512 values independently for each token.\n",
    "\n",
    "---\n",
    "\n",
    "### Simple example\n",
    "\n",
    "Imagine one token has this small 4-dimensional representation:\n",
    "\n",
    "$$\n",
    "x\n",
    "=\n",
    "[2,\\ 4,\\ 6,\\ 8]\n",
    "$$\n",
    "\n",
    "First calculate the mean:\n",
    "\n",
    "$$\n",
    "\\mu\n",
    "=\n",
    "\\frac{2+4+6+8}{4}\n",
    "=\n",
    "5\n",
    "$$\n",
    "\n",
    "Then subtract the mean:\n",
    "\n",
    "$$\n",
    "x-\\mu\n",
    "=\n",
    "[-3,\\ -1,\\ 1,\\ 3]\n",
    "$$\n",
    "\n",
    "Then calculate the variance and standard deviation.\n",
    "\n",
    "The values are then normalized using:\n",
    "\n",
    "$$\n",
    "\\hat{x}\n",
    "=\n",
    "\\frac{x-\\mu}\n",
    "{\\sqrt{\\sigma^2+\\epsilon}}\n",
    "$$\n",
    "\n",
    "After normalization, the values become centered approximately around:\n",
    "\n",
    "$$\n",
    "0\n",
    "$$\n",
    "\n",
    "with variance approximately:\n",
    "\n",
    "$$\n",
    "1\n",
    "$$\n",
    "\n",
    "So instead of having values with arbitrary scale:\n",
    "\n",
    "$$\n",
    "[2,\\ 4,\\ 6,\\ 8]\n",
    "$$\n",
    "\n",
    "we get a more standardized representation.\n",
    "\n",
    "---\n",
    "\n",
    "### LayerNorm formula\n",
    "\n",
    "For a token representation:\n",
    "\n",
    "$$\n",
    "x\n",
    "=\n",
    "[x_1,x_2,\\dots,x_{512}]\n",
    "$$\n",
    "\n",
    "the mean is:\n",
    "\n",
    "$$\n",
    "\\mu\n",
    "=\n",
    "\\frac{1}{d_{\\text{model}}}\n",
    "\\sum_{i=1}^{d_{\\text{model}}}x_i\n",
    "$$\n",
    "\n",
    "The variance is:\n",
    "\n",
    "$$\n",
    "\\sigma^2\n",
    "=\n",
    "\\frac{1}{d_{\\text{model}}}\n",
    "\\sum_{i=1}^{d_{\\text{model}}}\n",
    "(x_i-\\mu)^2\n",
    "$$\n",
    "\n",
    "Then normalize:\n",
    "\n",
    "$$\n",
    "\\hat{x}\n",
    "=\n",
    "\\frac{x-\\mu}\n",
    "{\\sqrt{\\sigma^2+\\epsilon}}\n",
    "$$\n",
    "\n",
    "Finally, LayerNorm applies two learnable parameters:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{LayerNorm}(x)\n",
    "=\n",
    "\\gamma\n",
    "\\frac{x-\\mu}\n",
    "{\\sqrt{\\sigma^2+\\epsilon}}\n",
    "+\n",
    "\\beta\n",
    "}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $\\mu$ = mean of the features\n",
    "- $\\sigma^2$ = variance of the features\n",
    "- $\\epsilon$ = small value for numerical stability\n",
    "- $\\gamma$ = learnable scaling parameter\n",
    "- $\\beta$ = learnable bias parameter\n",
    "\n",
    "---\n",
    "\n",
    "### Why do we need $\\gamma$ and $\\beta$?\n",
    "\n",
    "If LayerNorm always forced the representation to have exactly the same normalized scale, it could be too restrictive.\n",
    "\n",
    "So the model learns:\n",
    "\n",
    "$$\n",
    "\\gamma\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\beta\n",
    "$$\n",
    "\n",
    "to adjust the normalized values when useful.\n",
    "\n",
    "You can think of it as:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Normalize first}\n",
    "\\rightarrow\n",
    "\\text{then let the model rescale and shift}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### What does LayerNorm normalize?\n",
    "\n",
    "This is very important.\n",
    "\n",
    "Suppose:\n",
    "\n",
    "$$\n",
    "X\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "We have:\n",
    "\n",
    "- 5 tokens\n",
    "- 512 features per token\n",
    "\n",
    "LayerNorm processes each token independently across its 512 features.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\text{Token 1}\n",
    "\\rightarrow\n",
    "\\text{normalize its 512 features}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\text{Token 2}\n",
    "\\rightarrow\n",
    "\\text{normalize its 512 features}\n",
    "$$\n",
    "\n",
    "and so on.\n",
    "\n",
    "It does **not** normalize across all tokens together.\n",
    "\n",
    "---\n",
    "\n",
    "### Example\n",
    "\n",
    "For:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "we have:\n",
    "\n",
    "$$\n",
    "X\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "X_{\\text{the}} \\\\\n",
    "X_{\\text{cat}} \\\\\n",
    "X_{\\text{chased}} \\\\\n",
    "X_{\\text{the}} \\\\\n",
    "X_{\\text{mouse}}\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Each row has:\n",
    "\n",
    "$$\n",
    "512\n",
    "$$\n",
    "\n",
    "features.\n",
    "\n",
    "LayerNorm independently normalizes:\n",
    "\n",
    "$$\n",
    "X_{\\text{the}}\n",
    "$$\n",
    "\n",
    "then:\n",
    "\n",
    "$$\n",
    "X_{\\text{cat}}\n",
    "$$\n",
    "\n",
    "then:\n",
    "\n",
    "$$\n",
    "X_{\\text{chased}}\n",
    "$$\n",
    "\n",
    "and so on.\n",
    "\n",
    "---\n",
    "\n",
    "### Why do we use LayerNorm?\n",
    "\n",
    "LayerNorm provides several benefits.\n",
    "\n",
    "#### 1. Keeps activations more stable\n",
    "\n",
    "Values can become very large or very small after multiple transformations.\n",
    "\n",
    "LayerNorm helps keep the feature distributions more controlled.\n",
    "\n",
    "---\n",
    "\n",
    "#### 2. Makes training more stable\n",
    "\n",
    "Stable activations make optimization easier.\n",
    "\n",
    "This helps the model train more reliably.\n",
    "\n",
    "---\n",
    "\n",
    "#### 3. Works well for variable-length sequences\n",
    "\n",
    "LayerNorm operates independently on each token.\n",
    "\n",
    "It does not depend on other examples in the batch.\n",
    "\n",
    "This is one reason LayerNorm works very well in Transformers.\n",
    "\n",
    "---\n",
    "\n",
    "#### 4. Keeps the shape unchanged\n",
    "\n",
    "Before LayerNorm:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "After LayerNorm:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "Only the values change.\n",
    "\n",
    "The dimensions remain the same.\n",
    "\n",
    "---\n",
    "\n",
    "# Residual + LayerNorm together\n",
    "\n",
    "The complete operation is:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual}}\n",
    "=\n",
    "X_0\n",
    "+\n",
    "\\operatorname{MHA}(X_0)\n",
    "$$\n",
    "\n",
    "Then:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "(\n",
    "X_{\\text{residual}}\n",
    ")\n",
    "$$\n",
    "\n",
    "Combining both:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_1\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_0\n",
    "+\n",
    "\\operatorname{MHA}(X_0)\n",
    "\\right)\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Very simple mental model\n",
    "\n",
    "You can remember Residual Connection as:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Keep old information}\n",
    "+\n",
    "\\text{Add new information}\n",
    "}\n",
    "$$\n",
    "\n",
    "and LayerNorm as:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Keep the resulting values stable}\n",
    "}\n",
    "$$\n",
    "\n",
    "So together:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Original information}\n",
    "+\n",
    "\\text{Attention update}\n",
    "\\rightarrow\n",
    "\\text{Normalize}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Complete flow\n",
    "\n",
    "$$\n",
    "X_0\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{MultiHeadAttention}(X_0)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "X_0 + \\operatorname{MHA}(X_0)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{LayerNorm}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_1 \\in \\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "This output now moves into the **Feed-Forward Network (FFN)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "61c51a34",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:23.148452Z",
     "iopub.status.busy": "2026-10-03T18:57:23.148286Z",
     "iopub.status.idle": "2026-10-03T18:57:23.165608Z",
     "shell.execute_reply": "2026-10-03T18:57:23.165318Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭─────────────────────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">STEP 7 — ATTENTION RESIDUAL + LAYERNORM</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰─────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭─────────────────────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1mSTEP 7 — ATTENTION RESIDUAL + LAYERNORM\u001b[0m \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰─────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Residual</span>  X0 <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> + attention_output <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mResidual\u001b[0m  X0 \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m + attention_output \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                           X0 + attention_output   shape=(5, 512)                           </span>\n",
       "                                                                                            \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.0734   2.0924   -0.0730   -0.1774    0.0059    1.0691    0.4750   2.2076   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>   0.9017   1.6544    0.7414   -0.6497    0.7576    0.7125    1.2421   1.8017   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>   0.9612   0.7079    0.8503   -1.5516    0.9643   -0.1688    1.3747   0.9308   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.2354   0.1275    0.1966   -2.1340    0.3556   -0.8336    0.8943   0.2881   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.6947   0.4447   -0.7699   -2.0113   -0.5496   -0.7295   -0.0135   0.2617   …  \n",
       "                                                                                            \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                           X0 + attention_output   shape=(5, 512)                           \u001b[0m\n",
       "                                                                                            \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.0734   2.0924   -0.0730   -0.1774    0.0059    1.0691    0.4750   2.2076   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m   0.9017   1.6544    0.7414   -0.6497    0.7576    0.7125    1.2421   1.8017   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m   0.9612   0.7079    0.8503   -1.5516    0.9643   -0.1688    1.3747   0.9308   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.2354   0.1275    0.1966   -2.1340    0.3556   -0.8336    0.8943   0.2881   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.6947   0.4447   -0.7699   -2.0113   -0.5496   -0.7295   -0.0135   0.2617   …  \n",
       "                                                                                            \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                    X1 = LayerNorm(X0 + attention_output)   shape=(5, 512)                    </span>\n",
       "                                                                                              \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5102    1.8409   -0.6807   -0.8023   -0.5888    0.6493   -0.0425    1.9750   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>   0.4255    1.3339    0.2320   -1.4467    0.2516    0.1971    0.8363    1.5116   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>   0.5000    0.1921    0.3652   -2.5545    0.5038   -0.8736    1.0026    0.4629   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.3457   -0.4746   -0.3920   -3.1759   -0.2022   -1.6226    0.4413   -0.2828   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -1.3766   -0.0512   -1.4640   -2.9080   -1.2077   -1.4170   -0.5841   -0.2641   …  \n",
       "                                                                                              \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                    X1 = LayerNorm(X0 + attention_output)   shape=(5, 512)                    \u001b[0m\n",
       "                                                                                              \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5102    1.8409   -0.6807   -0.8023   -0.5888    0.6493   -0.0425    1.9750   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m   0.4255    1.3339    0.2320   -1.4467    0.2516    0.1971    0.8363    1.5116   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m   0.5000    0.1921    0.3652   -2.5545    0.5038   -0.8736    1.0026    0.4629   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.3457   -0.4746   -0.3920   -3.1759   -0.2022   -1.6226    0.4413   -0.2828   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -1.3766   -0.0512   -1.4640   -2.9080   -1.2077   -1.4170   -0.5841   -0.2641   …  \n",
       "                                                                                              \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "<span style=\"font-weight: bold\">LayerNorm sanity check per token</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\u001b[1mLayerNorm sanity check per token\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">      the: <span style=\"color: #808000; text-decoration-color: #808000\">mean</span>= <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.000000</span>, <span style=\"color: #808000; text-decoration-color: #808000\">variance</span>= <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.999986</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "      the: \u001b[33mmean\u001b[0m= \u001b[1;36m0.000000\u001b[0m, \u001b[33mvariance\u001b[0m= \u001b[1;36m0.999986\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">      cat: <span style=\"color: #808000; text-decoration-color: #808000\">mean</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.000000</span>, <span style=\"color: #808000; text-decoration-color: #808000\">variance</span>= <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.999986</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "      cat: \u001b[33mmean\u001b[0m=\u001b[1;36m-0.000000\u001b[0m, \u001b[33mvariance\u001b[0m= \u001b[1;36m0.999986\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">   chased: <span style=\"color: #808000; text-decoration-color: #808000\">mean</span>= <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.000000</span>, <span style=\"color: #808000; text-decoration-color: #808000\">variance</span>= <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.999985</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "   chased: \u001b[33mmean\u001b[0m= \u001b[1;36m0.000000\u001b[0m, \u001b[33mvariance\u001b[0m= \u001b[1;36m0.999985\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">      the: <span style=\"color: #808000; text-decoration-color: #808000\">mean</span>= <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.000000</span>, <span style=\"color: #808000; text-decoration-color: #808000\">variance</span>= <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.999986</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "      the: \u001b[33mmean\u001b[0m= \u001b[1;36m0.000000\u001b[0m, \u001b[33mvariance\u001b[0m= \u001b[1;36m0.999986\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">    mouse: <span style=\"color: #808000; text-decoration-color: #808000\">mean</span>= <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.000000</span>, <span style=\"color: #808000; text-decoration-color: #808000\">variance</span>= <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.999986</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "    mouse: \u001b[33mmean\u001b[0m= \u001b[1;36m0.000000\u001b[0m, \u001b[33mvariance\u001b[0m= \u001b[1;36m0.999986\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "def residual_add(x, sublayer_output):\n",
    "    return x + sublayer_output\n",
    "\n",
    "\n",
    "def manual_layer_norm(x, gamma, beta, eps=1e-5):\n",
    "    mean = x.mean(dim=-1, keepdim=True)\n",
    "    var = x.var(dim=-1, unbiased=False, keepdim=True)\n",
    "    x_hat = (x - mean) / torch.sqrt(var + eps)\n",
    "    return gamma * x_hat + beta\n",
    "\n",
    "# Trainable LayerNorm parameters for this educational pass.\n",
    "gamma1 = nn.Parameter(torch.ones(D_MODEL, device=device))\n",
    "beta1 = nn.Parameter(torch.zeros(D_MODEL, device=device))\n",
    "\n",
    "attention_residual = residual_add(x0, attention_output)\n",
    "x1 = manual_layer_norm(attention_residual, gamma1, beta1)\n",
    "\n",
    "console.print(Panel.fit(\"[bold]STEP 7 — ATTENTION RESIDUAL + LAYERNORM[/bold]\", border_style=\"green\"))\n",
    "print_shape_equation(\"Residual\", f\"X0 {tuple(x0.shape)} + attention_output {tuple(attention_output.shape)} -> {tuple(attention_residual.shape)}\")\n",
    "rich_tensor(\"X0 + attention_output\", attention_residual, row_labels=tokens)\n",
    "rich_tensor(\"X1 = LayerNorm(X0 + attention_output)\", x1, row_labels=tokens)\n",
    "\n",
    "means = x1.mean(dim=-1).detach().cpu()\n",
    "vars_ = x1.var(dim=-1, unbiased=False).detach().cpu()\n",
    "console.print(\"\\n[bold]LayerNorm sanity check per token[/bold]\")\n",
    "for tok, m, v in zip(tokens, means, vars_):\n",
    "    console.print(f\"  {tok:>7}: mean={float(m): .6f}, variance={float(v): .6f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7361333b",
   "metadata": {},
   "source": [
    "## 9. Feed-Forward Network (FFN)\n",
    "\n",
    "After Self-Attention + Residual + LayerNorm, the representation is passed through a **Feed-Forward Network (FFN)**.\n",
    "\n",
    "The FFN is a small neural network containing:\n",
    "\n",
    "1. First Linear Layer\n",
    "2. ReLU activation\n",
    "3. Second Linear Layer\n",
    "\n",
    "The complete equation is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{FFN}(x)\n",
    "=\n",
    "\\operatorname{ReLU}(xW_1+b_1)W_2+b_2\n",
    "}\n",
    "$$\n",
    "\n",
    "For our encoder:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "d_{\\text{ff}} = 2048\n",
    "$$\n",
    "\n",
    "Therefore, the dimensional transformation is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "\\rightarrow\n",
    "512\n",
    "}\n",
    "$$\n",
    "\n",
    "For our complete sentence containing 5 tokens:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "(5 \\times 512)\n",
    "\\rightarrow\n",
    "(5 \\times 2048)\n",
    "\\rightarrow\n",
    "(5 \\times 512)\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# What is the role of the FFN?\n",
    "\n",
    "Self-Attention and FFN perform two different jobs.\n",
    "\n",
    "### Self-Attention\n",
    "\n",
    "Self-Attention allows tokens to exchange information with other tokens.\n",
    "\n",
    "For example:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "the representation of **\"chased\"** can collect information from:\n",
    "\n",
    "- cat\n",
    "- mouse\n",
    "- the\n",
    "- other relevant tokens\n",
    "\n",
    "So we can think of attention as:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Communication between tokens}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Feed-Forward Network\n",
    "\n",
    "After the tokens communicate through attention, the FFN processes the features of **each token individually**.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Attention}\n",
    "=\n",
    "\\text{mix information between tokens}\n",
    "}\n",
    "$$\n",
    "\n",
    "while:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{FFN}\n",
    "=\n",
    "\\text{transform features inside each token}\n",
    "}\n",
    "$$\n",
    "\n",
    "This distinction is very important.\n",
    "\n",
    "---\n",
    "\n",
    "# Input to the FFN\n",
    "\n",
    "From the previous section, we obtained:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "x_{\\text{the}} \\\\\n",
    "x_{\\text{cat}} \\\\\n",
    "x_{\\text{chased}} \\\\\n",
    "x_{\\text{the}} \\\\\n",
    "x_{\\text{mouse}}\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Each token is represented by:\n",
    "\n",
    "$$\n",
    "x_i \\in \\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "The **same FFN** is applied independently to every token.\n",
    "\n",
    "For example:\n",
    "\n",
    "$$\n",
    "x_{\\text{cat}}\n",
    "\\rightarrow\n",
    "\\operatorname{FFN}(x_{\\text{cat}})\n",
    "$$\n",
    "\n",
    "$$\n",
    "x_{\\text{chased}}\n",
    "\\rightarrow\n",
    "\\operatorname{FFN}(x_{\\text{chased}})\n",
    "$$\n",
    "\n",
    "$$\n",
    "x_{\\text{mouse}}\n",
    "\\rightarrow\n",
    "\\operatorname{FFN}(x_{\\text{mouse}})\n",
    "$$\n",
    "\n",
    "The FFN does not directly mix **cat** with **mouse**.\n",
    "\n",
    "That token-to-token interaction already happened in Self-Attention.\n",
    "\n",
    "---\n",
    "\n",
    "# Step 1: First Linear Layer\n",
    "\n",
    "The first operation is:\n",
    "\n",
    "$$\n",
    "Z_1\n",
    "=\n",
    "X_1W_1+b_1\n",
    "$$\n",
    "\n",
    "The input has:\n",
    "\n",
    "$$\n",
    "512\n",
    "$$\n",
    "\n",
    "features.\n",
    "\n",
    "The first Linear Layer expands it to:\n",
    "\n",
    "$$\n",
    "2048\n",
    "$$\n",
    "\n",
    "features.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "W_1\n",
    "\\in\n",
    "\\mathbb{R}^{512 \\times 2048}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "b_1\n",
    "\\in\n",
    "\\mathbb{R}^{2048}\n",
    "$$\n",
    "\n",
    "For the complete sentence:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Multiplying by:\n",
    "\n",
    "$$\n",
    "W_1\n",
    "\\in\n",
    "\\mathbb{R}^{512 \\times 2048}\n",
    "$$\n",
    "\n",
    "gives:\n",
    "\n",
    "$$\n",
    "(5 \\times 512)\n",
    "(512 \\times 2048)\n",
    "\\rightarrow\n",
    "(5 \\times 2048)\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Z_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 2048}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# What is a Linear Layer?\n",
    "\n",
    "A Linear Layer performs:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "y=xW+b\n",
    "}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $x$ = input features\n",
    "- $W$ = trainable weight matrix\n",
    "- $b$ = trainable bias\n",
    "- $y$ = transformed output\n",
    "\n",
    "For our first FFN layer:\n",
    "\n",
    "$$\n",
    "x\n",
    "\\in\n",
    "\\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "$$\n",
    "W_1\n",
    "\\in\n",
    "\\mathbb{R}^{512 \\times 2048}\n",
    "$$\n",
    "\n",
    "$$\n",
    "b_1\n",
    "\\in\n",
    "\\mathbb{R}^{2048}\n",
    "$$\n",
    "\n",
    "and therefore:\n",
    "\n",
    "$$\n",
    "y\n",
    "\\in\n",
    "\\mathbb{R}^{2048}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# What does the weight matrix $W_1$ do?\n",
    "\n",
    "$W_1$ is learned during training.\n",
    "\n",
    "It learns how different input features should be combined to create new features.\n",
    "\n",
    "Suppose, just for intuition, a token contains features related to:\n",
    "\n",
    "- object information\n",
    "- action information\n",
    "- position information\n",
    "- semantic information\n",
    "- grammatical information\n",
    "\n",
    "A Linear Layer can learn different combinations of those features.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "\\text{Old Features}\n",
    "\\xrightarrow{W_1}\n",
    "\\text{New Feature Combinations}\n",
    "$$\n",
    "\n",
    "The model does not manually define what each feature means.\n",
    "\n",
    "These useful feature combinations are learned automatically during training.\n",
    "\n",
    "---\n",
    "\n",
    "# What is the bias $b_1$?\n",
    "\n",
    "The Linear Layer is not only:\n",
    "\n",
    "$$\n",
    "xW\n",
    "$$\n",
    "\n",
    "but:\n",
    "\n",
    "$$\n",
    "xW+b\n",
    "$$\n",
    "\n",
    "The bias allows the model to shift the transformed values.\n",
    "\n",
    "A very small scalar example:\n",
    "\n",
    "Suppose:\n",
    "\n",
    "$$\n",
    "x = 2\n",
    "$$\n",
    "\n",
    "$$\n",
    "W = 3\n",
    "$$\n",
    "\n",
    "$$\n",
    "b = 1\n",
    "$$\n",
    "\n",
    "Then:\n",
    "\n",
    "$$\n",
    "y=xW+b\n",
    "$$\n",
    "\n",
    "$$\n",
    "y=(2)(3)+1\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "y=7\n",
    "}\n",
    "$$\n",
    "\n",
    "Both the weights and biases are trainable parameters.\n",
    "\n",
    "---\n",
    "\n",
    "# Why expand from 512 to 2048?\n",
    "\n",
    "The first Linear Layer increases the feature space:\n",
    "\n",
    "$$\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "$$\n",
    "\n",
    "This gives the FFN a larger internal space in which it can learn more complex transformations.\n",
    "\n",
    "A simple intuition is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "512\n",
    "\\rightarrow\n",
    "\\text{expand}\n",
    "\\rightarrow\n",
    "2048\n",
    "}\n",
    "$$\n",
    "\n",
    "The model temporarily creates many more intermediate features.\n",
    "\n",
    "It can then detect and combine richer patterns before projecting the representation back to 512 dimensions.\n",
    "\n",
    "This intermediate dimension is called:\n",
    "\n",
    "$$\n",
    "d_{\\text{ff}}\n",
    "$$\n",
    "\n",
    "For our model:\n",
    "\n",
    "$$\n",
    "d_{\\text{ff}}=2048\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Step 2: ReLU Activation\n",
    "\n",
    "After the first Linear Layer:\n",
    "\n",
    "$$\n",
    "Z_1\n",
    "=\n",
    "X_1W_1+b_1\n",
    "$$\n",
    "\n",
    "we apply:\n",
    "\n",
    "$$\n",
    "\\operatorname{ReLU}(Z_1)\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "A_1\n",
    "=\n",
    "\\operatorname{ReLU}(Z_1)\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# What is ReLU?\n",
    "\n",
    "ReLU stands for:\n",
    "\n",
    "> **Rectified Linear Unit**\n",
    "\n",
    "Its formula is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{ReLU}(x)\n",
    "=\n",
    "\\max(0,x)\n",
    "}\n",
    "$$\n",
    "\n",
    "This means:\n",
    "\n",
    "- if $x>0$, keep $x$\n",
    "- if $x\\leq0$, output $0$\n",
    "\n",
    "For example:\n",
    "\n",
    "$$\n",
    "x\n",
    "=\n",
    "[-3,-1,0,2,5]\n",
    "$$\n",
    "\n",
    "After ReLU:\n",
    "\n",
    "$$\n",
    "\\operatorname{ReLU}(x)\n",
    "=\n",
    "[0,0,0,2,5]\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[-3,-1,0,2,5]\n",
    "\\rightarrow\n",
    "[0,0,0,2,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Why do we need ReLU?\n",
    "\n",
    "This is an important point.\n",
    "\n",
    "Suppose we only used two Linear Layers:\n",
    "\n",
    "$$\n",
    "xW_1W_2\n",
    "$$\n",
    "\n",
    "Both operations are linear.\n",
    "\n",
    "Multiple Linear Layers without a nonlinear activation can effectively collapse into another linear transformation.\n",
    "\n",
    "For example:\n",
    "\n",
    "$$\n",
    "(xW_1)W_2\n",
    "=\n",
    "x(W_1W_2)\n",
    "$$\n",
    "\n",
    "So simply stacking Linear Layers would not provide the same ability to learn nonlinear relationships.\n",
    "\n",
    "ReLU introduces **non-linearity**:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Linear}\n",
    "\\rightarrow\n",
    "\\text{ReLU}\n",
    "\\rightarrow\n",
    "\\text{Linear}\n",
    "}\n",
    "$$\n",
    "\n",
    "This allows the neural network to learn much more complex transformations.\n",
    "\n",
    "---\n",
    "\n",
    "# Simple ReLU intuition\n",
    "\n",
    "Imagine the first Linear Layer produces:\n",
    "\n",
    "$$\n",
    "[-1.7,\\ 2.4,\\ -0.3,\\ 5.1]\n",
    "$$\n",
    "\n",
    "ReLU converts it to:\n",
    "\n",
    "$$\n",
    "[0,\\ 2.4,\\ 0,\\ 5.1]\n",
    "$$\n",
    "\n",
    "Negative activations are suppressed.\n",
    "\n",
    "Positive activations remain active.\n",
    "\n",
    "You can loosely think of ReLU as:\n",
    "\n",
    "> **\"Keep positively activated features and suppress negative activations.\"**\n",
    "\n",
    "---\n",
    "\n",
    "# Does ReLU change the shape?\n",
    "\n",
    "No.\n",
    "\n",
    "Before ReLU:\n",
    "\n",
    "$$\n",
    "Z_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 2048}\n",
    "$$\n",
    "\n",
    "After ReLU:\n",
    "\n",
    "$$\n",
    "A_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 2048}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "(5,2048)\n",
    "\\rightarrow\n",
    "(5,2048)\n",
    "}\n",
    "$$\n",
    "\n",
    "Only the values change.\n",
    "\n",
    "The tensor shape remains unchanged.\n",
    "\n",
    "---\n",
    "\n",
    "# Step 3: Second Linear Layer\n",
    "\n",
    "After ReLU, we have:\n",
    "\n",
    "$$\n",
    "A_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 2048}\n",
    "$$\n",
    "\n",
    "Now we apply another Linear Layer:\n",
    "\n",
    "$$\n",
    "Z_2\n",
    "=\n",
    "A_1W_2+b_2\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "W_2\n",
    "\\in\n",
    "\\mathbb{R}^{2048 \\times 512}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "b_2\n",
    "\\in\n",
    "\\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "(5 \\times 2048)\n",
    "(2048 \\times 512)\n",
    "\\rightarrow\n",
    "(5 \\times 512)\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{FFN}(X_1)\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Why reduce 2048 back to 512?\n",
    "\n",
    "The encoder's main representation size is:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}}=512\n",
    "$$\n",
    "\n",
    "So although the FFN temporarily expands the representation:\n",
    "\n",
    "$$\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "$$\n",
    "\n",
    "it projects it back:\n",
    "\n",
    "$$\n",
    "2048\n",
    "\\rightarrow\n",
    "512\n",
    "$$\n",
    "\n",
    "giving:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "\\rightarrow\n",
    "512\n",
    "}\n",
    "$$\n",
    "\n",
    "This is important because the next operation is another residual connection:\n",
    "\n",
    "$$\n",
    "X_1+\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "Both tensors must therefore have the same shape:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(X_1)\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "so they can be added element-wise.\n",
    "\n",
    "---\n",
    "\n",
    "# Complete FFN calculation\n",
    "\n",
    "The complete process is:\n",
    "\n",
    "### First Linear Layer\n",
    "\n",
    "$$\n",
    "Z_1\n",
    "=\n",
    "X_1W_1+b_1\n",
    "$$\n",
    "\n",
    "$$\n",
    "(5,512)\n",
    "\\rightarrow\n",
    "(5,2048)\n",
    "$$\n",
    "\n",
    "### ReLU\n",
    "\n",
    "$$\n",
    "A_1\n",
    "=\n",
    "\\operatorname{ReLU}(Z_1)\n",
    "$$\n",
    "\n",
    "$$\n",
    "(5,2048)\n",
    "\\rightarrow\n",
    "(5,2048)\n",
    "$$\n",
    "\n",
    "### Second Linear Layer\n",
    "\n",
    "$$\n",
    "Z_2\n",
    "=\n",
    "A_1W_2+b_2\n",
    "$$\n",
    "\n",
    "$$\n",
    "(5,2048)\n",
    "\\rightarrow\n",
    "(5,512)\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{FFN}(X_1)\n",
    "=\n",
    "\\operatorname{ReLU}(X_1W_1+b_1)W_2+b_2\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# What happens to one token?\n",
    "\n",
    "Take the token:\n",
    "\n",
    "> **\"cat\"**\n",
    "\n",
    "After Self-Attention, suppose its contextual representation is:\n",
    "\n",
    "$$\n",
    "x_{\\text{cat}}\n",
    "\\in\n",
    "\\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "The FFN performs:\n",
    "\n",
    "$$\n",
    "512\n",
    "\\xrightarrow{\\text{Linear 1}}\n",
    "2048\n",
    "$$\n",
    "\n",
    "then:\n",
    "\n",
    "$$\n",
    "2048\n",
    "\\xrightarrow{\\text{ReLU}}\n",
    "2048\n",
    "$$\n",
    "\n",
    "then:\n",
    "\n",
    "$$\n",
    "2048\n",
    "\\xrightarrow{\\text{Linear 2}}\n",
    "512\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "x_{\\text{cat}}\n",
    ":\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "\\rightarrow\n",
    "512\n",
    "}\n",
    "$$\n",
    "\n",
    "The same process happens independently for:\n",
    "\n",
    "- the\n",
    "- cat\n",
    "- chased\n",
    "- the\n",
    "- mouse\n",
    "\n",
    "---\n",
    "\n",
    "# Is the FFN different for every token?\n",
    "\n",
    "No.\n",
    "\n",
    "This is important.\n",
    "\n",
    "The **same weights**:\n",
    "\n",
    "$$\n",
    "W_1,\\ b_1,\\ W_2,\\ b_2\n",
    "$$\n",
    "\n",
    "are reused for every token position.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(x_{\\text{the}})\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(x_{\\text{cat}})\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(x_{\\text{chased}})\n",
    "$$\n",
    "\n",
    "all use exactly the same FFN parameters.\n",
    "\n",
    "However, because each token has a different contextual representation, their FFN outputs are different.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Same network}\n",
    "+\n",
    "\\text{different token inputs}\n",
    "=\n",
    "\\text{different outputs}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Attention vs FFN\n",
    "\n",
    "This is one of the most important Transformer concepts.\n",
    "\n",
    "### Multi-Head Self-Attention\n",
    "\n",
    "Answers:\n",
    "\n",
    "> **\"What information should this token collect from other tokens?\"**\n",
    "\n",
    "It operates across the **sequence/token dimension**.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Token mixing}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Feed-Forward Network\n",
    "\n",
    "Answers:\n",
    "\n",
    "> **\"Now that I have contextual information, how should I transform my own features?\"**\n",
    "\n",
    "It operates independently on each token.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Feature transformation}\n",
    "}\n",
    "$$\n",
    "\n",
    "So a useful mental model is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Attention}\n",
    "=\n",
    "\\text{communication between tokens}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{FFN}\n",
    "=\n",
    "\\text{computation within each token}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Why is the FFN important?\n",
    "\n",
    "The FFN provides several important capabilities.\n",
    "\n",
    "### 1. Adds non-linearity\n",
    "\n",
    "ReLU allows the model to learn relationships that cannot be represented using only Linear Layers.\n",
    "\n",
    "$$\n",
    "\\text{Linear}\n",
    "\\rightarrow\n",
    "\\text{ReLU}\n",
    "\\rightarrow\n",
    "\\text{Linear}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 2. Expands the feature space\n",
    "\n",
    "The representation temporarily increases:\n",
    "\n",
    "$$\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "$$\n",
    "\n",
    "giving the network more capacity to construct useful intermediate features.\n",
    "\n",
    "---\n",
    "\n",
    "### 3. Transforms contextual information\n",
    "\n",
    "Self-Attention gathers information from other tokens.\n",
    "\n",
    "The FFN then processes that information and produces a richer representation for each token.\n",
    "\n",
    "---\n",
    "\n",
    "### 4. Keeps token positions independent\n",
    "\n",
    "Unlike attention, the FFN does not exchange information between positions.\n",
    "\n",
    "Every token is processed independently using the same learned network.\n",
    "\n",
    "---\n",
    "\n",
    "### 5. Preserves $d_{\\text{model}}$\n",
    "\n",
    "The final FFN output returns to:\n",
    "\n",
    "$$\n",
    "512\n",
    "$$\n",
    "\n",
    "dimensions so it remains compatible with the rest of the Transformer architecture.\n",
    "\n",
    "---\n",
    "\n",
    "# Complete dimensional flow\n",
    "\n",
    "For our complete sentence:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "First Linear Layer:\n",
    "\n",
    "$$\n",
    "(5,512)\n",
    "\\xrightarrow{W_1}\n",
    "(5,2048)\n",
    "$$\n",
    "\n",
    "ReLU:\n",
    "\n",
    "$$\n",
    "(5,2048)\n",
    "\\xrightarrow{\\operatorname{ReLU}}\n",
    "(5,2048)\n",
    "$$\n",
    "\n",
    "Second Linear Layer:\n",
    "\n",
    "$$\n",
    "(5,2048)\n",
    "\\xrightarrow{W_2}\n",
    "(5,512)\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "(5,512)\n",
    "\\rightarrow\n",
    "(5,2048)\n",
    "\\rightarrow\n",
    "(5,2048)\n",
    "\\rightarrow\n",
    "(5,512)\n",
    "}\n",
    "$$\n",
    "\n",
    "Or more simply:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "\\rightarrow\n",
    "512\n",
    "}\n",
    "$$\n",
    "\n",
    "for every token.\n",
    "\n",
    "---\n",
    "\n",
    "# Simple mental model\n",
    "\n",
    "Think of the FFN as a small processing factory inside every encoder block:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "512\\text{ features}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Linear 1: expand features}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "2048\\text{ features}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{ReLU: introduce non-linearity}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Linear 2: combine and compress features}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "512\\text{ transformed features}\n",
    "}\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{FFN}\n",
    "=\n",
    "\\text{Expand}\n",
    "\\rightarrow\n",
    "\\text{Activate}\n",
    "\\rightarrow\n",
    "\\text{Project Back}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Where does the FFN output go next?\n",
    "\n",
    "The FFN produces:\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(X_1)\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "The encoder then performs the **second residual connection**:\n",
    "\n",
    "$$\n",
    "X_1+\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "followed by LayerNorm:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{EncoderOutput}\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_1+\\operatorname{FFN}(X_1)\n",
    "\\right)\n",
    "}\n",
    "$$\n",
    "\n",
    "This gives the final output of one encoder block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "774d0088",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:23.167182Z",
     "iopub.status.busy": "2026-10-03T18:57:23.167025Z",
     "iopub.status.idle": "2026-10-03T18:57:23.191167Z",
     "shell.execute_reply": "2026-10-03T18:57:23.190814Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭───────────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">STEP 8 — FEED-FORWARD NETWORK</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰───────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭───────────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1mSTEP 8 — FEED-FORWARD NETWORK\u001b[0m \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰───────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Linear </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">1</span>  <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2048</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mLinear \u001b[0m\u001b[1;35m1\u001b[0m  \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m2048\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ReLU</span>  <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2048</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2048</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mReLU\u001b[0m  \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m2048\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m2048\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Linear </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">2</span>  <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2048</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mLinear \u001b[0m\u001b[1;35m2\u001b[0m  \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m2048\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                       FFN hidden after ReLU   shape=(5, 2048)                        </span>\n",
       "                                                                                      \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">      4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.0000   0.3149   0.0000   0.2961   0.0000   0.0000   0.7615   1.3679   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  0.0000   0.2076   0.0000   0.3593   0.0000   0.0000   0.5065   1.5295   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  0.0000   0.1231   0.0000   0.4952   0.0000   0.0000   0.2215   1.6012   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.0000   0.2199   0.0000   0.6805   0.0000   0.0000   0.0832   1.4915   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  0.0000   0.4194   0.0000   0.8200   0.0000   0.0000   0.1255   1.3570   …  \n",
       "                                                                                      \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                       FFN hidden after ReLU   shape=(5, 2048)                        \u001b[0m\n",
       "                                                                                      \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.0000   0.3149   0.0000   0.2961   0.0000   0.0000   0.7615   1.3679   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  0.0000   0.2076   0.0000   0.3593   0.0000   0.0000   0.5065   1.5295   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  0.0000   0.1231   0.0000   0.4952   0.0000   0.0000   0.2215   1.6012   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.0000   0.2199   0.0000   0.6805   0.0000   0.0000   0.0832   1.4915   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  0.0000   0.4194   0.0000   0.8200   0.0000   0.0000   0.1255   1.3570   …  \n",
       "                                                                                      \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                               FFN output   shape=(5, 512)                               </span>\n",
       "                                                                                         \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">      4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5075   0.5431   -0.4462    0.0692   0.6841   0.9204   1.8464   0.4487   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.3622   0.5667   -0.3646    0.1237   0.7159   0.8356   1.7501   0.5502   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.1602   0.4866   -0.3187   -0.0150   0.8043   0.8674   1.5251   0.7225   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.1811   0.3544   -0.2297   -0.2381   0.8927   0.9177   1.2827   0.7620   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.3319   0.2235   -0.1831   -0.3805   0.8718   0.8498   1.0886   0.7059   …  \n",
       "                                                                                         \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                               FFN output   shape=(5, 512)                               \u001b[0m\n",
       "                                                                                         \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5075   0.5431   -0.4462    0.0692   0.6841   0.9204   1.8464   0.4487   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.3622   0.5667   -0.3646    0.1237   0.7159   0.8356   1.7501   0.5502   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.1602   0.4866   -0.3187   -0.0150   0.8043   0.8674   1.5251   0.7225   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.1811   0.3544   -0.2297   -0.2381   0.8927   0.9177   1.2827   0.7620   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.3319   0.2235   -0.1831   -0.3805   0.8718   0.8498   1.0886   0.7059   …  \n",
       "                                                                                         \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "class ManualFeedForward(nn.Module):\n",
    "    def __init__(self, d_model, d_ff):\n",
    "        super().__init__()\n",
    "        self.fc1 = ManualLinear(d_model, d_ff)\n",
    "        self.fc2 = ManualLinear(d_ff, d_model)\n",
    "\n",
    "    def forward(self, x, return_hidden=False):\n",
    "        hidden_pre_activation = self.fc1(x)\n",
    "        hidden = torch.relu(hidden_pre_activation)\n",
    "        output = self.fc2(hidden)\n",
    "        if return_hidden:\n",
    "            return output, hidden_pre_activation, hidden\n",
    "        return output\n",
    "\n",
    "ffn = ManualFeedForward(D_MODEL, D_FF).to(device)\n",
    "ffn_output, ffn_pre, ffn_hidden = ffn(x1, return_hidden=True)\n",
    "\n",
    "console.print(Panel.fit(\"[bold]STEP 8 — FEED-FORWARD NETWORK[/bold]\", border_style=\"green\"))\n",
    "print_shape_equation(\"Linear 1\", f\"{tuple(x1.shape)} -> {tuple(ffn_pre.shape)}\")\n",
    "print_shape_equation(\"ReLU\", f\"{tuple(ffn_pre.shape)} -> {tuple(ffn_hidden.shape)}\")\n",
    "print_shape_equation(\"Linear 2\", f\"{tuple(ffn_hidden.shape)} -> {tuple(ffn_output.shape)}\")\n",
    "rich_tensor(\"FFN hidden after ReLU\", ffn_hidden, row_labels=tokens)\n",
    "rich_tensor(\"FFN output\", ffn_output, row_labels=tokens)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8b020cb",
   "metadata": {},
   "source": [
    "## 10. Second residual + LayerNorm = final encoder-block output\n",
    "\n",
    "The Feed-Forward Network also gets its own:\n",
    "\n",
    "1. **Residual connection**\n",
    "2. **Layer Normalization**\n",
    "\n",
    "This is the second Add & Norm operation inside one Transformer encoder block.\n",
    "\n",
    "The complete operation is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{EncoderOutput}\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_1 + \\operatorname{FFN}(X_1)\n",
    "\\right)\n",
    "}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $X_1$ = output from the first Residual + LayerNorm\n",
    "- $\\operatorname{FFN}(X_1)$ = transformed features produced by the Feed-Forward Network\n",
    "- $X_1 + \\operatorname{FFN}(X_1)$ = second residual connection\n",
    "- LayerNorm normalizes the combined representation\n",
    "\n",
    "---\n",
    "\n",
    "# What comes into this step?\n",
    "\n",
    "From the previous Attention + Residual + LayerNorm step, we have:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Then the Feed-Forward Network processes every token independently:\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(X_1)\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "So both tensors have exactly the same shape:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(X_1)\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "Because their shapes match, they can be added element-wise.\n",
    "\n",
    "---\n",
    "\n",
    "# Step 1: Second residual connection\n",
    "\n",
    "The residual operation is:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual-2}}\n",
    "=\n",
    "X_1\n",
    "+\n",
    "\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "So conceptually:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Previous Representation}\n",
    "+\n",
    "\\text{New FFN Transformation}\n",
    "}\n",
    "$$\n",
    "\n",
    "The FFN does not completely replace $X_1$.\n",
    "\n",
    "Instead, it contributes a learned update to it.\n",
    "\n",
    "---\n",
    "\n",
    "# Simple example\n",
    "\n",
    "Suppose one token currently has:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "[1.0,\\ 2.0,\\ 3.0]\n",
    "$$\n",
    "\n",
    "and the FFN produces:\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(X_1)\n",
    "=\n",
    "[0.4,\\ -0.2,\\ 0.8]\n",
    "$$\n",
    "\n",
    "Then:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual-2}}\n",
    "=\n",
    "[1.0,\\ 2.0,\\ 3.0]\n",
    "+\n",
    "[0.4,\\ -0.2,\\ 0.8]\n",
    "$$\n",
    "\n",
    "giving:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[1.4,\\ 1.8,\\ 3.8]\n",
    "}\n",
    "$$\n",
    "\n",
    "So the previous representation is preserved while the FFN contributes new information.\n",
    "\n",
    "---\n",
    "\n",
    "# Why do we need another residual connection?\n",
    "\n",
    "We already used a residual connection around Multi-Head Attention.\n",
    "\n",
    "Now we use another residual connection around the FFN.\n",
    "\n",
    "These two sublayers perform different jobs:\n",
    "\n",
    "### Attention sublayer\n",
    "\n",
    "$$\n",
    "X_0\n",
    "\\rightarrow\n",
    "\\operatorname{MHA}(X_0)\n",
    "$$\n",
    "\n",
    "Attention gathers information from other tokens.\n",
    "\n",
    "Residual:\n",
    "\n",
    "$$\n",
    "X_0+\\operatorname{MHA}(X_0)\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### FFN sublayer\n",
    "\n",
    "$$\n",
    "X_1\n",
    "\\rightarrow\n",
    "\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "The FFN transforms features inside each token.\n",
    "\n",
    "Residual:\n",
    "\n",
    "$$\n",
    "X_1+\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "So each major transformation in the encoder has its own residual path.\n",
    "\n",
    "---\n",
    "\n",
    "# Why is this useful?\n",
    "\n",
    "### 1. Preserves the representation from the attention block\n",
    "\n",
    "The FFN may significantly transform the features.\n",
    "\n",
    "The residual path keeps the representation learned by the attention stage available.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Attention representation}\n",
    "+\n",
    "\\text{FFN update}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 2. Helps deep Transformer networks train\n",
    "\n",
    "Transformers often stack many encoder blocks:\n",
    "\n",
    "$$\n",
    "Encoder_1\n",
    "\\rightarrow\n",
    "Encoder_2\n",
    "\\rightarrow\n",
    "Encoder_3\n",
    "\\rightarrow\n",
    "\\dots\n",
    "$$\n",
    "\n",
    "Residual paths help information and gradients flow through these deep networks.\n",
    "\n",
    "---\n",
    "\n",
    "### 3. FFN only needs to learn an improvement\n",
    "\n",
    "Instead of learning a completely new representation, the FFN can learn:\n",
    "\n",
    "> **\"What useful transformation should I add?\"**\n",
    "\n",
    "So we can think of:\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "as a learned update.\n",
    "\n",
    "---\n",
    "\n",
    "# Step 2: Apply LayerNorm\n",
    "\n",
    "After the residual addition:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual-2}}\n",
    "=\n",
    "X_1+\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "we apply Layer Normalization:\n",
    "\n",
    "$$\n",
    "X_2\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_{\\text{residual-2}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_2\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_1+\\operatorname{FFN}(X_1)\n",
    "\\right)\n",
    "}\n",
    "$$\n",
    "\n",
    "This $X_2$ is the final output of one encoder block.\n",
    "\n",
    "We can also write:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{EncoderOutput}\n",
    "=\n",
    "X_2\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Why apply LayerNorm again?\n",
    "\n",
    "The FFN changes the feature values.\n",
    "\n",
    "Then the residual addition combines:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "+\n",
    "\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "After this addition, the resulting feature values may have different magnitudes and distributions.\n",
    "\n",
    "LayerNorm helps keep these values well-behaved.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Residual Addition}\n",
    "\\rightarrow\n",
    "\\text{Normalize Features}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# What does LayerNorm normalize here?\n",
    "\n",
    "The encoder output has:\n",
    "\n",
    "$$\n",
    "5\n",
    "$$\n",
    "\n",
    "tokens and:\n",
    "\n",
    "$$\n",
    "512\n",
    "$$\n",
    "\n",
    "features per token.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual-2}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "LayerNorm independently normalizes the 512 features of each token.\n",
    "\n",
    "For example:\n",
    "\n",
    "$$\n",
    "X_{\\text{cat}}\n",
    "\\in\n",
    "\\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "gets its own normalization.\n",
    "\n",
    "Similarly:\n",
    "\n",
    "$$\n",
    "X_{\\text{chased}}\n",
    "\\in\n",
    "\\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "gets its own normalization.\n",
    "\n",
    "The tokens are **not normalized together**.\n",
    "\n",
    "---\n",
    "\n",
    "# Does the shape change?\n",
    "\n",
    "No.\n",
    "\n",
    "Before the second residual connection:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "FFN output:\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(X_1)\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "After residual:\n",
    "\n",
    "$$\n",
    "X_{\\text{residual-2}}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "After LayerNorm:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{EncoderOutput}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "(n,d_{\\text{model}})\n",
    "=\n",
    "(5,512)\n",
    "}\n",
    "$$\n",
    "\n",
    "The shape stays the same.\n",
    "\n",
    "Only the information stored inside each token representation changes.\n",
    "\n",
    "---\n",
    "\n",
    "# What does the final encoder output contain?\n",
    "\n",
    "At the beginning, each token mainly contained:\n",
    "\n",
    "$$\n",
    "\\text{Token Embedding}\n",
    "+\n",
    "\\text{Position Information}\n",
    "$$\n",
    "\n",
    "But after passing through:\n",
    "\n",
    "$$\n",
    "\\text{Self-Attention}\n",
    "$$\n",
    "\n",
    "$$\n",
    "+\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\text{Feed-Forward Network}\n",
    "$$\n",
    "\n",
    "the representation becomes much richer.\n",
    "\n",
    "For every token, the final encoder representation now contains information influenced by:\n",
    "\n",
    "- the token itself\n",
    "- its position\n",
    "- other tokens in the sentence\n",
    "- attention relationships\n",
    "- multiple attention heads\n",
    "- transformed nonlinear features from the FFN\n",
    "\n",
    "---\n",
    "\n",
    "# Example: representation of \"chased\"\n",
    "\n",
    "Initially:\n",
    "\n",
    "> **\"chased\"**\n",
    "\n",
    "has mainly its token embedding and positional information.\n",
    "\n",
    "After Self-Attention, its representation may incorporate information from:\n",
    "\n",
    "> **cat**\n",
    "\n",
    "and:\n",
    "\n",
    "> **mouse**\n",
    "\n",
    "because those tokens may be relevant to understanding the action.\n",
    "\n",
    "Then the FFN transforms those contextual features.\n",
    "\n",
    "So after the complete encoder block:\n",
    "\n",
    "$$\n",
    "X_{\\text{chased}}^{\\text{final}}\n",
    "$$\n",
    "\n",
    "is no longer just a representation of the isolated word:\n",
    "\n",
    "> **\"chased\"**\n",
    "\n",
    "It is now a **contextual representation of \"chased\" inside this particular sentence**.\n",
    "\n",
    "That is a major purpose of the encoder.\n",
    "\n",
    "---\n",
    "\n",
    "# Important: contextual representation\n",
    "\n",
    "Consider the word:\n",
    "\n",
    "> **\"bank\"**\n",
    "\n",
    "in:\n",
    "\n",
    "> **\"I deposited money in the bank.\"**\n",
    "\n",
    "and:\n",
    "\n",
    "> **\"I sat on the river bank.\"**\n",
    "\n",
    "The initial token embedding for **bank** may start from the same learned embedding.\n",
    "\n",
    "But after passing through the encoder, its contextual representation can become different because it attends to different surrounding tokens.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Same token}\n",
    "+\n",
    "\\text{Different context}\n",
    "\\rightarrow\n",
    "\\text{Different contextual representation}\n",
    "}\n",
    "$$\n",
    "\n",
    "This is one of the most important ideas behind Transformer encoders.\n",
    "\n",
    "---\n",
    "\n",
    "# Complete encoder block flow\n",
    "\n",
    "The complete flow of one encoder layer is now:\n",
    "\n",
    "$$\n",
    "X_0\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\text{Multi-Head Self-Attention}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "X_0+\\operatorname{MHA}(X_0)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{LayerNorm}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "X_1\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "X_1+\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\operatorname{LayerNorm}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{EncoderOutput}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Full mathematical view\n",
    "\n",
    "### Attention sublayer\n",
    "\n",
    "$$\n",
    "A\n",
    "=\n",
    "\\operatorname{MHA}(X_0)\n",
    "$$\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_0+A\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "### Feed-Forward sublayer\n",
    "\n",
    "$$\n",
    "F\n",
    "=\n",
    "\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_2\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "\\left(\n",
    "X_1+F\n",
    "\\right)\n",
    "}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_2\n",
    "=\n",
    "\\operatorname{EncoderOutput}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# What happens if we have multiple encoder layers?\n",
    "\n",
    "Usually, we do not use just one encoder block.\n",
    "\n",
    "We can stack many encoder blocks:\n",
    "\n",
    "$$\n",
    "X\n",
    "\\rightarrow\n",
    "Encoder_1\n",
    "\\rightarrow\n",
    "Encoder_2\n",
    "\\rightarrow\n",
    "Encoder_3\n",
    "\\rightarrow\n",
    "\\dots\n",
    "\\rightarrow\n",
    "Encoder_N\n",
    "$$\n",
    "\n",
    "The output of one encoder becomes the input to the next encoder:\n",
    "\n",
    "$$\n",
    "X^{(1)}\n",
    "=\n",
    "Encoder_1(X)\n",
    "$$\n",
    "\n",
    "$$\n",
    "X^{(2)}\n",
    "=\n",
    "Encoder_2(X^{(1)})\n",
    "$$\n",
    "\n",
    "$$\n",
    "X^{(3)}\n",
    "=\n",
    "Encoder_3(X^{(2)})\n",
    "$$\n",
    "\n",
    "and so on.\n",
    "\n",
    "Importantly, the shape can remain:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "through every encoder layer.\n",
    "\n",
    "What changes is the **quality and depth of the representation**.\n",
    "\n",
    "---\n",
    "\n",
    "# Simple mental model\n",
    "\n",
    "You can think of the whole encoder block as two major stages.\n",
    "\n",
    "### Stage 1 — Attention\n",
    "\n",
    "> **\"Talk to the other tokens.\"**\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Token Mixing}\n",
    "}\n",
    "$$\n",
    "\n",
    "### Stage 2 — FFN\n",
    "\n",
    "> **\"Process what you learned.\"**\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Feature Transformation}\n",
    "}\n",
    "$$\n",
    "\n",
    "Both stages are protected by:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Residual Connection}\n",
    "+\n",
    "\\text{LayerNorm}\n",
    "}\n",
    "$$\n",
    "\n",
    "So the complete idea becomes:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Communicate}\n",
    "\\rightarrow\n",
    "\\text{Preserve + Normalize}\n",
    "\\rightarrow\n",
    "\\text{Transform}\n",
    "\\rightarrow\n",
    "\\text{Preserve + Normalize}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Final encoder-block output\n",
    "\n",
    "For our sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "we finally obtain:\n",
    "\n",
    "$$\n",
    "\\operatorname{EncoderOutput}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "H_{\\text{the}} \\\\\n",
    "H_{\\text{cat}} \\\\\n",
    "H_{\\text{chased}} \\\\\n",
    "H_{\\text{the}} \\\\\n",
    "H_{\\text{mouse}}\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "where every contextual token representation has:\n",
    "\n",
    "$$\n",
    "512\n",
    "$$\n",
    "\n",
    "features.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{EncoderOutput}\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "}\n",
    "$$\n",
    "\n",
    "The encoder has therefore transformed:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{5 basic token representations}\n",
    "}\n",
    "$$\n",
    "\n",
    "into:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{5 context-aware token representations}\n",
    "}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "05f0f9e4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:23.192385Z",
     "iopub.status.busy": "2026-10-03T18:57:23.192153Z",
     "iopub.status.idle": "2026-10-03T18:57:23.204634Z",
     "shell.execute_reply": "2026-10-03T18:57:23.204327Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭───────────────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"font-weight: bold\">STEP 9 — FFN RESIDUAL + LAYERNORM</span> <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰───────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭───────────────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1mSTEP 9 — FFN RESIDUAL + LAYERNORM\u001b[0m \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰───────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Residual</span>  X1 <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> + <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">FFN</span><span style=\"font-weight: bold\">(</span>X1<span style=\"font-weight: bold\">)</span> <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span> -&gt; <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mResidual\u001b[0m  X1 \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m + \u001b[1;35mFFN\u001b[0m\u001b[1m(\u001b[0mX1\u001b[1m)\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m -> \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                               X1 + FFN(X1)   shape=(5, 512)                                </span>\n",
       "                                                                                            \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -1.0177    2.3840   -1.1269   -0.7331    0.0953    1.5697   1.8038   2.4237   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>   0.0633    1.9006   -0.1326   -1.3230    0.9674    1.0328   2.5864   2.0618   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>   0.3398    0.6786    0.0464   -2.5696    1.3080   -0.0062   2.5277   1.1855   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5267   -0.1202   -0.6218   -3.4139    0.6905   -0.7049   1.7241   0.4793   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -1.7085    0.1723   -1.6471   -3.2885   -0.3359   -0.5672   0.5044   0.4418   …  \n",
       "                                                                                            \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                               X1 + FFN(X1)   shape=(5, 512)                                \u001b[0m\n",
       "                                                                                            \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -1.0177    2.3840   -1.1269   -0.7331    0.0953    1.5697   1.8038   2.4237   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m   0.0633    1.9006   -0.1326   -1.3230    0.9674    1.0328   2.5864   2.0618   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m   0.3398    0.6786    0.0464   -2.5696    1.3080   -0.0062   2.5277   1.1855   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5267   -0.1202   -0.6218   -3.4139    0.6905   -0.7049   1.7241   0.4793   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -1.7085    0.1723   -1.6471   -3.2885   -0.3359   -0.5672   0.5044   0.4418   …  \n",
       "                                                                                            \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                        FINAL ENCODER BLOCK OUTPUT   shape=(5, 512)                         </span>\n",
       "                                                                                            \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.8747    2.0577   -0.9689   -0.6294    0.0848    1.3557   1.5576   2.0919   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>   0.0559    1.6358   -0.1126   -1.1361    0.8334    0.8895   2.2255   1.7744   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>   0.2937    0.5855    0.0412   -2.2115    1.1275   -0.0042   2.1777   1.0220   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.4492   -0.0985   -0.5312   -2.9400    0.6010   -0.6029   1.4926   0.4187   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -1.4636    0.1591   -1.4107   -2.8268   -0.2795   -0.4790   0.4456   0.3915   …  \n",
       "                                                                                            \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                        FINAL ENCODER BLOCK OUTPUT   shape=(5, 512)                         \u001b[0m\n",
       "                                                                                            \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.8747    2.0577   -0.9689   -0.6294    0.0848    1.3557   1.5576   2.0919   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m   0.0559    1.6358   -0.1126   -1.1361    0.8334    0.8895   2.2255   1.7744   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m   0.2937    0.5855    0.0412   -2.2115    1.1275   -0.0042   2.1777   1.0220   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.4492   -0.0985   -0.5312   -2.9400    0.6010   -0.6029   1.4926   0.4187   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -1.4636    0.1591   -1.4107   -2.8268   -0.2795   -0.4790   0.4456   0.3915   …  \n",
       "                                                                                            \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Final shape:</span> <span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span><span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\u001b[1;32mFinal shape:\u001b[0m \u001b[1m(\u001b[0m\u001b[1;36m5\u001b[0m, \u001b[1;36m512\u001b[0m\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "gamma2 = nn.Parameter(torch.ones(D_MODEL, device=device))\n",
    "beta2 = nn.Parameter(torch.zeros(D_MODEL, device=device))\n",
    "\n",
    "ffn_residual = residual_add(x1, ffn_output)\n",
    "encoder_output_manual = manual_layer_norm(ffn_residual, gamma2, beta2)\n",
    "\n",
    "console.print(Panel.fit(\"[bold]STEP 9 — FFN RESIDUAL + LAYERNORM[/bold]\", border_style=\"green\"))\n",
    "print_shape_equation(\"Residual\", f\"X1 {tuple(x1.shape)} + FFN(X1) {tuple(ffn_output.shape)} -> {tuple(ffn_residual.shape)}\")\n",
    "rich_tensor(\"X1 + FFN(X1)\", ffn_residual, row_labels=tokens)\n",
    "rich_tensor(\"FINAL ENCODER BLOCK OUTPUT\", encoder_output_manual, row_labels=tokens)\n",
    "console.print(f\"\\n[bold green]Final shape:[/bold green] {tuple(encoder_output_manual.shape)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "433295ff",
   "metadata": {},
   "source": [
    "## 11. Package the math into reusable encoder modules\n",
    "\n",
    "Up to this point, we implemented every Transformer encoder operation step-by-step:\n",
    "\n",
    "$$\n",
    "\\text{Embeddings}\n",
    "\\rightarrow\n",
    "\\text{Positional Encoding}\n",
    "\\rightarrow\n",
    "Q,K,V\n",
    "\\rightarrow\n",
    "\\text{Multi-Head Attention}\n",
    "\\rightarrow\n",
    "\\text{Residual + LayerNorm}\n",
    "\\rightarrow\n",
    "\\text{FFN}\n",
    "\\rightarrow\n",
    "\\text{Residual + LayerNorm}\n",
    "$$\n",
    "\n",
    "This is useful for understanding the mathematics.\n",
    "\n",
    "But in a real model, we do not want to manually write all of these steps every time.\n",
    "\n",
    "Instead, we package them into **reusable PyTorch modules**.\n",
    "\n",
    "---\n",
    "\n",
    "# Why create reusable modules?\n",
    "\n",
    "Imagine we want:\n",
    "\n",
    "$$\n",
    "6\n",
    "$$\n",
    "\n",
    "encoder layers.\n",
    "\n",
    "Without reusable modules, we would have to manually repeat:\n",
    "\n",
    "- Q, K, V projection\n",
    "- head splitting\n",
    "- attention calculation\n",
    "- Softmax\n",
    "- head concatenation\n",
    "- output projection\n",
    "- residual connection\n",
    "- LayerNorm\n",
    "- FFN\n",
    "- second residual connection\n",
    "- second LayerNorm\n",
    "\n",
    "six times.\n",
    "\n",
    "That would make the code:\n",
    "\n",
    "- repetitive\n",
    "- difficult to maintain\n",
    "- difficult to debug\n",
    "- difficult to scale\n",
    "\n",
    "Instead, we create reusable building blocks.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Mathematical operations}\n",
    "\\rightarrow\n",
    "\\text{Reusable PyTorch classes}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Main reusable components\n",
    "\n",
    "We can organize our encoder into several modules.\n",
    "\n",
    "### 1. Token Embedding\n",
    "\n",
    "Responsible for:\n",
    "\n",
    "$$\n",
    "\\text{Token ID}\n",
    "\\rightarrow\n",
    "\\text{512-dimensional vector}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 2. Positional Encoding\n",
    "\n",
    "Responsible for adding position information:\n",
    "\n",
    "$$\n",
    "X\n",
    "=\n",
    "\\text{TokenEmbedding}\n",
    "+\n",
    "\\text{PositionalEncoding}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 3. Multi-Head Self-Attention\n",
    "\n",
    "Responsible for:\n",
    "\n",
    "$$\n",
    "Q=XW_Q\n",
    "$$\n",
    "\n",
    "$$\n",
    "K=XW_K\n",
    "$$\n",
    "\n",
    "$$\n",
    "V=XW_V\n",
    "$$\n",
    "\n",
    "then:\n",
    "\n",
    "$$\n",
    "\\operatorname{Attention}(Q,K,V)\n",
    "=\n",
    "\\operatorname{softmax}\n",
    "\\left(\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "\\right)V\n",
    "$$\n",
    "\n",
    "followed by:\n",
    "\n",
    "$$\n",
    "\\text{Concatenate Heads}\n",
    "\\rightarrow\n",
    "W_O\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 4. Layer Normalization\n",
    "\n",
    "Responsible for:\n",
    "\n",
    "$$\n",
    "\\operatorname{LayerNorm}(x)\n",
    "=\n",
    "\\gamma\n",
    "\\frac{x-\\mu}\n",
    "{\\sqrt{\\sigma^2+\\epsilon}}\n",
    "+\n",
    "\\beta\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 5. Feed-Forward Network\n",
    "\n",
    "Responsible for:\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(x)\n",
    "=\n",
    "\\operatorname{ReLU}(xW_1+b_1)W_2+b_2\n",
    "$$\n",
    "\n",
    "with:\n",
    "\n",
    "$$\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "\\rightarrow\n",
    "512\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 6. Encoder Layer\n",
    "\n",
    "Combines everything into one complete Transformer encoder block:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Self-Attention}\n",
    "\\rightarrow\n",
    "\\text{Add + Norm}\n",
    "\\rightarrow\n",
    "\\text{FFN}\n",
    "\\rightarrow\n",
    "\\text{Add + Norm}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# What is an Encoder Layer?\n",
    "\n",
    "One encoder layer receives:\n",
    "\n",
    "$$\n",
    "X\n",
    "\\in\n",
    "\\mathbb{R}^{n \\times d_{\\text{model}}}\n",
    "$$\n",
    "\n",
    "and returns another tensor with the same shape:\n",
    "\n",
    "$$\n",
    "EncoderLayer(X)\n",
    "\\in\n",
    "\\mathbb{R}^{n \\times d_{\\text{model}}}\n",
    "$$\n",
    "\n",
    "For our example:\n",
    "\n",
    "$$\n",
    "X\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "EncoderLayer(X)\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "EncoderLayer\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "The shape remains the same, but the representation becomes more contextual.\n",
    "\n",
    "---\n",
    "\n",
    "# Internal structure of one Encoder Layer\n",
    "\n",
    "One encoder layer performs:\n",
    "\n",
    "### Step 1: Multi-Head Self-Attention\n",
    "\n",
    "$$\n",
    "A\n",
    "=\n",
    "\\operatorname{MHA}(X)\n",
    "$$\n",
    "\n",
    "### Step 2: First residual + LayerNorm\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "(\n",
    "X+A\n",
    ")\n",
    "$$\n",
    "\n",
    "### Step 3: Feed-Forward Network\n",
    "\n",
    "$$\n",
    "F\n",
    "=\n",
    "\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "### Step 4: Second residual + LayerNorm\n",
    "\n",
    "$$\n",
    "X_2\n",
    "=\n",
    "\\operatorname{LayerNorm}\n",
    "(\n",
    "X_1+F\n",
    ")\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "EncoderLayer(X)=X_2\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Why use PyTorch classes?\n",
    "\n",
    "PyTorch models are usually organized using:\n",
    "\n",
    "```python\n",
    "nn.Module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "107d5a38",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:23.205710Z",
     "iopub.status.busy": "2026-10-03T18:57:23.205559Z",
     "iopub.status.idle": "2026-10-03T18:57:23.213209Z",
     "shell.execute_reply": "2026-10-03T18:57:23.212947Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">╭────────────────────────────────────────────────╮\n",
       "│ <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Reusable encoder classes created successfully.</span> │\n",
       "╰────────────────────────────────────────────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "╭────────────────────────────────────────────────╮\n",
       "│ \u001b[1;32mReusable encoder classes created successfully.\u001b[0m │\n",
       "╰────────────────────────────────────────────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "class ManualMultiHeadSelfAttention(nn.Module):\n",
    "    def __init__(self, d_model, num_heads):\n",
    "        super().__init__()\n",
    "        assert d_model % num_heads == 0, \"d_model must be divisible by num_heads\"\n",
    "        self.d_model = d_model\n",
    "        self.num_heads = num_heads\n",
    "        self.d_head = d_model // num_heads\n",
    "\n",
    "        self.q_proj = ManualLinear(d_model, d_model)\n",
    "        self.k_proj = ManualLinear(d_model, d_model)\n",
    "        self.v_proj = ManualLinear(d_model, d_model)\n",
    "        self.o_proj = ManualLinear(d_model, d_model)\n",
    "\n",
    "    def forward(self, x, return_details=False):\n",
    "        Q = self.q_proj(x)\n",
    "        K = self.k_proj(x)\n",
    "        V = self.v_proj(x)\n",
    "\n",
    "        Qh = split_heads(Q, self.num_heads)\n",
    "        Kh = split_heads(K, self.num_heads)\n",
    "        Vh = split_heads(V, self.num_heads)\n",
    "\n",
    "        context_heads, scores, weights = scaled_dot_product_attention(Qh, Kh, Vh)\n",
    "        concatenated = combine_heads(context_heads)\n",
    "        output = self.o_proj(concatenated)\n",
    "\n",
    "        if return_details:\n",
    "            details = {\n",
    "                \"Q\": Q, \"K\": K, \"V\": V,\n",
    "                \"Qh\": Qh, \"Kh\": Kh, \"Vh\": Vh,\n",
    "                \"scores\": scores,\n",
    "                \"weights\": weights,\n",
    "                \"context_heads\": context_heads,\n",
    "                \"concatenated\": concatenated,\n",
    "            }\n",
    "            return output, details\n",
    "        return output\n",
    "\n",
    "\n",
    "class EncoderLayerFromScratch(nn.Module):\n",
    "    def __init__(self, d_model=512, num_heads=4, d_ff=2048):\n",
    "        super().__init__()\n",
    "        self.self_attn = ManualMultiHeadSelfAttention(d_model, num_heads)\n",
    "        self.ffn = ManualFeedForward(d_model, d_ff)\n",
    "\n",
    "        # Two manually implemented LayerNorms need trainable gamma/beta.\n",
    "        self.gamma1 = nn.Parameter(torch.ones(d_model))\n",
    "        self.beta1 = nn.Parameter(torch.zeros(d_model))\n",
    "        self.gamma2 = nn.Parameter(torch.ones(d_model))\n",
    "        self.beta2 = nn.Parameter(torch.zeros(d_model))\n",
    "\n",
    "    def forward(self, x, return_details=False):\n",
    "        attn_out, attn_details = self.self_attn(x, return_details=True)\n",
    "        after_attn_residual = residual_add(x, attn_out)\n",
    "        x1 = manual_layer_norm(after_attn_residual, self.gamma1, self.beta1)\n",
    "\n",
    "        ffn_out, ffn_pre, ffn_hidden = self.ffn(x1, return_hidden=True)\n",
    "        after_ffn_residual = residual_add(x1, ffn_out)\n",
    "        x2 = manual_layer_norm(after_ffn_residual, self.gamma2, self.beta2)\n",
    "\n",
    "        if return_details:\n",
    "            return x2, {\n",
    "                **attn_details,\n",
    "                \"attention_output\": attn_out,\n",
    "                \"after_attention_residual\": after_attn_residual,\n",
    "                \"after_attention_norm\": x1,\n",
    "                \"ffn_pre\": ffn_pre,\n",
    "                \"ffn_hidden\": ffn_hidden,\n",
    "                \"ffn_output\": ffn_out,\n",
    "                \"after_ffn_residual\": after_ffn_residual,\n",
    "                \"encoder_output\": x2,\n",
    "            }\n",
    "        return x2\n",
    "\n",
    "\n",
    "class EncoderFromScratch(nn.Module):\n",
    "    def __init__(self, vocab_size, d_model=512, num_heads=4, d_ff=2048, num_layers=1, max_seq_len=128):\n",
    "        super().__init__()\n",
    "        self.d_model = d_model\n",
    "        self.num_heads = num_heads\n",
    "        self.embedding = ManualEmbedding(vocab_size, d_model)\n",
    "        self.layers = nn.ModuleList([\n",
    "            EncoderLayerFromScratch(d_model, num_heads, d_ff)\n",
    "            for _ in range(num_layers)\n",
    "        ])\n",
    "        self.register_buffer(\n",
    "            \"positional_encoding\",\n",
    "            sinusoidal_positional_encoding(max_seq_len, d_model, torch.device(\"cpu\")),\n",
    "            persistent=False,\n",
    "        )\n",
    "\n",
    "    def forward(self, token_ids, return_details=False):\n",
    "        n = token_ids.shape[0]\n",
    "        token_emb = self.embedding(token_ids)\n",
    "        pe = self.positional_encoding[:n].to(token_ids.device)\n",
    "        x = add_embedding_and_position(token_emb, pe)\n",
    "\n",
    "        all_layer_details = []\n",
    "        for layer in self.layers:\n",
    "            if return_details:\n",
    "                x, details = layer(x, return_details=True)\n",
    "                all_layer_details.append(details)\n",
    "            else:\n",
    "                x = layer(x)\n",
    "\n",
    "        if return_details:\n",
    "            return x, {\n",
    "                \"token_embeddings\": token_emb,\n",
    "                \"positional_encoding\": pe,\n",
    "                \"embedding_plus_position\": token_emb + pe,\n",
    "                \"layers\": all_layer_details,\n",
    "            }\n",
    "        return x\n",
    "\n",
    "console.print(Panel.fit(\"[bold green]Reusable encoder classes created successfully.[/bold green]\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b089e818",
   "metadata": {},
   "source": [
    "## 12. Factory function — one call builds and runs the whole encoder\n",
    "\n",
    "So far, we built the Transformer encoder piece by piece:\n",
    "\n",
    "- tokenization\n",
    "- vocabulary creation\n",
    "- token-to-ID conversion\n",
    "- embedding lookup\n",
    "- positional encoding\n",
    "- self-attention\n",
    "- residual connections\n",
    "- LayerNorm\n",
    "- feed-forward network\n",
    "- stacking encoder layers\n",
    "\n",
    "This is excellent for learning.\n",
    "\n",
    "But in practice, we often want **one simple function** that can do all of this for us.\n",
    "\n",
    "That is the purpose of the **factory function**.\n",
    "\n",
    "---\n",
    "\n",
    "# What is a factory function?\n",
    "\n",
    "A factory function is a helper function that:\n",
    "\n",
    "1. receives configuration values\n",
    "2. builds the required objects\n",
    "3. runs the complete process\n",
    "4. returns the final output (and optionally intermediate details)\n",
    "\n",
    "So instead of manually writing many lines every time, we can just do:\n",
    "\n",
    "```python\n",
    "encoder_factory(sentence=\"The cat chased the mouse\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a0f6c7ce",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:23.214245Z",
     "iopub.status.busy": "2026-10-03T18:57:23.214097Z",
     "iopub.status.idle": "2026-10-03T18:57:23.309361Z",
     "shell.execute_reply": "2026-10-03T18:57:23.309015Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008080; text-decoration-color: #008080\">╭──────── Factory Configuration ────────╮</span>\n",
       "<span style=\"color: #008080; text-decoration-color: #008080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">ENCODER FACTORY</span>                       <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
       "<span style=\"color: #008080; text-decoration-color: #008080\">│</span> sentence = 'The cat chased the mouse' <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
       "<span style=\"color: #008080; text-decoration-color: #008080\">│</span> n = 5                                 <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
       "<span style=\"color: #008080; text-decoration-color: #008080\">│</span> d_model = 512                         <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
       "<span style=\"color: #008080; text-decoration-color: #008080\">│</span> num_heads = 4                         <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
       "<span style=\"color: #008080; text-decoration-color: #008080\">│</span> d_head = 128                          <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
       "<span style=\"color: #008080; text-decoration-color: #008080\">│</span> d_ff = 2048                           <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
       "<span style=\"color: #008080; text-decoration-color: #008080\">│</span> num_layers = 1                        <span style=\"color: #008080; text-decoration-color: #008080\">│</span>\n",
       "<span style=\"color: #008080; text-decoration-color: #008080\">╰───────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[36m╭─\u001b[0m\u001b[36m───────\u001b[0m\u001b[36m Factory Configuration \u001b[0m\u001b[36m───────\u001b[0m\u001b[36m─╮\u001b[0m\n",
       "\u001b[36m│\u001b[0m \u001b[1;36mENCODER FACTORY\u001b[0m                       \u001b[36m│\u001b[0m\n",
       "\u001b[36m│\u001b[0m sentence = 'The cat chased the mouse' \u001b[36m│\u001b[0m\n",
       "\u001b[36m│\u001b[0m n = 5                                 \u001b[36m│\u001b[0m\n",
       "\u001b[36m│\u001b[0m d_model = 512                         \u001b[36m│\u001b[0m\n",
       "\u001b[36m│\u001b[0m num_heads = 4                         \u001b[36m│\u001b[0m\n",
       "\u001b[36m│\u001b[0m d_head = 128                          \u001b[36m│\u001b[0m\n",
       "\u001b[36m│\u001b[0m d_ff = 2048                           \u001b[36m│\u001b[0m\n",
       "\u001b[36m│\u001b[0m num_layers = 1                        \u001b[36m│\u001b[0m\n",
       "\u001b[36m╰───────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                          Factory: token embeddings   shape=(5, 512)                          </span>\n",
       "                                                                                              \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.0245    0.0193   -0.0316    0.0134   -0.0012    0.0014   -0.0333   -0.0153   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.0170   -0.0140   -0.0041   -0.0156    0.0137    0.0157   -0.0222   -0.0621   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.0120   -0.0012    0.0059    0.0327   -0.0299   -0.0060   -0.0083   -0.0157   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.0245    0.0193   -0.0316    0.0134   -0.0012    0.0014   -0.0333   -0.0153   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.0135   -0.0378   -0.0368    0.0026   -0.0159    0.0246    0.0016    0.0361   …  \n",
       "                                                                                              \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                          Factory: token embeddings   shape=(5, 512)                          \u001b[0m\n",
       "                                                                                              \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.0245    0.0193   -0.0316    0.0134   -0.0012    0.0014   -0.0333   -0.0153   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.0170   -0.0140   -0.0041   -0.0156    0.0137    0.0157   -0.0222   -0.0621   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.0120   -0.0012    0.0059    0.0327   -0.0299   -0.0060   -0.0083   -0.0157   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.0245    0.0193   -0.0316    0.0134   -0.0012    0.0014   -0.0333   -0.0153   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.0135   -0.0378   -0.0368    0.0026   -0.0159    0.0246    0.0016    0.0361   …  \n",
       "                                                                                              \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                        Factory: positional encoding   shape=(5, 512)                         </span>\n",
       "                                                                                              \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.0000    1.0000    0.0000    1.0000    0.0000    1.0000    0.0000    1.0000   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>   0.8415    0.5403    0.8219    0.5697    0.8020    0.5974    0.7819    0.6234   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>   0.9093   -0.4161    0.9364   -0.3509    0.9581   -0.2863    0.9749   -0.2227   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.1411   -0.9900    0.2451   -0.9695    0.3428   -0.9394    0.4336   -0.9011   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.7568   -0.6536   -0.6572   -0.7537   -0.5486   -0.8361   -0.4342   -0.9008   …  \n",
       "                                                                                              \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                        Factory: positional encoding   shape=(5, 512)                         \u001b[0m\n",
       "                                                                                              \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.0000    1.0000    0.0000    1.0000    0.0000    1.0000    0.0000    1.0000   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m   0.8415    0.5403    0.8219    0.5697    0.8020    0.5974    0.7819    0.6234   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m   0.9093   -0.4161    0.9364   -0.3509    0.9581   -0.2863    0.9749   -0.2227   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.1411   -0.9900    0.2451   -0.9695    0.3428   -0.9394    0.4336   -0.9011   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.7568   -0.6536   -0.6572   -0.7537   -0.5486   -0.8361   -0.4342   -0.9008   …  \n",
       "                                                                                              \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                        Factory: embedding + position   shape=(5, 512)                        </span>\n",
       "                                                                                              \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.0245    1.0193   -0.0316    1.0134   -0.0012    1.0014   -0.0333    0.9847   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>   0.8245    0.5263    0.8178    0.5541    0.8157    0.6130    0.7597    0.5613   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>   0.8973   -0.4173    0.9424   -0.3182    0.9282   -0.2923    0.9666   -0.2384   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.1166   -0.9707    0.2135   -0.9561    0.3416   -0.9380    0.4004   -0.9164   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.7703   -0.6914   -0.6940   -0.7511   -0.5645   -0.8115   -0.4327   -0.8647   …  \n",
       "                                                                                              \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                        Factory: embedding + position   shape=(5, 512)                        \u001b[0m\n",
       "                                                                                              \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.0245    1.0193   -0.0316    1.0134   -0.0012    1.0014   -0.0333    0.9847   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m   0.8245    0.5263    0.8178    0.5541    0.8157    0.6130    0.7597    0.5613   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m   0.8973   -0.4173    0.9424   -0.3182    0.9282   -0.2923    0.9666   -0.2384   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.1166   -0.9707    0.2135   -0.9561    0.3416   -0.9380    0.4004   -0.9164   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.7703   -0.6914   -0.6940   -0.7511   -0.5645   -0.8115   -0.4327   -0.8647   …  \n",
       "                                                                                              \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────── </span>ENCODER LAYER <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────── \u001b[0mENCODER LAYER \u001b[1;36m1\u001b[0m\u001b[92m ──────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                    Q   shape=(5, 512)                                    </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">      2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.2420   0.3520   0.8823   0.6881   -0.5499    0.3434   0.1939   -0.3345   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.0232   0.2470   0.8416   0.7101   -0.6239    0.2577   0.1792   -0.0826   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.0912   0.1675   0.8518   0.7525   -0.5094    0.0991   0.2582    0.1603   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.0652   0.0258   0.8789   0.7544   -0.3073   -0.1305   0.3588    0.3885   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>   0.1399   0.0169   0.9198   0.6410   -0.0117   -0.1464   0.5031    0.4548   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                    Q   shape=(5, 512)                                    \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.2420   0.3520   0.8823   0.6881   -0.5499    0.3434   0.1939   -0.3345   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.0232   0.2470   0.8416   0.7101   -0.6239    0.2577   0.1792   -0.0826   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.0912   0.1675   0.8518   0.7525   -0.5094    0.0991   0.2582    0.1603   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.0652   0.0258   0.8789   0.7544   -0.3073   -0.1305   0.3588    0.3885   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m   0.1399   0.0169   0.9198   0.6410   -0.0117   -0.1464   0.5031    0.4548   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                   K   shape=(5, 512)                                    </span>\n",
       "                                                                                         \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">      4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>   0.2209   0.8819   -0.4400   0.6340   0.3152   0.6200   1.1130   -1.0775   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.0968   1.0331   -0.2886   0.9293   0.1714   0.7778   1.0469   -1.0041   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.2425   1.0368   -0.1817   1.0810   0.0698   0.9467   0.9014   -0.8800   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.2267   0.8996   -0.1362   0.9040   0.0373   1.1313   0.7261   -0.7024   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>   0.0525   0.7449   -0.2175   0.5866   0.1555   1.1017   0.5973   -0.5712   …  \n",
       "                                                                                         \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                   K   shape=(5, 512)                                    \u001b[0m\n",
       "                                                                                         \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m   0.2209   0.8819   -0.4400   0.6340   0.3152   0.6200   1.1130   -1.0775   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.0968   1.0331   -0.2886   0.9293   0.1714   0.7778   1.0469   -1.0041   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.2425   1.0368   -0.1817   1.0810   0.0698   0.9467   0.9014   -0.8800   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.2267   0.8996   -0.1362   0.9040   0.0373   1.1313   0.7261   -0.7024   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m   0.0525   0.7449   -0.2175   0.5866   0.1555   1.1017   0.5973   -0.5712   …  \n",
       "                                                                                         \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                    V   shape=(5, 512)                                    </span>\n",
       "                                                                                          \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">      0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">      5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.5705   0.6160    0.0298   0.4387   -1.1241   0.1134   -0.2389    0.5329   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  0.5968   0.7630   -0.0804   0.5254   -1.2284   0.2127   -0.2206    0.5499   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  0.4991   0.7863   -0.1270   0.4880   -1.1597   0.2153   -0.3086    0.3211   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  0.3702   0.5993   -0.1513   0.3479   -1.1130   0.1664   -0.4945   -0.0316   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  0.2347   0.3944   -0.1243   0.1479   -1.0659   0.0888   -0.6067   -0.3272   …  \n",
       "                                                                                          \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                    V   shape=(5, 512)                                    \u001b[0m\n",
       "                                                                                          \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.5705   0.6160    0.0298   0.4387   -1.1241   0.1134   -0.2389    0.5329   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  0.5968   0.7630   -0.0804   0.5254   -1.2284   0.2127   -0.2206    0.5499   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  0.4991   0.7863   -0.1270   0.4880   -1.1597   0.2153   -0.3086    0.3211   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  0.3702   0.5993   -0.1513   0.3479   -1.1130   0.1664   -0.4945   -0.0316   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  0.2347   0.3944   -0.1243   0.1479   -1.0659   0.0888   -0.6067   -0.3272   …  \n",
       "                                                                                          \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">   Layer 1 / Head 1 attention weights   shape=(5, 5)   </span>\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">    cat </span>│<span style=\"font-weight: bold\"> chased </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">  mouse </span>│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.1794 │ 0.1994 │ 0.2179 │ 0.2121 │ 0.1911 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ 0.1714 │ 0.1964 │ 0.2205 │ 0.2166 │ 0.1951 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ 0.1665 │ 0.1918 │ 0.2186 │ 0.2195 │ 0.2035 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.1675 │ 0.1881 │ 0.2142 │ 0.2188 │ 0.2114 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ 0.1739 │ 0.1856 │ 0.2078 │ 0.2151 │ 0.2176 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m   Layer 1 / Head 1 attention weights   shape=(5, 5)   \u001b[0m\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mchased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1794 │ 0.1994 │ 0.2179 │ 0.2121 │ 0.1911 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1714 │ 0.1964 │ 0.2205 │ 0.2166 │ 0.1951 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ 0.1665 │ 0.1918 │ 0.2186 │ 0.2195 │ 0.2035 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1675 │ 0.1881 │ 0.2142 │ 0.2188 │ 0.2114 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ 0.1739 │ 0.1856 │ 0.2078 │ 0.2151 │ 0.2176 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">   Layer 1 / Head 2 attention weights   shape=(5, 5)   </span>\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">    cat </span>│<span style=\"font-weight: bold\"> chased </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">  mouse </span>│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2230 │ 0.2287 │ 0.2068 │ 0.1836 │ 0.1580 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ 0.2164 │ 0.2247 │ 0.2073 │ 0.1887 │ 0.1629 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ 0.2111 │ 0.2176 │ 0.2049 │ 0.1937 │ 0.1727 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2086 │ 0.2107 │ 0.2018 │ 0.1960 │ 0.1829 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ 0.2076 │ 0.2048 │ 0.1985 │ 0.1967 │ 0.1924 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m   Layer 1 / Head 2 attention weights   shape=(5, 5)   \u001b[0m\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mchased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2230 │ 0.2287 │ 0.2068 │ 0.1836 │ 0.1580 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2164 │ 0.2247 │ 0.2073 │ 0.1887 │ 0.1629 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ 0.2111 │ 0.2176 │ 0.2049 │ 0.1937 │ 0.1727 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2086 │ 0.2107 │ 0.2018 │ 0.1960 │ 0.1829 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ 0.2076 │ 0.2048 │ 0.1985 │ 0.1967 │ 0.1924 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">   Layer 1 / Head 3 attention weights   shape=(5, 5)   </span>\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">    cat </span>│<span style=\"font-weight: bold\"> chased </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">  mouse </span>│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.1888 │ 0.1943 │ 0.2009 │ 0.2116 │ 0.2044 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ 0.1935 │ 0.1975 │ 0.2017 │ 0.2072 │ 0.2001 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ 0.1928 │ 0.1987 │ 0.2039 │ 0.2069 │ 0.1977 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.1898 │ 0.1993 │ 0.2076 │ 0.2083 │ 0.1950 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ 0.1848 │ 0.1986 │ 0.2117 │ 0.2120 │ 0.1929 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m   Layer 1 / Head 3 attention weights   shape=(5, 5)   \u001b[0m\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mchased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1888 │ 0.1943 │ 0.2009 │ 0.2116 │ 0.2044 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1935 │ 0.1975 │ 0.2017 │ 0.2072 │ 0.2001 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ 0.1928 │ 0.1987 │ 0.2039 │ 0.2069 │ 0.1977 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.1898 │ 0.1993 │ 0.2076 │ 0.2083 │ 0.1950 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ 0.1848 │ 0.1986 │ 0.2117 │ 0.2120 │ 0.1929 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">   Layer 1 / Head 4 attention weights   shape=(5, 5)   </span>\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│<span style=\"font-weight: bold\"> Q \\ K  </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">    cat </span>│<span style=\"font-weight: bold\"> chased </span>│<span style=\"font-weight: bold\">    the </span>│<span style=\"font-weight: bold\">  mouse </span>│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2265 │ 0.2134 │ 0.1985 │ 0.1833 │ 0.1782 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>│ 0.2427 │ 0.2227 │ 0.1978 │ 0.1734 │ 0.1634 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>│ 0.2653 │ 0.2333 │ 0.1929 │ 0.1594 │ 0.1490 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>│ 0.2800 │ 0.2366 │ 0.1867 │ 0.1516 │ 0.1451 │\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>│ 0.2843 │ 0.2351 │ 0.1827 │ 0.1500 │ 0.1479 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m   Layer 1 / Head 4 attention weights   shape=(5, 5)   \u001b[0m\n",
       "╭────────┬────────┬────────┬────────┬────────┬────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mQ \\ K \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   cat\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mchased\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m   the\u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1m mouse\u001b[0m\u001b[1m \u001b[0m│\n",
       "├────────┼────────┼────────┼────────┼────────┼────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2265 │ 0.2134 │ 0.1985 │ 0.1833 │ 0.1782 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2427 │ 0.2227 │ 0.1978 │ 0.1734 │ 0.1634 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m│ 0.2653 │ 0.2333 │ 0.1929 │ 0.1594 │ 0.1490 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m│ 0.2800 │ 0.2366 │ 0.1867 │ 0.1516 │ 0.1451 │\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m│ 0.2843 │ 0.2351 │ 0.1827 │ 0.1500 │ 0.1479 │\n",
       "╰────────┴────────┴────────┴────────┴────────┴────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                             Attention output   shape=(5, 512)                             </span>\n",
       "                                                                                           \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">      1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">      3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5401   0.8301   -0.3004   1.3188   -1.3323   -0.0377   0.5517   -1.5257   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.5469   0.8304   -0.3022   1.3169   -1.3376   -0.0326   0.5544   -1.5216   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.5542   0.8308   -0.3037   1.3110   -1.3397   -0.0269   0.5544   -1.5188   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.5572   0.8296   -0.3039   1.3068   -1.3377   -0.0218   0.5526   -1.5197   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.5570   0.8278   -0.3032   1.3046   -1.3329   -0.0186   0.5494   -1.5236   …  \n",
       "                                                                                           \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                             Attention output   shape=(5, 512)                             \u001b[0m\n",
       "                                                                                           \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5401   0.8301   -0.3004   1.3188   -1.3323   -0.0377   0.5517   -1.5257   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.5469   0.8304   -0.3022   1.3169   -1.3376   -0.0326   0.5544   -1.5216   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.5542   0.8308   -0.3037   1.3110   -1.3397   -0.0269   0.5544   -1.5188   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.5572   0.8296   -0.3039   1.3068   -1.3377   -0.0218   0.5526   -1.5197   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.5570   0.8278   -0.3032   1.3046   -1.3329   -0.0186   0.5494   -1.5236   …  \n",
       "                                                                                           \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                    After attention residual + LayerNorm   shape=(5, 512)                     </span>\n",
       "                                                                                              \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">       6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -1.2101    1.5401   -0.9451    2.0902   -2.0862    0.5311    0.0238   -1.1833   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.3028    0.9604   -0.0243    1.5624   -1.2387    0.0517    0.9105   -1.7519   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.2303   -0.1478    0.1159    0.5305   -1.1138   -1.0057    1.1491   -2.6896   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -1.0867   -0.7451   -0.6873   -0.1842   -1.7202   -1.6788    0.5026   -3.3624   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -1.9947   -0.3766   -1.6298    0.0846   -2.6249   -1.4451   -0.3983   -3.1677   …  \n",
       "                                                                                              \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                    After attention residual + LayerNorm   shape=(5, 512)                     \u001b[0m\n",
       "                                                                                              \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -1.2101    1.5401   -0.9451    2.0902   -2.0862    0.5311    0.0238   -1.1833   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.3028    0.9604   -0.0243    1.5624   -1.2387    0.0517    0.9105   -1.7519   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.2303   -0.1478    0.1159    0.5305   -1.1138   -1.0057    1.1491   -2.6896   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -1.0867   -0.7451   -0.6873   -0.1842   -1.7202   -1.6788    0.5026   -3.3624   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -1.9947   -0.3766   -1.6298    0.0846   -2.6249   -1.4451   -0.3983   -3.1677   …  \n",
       "                                                                                              \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                                FFN output   shape=(5, 512)                                 </span>\n",
       "                                                                                            \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">      7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.4567   -0.5697   -0.3160   -1.1164   -1.2255   -0.0489   0.7404   1.5363   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.2877   -0.6035   -0.2151   -0.9681   -1.2094   -0.0401   0.6915   1.3990   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.1356   -0.7691   -0.0931   -0.9240   -1.1844   -0.1716   0.8060   1.1801   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.0803   -0.9905   -0.1182   -0.8792   -1.1292   -0.3073   0.9588   1.0885   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -0.1620   -0.9855   -0.2753   -0.8246   -1.2791   -0.4135   1.0338   1.1202   …  \n",
       "                                                                                            \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                                FFN output   shape=(5, 512)                                 \u001b[0m\n",
       "                                                                                            \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.4567   -0.5697   -0.3160   -1.1164   -1.2255   -0.0489   0.7404   1.5363   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.2877   -0.6035   -0.2151   -0.9681   -1.2094   -0.0401   0.6915   1.3990   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.1356   -0.7691   -0.0931   -0.9240   -1.1844   -0.1716   0.8060   1.1801   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.0803   -0.9905   -0.1182   -0.8792   -1.1292   -0.3073   0.9588   1.0885   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -0.1620   -0.9855   -0.2753   -0.8246   -1.2791   -0.4135   1.0338   1.1202   …  \n",
       "                                                                                            \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                             Final layer output   shape=(5, 512)                             </span>\n",
       "                                                                                             \n",
       " <span style=\"font-weight: bold\"> row    </span> <span style=\"font-weight: bold\">       0 </span> <span style=\"font-weight: bold\">       1 </span> <span style=\"font-weight: bold\">       2 </span> <span style=\"font-weight: bold\">       3 </span> <span style=\"font-weight: bold\">       4 </span> <span style=\"font-weight: bold\">       5 </span> <span style=\"font-weight: bold\">      6 </span> <span style=\"font-weight: bold\">       7 </span> <span style=\"font-weight: bold\"> … </span> \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -1.4147    0.8784   -1.0619    0.8812   -2.8448    0.4538   0.6990    0.3415   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> cat    </span>  -0.4856    0.3410   -0.1793    0.5481   -2.1060    0.0396   1.4272   -0.2783   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> chased </span>  -0.2901   -0.7714    0.0494   -0.3143   -1.9780   -0.9989   1.7373   -1.2891   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> the    </span>  -0.9904   -1.4883   -0.6739   -0.8997   -2.4635   -1.7076   1.3109   -1.9596   …  \n",
       " <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> mouse  </span>  -1.8428   -1.1509   -1.6237   -0.6091   -3.3645   -1.5832   0.5887   -1.7478   …  \n",
       "                                                                                             \n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                             Final layer output   shape=(5, 512)                             \u001b[0m\n",
       "                                                                                             \n",
       " \u001b[1m \u001b[0m\u001b[1mrow   \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      0\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      1\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      2\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      3\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      4\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      5\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m     6\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m      7\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1m…\u001b[0m\u001b[1m \u001b[0m \n",
       " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -1.4147    0.8784   -1.0619    0.8812   -2.8448    0.4538   0.6990    0.3415   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mcat   \u001b[0m\u001b[1;36m \u001b[0m  -0.4856    0.3410   -0.1793    0.5481   -2.1060    0.0396   1.4272   -0.2783   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mchased\u001b[0m\u001b[1;36m \u001b[0m  -0.2901   -0.7714    0.0494   -0.3143   -1.9780   -0.9989   1.7373   -1.2891   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mthe   \u001b[0m\u001b[1;36m \u001b[0m  -0.9904   -1.4883   -0.6739   -0.8997   -2.4635   -1.7076   1.3109   -1.9596   …  \n",
       " \u001b[1;36m \u001b[0m\u001b[1;36mmouse \u001b[0m\u001b[1;36m \u001b[0m  -1.8428   -1.1509   -1.6237   -0.6091   -3.3645   -1.5832   0.5887   -1.7478   …  \n",
       "                                                                                             \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">╭───────────────────────────────────────╮</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Factory complete</span>                      <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">│</span> Final encoder output shape = (5, 512) <span style=\"color: #008000; text-decoration-color: #008000\">│</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">╰───────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m╭───────────────────────────────────────╮\u001b[0m\n",
       "\u001b[32m│\u001b[0m \u001b[1;32mFactory complete\u001b[0m                      \u001b[32m│\u001b[0m\n",
       "\u001b[32m│\u001b[0m Final encoder output shape = (5, 512) \u001b[32m│\u001b[0m\n",
       "\u001b[32m╰───────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "def encoder_factory(\n",
    "    sentence,\n",
    "    d_model=512,\n",
    "    num_heads=4,\n",
    "    d_ff=2048,\n",
    "    num_layers=1,\n",
    "    seed=7,\n",
    "    verbose=True,\n",
    "):\n",
    "    \"\"\"Build and run a complete educational Transformer encoder from scratch.\"\"\"\n",
    "    torch.manual_seed(seed)\n",
    "\n",
    "    toks = simple_tokenize(sentence)\n",
    "    voc = build_vocab(toks)\n",
    "    ids = tokens_to_ids(toks, voc)\n",
    "\n",
    "    model = EncoderFromScratch(\n",
    "        vocab_size=len(voc),\n",
    "        d_model=d_model,\n",
    "        num_heads=num_heads,\n",
    "        d_ff=d_ff,\n",
    "        num_layers=num_layers,\n",
    "        max_seq_len=max(128, len(toks)),\n",
    "    ).to(device)\n",
    "\n",
    "    output, details = model(ids, return_details=True)\n",
    "\n",
    "    if verbose:\n",
    "        console.print(Panel.fit(\n",
    "            f\"[bold cyan]ENCODER FACTORY[/bold cyan]\\n\"\n",
    "            f\"sentence = {sentence!r}\\n\"\n",
    "            f\"n = {len(toks)}\\n\"\n",
    "            f\"d_model = {d_model}\\n\"\n",
    "            f\"num_heads = {num_heads}\\n\"\n",
    "            f\"d_head = {d_model // num_heads}\\n\"\n",
    "            f\"d_ff = {d_ff}\\n\"\n",
    "            f\"num_layers = {num_layers}\",\n",
    "            title=\"Factory Configuration\",\n",
    "            border_style=\"cyan\",\n",
    "        ))\n",
    "\n",
    "        rich_tensor(\"Factory: token embeddings\", details[\"token_embeddings\"], row_labels=toks)\n",
    "        rich_tensor(\"Factory: positional encoding\", details[\"positional_encoding\"], row_labels=toks)\n",
    "        rich_tensor(\"Factory: embedding + position\", details[\"embedding_plus_position\"], row_labels=toks)\n",
    "\n",
    "        for layer_index, layer_details in enumerate(details[\"layers\"], start=1):\n",
    "            console.rule(f\"ENCODER LAYER {layer_index}\")\n",
    "            rich_tensor(\"Q\", layer_details[\"Q\"], row_labels=toks)\n",
    "            rich_tensor(\"K\", layer_details[\"K\"], row_labels=toks)\n",
    "            rich_tensor(\"V\", layer_details[\"V\"], row_labels=toks)\n",
    "\n",
    "            for h in range(num_heads):\n",
    "                rich_attention_matrix(\n",
    "                    f\"Layer {layer_index} / Head {h+1} attention weights\",\n",
    "                    layer_details[\"weights\"][h],\n",
    "                    toks,\n",
    "                )\n",
    "\n",
    "            rich_tensor(\"Attention output\", layer_details[\"attention_output\"], row_labels=toks)\n",
    "            rich_tensor(\"After attention residual + LayerNorm\", layer_details[\"after_attention_norm\"], row_labels=toks)\n",
    "            rich_tensor(\"FFN output\", layer_details[\"ffn_output\"], row_labels=toks)\n",
    "            rich_tensor(\"Final layer output\", layer_details[\"encoder_output\"], row_labels=toks)\n",
    "\n",
    "        console.print(Panel.fit(\n",
    "            f\"[bold green]Factory complete[/bold green]\\n\"\n",
    "            f\"Final encoder output shape = {tuple(output.shape)}\",\n",
    "            border_style=\"green\",\n",
    "        ))\n",
    "\n",
    "    return model, voc, ids, output, details\n",
    "\n",
    "\n",
    "factory_model, factory_vocab, factory_ids, factory_output, factory_details = encoder_factory(\n",
    "    SENTENCE,\n",
    "    d_model=D_MODEL,\n",
    "    num_heads=NUM_HEADS,\n",
    "    d_ff=D_FF,\n",
    "    num_layers=NUM_LAYERS,\n",
    "    seed=42,\n",
    "    verbose=True,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f95c7fe",
   "metadata": {},
   "source": [
    "\n",
    "# 13. Dimension audit — complete encoder with all dimensions\n",
    "\n",
    "For our exact example:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "we use:\n",
    "\n",
    "$$\n",
    "n = 5\n",
    "$$\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "$$\n",
    "h = 4\n",
    "$$\n",
    "\n",
    "$$\n",
    "d_{\\text{head}}\n",
    "=\n",
    "\\frac{d_{\\text{model}}}{h}\n",
    "=\n",
    "\\frac{512}{4}\n",
    "=\n",
    "128\n",
    "$$\n",
    "\n",
    "$$\n",
    "d_{\\text{ff}} = 2048\n",
    "$$\n",
    "\n",
    "Let:\n",
    "\n",
    "$$\n",
    "|V|\n",
    "=\n",
    "\\text{vocabulary size}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 1. Token IDs\n",
    "\n",
    "We have 5 tokens:\n",
    "\n",
    "```text\n",
    "[\"the\", \"cat\", \"chased\", \"the\", \"mouse\"]\n",
    "```\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Token IDs}\n",
    "=\n",
    "[5]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 2. Embedding table\n",
    "\n",
    "The complete trainable embedding table is:\n",
    "\n",
    "$$\n",
    "E\n",
    "\\in\n",
    "\\mathbb{R}^{|V| \\times 512}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "E\n",
    "=\n",
    "[|V|,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "After looking up the 5 token IDs:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_{\\text{embedding}}\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 3. Positional encoding\n",
    "\n",
    "One positional vector is created for every token position.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "PE\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 4. Embedding + positional encoding\n",
    "\n",
    "We calculate:\n",
    "\n",
    "$$\n",
    "X\n",
    "=\n",
    "X_{\\text{embedding}} + PE\n",
    "$$\n",
    "\n",
    "Dimensions:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "+\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Q, K, V projections\n",
    "\n",
    "## 5. Query projection parameters\n",
    "\n",
    "Query is calculated as:\n",
    "\n",
    "$$\n",
    "Q\n",
    "=\n",
    "XW_Q+b_Q\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "X\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "$$\n",
    "W_Q\n",
    "=\n",
    "[512,512]\n",
    "$$\n",
    "\n",
    "$$\n",
    "b_Q\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "(5 \\times 512)\n",
    "(512 \\times 512)\n",
    "\\rightarrow\n",
    "(5 \\times 512)\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 6. Key projection parameters\n",
    "\n",
    "$$\n",
    "K\n",
    "=\n",
    "XW_K+b_K\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "W_K\n",
    "=\n",
    "[512,512]\n",
    "$$\n",
    "\n",
    "$$\n",
    "b_K\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "K\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 7. Value projection parameters\n",
    "\n",
    "$$\n",
    "V\n",
    "=\n",
    "XW_V+b_V\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "W_V\n",
    "=\n",
    "[512,512]\n",
    "$$\n",
    "\n",
    "$$\n",
    "b_V\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "V\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Split into attention heads\n",
    "\n",
    "## 8. Reshape Q, K, V\n",
    "\n",
    "Before splitting:\n",
    "\n",
    "$$\n",
    "Q,K,V\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "Since:\n",
    "\n",
    "$$\n",
    "512\n",
    "=\n",
    "4 \\times 128\n",
    "$$\n",
    "\n",
    "we first reshape:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,4,128]\n",
    "$$\n",
    "\n",
    "Then transpose the head dimension:\n",
    "\n",
    "$$\n",
    "[5,4,128]\n",
    "\\rightarrow\n",
    "[4,5,128]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q_h\n",
    "=\n",
    "[4,5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "K_h\n",
    "=\n",
    "[4,5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "V_h\n",
    "=\n",
    "[4,5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 9. One individual attention head\n",
    "\n",
    "For one head:\n",
    "\n",
    "$$\n",
    "Q_i\n",
    "=\n",
    "[5,128]\n",
    "$$\n",
    "\n",
    "$$\n",
    "K_i\n",
    "=\n",
    "[5,128]\n",
    "$$\n",
    "\n",
    "$$\n",
    "V_i\n",
    "=\n",
    "[5,128]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q_i,K_i,V_i\n",
    "=\n",
    "[5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Scaled Dot-Product Attention\n",
    "\n",
    "## 10. Transpose Key\n",
    "\n",
    "For one head:\n",
    "\n",
    "$$\n",
    "K_i\n",
    "=\n",
    "[5,128]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "K_i^T\n",
    "=\n",
    "[128,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "Across all heads:\n",
    "\n",
    "$$\n",
    "K_h\n",
    "=\n",
    "[4,5,128]\n",
    "$$\n",
    "\n",
    "becomes:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "K_h^T\n",
    "=\n",
    "[4,128,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 11. Raw attention scores\n",
    "\n",
    "For one head:\n",
    "\n",
    "$$\n",
    "Q_iK_i^T\n",
    "$$\n",
    "\n",
    "Dimensions:\n",
    "\n",
    "$$\n",
    "(5 \\times 128)\n",
    "(128 \\times 5)\n",
    "\\rightarrow\n",
    "(5 \\times 5)\n",
    "$$\n",
    "\n",
    "Therefore each head produces:\n",
    "\n",
    "$$\n",
    "[5,5]\n",
    "$$\n",
    "\n",
    "Across 4 heads:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Attention Scores}\n",
    "=\n",
    "[4,5,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 12. Scaling\n",
    "\n",
    "We calculate:\n",
    "\n",
    "$$\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "d_k = 128\n",
    "$$\n",
    "\n",
    "Division by the scalar:\n",
    "\n",
    "$$\n",
    "\\sqrt{128}\n",
    "$$\n",
    "\n",
    "does not change the shape.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Scaled Scores}\n",
    "=\n",
    "[4,5,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 13. Softmax attention weights\n",
    "\n",
    "Softmax is applied along the Key-token dimension:\n",
    "\n",
    "$$\n",
    "A\n",
    "=\n",
    "\\operatorname{softmax}\n",
    "\\left(\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "Softmax changes the values but not the shape.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "A\n",
    "=\n",
    "[4,5,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "For one head:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "A_i\n",
    "=\n",
    "[5,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 14. Attention × Value\n",
    "\n",
    "For one head:\n",
    "\n",
    "$$\n",
    "A_i\n",
    "=\n",
    "[5,5]\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "V_i\n",
    "=\n",
    "[5,128]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "(5 \\times 5)\n",
    "(5 \\times 128)\n",
    "\\rightarrow\n",
    "(5 \\times 128)\n",
    "$$\n",
    "\n",
    "So one head produces:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Context}_i\n",
    "=\n",
    "[5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "Across all 4 heads:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Context Heads}\n",
    "=\n",
    "[4,5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Concatenate heads\n",
    "\n",
    "## 15. Combine all 4 attention heads\n",
    "\n",
    "Each head produces:\n",
    "\n",
    "$$\n",
    "[5,128]\n",
    "$$\n",
    "\n",
    "We have:\n",
    "\n",
    "$$\n",
    "4\n",
    "$$\n",
    "\n",
    "heads.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "4 \\times 128 = 512\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "[4,5,128]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Concatenated Heads}\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Output projection\n",
    "\n",
    "## 16. Output projection parameters\n",
    "\n",
    "After concatenation:\n",
    "\n",
    "$$\n",
    "C\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "The output projection uses:\n",
    "\n",
    "$$\n",
    "W_O\n",
    "=\n",
    "[512,512]\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "b_O\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "CW_O+b_O\n",
    "$$\n",
    "\n",
    "has dimensions:\n",
    "\n",
    "$$\n",
    "(5 \\times 512)\n",
    "(512 \\times 512)\n",
    "\\rightarrow\n",
    "(5 \\times 512)\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{MHAOutput}\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# First Residual + LayerNorm\n",
    "\n",
    "## 17. First residual connection\n",
    "\n",
    "Original encoder input:\n",
    "\n",
    "$$\n",
    "X\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "Attention output:\n",
    "\n",
    "$$\n",
    "\\operatorname{MHA}(X)\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "Residual addition:\n",
    "\n",
    "$$\n",
    "X+\\operatorname{MHA}(X)\n",
    "$$\n",
    "\n",
    "therefore gives:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_{\\text{residual-1}}\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 18. First LayerNorm parameters\n",
    "\n",
    "LayerNorm has learnable:\n",
    "\n",
    "$$\n",
    "\\gamma_1\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\beta_1\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "LayerNorm operates independently over the last dimension.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "\\xrightarrow{\\operatorname{LayerNorm}}\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_1\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Feed-Forward Network\n",
    "\n",
    "## 19. First FFN Linear Layer\n",
    "\n",
    "Input:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "First FFN weight:\n",
    "\n",
    "$$\n",
    "W_1\n",
    "=\n",
    "[512,2048]\n",
    "$$\n",
    "\n",
    "Bias:\n",
    "\n",
    "$$\n",
    "b_1\n",
    "=\n",
    "[2048]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "(5 \\times 512)\n",
    "(512 \\times 2048)\n",
    "\\rightarrow\n",
    "(5 \\times 2048)\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Z_1\n",
    "=\n",
    "[5,2048]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 20. ReLU activation\n",
    "\n",
    "We calculate:\n",
    "\n",
    "$$\n",
    "A_1\n",
    "=\n",
    "\\operatorname{ReLU}(Z_1)\n",
    "$$\n",
    "\n",
    "ReLU does not change the shape.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "A_1\n",
    "=\n",
    "[5,2048]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 21. Second FFN Linear Layer\n",
    "\n",
    "Input:\n",
    "\n",
    "$$\n",
    "A_1\n",
    "=\n",
    "[5,2048]\n",
    "$$\n",
    "\n",
    "Second FFN weight:\n",
    "\n",
    "$$\n",
    "W_2\n",
    "=\n",
    "[2048,512]\n",
    "$$\n",
    "\n",
    "Bias:\n",
    "\n",
    "$$\n",
    "b_2\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "(5 \\times 2048)\n",
    "(2048 \\times 512)\n",
    "\\rightarrow\n",
    "(5 \\times 512)\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{FFN}(X_1)\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Second Residual + LayerNorm\n",
    "\n",
    "## 22. Second residual connection\n",
    "\n",
    "Input from the first Add & Norm:\n",
    "\n",
    "$$\n",
    "X_1\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "FFN output:\n",
    "\n",
    "$$\n",
    "\\operatorname{FFN}(X_1)\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "X_1+\\operatorname{FFN}(X_1)\n",
    "$$\n",
    "\n",
    "gives:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X_{\\text{residual-2}}\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## 23. Second LayerNorm parameters\n",
    "\n",
    "The second LayerNorm has:\n",
    "\n",
    "$$\n",
    "\\gamma_2\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\beta_2\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "LayerNorm keeps the tensor shape unchanged:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{EncoderOutput}\n",
    "=\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Complete dimension audit table\n",
    "\n",
    "| # | Component | Input / Parameter Dimensions | Output Dimension |\n",
    "|---:|---|---|---|\n",
    "| 1 | Token IDs | 5 tokens | $[5]$ |\n",
    "| 2 | Embedding Table $E$ | $[|V|,512]$ | — |\n",
    "| 3 | Token Embedding Lookup | $[5]$ | $[5,512]$ |\n",
    "| 4 | Positional Encoding | — | $[5,512]$ |\n",
    "| 5 | Embedding + Position | $[5,512]+[5,512]$ | $[5,512]$ |\n",
    "| 6 | $W_Q$ | $[512,512]$ | — |\n",
    "| 7 | $b_Q$ | $[512]$ | — |\n",
    "| 8 | Query $Q$ | $[5,512]\\times[512,512]$ | $[5,512]$ |\n",
    "| 9 | $W_K$ | $[512,512]$ | — |\n",
    "| 10 | $b_K$ | $[512]$ | — |\n",
    "| 11 | Key $K$ | $[5,512]\\times[512,512]$ | $[5,512]$ |\n",
    "| 12 | $W_V$ | $[512,512]$ | — |\n",
    "| 13 | $b_V$ | $[512]$ | — |\n",
    "| 14 | Value $V$ | $[5,512]\\times[512,512]$ | $[5,512]$ |\n",
    "| 15 | Split $Q$ | $[5,512]$ | $[4,5,128]$ |\n",
    "| 16 | Split $K$ | $[5,512]$ | $[4,5,128]$ |\n",
    "| 17 | Split $V$ | $[5,512]$ | $[4,5,128]$ |\n",
    "| 18 | One Query Head | — | $[5,128]$ |\n",
    "| 19 | One Key Head | — | $[5,128]$ |\n",
    "| 20 | One Value Head | — | $[5,128]$ |\n",
    "| 21 | $K^T$ per Head | $[5,128]$ | $[128,5]$ |\n",
    "| 22 | $QK^T$ per Head | $[5,128]\\times[128,5]$ | $[5,5]$ |\n",
    "| 23 | Attention Scores | 4 heads | $[4,5,5]$ |\n",
    "| 24 | Scaled Scores | $[4,5,5]$ | $[4,5,5]$ |\n",
    "| 25 | Softmax Weights | $[4,5,5]$ | $[4,5,5]$ |\n",
    "| 26 | Context per Head | $[5,5]\\times[5,128]$ | $[5,128]$ |\n",
    "| 27 | All Context Heads | 4 heads | $[4,5,128]$ |\n",
    "| 28 | Concatenated Heads | $[4,5,128]$ | $[5,512]$ |\n",
    "| 29 | $W_O$ | $[512,512]$ | — |\n",
    "| 30 | $b_O$ | $[512]$ | — |\n",
    "| 31 | Output Projection | $[5,512]\\times[512,512]$ | $[5,512]$ |\n",
    "| 32 | First Residual | $[5,512]+[5,512]$ | $[5,512]$ |\n",
    "| 33 | LayerNorm $\\gamma_1$ | $[512]$ | — |\n",
    "| 34 | LayerNorm $\\beta_1$ | $[512]$ | — |\n",
    "| 35 | First LayerNorm | $[5,512]$ | $[5,512]$ |\n",
    "| 36 | FFN $W_1$ | $[512,2048]$ | — |\n",
    "| 37 | FFN $b_1$ | $[2048]$ | — |\n",
    "| 38 | FFN Linear 1 | $[5,512]\\times[512,2048]$ | $[5,2048]$ |\n",
    "| 39 | ReLU | $[5,2048]$ | $[5,2048]$ |\n",
    "| 40 | FFN $W_2$ | $[2048,512]$ | — |\n",
    "| 41 | FFN $b_2$ | $[512]$ | — |\n",
    "| 42 | FFN Linear 2 | $[5,2048]\\times[2048,512]$ | $[5,512]$ |\n",
    "| 43 | Second Residual | $[5,512]+[5,512]$ | $[5,512]$ |\n",
    "| 44 | LayerNorm $\\gamma_2$ | $[512]$ | — |\n",
    "| 45 | LayerNorm $\\beta_2$ | $[512]$ | — |\n",
    "| 46 | Second LayerNorm | $[5,512]$ | $[5,512]$ |\n",
    "| 47 | **Final Encoder Output** | — | **$[5,512]$** |\n",
    "\n",
    "---\n",
    "\n",
    "# Complete shape flow in one line\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[5]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[4,5,128]\n",
    "\\rightarrow\n",
    "[4,5,5]\n",
    "\\rightarrow\n",
    "[4,5,5]\n",
    "\\rightarrow\n",
    "[4,5,128]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,2048]\n",
    "\\rightarrow\n",
    "[5,2048]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "Or with component names:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\begin{aligned}\n",
    "\\text{Token IDs}\n",
    "&:[5] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{Embeddings}\n",
    "&:[5,512] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{+ Position}\n",
    "&:[5,512] \\\\\n",
    "\\downarrow \\\\\n",
    "Q,K,V\n",
    "&:[5,512] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{Split Heads}\n",
    "&:[4,5,128] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{Attention Scores}\n",
    "&:[4,5,5] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{Softmax}\n",
    "&:[4,5,5] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{Context Heads}\n",
    "&:[4,5,128] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{Concat}\n",
    "&:[5,512] \\\\\n",
    "\\downarrow \\\\\n",
    "W_O\n",
    "&:[5,512] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{Add + Norm}\n",
    "&:[5,512] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{FFN Expand}\n",
    "&:[5,2048] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{ReLU}\n",
    "&:[5,2048] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{FFN Project Back}\n",
    "&:[5,512] \\\\\n",
    "\\downarrow \\\\\n",
    "\\text{Add + Norm}\n",
    "&:[5,512]\n",
    "\\end{aligned}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Most important dimensions to remember\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X = [5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q=K=V=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q_h=K_h=V_h=[4,5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q_hK_h^T=[4,5,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Softmax Weights}=[4,5,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Context Heads}=[4,5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Concatenated Heads}=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{MHA Output}=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{FFN Hidden}=[5,2048]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{FFN Output}=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Final Encoder Output}=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## Final takeaway\n",
    "\n",
    "The encoder starts with:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "5 \\text{ token vectors}\n",
    "\\times\n",
    "512 \\text{ features}\n",
    "}\n",
    "$$\n",
    "\n",
    "and finishes with:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "5 \\text{ contextual token vectors}\n",
    "\\times\n",
    "512 \\text{ features}\n",
    "}\n",
    "$$\n",
    "\n",
    "So the encoder does not need to change the main tensor shape:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "Instead, it changes **what those 512 numbers represent**.\n",
    "\n",
    "At the beginning, each vector mainly contains:\n",
    "\n",
    "$$\n",
    "\\text{token identity}\n",
    "+\n",
    "\\text{position}\n",
    "$$\n",
    "\n",
    "At the end, each vector contains a much richer:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{context-aware representation of that token}\n",
    "}\n",
    "$$\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7eacf26d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-03T18:57:23.310538Z",
     "iopub.status.busy": "2026-10-03T18:57:23.310365Z",
     "iopub.status.idle": "2026-10-03T18:57:23.317162Z",
     "shell.execute_reply": "2026-10-03T18:57:23.316814Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">          End-to-End Shape Audit          </span>\n",
       "╭──────────────────────────┬─────────────╮\n",
       "│<span style=\"font-weight: bold\"> Stage                    </span>│<span style=\"font-weight: bold\"> Shape       </span>│\n",
       "├──────────────────────────┼─────────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Token IDs                </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5,)        </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Token embeddings         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Positional encoding      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Embedding + position     </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Q                        </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> K                        </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> V                        </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Q heads                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (4, 5, 128) </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> K heads                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (4, 5, 128) </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> V heads                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (4, 5, 128) </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Attention scores         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (4, 5, 5)   </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Attention weights        </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (4, 5, 5)   </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Context heads            </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (4, 5, 128) </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Concatenated heads       </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Attention output         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> After attention Add+Norm </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> FFN hidden               </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 2048)   </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> FFN output               </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Final encoder output     </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\"> (5, 512)    </span>│\n",
       "╰──────────────────────────┴─────────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m          End-to-End Shape Audit          \u001b[0m\n",
       "╭──────────────────────────┬─────────────╮\n",
       "│\u001b[1m \u001b[0m\u001b[1mStage                   \u001b[0m\u001b[1m \u001b[0m│\u001b[1m \u001b[0m\u001b[1mShape      \u001b[0m\u001b[1m \u001b[0m│\n",
       "├──────────────────────────┼─────────────┤\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mToken IDs               \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5,)       \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mToken embeddings        \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mPositional encoding     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mEmbedding + position    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mQ                       \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mK                       \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mV                       \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mQ heads                 \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(4, 5, 128)\u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mK heads                 \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(4, 5, 128)\u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mV heads                 \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(4, 5, 128)\u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mAttention scores        \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(4, 5, 5)  \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mAttention weights       \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(4, 5, 5)  \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mContext heads           \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(4, 5, 128)\u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mConcatenated heads      \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mAttention output        \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mAfter attention Add+Norm\u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mFFN hidden              \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 2048)  \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mFFN output              \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36mFinal encoder output    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;37m \u001b[0m\u001b[1;37m(5, 512)   \u001b[0m\u001b[1;37m \u001b[0m│\n",
       "╰──────────────────────────┴─────────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "audit = Table(title=\"End-to-End Shape Audit\", box=box.ROUNDED)\n",
    "audit.add_column(\"Stage\", style=\"bold cyan\")\n",
    "audit.add_column(\"Shape\", style=\"bold white\")\n",
    "\n",
    "layer0 = factory_details[\"layers\"][0]\n",
    "shape_rows = [\n",
    "    (\"Token IDs\", tuple(factory_ids.shape)),\n",
    "    (\"Token embeddings\", tuple(factory_details[\"token_embeddings\"].shape)),\n",
    "    (\"Positional encoding\", tuple(factory_details[\"positional_encoding\"].shape)),\n",
    "    (\"Embedding + position\", tuple(factory_details[\"embedding_plus_position\"].shape)),\n",
    "    (\"Q\", tuple(layer0[\"Q\"].shape)),\n",
    "    (\"K\", tuple(layer0[\"K\"].shape)),\n",
    "    (\"V\", tuple(layer0[\"V\"].shape)),\n",
    "    (\"Q heads\", tuple(layer0[\"Qh\"].shape)),\n",
    "    (\"K heads\", tuple(layer0[\"Kh\"].shape)),\n",
    "    (\"V heads\", tuple(layer0[\"Vh\"].shape)),\n",
    "    (\"Attention scores\", tuple(layer0[\"scores\"].shape)),\n",
    "    (\"Attention weights\", tuple(layer0[\"weights\"].shape)),\n",
    "    (\"Context heads\", tuple(layer0[\"context_heads\"].shape)),\n",
    "    (\"Concatenated heads\", tuple(layer0[\"concatenated\"].shape)),\n",
    "    (\"Attention output\", tuple(layer0[\"attention_output\"].shape)),\n",
    "    (\"After attention Add+Norm\", tuple(layer0[\"after_attention_norm\"].shape)),\n",
    "    (\"FFN hidden\", tuple(layer0[\"ffn_hidden\"].shape)),\n",
    "    (\"FFN output\", tuple(layer0[\"ffn_output\"].shape)),\n",
    "    (\"Final encoder output\", tuple(factory_output.shape)),\n",
    "]\n",
    "for stage, shape in shape_rows:\n",
    "    audit.add_row(stage, str(shape))\n",
    "console.print(audit)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "737ed20f",
   "metadata": {},
   "source": [
    "#  PyTorch building blocks used in our encoder\n",
    "\n",
    "Before the final conclusion, it is useful to summarize **exactly which PyTorch components we used**, why we used them, and where they appear in our Transformer encoder.\n",
    "\n",
    "An important point is that we intentionally built most of the Transformer mathematics ourselves.\n",
    "\n",
    "We did **not** use ready-made Transformer components such as:\n",
    "\n",
    "```python\n",
    "nn.Embedding\n",
    "nn.Linear\n",
    "nn.LayerNorm\n",
    "nn.MultiheadAttention\n",
    "nn.TransformerEncoderLayer\n",
    "nn.TransformerEncoder\n",
    "```\n",
    "\n",
    "Instead, we used basic PyTorch building blocks and manually implemented:\n",
    "\n",
    "- embedding lookup\n",
    "- Linear Layers\n",
    "- Q, K, V projections\n",
    "- Multi-Head Attention\n",
    "- LayerNorm\n",
    "- FFN\n",
    "- residual connections\n",
    "- encoder layers\n",
    "\n",
    "So our implementation sits at a useful educational level:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{PyTorch tensor operations}\n",
    "+\n",
    "\\text{manual Transformer mathematics}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# PyTorch imports used\n",
    "\n",
    "At the beginning of the notebook we imported:\n",
    "\n",
    "```python\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "```\n",
    "\n",
    "The main components actually used are:\n",
    "\n",
    "- `torch`\n",
    "- `torch.nn`\n",
    "- `nn.Module`\n",
    "- `nn.Parameter`\n",
    "- `nn.ModuleList`\n",
    "- `nn.init.xavier_uniform_`\n",
    "- basic tensor creation and mathematical operations\n",
    "\n",
    "`torch.nn.functional as F` was imported, but our current implementation does not actually need it because we directly use operations such as:\n",
    "\n",
    "```python\n",
    "torch.relu(...)\n",
    "torch.softmax(...)\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# 1. `nn.Module`\n",
    "\n",
    "The most important PyTorch class we used is:\n",
    "\n",
    "```python\n",
    "nn.Module\n",
    "```\n",
    "\n",
    "It is the base class for neural-network components in PyTorch.\n",
    "\n",
    "Our custom classes inherit from it:\n",
    "\n",
    "```python\n",
    "class ManualEmbedding(nn.Module):\n",
    "```\n",
    "\n",
    "```python\n",
    "class ManualLinear(nn.Module):\n",
    "```\n",
    "\n",
    "```python\n",
    "class ManualFeedForward(nn.Module):\n",
    "```\n",
    "\n",
    "```python\n",
    "class ManualMultiHeadSelfAttention(nn.Module):\n",
    "```\n",
    "\n",
    "```python\n",
    "class EncoderLayerFromScratch(nn.Module):\n",
    "```\n",
    "\n",
    "```python\n",
    "class EncoderFromScratch(nn.Module):\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Why do we use `nn.Module`?\n",
    "\n",
    "It allows PyTorch to understand that our class is part of a neural network.\n",
    "\n",
    "It automatically helps manage:\n",
    "\n",
    "- trainable parameters\n",
    "- nested layers\n",
    "- moving the model to CPU/GPU\n",
    "- saving and loading model weights\n",
    "- training and evaluation modes\n",
    "- recursive parameter collection\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "model.parameters()\n",
    "```\n",
    "\n",
    "can find trainable parameters even when they are deeply nested inside:\n",
    "\n",
    "```text\n",
    "EncoderFromScratch\n",
    "        ↓\n",
    "EncoderLayerFromScratch\n",
    "        ↓\n",
    "ManualMultiHeadSelfAttention\n",
    "        ↓\n",
    "ManualLinear\n",
    "        ↓\n",
    "weight + bias\n",
    "```\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "nn.Module\n",
    "=\n",
    "\\text{foundation of our reusable PyTorch architecture}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# 2. `super().__init__()`\n",
    "\n",
    "Inside every custom `nn.Module`, we call:\n",
    "\n",
    "```python\n",
    "super().__init__()\n",
    "```\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "class ManualLinear(nn.Module):\n",
    "    def __init__(self, in_features, out_features):\n",
    "        super().__init__()\n",
    "```\n",
    "\n",
    "This initializes the parent `nn.Module`.\n",
    "\n",
    "Without it, PyTorch would not correctly register:\n",
    "\n",
    "- parameters\n",
    "- submodules\n",
    "- buffers\n",
    "\n",
    "So every custom PyTorch module should initialize its parent module.\n",
    "\n",
    "---\n",
    "\n",
    "# 3. `nn.Parameter`\n",
    "\n",
    "We used:\n",
    "\n",
    "```python\n",
    "nn.Parameter(...)\n",
    "```\n",
    "\n",
    "whenever we wanted to tell PyTorch:\n",
    "\n",
    "> **\"This tensor is a trainable model parameter.\"**\n",
    "\n",
    "For example, our manually implemented embedding uses:\n",
    "\n",
    "```python\n",
    "self.weight = nn.Parameter(\n",
    "    torch.randn(vocab_size, d_model) * 0.02\n",
    ")\n",
    "```\n",
    "\n",
    "So the embedding matrix is trainable.\n",
    "\n",
    "Its shape is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[|V|,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "- $|V|$ = vocabulary size\n",
    "- $512$ = `d_model`\n",
    "\n",
    "---\n",
    "\n",
    "## `nn.Parameter` in our Linear Layer\n",
    "\n",
    "Our manual Linear Layer uses:\n",
    "\n",
    "```python\n",
    "self.weight = nn.Parameter(\n",
    "    torch.empty(in_features, out_features)\n",
    ")\n",
    "\n",
    "self.bias = nn.Parameter(\n",
    "    torch.zeros(out_features)\n",
    ")\n",
    "```\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "W\n",
    "=\n",
    "[in\\_features,out\\_features]\n",
    "}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "b\n",
    "=\n",
    "[out\\_features]\n",
    "}\n",
    "$$\n",
    "\n",
    "These tensors are automatically included when PyTorch performs:\n",
    "\n",
    "```python\n",
    "model.parameters()\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## `nn.Parameter` in LayerNorm\n",
    "\n",
    "Because we manually implemented LayerNorm, we also manually created its trainable:\n",
    "\n",
    "$$\n",
    "\\gamma\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\beta\n",
    "$$\n",
    "\n",
    "parameters.\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "self.gamma1 = nn.Parameter(torch.ones(d_model))\n",
    "self.beta1 = nn.Parameter(torch.zeros(d_model))\n",
    "```\n",
    "\n",
    "and:\n",
    "\n",
    "```python\n",
    "self.gamma2 = nn.Parameter(torch.ones(d_model))\n",
    "self.beta2 = nn.Parameter(torch.zeros(d_model))\n",
    "```\n",
    "\n",
    "For:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}}=512\n",
    "$$\n",
    "\n",
    "their dimensions are:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\gamma_1,\\beta_1,\\gamma_2,\\beta_2\n",
    "=\n",
    "[512]\n",
    "}\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "nn.Parameter\n",
    "=\n",
    "\\text{make manually created tensors trainable}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# 4. `nn.ModuleList`\n",
    "\n",
    "Inside the complete encoder we use:\n",
    "\n",
    "```python\n",
    "self.layers = nn.ModuleList([\n",
    "    EncoderLayerFromScratch(\n",
    "        d_model,\n",
    "        num_heads,\n",
    "        d_ff\n",
    "    )\n",
    "    for _ in range(num_layers)\n",
    "])\n",
    "```\n",
    "\n",
    "`nn.ModuleList` stores multiple neural-network modules while ensuring PyTorch knows about all of them.\n",
    "\n",
    "For example, if:\n",
    "\n",
    "```python\n",
    "num_layers = 3\n",
    "```\n",
    "\n",
    "then conceptually:\n",
    "\n",
    "```text\n",
    "ModuleList\n",
    "│\n",
    "├── EncoderLayer 1\n",
    "├── EncoderLayer 2\n",
    "└── EncoderLayer 3\n",
    "```\n",
    "\n",
    "Each encoder layer has its own trainable parameters.\n",
    "\n",
    "---\n",
    "\n",
    "## Why not use a normal Python list?\n",
    "\n",
    "A normal list such as:\n",
    "\n",
    "```python\n",
    "self.layers = [\n",
    "    layer1,\n",
    "    layer2,\n",
    "    layer3\n",
    "]\n",
    "```\n",
    "\n",
    "would not be the recommended way to register trainable submodules.\n",
    "\n",
    "Using:\n",
    "\n",
    "```python\n",
    "nn.ModuleList(...)\n",
    "```\n",
    "\n",
    "ensures that PyTorch recursively registers them.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "nn.ModuleList\n",
    "=\n",
    "\\text{registered collection of neural-network modules}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# 5. `nn.init.xavier_uniform_`\n",
    "\n",
    "Our manually implemented Linear Layer initializes its weights using:\n",
    "\n",
    "```python\n",
    "nn.init.xavier_uniform_(self.weight)\n",
    "```\n",
    "\n",
    "This is **Xavier/Glorot initialization**.\n",
    "\n",
    "Instead of starting all weights with zeros, Xavier initialization gives the weights sensible random initial values.\n",
    "\n",
    "---\n",
    "\n",
    "## Why do we need weight initialization?\n",
    "\n",
    "Suppose all weights started at:\n",
    "\n",
    "$$\n",
    "0\n",
    "$$\n",
    "\n",
    "Then neurons could initially behave identically, which is not useful for learning.\n",
    "\n",
    "Instead, we initialize the weights randomly while controlling their scale.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{good initialization}\n",
    "\\rightarrow\n",
    "\\text{better signal flow at the beginning of training}\n",
    "}\n",
    "$$\n",
    "\n",
    "We use Xavier initialization for:\n",
    "\n",
    "- $W_Q$\n",
    "- $W_K$\n",
    "- $W_V$\n",
    "- $W_O$\n",
    "- $W_1$\n",
    "- $W_2$\n",
    "\n",
    "because all of those transformations use our `ManualLinear` class.\n",
    "\n",
    "---\n",
    "\n",
    "# 6. `torch.randn`\n",
    "\n",
    "Our embedding matrix is initialized using:\n",
    "\n",
    "```python\n",
    "torch.randn(vocab_size, d_model)\n",
    "```\n",
    "\n",
    "This generates values from approximately a standard normal distribution.\n",
    "\n",
    "We then scale them:\n",
    "\n",
    "```python\n",
    "torch.randn(vocab_size, d_model) * 0.02\n",
    "```\n",
    "\n",
    "For our embedding:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "E\n",
    "\\in\n",
    "\\mathbb{R}^{|V|\\times512}\n",
    "}\n",
    "$$\n",
    "\n",
    "This gives every vocabulary token an initial random trainable vector.\n",
    "\n",
    "---\n",
    "\n",
    "# 7. `torch.empty`\n",
    "\n",
    "Inside `ManualLinear`, we create the weight matrix using:\n",
    "\n",
    "```python\n",
    "torch.empty(in_features, out_features)\n",
    "```\n",
    "\n",
    "For example, the Query projection uses:\n",
    "\n",
    "$$\n",
    "W_Q\n",
    "\\in\n",
    "\\mathbb{R}^{512\\times512}\n",
    "$$\n",
    "\n",
    "so conceptually:\n",
    "\n",
    "```python\n",
    "torch.empty(512, 512)\n",
    "```\n",
    "\n",
    "`torch.empty()` allocates memory without first assigning meaningful values.\n",
    "\n",
    "We immediately initialize it with:\n",
    "\n",
    "```python\n",
    "nn.init.xavier_uniform_(self.weight)\n",
    "```\n",
    "\n",
    "So:\n",
    "\n",
    "```text\n",
    "torch.empty(...)\n",
    "        ↓\n",
    "allocate tensor\n",
    "        ↓\n",
    "xavier_uniform_(...)\n",
    "        ↓\n",
    "proper initialized weights\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# 8. `torch.zeros`\n",
    "\n",
    "We use:\n",
    "\n",
    "```python\n",
    "torch.zeros(...)\n",
    "```\n",
    "\n",
    "in several places.\n",
    "\n",
    "### Linear bias\n",
    "\n",
    "```python\n",
    "self.bias = nn.Parameter(\n",
    "    torch.zeros(out_features)\n",
    ")\n",
    "```\n",
    "\n",
    "So the bias initially starts at zero.\n",
    "\n",
    "---\n",
    "\n",
    "### LayerNorm beta\n",
    "\n",
    "```python\n",
    "self.beta1 = nn.Parameter(torch.zeros(d_model))\n",
    "```\n",
    "\n",
    "and:\n",
    "\n",
    "```python\n",
    "self.beta2 = nn.Parameter(torch.zeros(d_model))\n",
    "```\n",
    "\n",
    "because LayerNorm initially uses:\n",
    "\n",
    "$$\n",
    "\\beta=0\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### Positional encoding matrix\n",
    "\n",
    "We also create the positional encoding matrix using:\n",
    "\n",
    "```python\n",
    "pe = torch.zeros(seq_len, d_model)\n",
    "```\n",
    "\n",
    "For our example:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "PE=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "Then we fill its even and odd columns with sine and cosine values.\n",
    "\n",
    "---\n",
    "\n",
    "# 9. `torch.ones`\n",
    "\n",
    "LayerNorm's learnable scaling parameter $\\gamma$ is initialized using:\n",
    "\n",
    "```python\n",
    "torch.ones(d_model)\n",
    "```\n",
    "\n",
    "So initially:\n",
    "\n",
    "$$\n",
    "\\gamma=1\n",
    "$$\n",
    "\n",
    "This means LayerNorm initially performs:\n",
    "\n",
    "$$\n",
    "1\\cdot\\hat{x}+0\n",
    "$$\n",
    "\n",
    "because:\n",
    "\n",
    "$$\n",
    "\\gamma=1\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\beta=0\n",
    "$$\n",
    "\n",
    "Later, both can be learned during training.\n",
    "\n",
    "---\n",
    "\n",
    "# 10. `torch.tensor`\n",
    "\n",
    "Our tokenizer produces Python token IDs.\n",
    "\n",
    "We convert them into a PyTorch tensor using:\n",
    "\n",
    "```python\n",
    "torch.tensor(\n",
    "    [...],\n",
    "    dtype=torch.long,\n",
    "    device=device\n",
    ")\n",
    "```\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "token_ids = torch.tensor(\n",
    "    [2, 3, 4, 2, 5],\n",
    "    dtype=torch.long\n",
    ")\n",
    "```\n",
    "\n",
    "The shape is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[5]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## Why `dtype=torch.long`?\n",
    "\n",
    "Token IDs are integer indices.\n",
    "\n",
    "Embedding lookup requires integer indices, so we use:\n",
    "\n",
    "```python\n",
    "dtype=torch.long\n",
    "```\n",
    "\n",
    "instead of floating-point values.\n",
    "\n",
    "---\n",
    "\n",
    "# 11. `torch.arange`\n",
    "\n",
    "For positional encoding, we need token positions:\n",
    "\n",
    "$$\n",
    "0,1,2,3,4\n",
    "$$\n",
    "\n",
    "We create them using:\n",
    "\n",
    "```python\n",
    "torch.arange(seq_len)\n",
    "```\n",
    "\n",
    "For our sentence:\n",
    "\n",
    "```text\n",
    "[0, 1, 2, 3, 4]\n",
    "```\n",
    "\n",
    "We also use:\n",
    "\n",
    "```python\n",
    "torch.arange(0, d_model, 2)\n",
    "```\n",
    "\n",
    "to generate the even embedding dimensions used in the sinusoidal positional-encoding formula.\n",
    "\n",
    "So `torch.arange()` is used to create regularly spaced integer positions.\n",
    "\n",
    "---\n",
    "\n",
    "# 12. `.unsqueeze()`\n",
    "\n",
    "The positional vector is created as:\n",
    "\n",
    "```python\n",
    "position = torch.arange(seq_len).unsqueeze(1)\n",
    "```\n",
    "\n",
    "Initially:\n",
    "\n",
    "$$\n",
    "[5]\n",
    "$$\n",
    "\n",
    "After:\n",
    "\n",
    "```python\n",
    "unsqueeze(1)\n",
    "```\n",
    "\n",
    "we get:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[5,1]\n",
    "}\n",
    "$$\n",
    "\n",
    "This makes broadcasting possible when multiplying position values with the sinusoidal frequency terms.\n",
    "\n",
    "---\n",
    "\n",
    "# 13. `torch.exp`\n",
    "\n",
    "The sinusoidal positional encoding requires different frequencies across embedding dimensions.\n",
    "\n",
    "We calculate:\n",
    "\n",
    "```python\n",
    "div_term = torch.exp(...)\n",
    "```\n",
    "\n",
    "This helps implement the equivalent of:\n",
    "\n",
    "$$\n",
    "\\frac{1}\n",
    "{10000^{2i/d_{\\text{model}}}}\n",
    "$$\n",
    "\n",
    "used in:\n",
    "\n",
    "$$\n",
    "PE(pos,2i)\n",
    "=\n",
    "\\sin\n",
    "\\left(\n",
    "\\frac{pos}\n",
    "{10000^{2i/d_{\\text{model}}}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "PE(pos,2i+1)\n",
    "=\n",
    "\\cos\n",
    "\\left(\n",
    "\\frac{pos}\n",
    "{10000^{2i/d_{\\text{model}}}}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# 14. `torch.sin`\n",
    "\n",
    "For even positional-encoding dimensions we use:\n",
    "\n",
    "```python\n",
    "torch.sin(position * div_term)\n",
    "```\n",
    "\n",
    "which implements:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "PE(pos,2i)\n",
    "=\n",
    "\\sin\n",
    "\\left(\n",
    "\\frac{pos}\n",
    "{10000^{2i/d_{\\text{model}}}}\n",
    "\\right)\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# 15. `torch.cos`\n",
    "\n",
    "For odd dimensions we use:\n",
    "\n",
    "```python\n",
    "torch.cos(position * div_term)\n",
    "```\n",
    "\n",
    "which implements:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "PE(pos,2i+1)\n",
    "=\n",
    "\\cos\n",
    "\\left(\n",
    "\\frac{pos}\n",
    "{10000^{2i/d_{\\text{model}}}}\n",
    "\\right)\n",
    "}\n",
    "$$\n",
    "\n",
    "Together:\n",
    "\n",
    "```text\n",
    "torch.sin()\n",
    "     +\n",
    "torch.cos()\n",
    "     ↓\n",
    "Sinusoidal Positional Encoding\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# 16. Matrix multiplication with `@`\n",
    "\n",
    "Our manual Linear Layer performs:\n",
    "\n",
    "```python\n",
    "return x @ self.weight + self.bias\n",
    "```\n",
    "\n",
    "The `@` operator performs matrix multiplication.\n",
    "\n",
    "For Query:\n",
    "\n",
    "$$\n",
    "X\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "W_Q\n",
    "=\n",
    "[512,512]\n",
    "$$\n",
    "\n",
    "therefore:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "@\n",
    "[512,512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "We use this same `ManualLinear` operation for:\n",
    "\n",
    "- Query projection\n",
    "- Key projection\n",
    "- Value projection\n",
    "- Output projection\n",
    "- FFN Linear 1\n",
    "- FFN Linear 2\n",
    "\n",
    "---\n",
    "\n",
    "# 17. `torch.matmul`\n",
    "\n",
    "Inside attention, we explicitly use:\n",
    "\n",
    "```python\n",
    "torch.matmul(...)\n",
    "```\n",
    "\n",
    "First:\n",
    "\n",
    "```python\n",
    "scores = torch.matmul(\n",
    "    Qh,\n",
    "    Kh.transpose(-2, -1)\n",
    ")\n",
    "```\n",
    "\n",
    "This calculates:\n",
    "\n",
    "$$\n",
    "QK^T\n",
    "$$\n",
    "\n",
    "Dimensions:\n",
    "\n",
    "$$\n",
    "[4,5,128]\n",
    "\\times\n",
    "[4,128,5]\n",
    "\\rightarrow\n",
    "[4,5,5]\n",
    "$$\n",
    "\n",
    "Then we use `torch.matmul()` again:\n",
    "\n",
    "```python\n",
    "context = torch.matmul(\n",
    "    attention_weights,\n",
    "    Vh\n",
    ")\n",
    "```\n",
    "\n",
    "Dimensions:\n",
    "\n",
    "$$\n",
    "[4,5,5]\n",
    "\\times\n",
    "[4,5,128]\n",
    "\\rightarrow\n",
    "[4,5,128]\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "torch.matmul\n",
    "=\n",
    "\\text{core attention matrix multiplication}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# 18. `.transpose()`\n",
    "\n",
    "We use transpose in several places.\n",
    "\n",
    "For attention:\n",
    "\n",
    "```python\n",
    "Kh.transpose(-2, -1)\n",
    "```\n",
    "\n",
    "changes:\n",
    "\n",
    "$$\n",
    "[4,5,128]\n",
    "$$\n",
    "\n",
    "into:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[4,128,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "so that:\n",
    "\n",
    "$$\n",
    "QK^T\n",
    "$$\n",
    "\n",
    "can be calculated.\n",
    "\n",
    "---\n",
    "\n",
    "## Transpose when splitting heads\n",
    "\n",
    "We also use:\n",
    "\n",
    "```python\n",
    "x.transpose(0, 1)\n",
    "```\n",
    "\n",
    "to transform:\n",
    "\n",
    "$$\n",
    "[5,4,128]\n",
    "$$\n",
    "\n",
    "into:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[4,5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# 19. `.reshape()`\n",
    "\n",
    "To split the model dimension into multiple heads we use:\n",
    "\n",
    "```python\n",
    "x.reshape(\n",
    "    n,\n",
    "    num_heads,\n",
    "    d_head\n",
    ")\n",
    "```\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "becomes:\n",
    "\n",
    "$$\n",
    "[5,4,128]\n",
    "$$\n",
    "\n",
    "because:\n",
    "\n",
    "$$\n",
    "512=4\\times128\n",
    "$$\n",
    "\n",
    "Then after attention we use reshape again to combine the heads:\n",
    "\n",
    "$$\n",
    "[5,4,128]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# 20. `.contiguous()`\n",
    "\n",
    "When combining heads we use:\n",
    "\n",
    "```python\n",
    "x.transpose(0, 1).contiguous().reshape(...)\n",
    "```\n",
    "\n",
    "After transpose, a tensor may have a different underlying memory layout.\n",
    "\n",
    "`.contiguous()` creates a tensor whose values are stored continuously in the expected memory order.\n",
    "\n",
    "This makes the following `reshape()` safe and predictable.\n",
    "\n",
    "So:\n",
    "\n",
    "```text\n",
    "Transpose\n",
    "   ↓\n",
    "contiguous()\n",
    "   ↓\n",
    "reshape()\n",
    "```\n",
    "\n",
    "is a common PyTorch pattern.\n",
    "\n",
    "---\n",
    "\n",
    "# 21. `torch.softmax`\n",
    "\n",
    "After calculating scaled attention scores, we use:\n",
    "\n",
    "```python\n",
    "attention_weights = torch.softmax(\n",
    "    scores,\n",
    "    dim=-1\n",
    ")\n",
    "```\n",
    "\n",
    "Softmax converts the scores into attention probabilities.\n",
    "\n",
    "For every Query token:\n",
    "\n",
    "$$\n",
    "\\sum_j A_{ij}=1\n",
    "$$\n",
    "\n",
    "For example:\n",
    "\n",
    "$$\n",
    "[2.1,0.7,1.4]\n",
    "$$\n",
    "\n",
    "might become something like:\n",
    "\n",
    "$$\n",
    "[0.58,0.14,0.28]\n",
    "$$\n",
    "\n",
    "The shape remains:\n",
    "\n",
    "$$\n",
    "[4,5,5]\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "torch.softmax\n",
    "=\n",
    "\\text{convert attention scores into attention weights}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## Why `dim=-1`?\n",
    "\n",
    "Our attention matrix has:\n",
    "\n",
    "$$\n",
    "[heads,query,key]\n",
    "$$\n",
    "\n",
    "or:\n",
    "\n",
    "$$\n",
    "[4,5,5]\n",
    "$$\n",
    "\n",
    "The last dimension represents Key tokens.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "```python\n",
    "dim=-1\n",
    "```\n",
    "\n",
    "means:\n",
    "\n",
    "> Normalize the scores across all Key tokens for each Query.\n",
    "\n",
    "---\n",
    "\n",
    "# 22. `torch.relu`\n",
    "\n",
    "Inside the Feed-Forward Network we use:\n",
    "\n",
    "```python\n",
    "hidden = torch.relu(hidden_pre_activation)\n",
    "```\n",
    "\n",
    "ReLU is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{ReLU}(x)=\\max(0,x)\n",
    "}\n",
    "$$\n",
    "\n",
    "For example:\n",
    "\n",
    "$$\n",
    "[-2,3,-1,5]\n",
    "$$\n",
    "\n",
    "becomes:\n",
    "\n",
    "$$\n",
    "[0,3,0,5]\n",
    "$$\n",
    "\n",
    "It introduces non-linearity between the two Linear Layers.\n",
    "\n",
    "Our FFN therefore performs:\n",
    "\n",
    "$$\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "\\rightarrow\n",
    "\\operatorname{ReLU}\n",
    "\\rightarrow\n",
    "512\n",
    "$$\n",
    "\n",
    "More precisely:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{FFN}(x)\n",
    "=\n",
    "\\operatorname{ReLU}(xW_1+b_1)W_2+b_2\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# 23. `.mean()`\n",
    "\n",
    "Our manually implemented LayerNorm calculates:\n",
    "\n",
    "```python\n",
    "mean = x.mean(\n",
    "    dim=-1,\n",
    "    keepdim=True\n",
    ")\n",
    "```\n",
    "\n",
    "This computes the mean across each token's feature dimension.\n",
    "\n",
    "For:\n",
    "\n",
    "$$\n",
    "X=[5,512]\n",
    "$$\n",
    "\n",
    "LayerNorm calculates a separate mean for each of the 5 tokens across its 512 features.\n",
    "\n",
    "---\n",
    "\n",
    "# 24. `.var()`\n",
    "\n",
    "We calculate LayerNorm variance using:\n",
    "\n",
    "```python\n",
    "var = x.var(\n",
    "    dim=-1,\n",
    "    unbiased=False,\n",
    "    keepdim=True\n",
    ")\n",
    "```\n",
    "\n",
    "This implements:\n",
    "\n",
    "$$\n",
    "\\sigma^2\n",
    "=\n",
    "\\frac{1}{d_{\\text{model}}}\n",
    "\\sum_i(x_i-\\mu)^2\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## Why `unbiased=False`?\n",
    "\n",
    "For LayerNorm we want the population-style variance over the features:\n",
    "\n",
    "$$\n",
    "\\frac{1}{N}\n",
    "\\sum(x_i-\\mu)^2\n",
    "$$\n",
    "\n",
    "rather than the sample variance that divides by:\n",
    "\n",
    "$$\n",
    "N-1\n",
    "$$\n",
    "\n",
    "So we use:\n",
    "\n",
    "```python\n",
    "unbiased=False\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# 25. `torch.sqrt`\n",
    "\n",
    "Our LayerNorm contains:\n",
    "\n",
    "```python\n",
    "torch.sqrt(var + eps)\n",
    "```\n",
    "\n",
    "which implements:\n",
    "\n",
    "$$\n",
    "\\sqrt{\\sigma^2+\\epsilon}\n",
    "$$\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\hat{x}\n",
    "=\n",
    "\\frac{x-\\mu}\n",
    "{\\sqrt{\\sigma^2+\\epsilon}}\n",
    "}\n",
    "$$\n",
    "\n",
    "We use:\n",
    "\n",
    "```python\n",
    "eps = 1e-5\n",
    "```\n",
    "\n",
    "for numerical stability.\n",
    "\n",
    "---\n",
    "\n",
    "# 26. `.sum()`\n",
    "\n",
    "We use:\n",
    "\n",
    "```python\n",
    "attention_weights[h].sum(dim=-1)\n",
    "```\n",
    "\n",
    "to verify our Softmax output.\n",
    "\n",
    "Each attention row should approximately satisfy:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\sum_j A_{ij}=1\n",
    "}\n",
    "$$\n",
    "\n",
    "This is primarily a debugging and educational sanity check.\n",
    "\n",
    "---\n",
    "\n",
    "# 27. `torch.allclose`\n",
    "\n",
    "We use:\n",
    "\n",
    "```python\n",
    "torch.allclose(...)\n",
    "```\n",
    "\n",
    "to compare tensors.\n",
    "\n",
    "For example, before positional encoding:\n",
    "\n",
    "```python\n",
    "torch.allclose(\n",
    "    token_embeddings[0],\n",
    "    token_embeddings[3]\n",
    ")\n",
    "```\n",
    "\n",
    "checks whether the two occurrences of:\n",
    "\n",
    "> **\"the\"**\n",
    "\n",
    "have the same embedding.\n",
    "\n",
    "Initially they do.\n",
    "\n",
    "After positional encoding, we perform another comparison and observe that the vectors differ because:\n",
    "\n",
    "$$\n",
    "PE_0\\neq PE_3\n",
    "$$\n",
    "\n",
    "So this operation is used for educational verification.\n",
    "\n",
    "---\n",
    "\n",
    "# 28. `torch.manual_seed`\n",
    "\n",
    "We use:\n",
    "\n",
    "```python\n",
    "torch.manual_seed(7)\n",
    "```\n",
    "\n",
    "and inside our factory:\n",
    "\n",
    "```python\n",
    "torch.manual_seed(seed)\n",
    "```\n",
    "\n",
    "This controls PyTorch's random-number generation.\n",
    "\n",
    "It makes random initialization reproducible.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{same seed}\n",
    "+\n",
    "\\text{same configuration}\n",
    "\\rightarrow\n",
    "\\text{same initial random parameters}\n",
    "}\n",
    "$$\n",
    "\n",
    "This is especially useful for educational notebooks and debugging.\n",
    "\n",
    "---\n",
    "\n",
    "# 29. `torch.device`\n",
    "\n",
    "We define:\n",
    "\n",
    "```python\n",
    "device = torch.device(\n",
    "    \"cuda\"\n",
    "    if torch.cuda.is_available()\n",
    "    else \"cpu\"\n",
    ")\n",
    "```\n",
    "\n",
    "This determines where tensors and model parameters are stored.\n",
    "\n",
    "Possible examples:\n",
    "\n",
    "```text\n",
    "cpu\n",
    "```\n",
    "\n",
    "or:\n",
    "\n",
    "```text\n",
    "cuda\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# 30. `torch.cuda.is_available()`\n",
    "\n",
    "We use:\n",
    "\n",
    "```python\n",
    "torch.cuda.is_available()\n",
    "```\n",
    "\n",
    "to check whether a CUDA-compatible GPU is available.\n",
    "\n",
    "If yes:\n",
    "\n",
    "```python\n",
    "device = \"cuda\"\n",
    "```\n",
    "\n",
    "otherwise:\n",
    "\n",
    "```python\n",
    "device = \"cpu\"\n",
    "```\n",
    "\n",
    "This makes the notebook portable across CPU and GPU systems.\n",
    "\n",
    "---\n",
    "\n",
    "# 31. `.to(device)`\n",
    "\n",
    "We move our modules to the selected device using:\n",
    "\n",
    "```python\n",
    "model.to(device)\n",
    "```\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "embedding_layer = ManualEmbedding(...).to(device)\n",
    "```\n",
    "\n",
    "and:\n",
    "\n",
    "```python\n",
    "model = EncoderFromScratch(...).to(device)\n",
    "```\n",
    "\n",
    "This moves trainable parameters to:\n",
    "\n",
    "- CPU, or\n",
    "- GPU\n",
    "\n",
    "depending on the selected device.\n",
    "\n",
    "---\n",
    "\n",
    "# 32. `.detach()`\n",
    "\n",
    "For visualization and debugging we sometimes use:\n",
    "\n",
    "```python\n",
    "tensor.detach()\n",
    "```\n",
    "\n",
    "A tensor produced during model computation may be connected to PyTorch's automatic-gradient graph.\n",
    "\n",
    "`.detach()` creates a view of the tensor that is no longer tracked for gradient computation.\n",
    "\n",
    "This is useful when we only want to:\n",
    "\n",
    "- print values\n",
    "- inspect them\n",
    "- move them to CPU\n",
    "- display them with Rich\n",
    "\n",
    "and do not need gradients.\n",
    "\n",
    "---\n",
    "\n",
    "# 33. `.cpu()`\n",
    "\n",
    "For display operations we use:\n",
    "\n",
    "```python\n",
    "tensor.cpu()\n",
    "```\n",
    "\n",
    "This moves the tensor to CPU memory.\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "row_sums = (\n",
    "    attention_weights[h]\n",
    "    .sum(dim=-1)\n",
    "    .detach()\n",
    "    .cpu()\n",
    ")\n",
    "```\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "```text\n",
    "Tensor used by model\n",
    "       ↓\n",
    "detach()\n",
    "       ↓\n",
    "No gradient tracking\n",
    "       ↓\n",
    "cpu()\n",
    "       ↓\n",
    "Easy Python / Rich inspection\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# 34. `.tolist()`\n",
    "\n",
    "We use:\n",
    "\n",
    "```python\n",
    "token_ids.tolist()\n",
    "```\n",
    "\n",
    "to convert a PyTorch tensor into a regular Python list.\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "tensor([2, 3, 4, 2, 5])\n",
    "```\n",
    "\n",
    "becomes:\n",
    "\n",
    "```python\n",
    "[2, 3, 4, 2, 5]\n",
    "```\n",
    "\n",
    "This is mostly used for display.\n",
    "\n",
    "---\n",
    "\n",
    "# 35. `register_buffer()`\n",
    "\n",
    "Inside our complete encoder we use:\n",
    "\n",
    "```python\n",
    "self.register_buffer(\n",
    "    \"positional_encoding\",\n",
    "    positional_encoding,\n",
    "    persistent=False,\n",
    ")\n",
    "```\n",
    "\n",
    "This is an important PyTorch feature.\n",
    "\n",
    "Our sinusoidal positional encoding is:\n",
    "\n",
    "- part of the model\n",
    "- needed during the forward pass\n",
    "- not trainable\n",
    "\n",
    "Therefore it should **not** be an `nn.Parameter`.\n",
    "\n",
    "Instead, we register it as a **buffer**.\n",
    "\n",
    "---\n",
    "\n",
    "## Parameter vs Buffer\n",
    "\n",
    "A parameter is trainable:\n",
    "\n",
    "```python\n",
    "nn.Parameter(...)\n",
    "```\n",
    "\n",
    "Example:\n",
    "\n",
    "$$\n",
    "W_Q\n",
    "$$\n",
    "\n",
    "A buffer belongs to the model but is not trainable:\n",
    "\n",
    "```python\n",
    "register_buffer(...)\n",
    "```\n",
    "\n",
    "Example:\n",
    "\n",
    "$$\n",
    "PE\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Parameter}\n",
    "=\n",
    "\\text{stored + trained}\n",
    "}\n",
    "$$\n",
    "\n",
    "while:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Buffer}\n",
    "=\n",
    "\\text{stored with model + not trained}\n",
    "}\n",
    "$$\n",
    "\n",
    "This is exactly appropriate for fixed sinusoidal positional encoding.\n",
    "\n",
    "---\n",
    "\n",
    "# 36. Tensor indexing\n",
    "\n",
    "Our manual embedding lookup is simply:\n",
    "\n",
    "```python\n",
    "return self.weight[token_ids]\n",
    "```\n",
    "\n",
    "Suppose:\n",
    "\n",
    "$$\n",
    "E\n",
    "=\n",
    "[|V|,512]\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "token\\_ids=[5]\n",
    "$$\n",
    "\n",
    "then:\n",
    "\n",
    "```python\n",
    "self.weight[token_ids]\n",
    "```\n",
    "\n",
    "selects 5 rows from the embedding matrix.\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "[5]\n",
    "\\rightarrow\n",
    "\\boxed{\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "This is how we manually reproduce the basic behavior of `nn.Embedding`.\n",
    "\n",
    "---\n",
    "\n",
    "# 37. Tensor slicing\n",
    "\n",
    "When running the encoder we use:\n",
    "\n",
    "```python\n",
    "pe = self.positional_encoding[:n]\n",
    "```\n",
    "\n",
    "If the stored positional encoding is:\n",
    "\n",
    "$$\n",
    "[128,512]\n",
    "$$\n",
    "\n",
    "but our sentence has only:\n",
    "\n",
    "$$\n",
    "n=5\n",
    "$$\n",
    "\n",
    "tokens, we select only:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "using:\n",
    "\n",
    "```python\n",
    "[:5]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# 38. Broadcasting\n",
    "\n",
    "Several operations in our encoder rely on PyTorch broadcasting.\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "x @ weight + bias\n",
    "```\n",
    "\n",
    "Suppose:\n",
    "\n",
    "$$\n",
    "xW\n",
    "=\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "bias\n",
    "=\n",
    "[512]\n",
    "$$\n",
    "\n",
    "PyTorch automatically applies the same bias vector to all 5 token rows:\n",
    "\n",
    "$$\n",
    "[5,512]+[512]\n",
    "\\rightarrow\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "Similarly, LayerNorm parameters:\n",
    "\n",
    "$$\n",
    "\\gamma=[512]\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\beta=[512]\n",
    "$$\n",
    "\n",
    "are automatically applied to every token.\n",
    "\n",
    "---\n",
    "\n",
    "# PyTorch components used in our custom modules\n",
    "\n",
    "We can now map everything directly to our architecture.\n",
    "\n",
    "---\n",
    "\n",
    "## `ManualEmbedding`\n",
    "\n",
    "Our code:\n",
    "\n",
    "```python\n",
    "class ManualEmbedding(nn.Module):\n",
    "```\n",
    "\n",
    "uses:\n",
    "\n",
    "- `nn.Module`\n",
    "- `nn.Parameter`\n",
    "- `torch.randn`\n",
    "- tensor indexing\n",
    "\n",
    "Trainable parameter:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "E=[|V|,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "Purpose:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Token IDs}\n",
    "\\rightarrow\n",
    "\\text{Token vectors}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## `ManualLinear`\n",
    "\n",
    "Our code:\n",
    "\n",
    "```python\n",
    "class ManualLinear(nn.Module):\n",
    "```\n",
    "\n",
    "uses:\n",
    "\n",
    "- `nn.Module`\n",
    "- `nn.Parameter`\n",
    "- `torch.empty`\n",
    "- `torch.zeros`\n",
    "- `nn.init.xavier_uniform_`\n",
    "- matrix multiplication `@`\n",
    "- broadcasting\n",
    "\n",
    "It implements:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "y=xW+b\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## `ManualMultiHeadSelfAttention`\n",
    "\n",
    "Our code:\n",
    "\n",
    "```python\n",
    "class ManualMultiHeadSelfAttention(nn.Module):\n",
    "```\n",
    "\n",
    "contains four `ManualLinear` modules:\n",
    "\n",
    "```python\n",
    "self.q_proj\n",
    "self.k_proj\n",
    "self.v_proj\n",
    "self.o_proj\n",
    "```\n",
    "\n",
    "Their dimensions are:\n",
    "\n",
    "$$\n",
    "W_Q=[512,512]\n",
    "$$\n",
    "\n",
    "$$\n",
    "W_K=[512,512]\n",
    "$$\n",
    "\n",
    "$$\n",
    "W_V=[512,512]\n",
    "$$\n",
    "\n",
    "$$\n",
    "W_O=[512,512]\n",
    "$$\n",
    "\n",
    "It also uses:\n",
    "\n",
    "- `.reshape()`\n",
    "- `.transpose()`\n",
    "- `.contiguous()`\n",
    "- `torch.matmul()`\n",
    "- `torch.softmax()`\n",
    "\n",
    "Purpose:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Learn relationships between tokens}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## `ManualFeedForward`\n",
    "\n",
    "Our code:\n",
    "\n",
    "```python\n",
    "class ManualFeedForward(nn.Module):\n",
    "```\n",
    "\n",
    "contains:\n",
    "\n",
    "```python\n",
    "self.fc1 = ManualLinear(d_model, d_ff)\n",
    "```\n",
    "\n",
    "and:\n",
    "\n",
    "```python\n",
    "self.fc2 = ManualLinear(d_ff, d_model)\n",
    "```\n",
    "\n",
    "For our configuration:\n",
    "\n",
    "### First Linear Layer\n",
    "\n",
    "$$\n",
    "W_1=[512,2048]\n",
    "$$\n",
    "\n",
    "$$\n",
    "b_1=[2048]\n",
    "$$\n",
    "\n",
    "### Second Linear Layer\n",
    "\n",
    "$$\n",
    "W_2=[2048,512]\n",
    "$$\n",
    "\n",
    "$$\n",
    "b_2=[512]\n",
    "$$\n",
    "\n",
    "It also uses:\n",
    "\n",
    "```python\n",
    "torch.relu(...)\n",
    "```\n",
    "\n",
    "Therefore:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "\\rightarrow\n",
    "512\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## `EncoderLayerFromScratch`\n",
    "\n",
    "Our code:\n",
    "\n",
    "```python\n",
    "class EncoderLayerFromScratch(nn.Module):\n",
    "```\n",
    "\n",
    "contains:\n",
    "\n",
    "```python\n",
    "self.self_attn\n",
    "```\n",
    "\n",
    "and:\n",
    "\n",
    "```python\n",
    "self.ffn\n",
    "```\n",
    "\n",
    "plus four manual LayerNorm parameters:\n",
    "\n",
    "```python\n",
    "self.gamma1\n",
    "self.beta1\n",
    "self.gamma2\n",
    "self.beta2\n",
    "```\n",
    "\n",
    "Each has:\n",
    "\n",
    "$$\n",
    "[512]\n",
    "$$\n",
    "\n",
    "The layer implements:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Attention}\n",
    "\\rightarrow\n",
    "\\text{Add + Norm}\n",
    "\\rightarrow\n",
    "\\text{FFN}\n",
    "\\rightarrow\n",
    "\\text{Add + Norm}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## `EncoderFromScratch`\n",
    "\n",
    "Our complete encoder:\n",
    "\n",
    "```python\n",
    "class EncoderFromScratch(nn.Module):\n",
    "```\n",
    "\n",
    "contains:\n",
    "\n",
    "```python\n",
    "self.embedding\n",
    "```\n",
    "\n",
    "```python\n",
    "self.layers\n",
    "```\n",
    "\n",
    "and:\n",
    "\n",
    "```python\n",
    "self.positional_encoding\n",
    "```\n",
    "\n",
    "where:\n",
    "\n",
    "- embedding = `ManualEmbedding`\n",
    "- layers = `nn.ModuleList`\n",
    "- positional encoding = registered buffer\n",
    "\n",
    "So:\n",
    "\n",
    "```text\n",
    "EncoderFromScratch\n",
    "│\n",
    "├── ManualEmbedding\n",
    "│\n",
    "├── Positional Encoding Buffer\n",
    "│\n",
    "└── ModuleList\n",
    "    │\n",
    "    ├── EncoderLayer 1\n",
    "    ├── EncoderLayer 2\n",
    "    ├── ...\n",
    "    └── EncoderLayer N\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# Complete trainable parameters in one encoder layer\n",
    "\n",
    "For one encoder layer with:\n",
    "\n",
    "$$\n",
    "d_{\\text{model}}=512\n",
    "$$\n",
    "\n",
    "$$\n",
    "d_{\\text{ff}}=2048\n",
    "$$\n",
    "\n",
    "we have the following major trainable tensors.\n",
    "\n",
    "| Parameter | Shape | Purpose |\n",
    "|---|---:|---|\n",
    "| $W_Q$ | $[512,512]$ | Query projection |\n",
    "| $b_Q$ | $[512]$ | Query bias |\n",
    "| $W_K$ | $[512,512]$ | Key projection |\n",
    "| $b_K$ | $[512]$ | Key bias |\n",
    "| $W_V$ | $[512,512]$ | Value projection |\n",
    "| $b_V$ | $[512]$ | Value bias |\n",
    "| $W_O$ | $[512,512]$ | Mix concatenated attention heads |\n",
    "| $b_O$ | $[512]$ | Output projection bias |\n",
    "| $\\gamma_1$ | $[512]$ | First LayerNorm scale |\n",
    "| $\\beta_1$ | $[512]$ | First LayerNorm shift |\n",
    "| $W_1$ | $[512,2048]$ | FFN expansion |\n",
    "| $b_1$ | $[2048]$ | FFN first bias |\n",
    "| $W_2$ | $[2048,512]$ | FFN projection back |\n",
    "| $b_2$ | $[512]$ | FFN second bias |\n",
    "| $\\gamma_2$ | $[512]$ | Second LayerNorm scale |\n",
    "| $\\beta_2$ | $[512]$ | Second LayerNorm shift |\n",
    "\n",
    "The embedding matrix is also trainable:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "E=[|V|,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "The sinusoidal positional encoding is **not trainable**.\n",
    "\n",
    "---\n",
    "\n",
    "# Complete PyTorch API summary\n",
    "\n",
    "| PyTorch component | Why we used it |\n",
    "|---|---|\n",
    "| `nn.Module` | Base class for all reusable neural-network modules |\n",
    "| `nn.Parameter` | Register manually created tensors as trainable parameters |\n",
    "| `nn.ModuleList` | Store multiple encoder layers while registering them with PyTorch |\n",
    "| `nn.init.xavier_uniform_` | Initialize Linear Layer weights |\n",
    "| `torch.tensor` | Create token-ID tensors |\n",
    "| `torch.randn` | Initialize embedding weights |\n",
    "| `torch.empty` | Allocate Linear Layer weights before initialization |\n",
    "| `torch.zeros` | Initialize biases, beta, and positional encoding |\n",
    "| `torch.ones` | Initialize LayerNorm gamma |\n",
    "| `torch.arange` | Create positions and positional-encoding dimension indices |\n",
    "| `torch.exp` | Build sinusoidal positional frequencies |\n",
    "| `torch.sin` | Even positional-encoding dimensions |\n",
    "| `torch.cos` | Odd positional-encoding dimensions |\n",
    "| `torch.matmul` | Calculate $QK^T$ and Attention $\\times V$ |\n",
    "| `torch.softmax` | Convert attention scores into attention probabilities |\n",
    "| `torch.relu` | Introduce FFN non-linearity |\n",
    "| `torch.sqrt` | LayerNorm denominator |\n",
    "| `.mean()` | Calculate LayerNorm mean |\n",
    "| `.var()` | Calculate LayerNorm variance |\n",
    "| `.reshape()` | Split and recombine attention heads |\n",
    "| `.transpose()` | Rearrange head/token dimensions and create $K^T$ |\n",
    "| `.contiguous()` | Prepare transposed tensor for reshaping |\n",
    "| `.unsqueeze()` | Add positional-encoding dimension |\n",
    "| `.sum()` | Verify Softmax rows sum to 1 |\n",
    "| `torch.allclose` | Compare tensors for educational checks |\n",
    "| `torch.manual_seed` | Reproducible parameter initialization |\n",
    "| `torch.device` | Choose CPU or GPU |\n",
    "| `torch.cuda.is_available` | Check whether CUDA GPU is available |\n",
    "| `.to(device)` | Move model/tensors to CPU or GPU |\n",
    "| `.detach()` | Remove tensor from autograd graph for inspection |\n",
    "| `.cpu()` | Move inspection tensors back to CPU |\n",
    "| `.tolist()` | Convert tensors to Python lists for display |\n",
    "| `register_buffer()` | Store fixed positional encoding as part of the model |\n",
    "\n",
    "---\n",
    "\n",
    "# What did we intentionally NOT use?\n",
    "\n",
    "To understand the Transformer internals, we deliberately avoided:\n",
    "\n",
    "```python\n",
    "nn.Embedding\n",
    "```\n",
    "\n",
    "Instead, we manually used:\n",
    "\n",
    "```python\n",
    "nn.Parameter + tensor indexing\n",
    "```\n",
    "\n",
    "We avoided:\n",
    "\n",
    "```python\n",
    "nn.Linear\n",
    "```\n",
    "\n",
    "and manually implemented:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "xW+b\n",
    "}\n",
    "$$\n",
    "\n",
    "We avoided:\n",
    "\n",
    "```python\n",
    "nn.LayerNorm\n",
    "```\n",
    "\n",
    "and manually implemented:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\gamma\n",
    "\\frac{x-\\mu}\n",
    "{\\sqrt{\\sigma^2+\\epsilon}}\n",
    "+\n",
    "\\beta\n",
    "}\n",
    "$$\n",
    "\n",
    "We avoided:\n",
    "\n",
    "```python\n",
    "nn.MultiheadAttention\n",
    "```\n",
    "\n",
    "and manually implemented:\n",
    "\n",
    "$$\n",
    "Q,K,V\n",
    "\\rightarrow\n",
    "QK^T\n",
    "\\rightarrow\n",
    "\\text{Scaling}\n",
    "\\rightarrow\n",
    "\\text{Softmax}\n",
    "\\rightarrow\n",
    "AV\n",
    "\\rightarrow\n",
    "\\text{Concatenate}\n",
    "\\rightarrow\n",
    "W_O\n",
    "$$\n",
    "\n",
    "We also avoided:\n",
    "\n",
    "```python\n",
    "nn.TransformerEncoderLayer\n",
    "```\n",
    "\n",
    "and:\n",
    "\n",
    "```python\n",
    "nn.TransformerEncoder\n",
    "```\n",
    "\n",
    "because those would hide most of the architecture we are trying to understand.\n",
    "\n",
    "---\n",
    "\n",
    "# Built-in version vs our manual version\n",
    "\n",
    "| Transformer component | Ready-made PyTorch option | What we used |\n",
    "|---|---|---|\n",
    "| Embedding | `nn.Embedding` | `ManualEmbedding` |\n",
    "| Linear Layer | `nn.Linear` | `ManualLinear` |\n",
    "| LayerNorm | `nn.LayerNorm` | `manual_layer_norm()` |\n",
    "| Multi-Head Attention | `nn.MultiheadAttention` | `ManualMultiHeadSelfAttention` |\n",
    "| FFN | Linear modules | `ManualFeedForward` |\n",
    "| Encoder Layer | `nn.TransformerEncoderLayer` | `EncoderLayerFromScratch` |\n",
    "| Complete Encoder | `nn.TransformerEncoder` | `EncoderFromScratch` |\n",
    "| Positional Encoding | No single required built-in | Manual sinusoidal encoding |\n",
    "\n",
    "This is why our notebook is genuinely useful for learning:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{we use PyTorch}\n",
    "}\n",
    "$$\n",
    "\n",
    "but:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{we do not let PyTorch hide the Transformer mathematics}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# PyTorch vs our own functions\n",
    "\n",
    "It is also useful to distinguish between the two.\n",
    "\n",
    "### PyTorch provides the tensor engine\n",
    "\n",
    "PyTorch handles:\n",
    "\n",
    "- tensors\n",
    "- matrix multiplication\n",
    "- gradients\n",
    "- trainable parameters\n",
    "- CPU/GPU execution\n",
    "- mathematical functions\n",
    "\n",
    "### We implemented the Transformer logic\n",
    "\n",
    "We wrote:\n",
    "\n",
    "```python\n",
    "simple_tokenize()\n",
    "```\n",
    "\n",
    "```python\n",
    "build_vocab()\n",
    "```\n",
    "\n",
    "```python\n",
    "tokens_to_ids()\n",
    "```\n",
    "\n",
    "```python\n",
    "sinusoidal_positional_encoding()\n",
    "```\n",
    "\n",
    "```python\n",
    "add_embedding_and_position()\n",
    "```\n",
    "\n",
    "```python\n",
    "split_heads()\n",
    "```\n",
    "\n",
    "```python\n",
    "combine_heads()\n",
    "```\n",
    "\n",
    "```python\n",
    "scaled_dot_product_attention()\n",
    "```\n",
    "\n",
    "```python\n",
    "residual_add()\n",
    "```\n",
    "\n",
    "```python\n",
    "manual_layer_norm()\n",
    "```\n",
    "\n",
    "and our custom neural-network classes.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{PyTorch}\n",
    "=\n",
    "\\text{numerical + training framework}\n",
    "}\n",
    "$$\n",
    "\n",
    "while:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Our code}\n",
    "=\n",
    "\\text{Transformer architecture logic}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# One final architectural view\n",
    "\n",
    "Our PyTorch hierarchy is:\n",
    "\n",
    "```text\n",
    "EncoderFromScratch(nn.Module)\n",
    "│\n",
    "├── ManualEmbedding(nn.Module)\n",
    "│   └── nn.Parameter → embedding weights\n",
    "│\n",
    "├── register_buffer()\n",
    "│   └── sinusoidal positional encoding\n",
    "│\n",
    "└── nn.ModuleList\n",
    "    │\n",
    "    └── EncoderLayerFromScratch(nn.Module)\n",
    "        │\n",
    "        ├── ManualMultiHeadSelfAttention(nn.Module)\n",
    "        │   │\n",
    "        │   ├── ManualLinear → Q\n",
    "        │   ├── ManualLinear → K\n",
    "        │   ├── ManualLinear → V\n",
    "        │   └── ManualLinear → Output Projection\n",
    "        │\n",
    "        ├── nn.Parameter → gamma1\n",
    "        ├── nn.Parameter → beta1\n",
    "        │\n",
    "        ├── ManualFeedForward(nn.Module)\n",
    "        │   ├── ManualLinear → 512 → 2048\n",
    "        │   ├── torch.relu\n",
    "        │   └── ManualLinear → 2048 → 512\n",
    "        │\n",
    "        ├── nn.Parameter → gamma2\n",
    "        └── nn.Parameter → beta2\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# Final takeaway\n",
    "\n",
    "We did **not** use PyTorch as a black-box Transformer implementation.\n",
    "\n",
    "Instead:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{PyTorch gave us the building blocks}\n",
    "}\n",
    "$$\n",
    "\n",
    "and:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{we assembled the Transformer Encoder ourselves}\n",
    "}\n",
    "$$\n",
    "\n",
    "The most important PyTorch concepts used in this notebook are:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "nn.Module\n",
    "}\n",
    "$$\n",
    "\n",
    "for reusable neural-network components,\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "nn.Parameter\n",
    "}\n",
    "$$\n",
    "\n",
    "for trainable tensors,\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "nn.ModuleList\n",
    "}\n",
    "$$\n",
    "\n",
    "for stacking encoder layers,\n",
    "\n",
    "and basic tensor operations such as:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{matmul, softmax, reshape, transpose, mean, variance, ReLU}\n",
    "}\n",
    "$$\n",
    "\n",
    "for implementing the actual Transformer mathematics.\n",
    "\n",
    "This gives us the best of both worlds:\n",
    "\n",
    "> **PyTorch handles tensors, autograd, parameters, and hardware.**\n",
    "\n",
    "while:\n",
    "\n",
    "> **we remain in control of every important Transformer operation.**\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f952188",
   "metadata": {},
   "source": [
    "\n",
    "# 15. Final conclusion — putting the complete encoder together\n",
    "\n",
    "We have now built a complete **Transformer Encoder from scratch** using PyTorch while keeping every important mathematical step visible.\n",
    "\n",
    "Starting from the raw sentence:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "the encoder gradually converts simple token representations into rich contextual representations.\n",
    "\n",
    "The full journey is:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Raw Text}\n",
    "\\rightarrow\n",
    "\\text{Tokens}\n",
    "\\rightarrow\n",
    "\\text{Embeddings}\n",
    "\\rightarrow\n",
    "\\text{Positional Encoding}\n",
    "\\rightarrow\n",
    "\\text{Self-Attention}\n",
    "\\rightarrow\n",
    "\\text{FFN}\n",
    "\\rightarrow\n",
    "\\text{Contextual Encoder Output}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## What did the encoder actually learn to do?\n",
    "\n",
    "At the beginning, each token mainly knows:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Who am I?}\n",
    "}\n",
    "$$\n",
    "\n",
    "The token embedding provides the identity and learned semantic features of the token.\n",
    "\n",
    "Then positional encoding adds:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Where am I?}\n",
    "}\n",
    "$$\n",
    "\n",
    "After Self-Attention, each token can ask:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Which other tokens are important to me?}\n",
    "}\n",
    "$$\n",
    "\n",
    "The Query, Key, and Value mechanism allows every token to interact with the other tokens in the sequence.\n",
    "\n",
    "Then the Feed-Forward Network asks:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{How should I transform the information I have collected?}\n",
    "}\n",
    "$$\n",
    "\n",
    "Finally, the residual connections and LayerNorm help preserve useful information and keep the transformations stable.\n",
    "\n",
    "So the final token representation answers something much richer:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Who am I in the context of this entire sentence?}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Complete encoder pipeline\n",
    "\n",
    "For our configuration:\n",
    "\n",
    "$$\n",
    "n = 5\n",
    "$$\n",
    "\n",
    "$$\n",
    "d_{\\text{model}} = 512\n",
    "$$\n",
    "\n",
    "$$\n",
    "h = 4\n",
    "$$\n",
    "\n",
    "$$\n",
    "d_{\\text{head}} = 128\n",
    "$$\n",
    "\n",
    "$$\n",
    "d_{\\text{ff}} = 2048\n",
    "$$\n",
    "\n",
    "the complete flow is:\n",
    "\n",
    "```text\n",
    "\"The cat chased the mouse\"\n",
    "            ↓\n",
    "        Tokenization\n",
    "            ↓\n",
    "         Token IDs\n",
    "            ↓\n",
    "      Token Embeddings\n",
    "          [5,512]\n",
    "            ↓\n",
    "    + Positional Encoding\n",
    "          [5,512]\n",
    "            ↓\n",
    "      Encoder Input X\n",
    "          [5,512]\n",
    "            ↓\n",
    "        Create Q, K, V\n",
    "      each [5,512]\n",
    "            ↓\n",
    "       Split into 4 heads\n",
    "       [4,5,128]\n",
    "            ↓\n",
    "   Scaled Dot-Product Attention\n",
    "       Scores [4,5,5]\n",
    "            ↓\n",
    "           Softmax\n",
    "       Weights [4,5,5]\n",
    "            ↓\n",
    "       Attention × Value\n",
    "       [4,5,128]\n",
    "            ↓\n",
    "      Concatenate 4 heads\n",
    "          [5,512]\n",
    "            ↓\n",
    "       Output Projection\n",
    "          [5,512]\n",
    "            ↓\n",
    "      Residual + LayerNorm\n",
    "          [5,512]\n",
    "            ↓\n",
    "      Feed-Forward Network\n",
    "    512 → 2048 → 512\n",
    "            ↓\n",
    "      Residual + LayerNorm\n",
    "            ↓\n",
    "     Final Encoder Output\n",
    "          [5,512]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "# The most important conceptual flow\n",
    "\n",
    "The encoder can be remembered in five major ideas.\n",
    "\n",
    "### 1. Embedding\n",
    "\n",
    "> **What token is this?**\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Token ID}\n",
    "\\rightarrow\n",
    "\\text{Meaning vector}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 2. Positional Encoding\n",
    "\n",
    "> **Where is this token located?**\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Meaning}\n",
    "+\n",
    "\\text{Position}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 3. Self-Attention\n",
    "\n",
    "> **Which other tokens should this token pay attention to?**\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q\n",
    "\\rightarrow\n",
    "\\text{asks}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "K\n",
    "\\rightarrow\n",
    "\\text{matches}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "V\n",
    "\\rightarrow\n",
    "\\text{provides information}\n",
    "}\n",
    "$$\n",
    "\n",
    "Then:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\operatorname{Attention}(Q,K,V)\n",
    "=\n",
    "\\operatorname{softmax}\n",
    "\\left(\n",
    "\\frac{QK^T}{\\sqrt{d_k}}\n",
    "\\right)V\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "### 4. Feed-Forward Network\n",
    "\n",
    "> **How should each token transform the information it has collected?**\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "512\n",
    "\\rightarrow\n",
    "2048\n",
    "\\rightarrow\n",
    "512\n",
    "}\n",
    "$$\n",
    "\n",
    "The FFN processes every token independently.\n",
    "\n",
    "---\n",
    "\n",
    "### 5. Residual + LayerNorm\n",
    "\n",
    "Residual connections preserve the existing representation while adding newly learned information:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Old Information}\n",
    "+\n",
    "\\text{New Information}\n",
    "}\n",
    "$$\n",
    "\n",
    "LayerNorm then keeps the resulting activations stable:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Add}\n",
    "\\rightarrow\n",
    "\\text{Normalize}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Attention and FFN have different jobs\n",
    "\n",
    "One of the most important lessons from the encoder architecture is the difference between Attention and FFN.\n",
    "\n",
    "### Self-Attention\n",
    "\n",
    "Self-Attention performs:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Token Mixing}\n",
    "}\n",
    "$$\n",
    "\n",
    "It allows information to move between tokens.\n",
    "\n",
    "For example:\n",
    "\n",
    "> **chased**\n",
    "\n",
    "can collect useful information from:\n",
    "\n",
    "> **cat**\n",
    "\n",
    "and:\n",
    "\n",
    "> **mouse**\n",
    "\n",
    "---\n",
    "\n",
    "### Feed-Forward Network\n",
    "\n",
    "The FFN performs:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Feature Transformation}\n",
    "}\n",
    "$$\n",
    "\n",
    "It takes the contextual information already collected by each token and transforms its internal features.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Attention}\n",
    "=\n",
    "\\text{communication between tokens}\n",
    "}\n",
    "$$\n",
    "\n",
    "while:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{FFN}\n",
    "=\n",
    "\\text{computation inside each token}\n",
    "}\n",
    "$$\n",
    "\n",
    "This is a very useful mental model for understanding Transformers.\n",
    "\n",
    "---\n",
    "\n",
    "# Why are multiple attention heads useful?\n",
    "\n",
    "Instead of having one single attention mechanism, Multi-Head Attention gives the model several parallel attention spaces.\n",
    "\n",
    "For our example:\n",
    "\n",
    "$$\n",
    "h = 4\n",
    "$$\n",
    "\n",
    "Each head receives:\n",
    "\n",
    "$$\n",
    "d_{\\text{head}} = 128\n",
    "$$\n",
    "\n",
    "features.\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "512\n",
    "=\n",
    "4 \\times 128\n",
    "$$\n",
    "\n",
    "Different heads can learn different patterns or relationships.\n",
    "\n",
    "Then:\n",
    "\n",
    "$$\n",
    "\\text{Head}_1\n",
    "\\oplus\n",
    "\\text{Head}_2\n",
    "\\oplus\n",
    "\\text{Head}_3\n",
    "\\oplus\n",
    "\\text{Head}_4\n",
    "$$\n",
    "\n",
    "are concatenated and mixed through:\n",
    "\n",
    "$$\n",
    "W_O\n",
    "$$\n",
    "\n",
    "to produce:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "again.\n",
    "\n",
    "---\n",
    "\n",
    "# Why does the encoder output remain `[5,512]`?\n",
    "\n",
    "At the beginning:\n",
    "\n",
    "$$\n",
    "X\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "At the end:\n",
    "\n",
    "$$\n",
    "EncoderOutput\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "The shape is the same.\n",
    "\n",
    "But the information is not.\n",
    "\n",
    "Initially:\n",
    "\n",
    "$$\n",
    "\\text{Token Representation}\n",
    "\\approx\n",
    "\\text{Identity}\n",
    "+\n",
    "\\text{Position}\n",
    "$$\n",
    "\n",
    "After the encoder:\n",
    "\n",
    "$$\n",
    "\\text{Token Representation}\n",
    "\\approx\n",
    "\\text{Identity}\n",
    "+\n",
    "\\text{Position}\n",
    "+\n",
    "\\text{Context}\n",
    "+\n",
    "\\text{Relationships}\n",
    "+\n",
    "\\text{Learned Features}\n",
    "$$\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Same dimensionality}\n",
    "\\neq\n",
    "\\text{same representation}\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Example: \"chased\"\n",
    "\n",
    "Before the encoder:\n",
    "\n",
    "$$\n",
    "X_{\\text{chased}}\n",
    "\\in\n",
    "\\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "mainly represents:\n",
    "\n",
    "> **\"This is the token 'chased' at this position.\"**\n",
    "\n",
    "After the encoder:\n",
    "\n",
    "$$\n",
    "H_{\\text{chased}}\n",
    "\\in\n",
    "\\mathbb{R}^{512}\n",
    "$$\n",
    "\n",
    "can contain information influenced by the rest of the sentence.\n",
    "\n",
    "For example:\n",
    "\n",
    "> **cat**\n",
    "\n",
    "may help provide information about who performs the action.\n",
    "\n",
    "> **mouse**\n",
    "\n",
    "may help provide information about what the action is directed toward.\n",
    "\n",
    "The encoder therefore transforms an isolated token vector into a contextual token vector.\n",
    "\n",
    "---\n",
    "\n",
    "# Why is the final encoder output useful?\n",
    "\n",
    "The final output is:\n",
    "\n",
    "$$\n",
    "H\n",
    "\\in\n",
    "\\mathbb{R}^{5 \\times 512}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "H\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "H_{\\text{the}} \\\\\n",
    "H_{\\text{cat}} \\\\\n",
    "H_{\\text{chased}} \\\\\n",
    "H_{\\text{the}} \\\\\n",
    "H_{\\text{mouse}}\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Every row is now a contextual representation.\n",
    "\n",
    "This representation can be reused for many downstream tasks.\n",
    "\n",
    "---\n",
    "\n",
    "## Text classification\n",
    "\n",
    "For tasks such as:\n",
    "\n",
    "- sentiment classification\n",
    "- spam detection\n",
    "- intent classification\n",
    "- topic classification\n",
    "\n",
    "we can pool the encoder outputs into one sequence representation and send it to a classifier.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "\\rightarrow\n",
    "\\text{Pooling}\n",
    "\\rightarrow\n",
    "[512]\n",
    "\\rightarrow\n",
    "\\text{Classifier}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## Token classification\n",
    "\n",
    "For tasks where every token needs its own prediction, we can directly use each contextual token vector.\n",
    "\n",
    "Examples include:\n",
    "\n",
    "- Named Entity Recognition\n",
    "- Part-of-Speech tagging\n",
    "- token labeling\n",
    "\n",
    "So:\n",
    "\n",
    "$$\n",
    "H_i\n",
    "\\rightarrow\n",
    "\\text{Classifier}\n",
    "\\rightarrow\n",
    "\\text{Label for token } i\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "## Semantic search and retrieval\n",
    "\n",
    "The token representations can be pooled to produce a sentence or document embedding.\n",
    "\n",
    "Then two texts can be compared using their vector representations.\n",
    "\n",
    "Conceptually:\n",
    "\n",
    "$$\n",
    "\\text{Text}\n",
    "\\rightarrow\n",
    "\\text{Encoder}\n",
    "\\rightarrow\n",
    "\\text{Embedding}\n",
    "$$\n",
    "\n",
    "This is useful for:\n",
    "\n",
    "- semantic search\n",
    "- retrieval systems\n",
    "- similarity matching\n",
    "- RAG retrieval\n",
    "\n",
    "---\n",
    "\n",
    "## Encoder-decoder Transformer\n",
    "\n",
    "In a full encoder-decoder Transformer, the final encoder output becomes the decoder's **encoder memory**.\n",
    "\n",
    "The encoder produces:\n",
    "\n",
    "$$\n",
    "H_{\\text{encoder}}\n",
    "$$\n",
    "\n",
    "Then during decoder cross-attention:\n",
    "\n",
    "$$\n",
    "Q\n",
    "=\n",
    "\\text{Decoder representation}\n",
    "$$\n",
    "\n",
    "while:\n",
    "\n",
    "$$\n",
    "K,V\n",
    "=\n",
    "\\text{projections of encoder output}\n",
    "$$\n",
    "\n",
    "So the decoder can ask:\n",
    "\n",
    "> **\"Which parts of the encoded source sentence are useful for generating the next output token?\"**\n",
    "\n",
    "This is especially important in tasks such as translation.\n",
    "\n",
    "---\n",
    "\n",
    "# Factory analogy recap\n",
    "\n",
    "You can think of our encoder like a factory.\n",
    "\n",
    "### Tokenizer\n",
    "\n",
    "Receives raw material and labels the pieces.\n",
    "\n",
    "$$\n",
    "\\text{Text}\n",
    "\\rightarrow\n",
    "\\text{Tokens}\n",
    "$$\n",
    "\n",
    "### Embedding\n",
    "\n",
    "Converts each piece into a machine-readable representation.\n",
    "\n",
    "$$\n",
    "\\text{Token}\n",
    "\\rightarrow\n",
    "\\text{Feature Vector}\n",
    "$$\n",
    "\n",
    "### Positional Encoding\n",
    "\n",
    "Adds a position stamp.\n",
    "\n",
    "> **\"Where did this item appear?\"**\n",
    "\n",
    "### Self-Attention\n",
    "\n",
    "Every station examines the other items and decides:\n",
    "\n",
    "> **\"Which ones are relevant to me?\"**\n",
    "\n",
    "### Multi-Head Attention\n",
    "\n",
    "Several inspectors examine relationships from different learned perspectives.\n",
    "\n",
    "### Residual Connection\n",
    "\n",
    "Keeps the original information on a bypass path.\n",
    "\n",
    "### LayerNorm\n",
    "\n",
    "Keeps the representation numerically stable.\n",
    "\n",
    "### Feed-Forward Network\n",
    "\n",
    "Performs deeper feature processing on every individual token.\n",
    "\n",
    "### Encoder Output\n",
    "\n",
    "Produces the final context-aware representation ready for the next task.\n",
    "\n",
    "---\n",
    "\n",
    "# From mathematics to reusable code\n",
    "\n",
    "In this notebook, we did not stop at the equations.\n",
    "\n",
    "We converted the mathematics into reusable PyTorch components.\n",
    "\n",
    "The progression was:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Mathematics}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Functions}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{PyTorch Modules}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Encoder Layer}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Complete Encoder}\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\downarrow\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Factory Function}\n",
    "}\n",
    "$$\n",
    "\n",
    "This gives us both:\n",
    "\n",
    "- a clear understanding of the Transformer mathematics\n",
    "- a reusable implementation that can actually be executed\n",
    "\n",
    "---\n",
    "\n",
    "# What we implemented from scratch\n",
    "\n",
    "Throughout this notebook, we manually implemented the major encoder operations:\n",
    "\n",
    "- token embeddings\n",
    "- sinusoidal positional encoding\n",
    "- Query projection\n",
    "- Key projection\n",
    "- Value projection\n",
    "- splitting into attention heads\n",
    "- scaled dot-product attention\n",
    "- Softmax\n",
    "- Value aggregation\n",
    "- head concatenation\n",
    "- output projection\n",
    "- residual connections\n",
    "- Layer Normalization\n",
    "- Feed-Forward Network\n",
    "- stacking reusable encoder modules\n",
    "- configurable factory function\n",
    "\n",
    "We deliberately avoided hiding the important logic inside high-level components such as:\n",
    "\n",
    "```python\n",
    "nn.MultiheadAttention\n",
    "```\n",
    "\n",
    "or:\n",
    "\n",
    "```python\n",
    "nn.TransformerEncoder\n",
    "```\n",
    "\n",
    "so that the internal mechanics remain visible.\n",
    "\n",
    "---\n",
    "\n",
    "# Final dimensional summary\n",
    "\n",
    "Our complete encoder uses:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "n=5\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "d_{\\text{model}}=512\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "h=4\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "d_{\\text{head}}=128\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "d_{\\text{ff}}=2048\n",
    "}\n",
    "$$\n",
    "\n",
    "The main dimensions are:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "X=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q=K=V=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "Q_h=K_h=V_h=[4,5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Attention Scores}=[4,5,5]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Context Heads}=[4,5,128]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{MHA Output}=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{FFN Hidden}=[5,2048]\n",
    "}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Final Encoder Output}=[5,512]\n",
    "}\n",
    "$$\n",
    "\n",
    "---\n",
    "\n",
    "# Final mental model\n",
    "\n",
    "If you remember only one flow from this notebook, remember:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{Token}\n",
    "\\rightarrow\n",
    "\\text{Meaning}\n",
    "\\rightarrow\n",
    "\\text{Position}\n",
    "\\rightarrow\n",
    "\\text{Attention}\n",
    "\\rightarrow\n",
    "\\text{Context}\n",
    "\\rightarrow\n",
    "\\text{Feature Transformation}\n",
    "\\rightarrow\n",
    "\\text{Contextual Representation}\n",
    "}\n",
    "$$\n",
    "\n",
    "Or even more simply:\n",
    "\n",
    "> **Embedding:** Who am I?\n",
    "\n",
    "> **Position:** Where am I?\n",
    "\n",
    "> **Query:** What am I looking for?\n",
    "\n",
    "> **Key:** What information do I contain?\n",
    "\n",
    "> **Attention:** Who should I listen to?\n",
    "\n",
    "> **Value:** What information should I collect?\n",
    "\n",
    "> **FFN:** How should I transform what I learned?\n",
    "\n",
    "> **Encoder Output:** Who am I in the context of the entire sequence?\n",
    "\n",
    "---\n",
    "\n",
    "# Final takeaway\n",
    "\n",
    "The Transformer encoder is fundamentally a system for turning:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{independent token representations}\n",
    "}\n",
    "$$\n",
    "\n",
    "into:\n",
    "\n",
    "$$\n",
    "\\boxed{\n",
    "\\text{context-aware token representations}\n",
    "}\n",
    "$$\n",
    "\n",
    "For:\n",
    "\n",
    "> **\"The cat chased the mouse\"**\n",
    "\n",
    "we started with 5 token vectors:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "and finished with 5 contextual token vectors:\n",
    "\n",
    "$$\n",
    "[5,512]\n",
    "$$\n",
    "\n",
    "The dimensions stayed the same.\n",
    "\n",
    "The **information inside them became much richer**.\n",
    "\n",
    "That is the core purpose of the Transformer encoder.\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
