Attention mechanism in detail

Direct answer

Attention lets each token build a new representation by taking a weighted mixture of value vectors. The weights come from how strongly that token’s query matches every permitted key. In scaled dot-product attention, the model computes query-key scores, scales them by the square root of the key dimension, applies a mask when needed, normalizes with softmax, and uses the result to combine values.

The three roles

The input representation is projected into three learned views:

• A query describes what the current position is looking for.
• A key describes what each candidate position offers for matching.
• A value carries the information that will be combined if its key receives weight.

Queries and keys decide where to read. Values decide what is read. They are produced by different learned matrices, so a high match does not mean the query and value vectors are numerically identical.

From scores to context

For query matrix $Q$, key matrix $K$, value matrix $V$, and key dimension $d_k$:

The mask $M$ is zero for allowed connections and a very negative value for blocked connections. A padding mask prevents reading placeholder tokens. A causal mask prevents a decoder position from reading future tokens.

The sequence is:

1. Project token representations into queries, keys, and values.
2. Compute every allowed query-key dot product.
3. Divide by $\sqrt{d_k}$ so score magnitude does not grow unchecked with width.
4. Add the mask before normalization.
5. Apply softmax across the candidate-key dimension.
6. Multiply the normalized weights by values and sum.

Small numerical example

Suppose one query produces raw scores $[2, 1, 0]$ against three keys. After softmax, the weights are approximately $[0.665, 0.245, 0.090]$. If the first value carries the most relevant signal, it contributes the largest share, but the output still blends all unmasked values.

import math

scores = [2.0, 1.0, 0.0]
maximum = max(scores)
unnormalized = [math.exp(score - maximum) for score in scores]
total = sum(unnormalized)
weights = [value / total for value in unnormalized]  

print([round(weight, 3) for weight in weights])

Output: [0.665, 0.245, 0.09].

Subtracting the maximum does not change the softmax result; it keeps exponentials in a safer numerical range.

Self-attention and cross-attention

In self-attention, queries, keys, and values originate from the same sequence, although their projections differ. In encoder-decoder cross-attention, decoder states supply the queries while encoder outputs supply keys and values. This allows each generated position to retrieve relevant source information.

Limitations and common mistakes

Attention weights are routing coefficients, not a guaranteed explanation of model reasoning. Masks must be applied on the correct axis, and an implementation should use numerically stable softmax. The $\sqrt{d_k}$ scaling controls score variance; it does not replace normalization, masking, or learned projections. Standard dense attention also compares every query with every key, so its score matrix grows quadratically with sequence length.

Interview takeaway

Scaled dot-product attention is a learned read operation: queries match keys, normalized matches weight values, and masks define which reads are legal.