What is attention

Attention lets each token in a sequence take information from the others. For token i it builds a query vector q_i. Every token j offers a key k_j and a value v_j. The score of j for i is the dot product q_i · k_j, scaled by 1/√d. A softmax turns each row of scores into weights that sum to one. The output for token i is the weighted sum of the values.

In matrix form, with N tokens of dimension d:

O = softmax(Q Kᵀ / √d) V

The middle term, S = Q Kᵀ, is an N × N matrix. For a 2,048-token context that is about four million scores for every head. Storing it is what makes naive attention expensive in memory, and it is the reason FlashAttention exists.

Causal masks

In a language model, token i must not see later tokens. A causal mask sets the scores for j > i to minus infinity before the softmax, so their weights are zero. The engine has a causal mode; with it, half of the score matrix is never computed.

Try it

The demo uses the float reference, not the engine. Pick a preset and a query token. In the toy sentence, the pronoun "it" gives "cat" about 0.66 of its weight and "mouse" about 0.15.