Learn · Causal Attention
Causal (Masked) Self-Attention
Decoder models like GPT generate text one token at a time. To do that honestly, each token is only allowed to attend to itself and the words before it — never the future. Flip the mask below and watch the upper triangle go dark.
Attention Weights
Each row is a query token; each column is a key token. Brighter cyan = stronger attention. Hatched/✕ cells are masked — the query is not allowed to look there. Hover or focus a row to read its distribution.
Why mask the future?
During training a decoder learns to predict the next token. If a token could peek
at the tokens it is supposed to predict, that would be cheating — the model would score perfectly
and learn nothing useful. The mask blocks positions j > i before softmax.
Autoregressive generation
Because every token only depends on earlier tokens, the model can generate one word, append it, and feed the sequence back in — over and over. This left-to-right dependency is exactly what makes text generation autoregressive.
Encoders see everything
Encoder models like BERT use full bidirectional attention — every token attends to every other token, past and future. Great for understanding a fixed sentence, but not for generating one token at a time. Toggle the mode above to compare.
This is a toy illustration — real transformers have learned query/key/value projections across
multiple heads and layers. The query and key vectors here are fixed pseudo-random values seeded by each
word, so the pattern is deterministic but not trained. The masking maths (scaled dot-product scores with
-∞ on future positions, then softmax per row) is the real thing. Compare with the
bidirectional Attention Visualiser,
or dig into the maths on Learn.