Learn · Demo
Cross-Attention: Encoder → Decoder
In a sequence-to-sequence transformer — translation, summarisation, RAG — the decoder writes the output while looking back at the encoded input. That backward glance is cross-attention. Below, a French source is softly aligned to its English translation, one decoder token at a time.
Source · Encoder (French)
le chat noir
Each source token exposes a key.
Target · Decoder (English)
the black cat
Each target token issues a query.
Pick a decoder token to see what it attends to in the source:
Cross-Attention Weights
Rows = target / decoder tokens (queries). Columns = source / encoder tokens (keys). Brighter = stronger attention. Hover a cell, or hover/click a row to draw its soft alignment. Each row sums to 100%.
Three kinds of attention in one transformer
A full encoder–decoder transformer runs three distinct attention operations. They share the same scaled dot-product maths but differ in where the queries and keys come from.
Encoder self-attention
Every source token attends to every other source token. Queries and keys both come from the input, so the encoder builds a context-rich representation of the whole source at once.
Decoder causal self-attention
Target tokens attend to earlier target tokens only — a causal mask hides the future so the model can't peek at words it hasn't generated yet. Queries and keys both come from the output-so-far.
Decoder → encoder cross-attention
The one shown above. Queries come from the decoder, keys and values from the encoder. This is the bridge that lets the output read the input.
Why cross-attention matters
Cross-attention is what makes the output conditioned on the input. In translation it aligns each generated word with the source words that inform it (here cat ↔ chat, black ↔ noir). The same mechanism lets a summariser point back at the document it's condensing, and lets a RAG system ground each generated token in retrieved passages. Without it, the decoder would be a plain language model with no view of what it's supposed to be transforming.
This is an illustrative toy. The scaled dot-product maths (dot product, ÷√d, softmax per row) is real, but the 4-dim query/key vectors are hand-tuned so a sensible word alignment emerges — real transformers learn these projections across many heads and layers, and true alignments are rarely this clean. For the underlying maths see Learn, and for self-attention within a single sequence see the Attention Visualiser.