Playground · Positional Encodings
Positional encodings
Self-attention treats a sequence as an unordered bag of tokens — shuffle the words and it computes the same thing. So Transformers add a positional code to every token's vector. The original paper builds that code from sines and cosines of many frequencies. Below is the real formula, drawn live.
The encoding
PE(pos, 2i) = sin( pos / 100002i/d )
PE(pos, 2i+1) = cos( pos / 100002i/d )
pos = token position · i = dimension-pair index · d = model dimension. Each pair of columns shares one frequency; low dimension-pairs use short wavelengths (fast oscillation), high pairs use long wavelengths (slow oscillation).
Encoding matrix
Rows = positions (top = 0), columns = dimensions (left = 0). Colour shows the value: cyan = negative, magenta = positive; brighter = larger magnitude.
Hover a cell to read its position, dimension and encoding value.
A few dimensions as waves
Each selected (even) dimension is a sine wave across positions. Notice the wavelength grows as the dimension index rises — that spread of frequencies is what makes every position's code unique.
Attention ignores order
Self-attention is permutation-invariant: it weighs tokens by content, not by where they sit. "Dog bites man" and "man bites dog" would look identical without extra help.
A unique fingerprint
Stacking sines and cosines of many frequencies gives each position a distinct, smoothly-varying vector — like binary counting, but continuous. Nearby positions have similar codes.
Relative offsets are linear
Because of trig identities, the code for pos+k is a fixed linear function of the code for pos. That lets the model learn to attend by relative distance, and extend past training lengths.
This computes the exact sinusoidal formula from "Attention Is All You Need". Real models add these vectors to token embeddings before the first attention layer; many modern Transformers instead use learned or rotary (RoPE) position encodings. Want the theory? See the videos on our Learn shelf.