VideosLearn & research
Attention in transformers: queries, keys and context
3Blue1Brown explains how attention updates token representations using surrounding words.
What you will learn
Grant Sanderson follows query, key and value vectors through softmax, masking and multiple attention heads. The adjective-and-noun example illustrates possible behavior, rather than explaining every learned head.
Who it suits
Before starting, use the preceding transformer lesson to distinguish tokens from their vector embeddings. It explains how representations carry meaning and how context changes them. This makes the attention diagrams easier to follow.
Published in April 2024, this foundations lesson uses GPT-3 examples. This guide relies on the creator’s written adaptation.
Sources(3)
- Attention — official videoChecked:
- Attention — written lessonChecked:
- Transformers — preceding lessonChecked:
