Skip to content
Handbook navigation

Full-access lesson

Self-Attention

> Remove one factor — `1/sqrt(d_k)` — from the attention formula, and at a realistic per-head dimension of 64, softmax collapses from a balanced 42%/30%/29% split to an 89%/6%/5% near-one-hot distribution, on the exact same underlying scores.

The complete curriculum

All 12 volumes, companion resources, interviews, and architecture reviews.

Executable engineering practice

Subscriber-only Python, Java, TypeScript, and SQL labs in the isolated runner.

Self-Attention | KnowledgeOS