Docs

Six pages, each with a live demo of the part it explains.

• What is attention: queries, keys and values, and the score matrix.
• Online softmax: how a softmax over a row can be computed a tile at a time.
• FlashAttention: the tile schedule, and the memory traffic it saves.
• Fixed point: the number formats, the exp2 unit, and the measured error.
• Architecture: the blocks of the RTL and how they connect.
• Verification: what is tested, how, and the results.