Transformer

Attention mechanism

• Intuitions of Q, K, V
  • The queries try to figure out the context of the target word. E.g. what’s happening with Africa?
  • The keys are the surrounding words used for identifying the context associated with the query word based on the similarity (between key and value).
  • The values are the customised representation of the target word based on the associated context.

References

• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)

• The Illustrated Transformer