$\color{black}\rule{365px}{3px}$

“Attention Is All You Need” - 2017

Link to Paper:

arxiv.org

Introduction

$\color{black}\rule{365px}{3px}$

• First model to rely solely on attention mechanisms for sequence modelling.

• Attention Mechanism:

  • Scaled Dot-Product Attention:

  Computes relevance between tokens using the formula:

  $$ \text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^T}{\sqrt{d_k}})V $$

• Self-Attention:

  • Each token in a sequence attends to all other tokens, including itself.
  • Captures global dependencies and relationships within the sequence.

• Multi-Head Attention:

  • Splits attention computation into multiple "heads" to learn diverse patterns.
  • Outputs from all heads are concatenated and projected for richer representation.

Attention in Seq2Seq vs. Attention in Transformer

$\color{black}\rule{365px}{3px}$