notes

Previous architectures

Self attention - Relates different positions of a single sequence in order to compute a representation of the sequence

Input - $(x_{1}, \dots, x_{n})$ Encoding - $z = (z_{1}, \dots, z_{n})$ Output - $(y_{1}, \dots, y_{m})$

Transduction models have encoder/decoder structure

Autoregresssive - consumes the previously generated symbol as additional input

Transformer architecture

Decoder uses masked self-attention to prevent positions from attending to subsequent positions.
- Ensures prediction for position $i$ can only depend on known outputs at positions less than $i$

Scaled dot-product attention

Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V

Multi-head attention

Instead of $d_{m o d e l}$ dimensional keys for queries, values and keys, split them $h$ times, perform attention parallely and then concatenated and projected
This allows the model to attend information from different representations at different positions

MultiHead (Q, K, V) = Concat (head_{1}, \dots, head_{h}) W^{O}

where

$head_{i} = Attention (Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V})$
$W_{i}^{Q} \in R^{d_{model} \times d_{k}}$ , … and $W^{O} \in R^{h d_{v} \times d_{model}}$

Attention uses in Transformers

Encoder-decoder attention layers - queries come from previous decoder and keys, values comes form the output of encoder
- Allows each position in decoder to attend over all positions of input
Encoder’s self attention - each position in encoder can attend to all positions in previous layer of encoder
Decoder self attention - uses masked attention so that each position can only attend to positions up to and including that position

Feed forward networks

F F N (x) = max (0, x W_{1} + b_{1}) W_{2} + b_{2}

Softmax

Use a learnt linear transformation and softmax to turn the decoder output to predicted next-token probabilities.

Positional encoding

Paper Notes