Transformer Architecture Explained: Self-Attention, Encoders, and Decoders

Transformers are a sequence-to-sequence model : given an input sequence, produce an output sequence. Architecture: an Encoder processes the input; a Decoder generates the output autoregressively. \[\text{(En) "I am sorry"} \xrightarrow{\text{Encoder}} \xrightarrow{\text{Decoder}} \texttt{<start>}\ \text{Je suis désolé}\ \texttt{<end>}\] Autoregressive : the decoder generates one token at a time,…