How LLMs Work, Part 2: Inside a Transformer Block

The previous write-up ended with the output from the embedding lookup (1 and 2 below). The previous write-up showed that the output from the embedding layer gives the numerical representation for the token IDs, but these embeddings themselves are not yet contextual. Details are in this section of the previous write-up:…