Parallelizing Multi-Head Attention

In the multi-head attention mechanism, why after reshaping the projection matrices for Q/K/V from 3 dimensions to 4, we need to transpose the tokens dimension with the heads dimension? Using the canonical example code for Attention Heads below as an example, why do we need Q = self.W_q(x).reshape(B, T, self.num_heads, self.head_dim).transpose(1, 2) ? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19…