Matrices

OpenAI released GPT-2 in February 2019 with TensorFlow code in which every projection is a function called conv1d. Despite its name it is an ordinary matrix multiplication, but it stores each weight matrix with the input dimension first: the attention output projection of the smallest model is a 768 × 768 array whose rows are indexed by the 768 inputs. PyTorch's nn.Linear, which most reimplementations use, stores the same matrix the other way round, one row per output. Hugging Face kept OpenAI's layout in a class it also calls Conv1D, documented as "Basically works like a linear layer but the weights are transposed". Anyone who loads those weights into nn.Linear layers has to transpose 48 of the model's 148 tensors.