On Architecture Research

Strong architectures are results of accidental/intentional hardware-software codesign. The dogma, to me, appears to be: how much useful compute/time can we throw at the model? (exemplified by: "the models, they just want to learn" - Ilya Sutskever) In a sense, the principal inductive bias that we adhere to is the fact that we run learning systems on the arbitrary computing substrate that we have…