A Simple Switch Makes Code Differentiable

Introduction One game I like to play is decomposing code and algorithms as compositions of simpler pieces even when they already seem as simple as they could be. One example is the observation that adjoint mode automatic differentiation isn't a separate algorithm to forward mode automatic differentiation but a composition of forward mode and transposition. I talked about this in an old paper of…