Activation functions and empiricism

In deep learning literature, there&rsquo;s a veritable menagerie of different activation functions that are commonly employed betwixt layers. Proponents of one or another class of functions will usually proffer up some rationalisation for what makes their choice grounded: differentiability, smoothness, computational complexity, numerical stability, concision&hellip; Today I was reading through GLU…