Neural networks are getting spiky again

Apologies for my silence the last few weeks. I’ve been in Brittany doing some research. And eating butter. So much butter. Maybe I’ll say something about that soon (the research, not the butter), but I thought I’d start the year with a topic I’ve been contemplating for some time: spiking neural networks (SNNs).

Back at the turn of the millennium, when I was a PhD student, there was a lot of interest in using SNNs as a replacement for classical ANNs. The main pull factors were biological plausibility and efficiency, and the main challenge was training them effectively, since the discrete manner in which spiking neurons communicate precludes direct use of classical backpropagation.

Yet despite this interest, SNNs turned out to be the road largely not taken in the intervening time. Instead, the mainstream followed the path of upscaling classical ANNs. Nevertheless, those dedicated to the SNN cause continued to hack away in the background, and there have been some pretty interesting developments since, resulting in SNNs that are competitive against mainstream deep learning architectures. Given the issues around efficiency and power usage, we’re again at the stage where people are positing SNNs as a plausible way forward. History does tend to repeat itself.

So what is an SNN? Basically it’s a neural network in which the individual neurons are modelled on the actual behaviour of biological neurons, more or less. Unlike the neurons you typically find in ANNs, they don’t have continuous-valued activation levels. That is, when they fire, they don’t maintain a constant output which is copied to downstream neurons and then multiplied by a weight. Rather, they emit a spike — a signal with a typically fixed value that only exists for a short period of time — which then travels to downstream neurons. When a downstream neuron receives this signal, it accumulates it (i.e. sums it) on top of any other signals it has received since it last fired, and when this accumulated value reaches a threshold, it then fires and sends out its own spike. Which then gets conveyed to its downstream neurons. And so on.

So one key difference is that a neuron only needs to sum its inputs, rather than multiplying by a weight and then summing. This is a much cheaper operation, especially if you’re dealing with a large neural network. Another more subtle benefit is that if a neuron isn’t receiving any spikes, then it doesn’t have to do anything. This means that parts of a neural network could remain in a quiescent state for a significant period of time, not using processing resources1.

You could also argue that SNNs are better because they’re biologically-motivated. That is, our brains use similar things, and they seem to work pretty well. However, I’m generally wary of simplistic “biology is better” arguments, since (a) biology may not be optimal and (b) our models might not capture the particular magic that makes biological systems work so well. Especially when we don’t really understand how our brains work.

One of the problems with following the biology is that optimisers designed for classical neural networks just don’t work with SNNs. This is because they’re based around calculating the gradients of activation functions, and this can’t be done in a straightforward or reliable manner when you’re dealing with discrete processes like accumulating spikes2. Instead, the early work on SNNs used optimisers that were motivated by ideas of how brains learn, particularly something called Hebbian learning. But — in part due to being unsupervised rather than supervised methods —these are far less effective than backpropagation, especially when it comes to training larger neural networks. Hence why they dropped out of the limelight.

But academics never let go of an interesting idea, and whilst the mainstream and its dog were obsessing over CNNs and transformers, those who continued to chip away at SNNs came up with two viable approaches to training larger networks. Two quite different approaches, but both leveraging classical ANN methods.

The first rests on the nowadays commonplace idea of transfer learning. Specifically, train a conventional ANN using a conventional ANN optimiser, and then turn it into an SNN. This can be done using surprisingly simple mappings between conventional activation functions (such as ReLU) and spiking mechanisms, and the resulting SNNs more or less maintain the accuracies of the original networks. Yet this approach is dissatisfying in that it still involves training a conventional ANN — which, for modern architectures like transformers, can be prohibitively expensive. But the upside is that you get a finished product with potentially much lower inference-time costs, without sacrificing accuracy.

The second approach avoids the need to train a conventional ANN by instead adapting backpropagation so that it works with SNNs. This involves something called surrogate gradients, basically mathematical approximations of the derivatives of spike-based mechanisms. Again, it’s not particularly complicated to implement, but required a fair amount of research to find surrogates that were close enough to the true gradients to guide learning whilst being well-behaved enough that they didn’t cause optimisation to grind to a halt.

So, using either of these approaches, it’s now possible to create large-scale SNNs. But there’s an important ingredient that stands in the way of broader adoption: hardware. More through luck than design3, conventional ANNs are able to leverage the massively-parallel matrix-multiplication abilities of GPUs to seriously speed up training and inference. But of course we could develop and deploy hardware architectures designed specifically for running SNNs. There are already concrete implementations, such as SpiNNaker, which was used in the Human Brain Project4, and IBM’s Loihi platform.

Another barrier is transformers, the core components of generative AI. These don’t just contain classic ANN elements; they also contain self-attention layers, which involve matrix multiplication and softmax operations. Early proposals to integrate SNNs in transformers involved just replacing the classical ANN elements, but this meant the resulting architectures were neither efficient for GPUs or hardware designed for SNNs. To address this, current research is focusing on replacing the self-attention layers with spiking mechanisms. There’s been a whole bunch of papers published on this topic in the last year, with some significant progress in the area of vision transformers, where SNN-based models are reportedly becoming competitive against classical approaches.

Which is good news, since transformers have become a major sink on global energy resources, and a major environmental risk factor. Replacing them with spiking models could reduce their energy consumption by orders of magnitude. However, spiking transformers have yet to prove themselves in the big league of generative AI. Maybe it will take buy-in from the big tech companies to make this happen, since much of the work on SNNs is currently being done by academic research groups, who generally lack the resources to scale up their models. Watch this space I guess. But as someone who’s been watching this space for a long time, I’m happy to see that SNNs finally have momentum.