Datacenters Aren’t the Same Anymore

Over the last little while, I’ve been doing a bit of a deep dive into what is happening inside datacenters because of AI.

Datacenters. Some people want them built everywhere and some don’t. At any rate, they’re hard to miss in the news. But what is maybe less obvious is this: what do these new datacenters actually look like? Because they do not really look like the old datacenters.

For a long time, the basic unit of the datacenter was the server. And the server was, in a real sense, just a fancy personal computer. Some CPU, some memory, some disk, a motherboard, maybe a network card. Put a bunch of them in racks, connect them with a network, and you have a datacenter.

Google’s original infrastructure design is probably the most famous version of this idea. The insight was basically: what if we made servers even more like cheap PCs, packed them together at massive scale, and used software to make the whole thing reliable?

That was the hyperscale datacenter model. Lots and lots of relatively ordinary machines, stitched together by software.

But now they look quite different, partly because we need the memory, the RAM, to be much closer to the CPU so that AI inference can happen faster. For regular data centres, it was fine to have more individualised components, but AI inference and training are different and the bottleneck moves to memory.

This is why NVIDIA’s AI data centre designs look less like a bunch of individual servers and more like one giant computer. The physical shape of the machine changes; it’s still a rack, but an individual rack rather than an individual computer.

(Slides from my internal slide deck on new hardware.)

We also see this hardware change in personal computers, particularly Apple computers with their “unified memory” concept. This design brings the memory closer to the CPU, coincidentally making it easier to run larger language models.

More directly this is obvious in some NVIDIA systems, such as the DGX Spark (see image) which is meant to be a tiny, somewhat personal, version of an NVIDIA DC or rack system.

NVIDIA is set to win a significant share of the new data centre market. Any stocks related to the construction of these AI data centres are soaring. Look at Micron for example.

.