How GPUs Talk to Each Other (And Why It Matters More Than You Think)

When people think about AI performance, they usually think about which GPU they are using.

But here’s the real question:

When GPUs need to talk to each other, how do they do it—and how fast is each method?

Because in modern AI training, GPUs don’t work alone.
They constantly share data, sync results, and coordinate with each other.

And that communication layer?
That’s what actually unlocks performance.

Think of GPU communication like sending packages.
Some methods are slow but cheap. Others are fast but expensive.

• This is the regular internet

• GPUs may be in different cities or data centers

• Communication happens using TCP/IP

Result:

• High latency

• Slow data transfer

✔ Good for testing
Not suitable for serious AI training

• Uses standard Ethernet, but smarter

• Data goes directly from GPU memory to GPU memory (skips CPU)

Result:

• ~10–100× faster than public internet

Used in cloud services like AWS EFA and Azure RDMA

• A dedicated network built for high-performance computing

• Custom cables, switches, and optimized stack

Result:

• Extremely low latency (sub-microsecond)

• High throughput

This is what serious AI clusters use between servers

• No external network at all

• GPUs are connected directly inside the same machine

Result:

• ~900 GB/s bandwidth

• ~100× faster than InfiniBand

Limitation:
Only works within a single server

All of this complexity is handled by one powerful layer: NCCL.

• Frameworks like PyTorch or JAX simply call NCCL

• You write normal training code

• NCCL automatically chooses the fastest communication path

Priority:

1. NVLink (same machine)

2. InfiniBand (across machines)

3. RoCE

4. Ethernet (last resort)

So you don’t have to think about networking—
NCCL optimizes it for you.

Every training step requires GPUs to sync data.

NCCL handles this using a few core operations:

• AllReduce → Combine gradients from all GPUs and share results

• AllGather / ReduceScatter → Used when the model is split across GPUs

• Broadcast / P2P → Send data from one GPU to others

This happens after every backward pass

Most people focus on GPUs.

But the real performance unlock comes from:

How GPUs talk to each other

• NVLink → fastest (inside a server)

• InfiniBand / RoCE → fast (between servers)

• Ethernet → fallback

If communication is slow, GPUs sit idle.
If communication is fast, GPUs stay busy.

And that’s what makes AI training truly scale