How CUDA Graph Works in torch.compile

This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.

📌 Introduction Link to heading CUDA Graphs capture a sequence of GPU operations and replay them as a single unit, eliminating the CPU launch overhead for each individual kernel. This is especially beneficial for workloads with small batches or models with many small kernels, where CPU overhead can become a bottleneck. When you use torch.compile(mode="reduce-overhead") or enable…

Read on fkong.tech ↗