Lequn Chen, abcdabcd987
Last week, our team summarized some recent progress we made on point-to-point communication for LLM systems and posted a paper on arXiv. We also open-sourced the code on GitHub. We built an RDMA communication library based on the idea of Unordered Reliable Datagram (URD) semantics. It runs on both AWS EFA and NVIDIA ConnectX. We applied this library to three scenarios: KvCache transfer in…
In the previous blog post, I walked through how we achieved cross-machine RL weight updates in just 2 seconds. This post is a quick follow-up with a few extra details: For Kimi-K2 (1T params), with 256 GPUs in BF16 training and 128 GPUs in FP8 inference, weight updates take less than 1.3 seconds. The pipeline for parameter updates has been tuned a bit more, adding two parallelizable steps: H2D…
I just spent the past two weeks getting cross-machine parameter updates for Qwen3-235B (BF16 training, FP8 inference) to run in just 2 seconds (128 GPUs for training, 32 GPUs for inference). Instead of writing a “here’s the solution” kind of post, I want to share my exploration process and thoughts along the way. I’ll post a shorter, polished version on the company blog in a few days.
In the previous chapter, by submitting new WRITE operations in batches, we achieved a transmission speed of 2589.488 Gbps, reaching 80.9% of the total 3200 Gbps bandwidth. Now, the full 3200 Gbps speed is within reach. Notice that every time we submit a WRITE operation, the PostWrite() function calls ProgressPendingOps(). It’s conceivable that if ProgressPendingOps() cannot progress in the first…
In the previous chapter, we sharded the states across different threads to avoid synchronization between threads, which increased the transmission speed to 1522.738 Gbps. Unfortunately, this still only reached 47.6% of the total 3200 Gbps bandwidth. Clearly, the CPU is still the bottleneck and we need to further optimize. In the ContinuePostWrite() function, when we submit new WRITE operations, we…
In the previous chapter, we bound CPU cores for the multi-threaded version, which boosted the transmission speed to 1237.738 Gbps. Unfortunately, this still only reached 38.7% of the total 3200 Gbps bandwidth. Clearly, the CPU remains a bottleneck, and we need to optimize further. Looking back at our process of converting the program from single-threaded to multi-threaded, we changed some…
In the previous chapter, we enabled multi-threading and created a thread for each GPU. However, the transmission speed was still only 355.301 Gbps, reaching just 11.1% of the total 3200 Gbps bandwidth. If we carefully compare the execution process of the multi-threaded version in the previous chapter with the single-threaded version from two chapters ago, we’ll notice a problem: the random number…
In the previous chapter, we solved the pre-test warmup issue, yet the transmission speed was still only 293.461 Gbps, reaching just 9.2% of the total 3200 Gbps bandwidth. We can speculate that the CPU might be unable to promptly process the completion queues of 32 network cards and submit new operations, leading to network card idling. In this chapter, we’ll attempt to solve this problem using…
In the previous chapter, we were indeed able to use 8 GPUs and 32 network cards, however the transmission speed was only 287.089 Gbps, reaching just 9.0% of the total bandwidth of 3200 Gbps. If we carefully observe the running process of the previous chapter’s program, we’ll see that the transmission speed starts around 100 Gbps at the beginning, quickly rises to around 260 Gbps, and then…
In the previous chapter, we clarified the system topology. And in chapter 7, we were able to achieve 97.844 Gbps bandwidth on a single network card. In this chapter, we will use 32 network cards corresponding to 8 GPUs and see how much bandwidth our program can achieve. We’ll name this program 9_multinet.cpp. Network We’ll first make a small modification to the Network class. First, we’ll record…
In the previous chapter, we achieved a transmission speed of 97.844 Gbps on a single network card. In Chapter 2, we introduced that the best transmission speed is achieved by pairing GPUs with nearby network cards. Before we start using multiple network cards, we’ll first clarify the machine’s system topology in this chapter. We’ll name this program 8_topo.cpp. Obtaining System Topology To obtain…
In the previous chapter, we implemented GPUDirect RDMA WRITE. However, in the previous chapter’s program, we only submitted 16 WRITE operations at once. If we transmit more data, fi_writemsg() would return -EAGAIN due to network congestion, and our program did not handle this situation. In this chapter, we will implement an operation queue that temporarily stores operations that cannot be…
In the previous chapter, we implemented bidirectional RECV and SEND, which are two-sided RDMA operations. In this chapter, we will extend the previous program to implement WRITE, which directly writes to remote memory. WRITE is a one-sided RDMA operation (One-sided RDMA) that does not require the participation of the remote CPU. In this chapter, we will directly write to GPU memory. From the…
In the previous chapter, we implemented unidirectional receive and send. In this chapter, we will extend the previous program to implement bidirectional receive and send. After the server receives a message from the client, it will reverse the message and send it back to the client. We’ll name this program 5_reverse.cpp. Different Types of Messages To allow the server to send a message to the…
After the groundwork in previous chapters, we can finally start writing code. This chapter’s goal is to implement unidirectional RECV and SEND between two machines using a single network card. Although libfabric is written in C, since I’m not familiar with C, I’ll be using C++ which I’m more comfortable with. Also, since this series is a tutorial, I won’t deliberately encapsulate C++ classes but…
libfabric is a generic high-performance network interface that’s very similar in style to the RDMA ibverbs interface mentioned earlier, but is easier to use. Applications only need to call libfabric’s high-level interface, while specific protocols are implemented by different Providers. libfabric’s official Providers include tcp, udp, shm (shared memory), verbs (i.e., RDMA ibverbs), and most…
Before introducing libfabric, let’s think about high-performance network library design from a higher level, which will help us understand the interfaces and programming models of both ibverbs and libfabric. On this topic, the libfabric official introduction article is very well written, and I recommend interested readers to read it carefully. In early 2021, I wrote a simple RDMA communication…
Before we start writing code to operate the 3200 Gbps network, let’s understand the hardware, software, and system design philosophy required to drive this high-performance network. RDMA Most networks we use daily are based on TCP/IP protocols. Applications interact with the Linux kernel through sockets. The Linux kernel maintains a complete TCP/IP protocol stack and is responsible for controlling…
Earlier this year, I had the fortune of joining Perplexity AI, where I finally got to use servers with the most powerful configuration—AWS p5 instances equipped with 8 NVIDIA H100 GPUs interconnected via NVSwitch. What excited me even more was the ultra-high-speed 3200 Gbps network between servers. I thought it would be incredibly cool if I could write a program that could utilize this full 3200…
As open-source pre-trained Large Language Models (LLMs) become more powerful and permissive, more and more users are incorporating LLMs into their projects. An essential adaptation step is the integration of domain-specific documents into the pre-trained model, known as fine-tuning. Often, the additional knowledge from domain-specific documents is minuscule compared to what the pre-trained model…