A new project on ECC and Ethernet

One of my interests is error correction, from time to time I consult on ECC mostly for DRAM, but I noticed that Ethernet has some awesome ECC specs. You can read more about them here:

https://www.signalintegrityjournal.com/articles/3405-200-gbps-ethernet-forward-error-correction-fec-analysis

What Dr. Liu passes over is how difficult that level of ECC is. The ECC can correct any 15 symbols, each 10 bits, in a 5440-bit FLIT, and finishes the work about 30ns after the end of the FLIT. By comparison, ECC on DDR5 typically corrects one 64-bit symbol in a 640-bit memory transfer. The DDR5 may be implemented as 8-way interleave of single corrections of 8-bit symbols.

Interleave is relatively easy. Each is correcting one symbol per 80 bits. Single symbol is easy and there are just 10 positions to be considered in each 80 bit interleave.

By comparison 15 symbols have an astronomical number of possible combinations out of 5440 bits (544 positions, drawn 15 times). And it is achieved in about 56ns which includes the 26ns it takes for the FLIT to arrive at 224Gbps. This requires top-notch skills and a LOT of parallel silicon. The clock rate, incidentally, is typically 1.56 GHz with the ECC logic consuming 136 bits in parallel.

So I got to wondering, just how much advantage is there in correcting 15 symbols with no interleave? Interleave could make the silicon simpler and the latency faster, so there must be a good reason. So, I wrote a small project to compare 15-symbols at any position, to 15 symbols handled as 5 interleaves of 3, or as 3 interleaves of 5.

https://github.com/TanjIsGray/Ethernet-ECC-interleave

As you can see in the PNG file

https://github.com/TanjIsGray/Ethernet-ECC-interleave/blob/main/failure_rates.png

All 3 solutions reach very high correction rates and reliable FLIT transmission - eventually. But true 15-symbol correction gets to near perfection even with 3x more burst errors than a 3-way interleave, and 10x more bursts than the 5-way interleave.

This no-excuses approach to ECC allows ethernet to extend to longer cables and weaker signals than something like PCIe Gen6 or CXL, which protect their FLITs with a triple interleave of single-symbol corrections. These PCIe-based transports do have hop-by-hop retry, so they are able to deliver reliably even with higher rates of FLIT failure, and they are tuned for smaller FLITs and easier ECC in order to keep latencies low. Still, those tradeoffs may not scale. As we get to Gen7 and Gen8 speeds (128G and 256G) they will need to deal with higher burst error rates due to worse insertion losses at higher frequency, putting them on essentially the same set of compromises that Ethernet has been on for a few years now. Meanwhile, Ethernet whether with UAlink, SUE, UEC, or Spectrum-X, is moving their retry mechanisms down to the transport layer where, just like PCIe, they offer transparency with minimal jitter. The capabilities are converging as they both reach the same physical limits on error rate and low inherent latencies in single-lane FLITs at such high speed. It raises the question of whether we really will continue with two ecosystems for physical transport, or PCIe/CXL should just pivot to carriage on the same transport that Ethernet has pioneered.

I will leave you with the thought that the ECC in Ethernet is a monster hero. It is just 40 years since NASA figured out how to decode Reed-Solomon fast enough to make the inclusion in Voyager - with R-S capabilities that were not solved at the time they launched - a very successful bet at thousands or even hundreds of baud. Now we have a few square mm of silicon at the end of an ethernet cable that can handle 15 symbols at 224G on 8 lanes and finish the decode in 30ns. This is a large part of what has kept copper in play, and allows glass fiber to reach 30km or more between repeaters. Plus, talking to StarLink and NASA satellites at modern data rates.

All based on some really pure math, Galois fields. Right up there with Fourier analysis which is running in the digital signal processors which untangle the signals to recover the individual bits from noise and dispersion. It seems that abstract math has some amazingly concrete benefits.

PCIe Gen6 (which is shared by CXL at 64Gb/s) uses a flit with 128 bytes and 3-way interleaved correction for 1 byte on each interleave. I compare the flit success rate with that for using the same bit budget with a single Reed-Solomon(128,122) code that can correct 3 bytes at any location. Just as with Ethernet, the single stream with stronger code is the better choice.

The surprise is how much better. The simulation predicts that when burst error rates are good, the FLIT error rate will drop 1,000x using the better correction. A t=3 R-S code will only take a few ns extra time to decode with modern logic.

Interleaves in general seem a weak choice given the power of modern logic.