We found that by outsourcing the development of matrix compression algorithms, Apex achieved a ~3x reduction in data transmission size. Our winning algorithm strictly improves latency for network speeds < 50 MB/s.
Therefore, in decentralized training scenarios involving consumer devices connected over the public internet, our compression results are expected to reduce communication latency and improve overall system throughput.
We recently concluded our matrix compression competitions on Apex. The purpose of these competitions was to increase the training speed of IOTA by speeding up data transfer.
We ran both a lossless and a lossy version in order to capture good solutions with minimal degradation of the data.
Once we launch competitions on Apex, anyone can submit a solution (both humans and agents). All innovation is completely outsourced; anyone who wants to take a shot can upload a solution (or many). Our infrastructure continuously evaluates the submissions on real data taken from IOTA. All solutions are open-sourced programmatically after a day or so, which allows other participants to copy and improve their designs. The continuous sharing of results + competitive dynamics drive an evolutionary optimization process through the solution space.
With the competitions closed, we’ve collated our findings, revealing how we’re able to improve our work on distributed training.
V1 (Lossless): 61 days, 30 rounds, 1,638 distinct hotkeys, 16,028 submissions.
Best score: 0.381.
V2 (Lossy): 58 days, 29 rounds, 1,229 distinct hotkeys, 13,445 submissions.
Best score: 0.695.
V2’s opening-round score (0.472) already beat V1’s all-time best.
In order to show the impact of these results we downloaded the winning submission and benchmarked its performance on a MacBook M3 Max.
Let
• S = file size (MB)
• v = network speed (MB/s)
• r = compression ratio (compressed/original)
• k = codec time per MB (compression + decompression)
• b = constant overhead
The time required to transfer an uncompressed matrix is
\(T_{\text{base}}(S) = \frac{S}{v} + b\)
The time required to transfer a compressed matrix is
\(T_{\mathrm{comp}}(S) = \frac{rS}{v} + kS + b\)
Compression is beneficial when
\(T_{\mathrm{comp}} < T_{\mathrm{base}}\)
Equating the two:
\(\frac{S}{v} = \frac{rS}{v} + kS\)
\(\frac{1-r}{v} = k\)
Thus the critical network speed is:
\(v^{*} = \frac{1-r}{k}\)
• If v < v∗: compression reduces latency.
• If v > v∗: compression increases latency.
Note that the file size S cancels; therefore the decision depends only on network speed and compression ratio.
Measured on an Apple M3 Max:
\(t_{\mathrm{comp}}(S) = 0.0107\,S - 0.0025 \tag{1}\)
\(t_{\mathrm{decomp}}(S) = 0.0021\,S + 0.0014 \tag{2}\)
Thus
\(k = 0.0128\ \mathrm{s/MB}\)
For compression ratio r = 0.35 (approx. best compression rate achieved across both compression competitions on Apex):
\(v^{*} = \frac{0.65}{0.0128} \approx 50.8\ \mathrm{MB/s}\)
Therefore:
• Network speed < 50 MB/s: compression improves latency.
• Network speed > 50 MB/s: sending raw data is faster.
However, the lossless compression competition produced results with compression rates which were closer to 0.55−0.6, which reduces the critical network speed to around 35 MB/s. Clearly, it is important to quantify the signal loss from the best lossy competition results.
\(\textbf{Compress if } v < \frac{1-r}{k}\)
With the measured codec:
\(\textbf{Compress if network speed } 50\ \mathrm{MB/s}\ \textbf{ (for } r \approx 0.35\textbf{).}\)
More generally, the table and figure below shows the maximum network speed for compression to be useful as a function of the compression ratio.
\(\begin{array}{c c} \hline \text{Compression Ratio } r & \text{Break-even Network Speed } v^* \text{ (MB/s)} \\ \hline 0.1 & 70.3 \\ 0.2 & 62.5 \\ 0.3 & 54.7 \\ 0.4 & 46.9 \\ 0.5 & 39.1 \\ 0.6 & 31.3 \\ 0.7 & 23.4 \\ 0.8 & 15.6 \\ 0.9 & 7.8 \\ \hline \end{array}\)
Table 1: Break-even network speed as a function of compression ratio, assuming codec cost k = 0.0128 s/MB. Compression improves latency when network speed is below v∗
The measured codec cost is
\(k = 0.0128\ \mathrm{s/MB}.\)
For a representative compression ratio of r = 0.35, the critical network speed is
\(v^{*} \approx 50.8\ \mathrm{MB/s}.\)
Compression reduces end-to-end latency whenever the available upload bandwidth is below this threshold. Typical residential internet connections provide upload speeds between 10–200 Mbps (1.25–25 MB/s), which are significantly below this break-even point. For example, a measured residential connection with 74.9 Mbps upload corresponds to approximately 9.4 MB/s, well within the regime where compression is advantageous.
Therefore, in decentralized training scenarios involving consumer devices connected over the public internet, compression is expected to reduce communication latency and improve overall system throughput.
Only in environments with very high-bandwidth connections such as datacenter networks or high-end symmetric fiber links would transmitting uncompressed data likely be faster.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.