Putting Together a Deep Learning Rig

Of late I’ve been quite into playing with and training neural networks, unfortunately the constraints of my main computer precluded experimenting with larger networks—training a network, even up to a few million params, is intractably slow. And so I decided that I need a computer with a decent GPU in it.

Putting together this computer ended up being an unanticipated ordeal. In this post I’ve gone over why I went with this particular build, the pains in sourcing the components, and finally setting it up and ensuring everything works.

Unfortunately this post is neither sufficiently short nor sufficiently descriptive, it sits in the uncomfortable middle-ground between the two. So, considering that it should have been like three or four separate posts, feel free to jump to whatever section interests you from the index below.

Note: All prices are in USD, for INR multiply by 100.

The desire for new technology springs forth, first, from the irrational alcoves of the heart. In its wake, the mind is then left scrambling for a justification to the part that enquires: don’t you already have a good enough computer?

Justifying the GPU

My daily driver is an M1 Pro Apple laptop with 32GB of Low Power DDR5 unified memory. Contrary to what HN comments may lead one to believe, not all deep learning workloads are bandwidth constrained. With 200GB/s of bandwidth, and 10.6 TFLOPS of FP16 compute1, for my purposes, this machine happens to be compute bound.

Consider training a 50M parameter model, each training step requires moving around 44GB of data to perform 15 trillion operations. So at M1 Pro numbers, moving data takes 220ms, and compute takes 1,415ms.

Practically what this means is, if I have to train even such a tiny model, for say 100k steps, it would take me nearly 2 days just for the compute.

This kinda would’ve been fine, if it wasn’t for the laptop being close to unusable during the entire run. Had it not been compute bound, the same workload would’ve taken less than half a day, so at least overnight training would’ve been feasible.

A superficially-sensible suggestion I’ve come across is to rent GPUs when I need them. An H100 SXM—with nearly 1PFLOPS of compute and more than 3TB/s of bandwidth2—would get the training run done in less than half an hour for under $4/h. If I set a budget of $3000, that’s 750 such runs.

This is only superficially-sensible cause I’ve found, for me, usage based pricing impedes experimentation. It puts me into a parsimonious state of mind, rationing out compute instead of throwing things at the wall to see what happens. This doesn’t happen if I own the device.

To understand my point, consider how your life would be different if the only way you could learn programming was by renting out AWS EC2 instances. The cheapest one is at $0.0042/h.

Deciding the specs

Comically enough, my plan to build a PC became a concrete one only after I almost got scammed3. Through that incident, I learned of a few harsh realities. The apposite one being that, barring a miracle there’s no way I’m finding a sensibly priced 4090 or 5090.

So realistically, 3090 was my only choice. Inspired by Chan Kha Vu’s Iva, I decided to spec out my system around dual a 3090 setup.

Considering deep-stupidity going on with RAM prices, 64GB of DDR5 RAM was above $800. I did not want pay that much, so I decided to go with a DDR4 system. Defaulted to AMD cause better power efficiency, and cost per core. Generally a much better deal than Intel.

DDR4 meant I had to build around the, nearly 10 year old, AM4 platform. This also meant no PCIe 5.0, which was fine cause 3090 is a PCIe 4.0 card. Considering the age of this platform I knew parts weren’t gonna be easily available.

For the CPU I decided anything above the Ryzen 5600X would by fine by me. And for the motherboard—to be able to run distributed workloads—I’d need two PCIe slots that run at same speeds5. My best option was the x570 motherboard; higher end variants allowed for a dual card setup with PCIe 4.0 x8/x8 configuration.

Besides this and 64GB DDR4 RAM, rest of the specs mattered less, just needed to hit the sufficiency mark on those, I’d pick up whatever reasonably good components I find a decent deal on. To list it out, this was my plan:

1. GPUs: 2x Nvidia RTX 3090
2. CPU: One of Ryzen 5600X, 5700X, 5900X, which ever I find at a good price
3. Motherboard: Any x570 that support PCIe 4.0 x8/x8
4. RAM: 64GB DDR4
5. PSU: 1300W Gold, Platinum may not be worth depending on the cost6

Sourcing sadness and component compromises

Sourcing started with the GPU; CPU and RAM were a self-contained ordeal; the rest followed after.

The GPU

Soon after getting almost scammed3, I found a 3090 on OLX. I called up the guy and, since I didn’t have a PC to test the card out, decided to meet him at Lamington Road. He knew someone there who’d put the GPU through the gauntlet that is FurMark.

I met the guy, and tested the card out. It ran hotter than I’d liked, 105°C at the memory junction after about a minute or so of FurMark. Stale thermals perhaps, I wasn’t aware of how much this would cost, so lobbed off just $10 from the asking price. I ended up paying $710 for the card, and since the fellow seemed nice, I picked up a used 2TB Crucial BX500 from him for $110.

After meeting the guy, I decided to do a vibe-check on RAM prices—I had not decided yet on the other components. The first store I walked into quoted me $60 for a Samsung stick of DDR4, the next one $50. Whoaa! I thought, this road remains untainted by the AI-induced nonsense of the outside world! How much for a used 3090? $700. Insane! Did I just overpay?

After having endured an hour and a half of local trains, I was rather exhausted, so decided to hold off on any purchases, and promised myself to return soon.

Return to Lamington Road

Two days later, I returned with my plan in place. Walked into the first store, asked about the processors, the motherboard, and the RAM—Processor is more expensive than I’d anticipated, motherboard is not available, RAM is now above $60 per stick. No worries, I’ll just check the next store.

More than 20 stores and no motherboard. Wtf is going on? I thought Lamington Road had my back?! After spending more than 4 hours, walking from store to store, I’d grown tremendously weary.

Even though the RAM here was more expensive than day-before, it was still cheaper than online. And I wasn’t finding any of the CPUs online either. So, decided to pick up 32GB of DDR4, and a 5700X.

The sketchy stores and fake RAM of Lamington Road

Store A, best rates at the sketchiest store

I return to the store with the lowest total, tell him I want to buy.

Before I’d left the store the first time, I’d asked him if what he quoted was his final rate, he said yes. Now he applies 18% GST on both the items. The deal was worse than what he’d made it out to be. I’m annoyed, but I persist.

The guy doesn’t have the components at his store, so he sends out a lackey to fetch the CPU. The lackey returns, ten minutes later, with a bare CPU in his hand, as though he was returning with a mango he’d plucked off a tree. The store owner takes the CPU and places it in a blister pack that says AM5. Annoyance amplifies. As he places it in a the blister, he tells me that this is how they come, cause they’re OEM pieces, so there is no box. Okay.

Meanwhile, the store had (surprisingly) managed to attract a few other people. One of them was a guy I’d noticed at two other stores, asking around for an AMD GPU and some RAM. He looks at the blister pack, points at it and asks me whether it’s a used piece.

In response to his inquiry, the store owner lowkey flips out at the guy, telling him to not disturb the customers, and to either buy something or to step back from the counter. By this point, I was pretty sketched out by the owner.

I told him I won’t be buying the CPU, and asked for the RAM. The owner seemed pretty flustered by this. He asks me to first pay for the RAM before he sends out his lackey to fetch it. How preposterous, make a deal without seeing the goods? I tell him I cannot do that, and walk out of the store.

Store B, don’t buy the fake RAM

It had started getting dark, and only smoke remained of the dying embers of my patience. I wanted to wrap up soon.

I go to the next best store. Here too the owner gets a lackey to pullup with a 5700X in hand, places it in a blister with AM5 embossed beneath where the CPU sits. He said it comes with 3-year AMD warranty, and the CPU looked new— unblemished. So sans patience, I acquiesced to the deal. $170 for the dense-feeling chip.

I asked him about the RAM, he quotes me a higher price than a few other stores. I tell him if he can match the price, I’ll just buy it off of him.

He responds saying that the other stores are selling fake RAM, and that I’ll get blue screens if I buy it. Huh, fake RAM? I tell him I’ll check it out and return.

I query one of the LLMs, it tells me to check if the store is offering manufacture warranty or store warranty. Gotcha.

Store C, there is only fake RAM

I step into the second shop I’d visited—cause 32GB for $130. I tell him I want to buy RAM, and I ask him whether it comes with warranty. Yes he tells me. Manufacture warranty right? Nope it’s store warranty. omg these people are selling fake RAM

I ask him if it’s fake RAM. He calls it “pulled RAM” and gives me a vague account of its provenance. It’s pulled from used office computers. So it’s used? No, no it’s brand new Kingston 3200MHz RAM, it’s just pulled. Wtf, how is that not used or fake? Sounds like both to me!

I ask him how much for proper non-fake RAM, he says $220. Oh… I tell him how the last store I went to is offering me legit RAM for like $160. It’s fake RAM, he responds. I tell him I’ll check and return.

I get back to Store B, from where I picked up the CPU, and ask him whether his RAM comes with manufacture warranty. Nope, store warranty. Fml, these people have put up a facade, nothing is as it seems at first blush. It’s all fake RAM! As I’m leaving his store, the owner tells me his RAM is made with original good quality chips.

There are precisely 3 companies in the world that make the DDR chips, it’s not like the others would be getting theirs from some artisan in Dharavi.

I succumb to the price

I return to Store B, even with GST applied (18%), it was still cheaper than whatever I’d seen online. I tell him that he was right, it was fake RAM, he responds with a smug itoldyouso smile and asks me to wait.

This was the busiest store I’d been to, there were like 8 people—buying, enquiring, getting things fixed—standing in the 16sq.ft area behind the counter.

I end up waiting for ten minutes until the finally one of the shop lackeys return with two sticks of DDR4 RAM. Each RAM in it’s own plastic case with a transparent front, each of them with very obviously fake Kingston stickers. Man this entire road is so sketchy.

I ask him if he’ll test it out, he says there’s three years warranty, if there’s any issue I can return to the store. Okay, whatever. I tell him I’ll take it. To my surprise, the $130 included the GST amount in it. That ends up being the final amount I pay for the RAM.

I leave the store, and Lamington Road, hoping I never have to return.

Lamington Road—a frustrating street of unscrupulous knaves—ended up being more expensive than I had expected. My return would’ve been entirely worthless if it wasn’t for the 5700X in the AM5 blister pack, or the probably-used 32GB DDR4 RAM with the fake Kingston stickers that I begrudgingly purchased.

Compromises and final purchases

I spent the next couple of days, once again, scouring the internet for the mythical x8/x8 x570. No luck. It looked like I was gonna have to compromise on my plan. I was pretty bummed out, I was really looking forward to distributed training across 48GB of VRAM.

After coming to terms with the situation, I decided that if I’m gonna build a single card system, I don’t want it ensconced in some megalith of a cabinet. Devices of dense capability soothe my aesthetic sensibilities. I am averse to things that take up more space than they need to.

So I decide to go for the B550 smallest motherboard I can find, and a case that’s just large enough to fit the behemoth that is 3090 Founders Edition—a card Nvidia has referred to as their Big Ferocious GPU7.

Within a days time I figured out the rest of the components for the build. No way in hell I was returning to Lamington Road, so I decided to just source the rest online.

Found the remaining components on mdcomputers—a store from Kolkata—and placed the order. Within a day’s time, other than the cabinet, I received all the components. The cabinet took another six days, that excruciating wait was spent in a state of arrested anticipation.

The final list

Total cost $1464.3. 60% compute, 11% storage, 9% memory, 4% cooling, and 16% to bring it all together

Lamentations

I set out to build my rig with dual GPUs in mind; and to accommodate that, a generous budget of up to $3000.

If it wasn’t for the RAMageddon, I’d have been able to build a dual 4090, 64GB DDR5, rig without hitting the limit—used 4090s around $800 each4, 9700X for $250, 64GB DDR5 under $200, x870E for $400. With enough left over for the case, cooling, and storage.

For the ~$1,500 I’ve spent now, I’d have been able to bag a single 4090, 32GB DDR5 build.

I have reason to lament, not only for the lost dual 4090 build, but also the fact that even a dual 3090 wasn’t possible. The AM4 platform is too old, supply for dual GPU supporting x570 motherboards seem to have entirely run dry.

AM4 Acceptance

All things considered, I do think I was able to build very capable computer.

Since the build was an AM4 one, it wasn’t that affected by the current situation. If the inferiority of certain components may be ignored, I maybe paid a premium of of around $170, roughly 13%. If I was putting together a current-gen AM5 build (same GPU, similar CPU, etc), this would be around 20% extra over standard.

Ultimately for my purposes what matters is the GPU. Rest or the components, as long as they meet some sufficiency criteria 11, it doesn’t really matter. If you have spec’d out your computer well enough, the GPU will be the bottleneck for any deep learning workload you throw at it. So improving the rest won’t matter as much.

Aside: I think older hardware—AM4, DDR4, older GPUs14—may be making a come back. A few days ago AMD relaunched their 5800X3D, and Nvidia is restarting production of the RTX 3060. Wonder if dual GPU AM4 will be restocked as well?.

The build

Like most things pertaining to this journey, assembling the computer too wasn’t a straightforward process. The complications stemmed from my insistence on fitting a 3 slot wide, 313mm long card in a 375mm long micro-ATX case.

Everything started off fine:

1. Removed the case’s top and side panels.
2. Removed the PSU cage, fastened the PSU to it.
3. Placed the CPU in the motherboard socket.
4. Placed the NVMe in its holster on the motherboard.
5. Placed the motherboard on the risers in the case.
6. Screwed the SSD to its case mount.
7. Wired up the SATA cable.
8. Installed the modular PSU + cage in the case.
9. Wired up all the required power-cables to the PSU.
10. Installed the rear case fan.
11. Placed the GPU in the… uh oh

The GPU doesn’t fit in the case. After plugging in the needed modular cables into the PSU, there isn’t enough clearance underneath it. The GPU can’t be seated with the PSU cables being where its butt should be.

I was starting to get anxious, worried that I picked up too small a case. I’d spent a couple of hours or so on the assembly, so decided to take a 20 minute break.

I get back to the assembly, take off all the PSU cables, the GPU now fits. I start reinstalling the cables, this time I use the lower slots (closer to the front), for the SATA power cables. Everything but the motherboard cables fit. There isn’t space for me to plug in the motherboard cable or pull the other end to the motherboard. I take it all off again.

After struggling for five minutes, I manage to wire up the motherboard cable. Nice! Now onto the others, thirty minutes later, all the power-cables have been wired up!

Powering the 3090 Founders Edition

TLDR: The PSU’s 12VHPWR will work with the 3090 FE if it fits and supplies at least 350W.

The 3090 Founders Edition comes with a non-standard 12pin power socket. To accommodate this, Nvidia provides an adapter in the box. One that splits the 12 pin into two 8 pin inputs.

The seller from whom I bought did not have this cable. And so I was worried about how I’d power this GPU. Later generation large GPUs made use of the 12VHPWR, this was a 16 pin socket that looks overtly incompatible with the 3090.

Most modern PSUs come with this cable—the 12VHPWR—in the box, none come with a cable that is meant for the 3090. It looked like I would have to shell out more for a moddiy cable and shipping, the worst part of course being the wait.

Luckily, I came across a Reddit thread where the poster faced a similar problem, but concluded in the comments saying that the 12VHPWR works with the 3090. I was a bit relieved, but want to blindly trust a good willing internet stranger. So I quickly checked the pin outs.

Note: the images are mirrored and the socket shape for pin 7 is incorrect on one of them.

The pin outs seemed to match, so as long as the sense pins cleared the socket and the remaining 12 pins fit the 3090, I should be good to go.

As it turned out later, the PSU cable did fit and my GPU was able to draw power from it.

I then plug in in the Fake RAM into the appropriate DIMM slots12. The sticker placement on the sticks throw me off initially, but a minute later it’s seated securely.

Next I screw in the higher pressure ceiling fans; a bit annoying cause it fell into the case more than once.

After fastening the fans, all the components were in place. Finished off the wiring up by plugging the front panel USB and other cables to the mobo, then plugged in the fan cables which are concealed along with the power cable behind a shroud.

I’d spent nearly half a day, it was past 2 in the morning, I just needed to power on the computer, see it light up, fans spinning, and then I could go to bed.

Unfortunately, not so soon. The PSU came with a fat 16A plug, not a standard 6A one. There are only two 16A sockets in my house—one in the kitchen for the oven, and one in the bathroom for the heater. I contemplated setting up in the kitchen, but was thwarted by my monitor being fastened to the wall.

Looked like I’m gonna have to wait at least another day. I left the PC open and went to bed. Should anything not be proper, I didn’t want to unscrew the panels.

Next day, first thing in the morning I head out to fetch either a 6A plug PSU cable, or a 16A to 6A adapter—a footgun of a contraption which I’m sure has lead to countless fire alarms being set off.

On 16A plugs and the adapter

Considering just 750W, the 16A plug seemed like an overkill. A smaller standard 6A plug would’ve sufficed just as well.

I surmised this was for compatibility reasons. In India, we get 250V, so for 750W the current draw would be around 3A. But in other countries that use similar D type plugs, but at a lower voltage, say ~120V, current draw would be much higher, above 6A.

I figured, even accounting for say ~80% PSU efficiency, peak power draw at 940W, it gets only up to 3.8A. And considering 2x power spikes at peak, up to 7.5A. So I should be mostly fine fine using 6A plug—highly unlikely the computer would draw even 500W at peak operation (sans spike).

I found an adapter for really cheap, $0.85, decided to use it until I got a 10A plugged power cord.

Note: I later purchased a power meter to check peak GPU power draw, when running Furmark, the computer pulls under 420W. Note that this does’t stress the CPU so still not at peak draw but at least gives an idea.

After using the 16A to 6A adapter, the computer lights up, fans spinning. Few seconds of memory training, the screen—plugged to the GPU through HDMI—lights up. IT LIVES!

The PC boots into a broken windows install asking for recovery, tf?! Turns out the SSD had some stale MBR partition. Okay, I’ll deal with that later. Boot into BIOS to check if everything is being detected.

Nope… things are weird: RAM is at 2933MHz, and NVMe is not being detected. Dammit, it’s one thing after another.

I lay the case on its side, and without removing anything else take the NVMe out—in the process, losing one of the screws in the case. I then, place it in my external enclosure and plug it into my laptop. Works, looks like a seating issue.

Spend 30 minutes carefully putting it back without disturbing any other components, and reboot. It’s being detected.

RAM running at lower than advertized clock speed is normal, unfortunately it didn’t come with any XMP/EXPO profiles, so I’d have to manually increase the clock speed.

Now that everything is being detected, I can finally case it up! I screw on the top panel and the side glass panel, and move on to the next stage.

Using another laptop, I flash Ventoy onto an old pen drive, move Fedora Workstation 44 onto it, boot from the pen drive and install the OS. Works flawlessly on the second try. I restart the computer, and it boots into the OS!

System checks

Since I bought a bunch of probably sketchy components, I wasn’t comfortably trusting them and decided first to run a couple of system checks.

Not being on Windows means I couldn’t rely on some of the standard tools like GPU-Z, but whatever I did end up using worked well enough for my purposes.

SSD Checks

First was the SSD, I’d almost picked it up on a whim, and so was at a bit of unease regarding whether I’d made a mistake or not. I used smartctl from smartmontools to check its SMART stats—the SSD self reports, and keeps track of, various metrics that help in determining how reliable or worn out it is.

You can see the relevant part of the smartctl -a /dev/sda dump below:

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          VALUE WORST THRESH TYPE      UPDATED RAW_VALUE
  1 Raw_Read_Error_Rate     100   100   000    Pre-fail  Always  0
  5 Reallocate_NAND_Blk_Cnt 100   100   010    Old_age   Always  0
  9 Power_On_Hours          100   100   000    Old_age   Always  2353
 12 Power_Cycle_Count       100   100   000    Old_age   Always  47
171 Program_Fail_Count      100   100   000    Old_age   Always  0
172 Erase_Fail_Count        100   100   000    Old_age   Always  0
173 Ave_Block-Erase_Count   099   099   000    Old_age   Always  12
174 Unexpect_Power_Loss_Ct  100   100   000    Old_age   Always  29
180 Unused_Reserve_NAND_Blk 100   100   000    Pre-fail  Always  35
183 SATA_Interfac_Downshift 100   100   000    Old_age   Always  0
184 Error_Correction_Count  100   100   000    Old_age   Always  0
187 Reported_Uncorrect      100   100   000    Old_age   Always  0
194 Temperature_Celsius     066   063   000    Old_age   Always  34 (Min/Max 20/37)
196 Reallocated_Event_Count 100   100   000    Old_age   Always  0
197 Current_Pending_ECC_Cnt 100   100   000    Old_age   Always  0
198 Offline_Uncorrectable   100   100   000    Old_age   Offline 0
199 UDMA_CRC_Error_Count    100   100   000    Old_age   Always  0
202 Percent_Lifetime_Remain 099   099   001    Old_age   Offline 1
206 Write_Error_Rate        100   100   000    Old_age   Always  0
210 Success_RAIN_Recov_Cnt  100   100   000    Old_age   Always  0
246 Total_LBAs_Written      100   100   000    Old_age   Always  12353141681
247 Host_Program_Page_Count 100   100   000    Old_age   Always  386035677
248 FTL_Program_Page_Count  100   100   000    Old_age   Always  201554944
249 Total_Refresh_ISP_Count 100   100   000    Old_age   Always  0
251 Tot_NAND_Rd_Plane_Cnt_L 100   100   000    Old_age   Always  0x0000a30585ea
252 Tot_NAND_Rd_Plane_Cnt_H 100   100   000    Old_age   Always  0x000000000000
253 Tot_Blk_Remap_Pass_Cnt  100   100   000    Old_age   Always  0

In short, over 98 days of usage, the SSD has been through 47 power cycles, has seen no failures (read, write, or erase), and has had 5.75TB written, i.e. less than 1% of it’s TBW of 720TB.

Looks like this was a good deal!

RAM Checks

The RAM was running at 2933MHz instead of the 3200MHz that I’d paid for, I manually set it to 3200MHz, leaving the voltages at 1.2V. I was fully expecting the change to not sit well with the computer, but to my surprise, it just worked!

I then downloaded memtest86+, put it on the Ventoy pen drive and booted from it, and set it to test the sticks.

In less than 30 minutes, memtest86+ completed a pass without any errors. Unknown manufacturer notwithstanding, RAM seemed to be fine too!

GPU Checks

The 105°C I saw before buying the GPU had been bothering me ever since I read up on it (after the fact). And it looked like I was gonna have to spend more time and money redoing the thermals. But before I dove further into that I thought I’d rerun the tests.

So I ran Furmark to boil the GPU, and tracked it using LACT:

It looked like the card was being power throttled—SW_POWER_CAP in the second screenshot—to prevent it from being boiled alive. Temperature curves stabilized once the fan kicked into a higher gear.

Probably the test bench where I saw the GPU hit 105°C was running older drivers? Not sure, but power throttling is fine by me. I was planning on either under-volting or power-limiting the card anyways. This seems to have removed the need for it. Also, looks like I won’t be needing to redo the thermals.

Note: I later ran gpu-burn, check the pseudo-undervolting section for this.

Edit: the aforementioned 105°C pertained to memory junction temperature. This isn’t reported on Linux, and is different from the VRAM temperature in the screenshot above. Also, the SW_POWER_CAP is normal, the 3090 is capped to the factory set value of 350W, and the throttling is kicking in cause it’s reached that limit. For overclocking purposes the LACT slider goes upto 400W.

Perf checks

The handful of system checks I ran above were sufficient to satisfy me. I was good to go.

I christened the system astora, and spent the next couple of days porting my dotfiles and setting up the system to my liking. I found Fedora to be a rather amiable OS—no conspicuous bloat, doesn’t get in the way, and most configurations don’t require reaching for a terminal, super keyboard friendly. Shame that I’d largely be using this system remotely.

Finally, I’d gotten to the point where I can test the GPU—count the FLOPS!

Counting FLOPS

To count FLOPS I wrote a simple pytorch function, and ran it on the laptop, and the computer.

import torch
import time


def count_flops(name=None, device="cpu", duration=10, n=1024, dtype=torch.float32):
    synchronize = lambda: None
    if device == "mps":
        synchronize = torch.mps.synchronize
    if device == "cuda":
        synchronize = torch.cuda.synchronize

    # Warmup run
    for _ in range(5):
        a = torch.randn(n, n, dtype=dtype, device=device)
        b = torch.randn(n, n, dtype=dtype, device=device)
        _ = a @ b
    synchronize()

    # Actual run
    durations = []
    start_ = time.perf_counter()
    while (time.perf_counter() - start_) < duration:
        a = torch.randn(n, n, dtype=dtype, device=device)
        b = torch.randn(n, n, dtype=dtype, device=device)
        synchronize()

        start = time.perf_counter_ns()
        _ = a @ b
        synchronize()
        dur = time.perf_counter_ns() - start

        durations.append(dur)

    durs = torch.tensor(durations, dtype=torch.float64)
    tops = (n * n * (2 * n - 1)) / 1e12  # trillion operations per mat mul

    tflops = tops / durs.div(1e9)
    durs_ = durs.div(1e6)  # from ns to ms
    m = dict(
        name=name,
        device=device,
        test_n=n,
        test_duration=duration,
        dtype=str(dtype).split(".")[1],
        count=len(durations),
        single=tops,
        # FLOPS
        mean_tflops=tflops.mean().item(),
        min_tflops=tflops.min().item(),
        max_tflops=tflops.max().item(),
        p95_tflops=tflops.quantile(1 - 0.95).item(),
        p99_tflops=tflops.quantile(1 - 0.99).item(),
        # DURATIONS (ms)
        mean_dur=durs_.mean().item(),
        min_dur=durs_.min().item(),
        max_dur=durs_.max().item(),
        p95_dur=durs_.quantile(1 - 0.95).item(),
        p99_dur=durs_.quantile(1 - 0.99).item(),
    )

    print(
        f"{name=} {device=} {dtype=} {n=}\n"
        f"  - count: {m['count']:10,}\n"
        f"  - single:{m['single']:10.6f} TOps\n"
        f"  TFLOPS\n"
        f"  - mean:  {m['mean_tflops']:10.3f} TFLOPS\n"
        f"  - min:   {m['min_tflops']:10.3f} TFLOPS\n"
        f"  - max:   {m['max_tflops']:10.3f} TFLOPS\n"
        f"  - p95:   {m['p95_tflops']:10.3f} TFLOPS\n"
        f"  - p99:   {m['p99_tflops']:10.3f} TFLOPS\n"
        f"  DURATIONS\n"
        f"  - mean:  {m['mean_dur']:10.3f} ms\n"
        f"  - min:   {m['min_dur']:10.3f} ms\n"
        f"  - max:   {m['max_dur']:10.3f} ms\n"
        f"  - p95:   {m['p95_dur']:10.3f} ms\n"
        f"  - p99:   {m['p99_dur']:10.3f} ms\n"
    )
    return m

Here’re the results:

Notes:

• TFLOP numbers are P99 values, the variance is low so mean is only slightly higher.
• Number of iterations (iter column) is not the same cause I was controlling for duration, as many runs in 10s. I did multiple of these runs, results were largely similar, very little variance.
• 3090* uses tensor cores by setting torch.backends.cuda.matmul.allow_tf32 for 32 bit floating point (else it does not use tensor cores). fp16 by default uses tensor cores.
• MPS uses Metal Performance Shaders by setting device="mps".
• xMPS column shows performance relative to the M1 Pro MPS numbers.
• M1 Pro is the 16 core variant.

I was a bit annoyed at first cause the 30 series Wikipedia page says that the 3090 hits 142 TFLOPS of Tensor compute, and I’d come to expect that. This is apparently incorrect, it’s Nvidia’s marketing number pertaining to specifically formatted sparse matrices. But after the annoyance faded, I was pretty thrilled that the card computes 15 times faster than what I was using.

This is a synthetic benchmark, actual relative performance may be higher considering memory bandwidth is nearly 5 times higher then the M1. I wanted to run a few steps of the setup I’d described earlier, but this post is already too long so decided to keep it for later.

Pseudo-undervolting

I wasn’t initially planning on undervolting, but considering I was worried about the memory junction temperatures, I thought I’d test it out, and eventually ended up keeping the settings.

The usual way of undervolting (ref)—applying a negative offset on the GPU clock, then pulling the VF curve up at 850mV—didn’t seem to have the intended effect. The performance peak went up a bit, but power consumption didn’t drop, it still hit 350W. So it’s not like the GPU would be running cooler cause of the undervolt.

I came across the idea of instead:

1. applying a powerlimit to ensure that the GPU runs cooler, then
2. boosting GPU clock to make up for the drop in performance

So, using LACT I did just that. This wasn’t an explicit undervolt because you don’t touch the VF-curve, but the effect should be similar, while also being explicit about the power limits.

I used gpu-burn to test the stability of my settings. To check that it works, I set some unreasonable values and surely the number of errors started racking up before the computer stopped responding. gpu-burn works, so I used to dial in reasonable settings—set power limit to 300W, and added a GPU clock offset of +225MHz, and while I was at it a +500MHz on the VRAM clock as well13.

As you can see above, no errors! But the sharp-eyed may notice FLOPS dropping, from 57.3TFLOPS to 52.7TFLOPS. What gives? Turns out I was right to be worried about the memory junction temperatures. After about 5 minutes of gpu-burn, thermal throttling was kicking in to cool the GPU. This most certainly shouldn’t be happening on a power limit.

Being on linux I couldn’t check the memory junction temperatures, the drivers don’t expose it like on Windows (where GPU-Z can be used). So I was going by just GPU and VRAM temperatures, which seemed to be very much under control, both under 65°C. And since those were under control, the GPU fans weren’t running fast enough, only around 50%.

It was obvious that the thermals were a bit stale, eventually I’d have to change the thermal pads and paste for a 10°C drop. But for now I decided that I’ll just set more aggressive fan speed/temperatures curves.

This seemed to have solved the problem. Running gpu-burn again, I saw that, GPU temperatures stayed consistently around 55°C, and compute around 56.8TFLOPS.

After applying my settings, I reran my FLOP counter and got 66.012TFLOPS. A 7.4% drop in performance for a 14.3% drop in power consumption, and cooler GPU. A reasonable exchange!

Closing thoughts

I had, at the same time, underestimated the amount of effort taken to put together a PC, and overestimated the kinda of PC I’d be able to put together. I thought I’d be able to put together a formidable, dual GPU’d, computer in like a handful of days by sourcing the parts from online stores. I was pretty off the mark.

Nearly a decade ago, I’d helped my brother build his PC, things were pretty straightforward, whether a part will be available or not was not a concern back then.

Now at the tail-end of this nearly three week long side-quest, I can see how bad the PC enthusiast scene has gotten. I had not anticipated scams, sketchy computer stores, malware, fake RAM, mobo shortages, or that all this would even take so long.

This is truly one of the worst times to build a PC.

Even though the process was unfun, and I ended up with a gaming PC, I’m super happy with the end result!

Just having a separate computer dedicated to this singular purpose is incredibly freeing! For instance, now I don’t worry about running out of space whilst downloading large datasets or models, or running code that will freeze my computer when I need it. Neither do I have to worry about raking up expenses on some GPU provider platform.

It feels like my mind has been freed up to focus on what matters!

I’m incredibly satisfied with the build! I didn’t succumb to any RGB components, the power requirements are surprisingly manageable, it’s small enough to sit rather impressively on my desk—only 23L! And finally, it’s powerful enough to do what I need it to; more than a decade later the 3090 is still a beast of a GPU.

This post has been rather long, I shall now go squeeze my computer for all its worth. ✌️

Footnotes