· Moving my homelab off 1 Gbps and onto a 10 Gbps backbone.
Table of Contents
1. Why 10 Gbps?
The short answer: fiber finally became available at my current residence, so I jumped on it immediately.
The slightly longer answer is that I'd previously enjoyed proper 1 Gbps fiber, but for a while now I'd been stuck limping along on "1 Gbps" cable from Cox that, in reality, often crawled along at 100 Mbps regardless of what I did to fix it. New modem, new cables, different ports, calls to support - nothing mattered. It ran at whatever speed it felt like that day.
So when Google Fiber showed up in my neighborhood, I ordered the 8 Gbps package immediately. I'll upgrade to their 10 Gbps package the moment it's released, but I'm not exactly suffering in the meantime.
A quick note on the title before we go further: my internet is 8 Gbps for now, but this post is really about rebuilding my internal network to 10 Gbps end to end. The WAN is just the excuse to upgrade everything.
In the screenshot above, you can see that Google Fiber was installed on 7/20 and UniFi entered WAN failover mode (the blue bar) and then I unplugged Cox on 7/22, resulting in the green Google Fiber bar.
As you can also see, Cox was a train wreck - cycling between yellow (disconnections, lag, etc.) and red (no internet connection) for the previous month.
2. What I Was Running Before
Nothing exotic. My network was built entirely on 1 Gbps gear:
- A 24-port UniFi switch in the rack.
- Two 8-port UniFi switches serving other rooms.
- A Cox "1 Gbps" WAN that, as mentioned, frequently delivered closer to 100 Mbps.
It worked. For years, it was more than enough - I've written before about how my homelab basically maintains itself, and the network was no exception. But once I had real bandwidth coming into the house, 1 Gbps internally became the bottleneck instead of the ISP.
As a result, I installed the hardware I'll list in the next section. You can see the 10G connections in blue and the 1G connections in green.
3. The Hardware
Everything here is UniFi. I'm already knee-deep in the ecosystem, and at this point fighting it would cost me more time than it's worth. Staying in one platform means everything shows up in a single controller and I don't have to think about it - which is exactly how I like my homelab.
This rack shows:
- UDM Pro router.
- 10G 24-port switch.
- Switch panel with keystone jacks.
- 1G 24-port switch.
- My server.
This is before the Google Fiber installation, so you can see that the WAN RJ45 port is in use on the UDM Pro instead of the SFP+ port.
I also removed the old 1G switch after this picture and plan to sell it. I just didn't want to box it up quite yet when I took this photo.
3.1. Switch
Two switches did the heavy lifting for the upgrade:
- USW Pro HD 24 PoE: the new core switch in the rack, replacing my old 1 Gbps 24-port unit.
- USW Pro XG 8 PoE: a smaller 10G switch serving another room, replacing one of the old 8-port units.
3.2. NICs
Neither of my two most important machines shipped with 10G, so both needed a new network card:
Windows Gaming PC: an Intel X520-DA1, a 10G SFP+ card.
Server: an Intel X710-DA2, a dual-port 10G SFP+ card.
The Ubuntu server already had Intel drivers installed, so I wasn't required to install anything. I simply slotted the NIC into the PCIe port on the motherboard and it worked.
However, the Windows machine was a different story. It did come with an installation CD-ROM, but I of course ignored that since I have no disc drive in my PC. I tried to find the right drivers online, but Intel's websites are a vast wasteland of useless downloads and circular links.
Eventually, I pulled a USB-C disc reader from storage and used the CD, which magically installed on the first try and everything started working from there.
That'll teach me (not really).
3.3. Cabling
This is where 10G gets fiddly, and where most of my shopping list ended up. Going 10G means leaving RJ45 behind for SFP+ almost everywhere:
- SFP+ to RJ45 Adapter: bridges the Google Fiber RJ45 handoff into the SFP+ port on my UDM Pro.
- UniFi Etherlighting Patch Cable: short in-rack runs between the switch, the other rack devices, and the fiber modem.
- 10G Direct Attach Cable (DAC): the short SFP+ runs from the rack switch to the server, and from the second switch to the gaming PC.
- 10G Active Optical Cable (AOC): the long run between the rack and the 8-port switch in the other room, where a copper DAC won't reach.
3.4. Cost
None of this is cheap, so here's the full damage:
| Item | Qty | Price |
|---|---|---|
| USW Pro HD 24 PoE | 1 | $999.00 |
| USW Pro XG 8 PoE | 1 | $499.00 |
| SFP+ to RJ45 Adapter | 1 | $65.00 |
| UniFi Etherlighting Patch | 27 | $112.97 |
| Cat6A Keystone Coupler | 2 | $78.00 |
| 10G Direct Attach Cable | 3 | $43.00 |
| 10G Long-Range DAC (AOC) | 1 | $55.00 |
| Intel X520-DA1 NIC | 1 | $45.00 |
| Intel X710-DA2 NIC | 1 | $145.00 |
| Total | $2,041.97 |
4. The Upgrade
With the parts in hand, the actual topology is straightforward. Bandwidth flows in from the street and fans out through the rack:
Google Fiber
│ (RJ45 → SFP+ adapter)
▼
UDM Pro
│ (Etherlighting patch)
▼
USW Pro HD 24 (core switch)
├─ DAC ──────────► Server
├─ AOC ──────────► USW Pro XG 8 (other room)
│ └─ DAC ──► Windows Gaming PC
└─ ETH ──────────► USW 8-port (1G switch)The UDM Pro takes the Google Fiber handoff through the RJ45-to-SFP+ adapter, then hands everything down to the new 24-port switch over an Etherlighting patch cable. From there, the server hangs directly off the core switch on a short DAC, and a long active optical cable carries 10G across the house to the second switch, which finally feeds my gaming PC. My surviving 1 Gbps 8-port switch still hangs off the core switch too, handling everything that doesn't need 10G.
The physical installation was pretty easy. I needed to replace a drive in my server's ZFS pool, so I already needed to get inside the rack. Since I don't use rails, I had to unscrew and pull out each device by hand.
From there, the NIC (and HDD) installation in the server was seamless. Just pop out one of the metal slots on the back of the server, slot the NIC into the PCIe lane, and line up the port with the new opening in the back panel.
All in all, it was easy and simple. No physical snags, only one driver issue on the Windows machine (which was my fault for ignoring the provided CD), cabling runs were easy, and UniFi updates were minimal.
5. Benchmarks
Let's nerd out on the numbers.
5.1. Before
I don't actually have any speed tests from before the upgrade, but the Cox connection had degraded even further by the end - down to roughly 70 Mbps for the month prior to the upgrade.
Occasionally, I was able to get ~800 Mbps when the connection would work properly.
5.2. After
So, here's the catch. I'm not getting the full 8 Gbps currently. When the Google tech installed the modem, we confirmed ~ 8500 Mbps together, which means the fiber is working through the ONT and the modem.
However, they would not connect directly to my router, which means the Google modem is in bridge mode for now.
The speed test below shows an average speed, where I'm getting 5 to 6 Gbps to my router. While not the full 8 Gbps, I'm not saturating it anyway, so I haven't dug into my UniFi settings to determine what's causing the difference.
A supplementary test showing speeds directly to my server:
6. Was It Worth It?
Here's the honest part.
Do I need 10 Gbps between my gaming PC and my server? Probably not. The number of times a day I move enough data to saturate even 1 Gbps is small, and for most of what I do, this changes nothing I'll consciously notice.
But that was never really the point. I finally have an internet connection that delivers what I pay for, and a network that won't be the bottleneck for years to come. After spending so long fighting a "1 Gbps" connection that never once hit 1 Gbps, there's something deeply satisfying about watching a transfer actually move as fast as the hardware allows.
Instead of constantly monitoring speeds and pausing services so our work-from-home connections aren't impacted, I can let my server idle at 400 Mbps 24/7, not to mention all the other devices in my home.
Was it worth it? For the peace of mind alone: yes.
...or, comment on this post on Bubbles!
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.