October Updates and Metrics

Oh boy, how things have changed. This has been one of the most aggressive few months I've had for upgrades and changes.

Updates:

• Stor1.lain.local is now fully operational, with a usable storage pool of 115TB.
• Stor2.lain.local is now fully operational, with a usable storage pool of 90TB.

• The Pomf edge caches are now 8x the size, allowing for much better cache hit ratios. This means less loading times for you!
• Francisco enabled 10Gbit networking for all Pomf nodes, meaning a theoretical 40Gbit capacity for outbound traffic. I don't expect to actually get 10Gbit, but still. Huge, huge upgrade. (quick update: CPUs start to scream at 2gbit, so 8gbit is our max.)

• The witness script has been upgraded yet again to fix some bugs. It has now gotten heavy use and is fully functional.
• I have standardized on Broadcom 57810 10Gb SFP+ NICs for all servers. These are actually supported by ESXi 7.0.
• Invidious is the newest service to join the fleet, at Invidious.lain.la
• The Generac automatic backup generator project is being scoped out as I type this! The estimate is... pricey.
• New environmental monitoring in the form of an APC Netbotz system will be installed as soon as I find a serial cable...
• Pomf will now allow files up to 1GB.

Metrics:

• Cluster stats:

• VM Count: 34 On-prem, 11 unique Hentai nodes, 7 endpoints, total of 52 VMs.
• Edge bandwidth - We're still running pretty hard here. 400TB extrapolated across all 4 edge nodes, plus Hentai and WAN usage. A single node is below:

• Internal bandwidth between edge and datacenter (Cache ratio - approx. 100x, a huge improvement over the last metrics article. Only 5TB!):

• Costs: Edge node costs have gone up a bit with the new caching, and power costs have significantly increased with the new storage arrays. We are now at $386.50 a month
• Pomf: 

  Filesystem          1G-blocks  Used Available Use% Mounted on
  /dev/mapper/volume1     9987G 4425G     5558G  45% /mnt/storage
  Total File Count is
  229468

• Uptime: I expect this won't improve past 3 9's for right now due to the scale of changes I am making. The next maintenance window is very soon as well.

That's all for now!