High-performance cloud computing, bioinformatics, and other fun stuff.
Let's say you need to sum a large array of floating-point numbers, maybe because you're calculating the arithmetic mean or variance. This post shows how to do so accurately and quickly. Here's what the accuracy-naïve, performance-naïve implementation looks like ( Godbolt ): Improving Accuracy When summing floating-point numbers, we need to consider the following: Overflow Adding two numbers that…
We recently had a need to create new DNS records that are visible on the authoritative server as soon as possible. This is a short and simple post showing measurements of how long it takes for new records to propagate after making the API call to add those records to Google Cloud Platform's or AWS's Route 53 DNS servers. GCP has guidance for monitoring changes to a DNS zone. They don't say what it…
I’m building a new cloud product that quickly processes large amounts of scientific data. Our largest customer dataset so far is about 3,000 tables, each 40 to 80 MB and totaling 150 GB, which we aim to process in 10 seconds or less. Each table can be processed independently, so we parallelize heavily to reach this goal — our deployment uses 1000 vCPUs or more as needed. The tricky part is rapidly…