Recent content on Phil Pearl's Blog
We have an annual hackathon at Ravelin in the lead up to Christmas each year. This year I chose to investigate “Swiss tables”, the new hash table idea behind recent improvements to Go’s maps. 
 Why am I interested in this? I’ve written a number of specialist hash tables, and I wondered if these approaches would lead to any performance improvements. In particular, I’m responsible for a…
Recently at work I ran into an interesting performance problem.
We have a data processing job that is basically a series of count-distinct operations.
It uses a map to help work out the distinct sets of values. 
 In the past we’ve done some performance work on this, and we’ve arranged things
so that we re-use the map between operations. 
 Re-using the map was great!…
I seem to be writing about one blog post a year at the moment.
I always tell myself I want to write more, but it doesn’t happen.
This, however, is the first post in a series.
But I only have two ideas for posts, and one seems like a bad idea, so it could be a short series! 
 Every now and again I visit the Go slack, in particular the #performance channel
and the #darkarts…
Me: What’s the audience for this post? 
 Also Me: People who write code in Go and care what a function variable actually is. 
 Me: … 
 Also Me: I mean right down to the bits and bytes 
 Me: … 
 Also Me: … 
 What is it that got me interested in writing this blog post, given I think it’s likely to have an incredibly small audience?
Well, I…
Acute impostor syndrome. When something you’ve built, something you’re proud of, built against all advice, full of unsafe, goes deeply, horribly wrong. Unspeakably wrong. You’ve built your company’s data pipeline around this code. If it is wrong then everything might be wrong. How could it go so wrong? 
 Your belief in yourself crumbles away. The ghost of Rob Pike…
Here at Ravelin we have a lot of data with a lot of timestamps.
Most of the timestamps are stored as strings in BigQuery, and most of our Go structs represent time with the Go time.Time type. 
 I say these facts above with regret.
We really do have a lot of data. And we really do have a lot of timestamps.
For some time I’ve been circling around a conclusion, and as more time…
When I write text I work hard to avoid being too terse.
I’ve learnt the hard way that for me it’s very difficult to write too much.
My natural level is very terse. So what feels like too much to me is often not
nearly enough. 
 So I try to write more. I write not only about what I’m doing, but why I’m doing
it. I try to explain not only what my changes…
Don’t use reflection. Unless you really have to. But when you’re not using
reflection, don’t think that it is because reflection is fundamentally slow. It
doesn’t have to be slow. 
 Reflection allows you to obtain information about Go types at runtime. We’ll
look at how you can use it to populate structs if you were ever foolish enough to
try to…
Everyone loves io.Reader . It’s often touted as people’s favourite thing in Go. But it’s not the best abstraction when you get down to the small. When you’re doing lots of small reads to parse a protocol the overheads are too high, even if you’re using a buffered reader. I think the best abstraction at this point may be []byte , which is essentially no abstraction at…
Go can have nil slices and empty slices, and they’re different. What’s up with that? 
 Regular readers of my blog (a select group if ever there was one) will know by now that a slice is syntactic sugar for a struct with a Data pointer, a Length and a Capacity. There’s a definition of this struct in reflect . It looks like this. 
 type SliceHeader struct {
 Data…
Both of my regular readers may be wondering what happened after my last blog about potentially forking encoding/json. Did I fork it? No I didn’t. 
 This is partly because I discovered https://github.com/json-iterator/go , which looks like it is the JSON library I was looking for. And partly because I spent my enthusiasm writing plenc - a serialisation library based around the protobuf…
It feels trite to write about the famous quote about premature optimisation being the root of all evil (I double-checked the definition of “trite” before I wrote that sentence!). But I do have a strongly-held opinion about it. What I like to call “normal” levels of optimisation are not only not evil: they’re entirely necessary in many circumstances. 
 I keep…
So in my mind the conversation went a little like this. 
 Phil : I’d like to make encoding/json better. I’d like to save allocations when marshaling by adding a new interface for custom marshalers. 
 World : Given that this can be explored in 3rd-party packages easily, this seems like a likely decline. Leaving open for a week for final comments. 
 Phil : Oh, interesting,…
In my last post I whinged endlessly about encoding/json and in particular about the poor performance of anything with a custom marshaler . I decided to try to do something about it and I raised a proposal for a new marshaler interface. 
 As I half expected I wasn’t immediately welcomed as the new Go Messiah and my proposal has been effectively rejected (stop press - actually rejected).…
This is a story about encoding/json in the Go standard library. I’m not going to say this is bad Go. That would be heresy. But there is an aspect of marshalling that could be improved. Because it is in the standard library it isn’t bad Go, but if you followed the pattern in your own code then that would be a mistake. Outside of the standard library it would lose its magical aura, and…
This is the 5th in a series of posts about Bad Go - a clickbaity title for posts about Go code that I’ve found frustrating because it could just be that little bit better. Better in my mind means more performant with less impact on GC, without being more complex or harder to read. 
 In this post I’ll rant about a problem I’ve seen in a number of blog posts. I’ve seen it…
This is the 4th in a series of posts about Bad Go - a clickbaity title for posts about Go code that I’ve found frustrating because it could just be that little bit better. Better in my mind means more performant with less impact on GC, without being more complex or harder to read. 
 In this post we’ll look at a very common issue - not setting the size of a slice when you know how…
This is the 3rd in a series of posts about Bad Go - a clickbaity title for posts about Go that I’ve found frustrating because it could just be that little bit better. Better in my mind is often more performant with less impact on GC without being more complex or harder to read. 
 The first two posts are about slices of pointers and pointer returns from functions 
 This one is about…
As an old C programmer I struggle with this one: it feels completely normal for functions to return pointers to structs. But I’ve a feeling this is bad Go, and that we’re normally better off returning struct values. I’m going to see if I can prove that returning struct values is just plain better, and that returning pointers is bad Go. 
 I’m going to define a struct…
This is the first of what may be a series of blog posts on uses of Go that I’ve found frustrating. They’re mostly minor things that could just be better without being more complicated. I’m going to try to not only explain why they are bad but also demonstrate it. 
 First up is slices of pointers. Things like []*MyStruct . Unless you need to express that certain indices in the…
At Ravelin we’ve migrated to Kubernetes (on GKE). This has been very successful. We’ve got pod disruption budgets coming out of our ears, our statefulsets are very stately, and rolling node replacements run without a hitch. 
 The last piece of the puzzle is to move our API layer from the old VMs into our kubernetes cluster. For this we need to set up an Ingress so the API can be…
There’s a cost associated with using interfaces. What is that cost? Let’s try and work out some of it. 
 Let’s start with the basic overhead of calling a method via an interface. We’ll define a very simple interface with a single method and a very simple implementation. We’ll also mark the method so it isn’t inlined by the compiler. We do this so that the…
Did everyone else already know this? Why didn’t you tell me? I got very confused the other day with some apparently simple JSON encoding. Here’s a simplified version, showing marshalling a struct with an embedded struct inside it. 
 package main
 
 import (
 	 'encoding/json' 
 	 'fmt' 
 )
 
 type Inner struct {
 	InnerField string…
Regular readers of my blog will all be aware that in Go a string is in fact a struct with a pointer to an area of memory containing the byte content of the string and an integer Len that tells you how many bytes make up the string. 
 Ever wondered why? Probably not, as it seems “obvious” that you need both the length of the string and the bytes of the string to make a string. But it wasn’t…
The latest issue is a problem with the bulk feature extraction process we use at Ravelin (yes, we’re hiring ! So if you like Go and you’re anywhere near London drop us a line). For our larger clients we’ve found this process just uses more and more memory, so we keep having to run it on more and more expensive boxes. I presumed it was a memory leak of some kind, so I decided to investigate.
Two years ago I wrote a post about building up strings with bytes.Buffer . I wondered what’s changed over the past two years? 
 Here are the benchmarks taken from the original post. 
 BenchmarkCopyKey-8 114 ns/op 31 B/op 1 allocs/op
BenchmarkSimpleKey-8 141 ns/op 31 B/op 1 allocs/op
BenchmarkSimpleMultilineKey-8 256 ns/op 63 B/op 4 allocs/op
BenchmarkSprintfKey-8 392 ns/op 79…
Many of us know that you can run Go binaries in “scratch” containers . Your container doesn’t need to be based on Alpine or Ubuntu. It can be based on nothing and contain just the binary you built from Go source. This is largely because Go code can be statically linked, and so requires no installed libraries. 
 But what about VMs? Normally you start from Ubuntu, or Alpine or whatever and then…
At Ravelin we build Go binaries, package them into scratch Docker containers and upload them to Google Cloud Registry so we can use them in a GKE cluster. We develop on Mac laptops, so we use Docker for Mac for this. But building and pushing the container images is very slow, particularly if we want to build 44 at once. Given we just want to package up single binaries the process seemed a bit…
If you actually want to use the memory on your computer with Go — really use it, with gigabytes of it allocated — then you may pay a big penalty for the Go garbage collector (GC). But there are things you can do about it. 
 The Go GC checks what parts of the memory you have allocated are still in use. It does this by looking at all the memory for references to other pieces of memory. If you’ve…
Some time ago I wrote about how I went to great lengths moving allocations off the Go heap into memory allocated directly from the OS in order to reduce the overhead of GC. This was in a new Graph Database I was developing at Ravelin to catch bad people more efficiently. At the time I wasn’t entirely certain that the GC CPU overhead was a terrible thing, but it was untidy and I didn’t want to risk…