Recent content on rakyll.org
I said that it’s likely that AI layers commoditize like everything else has. The harness, the planning loop, tools use patterns, the memory files, the protocols, context management, etc. I have watched ideas in this layer go from novel to table stakes in a few months. Everyone follows the same patterns. And the ones that actually work get absorbed into the model in the next training run, at…
I’m relatively new to ML/AL. I spent most of my career in commodity layers claiming to do boring things, and I’m still learning this field’s history, its culture, and its assumptions. So take this as an outsider’s observation. There is one assumption I keep running into that contradicts what I have seen in the past: the belief that AI is a zero-sum game. It impacts…
My thoughts on coding agents are evolving but they significantly shifted in December 2025. The models are being trained to produce trajectories that are effective for people who already know where they want to go. The value is the model reducing the distance between a clear goal and its execution. Once you see it this way, the interesting question is “how cheap is it now to get from a…
Go 1.18 is going to be released with generics support. Adding generics to Go was a multi-year effort and was a difficult one. Go type system is not a traditional type system and it was not possible just to bring an existing generics implementation from other language and be done. The current proposal was accepted after years of user research, experiments and discussions. The proposal got iterated…
Shard coordination has been one of the bigger challenges to design sharded systems especially for engineers with little experience in the subject. Companies like Facebook have been using general purpose shard coordinators, e.g. Shard Manager, and suggesting that general purpose sharding policies have been widely successful. A general purpose sharding coordinator is not a solution to all advanced…
Spanner is a distributed database Google initiated a while ago to build a highly available and highly consistent database for its own workloads. Spanner was initially built to be a key/value and was in a completely different shape than it is today and it had different goals. Since the beginning, it had transactional capabilities, external consistency and was able to failover transparently. Over…
Update: The proposal draft has been revisited to use brackets instead of parenthesis. This article will be updated with the new syntax soon. Ian Lance Taylor and Robert Griesemer have been working on a generics proposal for Go for a while. Unlike other proposals, a highly significant language change as such generics will require experimentation and comprehensive feedback before it can be finalized…
Spanner is a relational database with 99.999% availability which is roughly 5 mins a year. Spanner is a distributed system and can span multiple machines, multiple datacenters (and even geographical regions when configured). It splits the records automatically among its replicas and provides automatic failover. Unlike traditional failover models, Spanner doesn’t failover to a secondary cluster but…
Go’s defer keyword allows us to schedule a function to run before a function returns. Multiple functions can be deferred from a function. defer is often used to cleanup resources, finish function-scoped tasks, and similar. Deferring functions are great for maintability. By deferring, for example, we reduce the risk of forgetting to close the file in the rest of the program: func main() { f,…
Non-uniform memory access (NUMA) is an approach to optimize memory access time in multi-processor architectures. In NUMA architectures, processors can access to the memory chips near them instead of going to the physically distant ones. In the distant past CPUs generally ran slower than the memory. Today, CPUs are quite faster than the memory they use. This became a problem because processors…
It is real struggle to work with a new language, especially if the type doesn’t resemble what you have previously seen. I have been there with Go and lost my interest in the language when it first came out due to the reason I was pretending it is something I already knew. Go is considered as an object-oriented language even though it lacks type hierarchy. It has an unconventional type…
pprof now is coming with a Web UI. In order to try it out, go get the pprof tool: $ go get github.com/google/pprof The tool launches a web UI if -http flag is provided. For example, in order to launch the UI with an existing profile data, run the following command: $ pprof -http=:8080 profile.out You can focus, ignore, hide, and show with regexp. As well as clicking on the boxes and using the…
Note: This article contains non-finalized ideas; we may end up not implementing any of this but ideally we should do work towards the direction explained here. Go is the language to write servers, Go is the language to write microservices. Yet, we haven’t done much in the past for latency analysis and observability/diagnostics of request/RPC performance. GopherCon 2017 was an opportunity for…
Go scheduler’s job is to distribute runnable goroutines over multiple worker OS threads that runs on one or more processors. In multi-threaded computation, two paradigms have emerged in scheduling: work sharing and work stealing. Work-sharing: When a processor generates new threads, it attempts to migrate some of them to the other processors with the hopes of them being utilized by the…
Go 1.9 is introducing profiler labels, a way to add arbitrary key-values to the samples collected by the CPU profiler. CPU profilers collect and output hot spots where the CPU spent most time in when executing. A typical CPU profiler output is primarily reports the location of these spots as function name, source file/line, etc. By looking at the data, you can also examine which parts of the code…
Go provides several pprof profiles out of thet box to gather profiling data from Go programs. The builtin profiles provided by the runtime/pprof package: profile: CPU profile determines where a program spends its time while actively consuming CPU cycles (as opposed while sleeping or waiting for I/O). heap: Heap profile reports the currently live allocations; used to monitor current memory usage or…
Debugging is highly useful to examine the execution flow and to understand the current state of a program. A core file is a file that contains the memory dump of a running process and its process status. It is primarily used for post-mortem debugging of a program, as well as to understand a program’s state while it is still running. These two cases make debugging of core dumps a good…
In monolithic systems, it is relatively easy to collect diagnostic data from the building blocks of a program. All modules live within one process and share common resources to report logs and errors. Once you are distributing your system into microservices, it becomes harder to follow a call starting from the user’s entry point until a response is served. To address this problem, Google…
Go doesn’t specifically enforce you how you choose your test names. Tests are a significant contributors for the maintainability of your code. Tests not just providing correctness checking but also are useful in self documenting your code and its usage. On top of that, tests are the single best source to read about responsibilities of a type, function, etc. This is where naming tests better…
Go is about naming and organization as much as everything else in the language. Well-organized Go code is easy to discover, use and read. Well-organized code is as critical as well designed APIs. The location, name, and the structure of your packages are the first elements your users see and interact with. This document’s goal is to guide you with common good practices not to set rules. You…
Go 1.8 is going to to launched in February 2017. There is a sizable list of new features and improvements on the release notes. While these notes is the best summary to see what has happened in the last 6 months, I will try to give you some stats to give you a sense of the size of the work. I have examined all the changes merged into the tree during the Go 1.
Go 1.8 introduces a new profile, the contended mutex profile, that allows you to capture a fraction of the stack traces of goroutines with contended mutexes. You need to set the sampling fraction by calling runtime.SetMutexProfileFraction to a value above zero to enable collection. Consider the following program: import _ "net/http/pprof" var mu sync.Mutex var items = make(map[int]struct{})…
Go 1.8 will set a default GOPATH if the GOPATH env variable is not set. The requirement of setting a GOPATH has been a major issue for Go users who installed the Go tools for the first time and got the “you have to set a GOPATH” error in their initial experience with the tools. Explaining the GOPATH is and instructing how to set this env variable were both distracting new users away…
Go 1.8 is going to feature support for HTTP/2 server push. HTTP/2 has many features designed to make the Web faster. One of those features is the server push, the ability to send resources before the client asks for it. This feature enables websites to push assets like JavaScript and CSS files before waiting for the web page to be loaded and asking for those resources. net/http package will…
In Go, for a long time, we didn’t have a convention to label the deprecated APIs. In the past years, there is new convention emerged to add deprecation notices to the docs. Today, standard library uses this specific format. As an example, Go 1.8 deprecates sql/driver.Execer and adds a deprecation notice to its godoc. // Execer is an optional interface that may be implemented by a Conn. // //…
The context package makes it possible to manage a chain of calls within the same call path by signaling context’s Done channel. In this article, we will examine how to use the context package to avoid leaking goroutines. Assume, you have a function that starts a goroutine internally. Once this function is called, the caller may not be able to terminate the goroutine started by the function.…
Last week, I was at dotGo. I gave a very short lightning talk about inspection of code generation with the tools already available in the toolchain. This post goes through the talk for those who didn’t have the privilege to be at the conference. Slides are also available at go-talks. Throughout this article, we will use the following program: package main import "fmt" func main() { sum := 1…
Disclaimer: I forked my opinions on this one from a barely readable Twitter thread and wanted to write it down how I feel about keeping the language internals away from the users, especially from the newcomers. This is not a skill-level concern, it is a core goal of Go to provide a high-level programming language that saves users from excessive mental overhead. Note that these are personal…
New to the Go tools? Or do you want to expand your knowledge? This article is about the flags for the Go tools everyone should know. Disclaimer: This article might be slightly biased. This is a collection of flags I personally use and flags people around me having trouble finding references for. If you have more ideas, ping me on Twitter. $ go build -x -x lists all the commands go build invokes.
Go programming language provides many unique good features to write and maintain examples for your packages backed by the testing tools. As an addition to the test coverage and test coverage report, go test also can provide coverage for testable examples. Use the following commands in your package to use the -run flag to only the match the example tests and view the results in your browser. $ go…
With Go 1.7, testing package supports sub-tests that allows you to run multiple smaller tests from a test case. Each sub test is reported independently in the go test output. More information about these recent additions can be found at Marcel van Lohuizen’s recent talk from GolangUK 2016. These additions to Go 1.7 enabled reporting and other testing.T functionality for subtests. One of the…
Apple has a suite of instrumentation and tracing tools for performance analysis available as a part of their Xcode tooling set. In this article, we will use Instruments to record and analyze the CPU profile of a Go program. Instruments also provide a large set of macOS-specific tracing and profiling if you have performance issues specifically on darwin. Some of these specific profiles are: System…
Disclaimer: This article is not about a core Go package or tool but gRPC. gRPC provides support for implementing streaming endpoints as well as streaming support in their clients. Bidirectional streaming is useful if you want both server and client to be able to communicate to the other side independently in a full duplex fashion. In this article, I will dig into how to use the streaming gRPC Go…
About the author I work at Google working on our AI infrastructure. I’ve spent many years at Google where I worked on Google Drive, the API infrastructure, Go Programming Language, and our monitoring stack. I live in San Francisco, CA and I’m a polyglot. Contact Email: jbd@google.com, rakyll@rakyll.org Github: rakyll Twitter: rakyll
If you are willing to make large scale refactoring in your Go programs, automating the refactoring tasks is more desirable than manual editing. eg is a program that allows you to perform transformations based on template Go files. To install the tool, run the following: $ go get golang.org/x/tools/cmd/eg eg requires a template file to look for which transformation it should apply to your source…
Note: Swift bindings are highly experimental and subject to change. This work must currently be classified as preliminary work and we will be improving APIs in the long term. As a part of the Go Mobile, we have announced tools and packages that make language bindings from Java to Go and Objective-C to Go available. A relatively new and less documented aspect of the bindings is the availability of…
Note: This article extends Dave Cheney’s Go 1.5 cross compilers post. Cross compilers got easier with Go 1.5. You don’t have to bootstrap the standard library and toolchain as you used to do prior to 1.5. If cgo is not required The go tool won’t require any bootstrapping if cgo is not required. That allows you to target the following program to any GOOS/GOARCH without requiring you to do any…
If there is one standout language feature in Go, it is interfaces. Go’s internals combine useful ideas from various type systems, inevitably piquing curiosity. I recently surveyed globally available open source Go code bases for interface declarations, and the results indicate that Go developers often pollute their codebases with interfaces no one needs or will ever use. Don’t export any…