Summary: I joined Meta 2.5 years ago to work on build systems. I’m enjoying it. I joined Meta over two years ago when an opportunity arose to work on build systems full time. I started the Shake build system at Standard Chartered over 10 years ago, and then wrote an open source version a few years later. Since then, I’ve always been dabbling in build systems, at both moderate and small scale. I…
Summary: Shake won't scale to millions of files, this post says what would be required to make it do so. While Shake has compiled projects with hundreds of thousands of files, it's never scaled out to millions of files, and it would be unlikely to work well at that size. The most popular build systems that operate at that scale are Buck (from Facebook) and Bazel (from Google). In this post I go…
Summary: Forward build systems might work better for small projects. Yesterday's post talked about how Shake is a good medium sized build system - but what about smaller projects? Is the Shake model right for them? Shake can be considered a backwards build system . Each rule says how to produce a file, given some input files (which are dependencies) and an action. Almost all build systems (e.g.…
Summary: As a medium-sized build system, Shake has some good bits and some bad bits. I first developed the Shake build system at Standard Chartered in 2008, rewriting an open source version in my spare time in 2011. I wrote a paper on Shake for ICFP 2012 and then clarified some of the details in a JFP 2020 paper . Looking back, over a decade later, this post discusses what went well and what could…
Summary: Use OBS, Camo and Audacity . I recently needed to record a presentation which had slides and my face combined, using a Mac. Based on suggestions from friends and searching the web, I came up with a recipe that worked reasonably well. I'm writing this down to both share that recipe, and so I can reuse the recipe next time. Slide design: I used a slide template which had a vertical…
Summary: Monadic and early cut-off? Use a sequence of sets. In the Build Systems a la Carte paper we talk about the expressive power of various types of build systems. We deliberately simplify away parallelism and implementation concerns, but those details matter. In this post I'm going to discuss some of those details, specifically the representation of dependencies. Applicative build systems In…
Summary: Some languages ban recursion to ensure programs "terminate". That's technically true, but usually irrelevant. In my career there have been three instances where I've worked on a programming language that went through the evolution: Ban recursion and unbounded loops. Proclaim the language is "Turing incomplete" and that all programs terminate. Declare that Turing incomplete programs are…
Summary: I recommend people use the Haskell Language Server IDE. Just over a year ago, I recommended people looking for a Haskell IDE experience to give Ghcide a try . A few months later the Haskell IDE Engine and Ghcide teams agreed to work together on Haskell Language Server - using Ghcide as a library as the core, with the plugins/installer experience from the Haskell IDE Engine (by that stage…
Interviewing usually involves some level of subjectivity. I once struggled to decide about a candidate, and after some period of reflection, the only cause I can see is that I was biased against the candidate. That wasn't a happy realisation, but even so, it's one I think worth sharing. Over my years, I've interviewed hundreds of candidates for software engineering jobs (I reckon somewhere in the…
Summary: Hoogle searches packages on Stackage. Haskell (as of 27 July 2020) has 14,490 packages in the Hackage package repository . Hoogle (the Haskell API search engine) searches 2,463 packages. This post explains which packages are searched, why some packages are excluded, and thus, how you can ensure your package is searched. The first filter is that Hoogle only searches packages on Stackage .…
Summary: You can divide extensions into yes, no and maybe, and then use HLint to enforce that. I've worked in multiple moderately sized multinational teams of Haskell programmers. One debate that almost always comes up is which extensions to enable. It's important to have some consistency, so that everyone is using similar dialects of Haskell and can share/review code easily. The way I've solved…
Summary: In previous companies I had a lot of freedom to design an interview. This article describes what I came up with. Over the years, I've interviewed hundreds of candidates for software engineering jobs (at least 500, probably quite a bit more). I've interviewed for many companies, for teams I was setting up, for teams I was managing, for teams I worked in, and for different teams at the same…
Summary: Run cmdargs-browser hlint and you can fill out arguments easily. The Haskell command line parsing library cmdargs contains a data type that represents a command line . I always thought it would be a neat trick to transform that into a web page, to make it easier to explore command line options interactively - similar to how the custom-written wget::gui wraps wget . I wrote a demo to do…
Summary: I came up with a clever trick to encode lifetimes of allocated values in Rust. It doesn't work. Let's imagine we are using Rust to implement some kind of container that can allocate values, and a special value can be associated with the container. It's a bug if the allocated value gets freed while it is the special value of a container. We might hope to use lifetimes to encode that…
Summary: HLint has a match engine which powers most of the rules. The Haskell linter HLint has two forms of lint - some are built in written in Haskell code over the GHC AST (e.g. unused extension detection ), but 700+ hints are written using a matching engine. As an example, we can replace map f (map g xs) with map (f . g) xs . Doing so might be more efficient, but importantly for HLint, it's…
Summary: Hoogle 5 has three interesting parts, a pipeline, database and search algorithm. The Haskell search engine Hoogle has gone through five major designs, the first four of which are described in these slides from TFP 2011 . Hoogle version 5 was designed to be a complete rewrite which simplified the design and allowed it to scale to all of Hackage . All versions of Hoogle have had some…
Summary: IO evaluation caught me off guard when trying to write some benchmarks. I once needed to know a quick back-of-the-envelope timing of a pure operation, so hacked something up quickly rather than going via criterion . The code I wrote was: main = do (t, _) <- duration $ replicateM_ 100 $ action myInput print $ t / 100 {-# NOINLINE action #-} action x = do evaluate $ myOperation x return ()…
Summary: HLint --cross was accidentally quadratic in the number of files. One of my favourite blogs is Accidentally Quadratic , so when the Haskell linter HLint suffered such a fate, I felt duty bound to write it up. Most HLint hints work module-at-a-time (or smaller scopes), but there is one hint that can process multiple modules simultaneously - the duplication hint. If you write a sufficiently…
Summary: A performance investigation uncovered a memory leak in unordered-containers and performance issues with Ghcide. Over the bank holiday weekend, I decided to devote some time to a possible Shake build system performance issue in Ghcide Haskell IDE . As I started investigating (and mostly failed) I discovered a space leak which I eventually figured out, solved, and then (as a happy little…
Summary: The new version of Shake has some tweaks to how stdin works with cmd . I've just released Shake 0.19 , see the full change log . Most of the interesting changes in this release are around the cmd / command functions, which let you easily run command lines. As an example, Shake has always allowed: cmd "gcc -c" [source] "-o" [output] This snippet compiles a source file using gcc . The cmd…
Summary: Four improvements to Haskell I'm not going to raise as GHC proposals. Writing a GHC proposal is a lot of hard work. It requires you to fully flesh out your ideas, and then defend them robustly. That process can take many months . Here are four short proposals that I won't be raising, but think would be of benefit (if you raise a proposal for one of them, I'll buy you a beer next time we…
Summary: It is useful to trace files accessed by a command. Shake and FSATrace provide some tools to do that. When writing a build system, it's useful to see which files a command accesses. In the Shake build system , we use that information for linting , an auto-deps feature and a forward build mode . What we'd like is a primitive which when applied to a command execution: Reports which files are…
Summary: HLint 3.0 uses the GHC parser. In June 2019 I posted about our intention to move HLint to the GHC parser. Since then a small group of us have been hard at work making the conversion -- first by parsing with both GHC and haskell-src-exts, and finally, with the newly released HLint 3.0 , parsing only with GHC . As of now, if your code can be parsed with GHC, it can probably be parsed with…
Summary: Interpretation by closure is a lot faster than I expected. Let's imagine we have an imperative language (expressions, assignments, loops etc.) and we want to write an interpreter for it. What styles of interpreter are there? And how fast do they perform? I was curious, so I wrote a demo with some benchmarks. The full code, in Rust, is available here . First, let's get a taste of the the…
Summary: Sometimes <- pure makes a lot of sense, avoiding some common bugs. In Haskell, in a monadic do block, you can use either <- to bind monadic values, or let to bind pure values. You can also use pure or return to wrap a value with the monad, meaning the following are mostly equivalent: let x = myExpression x <- pure myExpression The one place they aren't fully equivalent is when…