uTest is a small unit testing library I maintain that aims for simplicity and convenience. This blog post explores the Golden testing feature newly-added in uTest 0.9.0: why it is necessary, what it does, and how it works internally. This feature was inspired by the Jane Street blog post What if writing tests was a joyful experience .
The com.lihaoyi platform is an ecosystem of Scala tools and libraries that allow you to write Scala in a way that is easy , rather than powerful. This post discusses the history of the com.lihaoyi platform, how it came about and evolved over the past decade, and what role I think it may play in the future of the Scala language.
This blog post discusses the introduction of the ScalaSql query library. Why it is needed, what it brings to the table over the many existing database query libraries in the Scala ecosystem, and what makes it fit nicely into the com-lihaoyi philosophy of "Executable Scala Pseudocode that's Easy, Boring, and Fast"
Mill is a Scala build tool that offers an alternative to the venerable SBT toolchain. Mill aims for simplicity by reusing concepts you are already familiar with, borrowing ideas from Functional Programming and modern tools like Bazel. Feedback from users of Mill is often surprisingly positive, with people saying it is "intuitive" or feels "just right" . In this post, we'll explore why Mill is such…
Python has a reputation as "Executable Pseudocode": code that fits just as easily on a whiteboard during a discussion, as it does in a codebase deploying to production. Scala is a language that can be just as concise a pseudocode as Python, and arguably better at the "executable" part: faster, safer, and with better tooling. This blog post will explore how the Scala libraries from the com.lihaoyi…
Last year I wrote and self-published my first technical book Hands-on Scala Programming . It has gone on to be relatively successful, selling thousands copies, both digital and physical. This blog post will explore how the process of writing and publishing Hands-on Scala went: from its inception, to writing, editing, and finally publishing in multiple formats and mediums.
Today marks my 4th year anniversary at my current employer, Databricks. It's been a good run, and this is a great chance to reflect on the journey so far. This post is as much for me to organize my own thoughts as it is for anyone else to read. This is a bit of change from the normal programming content on this blog, but given that my job is a programming job, writing lots of Scala, I suppose it…
This is a short blog post to introduce the new com-lihaoyi Github organization: a place for all of my most production-ready projects will live to be easily maintained and discovered. We will discuss the status quo, what is changing, and what I hope to achieve with this change.
Scala, first appearing in 2004, is neither an old stalwart nor a new player in the programming language market. This post will discuss the unique combination of features that Scala provides and how it compares to other languages on the market, diving beneath the superficial experience to explore the fundamentals of the language. From this, you will learn why you might consider including Scala as a…
This blog post is a chapter 16 excerpt from the book Hands-on Scala Programming class SimpleUploadActor()(implicit cc: castor.Context) extends castor.SimpleActor[String]{ def run(msg: String) = { val res = requests.post("https://httpbin.org/post", data = msg) println("response " + res.statusCode) } } Snippet 16.1: a simple actor implemented in Scala using the Castor library Message-based…
A recent tweet by a friend of mine noted how the public interest in the Scala programming language seems to have plateaued or waned, which matches my feeling of the latest trends and zeitgeist. This blog post will go into why I think that has happened, where Scala stands now, and what the future holds for the Scala community.
The latest version of my open source libraries have standardized on two new interfaces - Writable and Readable - that allow efficient streaming data exchange between libraries. This blog post will explore the origin of these two interfaces, what purpose they serve, and how they can enable more efficient inter-operabilty between a wide range of different libraries and frameworks.
The Scala language, following its Java heritage, allows you to compare any two values for equality regardless of their respective types. This can be very error prone, since a refactor that changes the type of one of your values may silently result in an equality check that can never return true. This blog post explores how a "safer" equality check is just one of many features that go against the…
Optimizing compilers are a mainstay of modern software: allowing a programmer to write code in a language that makes sense to them, while transforming it into a form that makes sense for the underlying hardware to run efficiently. The optimizing compiler's job is to figure out what your input program does, and create an output program that it knows will do the same thing, just faster. This post…
Most modern systems are backed by relational databases. This tutorial will walk you through the basis of using a relational database from Scala, using the Quill query library. We will work through small self-contained examples of how to store and query data within a Postgres database, and finish with converting an interactive chat website to use a Postgres database for data storage.
Not every website exposes their data through a JSON API: in many cases the HTML page shown to users is all you get. This tutorial will walk you through using Scala to scrape useful information from human-readable HTML pages, unlocking the ability to programmatically extract data from online websites or services that were never designed for programmatic access via an API.
Web and API servers are the backbone of internet systems: they provide the basic interface for computers to interact over a network, especially at the boundary between different companies and organizations. This tutorial will teach you the basics of setting up a simple HTTP server in Scala to serve Web and API requests, and walk you through a complete example of building a simple real-time chat…
One strength of Scala is implementing programming languages. Even if your goal is not to implement an entirely new programming language, these techniques are still useful: for writing linters, program analyzers, query engines, and other such tools. This blog post will walk you through the process of implementing a simple programming language in Scala, introduce some of the fundamental concepts…
The Scala programming language comes with a Futures API. Futures make parallel programming much easier to handle than working with traditional techniques of threads, locks, and callbacks. This blog post dives into Scala's Futures: how to use them, how they work, and how they can give you much more flexibility to leverage parallelism in your code.
Build pipelines are a common pattern, where you have files and assets you want to process but want to do so efficiently and incrementally. Usually that means only re-processing files when they change, and otherwise re-using the already-processed assets as much as possible. This blog post will walk through how to use the Mill build tool to set up these build pipelines, using a real-world use case,…
JSON is one of the most common data interchange formats: a human-readable way of exchanging structured data that is ubiquitous throughout industry. This tutorial will walk you through how to work effectively with JSON data in Scala, walking through a few common workflows on a piece of real-world JSON data.
JSON HTTP APIs have become the standard for any organization exposing parts of their system publicly for external developers to work with. This tutorial will walk you through how to access JSON HTTP APIs in Scala, building up to a simple use case: migrating Github issues from one repository to another using Github's public API.
Most complex systems are made of multiple processes: often the tool you need is not easily usable as a library within your program, but can be easily started as a subprocess to accomplish the task you need it to do. This tutorial will walk through how to easily work with such subprocesses from the Scala programming language, to allow you to interact with the rich ecosystem of third-party tools and…
Working with files and the filesystem is one of the most common things you do when programming. This tutorial will walk through how to easily work with files in the Scala programming language, in a way that scales from interactive usage in the REPL, to your first Scala scripts, to usage in a production system or application.
Pretty-printing hierarchical data into a human-readable form is a common thing to do. While a computer doesn't care exactly how you format the same textual data, a human would want to view data that is nicely laid out and indented, yet compact enough to make full use of the width of the output window and avoid wasting horizontal screen space. This post presents an algorithm which achieves optimal…
The latest version 0.7 of the uPickle Scala serialization library lets you easily serialize your Scala values to the binary MessagePack format, in addition to the existing JSON serialization format. This gives you the option of compact, high-performance, binary serialization entirely for free, for any value you were previously JSON serializing. This blog post will explore the benefits of binary…
This blog post introduces Fastparse 2 , a new major version of the FastParse parser-combinator library for Scala. In exchange for some minor tweaks in the public API FastParse 2 gives you parsers that run 2-4x faster on real-world parsers than FastParse 1. This brings Fastparse - already one of the fastest Scala parsing libraries - close to the speed of hand-written parsers. This blog post will…
The Visitor Pattern is one of the most mis-understood of the classic design patterns. While it has a reputation as a slightly roundabout technique for doing simple processing on simple trees, it is actually an advanced tool for a specific use case: flexible, streaming, zero-overhead processing of complex data structures. This blog post will dive into what makes the Visitor Pattern special, and why…
You have an idea. Your boss is indifferent, your team-mates apprehensive, and that other team whose help you need are dubious. You are an individual-contributor with no direct-reports. You still think it's a good idea, but cannot make it happen alone. What next? Driving change within an technical organization is hard, especially as someone with no rank or authority, but is a skill that can be…
uJson is a new JSON library for the Scala programming language. It serves as the back-end for the uPickle serializaiton library, but can be used standalone to manipulate JSON in a way that is fast, flexible and intuitive, far more than the existing JSON libraries in the Scala library ecosystem. This post will go over what makes uJson an improvement over other JSON libraries that are available, and…
Mill is a new build tool for Scala: it compiles your Scala code, packages it, runs it, and caches things to avoid doing unnecessary work. Mill aims to be better than Scala's venerable old SBT build tool , learning from it's mistakes and building upon ideas from functional programming to come up with a build tool that is fast, flexible, and easy to understand and use. This post will explore what…
The term "build tool" is used to describe a host of different pieces of software, each with their own features, uses and complexity. It is sometimes hard to see the forest for the trees: in the midst of minifying Javascript, compiling C++, or generating protobuf sources, what is a build tool all about? In this post, I will argue that fundamentally a build-tool models the same thing as a pure…
SBT is the default build tool for the Scala programming community: you can build Scala using other tools, but the vast majority of the community uses SBT. Despite that, nobody seems to like SBT: people say it's confusing, complicated, and opaque. This post will deeply analyze what exactly it is about SBT that people don't like, so we can build a consensus around the problems and a foundation for…
Fastparse is a Scala library for parsing strings and bytes into structured data. This lets you easily write a parser for any arbitrary textual data formats (e.g. program source code, JSON, ...) and have the parsers run at an acceptable speed, with great error debuggability and error reporting. This post goes over the history and motivations of Fastparse, and what to expect with the project's…
uTest is a testing framework for the Scala programming language. uTest aims to be both simple and convenient to use, to allow you to focus on what's most important: your tests and your code. This post will explore what makes uTest interesting, and why you should consider using it to build the test suite in your next Scala project.
Any software engineer who has ever looked for a job has had the interview experience: being locked in a small room for an hour, asked to write code to solve some arbitrary programming task, while being grilled by one or two interviewers whether or why the code they've written is correct. You can find lots of material online about how to ace the interview as an interviewee , but very little has…
One oft-repeated fact is that in the Scala language, immutable Vectors are implemented using a 32-way trees. This makes many operations take O(log32(n)) time, and given a maximum size of 2^31 elements, it never takes more than 6 steps to perform a given operation. They call this "effectively constant" time. This fact has found its way into books , blog posts , StackOverflow answers , and even the…
Automated testing is a core part of writing reliable software; there's only so much you can test manually, and there's no way you can test things as thoroughly or as conscientiously as a machine. As someone who has spent an inordinate amount of time working on automated testing systems, for both work and open-source projects, this post covers how I think of them. Which distinctions are meaningful…
Ammonite: Scala Scripting is an open-source project that lets you use the Scala programming language for "scripting" purposes: as an interactive REPL, small scripts, or a systems shell. Scala has traditionally been a "heavy, powerful" language with "heavy, powerful" tools, and Ammonite aims to let you use it for small, simple tasks as well. While people have been using Ammonite continuously over…
Scala is my current favorite general-purpose programming language. However, it definitely has its share of flaws. While some are deep trade-offs in the design of the language, others are trivial, silly issues which cause frustration far beyond their level of sophistication: "warts". This post will explore some of what, in my opinion, are the warts of the Scala programming language, to hopefully…
Platforms like Github , Bitbucket , and Phabricator all provide ways of browsing and searching your project's source code online, as part of their larger suite of collaboration features. While their overall platform is rich and valuable, my experience with their online code-explorers has always been mediocre and underwhelming. This post contrasts the experience of exploring code online with that…
There are many descriptions floating around the internet, trying to explain functional programming in simple terms. Unfortunately, most discuss details only loosely related to functional programming, while others focus on topics that are completely irrelevant. So of course, I had to write my own! This post is my own understanding of what is the "core" of "functional programming", how it differs…
Apart from the old design patterns from the 1990s, the Scala programming language in 2016 has a whole new set of design patterns that apply to it. These are patterns that you see in Scala code written across different organizations, cultures and communities. This blog post will describe some of these design patterns around the use of Scala implicits: specifically around the use of implicit…
A Design Pattern is something you do over and over when building software, but isn't concrete enough to be made into a helper method, class or other abstraction. The "Gang of Four" Design Patterns book popularized the idea, and discussed in detail many of the design patterns which are common in C++, Java and similar "Object Oriented" languages. The Scala programming language in 2016 is different…
This post will dive into the runtime characteristics of the Scala collections library, from an empirical point of view. While a lot has been written about the Scala collections from an implementation point of view ( inheritance hierarchies , CanBuildFrom , etc... ) surprisingly little has been written about how these collections actually behave under use. Are List s faster than Vector s for what…
Parsing structured text into data structures has always been a pain. If you are like me, you may have wondered: why do all the parsing tools seem to be parser-generators instead of just configurable-parsers ? After all, when you look at, say, a 2D physics library like Chipmunk2D , you just get a bunch of classes and functions you can call. In contrast, parsing libraries like YACC or ANTLR often…
Three years ago, I downloaded the nascent Scala.js compiler and tried to use it on a toy project. Since then, it has matured greatly: the compiler itself is rock-solid. It has a huge ecosystem of libraries. It has a vibrant community, been adopted by some of the largest commercial users of the Scala language , and is playing a key role in shaping evolution of the language. By any measure, it is a…
The Scala programming language has traditionally been a tool used for building "serious business" systems: compilers, big data, distributed systems or large web applications. With the advent of Scala.js , people are starting to use it for front-end Web work, while the Ammonite-REPL has turned it into a pleasant interactive experience. This post will explore the new Scala Scripting functionality in…
Everyone is used to programs printing out output in a terminal that scrolls as new text appears, but that's not all your can do: your program can color your text, move the cursor up, down, left or right, or clear portions of the screen if you are going to re-print them later. This is what lets programs like Git implement its dynamic progress indicators, and Vim or Bash implement their editors that…
When programming in Scala, there are two main ways of avoiding repetition: you can define functions to represent commonly-used procedures or computations, and you can define data-types , e.g. using class es or case class es, to represent commonly-used bundles of data that you tend to pass around or use together. Lots of people have opinions about functions : they should be "pure", not too long,…