When making a git commit I’ve been looking for a way to record what AI model or coding agent harness I used to help me write the commit, ideally in a machine-readable way. For AI-drafted code that I’ve reviewed or self-drafted code with significant AI revisions , I add one of the following trailers to my git commit message: Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>…
Never write your own database or filesystem. It’s way harder than you expect. — some wise programmer 1 Databases and filesystems are hard to write in part because they have very strong requirements around never losing data (durability) even when unusual events happen such as unexpected process termination, hardware failure, out of disk space, or out of memory. I realized recently that…
I’ve noticed that the GitHub issue count on my software project Crystal always seems to go up, even though I’m the only one who ever files issues (for now) and even though I’m actually completing issues occasionally. I think I’ve figured out why: Often when I complete an issue, I only complete the first 80% of the issue which is most valuable to me and file smaller issues…
A Philosophy of Software Design is my favorite book I’ve read to date about designing large long-lived maintainable software programs. Here’s what I learned: Complexity Complexity is anything related to the structure of a software system that makes it hard to understand & modify the system. Symptoms of complexity: Change Amplification - A seemingly simple change requires code…
There are a few languages that offer a symbol type , notably JavaScript, Ruby, Lisp, and Erlang. A symbol is similar to a string but guarantees that only one copy of the symbol exists with the same value. In JavaScript you can create a symbol with the code: Symbol.for('hello') // Symbol(hello) Symbols can be compared with each other for equality very quickly because only the object references need…
When a program I’m debugging prints something unexpected like an error message, I can usually search for a fragment of the message in the program’s code to figure out why it was printed. However sometimes I’m not able to locate the error message in the code at all! In that case one big hammer I have for isolating the offending code is to alter print() , sys.stdout.write() , and…
Software engineering research has identified 1 some techniques and rules of thumb for making code reviews more effective at identifying defects: Limit code reviews to 1 hour or less . After 60-90 minutes into a code review the ability to find further defects drops off dramatically due to focus fatigue. If a code review in progress is dragging on past 1 hour, consider taking a break and continuing…
I was happy to be able to attend PyCon US 2024 this past year, a prominent conference for the Python programming language. I got to meet in person a number of folks I’ve interacted with online, from the Python Typing community and elsewhere. Favorite Talks I particularly liked the following talks, which I’ve organized by topic. Most video links below are to the Hublio platform, which…
I find the fire & heat around Redis changing licensing surprising. Maintaining Redis takes effort which cannot be free in a sustainable fashion. Consider this post which complains Redis isn’t “uphold[ing] the ideals of Free and Open Source Software”, as if it was created as a vehicle to evangelize the FSF and OSI missions. But it was not, it was created to be useful , as a data…
Some seafaring ships divide their body into multiple watertight compartments, so that if one compartment becomes flooded the rest of the ship will remain floodfree and intact. I got the idea of using a similar pattern of “bulkheads” in software to limit the damage caused by an unhandled exception . Normally an unhandled exception will cause its thread to print diagnostic information to…
I recently encountered a deadlock while running the automated tests of my website downloader, Crystal . The process for investigating and fixing this deadlock was interesting, so I thought I’d share it: What does a deadlock look like? One day after making some changes to my program, when I ran the program’s automated tests the program just completely froze, becoming unresponsive to all…
I recently extended my website downloader, Crystal , so that the projects it creates have a proper icon on macOS, Windows, and Linux. It was a lot more challenging than I expected! Crystal organizes a group of downloaded web pages into a project , which is a special folder containing a particular arrangement of files: $ tree xkcd.crystalproj xkcd.crystalproj ← project ├── database.sqlite ← web…
The official instructions for making a .deb Debian package 1 I found to be rather long-winded and fiddly, so I came up with a hopefully more-straightforward method of building a .deb package, using Docker. What’s in a binary .deb package? A binary .deb package is kind of like a .zip file whose contents are unpacked to / during the installation process. For example, a simple .deb package that…
Sometimes when I have a problem with a bash script I want the ability to run each line of the script one by one to see what is wrong. There is a debugger called bashdb that can be used in this way. It behaves similarly to gdb (used for debugging C and C++ programs) and pdb (used for debugging Python programs). How to install bashdb First you need to determine what version of bash you are running,…
Crystal is a website downloader program that is intended to save websites for long-term archival, even after the original site has fallen off the internet. Today’s release brings the ability to download sites requiring login, better-support for infinite-scrolling sites, a read-only mode , and a CLI shell . See the release notes for more information. Download Crystal for Windows 7, 8, and 10…
I’m proud to release django-mysql-compressed-fields : a new library that provides a compressed version of Django’s TextField for use with a MySQL 1 database! This is the first library from TechSmart to be open-sourced. 🎉 In particular you can replace a TextField or CharField like: from django.db import models class ProjectTextFile(models.Model): content = models.TextField(blank=True)…
In March 2021 I was interviewed by Vladislav from softdroid.net. Since it gives a bit of my personal background, it may be of interest to others: I’m David Foster, software engineer, writer, and educator. I’m the CTO of TechSmart, where we seek to bring world-class computer science education to the next generation of K-12 students & teachers. What is your programming background?…
Crystal is a website downloader program that is intended to save websites for long-term archival, even after the original site has fallen off the internet. Today I’m pleased to announce Crystal’s first beta release, bringing support for downloading more complex static sites than ever before, and generally being stable enough for a full public release. Download Crystal for Windows 7, 8,…
The general availability of easy-to-use centralized continuous integration 1 (CI) solutions in recent years - from GitHub Actions 2 and Travis as hosted solutions, and from Jenkins 3 and Hudson as on-premises solutions - has been wonderful for allowing software to be tested continuously throughout development, catching errors early before changes are merged to shared development and mainline…
Any time a backend Django or Rails function calculates something complex from the database to send to the frontend as part of a view, there is a chance the database will be modified concurrently before the view is displayed to the visitor, causing the site visitor to see outdated information. In many cases displaying stale information is fine. After all refreshing the page will bring the latest…
It’s easy to build a simple chat server in Channels with real-time updates 1 but it’s a bit more complicated to design a system for a more realistic (and complex) data model that has real-time updates. Here, I will show a publish-subscribe (or “pub-sub” ) pattern using WebSockets and Channels that can be used by your frontend to watch elements of your backend data model for updates in…
1 server or 2 servers? 1-server approach Bundling strategies Concatenated Bundling Import-Traced Bundling Transpiled Bundling Render baseline HTML with Django Enhance baseline HTML with Vue 2-server approach Conclusion Vue and Django are both fantastic for building modern web apps - bringing declarative functional reactive programming to the frontend, and an integrated web app platform, ecosystem,…
If you’ve got a moderate-sized Django web application then you’re probably already writing automated tests to make sure none of its pages break unexpectedly when you’re making changes to them. That is, you’re testing page functionality . However another way that pages can break is that they take too long to display, or otherwise don’t have enough performance . The…
Google is working on a new technology called Privacy Sandbox to replace the need for advertisers to track individuals with third party cookies. This is interesting to me for a couple of reasons: Google is an advertising company, so they definitely want the ability to continue being able to classify users into cohorts based on their behavior in order to deliver targeted ads to them. But they are…
Automated tests are usually used for testing functional requirements of your product code. But they can also be used to enforce other policies and coding practices as well. If writing a web application it’s likely you already have a rule like “all automated tests must pass before any new version of the web application can be deployed to customers on the production environment”.…
You may have heard that TypeScript has been taking the web development space by storm in the last few years , bringing to it static types. I believe the same thing is starting to happen in the world of Python, where type checkers like mypy , Pyre , and Pyright are increasingly used, at least where Python is used by companies to write large systems. For the last several releases of Python, there…
I know a number of folks use The Great Suspender to automatically suspend inactive browser tabs in Chrome. Apparently recent versions of this extension have been taken over by a shady anonymous entity and is now flagged by Microsoft as malware . Notably the most recent version of the extension (v7.1.8) has added integrated analytics that can track all of your browsing activity across all sites.…
For difficult problems 1 I have used naps at home and work to recharge quickly or to temporarily alter my information processing mode 2 . When to power nap: When feeling mentally fuzzy. When slightly tired but still there is more that must be done before sleep. How to power nap: Find a comfortable position . Ensure it is materially different than a regular sleeping position, to avoid accidentally…
What is stress exactly? Stress is an abnormal alertness (and high rate of bodily energy drain) that is felt by an individual to be required in order to respond to the demands placed on them from their environment. Stress can generally be felt from negative situations ( distress ) or positive situations ( eustress ). Below I will only consider distress. An environment is stressful to a particular…
In recent years I’ve increasingly noticed software being written that cannot get the performance it needs unless it bypasses usual operating system services. Concurrency For example let’s consider a program that wants to do many tasks at the same time. Traditionally you would either create multiple threads or multiple processes for each parallel line of execution. But threads and…
Dependent types are a feature of certain programming language type systems that are unusually powerful, expressive, and precise, compared with other kinds of types. Dependently-typed programming languages such as Coq, Agda, and Idris appear occasionally in academia but not at all in mainstream languages used by software practitioners. I have been curious about dependent types for some time because…
What is a timechart? Why are they useful? A timechart is an organizational device of my own invention that helps you track what you are working on from minute-to-minute during the day. It provides time-awareness , the sense of time passing. It keeps you on task , as any distracting activities show up immediately on the chart. For someone like myself, who is easily distracted and often goes down…
I have frequently wanted to use the external link symbol ( ) in text that I’m writing, without having to resort to including an image. Normally the solution there would be to find the graphical symbol in Unicode and figure out how to type it. Apparently at least one proposal for the external link symbol was submitted to the Unicode Consortium as far back as August 2006. It was reviewed at a…
There are several patterns for structuring modules in JavaScript. The most common ones I see talked about is the original JavaScript Module Pattern , AMD Modules , CommonJS Modules , and the emerging Harmony Modules . The large JavaScript application that I work on at work - an in-browser IDE - does not presently use any of these well-known patterns but instead uses what I’m going to call…
In this article I will describe the theory of performance testing 1 and how we conduct such testing in practice at my company TechSmart, in the context of rich web applications and web services. By the end of this article you should understand how we define performance testing and perhaps get some ideas for implementing or customizing your own tools for performance-testing your own web service.…
The conventional wisdom in the academic community appears to be that a type system is not useful if it cannot be proven to be sound 1 . However as a software practitioner I definitely find unsound type systems to still be quite useful. The principal benefits that I get out of a type system include: Identification of many common types of errors , in particular: misspelled names of functions and…
I just started implementing my first big new software feature since the start of the New Year. I thought I’d outline my process since I don’t think I’ve written it down before. Let’s begin: A feature request comes in. In my case: Dynamic Calendar for Teachers: Extend our existing Plan page, which shows a calendar of daily activities, such that classroom teachers (our…
When trying a new recipe for the first time I annotate it to highlight certain important parts. An annotated recipe is easier to follow when cooking. And the process of annotating a recipe forces me to actually read the recipe. Here is a recipe that I’ve already annotated: In the ingredients list Underline implicit preparation steps like “chopped”, “minced”,…
This year I spent a lot of time teaching myself to cook more complicated dishes from cookbooks, doing meal planning, and learning related skills. In this article I’d like to share my system for learning to cook better. Commit to becoming skilled at cooking This is all in your head: Decide that you really want to get better at cooking. Get excited. Imagine your favorite tasty dish from a…
You’ve probably heard that global variables are bad. Today I want to explain why they are problematic, with examples, and give some design alternatives where you might be tempted to use a global variable. What is a global? A global is a variable from the environment that a function can directly read or write without going through a function parameter. For example, here is a warn function…
I spend a lot of time as a professional coder working on very large programs, attempting to grow them while also keeping them from collapsing under their own weight. This is hard. Abstraction and Encapsulation The Challenge A large program has a lot of behavior to specify. The complexity of the behavior specified by a program is roughly proportional to its size. 1 However a coder, being only…
The OS X Photos program has a built-in Share button that can post selected photos directly to Facebook. However it has the following limitations: Photos are not uploaded in high quality. Photos are not uploaded in the correct order. Neither the order of selection nor the order the photos were taken is used exactly. Photos cannot be uploaded to a new album, only an existing one. Therefore I…
Why Functional Programming Matters is a famous paper on the merits of functional programming (FP). It argues that FP has two big special tools for glueing programs together: Higher-order functions can be easily composed with other functions to create powerful composite functions. Lazy evaluation allows efficient processing of streams and large data structures. Higher-Order Functions, meet List…
Read a very interesting article about error handling in Midori recently, and it got me thinking about errors again. I’ve thought about errors a lot in the past, as you can see in my old Error Handling article from 2013. Midori mentions a few mechanisms for handling errors: error codes , unchecked exceptions , checked exceptions , and abandonment . Midori has chosen to run with abandonment…
Why wear a hat? protect against cold air protect against cold wind protect against sun be easy to spot or recognize look cool; look distinctive declare social identity to others; attract folks with similar identity When to especially wear a hat? extreme weather very cold or windy very sunny long exposure to weather going boating going hiking for > 2 hours weak or missing natural protections…
I recently picked up a copy of The Clean Coder : An excellent book about the non-technical aspects of being a senior software engineer. Below are my notes taken while reading the book. Introduction xvi. Management sometimes doesn’t view software engineers as professionals in the same way, for example, they treat lawyers as professionals. In this situation management is likely to babysit the…
I currently work at a software startup and wear a lot of hats. Here are some of the hats I wear and some of the hats I interact with. Product Manager A product manager envisions new things to create. Is either a stakeholder proxy or a direct stakeholder : Stakeholder Proxy : Represents the desires of external stakeholders, such as customers. This is an outbound role. Direct Stakeholder :…
Some programming languages are better at some tasks than others. Below I have presented my own assessment of how various languages stack up against each other for the following common classes of tasks: Programmer-bound : Degree of expressive power. Magnitude of what you can implement with a small development staff. Affected by design simplicity, platform stability, and library availability.…
In general I feel that deep algorithm knowledge is overrated in the software industry. In the early 90’s, one needed to know about common algorithms because you needed to actually implement them. Today in the 00’s and 10’s one mainly needs to know which algorithms are appropriate to use , since most common algorithms are already implemented in the standard library of modern…
Handling international and Unicode text correctly in modern programming languages remains a poorly understood topic. Read more…