If you listen real close you can hear the type whispers in there. Peewee 4.x is a picture, in code, of me eating my hat. We got async and are shipping a significantly better type whispering stub based on typeshed's original, which finally gets the whole Model.field vs instance.field distinction. Beyond those, I've added a core JSONField which exposes a cross-backend API for working with JSON data,…
A todo list full of secrets. Load-bearing secrets. This secret terminal was embedded in the app. Let's build a mid-century country-club-core todo-list app with Python, I thought. But, wait, another voice seemed to say, let's have Claude build us the mid-century country-club-core todo-list app with Python . I put down my dino grabber (🦖), rolled up my sleeves and began writing some prompts. I ran…
The one great principle of the English law is, to make business for itself. At work we were talking about metrics. Well, they were talking about metrics, and then when they realized they had none, they asked me to generate some. I spent an hour or so putting together a script that pulled all the relevant historical data from our database, cleaned and normalized into a CSV. Then I fed the CSV into…
Who is he that hideth counsel without knowledge? therefore have I uttered that I understood not; things too wonderful for me, which I knew not. "Hold on a sec-". My new boss' now-familiar MacOS desktop appeared in the video call, browser with Claude open, dominating the screen. I watched as he copied the transcript of our call up to that point (he records transcripts of every call in order to feed…
A year ago, as I was going through a mound of keepsakes my Mom transferred to my custody (I have reached that age, yes), I came across a little book I made in kindergarten describing my first bicycle crash, which I attributed to rolling over a pine-cone. I have very little recollection of the crash itself - I only recall sitting, high up on the passenger seat of the minivan, with a towel pressed…
And they said, Go to, let us build us a city and a tower, whose top may reach unto heaven; and let us make us a name, lest we be scattered abroad upon the face of the whole earth. I recently skimmed antirez's patch for adding an array type to Redis. The patch itself is not particularly noteworthy except as an example of how AI-assisted tooling can augment the abilities of a talented and tasteful…
Back in the spring of 2019, I began working on cysqlite , a from-scratch DB-API compatible SQLite driver. I intended one day to use it as a replacement for pysqlite3 . I began experimenting with the library and eventually got it to a point where it was working fairly well, but it needed a lot more time, effort and polish before I would think of using it. Life got busy, we were blessed with two…
A man's at odds to know his mind cause his mind is aught he has to know it with. He can know his heart, but he dont want to. Rightly so. Best not to look in there. It aint the heart of a creature that is bound in the way that God has set for it. You can find meanness in the least of creatures, but when God made man the devil was at his elbow. A creature that can do anything. Make a machine. And a…
I feel that it is high time that Peewee had an assyncio story. I've avoided this for years. Database operations and transactions, while needing to support some degree of concurrency, are still rather heavy operations. Postgres default config specifies a max of 100 connections, for example, and SQLite doesn't even support concurrent writers. Nonetheless, the writing has been on the wall for years…
Edit 2026-01: peewee now supports asyncio , and a bit more detail here . I'd like to put forth my current thinking about asyncio . I hope this will answer some of the questions I've received as to whether Peewee will one day support asyncio, but moreso I hope it will encourage some readers (especially in the web development crowd) to question whether asyncio is appropriate for their project, and…
I'd like to share a simple trick I use to reduce roundtrips pulling data from a cache server (like Redis or Kyoto Tycoon . Both Redis and Kyoto Tycoon support efficient bulk-get operations, so it makes sense to read as many keys from the cache as we can when performing an operation that may need to access multiple cached values. This is especially true in web applications, as a typical web-page…
I'm pleased to announce that Peewee now supports CockroachDB (CRDB), the distributed, horizontally-scalable SQL database. I'm excited about this release, because it's now quite easy to get up-and-running with a robust SQL database that can scale out with minimal effort ( documentation ). Here is how you configure a CockroachDatabase instance: from playhouse.cockroachdb import CockroachDatabase db…
With the release of Python 3.8 coming soon, the core development team has asked me to summarize our latest discussions on the new features planned for Python 4.0, codename "ouroboros: the snake will eat itself". This will be an exciting release and a significant milestone, many thanks to the hard work of over 100 contributors. After heated debate on the mailing list, the 79-character line limit…
I recently wrote about Kyoto Tycoon (KT), a fast key/value database server. KT databases may be ordered (B-Tree / red-black tree) or unordered (hash table), and persistent or stored completely in-memory. Among other things, I'm using KT's hash database as a cache for things like HTML fragments, RSS feed data, etc. KT supports automatic, time-based expiration, so using it as a cache is a natural…
I saw an excellent article recently describing how to implement the fire effect seen in the trailer for the N64/PlayStation ports of the DooM game. I figured this would be neat to put on the page displayed whenever there's a server error. I already have an awesome 404 page , and now I'm equally happy with the 500 page. I made a few small modifications to the code presented in the article . I'm…
I've been interested in using Kyoto Tycoon for some time. Kyoto Tycoon, successor to Tokyo Tyrant , is a key/value database with a number of desirable features: On-disk hash table for fast random access On-disk b-tree for ordered collections Server supports thousands of concurrent connections Embedded lua scripting Asynchronous replication, hot backups, update logging Exceptional performance This…
In this post I'll present a short code snippet demonstrating how to use Redis streams to implement a multi-process task queue with Python. Task queues are commonly-used in web-based applications, as they allow decoupling time-consuming computation from the request/response cycle. For example when someone submits the "contact me" form, the webapp puts a message onto a task queue, so that the…
Redis 5.0 contains, among lots of fixes and improvements, a new data-type and set of commands for working with persistent, append-only streams . Redis streams are a complex topic, so I won't be covering every aspect of the APIs, but hopefully after reading this post you'll have a feel for how they work and whether they might be useful in your own projects. Streams share some superficial…
In this post I'll provide instructions for building a standalone Python sqlite3 -compatible module which is powered by BerkeleyDB . This is possible because BerkeleyDB provides a SQL frontend , which essentially is the SQLite we all know and love with the b-tree code ripped out and replaced with BerkeleyDB. Why? Why would anyone use BerkeleyDB's SQLite in the first place? There are a couple of…
I began an unlikely adventure into Python packaging this week when I made what I thought were some innocuous modifications to the source distribution and setup.py script for the peewee database library. Over the course of a day, the setup.py more than doubled in size and underwent five major revisions as I worked to fix problems arising out of various differences in users environments. This was…
The upcoming SQLite 3.25.0 release adds support for one of my favorite SQL language features: window functions . Over the past few years SQLite has released many changes to improve the efficiency of the query planner and virtual machine, plus many extension modules which can provide additional functionality. For example: Window function support , which are available in trunk and will be included…
I recently open-sourced a Python library for working with the networked key/value databases Kyoto Tycoon and Tokyo Tyrant . These databases sit atop Kyoto Cabinet and Tokyo Cabinet , respectively, and provide fast DBM implementations. The Cabinet libraries expose a familiar key/value API backed by a number of different storage options, including persistent hash-tables and b-trees, as well as…
On Monday of this week I merged in the 3.0a branch of peewee, a lightweight Python ORM, marking the official 3.0.0 release of the project. Today as I'm writing this, the project is at 3.0.9, thanks to so many helpful people submitting issues and bug reports. Although this was pretty much a complete rewrite of the 2.x codebase, I have tried to maintain backwards-compatibility for the public APIs.…
The Python standard library sqlite3 driver comes with a barely-documented hook for implementing basic authorization for SQLite databases. Using this hook, it is possible to register a callback that signals, via a return value, what data can be accessed by a connection. SQLite databases are embedded in the same process as your application, so there is no master server process to act as a gatekeeper…
The other day the idea occurred to me that it would be neat to write a simple Redis -like database server. While I've had plenty of experience with WSGI applications, a database server presented a novel challenge and proved to be a nice practical way of learning how to work with sockets in Python. In this post I'll share what I learned along the way. The goal of my project was to write a simple…
Several months ago I was delighted to see significant progress on a relatively new extension in the SQLite source tree. The lsm1 extension is based on the LSM key/value database developed as an experimental storage engine for the now-defunct SQLite4 project. Since development has stopped on SQLite4 for the forseeable future, I was happy to see this technology being folded into SQLite3 and was…
Task queues are frequently deployed alongside websites to do background processing outside the normal request/response cycle. In the past I've used them for things like sending emails, generating thumbnails, warming caches, or periodically fetching remote resources. By pushing that work out of the request/response cycle, you can increase the throughput (and responsiveness) of your web application.…
In this post I'd like to share with you some techniques for effectively working with SQLite using Python. SQLite is a capable library, providing an in-process relational database for efficient storage of small-to-medium-sized data sets. It supports most of the common features of SQL with few exceptions . Best of all, most Python users do not need to install anything to get started working with…