Recent content on jpt.sh
I wound up doing some nice tool upgrades this year. The biggest improvement– I’m now using Linux again for most things. Below are my defaults, if I switched to it in the last year, I’ve also listed the tool it replaced. web email : Fastmail todo : tweek ( my own tool, tt ) calendar : Fastmail, Fantasical rss : miniflux , self-hosted git : codeberg ( GitHub ) hosting : bunnyCDN…
I just want to host some HTML… For years I hosted my static sites on AWS CloudFront , but about six years ago, in an effort to move away from Amazon, I wound up on Netlify. Netlify has mostly been great, I have recommended it to friends and, more recently, students looking for a place to host their portfolios & personal projects. While the interface & offerings have gotten far more complex…
I’ve released a new Python library for making HTTP requests to less-than-reliable websites : careful repo careful documentation It offers: configurable retries . retry on timeouts or other errors, with exponential back-off. simple request throttling . set a maximum number of requests per minute. a scraper development cache . configurable caching aimed at reducing redundant requests made…
learn to like your terminal I’ve met a lot of people that have learned to use the terminal just enough for it to remain a perpetually frustrating experience. There’s a lot to love about terminal programs in 2025: CLI tools are typically faster and less memory-intensive than (increasingly) bloated GUI tools the UIs are remarkably consistent over time, no installing an update and having…
I’ve had this post sitting in my drafts since April. I decided to finish it in light of the news that GitHub is no longer independent inside Microsoft and going full-tilt into AI slop. I’ve decided to migrate all of my personal projects 1 off GitHub— come join me. Going forward, my public work will be on https://codeberg.org/jpt/ Read on if you’d like to understand why, or you…
In the spring, I teach a project-based software engineering course titled Software Engineering for Civic Tech . Students take it in the last quarter of their time in the program, and it’s a lot of fun to see them put together everything they’ve learned. As the name implies, the focus of the course is on learning to build by listening to & understanding the community you intend to…
As mentioned in my last post , this was my third year teaching CAPP 30122 , the second CS course in the program I teach in. I love teaching this course because students get to build their first real projects with their new skills & tend to build really interesting data-driven projects. The projects are built in (less than) 9 weeks, and require them to bring together several sources of data, do…
I’ve decided to make the course notes public for a course I just finished teaching for the third time. The course assumes a few months worth of Python knowledge and practice, but no programming beyond that. It aims to help people expand their general programming knowledge, get better at Python, and learn to work effectively with real-world data (in a policy context). (what’s covered?)…
I’ve spent nearly twenty years working in civic tech. It feels like with everything going on right now, I should have something to say… This is my attempt to put into words what’s been swirling around in my head during this chaos. This is my perspective , which I know is just a small part of the picture. In particular, I am focused on the US here, and while I’ve worked…
( Notes from a brief introduction I gave at the 10th Annual Civic Scopeathon , put on by my students. Someone asked me to share it, so putting it up here without any real editing.) Civic Tech is a young field, about 20 years old. That means Scopeathon has been around for about half of that time. I want to take a second to talk about what has changed in civic tech in those past twenty years, and…
A while back I got asked to do a usesthis.com interview, and it is now up here: https://usesthis.com/interviews/james.turk/ . I’m a big fan of usesthis.com and other sites that focus on tools & this was fun to do. A lot has changed between when I wrote this & now, I suppose that means I should write a new post.
I’d had this draft ready and fortuitously found out today is World Wide Web Day even though nobody knows why . We tend take the web for granted, so it is nice that there is at least a nominal day for it. (The web is only 35 and it took Earth a few billion years to get a day named for it.) The web is an incredible achievement, an attempt to build a “universal linked information…
Back in 2010 , my colleague Michael created a Python library with C implementations of a bunch of common string distance algorithms. I remember wanting to call it strfry at one point but jellyfish won in the end. A year or so later I eventually took over maintenance of the library and didn’t give it much thought for a while & certainly didn’t expect it to become the most widely-used…
Media Party 2023 Presentation: Web Scraping with GPT Media Party is a conference that “facilitates the intersection between the media and technology industry in an innovative, international and hands-on way.” This year was their first one in the US, and it was conveniently here in my backyard. The organizers were nice enough to invite me to speak on the work I’ve been doing…
A couple of days ago I wrote about using GPT-4 to automate web scraping . I was surprised at how well it worked for a first attempt and have been curious how far I go to make it useful. I’m building an experimental interface here if you’d like to follow along: https://codeberg.org/jpt/scrapeghost Here’s what I’ve been working on: Command Line Interface If you’d like…
(Part 2 now available: Automated Scraping with GPT-4, Part 2 ) Like most people I know, I’ve been watching the pace of improvements to LLMs like ChatGPT and GPT-4 with a mix of awe and trepidation. I’ve been wanting a small project to get to explore the APIs better, and recently decided I’d try to see if I could use it to automate web scraping. For context, I’ve written a…
Update August 2025 VS Code is clearly deep down the path of enshittification. I can no longer recommend it for anyone. If you want a similar tool with less Microsoft bloat, try VSCodium . You might also try something different: Helix is my editor of choice, I’ve heard good things about Zed which now has an AI-free mode and seems worth a look, or look at JetBrains products if you prefer a…
What We Lost If you’re around my age, you remember an internet dominated by niche message boards. Gaby Del Valle’s article did a great job capturing what was special about them & how they differ from present day social media. In my teenage years, I was very active on the old GameDev.net. I wound up becoming a moderator, spending my time answering questions, talking about nonsense, and…
Earlier this week, Django 3.1 beta 1 was released . While the big news is the addition of asynchronous views and middleware, there’s a tiny feature I managed to get added that I wanted to bring a bit of attention to as it makes writing apps that use Postgres’ full text search a little easier & nicer. It’s a small line in the changelog , but the SearchQuery constructor now accepts…
May 29th, 2007 On May 29th 2007 I’d been living in DC for a little under a week. I liked it quite a bit, which was lucky as I’d made plans to move there sight unseen. In December of 2006 I’d accepted a position with Sunlight Foundation , an up-and-coming nonprofit with a mission I cared about and which already seemed to place an importance on technology. In fact my position was…
Like many developers, I’ve written my own blog software half a dozen times. I’ve probably spent a lot more time tweaking the software than writing posts so this time I decided to do something different. I wanted to keep things as simple as possible but still have an excuse to learn a few new things. I wound up settling on Hugo hosted on Amazon S3 w/ CloudFront in front of it. One of…
Note: Another old one that I thought was interesting enough to migrate. I’d meant to keep doing this every year. Maybe next February? Earlier today I noticed an email I was looking up was from exactly three years ago. From there I wound up on a random path that had me reading emails from February 24th for the past 6 years. It wound up being a pretty interesting look at what I’ve been…
Note: An old post I brought over because at least two people started small dotfiles projects based on this :) I now use chezmoi for this! If you find yourself logging into 3 or more machines in a given day (for me my laptop, work machine, and a few servers) you’ve probably faced the problem of being on a machine and not having some essential piece of functionality in your .vimrc or .bashrc…
Note: an old post from 2008… not the best post but a nostalgic look back at my introduction to organizing developers. This past weekend I had the pleasure of attending the (fantastically organized) Rochester BarCamp 3 on the RIT Campus. A BarCamp is basically a group of knowledgeable people that get together and give talks on subjects that they care about. It's quite possibly one of the…
Note: This is an old post from a blog I had in 2007, I put it up here for posterity. Hardware: My laptop is a PowerPro J 10:15 , which is a Compal IFL90 sold by the great people at PowerPro. I bought my first laptop from them years ago and it served me amazingly well, and when it came time last month to replace it, I went back to them (I’m not a huge fan of their site, but once you get past…