In 2024, I noticed that I'd let my blog languish. Since the issue was urgent, I made a firm new year's resolution to address the situation in 2025. Which is why, today, in January 2026, I'm writing this post. I've just released spacecurve , a new just-for-fun space-filling curve project. It's is the latest symptom of a long preoccupation with these beautiuful mathematical…
A couple of years ago a paper titled Progressive Growing of GANs for Improved Quality, Stability, and Variation cropped up on my reading list. It describes growing generative adversarial networks progressively, starting with low-resolution images, and then building up more detail as training goes on. It got quite a bit of press at the time because the authors used their idea to generate realistic,…
Outside the Al Huda Mosque near my home (by Mark McGuire ) A year ago, my wife and I decided to become citizens of New Zealand. Both of our sons were born here and are full, native Kiwis. It felt odd for our family not to have this in common, and besides, our own connection with New Zealand had grown strong over the happy decade we'd lived here. It was time to take the plunge. Forms were…
Six years after mitmproxy's first checkin, we've finally released version 1.0.0 of the project. Our version numbering persisted below 1.0 well into the project's maturity, for reasons that are a tad difficult to explain. My mental model of software development is of an eternal pilgrimage - the roadmap of possible improvements stretches on forever, and we never quite reach a point…
We've just released mitmproxy v0.18 ! Since the last release, the project has had 1399 commits by 40 contributors, resulting in 217 closed issues and 305 closed PRs, all of this in just over 189 days. This release is notable for a number of reasons. First, it contains significant contributions from our three excellent GSOC students this year. Shadab Zafar worked on Python 3 compatibility and…
Eight years ago my wife and I walked into the Cat Protection Society near our house in Sydney on a whim - just to look, we assured each other, and most definitely not to get another cat. Thirty minutes later we emerged with a box containing a tiny ball of scraggly orange fluff, a wee kitten we immediately named Hobbes. Circumstances had taken Hobbes away his mother far too early, and since I was…
I've just released modd , a new 1 project of mine. Like its sister project devd , it's distributed as a single, self-contained binary for all major platforms - get it while it's fresh . Modd is a simple tool that's hard to explain pithily. It triggers commands and manages daemons in response to filesystem changes - but that is a technically-correct mouthful that doesn't…
We've just released mitmproxy 0.15 . This is primarily a bugfix release, but with a few really juicy long-demanded features thrown in: Support for loading and converting older dumpfile formats (0.13 and up) Content views for inline script (@chrisczub) Better handling of empty header values (Benjamin Lee/@bltb) Fix a gnarly memory leak in mitmdump A number of bugfixes and small…
It's a universal rule that search over a sufficiently large body of user data poses security challenges. This follows naturally from the fact that humans - even smart, informed, careful humans - occasionally slip up. Given enough data, and the ability to pick out slip-ups with search, there will always be rich pickings for a malefactor. I wrote a short series of posts a while ago about…
I've just released devd 0.3 - a measured increment, with a modest set of bugfixes and new features. This is inline with my broad plan to keep devd a small, dependable, and focused tool. Everyone should update. -s (--tls) Generate a self-signed certificate, and enable TLS. The cert bundle is stored in ~/.devd.cert Add the X-Forwarded-Host header to reverse proxied traffic. Disable…
We've just released mitmproxy 0.14 ! Since the last release, the project has had 399 commits by 13 contributors, resulting in 79 closed issues and 37 closed PRs, all of this in just over 100 days. Docs: Greatly updated docs now hosted on ReadTheDocs Docs: Fixed Typos, updated URLs etc. (Nick Badger, Ben Lerner, Choongwoo Han, onlywade, Jurriaan Bremer) mitmdump: Colorized TTY output mitmdump:…
I've just released version 0.2 of devd , a local webserver for developers. This release contains a number of small improvement, and a few new features. -x (--exclude) flag to exclude files from livereload. -P (--password) flag for quick HTTP Basic password protection. -q (--quiet) flag to suppress all output from devd. Humanize file sizes in console logs. Improve directory indexes - better…
I've just released devd , a small, self-contained, command-line-only HTTP server for developers. It started as a weekend stress-relief hack (that's a thing where I'm from), but has now become my preferred "daily driver" for most web-ish things. It's simple, direct and does more or less exactly what I need. This isn't terribly surprising, since I wrote it to scratch my own…
This is a slightly late announcement of the release of mitmproxy v0.13 , which was pushed out the door earlier this week by my esteemed compatriots while I was tied up with other things. We have a number of big new features this time round. First, mitmproxy now has upstream certificate validation, thanks to the hard work of Kyle Morton . Mitmproxy is increasingly being used in user-oriented roles…
I've just released mitmproxy v0.12.1 . This release fixes a few crashing bugs that slipped through in the previous iteration, so everyone should upgrade. Also included are a number of small improvements. The most noticeable of these is mouse interaction for mitmproxy console - the screen capture above shows me scrolling with my mouse, clicking to view a flow and switch tabs. We pay a small…
Project News Before we get to the new release, I'd like to give a quick update on some internal project developments. First up, after a somewhat involved process that included a couple of rounds of community voting and much discussion, we have a new logo: This will be rolled out in all the places where it makes sense along with the 0.12 release. Second, the long-dormant @mitmproxy Twitter…
Over the years, I've written a number of posts on this blog on the topic of binary data visualisation. I looked at using space-filling curves to understand the structure of binary data , I've showed how entropy visualisation lets you trivially pick out compressed and encrypted sections , and I've drawn pretty pictures of malware . Unfortunately the tools I wrote ( code here ) all…
I've just pushed mitmproxy v0.11.2 out the door. This is primarily a bugfix release, but does have one very useful new feature: configuration files. All options available through command-line flags can now be set persistently in config files, for all the tools - see the documentation for more . Adding this was made much easier by ConfigArgParse , one of those small Python project gems that…
I'm happy to announce that we've just released v0.11 of both mitmproxy and pathod . This release features a huge revamp of mitmproxy's internals and a long list of important features. Pathod has much improved SSL support and fuzzing. Our thanks to the many testers and [contributors](https: //github.com/mitmproxy/mitmproxy/blob/master/CONTRIBUTORS)…
A few weeks ago, I posted that I had hacked up a version of mitmproxy that exploited CVE-2014-1266 , giving unrestricted access to nearly all HTTPS traffic on affected IOS and OSX devices. I chose not to release working code at the time, but a number of POCs have been floating about publicly almost since the issue was first discovered. So, the time has come to publish - as of yesterday,…
This post is a quick recap of work I've been discussing on Twitter in the last few hours. I've just finished putting together a version of mitmproxy that takes advantage of CVE-2014-1266 , Apple's critical SSL/TLS bug . We knew in theory that the issue should give access to all SSL traffic using Apple's broken implementation - I can now report that this is also true in…
I've just released v0.10 of both mitmproxy and pathod . This is chiefly a bugfix release, with a few nice additional features to sweeten the pot. Perhaps the most visible change has been a huge improvement in the recommended method for installing the mitmproxy certificates. Certs are now served straight from the web application hosted in mitmproxy, which means that in most cases cert…
Here's a riff on Malcolm Gladwell's rule of thumb about mastery : you don't really know a programming language until you've written 10,000 lines of production-quality code in it. Like the original this is a generalization that is undoubtedly false in many cases - still, it broadly matches my intuition for most languages and most programmers 1 . At the beginning of this year, I…
I've just released v0.9.2 of both mitmproxy and pathod . This is a bugfix release, chiefly to address two crashing issues affecting mitmproxy when relaying SSL traffic. A range of other fixes and improvements are also included - if you use mitmproxy, you should upgrade. CHANGELOG Improvements to the mitmproxywrapper.py helper script for OSX. Don't take minor version into account when…
choir.io Today, I'm raising the veil (slightly) on a new project - choir.io . The most succinct description of choir.io is that it is a service that turns events into sound. Why would you want to do that? Well, I believe that there are compelling reasons to make sound part of your monitoring stack. Let's see if I can convince you. The soundscape When I walk into my study every morning,…
I'm happy to announce the release of mitmproxy 0.9.1 . This is a bugfix release, with no significant changes in behaviour. As hinted in my previous release note, the project itself is also evolving. As of this release, mitmproxy and its sister projects ( pathod and netlib ) are housed under a separate organization on Github, rather than my own personal space: github.com/mitmproxy…
I've become a bit weary of the process of public vulnerability disclosure - I'm much more likely nowadays to just drop companies an anonymous notice and move on. Every so often, though, I come across an issue so egregious that talking about it publicly seems like an imperative. This is one of them. First, some background. Skout is a location-based mobile social network. The idea is to…
I started work on mitmproxy because I was frustrated with the available interception tools. I had a long list of minor complaints - they were insufficiently flexible, not programmable enough, mostly written in Java (a language I don't enjoy), and so forth. My most serious problem, though, was opacity. The best tools were all closed source and commercial. SSL interception is a complicated and…
I've just released pathod 0.9 , my toolset for crafting malicious and interesting HTTP traffic. Apart from the usual range of stability improvements and bugfixes, this release introduces a major new set of features: proxy support. Pathoc , the client, has sprouted support for vanilla proxy connections, and is also able to tunnel through proxies using CONNECT. Pathod , the server, will now…
I'm happy to announce the release of mitmproxy 0.9 . This is a major release, with huge improvements to mitmproxy pretty much across the board. So much has happened in the year since the last release that it's difficult to pick out the headlines. Mitmproxy is now faster, more scalable, and works in more tricky corner cases than ever before. Full transparent mode support has landed for…
Google has finally shut down a service I actually care about - Google Reader will die a graceless, undignified death on July 1, 2013 . The only way Google could inconvenience me more would be to shut down search itself, and yet - I'm not angry that Google is shutting Reader down. I'm furious that they ever entered the RSS game at all. Consider this quote from a TechCrunch article in…
I've been doing a series of posts looking at data gathered with ghrabber , a simple tool I wrote that lets you grab files matching a search specification from GitHub. Last week, I looked at shell history in the broad, and then specifically at pipe chains . Today, I move on to something different - custom aspell dictionaries. When aspell finds a word it doesn't recognize, the user is…
Earlier this week I published ghrabber , a simple tool that lets you grab files matching an arbitrary search specification from GitHub. I used ghrabber to retrieve all the bash_history and zsh_history files accidentally checked in to repos, and took a light look at the dataset with some simple graphs . In total, I obtained 234 shell history files with 165k individual command entries. This is a…
Github recently introduced hugely improved code search , one of those rare moments when a service I use adds a feature that directly and measurably measurably improves my life. Predictably, there was soon a flurry of breathless stories about the security implications. This shouldn't have been news to anyone - by now, it should be clear that better search in almost any context has security or…
There is something terribly awry with the social news ecosystem. This is a feeling that's been growing on me over the last few years, and is the reason why I've cut both Reddit and Hacker News (who together constitute pretty much all of "social news") out of my information diet. Although I've mulled over things in various conversations, I've never actually tried to put my…
Last week, I had the pleasure of attending Dropbox 's annual company hack fest . It was a great opportunity to get a look at how Dropbox works internally, and mingle with the smart and driven folks who make one of my favourite products. In the spirit of hack week, me and my friend @alexdong decided to do our project in Go. We'd both wanted to explore the language, but had never quite…
I've just released pathod 0.3 , which beefs up pathoc 's fuzzing capabilities, improves the spec language and includes lots of bugfixes and other small tweaks. Get it while it's hot! Better fuzzing A major focus of this release is to improve pathoc 's capabilities as a basic fuzzing tool. I've had fun breaking webservers with pathoc, and it's even come in handy in my…
A few months ago, I announced pathod , a pathological HTTP daemon. The project started as a testing tool to let me craft standards-violating HTTP responses while working on mitmproxy . It soon became a free-standing project, and has turned out to be incredibly useful in security testing, exploit delivery and general creative mischief. In the last release, I added pathoc - pathod's malicious…
I recently wrote a series of posts using the Hilbert curve to visualize binaries, culminating in a gallery showing regions of high entropy in malware . The fact that the Hilbert curve has excellent locality preservation means that one dimensional features are preserved (as much as they can be) in the two-dimensional layout. This lets us visually pick out features of interest, and makes it…
It's become quite a popular parlor game to guess who is responsible for the recent Antisec UDID leak. I've now seen no less than six separate apps named as the probable source (two of which came from Marco Arment ). Before we pick the next culprit, I think it's worth taking a step back to consider the list of things we don't know: We don't know that we're dealing with…
Something I've been worrying about for a long time has just happened: Antisec has leaked a database with more than a million UDIDs . The UDID issue has been a bit of a white whale of mine - I've written many blog posts about it and spent more hours than I care to think negotiating responsible disclosure with companies misusing UDIDs. Let's recap some of the posts I've written…
I've been living out of a bag for the last 3 weeks, working hard on a series of intense but fun audits. After running in high gear for a while I find that I need a mental palate cleanser - something to help me refocus and stop me from getting snowblind. I then grab my camera, strap on my macro rig, and walk out the door to try to catch the local wildlife in the act. It's become a bit of…
I've just pushed pathod 0.2 out the door. This is a huge release, with many new features: pathoc , pathod's evil client-side twin. libpathod.test , a framework for using pathod in your unit tests. Improved mini language , including many new abilities and improvements. A rewrite of the networking core. The project also has a new website at pathod.net . Yes, pathod is now self-hosting, so…
I've just released pathod , a pathological HTTP/S daemon useful for testing and torturing HTTP clients. At its core is a tiny, terse language for crafting HTTP responses. It also has a built-in web interface that lets you play with the response spec language, inspect logs, and access pathod's full help document. The rest of this post is a quick teaser showing some of pathod's…
I'm happy to announce the release of mitmproxy 0.8 . This release has a few major new features, big speedups, and many, many small bugfixes and improvements. Here are the headlines: Android interception The most prominent new feature is that we now have a supported way to intercept Android traffic. What's more, we can do this without a cumbersome transparent proxying rig - see the…
I'm happy to announce the release of mitmproxy 0.7 . The biggest visible change is a new structured editor for headers, query strings and form fields. Other new feature include a reverse proxy mode, extended script API that makes many common tasks much easier, and a myriad of improvements to the interface (including a massive increase in speed). Everybody still on 0.6 should upgrade - get it…
My leisurely Sunday activity today is to set up a new OpenBSD firewall for my mobile app testing lab. I haven't done a from-scratch OpenBSD install for years, so I spent some time reading through the change logs for the last few versions to catch up with what's changed. Although the project is clearly still making steady, well-engineered progress, I had the nagging feeling that the rate…
Edit: Since this post, I've created an interactive tool for binary visualisation - see it at binvis.io Hover and click for more. .malware { } .malware tr { border: 0; } .malware td { border: 0; position: relative; margin: 0 auto; width: 128px; height: 138px; } .malware td img { position: absolute; top:0; left:0; overflow: hidden; height: 128px; width: 128px; } .malware td .entropy { z-index:…
Edit: Since this post, I've created an interactive tool for binary visualisation - see it at binvis.io Last week, I wrote about visualizing binary files using space-filling curves , a technique I use when I need to get a quick overview of the broad structure of a file. Today, I'll show you an elaboration of the same basic idea - still based on space-filling curves, but this time using a…
I posted a link to an interesting visualization paper on Twitter today, prompting someone to ask me where I had found it . Sadly, I had to admit that I had no clue where I first saw it referenced, due to the way I consume links I find on the net. So, I thought I'd write a quick blog post to explain myself, and then pitch a product idea that could make my life (and maybe yours) much easier.…