Recent content on Justin Azoff
Generating clean room PCAP files can be a difficult orchestration problem. The NixOS Test framework makes this easy. The Problem I work with Zeek and Suricata which process live network traffic. When a performance or correctness issue arises on a customer network, a PCAP file is required to reproduce and resolve the issue. Unfortunately, capturing a live PCAP from a customer network is typically…
Error messages are hard. When a program logs an error message sometimes both of these things can be true: The error message can be 100% accurate. The error message can be completely unhelpful, if not outright misleading. Let’s look at some examples: python JSONDecodeError import json import sys with open ( sys . argv [ 1 ]) as f : contents = f . read () print ( json . loads ( contents ))…
A common goal in Go is wanting to ensure that calling a function results in a small number of allocations. Often you want to go even further and ensure that calling a function results in zero allocations. There are guides on how to write benchmarks that can tell you how many allocations a function is making. However, once you have such a benchmark, it’s up to you to notice if the allocation…
Similar to how continuous incremental scanning is a good thing for security tools, zfs supports a way to do incremental pool scrubbing. They don’t really call it that, but if you look at the options for a recent version of zpool you will see the following new option: -p Pause scrubbing. Scrub pause state and progress are periodically synced to disk. If the system is restarted or pool is…
Recently I needed a large PCAP of SMTP traffic for testing zeek scripts. Obtaining a capture from a production network would be a straightforward process. However, I needed a file that could be included in test suites, or copied to other hosts without having to worry about leaking sensitive data. I could have taken an archived MBOX file and replayed it into a server. This would have worked, but…
Many security tools could be better. The problem isn’t that they are buggy or don’t work properly, but that they are designed to work in a way that is not as useful to defenders as they could be. The primary example I’m going to use is ssh server weak credential scanning. There are many existing tools that do this, such as ncrack or hydra, but I wrote my own called ssh-auditor .…
Background eBPF maps are a core component of most XDP programs. I am working on a library called libflowbypass that uses some code and ideas from Suricata to implement flow cutoff inside the kernel using eBPF and XDP in a reusable way for any network monitoring tool. The core of this is 2 maps (one for v4 and one for v6) whose keys are the 5 tuple (proto, src, sport, dst, dstport) of each flow to…
Wiring an existing home for Ethernet is a fun project. There is a lot of information out there on how to accomplish this, unfortunately much of it is vague or outdated. It’s not uncommon to find guides that talk about using Cat5 cable or show pictures of ancient 10mbit networking devices. Other guides are more modern, but they gloss over specific details and fail to mention various pitfalls.…
When writing a component like a storage backend you often want a way to switch between different implementations like memory, sqlite, or redis. Another common example is authentication backends for LDAP or Kerberos. With interfaces in go you can have multiple implementations of a common API, but it takes a little more work to make them pluggable. 1 The goal is that we want to be able to have a…
I hate syslog. The protocol is terrible. The message format is terrible. The API is terrible. The protocol The proposed syslog RFC was updated in 2009. Despite being a recent standard, it is primary a formalization of current implementations and retains much of the baggage from the 1980s. There are alternatives , but nothing standardized. Most embedded devices or turn-key systems only support the…
My last post (over 2 years ago) was my initial tests of running my http_flood project under docker. At the time, there was a huge performance impact running through docker. Performance dropped by almost 90%. Two years later, things are a bit different. Bare metal: duration=3.364873568s megabytes=10000.0 speed=2971.9MB/s Inside docker: duration=7.283130136s megabytes=10000.0 speed=1373.0MB/s There…
I built a (somewhat bloated) docker image for http_flood with a simple Dockerfile root@dovm1:~# docker run -p 7070:7070 justinazoff/http-flood Unable to find image 'justinazoff/http-flood' (tag: latest) locally Pulling repository justinazoff/http-flood 0d71f044c41f: Download complete 511136ea3c5a: Download complete adfd622eb223: Download complete 9a776d8a79dd: Download complete 1819c4b85615:…
I spent a while the other day figuring out how to get websockets working on heroku, so I thought I’d write it up. First, Heroku doesn’t actually support websockets, so you must use something like socket.io which can fallback to various long polling mechanisms. Step 1, disable websocket support in socket.io Without this, socket.io tries to connect first using websockets and it takes a…
TL;DR Whatever you do, make sure you are using versioned python packages, even for simple tasks. And use pip+virtualenv. So you want to program in python.. It seems like only yesterday, and not 7 years ago, that I decided to learn python. I may not be the best python programmer, but I have made probably every mistake you can, so here are a bunch of things not to do, and a few things you should be…
os.popen uses the shell by default, and unlike subprocess.Popen, has no way of disabling it. Problems can occur when the program you are trying to run does not exist or is unable to be ran due to a permissions issue. Consider the following example function: def logged_in_users(): users = set() for line in os.popen("who"): users.add(line.split()[0]) return users This runs just fine when everything…
One of the first steps in groking ipv6 is getting a handle on ipv6 addresses. The ‘dotted quad’ notation for ipv4 is fairly simple, and other than possible zero padding issues, they all look the same. ipv6 addresses are a bit different. Rather than a dotted quad they are 8 hex groups, and there are a lot of rules for displaying the addresses. For working with ipv6 addresses there are…
A few days ago I installed Debian/kFreeBSD on my home server. It had been running opensolaris for years, but doing just about anything on that system was a complete pain in the ass. I had been meaning to give Debian/kFreeBSD a try, but had been putting it off thinking the changeover would break a lot of things, or I would have trouble importing the ZFS pools. The other day I had some free time so…
Nice page titles The first thing I wanted to do was fix the page titles. Blog posts should automatically have their page title set. This was a trivial change to head.mako: -<title>${bf.config.blog.name}</title> +<title> + BB.Net +%if post and post.title: +- ${post.title} +%endif +</title> Easy blogging The second thing I needed to do was write a script for easily adding a new post. newblog.py was…
I’ve been wondering why the web doesn’t have a mechanism for uniquely identifying a resource by a means other than its URL. I think if such a thing existed, then HTTP caches for common files could be shared between sites. There has been a push lately to let Google host common JS libraries for you. The main reason for this is increased performance, there are two cases where this helps:…
My ‘z’ key has been (physically) broken for a while now. Generally this isn’t a problem because there aren’t that many places where I need to type a ‘z’ that I can’t autocomplete it. Between tab completion in the shell, and the irssi dictcomplete plugin, it hasn’t bothered me that much. I finally got around to figuring out how to remap Caps lock to…
finding duplicate files This post is about my duplicate finding program, dupes . The program is a little bare, and needs a nicer API, but the method it uses is the most efficient one that I am aware of. There are a couple of different ways you can find duplicate files: Compute the hash of all the files, and look for duplicates This method works well if the files on disk are mostly static, and…
The problem The Xen documentation on live migration states : Currently, there is no support for providing automatic remote access to filesystems stored on local disk when a domain is migrated. Administrators should choose an appropriate storage solution (i.e. SAN, NAS, etc.) to ensure that domain filesystems are also available on their destination node. GNBD is a good method for exporting a volume…