tcpdump filters (pcap filters, really) for Fun and Profit?

I've been fighting what I'm pretty sure is a silly firewall problem while trying to get Synctrain to work when my iPhone isn't on my WiFi. Long time readers will know that when I am not on a trusted WiFi network my i{Pad,}OS devices automatically VPN into my network so they can tunnel all their traffic through my security apparatus (and multi-layer ad and tracking prevention systems). I'm getting ready to re-build one of my main servers and I'm trying to modernize some of the services it runs. According to the debian-installer logs the server was built on November, 5, 2011 in the early evening and I installed the recently released Debian 6.0 (squeeze) Linux distribution on it. One of the services it provides is a sync location for data from my i{Pad,}OS devices, specifically I use OwnCloud's app to sync my photos so I can edit them on my laptop and then organize them using my image-process python script. I want to get rid of OwnCloud and while the other services it provides (contact and calendar sync) are easily replaced, the file syncing is not. Enter syncthing.

Syncthing

While the Syncthing protocol seems to be a needlessly complex peer to peer thing it appears you can basically neuter most of the dumbness by just running a private relay and instructing your sync clients to use it. I don't need my data to be transported by anyone else and I don't understand why you would want that but thankfully I don't have to. I threw a relay on one of my servers in the same datacenter as my VPN hub and my laptop seems to be quite happy to talk to it. My phone seems to be a different story. When I am not on my local WiFi it tries to connect to the relay and gets a connection refused.

tcpdump

Of course at some point when troubleshooting network connections you're going to need to look at packets. I use OpenBSD for my firewall and VPN hubs so tcpdump is my go-to tool. Since I am using IKEv2 the OS has a handy enc0 interface that exposes the VPN traffic unencrypted and still wrapped in the ESP tunnel as it passes through the kernel. This is nice but proves to be a little tricky to filter when you have a lot of traffic (I am terminating more than just my iPhone's tunnel here) and are just looking for a couple packets (it turns out that Synctrain on my phone is only sending 3 SYN packets before giving up so it's a very short burst) in a sea of traffic.

tl;dr, if you are looking for a source address of 192.168.197.26 and a destination port of 22067 you can find that traffic with:

tcpdump -envi enc0 'ip[32:4] = 0xC0A8C512 and ip[42:2] = 0x5633'

Want to know why? Keep reading.

Protocol Decoding

To write a filter expression you need to first understand the pcap-filter(7) syntax, once you do looking at the command above you will see that it breaks down to: match 4 bytes at offset 32 with a hexadecimal encoded value of 0xC0A8C512 and 2 bytes at offset 42 with 0x5633. To understand how we arrived at those offsets and values we consult three RFCs and the OpenBSD manpage for enc(4).

RFC-791 describes the Internet Protocol and provides the header format for an IP version 4 packet.

RFC-793 describes the Transmission Control Protocol and the header format for a TCP/IP packet.

and RFC-4303 which will show you the packet format for an Encapsulating Security Payload packet.

Armed with the above you should be able to discern that 0xC0A8C51A is an IPv4 address stored as bytes (as opposed to the normal dotted quad notation of 192.168.197.26) and 0x5633 is 22067 (which is the syncthing relay port).

Now if we take a look at a packet from enc0 using an unfiltered tcpdump we will see that as described in the enc(4) manpage the packe is in fact decapsulated from the ESP packet but includes both the outer and inner IP headers (as an aside, this is why you can't just filter using the normal tcpdump src 192.168.197.26 and dport 22067 syntax).

The start of the packet I was analyzing to create the filter was dumped by tcpdump with -Xs 1500 and looks like:

  0000: 4500 0500 30a1 4000 4004 0000 a654 0724  E...0.@.@....T.$
  0010: ac38 04d1 4500 04ec 3854 4000 3306 ac94  .8..E...8T@.3...
  0020: 88f3 4f6d c0a8 c51a 0960 db74 1f0f c261  ..Om.....`.t...a
  0030: 870d 7fe3 8010 00f4 96ac 0000 0101 080a  ................

Broken into 4 byte blocks to match the RFC's IP header layout we see the outer IP header first, ending with the source and destination IP addresses in network byte order:

0x45 0x00 0x05 0x00
0xa1 0x40 0x00 0x40                          
0x04 0x00 0x00 0xa6 
0x54 0x07 0x24 0xac [  SRC ] 166.84.7.36 (0xA6540724)
0x38 0x04 0xd1 ---- [ DEST ] 172.56.4.209 (0xAC3804D1)

Then immediately following we get the next IP header which represents the payload of the ESP packet:

---- ---- ---- 0x45
0x00 0x04 0xec 0x38
0x54 0x40 0x00 0x33
0x06 0xAC 0x94 0x88 [  SRC ] 136.243.79.109 (0x88F34F6D)
0xF3 0x4F 0x6D 0xC0 [ DEST ] 192.168.197.26 (0xC0A8C51A)
0x8A 0xC5 0x1A 0x09 [  SRC PORT ] (0x0960)
0x60 0xDB 0x74 0x1F [ DEST PORT ] (0xDB74)

This IP header has the source and destination addresses in what looks to be host byte order at offset 32 and 36 respectively. We can tell from the byte in position 29 (0x06) that this IP packet contains a TCP packet so the values right after the IP addresses must be the source and destination ports at offsets 40 and 42. This particular packet decoded as:

11:51:56.460254 (authentic,confidential): SPI 0x08a855fe: 166.84.7.36 > 172.56.4.209: 136.243.79.109.2400 > 192.168.197.26.56180: . [tcp sum ok] 12081:13289(1208) ack 0 win 244 <nop,nop,timestamp 1275851728 3211940279> (DF) (ttl 51, id 14420, len 1260) (DF) (ttl 64, id 12449, len 1280, bad ip cksum 0! -> a6d7)

And so we can see that the TCP packet port numbers of 2400 and 56180 translate to 0x0960 and 0xDB74 which is what we see in the packet.

Building the filter

So now that we understand what we are looking at, writing a filter to match what we want is fairly trivial. I'm looking for traffic from my phone's IP (which at the time was 192.168.197.26) to port 22067. Converting the IP and port into hex gave me the values 0xC0A8C51A and 0x5633 and I know the offset and lengths so there we go, ip[32:4] = 0xC0A8C51A and ip[42:2] = 0x5633.

Conclusion

While there are other, friendlier tools like Wireshark to capture, parse and search network traffic tcpdump has been my first go-to for literally decades now when I have to troubleshoot complex network problems. The filter language is extremely powerful if you know what you are looking at and the fact that it's a command line utility on almost every Linux or BSD box ever made means you almost always have it to hand. You can even use it to capture packets to files that you can load in Wireshark later but tcpdump with a good filter expression is invaluable as you can watch an application's behavior in real time as you use it from wherever it is that you can get in between the application and it's destination.

Further, understanding the workings of the network that we all rely on day in and day out is important if you're going to build or maintain anything that uses, or is part of the Internet.