How do HTTP servers figure out Content-Length?

Anyone who has implemented a simple HTTP 1.1 server can tell you that it is a really simple protocol. Basically, it’s a text file that has some specific rules to make parsing it easier.

All HTTP requests look something like this: ¹

GET /path HTTP/1.1\r\n
Host: aarol.dev\r\n
Accept-Language: en,fi-FI\r\n
Accept-Encoding: gzip, deflate\r\n
\r\n

The first line is the “request line”, and it has the requested method, path and HTTP version. The following lines are headers, each terminated with “carriage return” and “line feed” characters. There is also an extra CRLF at the end, to mark the end of the header section. After that, there is the message body which can whatever data you want to send.

Here is a simple Go program to demonstrate this, using a raw TCP socket for the client:

func main() {
	// Setup an HTTP server to respond to the path "/test"
	http.HandleFunc("GET /test", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("Hello"))
		w.Write([]byte(" world!"))
	})
	go http.ListenAndServe("localhost:2024", nil)

	// Connect to the server using TCP
	conn, err := net.Dial("tcp", "localhost:2024")
	if err != nil {
		panic(err)
	}
	// Write the HTTP request (no body, only request line and Host header)
	_, err = conn.Write([]byte("GET /test HTTP/1.1\r\nHost: localhost\r\n\r\n"))
	if err != nil {
		panic(err)
	}

	buf := make([]byte, 1024) // 1 kb
	// Read the response
	n, err := conn.Read(buf)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(buf[:n]))
}

When I tried this, I got the following response:

HTTP/1.1 200 OK
Date: Sun, 06 Oct 2024 14:51:13 GMT
Content-Length: 12
Content-Type: text/plain; charset=utf-8

Hello world!

One interesting thing about this is the header Content-Length, which has a value of 12. That’s exactly the length of "Hello world!" in UTF-8. In the above Go code, writing the response happens in two parts: first we write"Hello", then we write " world!". Notice that we didn’t need to call any function to write the headers, but they are still in the response. In Go’s http package, a status of 200 and the headers are automatically written if w.Write() is called before w.WriteHeader(). Any calls to w.WriteHeader() after that are useless and will output a warning.

Remember that in HTTP, the headers are always written before the body. How is it possible that before writing “Hello” to the connection, the server already knows how long the response will be? What if I wanted to write one “Hello”, and then a thousand exclamation marks after that? Or a million? Does the server need to know how long every response is before sending it? It would mean that every response needs to be kept in memory for the entire duration of the handler.

I wanted to figure this out, and I didn’t have to look too deep. I found this amazing comment in the standard library’s net/http/client.go file. Basically, it says that if the response is small enough to fit into one “chunking buffer”, the length can be calculated very easily, and sent all at once. If the response is bigger than the buffer, it is sent in chunks. What does this mean in practice? I’ve modified the above code to demonstrate it:

func main() {
	http.HandleFunc("GET /test", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("Hello"))
		w.Write([]byte(strings.Repeat("!", 3000)))
	})

	go http.ListenAndServe("localhost:2024", nil)
	// removed all error handling for brevity
	conn, _ := net.Dial("tcp", "localhost:2024")

	conn.Write([]byte("GET /test HTTP/1.1\r\nHost: localhost\r\n\r\n"))

	buf := make([]byte, 1024) // 1 kb
	for {
		conn.SetReadDeadline(time.Now().Add(1 * time.Second))
		n, err := conn.Read(buf)
		if err != nil {
			break
		}

		fmt.Println(string(buf[:n]))
	}
}

The handler now returns “Hello!!!!!!!!!!! …” with 3000 exclamation marks. This is bigger than the configured chunk size, so we can see what happens. This is the response:

HTTP/1.1 200 OK
Date: Sun, 06 Oct 2024 16:43:28 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

800
Hello!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
3bd
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!        
0

There is no Content-Length header in the response now. Instead, we have a Transfer-Encoding: chunked. Our message is being chunked, sent in multiple parts, so that the server doesn’t need to fit the whole thing into memory at once. Clever!

The first line of the response data is now a number “800”. Why 800? As it turns out, the number is actually a hexadecimal number, 0x800 is 2048 in base 10. Similarly, 0x3bd, is 957 in base 10. 2048+957=3005, which is the full length of our message! Sending the length of the message before the actual message is a common method to efficiently transfer unknown lengths of data. It is used in the Redis protocol for example.

Chunked transfer encoding was added in HTTP 1.1.² This means that it’s very old and basically all HTTP servers & clients support it. Sending chunked responses also allows something called “trailers”, which are headers that are sent after the body. In Go, trailers need to be explicitly declared before sending them:

http.HandleFunc("GET /test", func(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Trailer", "My-Trailer")
	w.Write([]byte("Hello"))
	w.Write([]byte(strings.Repeat("!", 3000)))
	w.Header().Add("My-Trailer", "test")
})

This is useful for things like digital signatures, which need to be computed from the response body. In Go’s http handlers, adding trailers will also automatically make the response chunked even if it normally wouldn’t be.

HTTP/2 and HTTP/3 don’t support chunked transfer encoding, as they have their own streaming mechanisms.²

This is one of the many things that happen in the background when using the net/http package in Go. http.ResponseWriter is a remarkably simple interface, having just 3 methods on it. Behind this API is a lot of magic, including Content-Type sniffing and the implicit header writing I mentioned above. It’s great that these things are handled automatically, but in programming I think it’s always important to have an idea of what is happening at the layer below.

Comments? See the discussion on Hacker News.