HTTP integrity demos

The Repr-Digest field communicates a digest calculated over the entire selected representation data. Understanding HTTP representations can be hard. It requires a good understanding of core HTTP semantics. Getting things wrong can be a common source of problems when trying to use digest fields.

Representation data and metadata are defined in Section 8 of HTTP Semantics. One of the most important parts to get your head around, for the purposes of digest fields, is the interplay between representation data and content negotiation (Section 12 of HTTP Semantics). Chiefly, the "Accept-Encoding" and "Content-Encoding" fields.

Accept-Encoding allows an endpoint to express its preference for Content Coding in HTTP messages. For instance, a client can send Accept-Encoding in a request to indicate which content codings are acceptable in a response. If no Accept-Encoding is sent, any content coding is deemed acceptable. A client that sends "Accept-Encoding: gzip, deflate, br, zstd" indicates support for 4 different types of content coding; a server would pick one and respond e.g., "Content-Encoding: gzip". The response message content would be GZIP-compressed, and the client would transparently decode this before processing or rendering.

One of the key points to understand in HTTP is the difference between resource and representation. The above helps illustrate the notion of selected representation. A client issues a request to a resource identified by a URI and HTTP metadata helps select the most appropriate representation of that resource.

Imagine a plaintext file named "hello.txt" containing the string "hello world". A server makes this resource available via the URI "https://example.org/hello.txt". A simple HTTP user agent could access this resource using a GET request such as:

And the server could respond with the simplest selected representation of plaintext with no content coding:

import hashlib, base64

data = b"hello world"

digest = hashlib.sha256(data).digest()
print("sha-256 (base64):", base64.b64encode(digest).decode())

The Repr-Digest above is the SHA-256 hash over the bytes "hello world" because that is the selected representation. However, if the client used content negotiation to select a different representation, Repr-Digest would be different. For example, a more advanced HTTP user agent such as a web browser can support requesting compressed resources to save bandwidth and then transparently decompress them before presenting it to the end user. The URI does not change, only the HTTP request metadata.

If the server supports GZIP compression, it could select the GZIP-compressed representation and serve that as follows:

import gzip, io, hashlib, base64

data = b"hello world"
buf = io.BytesIO()
with gzip.GzipFile(fileobj=buf, mode="wb", mtime=0) as f:
    f.write(data)
compressed = buf.getvalue()

print("bytes:", len(compressed))
print("hex:", compressed.hex())

digest = hashlib.sha256(compressed).digest()
print("sha-256 (base64):", base64.b64encode(digest).decode())

Comparing the plaintext and GZIP-compressed examples, there are some notable differences. The Content-Encoding header is added to indicate GZIP. This is a different selected representation. The content itself is the binary output of the GZIP compression, presented as hex for readability; the actual bytes on the wire would be slightly different. The Content-Length is longer in the GZIP example due to standard header and trailer bytes being added. The Repr-Digest is different because it is calculated over the 31 compressed bytes, not the 11-byte plaintext. NB: A real GZIP encoder won't necessarily compress exactly the same (e.g., sometimes they embed a timestamp), hence the exact bytes above are illustrative and could vary across deployments.

Relationship between digest fields

The relationship between Content-Digest, Repr-Digest, and Unencoded-Digest can also be tricky to understand. Sometimes the values are the same, sometimes they are completely different. Let's reuse the prior examples to illustrate the concepts.

Content-Digest is calculated using the bytes of the message content, i.e., the actual bytes that were sent on the wire. Unencoded-Digest is calculated using the bytes of the selected representation without content coding.

Revisiting our first example that uses plaintext, we see all three digest fields are identical:

However, in the GZIP-compressed response example the Unencoded-Digest differs from the other two. This is because Content-Digest and Repr-Digest are both calculated over the 31 encoded bytes, while Unencoded-Digest is calculated over the 11-byte plaintext:

Range requests, partial content, and representations

One of the remaining brain teasers is how digest fields interplay with range requests or requests like HEAD that elicit a response with no content.

Let's return to the first plaintext example again and make it a range request for the first 5 bytes of the string "hello world" by sending a Range header field:

The server can satisfy the range request and respond with 206 (Partial Content) and a Content-Range header field:

import hashlib, base64

full = b"hello world"
sent = full[0:5]  # Range: bytes=0-4

content_digest = hashlib.sha256(sent).digest()
print("Content-Digest sha-256 (base64):", base64.b64encode(content_digest).decode())

repr_digest = hashlib.sha256(full).digest()
print("Repr-Digest / Unencoded-Digest sha-256 (base64):", base64.b64encode(repr_digest).decode())

There are several notable aspects of the above response. First, the Content-Digest differs from both Repr-Digest and Unencoded-Digest because it is calculated over the substring "hello". Second, even though the response only transfers a part of the selected representation, both Repr-Digest and Unencoded-Digest are calculated using the bytes of the whole representation.

We can extend this further to also consider a range request for a GZIP-compressed response:

A capable server would respond with the following where all digest fields differ:

import gzip, io, hashlib, base64

data = b"hello world"
buf = io.BytesIO()
with gzip.GzipFile(fileobj=buf, mode="wb", mtime=0) as f:
    f.write(data)
full_encoded = buf.getvalue()  # the full GZIP-compressed representation
sent = full_encoded[0:5]  # Range: bytes=0-4 addresses the *compressed* stream

content_digest = hashlib.sha256(sent).digest()
print("Content-Digest sha-256 (base64):", base64.b64encode(content_digest).decode())

repr_digest = hashlib.sha256(full_encoded).digest()
print("Repr-Digest sha-256 (base64):", base64.b64encode(repr_digest).decode())

unencoded_digest = hashlib.sha256(data).digest()
print("Unencoded-Digest sha-256 (base64):", base64.b64encode(unencoded_digest).decode())

Finally, let's look at switching the HEAD method into that last request:

The request still causes the server to select a representation, but the HEAD request means no response content is sent. However, the server sends all the same metadata as if the request were GET, with the exception of Content-Digest. This is one of digest's secret powers: you can transfer metadata about resources and representation data without needing to transfer that data as HTTP content: