Temporary page for testing. Not linked from any menu and attempted to be hidden from the bots ;-)

The core HTTP specifications don’t define their own means of providing integrity. Although “hop-by-hop” protocols underneath HTTP, such as TLS or QUIC, provide integrity for HTTP message transactions, it isn’t enough. HTTP powers massively distributed systems where intermediaries (aka proxies) are first-class citizens. Coupled with that are the powerful semantics of HTTP resources and representations.

This post explores the past, present, and future of HTTP extensions that allow for HTTP Integrity. We’ll cover core concepts, fields such as Content-MD5, Digest, Content-Digest, Repr-Digest, Unencoded-Digest, and more. However, some foundational HTTP understanding needs to be established before we can start. If you just want to jump into an interactive demo playground see my HTTP Integrity Demos page.

What Do We Mean By Integrity?

This is one of the most important things to define before we get into harder topics. I really like Wikipedia’s opening sentence in the Data Integrity article:

Data integrity is the maintenance of, and the assurance of, data accuracy and consistency over its entire life-cycle[1] and is a critical aspect to the design, implementation, and usage of any system that stores, processes, or retrieves data.

For the purposes of this post, we can assume data integrity relates to ensuring the bytes of a $thing are exactly the bytes that are expected. Issues such as data corruption, software program errors, or malicious modification could cause unexpected changes that damage integrity.

When it comes to HTTP, what the $thing is is often more complicated than your first intuition might suggest. This complexity arises from HTTP Semantics defined in RFC 9110.

In essence, HTTP semantics define concepts like actors (clients, servers, proxies), request/response messages (and their sub components like methods, status codes, data, and metadata), and resources and representations. If you are not familiar with this, I suggest reading both the RFCs (RFC 9110 et al) and related resources such as MDN’s HTTP pages (caveat: MDN can be incorrect, more on that later in the post. If ever in doubt defer to the RFCs).

For the remainder of this blog, I’ll use “HTTP” as shorthand for “HTTP Semantics defined in RFC 9110”.

Finally, deciding what to do if an integrity check fails is often something the upper-layers need to decide. It needs to factor in the threat model. A text file that is corrupted by one-byte may still be readable by a human and pose no security threat. A binary file that has been maliciously swapped out could be a major security threat. HTTP specifications for integrity tend to punt on these matters. Instead, we expect an “application profile” to define the nature of interactions between endpoints, what is acceptable or not, and how to deal with integrity validation failures.

Differences Between Resources and Representations

Section 3.1 of HTTP defines the term “resource”:

The target of an HTTP request is called a “resource”. HTTP does not limit the nature of a resource; it merely defines an interface that might be used to interact with resources. Most resources are identified by a Uniform Resource Identifier (URI), as described in Section 4.

Section 3.2 of HTTP defines the term “representation”.

A “representation” is information that is intended to reflect a past, current, or desired state of a given resource, in a format that can be readily communicated via the protocol. A representation consists of a set of representation metadata and a potentially unbounded stream of representation data (Section 8).

One of HTTP’s super powers is the ability for a single resource, identified by some URI like https://lucaspardue.com/example-resource, to generate multiple different representations. For example, different user agents could send different requests with different language preferences all to the same URI, and my server could return an appropriate translation to them all. This concept is formally summarized by the statement in Paragraph 4 of Section 3.2 of HTTP:

A target resource might be provided with, or be capable of generating, multiple representations that are each intended to reflect the resource’s current state. An algorithm, usually based on content negotiation (Section 12), would be used to select one of those representations as being most applicable to a given request. This “selected representation” provides the data and metadata for evaluating conditional requests (Section 13) and constructing the content for 200 (OK), 206 (Partial Content), and 304 (Not Modified) responses to GET (Section 9.3.1).

Metadata, Data … HTTP Fields and Content

Representation data and metadata are defined Section 8 of HTTP. Metadata is sent in the form of HTTP fields, which can appear in a header or trailer section. while data is sent in the form of HTTP content FKA “message payload” or “payload”.

I’ll insist on using the most up-to-date terms defined in HTTP because it is core to properly understanding the most recent Integrity RFCs. Folks will commonly use alternative terminology based on legacy understanding, domain convention, shorthand phrasing, etc. I can’t police them on that but it doesn’t aid this discussion.

Misusing terms can lead to misunderstanding HTTP semantics and in all likelihood incorrect application of HTTP Integrity. There’s no point trying to use it if the foundations are not set straight from the offset. The road is littered with broken attempts based on an insufficient common understanding of HTTP. The community has fixed that now and it is vital to operate in the present.

I ask you, dear reader, to embrace this blog post and shed any of your past (mis)conceptions of how this stuff all combines together. If I make a mistake, you can feel free to shout at me.

Understanding Representations Through the Lens of Content Negotiation

Let’s explore these concepts through the lens of content negotiation (Section 12 of HTTP). Chiefly, the “Accept-Encoding” and “Content-Encoding” fields.

Accept-Encoding allows an endpoint to express its preference for Content Coding in HTTP messages. For instance, a client can send Accept-Encoding in a request to indicate which content codings are acceptable in a response. If no Accept-Encoding is sent, any content coding is deemed acceptable. A client that sends Accept-Encoding: gzip, deflate, br, zstd indicates support for 4 different types of content coding; a server would pick one and respond e.g., Content-Encoding: gzip. The response message content would be GZIP-compressed, and the client would transparently decode this before processing or rendering.

Imagine a plaintext file named “hello.txt” containing the string “hello world”. A server makes this resource available via the URI “https://example.org/hello.txt". A simple HTTP user agent could access this resource using a GET request such as:

And the server could respond with the simplest selected representation of plaintext with no content coding:

If the client supports GZIP compression, it could advertise this using the Accept-Encoding header. A server could then select the GZIP-compressed representation and serve that as follows:

Comparing the plaintext and GZIP-compressed responses, there are some notable differences.

The Content-Encoding header is added to indicate GZIP. This is a different selected representation. The content itself is the binary output of the GZIP compression, presented as hex for readability; the actual bytes on the wire would be slightly different. This presentation format is used throughout the blog.

The Content-Length is longer in the GZIP example due to standard header and trailer bytes being added. NB: A real GZIP encoder won’t necessarily compress exactly the same (e.g., sometimes they embed a timestamp), hence the exact bytes above are illustrative and could vary across deployments.

Range requests, partial content, and representations

HTTP allows for something called range requests. A client can request part of a representation, indicated by a range.

Using the plaintext example again, we can change the request to ask for just the first 5 bytes of the string “hello world” by sending a Range header field:

The server can satisfy the range request and respond with 206 (Partial Content) and a Content-Range header field:

We can also have issued a range request for the GZIP-compressed representation:

A capable server would respond with the following:

Something important to highlight is that the range request causes the server to select the same representation as the regular request. The only difference is what bytes are sent in the response.

The Long and Winding History of HTTP RFCs

This topic is rich enough for a blog post in its own right. I’d suggest reading my blog post HTTP/3: From root to tip. As one example of the early days, this image shows the origins and refactors of the HTTP/1.x specs.

Image here

IETF RFCs are immutable. If changes to text are required or desired, a new RFC needs to be published that either updates or obsoletes an older one. HTTP/1.1 for instance has been defined as the following RFCs:

• RFC 2068
• RFC 2616
• RFC 7230
• RFC 9112 (latest)

Before we had HTTP/2 or HTTP/3, the RFCs used to intermingle details of general HTTP semantics with version-specific wire format details (how the messages are serialized and sent over networks). The latest family of documents makes this separation much cleaner. Mark Nottingham’s 2022 blog post goes into more detail but to summarize:

• RFC 9110 - Semantics
• RFC 9111 - Caching
• RFC 9112 - HTTP/1.1
• RFC 9113 - HTTP/2
• RFC 9114 - HTTP/3

Hashing and Integrity

Earlier I touched on what is meant by integrity but didn’t discuss methods that help check it. One common approach is to use a digest (often interchangeably called a hash, hash value, checksum, and so on). I won’t go deep into the computer science of hashing in this post, the Hash function Wikipedia article is a good launching point.

A hash function is any function that can be used to map data of arbitrary size to fixed-size values, though there are some hash functions that support variable-length output.

In short, a hash function can run an algorithm over the bytes of a $thing and return a digest value. There are many different hashing functions that have different properties.

For our purposes, ideally we’d like the digest value to be different if even one byte changes. However, that’s not guaranteed for all algorithms; see Hash collisions.

Broadly speaking, HTTP integrity operates by comparing the calculated digest value of locally-available bytes against an expected digest value. By including an expected digest value in a header field, alongside the content bytes, the receiver can detect an issue with the integrity of the bytes. This requires somewhat of a leap of faith that the digest value itself was not corrupted or maliciously changed. More on that topic later in the post.

The road is littered with past attempts to define fields for sending digest values, let’s look at them.

The First Stab and Second Stabs at Integrity: Content-MD5 and Digest

The concept of Content-MD5 was first defined for MIME (i.e., email) in 1993 by RFC 1544, and pretty rapidly obsoleted by the follow up RFC 1864 published in 1995.

The definition of the Content-MD5 HTTP header field was first defined in RFC 2068.

an MD5 digest of the entity-body for the purpose of providing an end-to-end message integrity check (MIC) of the entity-body.

The term “entity-body” is new to this post but it is what we now call HTTP content. To use Content-MD5 you hash the bytes of the content using the MD5 hash function and send the resulting value. For example, reusing the “hello world” string, the MD5 digest value is “XrY7u+Ae7tCTyyK7j1rNww==”. See the example Python code in the expandable section underneath for a simple reproduction (repeated throughout the post):

import hashlib, base64

data = b"hello world"

digest = hashlib.md5(data).digest()
print("Content-MD5 (base64):", base64.b64encode(digest).decode())

Content-MD5 sort of works ok. But it has some sharp edges and it failed to deliver in practice. The most salient description of the issues with Content-MD5 probably comes from RFC 3230 published in 2002:

HTTP/1.1 borrowed Content-MD5 from the MIME world based on an analogy between MIME messages (e.g., electronic mail messages) and HTTP messages (requests to or responses from an HTTP server).

As discussed in more detail in section 3, this analogy between MIME messages and HTTP messages has resulted in some confusion. In particular, while a MIME message is self-contained, an HTTP message might not contain the entire representation of the current state of a resource. (More precisely, an HTTP response might not contain an entire “instance”; see section 3 for a definition of this term.)

And later:

When an HTTP server sends a 206 (Partial Content) response, as defined in HTTP/1.1. The client may form its view of an instance (e.g., an HTML document) by combining a cache entry with the partial content in the message.

And finally:

The Content-MD5 header, of course, implies the use of the MD5 algorithm [15]. Other algorithms, however, might be more appropriate for some purposes. These include the SHA-1 algorithm [12] and various “fingerprinting” algorithms [7]. HTTP currently provides no standardized support for the use of these algorithms.

To condense those three statements, the re-use of Content-MD5 for HTTP had issues due to fundamental differences in how messages are exchanged, especially for range requests. The restriction to only using MD5 was also seen as a negative; even as far back as 1993 it was theorized to be vulnerable to collisions. Subsequent research has proven that. For solely integrity purposes it may be good enough but people like to do better.

It was decided to standardize a new solution to address the existing issues and support the new features. RFC 3230 was written in parallel with efforts to standardize Delta Encoding in HTTP.

RFC 3230 specifies the Digest field as:

The Digest message header field provides a message digest of the instance described by the message.

Digest = “Digest” “:” #(instance-digest)

The instance described by a message might be fully contained in the message-body, partially-contained in the message-body, or not at all contained in the message-body. The instance is specified by the Request-URI and any cache-validator contained in the message.

This definition syntax is ABNF, it means that the value of the Digest header is a list of instance-digests that are defined as:

instance-digest = digest-algorithm “=”

Brief Aside: You may have noticed the terms “entity-body”, “message-body” and “instance”. HTTP terminology has evolved over the years as we’ve gotten more practical deployment usage. RFC 3230 was written in the early 2000s where HTTP was defined by RFC 2616, before our snazzy new terminology. It had to spend a lot of text to define the “instance” concepts on top. Don’t spend any time trying to understand these old terms, RFC 9110 now uses things like “content”, “representation”, and “selected representation”.

Lets first look at a simple example of using Digest to send an MD5 digest value:

import hashlib, base64

data = b"hello world"

digest = hashlib.md5(data).digest()
print("Digest MD5 (base64):", base64.b64encode(digest).decode())

The header value contains the MD5 algorithm identifier “MD5” and the digest value “XrY7u+Ae7tCTyyK7j1rNww==”, which is identical to the value sent in Content-MD5. Pretty boring. We can spice up the example a bit by sending multiple digest values:

import hashlib, base64

data = b"hello world"

md5_digest = hashlib.md5(data).digest()
print("Digest MD5 (base64):", base64.b64encode(md5_digest).decode())

sha256_digest = hashlib.sha256(data).digest()
print("Digest SHA-256 (base64):", base64.b64encode(sha256_digest).decode())

This example uses SHA-256, which was later added to the list of allowed hashing functions by RFC 5843 in 2010. Why? Because the SHA-1 function specced for use in RFC 3230 was also found to be insecure. A lesson in why algorithm agility is important.

This example is very easy but Digest is a lot more powerful, which led to confusion over how to implement it and interoperate with others.

Confusion Over How to Implement Digest

HTTP semantics can be hard for folks to get their head around. In my experience, the developers of the last decade struggled to understand what Digest was trying to achieve. This led to different interpretations of what bytes of the $thing served as the input into the hash function.

Let’s dig into that some more by revisiting our prior example of a range-request for a GZIP-compressed object.

For the Digest header, what do you think the input bytes to the MD5 or SHA-256 algorithm should be? If you guessed “1F 8B 08 00 00”, then you’re wrong!

The “instance” that the Digest applies to is the “selected representation” as we call it now. Recall that a response to a range request might not transfer all of the bytes, but the representation is the same as a non-range-request.

The correct usage of Digest in this case would be as follows:

import gzip, io, hashlib, base64

data = b"hello world"
buf = io.BytesIO()
with gzip.GzipFile(fileobj=buf, mode="wb", mtime=0) as f:
    f.write(data)
full_encoded = buf.getvalue()  # the full GZIP-compressed representation, all 31 bytes
sent = full_encoded[0:5]  # only these 5 bytes are actually transferred

content_md5 = hashlib.md5(sent).digest()
print("Content-MD5 (base64, over the 5 sent bytes):", base64.b64encode(content_md5).decode())

digest_md5 = hashlib.md5(full_encoded).digest()
print("Digest MD5 (base64, over the full 31-byte instance):", base64.b64encode(digest_md5).decode())

Notice that Digest and Content-MD5 carry different MD5 values. Digest applies to the representation (the full 31 bytes of the GZIP-compressed version of “hello world”) but Content-MD5 applies to the content (the first 5 bytes). RFC 3230 explains the distinction, in old terms, as:

the MIC in a Content-MD5 header field applies only to the entity in that message, and not to the entire instance being reassembled

I’ve also experienced, first hand, other confusion on what bytes to input into Digest. One interpretation of the “instance” definition is that it applies to the bytes prior to transforms such as content coding. For instance, it would be calculated over the string “hello world” not the GZIP-compressed 31 bytes. This is incorrect. The purpose of Digest is to support HTTP integrity for different representations. As such, we’d have different digest values for each representation, as follows:

We can even show this further with an example using Brotli compression, a 3rd representation:

import brotli, hashlib, base64

data = b"hello world"
full_encoded = brotli.compress(data)  # the full Brotli-compressed representation
sent = full_encoded[0:5]  # Range: bytes=0-4

print("full bytes:", len(full_encoded))
print("full hex:", full_encoded.hex())
print("sent hex:", sent.hex())

digest = hashlib.md5(full_encoded).digest()
print("Digest MD5 (base64, over the full instance):", base64.b64encode(digest).decode())

The Third Stab: Content-Digest and Repr-Digest

I’d been playing with the Digest field starting around 2017 as part of my work at BBC R&D on multicast HTTP using QUIC. That design required all subscribers to share a symmetric key used for packet protection (or to use a NULL cipher, yuck). We therefore wanted a way to add HTTP integrity, found RFC 3230, and started using it.

It seems as if I wasn’t the only one doing integrity things at that time. Roberto Polli sent an email to the HTTP mailing list in April 2019 titled “Updating Digest header RFC3230 using “selected representation”:

as the Digest header is still used, I’m willing to refresh the old spec

• https://tools.ietf.org/html/rfc3230

using the new definitions of RFC723x (eg. selected representation) and untagling it from delta-encoding.

Is there somebody interested in supporting that effort?

To which I promptly responded as basically “yes”. Although I’d left the BBC to join Cloudflare and work on other things, finding a kindred spirit that felt as strongly about terminology as me was inspiration enough to try and make Digest clearer.

Our first document was published in May 2019 as draft-polli-resource-digests-http-00. This was very much focused on editorial changes. We didn’t want to change the design of the Digest header, just clarify all the text that was stale and/or confusing. That said, we also took the opportunity to make some small improvements:

1. Deprecate MD5 and SHA-1 as insecure.
2. Introduce the ability to send digest values of unencoded representations (i.e. the selected representation before content coding had been applied)

That second feature was first proposed to be implemented by defining new algorithm names. It then evolved into a discussion about a special “id-” prefix (Github issue 885). For instance, the algorithm identifier “ID-SHA-256” would mean using SHA-256 over the string “hello world”. This sounds complicated. It kind of is. The use case was sort of about helping software stacks that transparently removed content coding at one layer to be able to continue benefitting from HTTP integrity. For example, userspace JavaScript running in a browser context would not have access to the GZIP-compressed response bytes and so could not add any verification logic.

The document was adopted by the IETF HTTP Working Group in July 2019. Roberto and I worked on it at a moderate pace over the years (COVID didn’t really help).

In draft 07 (Nov 2021) we decided to punt the unencoded prefix feature thing. Issue 885 gathered quite a lot of discussion including Mark Nottingham saying “That’s really weird”. Punting allowed us, in theory, to keep the update doc minimal. The new feature could be defined in a separate follow up document.

Draft 07 also contained a big change though: the addition of a new Content-Digest field. In the previous section I highlighted that Digest and Content-MD5 operated on different bytes. Between 2019 and 2021 several issues and discussion threads had highlighted the differing schools of thought on what Digest did. See the presentation from the June meeting for more information. Content-Digest was ultimately added and it behaved like Content-MD5 but with the benefit of algorithm agility. To support this, we also created a new registry of algorithm identifiers (changing them to lowercase). Here’s an example, extending the same range-request scenario from before:

import gzip, io, hashlib, base64

data = b"hello world"
buf = io.BytesIO()
with gzip.GzipFile(fileobj=buf, mode="wb", mtime=0) as f:
    f.write(data)
full_encoded = buf.getvalue()  # the full GZIP-compressed representation, all 31 bytes
sent = full_encoded[0:5]  # only these 5 bytes are actually transferred

md5_digest = hashlib.md5(sent).digest()
print("Content-Digest md5 (base64):", base64.b64encode(md5_digest).decode())

The addition of Content-Digest was a net improvement because in June 2014, the definition of HTTP was updated in RFC7231 to state the following.

The Content-MD5 header field has been removed because it was inconsistently implemented with respect to partial responses.

In 2022, it was highlighted that the Digest and Content-Digest fields were still defined using ABNF, rather than newer Structured Fields. This defined a standard set of data types for common use cases, intended to help avoid people needing to write custom per-header parsers. To address this, the Working Group agreed to obsolete the Digest field and replace it with Repr-Digest. A name that would, hopefully, make it abundantly clear that the digest value applied to the representation. Content-Digest and Repr-Digest are now defined as Structured Fields Dictionary types. These are serialized as a comma-separated list of key/value pairs where the key is the digest algorithm and the value is the digest value e.g., Content-Digest: sha-256=:d435Qo+nKZ+gLcUHn7GQtQ72hiBVAgqoLsZnZPiTGPk=:, sha-512=:YMAam51Jz/jOATT6/zvHrLVgOYTGFy1d6GJiOHTohq4yP+pgk4vf2aCsyRZOtw8MjkM7iw7yZ/WkppmM44T3qg==:.

2022 also saw the release of another HTTP update in the form of RFC 9110 et al. We needed to do a fair amount of editorial work to make all the terminology consistent but it was a great improvement very aligned with Roberto’s original goal for the document. RFC 9110 also went a bit further with killing Content-MD5:

IANA has updated the “Content-MD5” entry in the new registry to have a status of ‘obsoleted’ with references to Section 14.15 of [RFC2616] (for the definition of the header field) and Appendix B of [RFC7231] (which removed the field definition from the updated specification).

In February 2024, the work was published as RFC 9530 with the title “Digest Fields”. It formally obsoletes the Digest field so you should never use that again.

Here’s the range-request scenario again, presented with RFC-correct field names and values:

import gzip, io, hashlib, base64

data = b"hello world"
buf = io.BytesIO()
with gzip.GzipFile(fileobj=buf, mode="wb", mtime=0) as f:
    f.write(data)
full_encoded = buf.getvalue()  # the full GZIP-compressed representation, all 31 bytes
sent = full_encoded[0:5]  # only these 5 bytes are actually transferred

repr_digest = hashlib.sha256(full_encoded).digest()
print("Repr-Digest sha-256 (base64):", base64.b64encode(repr_digest).decode())

content_digest = hashlib.sha256(sent).digest()
print("Content-Digest sha-256 (base64):", base64.b64encode(content_digest).decode())

For the avoidance of doubt, and to reinforce the text in the RFC, this example shows:

1. A resource that is a text file with the string value “hello world”
2. The outcome of content negotiation selecting a GZIP-compressed representation that is 31 bytes
3. The outcome of a range request that transfers the first 5 bytes of the representation
4. The SHA-256 algorithm taking bytes from (2) for Repr-Digest, and (3) for Content-Digest

The Repr-Digest applies to the full 31 bytes of the GZIP-compressed representation. The Content-Digest applies to the first 5 bytes of the GZIP-compressed representation. RFC 9530 defined no special means to communicate the digest of an unencoded selected representation; the only way to do that would be to select it explicitly (e.g, send a request with Accept-Encoding: identity).

Validation of Integrity

As noted previously in the section “What Do We Mean By Integrity?”, RFC 9530 purposefully omits any specification about what to do with integrity validation failures.

However, there are common considerations for applications to make. The ability to send multiple digests opens up some possibilities:

1. If a peer supports multiple digest algorithms at the same preference (see below), do you spend CPU to calculate all digest values?
2. If you receive multiple digest values do you validate all of them? If so, and only one fails, what should happen?
3. Is a validation failure treated as “fail open”, “fail closed”, etc.

Why is Sending a Repr-Digest Useful?

You may have got this far and understood the concepts but still not any wiser why Repr-Digest is useful. That’s fair, it can be confusing.

One example use case is resumable or parallel downloads. Imagine you were downloading parts of a large file by issuing multiple range requests. Content-Digest would allow the validation of the bytes that were transferred in the response, great. However, it wouldn’t allow for understanding if the bytes were pieced back together correctly.

Repr-Digest doesn’t even need any content. A client can send a HEAD request and obtain one. This is one of the lesser-known superpowers.

This works because HEAD elicits a response as if the request were GET, except with no content. In other words, we can select a representation without having to spend any bandwidth serving it.

Integrity for HTTP Requests

All the examples so far have concentrated on integrity for HTTP responses. However, requests can also benefit.

Let’s use a very complicated example that takes inspiration from RFC 9530’s Figure 8: POST request. The client is sending the JSON object {"author": "Camilleri"} with GZIP-compression.

import gzip, io, hashlib, base64

data = b'{"author": "Camilleri"}'
buf = io.BytesIO()
with gzip.GzipFile(fileobj=buf, mode="wb", mtime=0) as f:
    f.write(data)
compressed = buf.getvalue()

print("bytes:", len(compressed))
print("hex:", compressed.hex())

digest = hashlib.sha256(compressed).digest()
print("sha-256 (base64):", base64.b64encode(digest).decode())

Content-Digest and Repr-Digest work exactly the same way for requests as they do for responses. The values are identical because they are calculated over the 43 GZIP-compressed bytes.

In this mock scenario, the server actions the POST request by processing it and creating a new resource at the URI “https://foo.example/authors/123". This is indicated by a 201 (Created) response status that includes Content-Location and Location fields. The response also includes the plaintext selected representation of that resource {"id": "123", "author": "Camilleri"}:

import hashlib, base64

data = b'{"id": "123", "author": "Camilleri"}'

digest = hashlib.sha256(data).digest()
print("sha-256 (base64):", base64.b64encode(digest).decode())

Once again Content-Digest and Repr-Digest are identical.

I could continue to add complicated examples to illustrate the concepts. RFC 9110 describes Partial PUT:

Some origin servers support PUT of a partial representation when the user agent sends a Content-Range header field (Section 14.4) in the request, though such support is inconsistent and depends on private agreements with user agents. In general, it requests that the state of the target resource be partly replaced with the enclosed content at an offset and length indicated by the Content-Range value, where the offset is relative to the current selected representation.

In such a case, the partial PUT’s Content-Digest and Repr-Digest would differ. Calculating the values is left as an exercise for the reader.

Integrity in Trailers

HTTP allows for trailer sections:

Fields (Section 5) that are located within a “trailer section” are referred to as “trailer fields” (or just “trailers”, colloquially). Trailer fields can be useful for supplying message integrity checks, digital signatures, delivery metrics, or post-processing status information.

Bingo. We can send Content-Digest or Repr-Digest in trailers for integrity as described.

One use case for this is dynamic-content generation. A sending endpoint can avoid needing to buffer bytes in order to calculate the digest value to stick in the headers. Instead, deffering the sending to the trailers (even if digest calculation happens in parallel to the content generation). One caveat is that the receiver might need to buffer the full set of bytes before it can validate the integrity and consume them. A potential way to work around that is some form of incremental integrity. Martin Thomson and Jeffrey Yasskin once proposed Merkle Integrity Content Encoding that might be able to satisfy that need.

Algorithm Agility and Integrity Preference Fields

So far I’ve taken care to omit any mention of Integrity Preference Fields. RFC 3230 added algorithm agility by allowing multiple different algorithms to be used to generate values for the Digest field. Hashing is not 100% free. There can be a cost to generate and validate digest values. Generating a digest with an algorithm that can’t be validated by the recipient is a waste.

RFC 3230 defined the Want-Digest field as:

The Want-Digest message header field indicates the sender’s desire to receive an instance digest on messages associated with the Request- URI.

Want-Digest = "Want-Digest" ":"
                 #(digest-algorithm [ ";" "q" "=" qvalue])

The idea being that a client could, for example, send something like Want-Digest: MD5;q=0.3, sha;q=1 in a request and the server would understand the client preferred the “sha” algorithm and generate a Digest response header field using only that algorithm.

In the 5 years it took to author RFC 9530, neither Roberto or I could find anyone that seemed to have used or rely on Want-Digest. I can take that to mean that its practical deployment on the Internet is next-to-none. However, to maintain feature parity the Want-Content-Digest and Want-Repr-Digest fields are defined in Section 4. Even if we don’t think anyone will use them.

HTTP Signatures and Digests

Observant readers will have noticed that Content-Digest and Repr-Digest are sent in HTTP messages that could be affected by corruption or malicious modification. That sort of defeats the point right?

One mitigation is defense in depth - lower layers like TLS or QUIC can protect each hop, and digest fields can add some end-to-end protection. That’s somewhat complementary. But it doesn’t completely protect from manipulation at the points between hops. Anyone that has access to modify an HTTP message could change the content and the headers and cause validation of the “bad thing” to succeed, which could be disastrous.

Fortunately, in parallel to the work on RFC 9530 that could protect content and representation integrity, a method to protect HTTP metadata was being standardized in the form of “Signatures”

This wasn’t a new concept. Going back to my work in BBC R&D, we’d identified the need and a potential solution. The multicast QUIC draft described how the non-standard “Cavage Signatures” could achieve it.

Signatures are a complex topic that deserve a long blog post of their own by the relevant experts (i.e., not me). I’ll try to summarize some interesting things though.

draft-cavage-http-signatures-00 dates back to May 2013. It defined a Signature header field that could communicate a digital signature calculated over other HTTP fields. Use of “secret” asymmetric keys would help protect against modifications; only holders of the private keys could legitimately change the signature. Receivers would access a public key from somewhere (potentially identified by an in-band key identifier, or out-of-band) and use that for signature validation. Any modification to the headers would cause a validation failure and trigger a choice of failure mode.

There have been several other similar approaches to HTTP signatures over the years. Some reusing the Signature name with different details on how to calculate or process the values. In 2019, Justin Richer presented a talk at the IETF 106 DISPATCH and SECDISPATCH sessions. Soon after, the HTTP WG adopted draft-ietf-httpbis-message-signatures-00, authored by Justin Richer, Annabelle Backman, and Manu Sporny. This document superseded draft-cavage-http-signatures and draft-richanna-http-message-signatures. In February 2024, the Signature was published as RFC 9421.

Since Signatures and Digest are both heavily grounded in modern HTTP semantic terminology, they fit hand in glove. Digests help protect data by putting information in metadata; Signatures help protect metadata. The following example of both working together is taken from Section 2.4 of RFC 9421, with some modification:

NOTE: '\' line wrapping per RFC 8792

POST /foo?param=Value&Pet=dog HTTP/1.1
Host: example.com
Date: Tue, 20 Apr 2021 02:07:55 GMT
Content-Digest: sha-512=:WZDPaVn/7XgHaAy8pmojAkGWoRx2UFChF41A2svX+T\
  aPm+AbwAgBWnrIiYllu7BNNyealdVLvRwEmTHWXvJwew==:
Content-Type: application/json
Content-Length: 18

{"hello": "world"}

HTTP/1.1 503 Service Unavailable
Date: Tue, 20 Apr 2021 02:07:56 GMT
Content-Type: application/json
Content-Length: 62
Content-Digest: sha-512=:0Y6iCBzGg5rZtoXS95Ijz03mslf6KAMCloESHObfwn\
  HJDbkkWWQz6PhhU9kxsTbARtY2PTBOzq24uJFpHsMuAg==:
Signature-Input: reqres=("@status" "content-digest" "content-type" \
  "@authority";req "@method";req "@path";req "content-digest";req)\
  ;created=1618884479;keyid="test-key-ecc-p256"
Signature: reqres=:dMT/A/76ehrdBTD/2Xx8QuKV6FoyzEP/I9hdzKN8LQJLNgzU\
  4W767HK05rx1i8meNQQgQPgQp8wq2ive3tV5Ag==:

{"busy": true, "message": "Your call is very important to us"}

There’s a lot in that example and I won’t even try to explain it. One cool thing is that "content-digest";req means that the Content-Digest of the request was used as input to the signature calculation, helping to tie together the request/response exchange and providing the possibility for the client to detect something had been manipulated before server processing.

To learn more about signatures, I recommend the excellent https://httpsig.org/ site.

The relationship between HTTP Integrity and Subresource Integrity

You may be familiar with something called Subresource Integrity (SRI), defined by the W3C:

This specification defines a mechanism by which user agents may verify that a fetched resource has been delivered without unexpected manipulation.

This sounds quite familiar but it works differently from HTTP Integrity. Early drafts of RFC 9530 contained some text to explain those differences but we decided to drop it; RFCs work best when they focus on describing what the protocol is and how it works, rather than trying to explain the world around the protocol. We dropped the text in September 2021 in this PR, which included the comment “[Lucas] will publish that under a blog post”. Well the time has finally come…

Rather than copy the text verbatim, I’ll point you to it in draft 05 Section 9.

What I think is interesting to highlight is that SRI’s model was focused on a “first party” declaring a digest value for a $thing that would be fetched from a “third party”. It only works for responses. For example, a website on example.org that loaded some JavaScript from example.com could use SRI via the integrity attribute:

<script src="https://example.com/example-framework.js"
        integrity="sha384-Li9vy3DqF8tnTXuiaAJuML3ky+er10rcgNR/VqsVpcw+ThHmYcwiB1pbOxEbzJr7"
        crossorigin="anonymous"></script>

This allows example.org to make a snapshot of the digest of “example-framework.js” so that any changes made by example.com could be detected. In contrast, HTTP integrity relies on sending digest values in the messages themselves (i.e., the response served by example.com); example.org wouldn’t have the means to make an assertion on the HTTP integrity.

Another difference is that SRI doesn’t consider HTTP content coding. Per Section 3.1 of SRI:

For example, given a script resource containing only the string alert(‘Hello, world.’);, an author might choose SHA-384 as a hash function. H8BRh8j48O9oYatfu5AZzq6A9RINhZO5H16dQZngK7T62em8MUt1FLm52t+eX6xO is the base64 encoded digest that results.

In order to validate SRI, any response Content-Encoding needs to be decoded. That means CPU needs to be spent decoding something that might have already been incorrect or corrupted. That’s an ok trade-off, by using the digest value of the unencoded object the integrity attribute will work no matter what HTTP representation is served; this avoids proliferation of digest values in the integrity field.

One of the downsides to SRI, however, is that it is fragile. Resources are often updated. Requiring example.org to update the value of the integrity field in its HTML every time that example.com makes a change doesn’t scale. And not updating the value would cause SRI validation to fail. The spec is more proscriptive about SRI validation and defines Integrity-Policy and Integrity-Policy-Report-Only fields to let websites decide how to handle failures. The worst case scenario though is that a change to the third party resource (e.g., to fix a bug or security issue) causes a validation failure and the website fails hard.

Although SRI is nice in theory, its fragility makes it risky to use for its intended purposes in practice. Mike West has been leading a new initiative called Signature-based Integrity to keep the intent but achieve it in a different way.

In a nutshell, rather than embed the digest itself, example.org can state the key that would be used to sign the digest. This harkens back to the Signature and Digests section. The first party is able to assert its trust in the third-party via the key and assuming the key is managed properly, all resources signed by that key can be trusted. To achieve this, the integrity field is tweaked slightly to something like:

<script src="https://example.com/example-framework.js"
        integrity="ed25519-[base64-encoded-public-key]"
        crossorigin="anonymous"></script>

Then responses from example.com use HTTP Signatures and Digests to carry the actual information that can be used for validation. There’s one wrinkle though; SRI depends on the unencoded bytes of $thing but both Content-Digest and Repr-Digest are calculated over encoded bytes…

Taking things further with Unencoded Digest

The early drafts of RFC 9530 had the capability for sending digests of unencoded bytes. However, if you recall we punted on that and dropped it. Mike’s Signature-Based Integrity was the use case we’d been waiting for.

Having learned a lot during the standards process of RFC 9530, we decided against using a special “id-” algorithm prefix. It would be clearer to define a new header with explicit definitions. draft-ietf-httpbis-unencoded-digest was adopted by the HTTP Working Group in July 2025 and is now with the RFC Editor awaiting publication. It defines a new Unencoded-Digest header that works almost identically to Repr-Digest except that the bytes of $thing are not encoded.

Here’s an example, using the GZIP-compressed string “hello world”:

import gzip, io, hashlib, base64

data = b"hello world"
buf = io.BytesIO()
with gzip.GzipFile(fileobj=buf, mode="wb", mtime=0) as f:
    f.write(data)
full_encoded = buf.getvalue()  # the full GZIP-compressed representation, all 31 bytes

content_digest = hashlib.sha256(full_encoded).digest()
print("Content-Digest sha-256 (base64):", base64.b64encode(content_digest).decode())

repr_digest = hashlib.sha256(full_encoded).digest()
print("Repr-Digest sha-256 (base64):", base64.b64encode(repr_digest).decode())

unencoded_digest = hashlib.sha256(data).digest()
print("Unencoded-Digest sha-256 (base64):", base64.b64encode(unencoded_digest).decode())

We can extend this example, so that all digest field are different, by using our favorite range request:

import gzip, io, hashlib, base64

data = b"hello world"
buf = io.BytesIO()
with gzip.GzipFile(fileobj=buf, mode="wb", mtime=0) as f:
    f.write(data)
full_encoded = buf.getvalue()  # the full GZIP-compressed representation, all 31 bytes
sent = full_encoded[0:5]  # Range: bytes=0-4 addresses the *compressed* stream

content_digest = hashlib.sha256(sent).digest()
print("Content-Digest sha-256 (base64, over the 5 sent bytes):", base64.b64encode(content_digest).decode())

repr_digest = hashlib.sha256(full_encoded).digest()
print("Repr-Digest sha-256 (base64, over the full 31-byte instance):", base64.b64encode(repr_digest).decode())

unencoded_digest = hashlib.sha256(data).digest()
print("Unencoded-Digest sha-256 (base64, over the 11-byte plaintext):", base64.b64encode(unencoded_digest).decode())

MDN Pages on Content-Digest and Repr-Digest Contain Errors

I mentioned MDN at the top of the blog post. It came to my attention last week that it has pages for Content-Digest and Repr-Digest. Unfortunately, at the time of writing this blog, the MDN pages contain some fundamental errors and explain the concepts incorrectly.

I’ve had people contact me asking if I made a mistake explaining HTTP integrity since it doesn’t agree with what MDN says. There is always a chance I make a mistake, this post probably has some of its own, but in this case both Roberto and I have reviewed MDN and agree it’s a bit wrong. For instance, there is an example called “HTTP responses where Repr-Digest and Content-Digest diverge” which is invalid. It states:

A server may compress the content for sending. In this case Content-Digest will depend on the Content-Encoding, and will therefore have a different value to the Repr-Digest header in a response:

Readers of the RFC and this blog should now understand why a compressed response that sends the full bytes would have identical Content-Digest and Repr-Digest values in such an example.

There’s no blame to make. HTTP semantics are hard to get right, I’ll be opening up some issues to help get the MDN articles fixed as soon as this blog is published. I’ll update the blog once that happens.

A Digest for PATCHing?

More recently, as part of my work helping the standardization of Resumable Uploads for HTTP some discussion around the PATCH method came up. I posted the following to the HTTP mailing list:

Imagine there’s an existing resource of X bytes, with a client wishing to use PATCH to append Y bytes to it. The server accepts the request, performs the patch, and the resource is now X+Y bytes.

The client initiating the request can indicate the digest(s) of the patch document: content, repr, or identity. The server can indicate the digests(s) related to the resource and/or transfers related to it. This could be prior to or after patching. (ETags also play a role, if aspects of RFC 5789 are in play [1]).

However, there’s no digesty way for a client to articulate what it might expect the outcome of the patch to be. In other words, it can’t say “If you apply patch document with bytes Y the hash of X+Y is foo. If it’s not foo, the operation failed”. This seems very niche and I don’t have a personal use case. So throwing it out to the list in case there’s other opinions.

Apart from me accidentally referring to Unencoded-Digest as “identity” hopefully readers now understand what this text is talking about. I’ve written up a proposal for a Patched-Digest in draft-pardue-httpbis-patched-digest that would work a little something like the following.

Let’s use the familiar /hello.txt resource that is the string “hello world”. Let’s say the client wants to append the string " again” to the resource so that it becomes “hello world again”. A PATCH request would support that, the request content is a “patch document” and a server that understands it would apply the document in a patch operation. However, if the resource on the server had somehow changed, appending data to it could cause a invalid result.

Today, a client could make a PATCH request that depends on the pre-condition of “/hello.txt” via an If-Match header field.

In our example, the client does a HEAD request to obtain the server-controlled ETag value.

Now it can use ETag as a pre-condition:

import hashlib, base64

patch_document = b" again"

digest = hashlib.sha256(patch_document).digest()
print("Content-Digest sha-256 (base64):", base64.b64encode(digest).decode())

The Content-Digest in the request relates to the 6-byte patch document " again”.

The If-Match precondition is satisfied and the server applies the patch before responding with an updated ETag and Repr-Digest:

import hashlib, base64

data = b"hello world again"

digest = hashlib.sha256(data).digest()
print("Repr-Digest sha-256 (base64):", base64.b64encode(digest).decode())

In order for the client to understand if the patch operation worked as intended, it would need to compare the received Repr-Digest against its own expectations. That’s ok but it means the server has already mutated the resource. If the client detects the operation did not succeed as expected, it’s not clear what the client should do next. Possibly ask the server to DELETE the resource and start over?!

Patched-Digest is intended to allow for front-loading a bit of the work. The server can stage the patch operation and return a success or fail response depending on the outcome. This could, potentially, catch corruption cases where the resource is changed and gets out of sync with the ETag.

Redo’ing the PATCH with Patched-Digestwould look like:

import hashlib, base64

expected_result = b"hello world again"

digest = hashlib.sha256(expected_result).digest()
print("Patched-Digest sha-256 (base64):", base64.b64encode(digest).decode())

Say, hypothetically, the first byte of the resource had been corrupted to be “jello world”, the server could detect the patch operation failure and serve an error:

import hashlib, base64

expected_result = b"hello world again"
expected_digest = hashlib.sha256(expected_result).digest()
print("Client's Patched-Digest sha-256 (base64):", base64.b64encode(expected_digest).decode())

actual_result = b"jello world again"  # patch applied to the corrupted resource
actual_digest = hashlib.sha256(actual_result).digest()
print("Server's computed sha-256 (base64):", base64.b64encode(actual_digest).decode())

It’s early days for the Patched-Digest proposal. It’s not adopted by the Working Group and may not be. Time will tell if we get a new header.

Problem Types for Digests

In the previous section, I presented an example hypothetical failure case for a non-standard proposal. In that example the server responds with an HTTP Problem Type.

But back in 2024, Marius Kleidl identified some real-world situations where communicating digest validation failures could be useful. Marius, Roberto and I spun up draft-ietf-httpapi-digest-fields-problem-types and worked to standardize it in the HTTP API Working Group. That document is now also with the RFC Editor and will be published soon.

Closing Remarks

The road to HTTP integrity is littered with past attempts. I wouldn’t go so far as to call them failures. Nor would I go so far as to claim the most recent Content-Digest, Repr-Digest or Unencoded-Digest are major successes. I’ve been tinkering in the area of HTTP integrity for about 10+ years now and I still have to double-check everything I say or do because getting it wrong has ramifications.

Understanding core HTTP semantics can be difficult; it takes a time investment, and mistakes or assumptions are easy to make. The current documents I’ve had a hand in have had a significant amount of input from various people in the HTTP community and I’m very grateful for the knowledge or experience they have shared. We only succeed together. I’m hopeful that we can help fine-tune the MDN pages to be a bit more correct and ensure there are good, diverse, resources that help communicate with the world.

I’ve made the HTTP Integrity Demos page in order to try and add another dimension to understanding digests. I’ve already gotten some great feedback I still need to action. I expect it to be a good companion to this blog post, so please do check it out and let me know if there’s things I can improve.

Note on AI usage: the prose in this blog was written first-hand by me, unless, of course, it is a quote. HTTP examples and Python code samples were prepared by Claude under my guidance and verified by me; mistakes may exist.