The hidden trap in Terraform's S3 backend (and why Cloudflare breaks it)

The hidden trap in Terraform's S3 backend (and why Cloudflare breaks it)

#terraform#s3#cloudflare#nginx#seaweedfs#devops

Recently while working on automating my homelab Proxmox VM and LXC container provisioning using my proxmox-utils tooling (which I covered in my post on Proxmox Terraform state isolation and live homelab discovery), I decided to move my Terraform state files onto a local S3 object store. Everything looked straightforward: I had a self-hosted S3 service running, an HTTPS domain (https://s3.example.com) exposed securely via a Cloudflare Tunnel, and valid access keys.

My original S3 backend was MinIO — the long-standing default for self-hosted S3-compatible storage. When I first saw SignatureDoesNotMatch errors in Terraform, my first instinct was to search the MinIO issue tracker for similar reports. That search turned up something unexpected: the MinIO repository had been archived. That sent me looking for alternatives, which I wrote about separately in Life after MinIO: Why I switched to SeaweedFS in my homelab. I eventually settled on SeaweedFS — only to find myself staring at the exact same error again.

I ran aws s3 ls --endpoint-url https://s3.example.com — it listed my buckets instantly. I opened https://s3.example.com in my browser — the XML bucket listing loaded without a single TLS or SSL error.

Then I ran terraform init, and was greeted by this wall of text:

Error: Failed to get existing workspaces: Unable to list objects in S3 bucket "homelab-tf" with prefix "env:/":
operation error S3: ListObjectsV2, https response error StatusCode: 403,
api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.

If the AWS CLI worked and the browser worked, why was Terraform getting rejected with a 403 SignatureDoesNotMatch?

It turned out to be a fascinating journey down the rabbit hole of AWS SDK v2, query parameter encoding, and reverse proxy header mutations.

Clue 1: The difference between aws-cli and Terraform

The first puzzling part was why aws s3 ls succeeded while terraform init failed.

To see what Terraform was actually doing, I enabled verbose debug logging:

export TF_LOG=DEBUG
terraform init

Looking at the raw HTTP trace revealed the exact request Terraform issued during initialization:

GET /homelab-tf?list-type=2&max-keys=1000&prefix=env%3A%2F HTTP/1.1
Host: s3.example.com
Authorization: AWS4-HMAC-SHA256 Credential=.../us-east-1/s3/aws4_request,
SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date,
Signature=...

Notice two critical things happening here:

  1. The Workspace Search Prefix (prefix="env:/"): On every init, plan, and apply, Terraform’s S3 backend automatically checks for existing workspace state files by querying ListObjectsV2 with prefix="env:/" (formatted by the SDK as prefix=env%3A%2F).
  2. The SignedHeaders List: Starting in Terraform 1.6+, HashiCorp upgraded the S3 backend to use AWS SDK for Go v2. Unlike aws-cli (which only signs basic headers like host and x-amz-date), AWS Go SDK v2 explicitly signs accept-encoding, amz-sdk-invocation-id, and amz-sdk-request.

Clue 2: Why Cloudflare Tunnels break AWS Signature V4

AWS Signature Version 4 (SigV4) works by hashing the HTTP method, canonical URI, query parameters, and all headers listed in SignedHeaders. If a single character or header value changes between the client sending the request and the S3 server receiving it, the cryptographic signatures won’t match.

When Terraform sends requests through a Cloudflare Tunnel:

  1. Terraform signs Accept-Encoding: identity.
  2. Cloudflare’s HTTP edge proxy receives the request and automatically alters Accept-Encoding (changing identity ➔ gzip, br) before forwarding the traffic down the tunnel.
  3. The destination S3 server receives Accept-Encoding: gzip, br and recalculates the SigV4 checksum using gzip, br.
  4. Because Terraform signed identity but S3 calculated on gzip, br, S3 rejects the request with SignatureDoesNotMatch.

aws-cli never gets broken by Cloudflare because it does not include accept-encoding in its list of signed headers!

Clue 3: Nginx proxy_pass URI unescaping

As if Cloudflare wasn’t enough, reverse proxies like Nginx Proxy Manager have a second trap for custom S3 endpoints:

If your Nginx location / block uses a proxy_pass directive with a trailing slash (proxy_pass http://backend:8333/), Nginx automatically unescapes URL-encoded characters in $request_uri before forwarding them.

When Terraform sends prefix=env%3A%2F (encoding the slash as %2F), Nginx rewrites it as prefix=env%3A/. S3 receives unescaped / when Terraform signed %2F, triggering another signature mismatch!

What the GitHub Issues Reveal

Digging into the issue trackers for SeaweedFS revealed that I was far from the first person to hit this wall. The signature failure is the result of a subtle bug in how S3 authentication handles proxied HTTPS connections.

SeaweedFS Port Appending in auth_signature_v4.go

Several reports in the SeaweedFS repository — notably SeaweedFS Issue #6761 and SeaweedFS Issue #6839 — highlight a specific issue inside SeaweedFS’s authentication engine (auth_signature_v4.go):

  1. Client Signs HTTPS: When Terraform connects to https://s3.example.com, it signs the request using Host: s3.example.com (omitting port 443 as the implicit default for HTTPS).
  2. Proxy Terminates HTTPS: Nginx Proxy Manager receives HTTPS on port 443 and forwards plain HTTP traffic upstream to SeaweedFS on port 30304.
  3. The isDefaultPort Check: Inside SeaweedFS’s auth_signature_v4.go, the server inspects the incoming request. Because it receives unencrypted HTTP on port 30304 (which is not standard HTTP port 80), SeaweedFS’s isDefaultPort check automatically appends :30304 to the Host header when reconstructing the internal CanonicalRequest!
  4. The Mismatch: Terraform signed Host: s3.example.com, but SeaweedFS calculated its signature on Host: s3.example.com:30304. Result: SignatureDoesNotMatch.

What I Tried (and Why Nothing Worked)

Online forums and documentation suggest a few ways to tackle signature mismatches when proxying S3 traffic. I systematically tested each workaround, only to discover that every proxied approach fell short in a Cloudflare Tunnel + Nginx Proxy Manager setup.

Attempt 1: Disabling Cloudflare Compression

The most common advice for fixing AWS Go SDK v2 errors behind Cloudflare is to create a Configuration Rule in the Cloudflare Dashboard for s3.example.com to Disable Compression.

  • What it fixed: Disabling compression successfully stopped Cloudflare from altering Accept-Encoding: identity to Accept-Encoding: gzip, br.
  • Why it failed: While disabling compression solved the Accept-Encoding header issue, Cloudflare’s edge proxy still stripped or modified AWS SDK v2’s custom telemetry headers (amz-sdk-invocation-id and amz-sdk-request) in transit. Because those headers were included in Terraform’s SignedHeaders list, altering or dropping them still resulted in a SignatureDoesNotMatch rejection.

Attempt 2: Tweaking Nginx Proxy Manager Headers & proxy_pass

Next, I focused on the internal reverse proxy. I added custom Nginx configuration headers (proxy_set_header Host $http_host; and proxy_set_header X-Forwarded-Proto $scheme;) and verified that proxy_pass was not unescaping slashes in $request_uri.

  • What it fixed: It ensured that Nginx preserved the original hostname and preserved encoded URI characters (%2F).
  • Why it failed: When traffic passed through Cloudflare first and then Nginx Proxy Manager second, NPM still failed to reliably route and forward S3 payload signatures across all operations.

Attempt 3: Disabling Cloudflare Proxying (DNS-only / “Grey Cloud”)

Another frequently recommended fix is simply bypassing Cloudflare’s HTTP proxy by setting the DNS record to “DNS Only” (the grey cloud in Cloudflare).

  • Why it was impossible: My homelab sits behind a dynamic residential IP address and relies on a Cloudflare Tunnel (cloudflared) for secure ingress without open inbound firewall ports. In a Cloudflare Tunnel architecture, all ingress traffic must be proxied through Cloudflare’s edge network. You cannot set a Tunnel hostname to “DNS Only.”

Attempt 4: Specifying the Direct S3 Port on the Domain Endpoint

Finally, I tried appending the direct S3 service port to the domain name endpoint (e.g., http://s3.example.com:30304).

  • Why it defeats the purpose: While appending the port number avoids Nginx URI unescaping, it requires direct network connectivity to that high port. That means your client machine must already be connected to your local network or a VPN like Tailscale. But the entire reason for setting up an HTTPS domain endpoint in the first place was to have a single, clean URL accessible anywhere without port numbers or network restrictions. If specifying a port number forces you onto Tailscale anyway, using a domain name adds zero benefit — you might as well use the direct IP address.

The Ultimate Fix: Direct Internal IP over Tailscale

After exhausting every proxied configuration, the verdict was clear: no combination of Cloudflare Tunnels and Nginx Proxy Manager reliably preserves AWS Signature V4 for Terraform’s S3 backend.

The only rock-solid solution that worked end-to-end was bypassing both Cloudflare and Nginx Proxy Manager completely. In my Terraform backend "s3" configuration, I pointed the endpoints.s3 URL directly to the internal S3 server IP address:

terraform {
  backend "s3" {
    bucket                      = "homelab-tf"
    key                         = "network/terraform.tfstate"
    region                      = "main"
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    use_path_style              = true

    endpoints = {
      s3 = "http://192.168.x.x:30304"
    }
  }
}

Because traffic travels directly over Tailscale (or local LAN) to the backend service:

  • Zero header modifications occur in transit.
  • URL-encoded parameters (prefix=env%3A%2F) remain pristine.
  • AWS SDK v2 signatures validate cleanly on every init, plan, and apply.

Summary of Workarounds

If you’re troubleshooting a similar setup, here is how the options stack up depending on your architecture:

Setup / Scenario Workaround Status
Cloudflare Tunnel + NPM + Terraform S3 Direct internal IP over Tailscale / LAN Only fully working solution
NPM Only (No Cloudflare) Custom Nginx headers (proxy_set_header Host $http_host;) & proxy_pass fix May work
Cloudflare DNS (Direct Public IP, No Tunnel) Set Cloudflare DNS record to “DNS Only” (Grey Cloud) May work
Cloudflare Tunnel + S3 CLI / Browser Disable Cloudflare Compression rule Works for aws-cli & browser, fails for Terraform

Wrap-up

Debugging AWS Signature V4 failures across modern SDKs, Cloudflare Tunnels, and reverse proxies can be infuriating because standard tools like aws-cli and web browsers work without a hitch.

Understanding what AWS Go SDK v2 requires in SignedHeaders — and recognizing when proxying adds more complexity than value — saves hours of dead-end troubleshooting. If you’re building a homelab automation pipeline with Terraform, skip the proxy gymnastics and point directly to your internal endpoint. You can check out my open-source wrapper scripts in the proxmox-utils repository, and read the full breakdown of how I structure per-VMID state isolation in my post on Proxmox Terraform state isolation and live homelab discovery.