SignatureDoesNotMatch when deployed behind haproxy with url rewrite?

[alexlovelltroy]

I'm just getting started with versity and looking to include it with a larger set of applications within a microservice infrastructure. When deploying it alone, the quickstart with s3cmd works fine. However, when deploying it within docker-compose and behind haproxy, I keep getting signature errors when trying to upload files. I've enabled debug flags everywhere, but I still can't figure out what is being signed and why it isn't working.

my guess is that there's a problem that the uri called by the client isn't the same as the one read by versity, but I can't figure out how to tell versity that it is behind haproxy.

I don't know what to check next. Hopefully there's enough information in this post to help me figure out what to change.

haproxy terminates SSL with a self-signed cert at a hostname I reference in /etc/hosts. For this test it was foobar.openchami.cluster

Here's my docker compose snippet:

services:
   versity:
      image: ghcr.io/versity/versitygw:v0.22 
      container_name: versity
      hostname:  versity
      restart: always
      networks:
        - internal
      volumes:
        - ./s3-storage/:/tmp/:rw
      environment:
        - ROOT_ACCESS_KEY=${VERSITY_ROOT_ACCESS_KEY} # Set in .env file
        - ROOT_SECRET_KEY={VERSITY_ROOT_SECRET_KEY} # Set in .env file
      command: -p :7070 posix /tmp # Uses port 7070 by default
      healthcheck:
        test: ["CMD", "pgrep", "versitygw"]
        interval: 30s
        timeout: 10s
        retries: 5

Here's the relevant parts of my haproxy config:

frontend my-frontend
  acl PATH_versity path_beg -i /s3
  use_backend versity if PATH_versity

backend versity
  server versity versity:7070 
  http-request replace-path ^/s3/(.*) /\1

My S3cmd config:

# Setup endpoint
host_base = foobar.openchami.cluster/s3
host_bucket = foobar.openchami.cluster/s3/%(bucket)
bucket_location = us-east-1
use_https = True

# Setup access keys
access_key = redacted-key
secret_key = redacted-secret

# Enable S3 v4 signature APIs
signature_v2 = False

Interestingly, I can make a bucket without issue:

[ochami]🌐 openchami in ~
❯ s3cmd --ca-certs ~/cacert.pem -c s3cmd.foobar mb s3://mybucket-test4
Bucket 's3://mybucket-test4/' created

Most anything else fails with a 403 Signature problem:

[ochami]🌐 openchami in ~
❯ s3cmd --ca-certs ~/cacert.pem -c s3cmd.foobar put cacert.pem s3://mybucket-test4
upload: 'cacert.pem' -> 's3://mybucket-test4/cacert.pem'  [1 of 1]
 643 of 643   100% in    0s    20.90 KB/s  done
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method.

Here's the expanded error with --debug added:

DEBUG: get_hostname(mybucket-test4): foobar.openchami.cluster/s3
DEBUG: ConnMan.get(): creating new connection: https://foobar.openchami.cluster/s3
DEBUG: endpoint path set to /s3
DEBUG: Using ca_certs_file /home/ochami/cacert.pem
DEBUG: Using ssl_client_cert_file None
DEBUG: Using ssl_client_key_file None
DEBUG: httplib.HTTPSConnection() has both context and check_hostname
DEBUG: non-proxied HTTPSConnection(foobar.openchami.cluster, None)
DEBUG: format_uri(): /s3/mybucket-test4/cacert.pem
 643 of 643   100% in    0s    19.36 KB/sDEBUG: ConnMan.put(): connection put back to pool (https://foobar.openchami.cluster/s3#1)
DEBUG: Response:
{'data': b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoe'
         b'sNotMatch</Code><Message>The request signature we calculated does no'
         b't match the signature you provided. Check your key and signing metho'
         b'd.</Message><Resource></Resource><RequestId></RequestId><HostId></Ho'
         b'stId></Error>',
 'headers': {'content-length': '281',
             'content-type': 'text/plain; charset=utf-8',
             'date': 'Wed, 08 May 2024 19:45:42 GMT',
             'etag': '',
             'server': 'VERSITYGW'},
 'reason': 'Forbidden',
 'size': 643,
 'status': 403}

[benmcclelland]

@alexlovelltroy I was able to reproduce the error you are seeing. The issue is the "http-request replace-path" directive. This is causing the host header to change in the requests, and causing the v4 signature check to fail.

Here are the relevant lines from the client (s3cmd) debug output using the command:
s3cmd --debug -c ./s3cfg put ./test s3://testbucket/test

DEBUG: Using signature v4
DEBUG: get_hostname(testbucket): 127.0.0.1:5000/s3
DEBUG: canonical_headers = content-type:application/vnd.debian.binary-package
host:127.0.0.1:5000/s3
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240509T040507Z
x-amz-meta-s3cmd-attrs:atime:1715226672/ctime:1715226659/gid:0/gname:root/md5:1e2886cf64297fd4c87bf17c71a750fe/mode:33188/mtime:1714756471/uid:0/uname:root
x-amz-storage-class:STANDARD

DEBUG: Canonical Request:
POST
/testbucket/test
uploads=
content-type:application/vnd.debian.binary-package
host:127.0.0.1:5000/s3
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240509T040507Z
x-amz-meta-s3cmd-attrs:atime:1715226672/ctime:1715226659/gid:0/gname:root/md5:1e2886cf64297fd4c87bf17c71a750fe/mode:33188/mtime:1714756471/uid:0/uname:root
x-amz-storage-class:STANDARD

content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

We can enable similar debug server side like this:
versitygw -a test -s test --debug posix /tmp/gw
With the relevant signature check debug:

SDK 2024/05/09 04:05:07 DEBUG Request Signature:
---[ CANONICAL STRING  ]-----------------------------
POST
/testbucket/test
uploads=
content-type:application/vnd.debian.binary-package
host:127.0.0.1:5000
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20240509T040507Z
x-amz-meta-s3cmd-attrs:atime:1715226672/ctime:1715226659/gid:0/gname:root/md5:1e2886cf64297fd4c87bf17c71a750fe/mode:33188/mtime:1714756471/uid:0/uname:root
x-amz-storage-class:STANDARD

content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20240509T040507Z
20240509/us-east-1/s3/aws4_request
6a9e691da39b334f8a0c9b509880c3ca6ffe25ad3dcac78ad112fa8a586678ff
-----------------------------------------------------

The client is calculating the request signature with host:127.0.0.1:5000/s3 and the server is getting the request header host:127.0.0.1:5000. This results in a different hash, so will fail the signature check.

To get matching host header you probably need the following in the s3cmd config:

host_base = foobar.openchami.cluster
host_bucket = foobar.openchami.cluster/%(bucket)

And this will require a change to the haproxy config to remove the "http-request replace-path" directive.

[alexlovelltroy]

That's what I suspected, but wasn't able to articulate. Thanks for the clear explanation and taking the time to replicate.

For our use case, the replace-path logic was kinda the point. If there's no way to allow verstity to be deployed such that all urls are expected to prepend host.domain.tld/s3/, we'll have to find another solution.

Is there any way to get versity to expect urls like this?

[benmcclelland]

This is not really a imitation of versitygw, but a limitation of the aws v4 authentication. The s3cmd client is sending a signed request that includes the host header, and then something in the middle is changing the value of this header. This would fail with any service using aws v4 auth. The only way to address this would be to modify the client to not include "host" in the signed headers. In my tests s3cmd including the following in the signed headers (meaning, none of these are allowed to change in transit):
content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class

This article has more details about the singed requests: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

[benmcclelland]

Looking further, I think aws stipulates that host must be part of the signed headers, so this would likely keep causing problems if you were to try to workaround this:

The CanonicalHeaders list must include the following:
 * HTTP host header.
 * If the Content-Type header is present in the request, you must add it to the CanonicalHeaders list.
 * Any x-amz-* headers that you plan to include in your request must also be added. For example, if you are using temporary security credentials, you need to include x-amz-security-token in your request. You must add this header in the list of CanonicalHeaders.

https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

[benmcclelland]

Another possibility might be to route requests based on the scope portion of the Authorization header.

For example in the following sample AWS Authorization header:

AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=34b48302e7b5fa45bde8084f4b7868a86f0a534bc59db6670ed5711ef69dc6f7

The Credential includes the access key (AKIAIOSFODNN7EXAMPLE) and scope (20130524/us-east-1/s3/aws4_request) where the scope is the following:

date.Format(<YYYYMMDD>) + "/" + <region> + "/" + <service> + "/aws4_request"

And for s3, the service is always "s3".