Signing HTTP Messages
There’s a new draft in the HTTP working group that deals with signing HTTP messages of all types. Why is it here, and what does that give us?
HTTP is irrefutably a fundamental building block of most of today’s software systems. Yet security and identity need to be layered alongside HTTP. The most common of these is simply running the HTTP protocol over an encrypted socket using TLS, resulting in HTTPS. While this is a powerful and important security component, TLS works only my protecting the stream of bits in transit. It does not allow for message-level and application-level security operations. But what if we could sign the messages themselves?
While it is possible to wrap the body of a request in a cryptographic envelope like JOSE or XML DSig, such approaches force developers to ignore most of the power and flexibility of HTTP, reducing it to a dumb transport layer. In order to sign a message but keep using HTTP as it stands, with all the verbs and headers and content types that it gives us, we will need a scheme that allows us to add a detached signature to the HTTP message. The cryptographic elements to the message can then be generated and validated separately from the request itself, providing a layered approach.
There have been numerous attempts at creating detached signature methods for HTTP over the years, one of the most famous being the Cavage draft which itself started as a community-facing version of Amazon’s SIGv4 method used within AWS. There were several other efforts, and all of them incompatible with each other in one way or another. To address this, the HTTP Working Group in the IETF stepped up and took on the effort of creating an RFC-track standard for HTTP message signatures that could be used across the variety of use cases.
As of the writing of this post, the specification is at version 04. While it’s not finished yet, it’s recently become a bit more stable and so it’s worth looking at it in greater depth.
Normalizing HTTP
As it turns out, the hardest part of signing HTTP messages isn’t the signing, it’s the HTTP. HTTP is a messy set of specifications, with pieces that have been built up by many authors over many years in ways that aren’t always that consistent. A recent move towards consistency has been the adoption of Structured Field Values for HTTP. In short, structured fields allow HTTP headers to house simple, non-recursive data structures with unambiguous parsing and deterministic serialization. These aspects made it perfect for use within the HTTP message signatures specification.
Previous efforts at HTTP message signing concentrated on creating a signature around HTTP headers, and the current draft is no exception in allowing that. On top of that, the current draft also allows for the definition of specialty fields that contain other pieces of constructed information not found in the headers themselves. These covered components are identified and combined with each other into a signature input string. To this string is added a field that includes all of the input parameters to this signature. For example, let’s say we want to sign parts of this HTTP request:
POST /foo?param=value&pet=dog HTTP/1.1
Host: example.com
Date: Tue, 20 Apr 2021 02:07:55 GMT
Content-Type: application/json
Digest: SHA-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=
Content-Length: 18
{"hello": "world"}We choose the components we want to sign, including the target of the request and a subset of the available headers, and create the following signature input string:
@request-target": post /foo?param=value&pet=dog
"host": example.com
"date": Tue, 20 Apr 2021 02:07:55 GMT
"content-type": application/json
"digest": SHA-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=
"content-length": 18
"@signature-params": ("@request-target" "host" "date" "content-type" "digest" "content-length");created=1618884475;keyid="test-key-rsa-pss"With a given HTTP message and a set of input parameters determining which parts of the message are covered with a signature, any party can re-generate this string with a reasonable level of success. Unsigned headers can be added to the message by intermediaries without invalidating the signature, and it’s even possible for an intermediary to add its own signature to the message on the way through — but we’ll get more into that advanced use case in a future post. The result of this is that the signer and verifier will re-create this signature input string independently of each other.
Now that we have a normalized string to sign, how do we actually sign it?
Signing and Verifying Content
Once we have the string, it’s a relatively straightforward matter of applying a key and signature function to the string. Any signature method that takes in and bunch of bytes and spits out a different set of bytes is technically feasible here.
How do the signer and verifier know which algorithm to use on a given method? It turns out that different deployments have drastically different needs in this regard. As a consequence, this is an aspect that is application specific by the specification, with several common methods called out:
- The signer and verifier can both be configured to expect only a specific algorithm, or have that algorithm identified by some aspect external to the protocol.
- The signer and verifier can identify the key used to do the signing and figure out the signature algorithm based on that. If an application’s using JSON Web Keys, the
algfield of the key provides an easy way to identify a signing mechanism. - If the signer and verifier need to signal the algorithm dynamically at runtime, there is an
algfield in the signature parameter set itself that points to a new registry. - And if two or more of these methods are applicable to a given message, the answers all have to match, otherwise something fishy is going on and the signature is invalidated.
Given the above signature input string and an RSA-PSS signing method, we end up with the following Base64-encoded bytes as the signature output:
NtIKWuXjr4SBEXj97gbick4O95ff378I0CZOa2VnIeEXZ1itzAdqTpSvG91XYrq5CfxCmk8zz1Zg7ZGYD+ngJyVn805r73rh2eFCPO+ZXDs45Is/Ex8srzGC9sfVZfqeEfApRFFe5yXDmANVUwzFWCEnGM6+SJVmWl1/jyEn45qA6Hw+ZDHbrbp6qvD4N0S92jlPyVVEh/SmCwnkeNiBgnbt+E0K5wCFNHPbo4X1Tj406W+bTtnKzaoKxBWKW8aIQ7rg92zqE1oqBRjqtRi5/Q6P5ZYYGGINKzNyV3UjZtxeZNnNJ+MAnWS0mofFqcZHVgSU/1wUzP7MhzOKLca1Yg==This gives us a signed object, and now we need to put that into our HTTP message.
Sending Signatures in Messages
The HTTP message signature specification defines two new headers to carry the signature, Signature and Signature-Input. Both of these use the Dictionary construct from the HTTP Structured Field Values standard to carry a named signature.
But first, why two headers? This construct allows us to easily separate the metadata about the signature — how it was made — from the signature value itself. This separation makes parsing simpler and also allows the HTTP message signatures specification to support multiple independent signatures on a given message.
The Signature-Input header contains all the parameters that went into the creation of the signature, including the list of covered content, identifiers for the key and algorithm, and items like timestamps or other application-specific flags. In fact, this is the same value used as the last line of the signature input string, and so its values are always covered by the signature. The Signature header contains the value of the signature itself as a byte array, encoded in Base64. The signer chooses a name for the signature object and adds both items to the headers. The name has no semantic impact, it just needs to be unique within a given request.
Let’s say this signature is named sig1. The signer adds both headers to the request above, resulting in the following signed request.
POST /foo?param=value&pet=dog HTTP/1.1
Host: example.com
Date: Tue, 20 Apr 2021 02:07:55 GMT
Content-Type: application/json
Digest: SHA-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=
Content-Length: 18
Signature-Input: sig1=("host" "date" "content-type");created=1618884475;keyid="test-key-rsa-pss"
Signature: sig1=:NtIKWuXjr4SBEXj97gbick4O95ff378I0CZOa2VnIeEXZ1itzAdqTpSvG91XYrq5CfxCmk8zz1Zg7ZGYD+ngJyVn805r73rh2eFCPO+ZXDs45Is/Ex8srzGC9sfVZfqeEfApRFFe5yXDmANVUwzFWCEnGM6+SJVmWl1/jyEn45qA6Hw+ZDHbrbp6qvD4N0S92jlPyVVEh/SmCwnkeNiBgnbt+E0K5wCFNHPbo4X1Tj406W+bTtnKzaoKxBWKW8aIQ7rg92zqE1oqBRjqtRi5/Q6P5ZYYGGINKzNyV3UjZtxeZNnNJ+MAnWS0mofFqcZHVgSU/1wUzP7MhzOKLca1Yg==:{"hello": "world"}
Note that none of the other headers or aspects of the message are modified by the signature process.
The verifier parses both headers, re-creates the signature input string from the request, and verifies the signature value using the identified key and algorithm. But how does the verifier know that this signature is sufficient for this request, and how does the signer know what to sign in the first place?
Applying the Message Signature Specification
As discussed above, the signer and verifier need to have a way of figuring out which algorithm and keys are appropriate for a given signed message. In many deployments, this information can be gleaned through context and configuration. For example, a key derivation algorithm based on the tenant identifier in the URL can be used to dereference the key needed for a given call. Or an application identifier passed in the body could point to a record giving both the expected algorithm and allowable key material.
In addition to defining a predictable way to determine this, an application of the HTTP message signatures specification also needs to define which parts of the message need to be signed. For example, and API might have very different behaviors based on a Content-Type header but not really care about the Content-Encoding. A security protocol like OAuth or GNAP would require signing the Authorization header that contains the access token as well as the @request-target specialty field.
The HTTP protocol is also designed to allow interception and proxy of requests and responses, with intermediaries fully allowed to alter the message in certain ways. Applications that need to account for such intermediaries can be picky about which headers and components are signed, allowing the signature to survive expected message modifications but protecting against unanticipated changes in transit.
This fundamentally means that no signature method will ever be perfect for all messages — but that’s ok. The HTTP message signature draft instead leans on flexibility, allowing applications to define how best to apply the signature methods to achieve the security needed.
Building the Standard
The HTTP message signatures specification is still a long way from being done. It’s taken in a number of different inputs and many years of collective community experience, and that initially resulted in some major churn in the specification’s syntax and structure. As of version 04 though, the major surgery seems to be behind us. While there will inevitably be some changes to the parameters, names, and possibly even structures, the core of this is pretty solid. It’s time to start implementing it and testing it out with applications of all stripes, and I invite all of you to join me in doing just that.
