Cache-Related HTTP

Cacheable Requests

[RFC 7231] section 4.2.3 defines the HTTP methods GET, HEAD and POST as cacheable, meaning, the responses to such requests can be subject to caching, however, in section 4.3.3 it describes caching of POST requests as „not widely implemented“. All other methods are non-cacheable meaning, their results may not be subject to caching.

It is reasonable for a cache to assume that requests using the methods POST, PUT and DELETE should influence affected cached resources.

„Freshness“ and „Staleness“

A cache considers a resource that it has cached „fresh“ if it has not exceeded its expiration lifetime. If the resource has exceeded expiration, the cache considers the cached version of the resource „stale“. If no lifetime information is available at all, a cache can assign a default value or consider the resource „stale“ immediately.

The expiration lifetime is determined by the cache as follows:

1. If the webserver has issued a response for the resource and if the resource states an expiry datetime with HTTP header Expires or an expiration lifetime with HTTP header Cache-Control, directive max-age (see below) the cache uses this information to determine the lifetime. Since Expires uses absolute times, whereas Cache-Control: max-age=SECONDS does not, max-age is preferable over Expires, because it is not subject to timezone and daylight saving time-shifts. If both Expires and max-age are present, max-age is used. If Cache-Control: s-maxage=SECONDS is present, a public cache prefers it over max-age and Expires.
2. If neither Expires nor max-age are present, but a Last-Modified header is present, caches can be configured to calculate a lifetime by determining the time between last modification and the time the response that is to be cached was received. They may also offer default expiration times for resources that do not state values that would allow for a calculation.

Squid can be configured to determine lifetime as a percentage of time since last modification time. [RFC 7234], section 4.2.2. „Calculating Heuristic Freshness“ states a value of 10 percent as „typical“. It also can be configured to calculate lifetimes as percentages of modification age and assign defaults using the directive refresh_pattern in squid.conf. Different percentages and defaults can be configured for different URL-matching regular expressions. See [Squid 2013 refresh_pattern] for details.

The „Cache-Control“ Header

The HTTP header Cache-Control implements crucial aspects of HTTP caching, stating limits for freshness and expiry (in seconds), demanding re-checks for more recent versions and and even preventing resources from being cached at all.

This header can be part of a request by a user agent and a cache and of the response of a cache and a webserver. It can occur multiple times, and the directives listed below can be concatenated to a comma-separated list.

In an HTTP request, the following Cache-Control header was valid:

Cache-Control: max-age=360, min-fresh=60, no-cache, no-transform

The exampe above let’s a user agent state to a cache that it is willing to accept a cached version of the requested resource if it has not been cached for more than 360 seconds (max-age=360) and will not expire from the cache in less than 60 seconds (min-fresh=60). Furthermore, the cache is instructed to check the webserver for a more recent version of the resource (no-cache), and the client will not accept a cached version of the resource that has been altered by a caching procedure (no-transform, for example an image resource that has been downsampled to a lower resolution).

Available cache control directives for HTTP requests are:

• Cache-Control: max-age=SECONDS
  The cache will not return the cached resource if the last time it was requested from the webserver is more than than SECONDS ago.
• Cache-Control: max-stale[=SECONDS]
  The client indicates that it will accept a cached resource that is past expiry. Optionally the time past the expiry can be limited to SECONDS seconds.
• Cache-Control: min-fresh=SECONDS
  The client indicates that it will not accept a cached resource that will expire in less than SECONDS seconds.
• Cache-Control: no-cache
  If a cache receives this request, it will always check on the webserver for a fresher version of the requested resource. It will only return a cached version of the resource if that check did not result in a fresher version.
• Cache-Control: no-store
  This directive behaves like no-cache above, and additionally, a cache receiving this request will not store the result of the request.
• Cache-Control: no-transform
  If a cache receives this request, it will not return a transformed (downsampled, recoded or otherwise modified) version of a resource.
• Cache-Control: only-if-cached
  If a cache receives this request, it will only return a version of the resource that is cached, and it will not return a version from the webserver. Use of this directive is obscure and rare.

Popular web browsers implement their „forced reload“ feature (usually accessible by holding the Shift-key while pushing the „reload page“ button) with the help of Cache-Control: no-cache, no store.

Available cache control directives for HTTP responses are:

• Cache-Control: public
  A public cache receiving a server response with this directive is allowed to cache the resource.
• Cache-Control: private
  A public cache receiving a server response with this directive is not allowed to cache the resource. A private cache (such as a user agent’s built in cache) is allowed to cache the resource.
• Cache-Control: must-revalidate
  Informs any cache receiving this server response that the resource must be checked with the server at the first request after its expiry time.
• Cache-Control: proxy-revalidate
  Public caches must check this resource with the server at the first request after expiry. Private caches can use this resource past its expiry time.
• Cache-Control: no-cache
  Caches may not serve this resource without checking with the webserver for a more recent version of this resource.
• Cache-Control: no-store
  Caches are not allowed to store this resource, they must make best effort to delete the resource from memory after having served the resource to the client.
• Cache-Control: no-transform
  Caches may not perform transformations (such as recoding or downsampling) on this resource, it must be delivered to clients as it was received by the cache.
• Cache-Control: max-age=SECONDS
  Caches are informed that the resource will expire not later than SECONDS seconds after the request.
• Cache-Control: s-maxage=SECONDS
  Same as max-age but only affects public caches.

The „If-Modified-Since“ Header

Clients that want to minimize data traffic by avoiding transfer of resources that are still fresh in the cache of the client can utilize the request header If-Modified-Since to make their GET requests „conditional“.

If a GET request is issued to a cache with a header If-Modified-Since set to a certain datetime, the cache will check if one of the following conditions is met:

• The cache currently holds a stored version of the requested resource where the Last-Modified header states a date earlier than the time given by If-Modified-Since, or
• the cache currently holds a stored version of the requested resource where the Date header states a date earlier than the time given by If-Modified-Since, or
• the cache currently holds a stored version of the requested resource that was retrieved earlier than the time given by If-Modified-Since.

If one of these conditions is true, the cache should reply with a status code of 304 „Not Modified“ (see below).

The „ETag“ Header

A resource as identified by an URL is rendered by an HTTP server in a specific state that possibly changes over time. A client can request the same resource in different encodings and languages using request headers. A server can decide to present different versions of the resource depending on properties of the client such as geographical location of the requesting IP address or the value of the User-Agent header. After all such transformations have been applied, the result – which would be the content of a reply to a requesting client – is said to be an „entity“ of that resource.

A server can decide to give a resource entity an individual „tag“ in the form of a string of text, which means that every time the state of the resource or any of the described choices of presentation changes, that „entity tag“ will change, too.

Only the webserver side can determine a useful algorithm – if any – that actually derives such a tag from the state and data of the application that produces the served resources. Therefore, there is no standard on how such an entity tag should be computed or what it should look like. However, the following general requirements are stated by [RFC 7232], section 2.3:

• The tag value is surrounded by double quotes.
• The tag value should consist of ASCII visible characters 0x21 and from 0x23 to 0x7E.
• The tag value may not contain the double quote, and it should not contain backslashes.
• If the tag value is considered a „weak“ indicator of the state of a resource, for example because its calculation is restricted to a clock with a precision of 1 second and could not reflect if the resource has changed state several times during a single second, it must be preceeded with „W“ followed by a forward slash „/“.

The following are syntactically valid ETag headers:

ETag: "helloworld ~ 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
ETag: W/"helloworld @ 2020-01-06 08:50:44+01:00"

The „If-None-Match“ Header

Just as with the If-Modified-Since header that is concerned with the age of cached resources, requests can be made conditional with regards to the entity tag of a resource using the If-None-Match header.

Cache-Related HTTP Status Codes

If for a GET or a HEAD request that is conditional using the header If-Modified-Since or If-None-Match, an HTTP server has determined that the requested resource would be available, but the condition is not met (meaning the resource has not been modified since the time stated by the client or the state of the resource has not changed), the server will reply to the client with a status code of 304 „Not Modified.“ Both caching proxies and webservers expose this behavior. A client receiving this response can then proceed to use the resource it already has cached as if it was the body of a 200 „OK“ reply.

If a result of a POST request was identical to a different resource, a server can choose to reply with a status code 303 „See Other“ and a Location header pointing to that other, possibly cacheable resource.

If a condition is not met for a request that has a method other than GET or HEAD, the server must reply with a status code of 412 „Precondition Failed“, POST being the only cacheable method affected by this requirement.

Caching Cookies

[RFC 2109] and its successor [RFC 2965] went to increasing lengths to mandate Cookie-specific mechanisms of cache control and validation, such as dedicated cache-control directives governing the Set-Cookie header and an alternative header Set-Cookie2 to distinguish „public“ from „private“ Cookie information. [RFC 6265], which obsoletes them both, in section 3 mandates that responses that carry Set-Cookie headers can principally be cached and presents no further, cookie-specific cache semantics.

Despite [RFC 6265] not being explicit about this, it can be assumed that the authoritative source of a response that carries private information in the Set-Cookie header should declare the response non-cacheable by the usual means of cache-control and validation.

[Squid 2013 cookies] describes how Squid deals with the caching of responses that have Set-Cookie headers: Respones are stored with their Set-Cookie headers, but upon retrieval by a client (cache hit) they are stripped from the response to the client of the cache.

User Agent Behavior

User agents can facilitate cache-control methods in request headers to provide special-purpose functions:

• A „page reload“ function of a web browser can cause resource requests to be re-issued unconditionally and with a Cache-Control header stating max-age=0, refreshing all locally cached resources.
• A „forced page reload“ can additionally set no-cache and no-store in the resulting resource requests, instructing intermediate caches to revalidate all requested resources and not to serve cached versions. This behavior addresses intermediate caches that ignore or optimistically override age information.
• A „disable cache“ setting can make this behavior permanent for all requests.

The exact behavior of user agents is not standardized; user agents expose implementation-specific differences.

User agents may maintain additional specific caches besides the private resource cache (see the section on specific caches below). If and how such caches are affected by the functions described above is implementation-specific.

Cache Busting

Implementing cache-compliant web applications can be a challenging task, as can be the implementation of standards-compliant caches. In practice, web application developers deal with the loading of cached resources that should be refreshed, but the official mechanisms of cache control do not work:

• Cache-control header directives like max-age, no-cache and no-store are ignored by misbehaving caches, or they can not be stated appropriately by clients or servers.
• No ETag or Last-Modified headers are set, or their values do not reflect a changed resource state, or weak qualifiers are treated as strong ones, leading to state changes not being reflected in conditional requests.

For such situations, web developers have established a set of „cache-busting“ methods with the intention to force the transfer of the most recent version of a resource from the authoritative source to the user agent. These are also used in combination:

• Appending GET requests with query strings containing random, unique substrings,
• using POST requeststo retrieve resources, because their cacheability is restricted,
• altering the URI of the resource to include random, unique substrings.

From a caching point of view all these methods are undesirable: Altering query strings or URI potentially „spams“ the cache with resources that the cache must regard as different while in reality they are not, while using POST potentially makes a resource non-cacheable that could be cached if control and validation semantics were implemented properly.

The „Vary“ Header

A HTTP resource as specified by a uniform resource locator can be subject to content negotiation, if a request for that resource includes one or more of the headers Accept (which specifies acceptable MIME types of the response), Accept-Charset (specifying acceptable charsets), Accept-Encoding (specifying acceptable transfer encodings) or Accept-Language (specifying an acceptable language). The Vary header of a respons informs intermediate caches that actual delivered content of the resource differs depending on one or more of these headers. For example, a resource that deploys different content to a client depending on MIME type and charset but not depending on the acceptable language or encoding could announce these dependencies with the following header:

Vary: Accept, Accept-Charset

A special Vary header value of * anounces that there are circumstances of an HTTP transaction that can not be specified in terms of HTTP header names.

The semantics described above are the ones mandated by the HTTP/1.1 specification. Some implementations extend this model and make other request headers eligible for being listed in Vary, such as User-Agent (content differs depending on user agent identifcation) or Referer (content differs depending on resource that referred the user agent to this resource).

Content that is subject to Vary semantics is also called „variable“. Content with a Vary header set to * is generally considered not cacheable; however, the exact behavior of an intermediate cache in this regard is implementation-specific, and it cannot be safely assumed that a response header of Vary: * will prevent all intermediate caching. Semantically, a response header of Vary: * does not mean „this content is not cacheable“, it means „responses to this request may vary“. [RFC 7234] section 4.1 specifically mentions that a cache may have stored multiple versions of a resource, where some of them could have no Vary header at all. The cache could eventually choose to deliver this (cached!) version to the client.

On the other hand, in practise, many implementations of intermediate caches consider content that is variable in any aspect as not cacheable; therfore, the use of the Vary header in responses can be expected to have detrimental effect on cache effectiveness.