Rectangle 27 1898

Verifying the actual HTTP response headers

The correct minimum set of headers that works across all mentioned clients (and proxies):

Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0

The Cache-Control is per the HTTP 1.1 spec for clients and proxies (and implicitly required by some clients next to Expires). The Pragma is per the HTTP 1.0 spec for prehistoric clients. The Expires is per the HTTP 1.0 and 1.1 spec for clients and proxies. In HTTP 1.1, the Cache-Control takes precedence over Expires, so it's after all for HTTP 1.0 proxies only.

If you don't care about IE6 and its broken caching when serving pages over HTTPS with only no-store, then you could omit Cache-Control: no-cache.

Cache-Control: no-store, must-revalidate
Pragma: no-cache
Expires: 0

If you don't care about IE6 nor HTTP 1.0 clients (HTTP 1.1 was introduced 1997), then you could omit Pragma.

If you don't care about HTTP 1.0 proxies either, then you could omit Expires.

Cache-Control: no-store, must-revalidate

On the other hand, if the server auto-includes a valid Date header, then you could theoretically omit Cache-Control too and rely on Expires only.

Date: Wed, 24 Aug 2016 18:32:02 GMT
Expires: 0

But that may fail if e.g. the enduser manipulates the operating system date and the client software is relying on it.

Other Cache-Control parameters such as max-age are irrelevant if the abovementioned Cache-Control parameters are specified. The Last-Modified header as included in most other answers here is only interesting if you actually want to cache the request, so you don't need to specify it at all.

header("Cache-Control: no-cache, no-store, must-revalidate"); // HTTP 1.1.
header("Pragma: no-cache"); // HTTP 1.0.
header("Expires: 0"); // Proxies.
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setHeader("Expires", "0"); // Proxies.
Response.Cache.SetCacheability(HttpCacheability.NoCache);  // HTTP 1.1.
Response.Cache.AppendCacheExtension("no-store, must-revalidate");
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.
Response.AppendHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.
Response.addHeader "Cache-Control", "no-cache, no-store, must-revalidate" ' HTTP 1.1.
Response.addHeader "Pragma", "no-cache" ' HTTP 1.0.
Response.addHeader "Expires", "0" ' Proxies.

Using Ruby on Rails, or Python on Flask:

response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
response.headers["Pragma"] = "no-cache" # HTTP 1.0.
response.headers["Expires"] = "0" # Proxies.
responseWriter.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate") // HTTP 1.1.
responseWriter.Header().Set("Pragma", "no-cache") // HTTP 1.0.
responseWriter.Header().Set("Expires", "0") // Proxies.
.htaccess
<IfModule mod_headers.c>
    Header set Cache-Control "no-cache, no-store, must-revalidate"
    Header set Pragma "no-cache"
    Header set Expires 0
</IfModule>
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="Expires" content="0" />
def my_api_call(context, request):

    # disable caching
    request.response.headerlist.extend(
        (
            ('Cache-Control', 'no-cache, no-store, must-revalidate'),
            ('Pragma', 'no-cache'),
            ('Expires', '0')
        )
    )

Important to know is that when a HTML page is served over a HTTP connection, and a header is present in both the HTTP response headers and the HTML <meta http-equiv> tags, then the one specified in the HTTP response header will get precedence over the HTML meta tag. The HTML meta tag will only be used when the page is viewed from local disk file system via a file:// URL. See also W3 HTML spec chapter 5.2.2. Take care with this when you don't specify them programmatically, because the webserver can namely include some default values.

Generally, you'd better just not specify the HTML meta tags to avoid confusion by starters, and rely on hard HTTP response headers. Moreover, specifically those <meta http-equiv> tags are invalid in HTML5. Only the http-equiv values listed in HTML5 specification are allowed.

To verify the one and other, you can see/debug them in HTTP traffic monitor of webbrowser's developer toolset. You can get there by pressing F12 in Chrome/Firefox23+/IE9+, and then opening the "Network" or "Net" tab panel, and then clicking the HTTP request of interest to uncover all detail about the HTTP request and response. The below screenshot is from Chrome:

First of all, this question and answer is targeted on "web pages" (HTML pages), not "file downloads" (PDF, zip, Excel, etc). You'd better have them cached and make use of some file version identifier somewhere in URI path or querystring to force a redownload on a changed file. When applying those no-cache headers on file downloads anyway, then beware of the IE7/8 bug when serving a file download over HTTPS instead of HTTP. For detail, see IE cannot download foo.jsf. IE was not able to open this internet site. The requested site is either unavailable or cannot be found.

This does not appear to be complete. I tried this solution on IE 8 and found that the browser will load a cached version when you hit the back button.

Likely your testing methodology was wrong. Maybe the page was already in the cache? Maybe the headers were incorrect/overriden? Maybe you were looking at the wrong request? Etc..

Actually, I confirm that this approach is incomplete and causes issues with IE8, or at least in some circumstances. Specifically, when using IE8 to fetch a resource over SSL, IE8 will refuse to fetch the resource a second time (either at all, or after a first try, depending on headers used). See EricLaw's blog, for instance.

I'd like to add that this is essentially what Bank of America does. If you look at their response headers and translate that into aspx, they're doing: Response.AppendHeader("Cache-Control", "no-cache, no-store, must-revalidate"); Response.AppendHeader("Expires", "Thu, 01 Dec 1994 16:00:00 GMT"); I figure, if it's good enough for them, it's good enough for me.

@John: That expires header is exactly the example value in HTTP 1.0 specification. It works, but it's somewhat ridiculous to take exactly that timestamp.

http - How to control web page caching, across all browsers? - Stack O...

http caching https http-headers
Rectangle 27 1873

Verifying the actual HTTP response headers

The correct minimum set of headers that works across all mentioned clients (and proxies):

Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0

The Cache-Control is per the HTTP 1.1 spec for clients and proxies (and implicitly required by some clients next to Expires). The Pragma is per the HTTP 1.0 spec for prehistoric clients. The Expires is per the HTTP 1.0 and 1.1 spec for clients and proxies. In HTTP 1.1, the Cache-Control takes precedence over Expires, so it's after all for HTTP 1.0 proxies only.

If you don't care about IE6 and its broken caching when serving pages over HTTPS with only no-store, then you could omit Cache-Control: no-cache.

Cache-Control: no-store, must-revalidate
Pragma: no-cache
Expires: 0

If you don't care about IE6 nor HTTP 1.0 clients (HTTP 1.1 was introduced 1997), then you could omit Pragma.

If you don't care about HTTP 1.0 proxies either, then you could omit Expires.

Cache-Control: no-store, must-revalidate

On the other hand, if the server auto-includes a valid Date header, then you could theoretically omit Cache-Control too and rely on Expires only.

Date: Wed, 24 Aug 2016 18:32:02 GMT
Expires: 0

But that may fail if e.g. the enduser manipulates the operating system date and the client software is relying on it.

Other Cache-Control parameters such as max-age are irrelevant if the abovementioned Cache-Control parameters are specified. The Last-Modified header as included in most other answers here is only interesting if you actually want to cache the request, so you don't need to specify it at all.

header("Cache-Control: no-cache, no-store, must-revalidate"); // HTTP 1.1.
header("Pragma: no-cache"); // HTTP 1.0.
header("Expires: 0"); // Proxies.
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setHeader("Expires", "0"); // Proxies.
Response.Cache.SetCacheability(HttpCacheability.NoCache);  // HTTP 1.1.
Response.Cache.AppendCacheExtension("no-store, must-revalidate");
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.
Response.AppendHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.
Response.addHeader "Cache-Control", "no-cache, no-store, must-revalidate" ' HTTP 1.1.
Response.addHeader "Pragma", "no-cache" ' HTTP 1.0.
Response.addHeader "Expires", "0" ' Proxies.

Using Ruby on Rails, or Python on Flask:

response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
response.headers["Pragma"] = "no-cache" # HTTP 1.0.
response.headers["Expires"] = "0" # Proxies.
responseWriter.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate") // HTTP 1.1.
responseWriter.Header().Set("Pragma", "no-cache") // HTTP 1.0.
responseWriter.Header().Set("Expires", "0") // Proxies.
.htaccess
<IfModule mod_headers.c>
    Header set Cache-Control "no-cache, no-store, must-revalidate"
    Header set Pragma "no-cache"
    Header set Expires 0
</IfModule>
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="Expires" content="0" />
def my_api_call(context, request):

    # disable caching
    request.response.headerlist.extend(
        (
            ('Cache-Control', 'no-cache, no-store, must-revalidate'),
            ('Pragma', 'no-cache'),
            ('Expires', '0')
        )
    )

Important to know is that when a HTML page is served over a HTTP connection, and a header is present in both the HTTP response headers and the HTML <meta http-equiv> tags, then the one specified in the HTTP response header will get precedence over the HTML meta tag. The HTML meta tag will only be used when the page is viewed from local disk file system via a file:// URL. See also W3 HTML spec chapter 5.2.2. Take care with this when you don't specify them programmatically, because the webserver can namely include some default values.

Generally, you'd better just not specify the HTML meta tags to avoid confusion by starters, and rely on hard HTTP response headers. Moreover, specifically those <meta http-equiv> tags are invalid in HTML5. Only the http-equiv values listed in HTML5 specification are allowed.

To verify the one and other, you can see/debug them in HTTP traffic monitor of webbrowser's developer toolset. You can get there by pressing F12 in Chrome/Firefox23+/IE9+, and then opening the "Network" or "Net" tab panel, and then clicking the HTTP request of interest to uncover all detail about the HTTP request and response. The below screenshot is from Chrome:

First of all, this question and answer is targeted on "web pages" (HTML pages), not "file downloads" (PDF, zip, Excel, etc). You'd better have them cached and make use of some file version identifier somewhere in URI path or querystring to force a redownload on a changed file. When applying those no-cache headers on file downloads anyway, then beware of the IE7/8 bug when serving a file download over HTTPS instead of HTTP. For detail, see IE cannot download foo.jsf. IE was not able to open this internet site. The requested site is either unavailable or cannot be found.

This does not appear to be complete. I tried this solution on IE 8 and found that the browser will load a cached version when you hit the back button.

Likely your testing methodology was wrong. Maybe the page was already in the cache? Maybe the headers were incorrect/overriden? Maybe you were looking at the wrong request? Etc..

Actually, I confirm that this approach is incomplete and causes issues with IE8, or at least in some circumstances. Specifically, when using IE8 to fetch a resource over SSL, IE8 will refuse to fetch the resource a second time (either at all, or after a first try, depending on headers used). See EricLaw's blog, for instance.

I'd like to add that this is essentially what Bank of America does. If you look at their response headers and translate that into aspx, they're doing: Response.AppendHeader("Cache-Control", "no-cache, no-store, must-revalidate"); Response.AppendHeader("Expires", "Thu, 01 Dec 1994 16:00:00 GMT"); I figure, if it's good enough for them, it's good enough for me.

@John: That expires header is exactly the example value in HTTP 1.0 specification. It works, but it's somewhat ridiculous to take exactly that timestamp.

http - How to control web page caching, across all browsers? - Stack O...

http caching https http-headers
Rectangle 27 1872

Verifying the actual HTTP response headers

The correct minimum set of headers that works across all mentioned clients (and proxies):

Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0

The Cache-Control is per the HTTP 1.1 spec for clients and proxies (and implicitly required by some clients next to Expires). The Pragma is per the HTTP 1.0 spec for prehistoric clients. The Expires is per the HTTP 1.0 and 1.1 spec for clients and proxies. In HTTP 1.1, the Cache-Control takes precedence over Expires, so it's after all for HTTP 1.0 proxies only.

If you don't care about IE6 and its broken caching when serving pages over HTTPS with only no-store, then you could omit Cache-Control: no-cache.

Cache-Control: no-store, must-revalidate
Pragma: no-cache
Expires: 0

If you don't care about IE6 nor HTTP 1.0 clients (HTTP 1.1 was introduced 1997), then you could omit Pragma.

If you don't care about HTTP 1.0 proxies either, then you could omit Expires.

Cache-Control: no-store, must-revalidate

On the other hand, if the server auto-includes a valid Date header, then you could theoretically omit Cache-Control too and rely on Expires only.

Date: Wed, 24 Aug 2016 18:32:02 GMT
Expires: 0

But that may fail if e.g. the enduser manipulates the operating system date and the client software is relying on it.

Other Cache-Control parameters such as max-age are irrelevant if the abovementioned Cache-Control parameters are specified. The Last-Modified header as included in most other answers here is only interesting if you actually want to cache the request, so you don't need to specify it at all.

header("Cache-Control: no-cache, no-store, must-revalidate"); // HTTP 1.1.
header("Pragma: no-cache"); // HTTP 1.0.
header("Expires: 0"); // Proxies.
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setHeader("Expires", "0"); // Proxies.
Response.Cache.SetCacheability(HttpCacheability.NoCache);  // HTTP 1.1.
Response.Cache.AppendCacheExtension("no-store, must-revalidate");
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.
Response.AppendHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.
Response.addHeader "Cache-Control", "no-cache, no-store, must-revalidate" ' HTTP 1.1.
Response.addHeader "Pragma", "no-cache" ' HTTP 1.0.
Response.addHeader "Expires", "0" ' Proxies.

Using Ruby on Rails, or Python on Flask:

response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
response.headers["Pragma"] = "no-cache" # HTTP 1.0.
response.headers["Expires"] = "0" # Proxies.
responseWriter.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate") // HTTP 1.1.
responseWriter.Header().Set("Pragma", "no-cache") // HTTP 1.0.
responseWriter.Header().Set("Expires", "0") // Proxies.
.htaccess
<IfModule mod_headers.c>
    Header set Cache-Control "no-cache, no-store, must-revalidate"
    Header set Pragma "no-cache"
    Header set Expires 0
</IfModule>
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="Expires" content="0" />
def my_api_call(context, request):

    # disable caching
    request.response.headerlist.extend(
        (
            ('Cache-Control', 'no-cache, no-store, must-revalidate'),
            ('Pragma', 'no-cache'),
            ('Expires', '0')
        )
    )

Important to know is that when a HTML page is served over a HTTP connection, and a header is present in both the HTTP response headers and the HTML <meta http-equiv> tags, then the one specified in the HTTP response header will get precedence over the HTML meta tag. The HTML meta tag will only be used when the page is viewed from local disk file system via a file:// URL. See also W3 HTML spec chapter 5.2.2. Take care with this when you don't specify them programmatically, because the webserver can namely include some default values.

Generally, you'd better just not specify the HTML meta tags to avoid confusion by starters, and rely on hard HTTP response headers. Moreover, specifically those <meta http-equiv> tags are invalid in HTML5. Only the http-equiv values listed in HTML5 specification are allowed.

To verify the one and other, you can see/debug them in HTTP traffic monitor of webbrowser's developer toolset. You can get there by pressing F12 in Chrome/Firefox23+/IE9+, and then opening the "Network" or "Net" tab panel, and then clicking the HTTP request of interest to uncover all detail about the HTTP request and response. The below screenshot is from Chrome:

First of all, this question and answer is targeted on "web pages" (HTML pages), not "file downloads" (PDF, zip, Excel, etc). You'd better have them cached and make use of some file version identifier somewhere in URI path or querystring to force a redownload on a changed file. When applying those no-cache headers on file downloads anyway, then beware of the IE7/8 bug when serving a file download over HTTPS instead of HTTP. For detail, see IE cannot download foo.jsf. IE was not able to open this internet site. The requested site is either unavailable or cannot be found.

This does not appear to be complete. I tried this solution on IE 8 and found that the browser will load a cached version when you hit the back button.

Likely your testing methodology was wrong. Maybe the page was already in the cache? Maybe the headers were incorrect/overriden? Maybe you were looking at the wrong request? Etc..

Actually, I confirm that this approach is incomplete and causes issues with IE8, or at least in some circumstances. Specifically, when using IE8 to fetch a resource over SSL, IE8 will refuse to fetch the resource a second time (either at all, or after a first try, depending on headers used). See EricLaw's blog, for instance.

I'd like to add that this is essentially what Bank of America does. If you look at their response headers and translate that into aspx, they're doing: Response.AppendHeader("Cache-Control", "no-cache, no-store, must-revalidate"); Response.AppendHeader("Expires", "Thu, 01 Dec 1994 16:00:00 GMT"); I figure, if it's good enough for them, it's good enough for me.

@John: That expires header is exactly the example value in HTTP 1.0 specification. It works, but it's somewhat ridiculous to take exactly that timestamp.

http - How to control web page caching, across all browsers? - Stack O...

http caching https http-headers
Rectangle 27 241

Getting header values from the Initial Page Request:

Unfortunately, there isn't an API to give you the HTTP response headers for your initial page request. That was the original question posted here. It has been repeatedly asked, too, because some people would like to get the actual response headers of the original page request without issuing another one.

If an HTTP request is made over AJAX, it is possible to get the response headers with the getAllResponseHeaders() method. It's part of the XMLHttpRequest API. To see how this can be applied, check out the fetchSimilarHeaders() function below. Note that this is a work-around to the problem that won't be reliable for some applications.

myXMLHttpRequest.getAllResponseHeaders();

This will not give you information about the original page request's HTTP response headers, but it could be used to make educated guesses about what those headers were. More on that is described next.

This question was first asked several years ago, asking specifically about how to get at the original HTTP response headers for the current page (i.e. the same page inside of which the javascript was running). This is quite a different question than simply getting the response headers for any HTTP request. For the initial page request, the headers aren't readily available to javascript. Whether the header values you need will be reliably and sufficiently consistent if you request the same page again via AJAX will depend on your particular application.

The following are a few suggestions for getting around that problem.

If the response is largely static and the headers are not expected to change much between requests, you could make an AJAX request for the same page you're currently on and assume that they're they are the same values which were part of the page's HTTP response. This could allow you to access the headers you need using the nice XMLHttpRequest API described above.

function fetchSimilarHeaders (callback) {
    var request = new XMLHttpRequest();
    request.onreadystatechange = function () {
        if (request.readyState === 4) {
            //
            // The following headers may often be similar
            // to those of the original page request...
            //
            if (callback && typeof callback === 'function') {
                callback(request.getAllResponseHeaders());
            }
        }
    };

    //
    // Re-request the same page (document.location)
    // We hope to get the same or similar response headers to those which 
    // came with the current page, but we have no guarantee.
    // Since we are only after the headers, a HEAD request may be sufficient.
    //
    request.open('HEAD', document.location, true);
    request.send(null);
}

This approach will be problematic if you truly have to rely on the values being consistent between requests, since you can't fully guarantee that they are the same. It's going to depend on your specific application and whether you know that the value you need is something that won't be changing from one request to the next.

There are some BOM properties (Browser Object Model) which the browser determines by looking at the headers. Some of these properties reflect HTTP headers directly (e.g. navigator.userAgent is set to the value of the HTTP User-Agent header field). By sniffing around the available properties you might be able to find what you need, or some clues to indicate what the HTTP response contained.

If you control the server side, you can access any header you like as you construct the full response. Values could be passed to the client with the page, stashed in some markup or perhaps in an inlined JSON structure. If you wanted to have every HTTP request header available to your javascript, you could iterate through them on the server and send them back as hidden values in the markup. It's probably not ideal to send header values this way, but you could certainly do it for the specific value you need. This solution is arguably inefficient, too, but it would do the job if you needed it.

RE update: ajax requests were a standard part of web development way back in 2008 as well -_-

3) you could stash them in the http cookie header, too. Then you wouldn't need to change the document markup.

There is a simple way to access the response header elements such as the link element: use document example here: gist.github.com/FunThomas424242/

Accessing the web page's HTTP Headers in JavaScript - Stack Overflow

javascript http http-headers
Rectangle 27 241

Getting header values from the Initial Page Request:

Unfortunately, there isn't an API to give you the HTTP response headers for your initial page request. That was the original question posted here. It has been repeatedly asked, too, because some people would like to get the actual response headers of the original page request without issuing another one.

If an HTTP request is made over AJAX, it is possible to get the response headers with the getAllResponseHeaders() method. It's part of the XMLHttpRequest API. To see how this can be applied, check out the fetchSimilarHeaders() function below. Note that this is a work-around to the problem that won't be reliable for some applications.

myXMLHttpRequest.getAllResponseHeaders();

This will not give you information about the original page request's HTTP response headers, but it could be used to make educated guesses about what those headers were. More on that is described next.

This question was first asked several years ago, asking specifically about how to get at the original HTTP response headers for the current page (i.e. the same page inside of which the javascript was running). This is quite a different question than simply getting the response headers for any HTTP request. For the initial page request, the headers aren't readily available to javascript. Whether the header values you need will be reliably and sufficiently consistent if you request the same page again via AJAX will depend on your particular application.

The following are a few suggestions for getting around that problem.

If the response is largely static and the headers are not expected to change much between requests, you could make an AJAX request for the same page you're currently on and assume that they're they are the same values which were part of the page's HTTP response. This could allow you to access the headers you need using the nice XMLHttpRequest API described above.

function fetchSimilarHeaders (callback) {
    var request = new XMLHttpRequest();
    request.onreadystatechange = function () {
        if (request.readyState === 4) {
            //
            // The following headers may often be similar
            // to those of the original page request...
            //
            if (callback && typeof callback === 'function') {
                callback(request.getAllResponseHeaders());
            }
        }
    };

    //
    // Re-request the same page (document.location)
    // We hope to get the same or similar response headers to those which 
    // came with the current page, but we have no guarantee.
    // Since we are only after the headers, a HEAD request may be sufficient.
    //
    request.open('HEAD', document.location, true);
    request.send(null);
}

This approach will be problematic if you truly have to rely on the values being consistent between requests, since you can't fully guarantee that they are the same. It's going to depend on your specific application and whether you know that the value you need is something that won't be changing from one request to the next.

There are some BOM properties (Browser Object Model) which the browser determines by looking at the headers. Some of these properties reflect HTTP headers directly (e.g. navigator.userAgent is set to the value of the HTTP User-Agent header field). By sniffing around the available properties you might be able to find what you need, or some clues to indicate what the HTTP response contained.

If you control the server side, you can access any header you like as you construct the full response. Values could be passed to the client with the page, stashed in some markup or perhaps in an inlined JSON structure. If you wanted to have every HTTP request header available to your javascript, you could iterate through them on the server and send them back as hidden values in the markup. It's probably not ideal to send header values this way, but you could certainly do it for the specific value you need. This solution is arguably inefficient, too, but it would do the job if you needed it.

RE update: ajax requests were a standard part of web development way back in 2008 as well -_-

3) you could stash them in the http cookie header, too. Then you wouldn't need to change the document markup.

There is a simple way to access the response header elements such as the link element: use document example here: gist.github.com/FunThomas424242/

Accessing the web page's HTTP Headers in JavaScript - Stack Overflow

javascript http http-headers
Rectangle 27 90

Here are some examples of HTTP API Rate Limiting HTTP Response headers. Taken from four common REST APIs: Github, Vimeo, Twitter and Imgur:

Note: Twitter uses headers with similar names like Vimeo, but has another dash in each name.

#=============================#=============================================#
# HTTP Header                 # Description                                 #
#=============================#=============================================#
| X-Rate-Limit-Limit          | The rate limit ceiling for that given       |
|                             | request                                     |
+-----------------------------+---------------------------------------------+
| X-Rate-Limit-Remaining      | The number of requests left for the         |
|                             | 15 minute window                            |
+-----------------------------+---------------------------------------------+
| X-Rate-Limit-Reset          | The remaining window before the rate limit  |
|                             | resets in UTC epoch seconds                 |
+-----------------------------+---------------------------------------------+
#=============================#=============================================#
# HTTP Header                 # Description                                 #
#=============================#=============================================#
| X-RateLimit-UserLimit       | Total credits that can be allocated         |
+-----------------------------+---------------------------------------------+
| X-RateLimit-UserRemaining   | Total credits available                     |
+-----------------------------+---------------------------------------------+
| X-RateLimit-UserReset       | Timestamp (unix epoch) for when the credits |
|                             | will be reset                               |
+-----------------------------+---------------------------------------------+
| X-RateLimit-ClientLimit     | Total credits that can be allocated for the |
|                             | application in a day                        |
+-----------------------------+---------------------------------------------+
| X-RateLimit-ClientRemaining | Total credits remaining for the application |
|                             | in a day                                    |
+-----------------------------+---------------------------------------------+

If you're designing your own rate-limit headers, the Best Current Practice BCP178 is a relevant resource, recommending that the X- prefix be deprecated. Check out the original RFC/BCP for more info. tools.ietf.org/html/bcp178

Great examples, I made a Node.js package that can be used with the request package: github.com/webjay/x-rate

Examples of HTTP API Rate Limiting HTTP Response headers - Stack Overf...

api http rest rate-limiting
Rectangle 27 118

The following changes allow you to remove these HTTP response headers in Azure without writing a custom HttpModule.

Most of the information on the net is out of date, and involves UrlScan (which has since been integrated into IIS7, but with the RemoveServerHeader=1 option removed). Below is the neatest solution I've found (thanks to this blog, this answer, and this blog combined).

Application_PreSendRequestHeaders

Edited April 2014: You can use the PreSendRequestHeaders and PreSendRequestContext events with native IIS modules, but do not use them with managed modules that implement IHttpModule. Setting these properties can cause issues with asynchronous requests. The correct version is to use BeginRequest event.

protected void Application_BeginRequest(object sender, EventArgs e)
    {
        var application = sender as HttpApplication;
        if (application != null && application.Context != null)
        {
            application.Context.Response.Headers.Remove("Server");
        }
    }

To remove X-AspNet-Version, in the web.config find/create <system.web> and add:

<system.web>
    <httpRuntime enableVersionHeader="false" />

    ...

To remove X-AspNetMvc-Version, go to Global.asax, find/create the Application_Start event and add a line as follows:

protected void Application_Start()
  {
      MvcHandler.DisableMvcResponseHeader = true;
  }

To remove X-Powered-By, in the web.config find/create <system.webServer> and add:

<system.webServer>
    <httpProtocol>
      <customHeaders>
        <remove name="X-Powered-By" />
      </customHeaders>
    </httpProtocol>

    ...

When used on IIS not Azure be aware that application pool has to be in Integrated mode. And .IsLocal should be removed when debugging locally.

There's no need for "Yoda conditions" in C# - it doesn't allow assignment in a conditional, en.wikipedia.org/wiki/Yoda_Conditions

Thanks for the detail answer ,however i did try and followed up the steps but each time i scan the site using asafweb, it still mentions an issue about the excessive header (X-AspNet-Version). I even used the URLRewrite to remove this header. Are they any other possibilities of removing it?

There is still the problem of requesting a non-existent file, e.g. "yoursite/foo.jpg. Since this request is not processed by MVC the response header "Server: IIS x.y" will still be there. One solution which works for Azure Web Sites (and apparently ONLY for azure web sites) is to add this under <system.webServer>: <security xdt:Transform="Insert"> <requestFiltering removeServerHeader="true"/> </security>

asp.net - Removing/Hiding/Disabling excessive HTTP response headers in...

asp.net iis-7 azure penetration-testing response-headers
Rectangle 27 118

The following changes allow you to remove these HTTP response headers in Azure without writing a custom HttpModule.

Most of the information on the net is out of date, and involves UrlScan (which has since been integrated into IIS7, but with the RemoveServerHeader=1 option removed). Below is the neatest solution I've found (thanks to this blog, this answer, and this blog combined).

Application_PreSendRequestHeaders

Edited April 2014: You can use the PreSendRequestHeaders and PreSendRequestContext events with native IIS modules, but do not use them with managed modules that implement IHttpModule. Setting these properties can cause issues with asynchronous requests. The correct version is to use BeginRequest event.

protected void Application_BeginRequest(object sender, EventArgs e)
    {
        var application = sender as HttpApplication;
        if (application != null && application.Context != null)
        {
            application.Context.Response.Headers.Remove("Server");
        }
    }

To remove X-AspNet-Version, in the web.config find/create <system.web> and add:

<system.web>
    <httpRuntime enableVersionHeader="false" />

    ...

To remove X-AspNetMvc-Version, go to Global.asax, find/create the Application_Start event and add a line as follows:

protected void Application_Start()
  {
      MvcHandler.DisableMvcResponseHeader = true;
  }

To remove X-Powered-By, in the web.config find/create <system.webServer> and add:

<system.webServer>
    <httpProtocol>
      <customHeaders>
        <remove name="X-Powered-By" />
      </customHeaders>
    </httpProtocol>

    ...

When used on IIS not Azure be aware that application pool has to be in Integrated mode. And .IsLocal should be removed when debugging locally.

There's no need for "Yoda conditions" in C# - it doesn't allow assignment in a conditional, en.wikipedia.org/wiki/Yoda_Conditions

Thanks for the detail answer ,however i did try and followed up the steps but each time i scan the site using asafweb, it still mentions an issue about the excessive header (X-AspNet-Version). I even used the URLRewrite to remove this header. Are they any other possibilities of removing it?

There is still the problem of requesting a non-existent file, e.g. "yoursite/foo.jpg. Since this request is not processed by MVC the response header "Server: IIS x.y" will still be there. One solution which works for Azure Web Sites (and apparently ONLY for azure web sites) is to add this under <system.webServer>: <security xdt:Transform="Insert"> <requestFiltering removeServerHeader="true"/> </security>

asp.net - Removing/Hiding/Disabling excessive HTTP response headers in...

asp.net iis-7 azure penetration-testing response-headers
Rectangle 27 22

Use the Net tab in Firebug to monitor network traffic. There you can see the complete headers as well as the timing and content of any network/HTTP queries.

Hi Vagrant. Firebug was my first choice too, but I don't seem to have a Net tab. Would you happen to know if this is a Mac OS X feature?

firebug - How to debug the http response headers from a HTTP call - St...

http firebug
Rectangle 27 22

Use the Net tab in Firebug to monitor network traffic. There you can see the complete headers as well as the timing and content of any network/HTTP queries.

Hi Vagrant. Firebug was my first choice too, but I don't seem to have a Net tab. Would you happen to know if this is a Mac OS X feature?

firebug - How to debug the http response headers from a HTTP call - St...

http firebug
Rectangle 27 6

Fiddler is the best tool I know of to do this and much more, including inspecting the entire request/response in many formats including a nice XML browser.

firebug - How to debug the http response headers from a HTTP call - St...

http firebug
Rectangle 27 6

Fiddler is the best tool I know of to do this and much more, including inspecting the entire request/response in many formats including a nice XML browser.

firebug - How to debug the http response headers from a HTTP call - St...

http firebug
Rectangle 27 343

The question bears re-reading. The actual question asked is not similar to vendor prefixes in CSS properties, where future-proofing and thinking about vendor support and official standards is appropriate. The actual question asked is more akin to choosing URL query parameter names. Nobody should care what they are. But name-spacing the custom ones is a perfectly valid -- and common, and correct -- thing to do.

Rationale: It is about conventions among developers for custom, application-specific headers -- "data relevant to their account" -- which have nothing to do with vendors, standards bodies, or protocols to be implemented by third parties, except that the developer in question simply needs to avoid header names that may have other intended use by servers, proxies or clients. For this reason, the "X-Gzip/Gzip" and "X-Forwarded-For/Forwarded-For" examples given are moot. The question posed is about conventions in the context of a private API, akin to URL query parameter naming conventions. It's a matter of preference and name-spacing; concerns about "X-ClientDataFoo" being supported by any proxy or vendor without the "X" are clearly misplaced.

There's nothing special or magical about the "X-" prefix, but it helps to make it clear that it is a custom header. In fact, RFC-6648 et al help bolster the case for use of an "X-" prefix, because -- as vendors of HTTP clients and servers abandon the prefix -- your app-specific, private-API, personal-data-passing-mechanism is becoming even better-insulated against name-space collisions with the small number of official reserved header names. That said, my personal preference and recommendation is to go a step further and do e.g. "X-ACME-ClientDataFoo" (if your widget company is "ACME").

IMHO the IETF spec is insufficiently specific to answer the OP's question, because it fails to distinguish between completely different use cases: (A) vendors introducing new globally-applicable features like "Forwarded-For" on the one hand, vs. (B) app developers passing app-specific strings to/from client and server. The spec only concerns itself with the former, (A). The question here is whether there are conventions for (B). There are. They involve grouping the parameters together alphabetically, and separating them from the many standards-relevant headers of type (A). Using the "X-" or "X-ACME-" prefix is convenient and legitimate for (B), and does not conflict with (A). The more vendors stop using "X-" for (A), the more cleanly-distinct the (B) ones will become.

Example: Google (who carry a bit of weight in the various standards bodies) are -- as of today, 20141102 in this slight edit to my answer -- currently using "X-Mod-Pagespeed" to indicate the version of their Apache module involved in transforming a given response. Is anyone really suggesting that Google should use "Mod-Pagespeed", without the "X-", and/or ask the IETF to bless its use?

Summary: If you're using custom HTTP Headers (as a sometimes-appropriate alternative to cookies) within your app to pass data to/from your server, and these headers are, explicitly, NOT intended ever to be used outside the context of your application, name-spacing them with an "X-" or "X-FOO-" prefix is a reasonable, and common, convention.

I'd appreciate it if any downvoters of my comment could explain what part of my answer they find objectionable. I don't care that much about my reputation score, but I'm genuinely curious. Where does the disagreement lie? Thanks.

I completely agree with your answer and it is the only answer here that answers the actual question asked. We ARE talking about custom, application-specific headers here, never to be standardized in the HTTP standards. Is there a common convention for these that people tend to use? (such as prefixing them with "_" perhaps? ie: ("_ClientDataFoo")

Thanks Marchy, yeah, the accepted answer doesn't address the question asked. IETF deprecation of "X-" prefix for non-standard (but generic) headers is irrelevant to custom app-specific headers that will never be standardized. To answer your question, in my opinion and experience (16 years of webdev), the best convention is to use the aforementioned "X-ACME-ClientData". "X-" bc it's not standard (nor will it ever be, which is why the IETF deprecation is moot here), "ACME-" to namespace it to your "ACME" company or specific app, and "ClientData" can be whatever semantic name you like. :)

@DarrelMiller ... hence the recommendation to use X-ACMECO-WIDGET-FOO. I insist that, for the OP's question as asked, use of X- is simply not contra-indicated by RFC-6648 and the like. If you're a vendor providing a framework, library or module for use in other peoples' projects, that's a different story, and by all means follow that RFC to a T. But it's just moot for individual one-off applications, where custom app-specific header naming conventions are effectively completely private APIs. How would they clash with "everyone else's" names? Whose would those be?

I'm honestly having a little trouble understanding the RFC reasoning. Granted that, if and when the parameter is standardized, there will be both x- and non-x- versions. That is only a problem if the behavior of the x- and non-x- versions is identical. I stumbled over here because I am looking at adding an "on behalf of" header to my API. It could become public some day (as it's a common kind of use case). If I used "On-Behalf-Of" and someday they add that as a standard header, what are the odds that my semantics will be identical to the standardized one?

Custom HTTP headers : naming conventions - Stack Overflow

http http-headers
Rectangle 27 343

The question bears re-reading. The actual question asked is not similar to vendor prefixes in CSS properties, where future-proofing and thinking about vendor support and official standards is appropriate. The actual question asked is more akin to choosing URL query parameter names. Nobody should care what they are. But name-spacing the custom ones is a perfectly valid -- and common, and correct -- thing to do.

Rationale: It is about conventions among developers for custom, application-specific headers -- "data relevant to their account" -- which have nothing to do with vendors, standards bodies, or protocols to be implemented by third parties, except that the developer in question simply needs to avoid header names that may have other intended use by servers, proxies or clients. For this reason, the "X-Gzip/Gzip" and "X-Forwarded-For/Forwarded-For" examples given are moot. The question posed is about conventions in the context of a private API, akin to URL query parameter naming conventions. It's a matter of preference and name-spacing; concerns about "X-ClientDataFoo" being supported by any proxy or vendor without the "X" are clearly misplaced.

There's nothing special or magical about the "X-" prefix, but it helps to make it clear that it is a custom header. In fact, RFC-6648 et al help bolster the case for use of an "X-" prefix, because -- as vendors of HTTP clients and servers abandon the prefix -- your app-specific, private-API, personal-data-passing-mechanism is becoming even better-insulated against name-space collisions with the small number of official reserved header names. That said, my personal preference and recommendation is to go a step further and do e.g. "X-ACME-ClientDataFoo" (if your widget company is "ACME").

IMHO the IETF spec is insufficiently specific to answer the OP's question, because it fails to distinguish between completely different use cases: (A) vendors introducing new globally-applicable features like "Forwarded-For" on the one hand, vs. (B) app developers passing app-specific strings to/from client and server. The spec only concerns itself with the former, (A). The question here is whether there are conventions for (B). There are. They involve grouping the parameters together alphabetically, and separating them from the many standards-relevant headers of type (A). Using the "X-" or "X-ACME-" prefix is convenient and legitimate for (B), and does not conflict with (A). The more vendors stop using "X-" for (A), the more cleanly-distinct the (B) ones will become.

Example: Google (who carry a bit of weight in the various standards bodies) are -- as of today, 20141102 in this slight edit to my answer -- currently using "X-Mod-Pagespeed" to indicate the version of their Apache module involved in transforming a given response. Is anyone really suggesting that Google should use "Mod-Pagespeed", without the "X-", and/or ask the IETF to bless its use?

Summary: If you're using custom HTTP Headers (as a sometimes-appropriate alternative to cookies) within your app to pass data to/from your server, and these headers are, explicitly, NOT intended ever to be used outside the context of your application, name-spacing them with an "X-" or "X-FOO-" prefix is a reasonable, and common, convention.

I'd appreciate it if any downvoters of my comment could explain what part of my answer they find objectionable. I don't care that much about my reputation score, but I'm genuinely curious. Where does the disagreement lie? Thanks.

I completely agree with your answer and it is the only answer here that answers the actual question asked. We ARE talking about custom, application-specific headers here, never to be standardized in the HTTP standards. Is there a common convention for these that people tend to use? (such as prefixing them with "_" perhaps? ie: ("_ClientDataFoo")

Thanks Marchy, yeah, the accepted answer doesn't address the question asked. IETF deprecation of "X-" prefix for non-standard (but generic) headers is irrelevant to custom app-specific headers that will never be standardized. To answer your question, in my opinion and experience (16 years of webdev), the best convention is to use the aforementioned "X-ACME-ClientData". "X-" bc it's not standard (nor will it ever be, which is why the IETF deprecation is moot here), "ACME-" to namespace it to your "ACME" company or specific app, and "ClientData" can be whatever semantic name you like. :)

@DarrelMiller ... hence the recommendation to use X-ACMECO-WIDGET-FOO. I insist that, for the OP's question as asked, use of X- is simply not contra-indicated by RFC-6648 and the like. If you're a vendor providing a framework, library or module for use in other peoples' projects, that's a different story, and by all means follow that RFC to a T. But it's just moot for individual one-off applications, where custom app-specific header naming conventions are effectively completely private APIs. How would they clash with "everyone else's" names? Whose would those be?

I'm honestly having a little trouble understanding the RFC reasoning. Granted that, if and when the parameter is standardized, there will be both x- and non-x- versions. That is only a problem if the behavior of the x- and non-x- versions is identical. I stumbled over here because I am looking at adding an "on behalf of" header to my API. It could become public some day (as it's a common kind of use case). If I used "On-Behalf-Of" and someday they add that as a standard header, what are the odds that my semantics will be identical to the standardized one?

Custom HTTP headers : naming conventions - Stack Overflow

http http-headers
Rectangle 27 343

The question bears re-reading. The actual question asked is not similar to vendor prefixes in CSS properties, where future-proofing and thinking about vendor support and official standards is appropriate. The actual question asked is more akin to choosing URL query parameter names. Nobody should care what they are. But name-spacing the custom ones is a perfectly valid -- and common, and correct -- thing to do.

Rationale: It is about conventions among developers for custom, application-specific headers -- "data relevant to their account" -- which have nothing to do with vendors, standards bodies, or protocols to be implemented by third parties, except that the developer in question simply needs to avoid header names that may have other intended use by servers, proxies or clients. For this reason, the "X-Gzip/Gzip" and "X-Forwarded-For/Forwarded-For" examples given are moot. The question posed is about conventions in the context of a private API, akin to URL query parameter naming conventions. It's a matter of preference and name-spacing; concerns about "X-ClientDataFoo" being supported by any proxy or vendor without the "X" are clearly misplaced.

There's nothing special or magical about the "X-" prefix, but it helps to make it clear that it is a custom header. In fact, RFC-6648 et al help bolster the case for use of an "X-" prefix, because -- as vendors of HTTP clients and servers abandon the prefix -- your app-specific, private-API, personal-data-passing-mechanism is becoming even better-insulated against name-space collisions with the small number of official reserved header names. That said, my personal preference and recommendation is to go a step further and do e.g. "X-ACME-ClientDataFoo" (if your widget company is "ACME").

IMHO the IETF spec is insufficiently specific to answer the OP's question, because it fails to distinguish between completely different use cases: (A) vendors introducing new globally-applicable features like "Forwarded-For" on the one hand, vs. (B) app developers passing app-specific strings to/from client and server. The spec only concerns itself with the former, (A). The question here is whether there are conventions for (B). There are. They involve grouping the parameters together alphabetically, and separating them from the many standards-relevant headers of type (A). Using the "X-" or "X-ACME-" prefix is convenient and legitimate for (B), and does not conflict with (A). The more vendors stop using "X-" for (A), the more cleanly-distinct the (B) ones will become.

Example: Google (who carry a bit of weight in the various standards bodies) are -- as of today, 20141102 in this slight edit to my answer -- currently using "X-Mod-Pagespeed" to indicate the version of their Apache module involved in transforming a given response. Is anyone really suggesting that Google should use "Mod-Pagespeed", without the "X-", and/or ask the IETF to bless its use?

Summary: If you're using custom HTTP Headers (as a sometimes-appropriate alternative to cookies) within your app to pass data to/from your server, and these headers are, explicitly, NOT intended ever to be used outside the context of your application, name-spacing them with an "X-" or "X-FOO-" prefix is a reasonable, and common, convention.

I'd appreciate it if any downvoters of my comment could explain what part of my answer they find objectionable. I don't care that much about my reputation score, but I'm genuinely curious. Where does the disagreement lie? Thanks.

I completely agree with your answer and it is the only answer here that answers the actual question asked. We ARE talking about custom, application-specific headers here, never to be standardized in the HTTP standards. Is there a common convention for these that people tend to use? (such as prefixing them with "_" perhaps? ie: ("_ClientDataFoo")

Thanks Marchy, yeah, the accepted answer doesn't address the question asked. IETF deprecation of "X-" prefix for non-standard (but generic) headers is irrelevant to custom app-specific headers that will never be standardized. To answer your question, in my opinion and experience (16 years of webdev), the best convention is to use the aforementioned "X-ACME-ClientData". "X-" bc it's not standard (nor will it ever be, which is why the IETF deprecation is moot here), "ACME-" to namespace it to your "ACME" company or specific app, and "ClientData" can be whatever semantic name you like. :)

Further to your recommendations, I'd suggest using a similar convention to package or namespace names in many languages, that is something similar to a reverse domain: com.mycompany.headers.custom:somevalue. I can't find anything anywhere that says you can't use dots in header names. You won't get collisions that way.

@DarrelMiller ... hence the recommendation to use X-ACMECO-WIDGET-FOO. I insist that, for the OP's question as asked, use of X- is simply not contra-indicated by RFC-6648 and the like. If you're a vendor providing a framework, library or module for use in other peoples' projects, that's a different story, and by all means follow that RFC to a T. But it's just moot for individual one-off applications, where custom app-specific header naming conventions are effectively completely private APIs. How would they clash with "everyone else's" names? Whose would those be?

I'm honestly having a little trouble understanding the RFC reasoning. Granted that, if and when the parameter is standardized, there will be both x- and non-x- versions. That is only a problem if the behavior of the x- and non-x- versions is identical. I stumbled over here because I am looking at adding an "on behalf of" header to my API. It could become public some day (as it's a common kind of use case). If I used "On-Behalf-Of" and someday they add that as a standard header, what are the odds that my semantics will be identical to the standardized one?

@Conan - I've always used just letters, numbers and hyphens (which map cleanly to Apache mod_setenvif underscores for corresponding env vars), since pairing headers to env vars in Apache is my primary use for custom headers. YMMV.

Custom HTTP headers : naming conventions - Stack Overflow

http http-headers
Rectangle 27 112

Not quite. Creating a Request object does not actually send the request, and Request objects have no Read() method. (Also: read() is lowercase.) All you need to do is pass the Request as the first argument to urlopen() and that will give you your response.

import urllib2
request = urllib2.Request("http://www.google.com", headers={"Accept" : "text/html"})
contents = urllib2.urlopen(request).read()

python - How do I send a custom header with urllib2 in a HTTP Request?...

python header urllib2
Rectangle 27 362

$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
// ...

$response = curl_exec($ch);

// Then, after your curl_exec call:
$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$header = substr($response, 0, $header_size);
$body = substr($response, $header_size);

Warning: As noted in the comments below, this may not be reliable when used with proxy servers or when handling certain types of redirects. @Geoffrey's answer may handle these more reliably.

list($header, $body) = explode("\r\n\r\n", $response, 2)

this is bad solution because if you use proxy server and your proxy server(fiddler for example) add own headers to response - this headers broke all offsets and you should use list($header, $body) = explode("\r\n\r\n", $response, 2) as only working variant

@msangel Your solution doesn't work when there are multiple headers in the response, such as when the server does a 302 redirect. Any suggestions?

@Nate, yes, i know this. AFAIK, but there is only one possible additional header - with code 100 (Continue). For this header you can go around with correctly defining request option: curl_setopt($ch, CURLOPT_HTTPHEADER, array('Expect:')); , disabling sending this header response. As for 302, this should not be happened, because 302 header is redirect, it not expecting body, however i know, sometimes servers send some body with 302 response, but it will be anyway ignored by browsers, so far, why curl should handle this?)

http - Can PHP cURL retrieve response headers AND body in a single req...

php http curl
Rectangle 27 35

I find that Fiddler is my weapon of choice, although there are firefox plugins as well. Fiddler will show you all the HTTP Requests, their response status, all the headers, with different views (raw, hex, image), a timeline view, HTTPS Connects, everything.

In their site, they offer to install some addons but after virustotal test, there's a trojan hic virustotal.com/file-scan/

http - Tool to view response headers - Stack Overflow

http browser networking http-headers
Rectangle 27 4

OpenURI open returns an object which exposes the HTTP response's headers, so you can get the expected byte count from the Content-Length header and compare it to the return value of IO.copy_stream:

require 'open-uri'
download = open 'http://cdn.sstatic.net/stackoverflow/img/apple-touch-icon.png'
bytes_expected = download.meta['content-length']
bytes_copied = IO.copy_stream download, 'image.png'
if bytes_expected != bytes_copied
  raise "Expected #{bytes_expected} bytes but got #{bytes_copied}"
end

It would be surprising if open executed without error and this check still failed, but you never know.

ruby - How to verify that IO.copy_stream was successful - Stack Overfl...

ruby
Rectangle 27 33

I use the httpfox Firefox extension to view all HTTP requests and responses (including header and body) initiated from the web browser in the interval I specify (by pressing the start-recording and the stop-recording buttons). httpfox presents the data in a structured way in which I can quickly find the request I am interested in. It nicely complements LiveHTTPHeaders, because httpfox can view requests in the past as well.

http - Tool to view response headers - Stack Overflow

http browser networking http-headers