Rectangle 27 0

python Is it a bug to omit an Accept ** header in an HTTP1.0 Request for a REST API?


Accept: */*
Be conservative in what you do, be liberal in what you accept from others (often reworded as "Be conservative in what you send, be liberal in what you accept").
  • Just adding header fields like Accept: */* which could be omitted increases network traffic by 11 Byte for every single request which might lead to performance issues. Having Accept: */* be default in the request might make it hard for developers to get it out of the header in order to save to 11 Byte.
  • There is a difference between a specification (or a standard) and a de facto standard. Obviously omitting the header field is perfect according to the specification on the other hand a lot of libraries seem to include this and services like the facebook API behave differently this can be seen as a de facto standard being created and you could jump into the loop and be part of creating it.

Bear in mind,"de facto" standards are great when there are no standards ruling. If there is a standard, there is no such a thing as a "de facto" standard, as anything not conforming with the standard is just a violation (an things conforming are just standard). This problem domain we are talking has a ruling standard. Let's just conform to it.

I have some general thoughts on this issue (which might also contribute to the bugfix discussion):

IMO in an RFC, "can" should not automatically be interpreted to "optional", as you are implying. There are key words like "MAY/MUST" for that. So I don't believe your first interpretation is a valid one, though the second quote does backs the optional case. This comment is not to say that that this answer is wrong. Based on the evidence provided, it seems it is optional. All I'm saying is that you should be careful how you interpret "can". That first quote alone, without more context, is ambiguous.

If no Accept header field is present, then it is assumed that the client accepts all media types.

It is interesting that the facebook response differs in both cases but I guess it is their failure of interpreting the protocol correctly. Though on the other side both responses are obviously correct responses to the request (Which I find a funny twist).

The Accept request-header field can be used to specify certain media types which are acceptable for the response.

This means that omitting the header SHOULD be equivalently interpreted by the server as sending Accept: */* in the sense that the client acceptes all media types in both cases.

This means that the header is optional because it says can be used.

When speaking HTTP/1.0: I would send the accept header since you said it is not specified in the HTTP/1.0 RFC and then I think Postel's law becomes more important. On the other side the Accept header is just optional in http 1.0. The Accept request-header field can be used to indicate a list of media ranges which are acceptable as a response to the request Why would you set an optional header by default?

When speaking HTTP/1.1: Even though (1) und (3) speak for fixing the urllib I would probably follow the specification and the performance argument (2) and omit the header. As stated above the response of facebook in both cases is correct since they are allowed to set the media type to whatever they like. (even though this behaviour seems unintended, weird, and by mistake)

as you pointed out ther RFC also says:

Note
Rectangle 27 0

python Is it a bug to omit an Accept ** header in an HTTP1.0 Request for a REST API?


Accept: */*
Be conservative in what you do, be liberal in what you accept from others (often reworded as "Be conservative in what you send, be liberal in what you accept").
  • Just adding header fields like Accept: */* which could be omitted increases network traffic by 11 Byte for every single request which might lead to performance issues. Having Accept: */* be default in the request might make it hard for developers to get it out of the header in order to save to 11 Byte.
  • There is a difference between a specification (or a standard) and a de facto standard. Obviously omitting the header field is perfect according to the specification on the other hand a lot of libraries seem to include this and services like the facebook API behave differently this can be seen as a de facto standard being created and you could jump into the loop and be part of creating it.

Bear in mind,"de facto" standards are great when there are no standards ruling. If there is a standard, there is no such a thing as a "de facto" standard, as anything not conforming with the standard is just a violation (an things conforming are just standard). This problem domain we are talking has a ruling standard. Let's just conform to it.

I have some general thoughts on this issue (which might also contribute to the bugfix discussion):

IMO in an RFC, "can" should not automatically be interpreted to "optional", as you are implying. There are key words like "MAY/MUST" for that. So I don't believe your first interpretation is a valid one, though the second quote does backs the optional case. This comment is not to say that that this answer is wrong. Based on the evidence provided, it seems it is optional. All I'm saying is that you should be careful how you interpret "can". That first quote alone, without more context, is ambiguous.

If no Accept header field is present, then it is assumed that the client accepts all media types.

It is interesting that the facebook response differs in both cases but I guess it is their failure of interpreting the protocol correctly. Though on the other side both responses are obviously correct responses to the request (Which I find a funny twist).

The Accept request-header field can be used to specify certain media types which are acceptable for the response.

This means that omitting the header SHOULD be equivalently interpreted by the server as sending Accept: */* in the sense that the client acceptes all media types in both cases.

This means that the header is optional because it says can be used.

When speaking HTTP/1.0: I would send the accept header since you said it is not specified in the HTTP/1.0 RFC and then I think Postel's law becomes more important. On the other side the Accept header is just optional in http 1.0. The Accept request-header field can be used to indicate a list of media ranges which are acceptable as a response to the request Why would you set an optional header by default?

When speaking HTTP/1.1: Even though (1) und (3) speak for fixing the urllib I would probably follow the specification and the performance argument (2) and omit the header. As stated above the response of facebook in both cases is correct since they are allowed to set the media type to whatever they like. (even though this behaviour seems unintended, weird, and by mistake)

as you pointed out ther RFC also says:

Note
Rectangle 27 0

python Is it a bug to omit an Accept ** header in an HTTP1.0 Request for a REST API?


application/ecmascript
application/javascript
application/json
application/json; charset=UTF-8
charset
javascript/json
text/ecmascript
text/javascript

A request without any Accept header field implies that the user agent will accept any media type in response.

It doesn't matter whether the request is HTTP/1.0 or HTTP/1.1, modern servers always respond with HTTP/1.1. So to be considered up-to-date by servers, user agents should include Accept header in requests. And also Accept headers participate in content negotiation.

It seems that the server considers user agents that omit an HTTP Accept header less capable and obsolete and from pre-json era. That may be the reason that it sends obsolete MIME media type text/javascript which is obsoleted by application/javascript.

JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32. The default encoding is UTF-8.

That said, Python is using HTTP/1.0 and the RFCs are silent about the effect of omitting the header. @raymond-hettinger

The asterisk "*" character is used to group media types into ranges, with "*/*" indicating all media types and "type/*" indicating all subtypes of that type. ...

This is a thoughtful and interesting answer; however, I don't see any part of the RFC that supports the interpretation that "any can be interpreted as most obsolete or most compatible and all can be interpreted as most recent media type". Do you have a specific link to anything that supports that interpretation? Even a link to some public REST server code that behaves that way would be helpful.

Note
Rectangle 27 0

python Is it a bug to omit an Accept ** header in an HTTP1.0 Request for a REST API?


Facebook has updated (and closed-off) its API since this question was asked, but at the time, here is the scenario that caused the observed effects. For backwards compatibility reasons, Facebook was using content negotiation and responding with text/javascript; charset=UTF-8 when getting the request that either omitted the Accept header or had a browser-like Accept: text/html;text/*;*/*. However, when it received Accept: */*, it returned the more modern application/json; charset=UTF-8. When a proxy server receives a request without an accept header, it can give either one of the cached responses; however, when it gets Accept: */*, it always gives the last response.

Reading-up about proxy servers (like NGinx and Varnish) helped me figure out what is going on.

So here is why you should include the Accept: */* header: If you do, then a caching proxy will alway return the same content type. If omit the header, the response can vary depending on the results of the last user's content negotiation. REST API clients tend to rely on always getting the same content type back every time.

While the presence of an Accept: */* header shouldn't make a difference to a server, it can and likely will make a difference to a proxy server when the response includes a Vary: Accept header. In particular, the proxy server is allowed to cache different results for different or omitted Accept headers.

Note
Rectangle 27 0

python Is it a bug to omit an Accept ** header in an HTTP1.0 Request for a REST API?


  • That not sending and Accept header is perfectly legal for an HTTP client.
  • That when this happens, the client intention is to accept any data representation and that the content negotiation on the service side decides which one to use.

As you pointed out, RFC 2616 already states what is the expected behavior of a service in the absence of an Accept header (That it is equivalent to sending Accept: */*). From the specification we can infer

So in your sample, neither your PHP client nor the service you are invoking are misbehaving. I guess there is nothing to fix there.

Note
Rectangle 27 0

python Is it a bug to omit an Accept ** header in an HTTP1.0 Request for a REST API?


Facebook has updated (and closed-off) its API since this question was asked, but at the time, here is the scenario that caused the observed effects. For backwards compatibility reasons, Facebook was using content negotiation and responding with text/javascript; charset=UTF-8 when getting the request that either omitted the Accept header or had a browser-like Accept: text/html;text/*;*/*. However, when it received Accept: */*, it returned the more modern application/json; charset=UTF-8. When a proxy server receives a request without an accept header, it can give either one of the cached responses; however, when it gets Accept: */*, it always gives the last response.

Reading-up about proxy servers (like NGinx and Varnish) helped me figure out what is going on.

So here is why you should include the Accept: */* header: If you do, then a caching proxy will alway return the same content type. If omit the header, the response can vary depending on the results of the last user's content negotiation. REST API clients tend to rely on always getting the same content type back every time.

While the presence of an Accept: */* header shouldn't make a difference to a server, it can and likely will make a difference to a proxy server when the response includes a Vary: Accept header. In particular, the proxy server is allowed to cache different results for different or omitted Accept headers.

Note