Please support automatic gunzip when fetching webpages

[Tim Janik]

Hi Tim Rühsen.

For a WebSocket server I am working on, I just looked into the handling
of gzip content in a number of related tools.
On my Ubuntu-22.04 system,  Firefox, WebKit, Chrome, w3m all request
gzip by default. That means they also automatically handle gzip content
responses out of the box.

Lynx-2.9, node-18, Python3-urllib-1.26 do not request gzip encoding by
default, but will auto unzip contents when they receive a
"Content-Encoding: gzip" response.
Wget-1.21.2 is the only http client implementation I could find that
will not honour the "Content-Encoding: gzip" header in the default
settings and just save the response compressed.

To quote Matthew Scharley on SO [1]:
        Supporting Content-Encoding: gzip isn't a requirement of any
        current HTTP specification [...]
        It's important to realise that HTTP request/response is a
        conversation and that most of the headers in a request are just
        that; a request. For the most part, the server on the other end
        is under no obligation to honor any particular headers, and as
        long as they return a valid response that makes sense the client
        on the other end should do their best to make sense of what was
        returned. This includes enabling gzip if the server responds
        that it has used it.

So while wget may be spec compliant by not automatically unzipping
contents, it'd be nice to bring it on par with the other http clients by
auto-unzipping by default if it encounters contents with gzip encoding,
even though it didn't ask for that.
For most users, that should be an improvement over the current behaviour
and will follow the principle of least surprise, esp given the behaviour
of all other HTTP clients.

[1] 
https://stackoverflow.com/questions/11653488/serving-gzipped-content-directly-bad-thing-to-do/11670562#11670562


On 03.02.23 14:40, Tim Rühsen wrote:
> Sorry about your experience. This feature have been added years back:
>    --compression=TYPE          choose compression, one of auto, gzip
> and none. (default: none)
>
> This feature is off by default, but you can add it to your ~/.wgetrc
> file to permanently enable it (see `man wget`).
>
> Nonetheless, no server should serve gzip compressed pages when not
> explicitly asked for via `Accept-Encoding: gzip`.
>
> Regards, Tim


--
Anklang Free Software DAW
https://anklang.testbit.eu/