Re: [Wget-dev] wget2 | Recursive downloads with HTTP/2 not uniformly divided (#397)

[Darshit Shah]

> I have never seen this. That means downloading single files would suffer from 
> this as well. And people are very keen on getting the bandwidth they pay for. 
> That would mean heavy trouble for an ISP.
> I think it's more likely that the server(s) themselves limit the bandwidth or 
> are limited in some way (bandwidth, cpu, ...). Using more parallel 
> connections can circumvent this if you think about load-balancers (which are 
> very common) or other means of distributing traffic to several servers.

I have first hand seen this happen with multiple servers that I know are 
capable of delivering higher bandwidth per connection. Unfortunately, the ISPs 
are getting away with this and a lot more. But I won't go into that now. Just 
that this is a problem for some parts in the world.


> 
> And please don't forget why HTTP/2 was introduced. The main reason was to put 
> _less_ load on the server by just having _one_ connection instead of many. 
> BTW, curl just raised it's default request window to 100 for HTTP/2 (from 20 
> or 30 before).
> 
Oh wow! I know what the point of HTTP/2 is. But a 100 simultaneous files seems 
like a little like overkill to me. What do you think?

> Then, we already have `--http2-request-window` to configure the max parallel 
> requests per HTTP/2 connection. If you really experience regular `throttling` 
> on a single connection, put this into your config file. HTTP/1.1 requests 
> will be shared through all threads anyways, since even with persistent 
> connections we do just one request/response cycle after the other. Putting 
> this value down to 1 means you are basically back at HTTP/1 speed and 
> behavior (maybe even worse since some HTTP/2 protocol overhead).

True. But this is exactly the reason I proposed a new switch. Since 
`http2-request-window` configures the max parallelism per thread and not the 
total max parallel files. The issue here is that, with the default of 5 
threads, it creates confusing output for the end user. Where 25 files are 
downloaded in the single thread while the other 4 threads download only a 
single file and then stop.

My proposal hence, was so that we give the end user the most options. If they 
set, `--max-parallel-downloads` then it automatically sets the number of 
threads accordingly. But if the user wants more control and would like to 
spread it parallelism over multiple threads, then they can manually set the 
thread count and request window options

-- 
Reply to this email directly or view it on GitLab: 
https://gitlab.com/gnuwget/wget2/issues/397#note_103416768
You're receiving this email because of your account on gitlab.com.