Re: [Wget-dev] wget2 | Recursive downloads with HTTP/2 not uniformly divided (#397)

[Tim Rühsen]

So we agree to first make some benchmarks.

Then, while `--max-threads` is easily understandable by normal users (we could 
use -j as short option here, that seems very common).

`--max-parallel-downloads` has an unpredictable behavior. You don't know in 
advance if a server supports HTTP/2 or not. Also, our queue can be filled with 
jobs for several servers, so that we get a mix of simultaneous HTTP/1 and 
HTTP/2 connections. So you can easily explain to a user what will happen if he 
uses this option and how it play together with `--max-threads`.

What we could do is calculating the size of the HTTP/2 window size dynamically. 
E.g. there are 30 jobs for one server/ip and 5 threads. The first thread in 
charge could just take 'number of available jobs' / 'number of idling threads' 
(rounded up). That would give each thread 6 jobs (=using a total of 5 
connections).
When the first download is ready, let's say another 10 new jobs are generated. 
Then that thread takes 2 of these jobs (10 / 5), effectively increasing it's 
own window size by 1 (10 / 5 minus the one that is already done). Sounds easy, 
but we might have some very interesting corner cases ;-).

Another, maybe much simpler, approach would be if a thread just takes one job, 
sends the request and then gives away scheduling to another task. That in turn 
makes the same and so on. If CPU is ~ evenly distributed between threads, we 
automatically get a distribution of jobs. We just have to introduce a call to 
`pthread_yield()` (or a no-op if pthreads are not available), which is elegant 
and easy to implement/test. But first benchmarks to have a comparison.

-- 
Reply to this email directly or view it on GitLab: 
https://gitlab.com/gnuwget/wget2/issues/397#note_103513550
You're receiving this email because of your account on gitlab.com.