Re: [Bug-wget] Benchmarking

[Tim Rühsen]

Hi Marcel,

good to hear it works for you :-)

I forgot: There are also two python scripts in contrib/ to convert the
csv to a tree view and to JSON, namely 'csv2tree' and 'csv2json'.

Regards, Tim

On 15.09.19 17:10, Marcel Schaible wrote:
> Hi Tim,
> 
> thanks for your help. Works like a charm!
> 
> Kind regards
> 
> Marcel
> 
> On 9/14/19 11:30 PM, Tim Rühsen wrote:
>> Hi Marcel,
>>
>> wget1.x can only download one file at a time. Of course you can use
>> multiple instances in parallel.
>>
>> Wget2 uses 5 parallel threads per default (can be tuned with
>> --max-threads=N), so no need for parallel instances. If you give e.g. 5
>> or less URLs, they are downloaded in parallel without the need for
>> tuning.
>> Here comes a little complication... HTTP/2 behaves slightly different
>> than HTTP/1.1 as it allows parallel requests (streams) on one connection
>> - if all URLs have the same domain.
>>
>> Next, Wget2 has a stats module that allows to generate one data record
>> per URL including timings (millisecond precision), see --stats-site.
>>
>> In the Wget2 repository are also some benchmark scripts in subdir
>> benchmarks/.
>>
>> Example:
>> $ wget2 --stats-site=csv:- www.gnu.org www.heise.de www.oracle.de
>> ...
>> ID,ParentID,URL,Status,Link,Method,Size,SizeDecompressed,TransferTime,ResponseTime,Encoding,Verification,Last-Modified,Content-Type
>>
>> 1,0,http://www.gnu.org,200,1,1,9168,28592,235,234,1,0,0,text/html
>> 2,0,https://www.heise.de,200,1,1,58158,434935,53,27,1,0,1568496351,text/html
>>
>> 3,0,http://www.oracle.de,301,1,1,217,270,341,341,1,0,0,text/html
>> 4,0,http://www.oracle.com/de/index.html,301,0,1,0,0,180,180,0,0,0,(null)
>> 5,0,https://www.oracle.com/de/index.html,200,0,1,8511,32215,160,160,1,0,1568450074,text/html
>>
>>
>> It also records the redirections as you can see. See docs/wget2.md for
>> more info regarding the fields.
>>
>> Regards, Tim
>>
>> On 14.09.19 17:33, Schaible, Marcel wrote:
>>> Hi,
>>>
>>>
>>> I am wondering if wget is capable of being used as a benchmark tool
>>> for web servers.
>>>
>>>
>>> I just come back from a programming course for beginners and we have
>>> setup for the  attendees a small webserver (nginx) as download hub
>>> for some installation packages. The server was dead slow and
>>> practically unusable. For the next course I want to test this upfront.
>>>
>>>
>>> So I am looking for some wget options to start a couple of
>>> concurrecnt connections to my server and download files, preferable a
>>> set of different files.
>>>
>>>
>>> Can this be done with wget.
>>>
>>>
>>> Thanks
>>>
>>>
>>> Marcel
>>>

signature.asc
Description: OpenPGP digital signature