[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Bug-wget] [bug #56946] Performance of `-r -A` is poor
|
From: |
Tim Ruehsen |
|
Subject: |
[Bug-wget] [bug #56946] Performance of `-r -A` is poor |
|
Date: |
Wed, 25 Sep 2019 08:18:56 -0400 (EDT) |
|
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0 |
Update of bug #56946 (project wget):
Assigned to: None => rockdaboot
Open/Closed: Open => Closed
_______________________________________________________
Follow-up Comment #2:
The reason is, due to -r wget has to make HEAD requests to *all* URLs
(>280.000) to find out about the content-type (in case it find text/html or
text/css, it will scan those as well and follow the found URLs).
What you want is to filter out the URLs before probing. Wget1.x doesn't have
this option. Wget2 has --filter-urls which does exactly that. Together with
-l1 the download with Wget2 takes here 3s (250Mbit/s). The list contains
Saving 'ca-certificates-mono_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_i386.deb'
Saving
'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_amd64.deb'
Saving 'libmono-cairo4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-codecontracts4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-corlib2.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-cscompmgd0.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-csharp4.0c-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-custommarshalers4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-db2-1.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-http4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-corlib4.5-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_amd64.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_arm64.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_ppc64el.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_arm64.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_armhf.deb'
Saving 'libmono-2.0-1_6.0.0.334-0xamarin1+ubuntu1604b1_armhf.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_i386.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_ppc64el.deb'
Saving
'libmono-accessibility4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-cil-dev_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_arm64.deb'
Saving 'libmono-2.0-dev_6.0.0.334-0xamarin1+ubuntu1604b1_amd64.deb'
Saving 'libmono-cecil-private-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_ppc64el.deb'
Saving
'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_i386.deb'
Saving
'libmono-btls-interface4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_armhf.deb'
Saving
'libmono-microsoft-build-utilities-v4.0-4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-corlib4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-compilerservices-symbolwriter4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-data-tds4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-debugger-soft4.0a-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-cjk4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-mideast4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-other4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-rare4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n-west4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n4.0-all_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-i18n4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-messaging4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-management4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-ldap4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-messaging-rabbitmq4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-microsoft-build-engine4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-microsoft-build-tasks-v4.0-4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-microsoft-build4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-microsoft-build-framework4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-microsoft-web-infrastructure1.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-parallel4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-peapi4.0a-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_i386.deb'
Saving 'libmono-rabbitmq4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-relaxng4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_ppc64el.deb'
Saving 'libmono-security4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-microsoft-visualc10.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving
'libmono-microsoft-csharp4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-oracle4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_amd64.deb'
Saving 'libmono-posix4.0-cil_6.0.0.334-0xamarin1+ubuntu1604b1_all.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_arm64.deb'
Saving 'libmono-profiler_6.0.0.334-0xamarin1+ubuntu1604b1_armhf.deb'
The solution with wget1.x is indeed to download the html file and manually
scan the wanted URLs and download those in an extra step.
_______________________________________________________
Reply to this item at:
<https://savannah.gnu.org/bugs/?56946>
_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/