Re: qemu.org server bandwidth report (May 2021)

[Daniel P . Berrangé]

On Mon, May 10, 2021 at 10:49:19AM +0100, Stefan Hajnoczi wrote:

> qemu.org bandwidth usage has been as follows:
> - Jan: 12.56 TB
> - Feb: 10.55 TB
> - Mar: 10.28 TB
> - Apr: 7.62 TB
> 
> In May qemu.org has averaged 232.25 GB/day so far putting it on track
> for 7 TB total this month.

That decrease seems to show we've had a big effect from moving to
gitlab. Not big enough yet though.

> Roughly 75% of traffic is git (https), 25% is tarball downloads, and
> the rest is wiki/web/miscellaneous traffic. Fun fact:
> qemu-4.2.0.tar.xz is the most popular download!

First git traffic...

When you say  "git (https)" are you exclusively meaning access of
git via https:// protocol URIs, or does that include git:// URIs
too ?

Or are git:/// URI traffic not accounted for at all in your 75%/25%
split there ?

For the https:// URIs should we setup a HTTP redirect ?

When git clones via https it fetches some specific paths which
I believe we have rules for in httpd conf:

  ScriptAliasMatch "^/git/(.*\.git/(HEAD|info/refs))$" \
    /usr/libexec/git-core/git-http-backend/$1
  ScriptAliasMatch "^/git/(.*\.git/git-(upload|receive)-pack)$" \
    /usr/libexec/git-core/git-http-backend/$1

If we set those URI path matches to send a HTTP 307 redirect
to gitlab, that would essentially kill off our git traffic on
qemu.org, while still allowing the qemu.org gitweb UI to
work normally. The downside is that people won't notice to
update their clone URIs. Still feels like an easy win and
we can easily remove the redirect if we use code 307.



Second tarball traffic...

The qemu-5.2.0.tar.xz file is 102 MB in size

This is quite ridiculous and is directly caused by the number of
binary blobs we're bundling and the corresponding need to bundle
their source.

In fact 66% of this is EDK2's fault - just removing the EDK2
ROMs and source code drops it to 38 MB.

Deleting capstone, dtc, slirp and meson saves 2 MB compressed.

Deleting all remaining contents or roms/ gets us down to 14 MB

Of course we likely want to provide ROMs as a convenience to
users who are not distro vendors, but we could perhaps do that
in a more flexible way.

Even users who want the ROMs likely don't need all of them.


Third, qemu 4.2.0....

I wonder why this is the most popular. Something must be linking
to this, as you would otherwise have to go out of your way to
search it out.

Do we have any stats on the referrer URLs ?

I wonder if there's some key page(s) that need updating ?

If we're unlucky there might be some CI system that hardcoded
use of qemu 4.2.0 that's frequently pulling it.

> I will send another update in 2 months so we can see where bandwidth
> usage finally settled. At that point we can decide whether more steps
> are necessary.




Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|