qemu.org server bandwidth report (May 2021)

[Stefan Hajnoczi]

Hi,
A few months ago qemu.org hit network bandwidth limits and incurred
costs for exceeding them. Since then we have implemented several
changes to make continuous integration systems more
bandwidth-efficient and reduce the biggest sources of traffic to
qemu.org.

During the Mar-Apr billing cycle qemu.org still exceeded its network
bandwidth limit but only by a small amount. Bandwidth consumption
needs to be under ~6-7 TB/month. Below are the details of how we're
doing.

Thank you to Paolo Bonzini, Thomas Huth, Philippe Mathieu-Daudé,
Daniel Berrangé, and everyone who helped with bandwidth reduction. The
main change was a move to GitLab.com, which now serves the main QEMU
git repository URLs. We also updated documentation and links to
encourage people to use these new URLs.

qemu.org bandwidth usage has been as follows:
- Jan: 12.56 TB
- Feb: 10.55 TB
- Mar: 10.28 TB
- Apr: 7.62 TB

In May qemu.org has averaged 232.25 GB/day so far putting it on track
for 7 TB total this month.

The top 3 web traffic users are Google Cloud and Amazon Web Services
IP addresses. This suggests that some continuous integration systems
are still accessing qemu.org git repositories. It is unlikely that
these are crawlers because User-Agent web stats show that crawlers
only consume a few GB whereas the top three hosts consume 10s or 100s
of GB each.

Roughly 75% of traffic is git (https), 25% is tarball downloads, and
the rest is wiki/web/miscellaneous traffic. Fun fact:
qemu-4.2.0.tar.xz is the most popular download!

I will send another update in 2 months so we can see where bandwidth
usage finally settled. At that point we can decide whether more steps
are necessary.

Thanks,
Stefan