[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images
From: |
Alberto Garcia |
Subject: |
[Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images |
Date: |
Thu, 27 Jun 2019 16:59:14 +0300 |
User-agent: |
NeoMutt/20170113 (1.7.2) |
Hi all,
a couple of years ago I came to the mailing list with a proposal to
extend the qcow2 format to add subcluster allocation.
You can read the original message (and the discussion thread that came
afterwards) here:
https://lists.gnu.org/archive/html/qemu-block/2017-04/msg00178.html
The description of the problem from the original proposal is still
valid so I won't repeat it here.
What I have been doing during the past few weeks was to retake the
code that I wrote in 2017, make it work with the latest QEMU and fix
many of its bugs. I have again a working prototype which is by no
means complete but it allows us to have up-to-date information about
what we can expect from this feature.
My goal with this message is to retake the discussion and re-evaluate
whether this is a feature that we'd like for QEMU in light of the test
results and all the changes that we have had in the past couple of
years.
=== Test results ===
I ran these tests with the same hardware configuration as in 2017: an
SSD drive and random 4KB write requests to an empty 40GB qcow2 image.
Here are the results when the qcow2 file is backed by a fully
populated image. There are 8 subclusters per cluster and the
subcluster size is in brackets:
|-----------------+----------------+-----------------|
| Cluster size | subclusters=on | subclusters=off |
|-----------------+----------------+-----------------|
| 2 MB (256 KB) | 571 IOPS | 124 IOPS |
| 1 MB (128 KB) | 863 IOPS | 212 IOPS |
| 512 KB (64 KB) | 1678 IOPS | 365 IOPS |
| 256 KB (32 KB) | 2618 IOPS | 568 IOPS |
| 128 KB (16 KB) | 4907 IOPS | 873 IOPS |
| 64 KB (8 KB) | 10613 IOPS | 1680 IOPS |
| 32 KB (4 KB) | 13038 IOPS | 2476 IOPS |
| 4 KB (512 B) | 101 IOPS | 101 IOPS |
|-----------------+----------------+-----------------|
Some comments about the results, after comparing them with those from
2017:
- As expected, 32KB clusters / 4 KB subclusters give the best results
because that matches the size of the write request and therefore
there's no copy-on-write involved.
- Allocation is generally faster now in all cases (between 20-90%,
depending on the case). We have made several optimizations to the
code since last time, and I suppose that the COW changes made in
commits b3cf1c7cf8 and ee22a9d869 are probably the main factor
behind these improvements.
- Apart from the 64KB/8KB case (which is much faster), the patters are
generally the same: subcluster allocation offers similar performance
benefits compared to last time, so there are no surprises in this
area.
Then I ran the tests again using the same environment but without a
backing image. The goal is to measure the impact of subcluster
allocation on completely empty images.
Here we have an important change: since commit c8bb23cbdb empty
clusters are preallocated and filled with zeroes using an efficient
operation (typically fallocate() with FALLOC_FL_ZERO_RANGE) instead of
writing the zeroes with the usual pwrite() call.
The effects of this are dramatic, so I decided to run two sets of
tests: one with this optimization and one without it.
Here are the results:
|-----------------+----------------+-----------------+----------------+-----------------|
| | Initialization with fallocate() | Initialization with
pwritev() |
|-----------------+----------------+-----------------+----------------+-----------------|
| Cluster size | subclusters=on | subclusters=off | subclusters=on |
subclusters=off |
|-----------------+----------------+-----------------+----------------+-----------------|
| 2 MB (256 KB) | 14468 IOPS | 14776 IOPS | 1181 IOPS | 260
IOPS |
| 1 MB (128 KB) | 13752 IOPS | 14956 IOPS | 1916 IOPS | 358
IOPS |
| 512 KB (64 KB) | 12961 IOPS | 14776 IOPS | 4038 IOPS | 684
IOPS |
| 256 KB (32 KB) | 12790 IOPS | 14534 IOPS | 6172 IOPS | 1213
IOPS |
| 128 KB (16 KB) | 12550 IOPS | 13967 IOPS | 8700 IOPS | 1976
IOPS |
| 64 KB (8 KB) | 12491 IOPS | 13432 IOPS | 11735 IOPS | 4267
IOPS |
| 32 KB (4 KB) | 13203 IOPS | 11752 IOPS | 12366 IOPS | 6306
IOPS |
| 4 KB (512 B) | 103 IOPS | 101 IOPS | 101 IOPS | 101
IOPS |
|-----------------+----------------+-----------------+----------------+-----------------|
Comments:
- With the old-style allocation method using pwritev() we get similar
benefits as we did last time. The comments from the test with a
backing image apply to this one as well.
- However the new allocation method is so efficient that having
subclusters does not offer any performance benefit. It even slows
down things a bit in most cases, so we'd probably need to fine tune
the algorithm in order to get similar results.
- In light of this numbers I also think that even when there's a
backing image we could preallocate the full cluster but only do COW
on the affected subclusters. This would the rest of the cluster
preallocated on disk but unallocated on the bitmap. This would
probably reduce on-disk fragmentation, which was one of the concerns
raised during the original discussion.
I also ran some tests on a rotating HDD drive. Here having subclusters
doesn't make a big difference regardless of whether there is a backing
image or not, so we can ignore this scenario.
=== Changes to the on-disk format ===
In my original proposal I described 3 different alternatives for
storing the subcluster bitmaps. I'm naming them here, but refer to
that message for more details.
(1) Storing the bitmap inside the 64-bit entry
(2) Making L2 entries 128-bit wide.
(3) Storing the bitmap somewhere else
I used (1) for this implementation for simplicity, but I think (2) is
probably the best one.
===========================
And I think that's all. As you can see I didn't want to go much into
the open technical questions (I think the on-disk format would be the
main one), the first goal should be to decide whether this is still an
interesting feature or not.
So, any questions or comments will be much appreciated.
Berto
- [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images,
Alberto Garcia <=
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Denis Lunev, 2019/06/27
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Alberto Garcia, 2019/06/27
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Alberto Garcia, 2019/06/27
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Kevin Wolf, 2019/06/28
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Alberto Garcia, 2019/06/28
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Kevin Wolf, 2019/06/28
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Alberto Garcia, 2019/06/28
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Kevin Wolf, 2019/06/28
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Alberto Garcia, 2019/06/28
- Re: [Qemu-block] [RFC] Re-evaluating subcluster allocation for qcow2 images, Denis Lunev, 2019/06/27