Re: [Qemu-devel] Adding a persistent writeback cache to qemu

[Sage Weil]

On Fri, 21 Jun 2013, Alex Bligh wrote:
> Sage,
> 
> --On 20 June 2013 08:58:19 -0700 Sage Weil <address@hidden> wrote:
> 
> > > I'd like to hear from Ceph folks what their position on kernel rbd vs
> > > librados is.  Why one do they recommend for QEMU guests and what are the
> > > pros/cons?
> > 
> > I agree that a flashcache/bcache-like persistent cache would be a big win
> > for qemu + rbd users.
> 
> Great.
> 
> I think Stefan was really after testing my received wisdom that
> ceph+librbd will be greater performance than ceph+blkdev+kernelrbd
> (even without the persistent cache), and if so why.

Oh, right.  At this point the performance differential is strictly related 
to the cache behavior.  If there were feature parity, I would not expect 
any significant difference.  There may be a net difference of a data copy, 
but I'm not sure it will be significant.

> > There are few important issues with librbd vs kernel rbd:
> > 
> >  * librbd tends to get new features more quickly that the kernel rbd
> >    (although now that layering has landed in 3.10 this will be less
> >    painful than it was).
> > 
> >  * Using kernel rbd means users need bleeding edge kernels, a non-starter
> >    for many orgs that are still running things like RHEL.  Bug fixes are
> >    difficult to roll out, etc.
> > 
> >  * librbd has an in-memory cache that behaves similar to an HDD's cache
> >    (e.g., it forces writeback on flush).  This improves performance
> >    significantly for many workloads.  Of course, having a bcache-like
> >    layer mitigates this..
> > 
> > I'm not really sure what the best path forward is.  Putting the
> > functionality in qemu would benefit lots of other storage backends,
> > putting it in librbd would capture various other librbd users (xen, tgt,
> > and future users like hyper-v), and using new kernels works today but
> > creates a lot of friction for operations.
> 
> To be honest I'd not even thought of putting it in librbd (which might
> be simpler). I suspect it might be easier to get patches into librbd
> than into qemu, and that ensuring cache coherency might be simpler.
> If I get time to look at this, would you be interested in taking patches
> for this?

Certainly!  It will be a bit tricky to integrate is a lightweight way, 
however, so I would be sure to sketch out a design before diving too far 
into the coding.  I suspect the best path forward would be to extend the 
ObjectCacher.  This has the added benefit that ceph-fuse and libcephfs 
could benefit at well.

The dev summit for the next release (emperor) is coming up in a few 
weeks.. this would be a great project to submit a blueprint for so we can 
discuss it then.

sage