Re: Can not set high msize with virtio-9p (Was: Re: virtiofs vs 9p performance)

[Christian Schoenebeck]

On Mittwoch, 3. März 2021 15:50:37 CET Dominique Martinet wrote:
> Christian Schoenebeck wrote on Wed, Mar 03, 2021 at 03:04:21PM +0100:
> > > We can definitely increase the default, for all transports in my
> > > opinion.
> > > As a first step, 64 or 128k?
> > 
> > Just to throw some numbers first; when linearly reading a 12 GB file on
> > guest (i.e. "time cat test.dat > /dev/null") on a test machine, these are
> > the results that I get (cache=mmap):
> > 
> > msize=16k: 2min7s (95 MB/s)
> > msize=64k: 17s (706 MB/s)
> > msize=128k: 12s (1000 MB/s)
> > msize=256k: 8s (1500 MB/s)
> > msize=512k: 6.5s (1846 MB/s)
> > 
> > Personally I would raise the default msize value at least to 128k.
> 
> Thanks for the numbers.
> I'm still a bit worried about too large chunks, let's go with 128k for
> now -- I'll send a couple of patches increasing the tcp max/default as
> well next week-ish.

Ok, sounds good!

> > Personally I don't find raw I/O the worst performance issue right now. As
> > you can see from the numbers above, if 'msize' is raised and I/O being
> > performed with large chunk sizes (e.g. 'cat' automatically uses a chunk
> > size according to the iounit advertised by stat) then the I/O results are
> > okay.
> > 
> > What hurts IMO the most in practice is the sluggish behaviour regarding
> > dentries ATM. The following is with cache=mmap (on guest side):
> > 
> > $ time ls /etc/ > /dev/null
> > real    0m0.091s
> > user    0m0.000s
> > sys     0m0.044s
> > $ time ls -l /etc/ > /dev/null
> > real    0m0.259s
> > user    0m0.008s
> > sys     0m0.016s
> > $ ls -l /etc/ | wc -l
> > 113
> > $
> 
> Yes, that is slow indeed.. Unfortunately cache=none/mmap means only open
> dentries are pinned, so that means a load of requests everytime.
> 
> I was going to suggest something like readdirplus or prefetching
> directory entries attributes in parallel/background, but since we're not
> keeping any entries around we can't even do that in that mode.
> 
> > With cache=loose there is some improvement; on the first "ls" run (when
> > its
> > not in the dentry cache I assume) the results are similar. The subsequent
> > runs then improve to around 50ms for "ls" and around 70ms for "ls -l".
> > But that's still far from numbers I would expect.
> 
> I'm surprised cached mode is that slow though, that is worth
> investigating.
> With that time range we are definitely sending more requests to the
> server than I would expect for cache=loose, some stat revalidation
> perhaps? I thought there wasn't any.

Yes, it looks like more 9p requests are sent than actually required for 
readdir. But I haven't checked yet what's going on there in detail. That's 
definitely on my todo list, because this readdir/stat/direntry issue ATM 
really hurts the most IMO.

> I don't like cache=loose/fscache right now as the reclaim mechanism
> doesn't work well as far as I'm aware (I've heard reports of 9p memory
> usage growing ad nauseam in these modes), so while it's fine for
> short-lived VMs it can't really be used for long periods of time as
> is... That's been on my todo for a while too, but unfortunately no time
> for that.

Ok, that's new to me. But I fear the opposite is currently worse; with 
cache=mmap and running a VM for a longer time: 9p requests get slower and 
slower, e.g. at a certain point you're waiting like 20s for one request. I 
haven't investigated the cause here either yet. It may very well be an issue 
on QEMU side: I have some doubts in the fid reclaim algorithm on 9p server 
side which is using just a linked list. Maybe that list is growing to 
ridiculous sizes and searching the list with O(n) starts to hurt after a 
while.

With cache=loose I don't see such tremendous slowdowns even on long runs, 
which might indicate that this symptom might indeed be due to a problem on 
QEMU side.

> Ideally if that gets fixed, it really should be the default with some
> sort of cache revalidation like NFS does (if that hasn't changed, inode
> stats have a lifetime after which they get revalidated on access, and
> directory ctime changes lead to a fresh readdir) ; but we can't really
> do that right now if it "leaks".
> 
> Some cap to the number of open fids could be appreciable as well
> perhaps, to spare server resources and keep internal lists short.

I just reviewed the fid reclaim code on 9p servers side to some extent because 
of a security issue on 9p server side in this area recently, but I haven't 
really thought through nor captured the authors' original ideas behind it 
entirely yet. I still have some question marks here. Maybe Greg feels the 
same.

Probably when support for macOS is added (also on my todo list), then the 
amount of open fids needs to be limited anyway. Because macOS is much more 
conservative and does not allow a large number of open files by default.

> Anyway, there definitely is room for improvement; if you need ideas I
> have plenty but my time is more than limited right now and for the
> forseeable future... 9p work is purely on my freetime and there isn't
> much at the moment :(
> 
> I'll make time as necessary for reviews & tests but that's about as much
> as I can promise, sorry and good luck!

I fear that applies to all developers right now. To my knowledge there is not 
a single developer either paid and/or able to spend reasonable large time 
slices on 9p issues.

From my side: my plans are to hunt down the worst 9p performance issues in 
order of their impact, but like anybody else, when I find some free time 
slices for that.

#patience #optimistic

Best regards,
Christian Schoenebeck