Re: [PATCH v2 000/109] virtiofs daemon [all]

[Philippe Mathieu-Daudé]

On 1/21/20 1:22 PM, Dr. David Alan Gilbert (git) wrote:
> From: "Dr. David Alan Gilbert" <address@hidden>
>
> Hi,
>    This is a full set for virtiofsd - a daemon
> that implements the user space side of virtiofs.
>
> I've addressed most review comments from V1; we've got
> 95 Reviewed-by's out of 110 - I'd like to get a PULL out ASAP.
>
> The set pulls in a big chunk of the upstream libfuse library
> (unmodified so that it's easy to check it really is upstream),
> chops all the stuff out we don't need and then adds the
> new transport we need.  I've formatted everything into qemu's code style -
> using indent and the clang tools for the files I've imported.
>
> We can't just link with libfuse, since we have to make ABI incompatible
> changes for the new transport and it's quite invasive; the library is
> designed to be the basis for multiple filesystems, but all on the same
> transport.
>
> Running this daemon is typically done with:
>
>     ./virtiofsd --socket-path=/path/socket -o source=/path/to/fs
>
> connected to a qemu that's then started with:
>     -chardev socket,id=char0,path=/path/socket -device 
> vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs
>
> and then in the guest mount with:
>     mount -t virtiofs myfs /mnt
>
> Our development branch is: https://gitlab.com/virtio-fs/qemu/tree/virtio-fs-dev

FWIW this branch contains more than this series, because it failed to 
build on 32-bit:

$ make virtiofsd
  CC      tools/virtiofsd/fuse_lowlevel.o
tools/virtiofsd/fuse_lowlevel.c: In function 'do_removemapping':
tools/virtiofsd/fuse_lowlevel.c:1956:60: error: format '%ld' expects 
argument of type 'long int', but argument 4 has type 'unsigned int' 
[-Werror=format=]
             "do_removemapping: invalid in, expected %d * %ld, has %ld 
- %ld\n",
                                                            ^
tools/virtiofsd/fuse_lowlevel.c:1956:69: error: format '%ld' expects 
argument of type 'long int', but argument 5 has type 'size_t {aka 
unsigned int}' [-Werror=format=]
             "do_removemapping: invalid in, expected %d * %ld, has %ld 
- %ld\n",
                                                                     ^
tools/virtiofsd/fuse_lowlevel.c:1956:75: error: format '%ld' expects 
argument of type 'long int', but argument 6 has type 'size_t {aka 
unsigned int}' [-Werror=format=]
             "do_removemapping: invalid in, expected %d * %ld, has %ld 
- %ld\n",

   ^

And the offending patch is not included in the 109:

$ git show 62d82cfbe5a
commit 62d82cfbe5a2ad05ae3215f2461b76aa9915108d
Author: Dr. David Alan Gilbert <address@hidden>
Date:   Fri Jul 6 18:03:49 2018 +0100

    DAX: virtiofsd: Add setup/remove mappings fuse commands

Consider pushing tags with version, so it is easier to diff 
(virtio-fs-dev got force-pushed).


We don't need libslirp/capstone/dtc but they are still built:

$ make virtiofsd
  GEN     config-host.h
make[1]: Entering directory 'slirp'
  CC      slirp/src/state.o
  CC      slirp/src/tcp_timer.o
  CC      slirp/src/dhcpv6.o
  CC      slirp/src/ip_input.o
  CC      slirp/src/ip6_icmp.o
  CC      slirp/src/bootp.o
  CC      slirp/src/ip6_input.o
  CC      slirp/src/slirp.o
  CC      slirp/src/vmstate.o
  CC      slirp/src/ip_output.o
  CC      slirp/src/ncsi.o
  CC      slirp/src/tcp_output.o
  CC      slirp/src/ndp_table.o
  CC      slirp/src/version.o
  CC      slirp/src/misc.o
  CC      slirp/src/ip6_output.o
  CC      slirp/src/mbuf.o
  CC      slirp/src/tftp.o
  CC      slirp/src/arp_table.o
  CC      slirp/src/util.o
  CC      slirp/src/socket.o
  CC      slirp/src/sbuf.o
  CC      slirp/src/stream.o
  CC      slirp/src/dnssearch.o
  CC      slirp/src/udp.o
  CC      slirp/src/tcp_input.o
  CC      slirp/src/if.o
  CC      slirp/src/cksum.o
  CC      slirp/src/tcp_subr.o
  CC      slirp/src/udp6.o
  CC      slirp/src/ip_icmp.o
  AR      slirp/libslirp.a
make[1]: Leaving directory 'slirp'
         DEP dtc/libfdt/fdt_overlay.c
         DEP dtc/libfdt/fdt_addresses.c
         DEP dtc/libfdt/fdt_empty_tree.c
         DEP dtc/libfdt/fdt_strerror.c
         DEP dtc/libfdt/fdt_rw.c
         DEP dtc/libfdt/fdt_sw.c
         DEP dtc/libfdt/fdt_wip.c
         DEP dtc/libfdt/fdt_ro.c
         DEP dtc/libfdt/fdt.c
         DEP dtc/util.c
         DEP dtc/fdtoverlay.c
         DEP dtc/fdtput.c
         DEP dtc/fdtget.c
         DEP dtc/fdtdump.c
         DEP convert-dtsv0-lexer.lex.c
         DEP dtc/srcpos.c
         BISON dtc-parser.tab.c
         DEP dtc-parser.tab.c
         LEX dtc-lexer.lex.c
         DEP dtc-lexer.lex.c
         DEP dtc/treesource.c
         DEP dtc/livetree.c
         DEP dtc/fstree.c
         DEP dtc/flattree.c
         DEP dtc/dtc.c
         DEP dtc/data.c
         DEP dtc/checks.c
         CC libfdt/fdt.o
         CC libfdt/fdt_ro.o
         CC libfdt/fdt_wip.o
         CC libfdt/fdt_sw.o
         CC libfdt/fdt_rw.o
         CC libfdt/fdt_strerror.o
         CC libfdt/fdt_empty_tree.o
         CC libfdt/fdt_addresses.o
         CC libfdt/fdt_overlay.o
         AR libfdt/libfdt.a
arm-linux-gnueabihf-ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
  CC      cs.o
  CC      utils.o
  CC      SStream.o
  CC      MCInstrDesc.o
  CC      MCRegisterInfo.o
  CC      arch/ARM/ARMDisassembler.o
  CC      arch/ARM/ARMInstPrinter.o
  CC      arch/ARM/ARMMapping.o
  CC      arch/ARM/ARMModule.o
  CC      arch/AArch64/AArch64BaseInfo.o
  CC      arch/AArch64/AArch64Disassembler.o
  CC      arch/AArch64/AArch64InstPrinter.o
  CC      arch/AArch64/AArch64Mapping.o
  CC      arch/AArch64/AArch64Module.o
  CC      arch/Mips/MipsDisassembler.o
  CC      arch/Mips/MipsInstPrinter.o
  CC      arch/Mips/MipsMapping.o
  CC      arch/Mips/MipsModule.o
  CC      arch/PowerPC/PPCDisassembler.o
  CC      arch/PowerPC/PPCInstPrinter.o
  CC      arch/PowerPC/PPCMapping.o
  CC      arch/PowerPC/PPCModule.o
  CC      arch/Sparc/SparcDisassembler.o
  CC      arch/Sparc/SparcInstPrinter.o
  CC      arch/Sparc/SparcMapping.o
  CC      arch/Sparc/SparcModule.o
  CC      arch/SystemZ/SystemZDisassembler.o
  CC      arch/SystemZ/SystemZInstPrinter.o
  CC      arch/SystemZ/SystemZMapping.o
  CC      arch/SystemZ/SystemZModule.o
  CC      arch/SystemZ/SystemZMCTargetDesc.o
  CC      arch/X86/X86DisassemblerDecoder.o
  CC      arch/X86/X86Disassembler.o
  CC      arch/X86/X86IntelInstPrinter.o
  CC      arch/X86/X86ATTInstPrinter.o
  CC      arch/X86/X86Mapping.o
  CC      arch/X86/X86Module.o
  CC      arch/XCore/XCoreDisassembler.o
  CC      arch/XCore/XCoreInstPrinter.o
  CC      arch/XCore/XCoreMapping.o
  CC      arch/XCore/XCoreModule.o
  CC      MCInst.o
  AR      libcapstone.a
  CC      tools/virtiofsd/buffer.o
  CC      tools/virtiofsd/fuse_opt.o
  CC      tools/virtiofsd/fuse_log.o
  CC      tools/virtiofsd/fuse_lowlevel.o
  ...

FYI Paolo queued this patch, so libfdt shouldn't get compiled anymore:
https://www.mail-archive.com/address@hidden/msg671667.html

> This code is going into tools/virtiofsd  based on the previous long
> discussion.
>
> Thank you to all those who have contributed code and to those who have
> reviewed the previous version.
>
> Dave
>
> Dr. David Alan Gilbert (35):
>    virtiofsd: Pull in upstream headers
>    virtiofsd: Pull in kernel's fuse.h
>    virtiofsd: Add auxiliary .c's
>    virtiofsd: Add fuse_lowlevel.c
>    virtiofsd: Add passthrough_ll
>    virtiofsd: Trim down imported files
>    virtiofsd: Format imported files to qemu style
>    virtiofsd: Fix fuse_daemonize ignored return values
>    virtiofsd: Fix common header and define for QEMU builds
>    virtiofsd: Trim out compatibility code
>    virtiofsd: Add options for virtio
>    virtiofsd: Open vhost connection instead of mounting
>    virtiofsd: Start wiring up vhost-user
>    virtiofsd: Add main virtio loop
>    virtiofsd: get/set features callbacks
>    virtiofsd: Start queue threads
>    virtiofsd: Poll kick_fd for queue
>    virtiofsd: Start reading commands from queue
>    virtiofsd: Send replies to messages
>    virtiofsd: Keep track of replies
>    virtiofsd: Add Makefile wiring for virtiofsd contrib
>    virtiofsd: Fast path for virtio read
>    virtiofs: Add maintainers entry
>    virtiofsd: Plumb fuse_bufvec through to do_write_buf
>    virtiofsd: Pass write iov's all the way through
>    virtiofsd: cap-ng helpers
>    docs:  Add docs/tools
>    virtiofsd: Handle reinit
>    virtiofsd: Handle hard reboot
>    virtiofsd: Kill threads when queues are stopped
>    vhost-user: Print unexpected slave message types
>    contrib/libvhost-user: Protect slave fd with mutex
>    virtiofsd: Clean up inodes on destroy
>    libvhost-user: Fix some memtable remap cases
>    virtiofsd: Convert lo_destroy to take the lo->mutex lock itself
>
> Eric Ren (1):
>    virtiofsd: fix incorrect error handling in lo_do_lookup
>
> Eryu Guan (3):
>    virtiofsd: print log only when priority is high enough
>    virtiofsd: convert more fprintf and perror to use fuse log infra
>    virtiofsd: stop all queue threads on exit in virtio_loop()
>
> Jiufei Xue (1):
>    virtiofsd: support nanosecond resolution for file timestamp
>
> Liu Bo (6):
>    virtiofsd: fix error handling in main()
>    virtiofsd: cleanup allocated resource in se
>    virtiofsd: fix memory leak on lo.source
>    virtiofsd: add helper for lo_data cleanup
>    virtiofsd: enable PARALLEL_DIROPS during INIT
>    Virtiofsd: fix memory leak on fuse queueinfo
>
> Masayoshi Mizuma (4):
>    virtiofsd: Add ID to the log with FUSE_LOG_DEBUG level
>    virtiofsd: Add timestamp to the log with FUSE_LOG_DEBUG level
>    virtiofsd: Prevent multiply running with same vhost_user_socket
>    virtiofsd: add some options to the help message
>
> Miklos Szeredi (10):
>    virtiofsd: passthrough_ll: add fallback for racy ops
>    virtiofsd: passthrough_ll: add renameat2 support
>    virtiofsd: passthrough_ll: disable readdirplus on cache=never
>    virtiofsd: passthrough_ll: control readdirplus
>    virtiofsd: rename unref_inode() to unref_inode_lolocked()
>    virtiofsd: fail when parent inode isn't known in lo_do_lookup()
>    virtiofsd: extract root inode init into setup_root()
>    virtiofsd: passthrough_ll: clean up cache related options
>    virtiofsd: passthrough_ll: use hashtable
>    virtiofsd: passthrough_ll: fix refcounting on remove/rename
>
> Misono Tomohiro (1):
>    virtiofsd: Fix data corruption with O_APPEND write in writeback mode
>
> Peng Tao (1):
>    virtiofsd: do not always set FUSE_FLOCK_LOCKS
>
> Stefan Hajnoczi (37):
>    virtiofsd: remove mountpoint dummy argument
>    virtiofsd: remove unused notify reply support
>    virtiofsd: add -o source=PATH to help output
>    virtiofsd: add --fd=FDNUM fd passing option
>    virtiofsd: make -f (foreground) the default
>    virtiofsd: add vhost-user.json file
>    virtiofsd: add --print-capabilities option
>    virtiofsd: passthrough_ll: add lo_map for ino/fh indirection
>    virtiofsd: passthrough_ll: add ino_map to hide lo_inode pointers
>    virtiofsd: passthrough_ll: add dirp_map to hide lo_dirp pointers
>    virtiofsd: passthrough_ll: add fd_map to hide file descriptors
>    virtiofsd: validate path components
>    virtiofsd: add fuse_mbuf_iter API
>    virtiofsd: validate input buffer sizes in do_write_buf()
>    virtiofsd: check input buffer size in fuse_lowlevel.c ops
>    virtiofsd: prevent ".." escape in lo_do_lookup()
>    virtiofsd: prevent ".." escape in lo_do_readdir()
>    virtiofsd: use /proc/self/fd/ O_PATH file descriptor
>    virtiofsd: sandbox mount namespace
>    virtiofsd: move to an empty network namespace
>    virtiofsd: move to a new pid namespace
>    virtiofsd: add seccomp whitelist
>    virtiofsd: set maximum RLIMIT_NOFILE limit
>    virtiofsd: fix libfuse information leaks
>    virtiofsd: add security guide document
>    virtiofsd: add --syslog command-line option
>    virtiofsd: use fuse_lowlevel_is_virtio() in fuse_session_destroy()
>    virtiofsd: prevent fv_queue_thread() vs virtio_loop() races
>    virtiofsd: make lo_release() atomic
>    virtiofsd: prevent races with lo_dirp_put()
>    virtiofsd: rename inode->refcount to inode->nlookup
>    virtiofsd: add man page
>    virtiofsd: introduce inode refcount to prevent use-after-free
>    virtiofsd: process requests in a thread pool
>    virtiofsd: prevent FUSE_INIT/FUSE_DESTROY races
>    virtiofsd: fix lo_destroy() resource leaks
>    virtiofsd: add --thread-pool-size=NUM option
>
> Vivek Goyal (6):
>    virtiofsd: Make fsync work even if only inode is passed in
>    virtiofsd: passthrough_ll: create new files in caller's context
>    virtiofsd: Parse flag FUSE_WRITE_KILL_PRIV
>    virtiofsd: Drop CAP_FSETID if client asked for it
>    virtiofsd: Support remote posix locks
>    virtiofsd: Reset O_DIRECT flag during file open
>
> Xiao Yang (2):
>    vitriofsd/passthrough_ll: fix fallocate() ifdefs
>    virtiofsd/passthrough_ll: Pass errno to fuse_reply_err()
>
> piaojun (2):
>    virtiofsd: add definition of fuse_buf_writev()
>    virtiofsd: use fuse_buf_writev to replace fuse_buf_write for better
>      performance
>
>   .gitignore                                |    1 +
>   MAINTAINERS                               |    8 +
>   Makefile                                  |   27 +-
>   Makefile.objs                             |    1 +
>   configure                                 |   16 +
>   contrib/libvhost-user/libvhost-user.c     |   57 +-
>   contrib/libvhost-user/libvhost-user.h     |    6 +
>   docs/index.rst                            |    1 +
>   docs/interop/vhost-user.json              |    4 +-
>   docs/tools/conf.py                        |   16 +
>   docs/tools/index.rst                      |   14 +
>   docs/tools/virtiofsd-security.rst         |  118 +
>   hw/virtio/vhost-user.c                    |    2 +-
>   include/standard-headers/linux/fuse.h     |  891 ++++++
>   scripts/update-linux-headers.sh           |    1 +
>   tools/virtiofsd/50-qemu-virtiofsd.json.in |    5 +
>   tools/virtiofsd/Makefile.objs             |   12 +
>   tools/virtiofsd/buffer.c                  |  351 +++
>   tools/virtiofsd/fuse.h                    | 1249 +++++++++
>   tools/virtiofsd/fuse_common.h             |  860 ++++++
>   tools/virtiofsd/fuse_i.h                  |  115 +
>   tools/virtiofsd/fuse_log.c                |   40 +
>   tools/virtiofsd/fuse_log.h                |   74 +
>   tools/virtiofsd/fuse_lowlevel.c           | 2779 +++++++++++++++++++
>   tools/virtiofsd/fuse_lowlevel.h           | 2023 ++++++++++++++
>   tools/virtiofsd/fuse_misc.h               |   60 +
>   tools/virtiofsd/fuse_opt.c                |  449 +++
>   tools/virtiofsd/fuse_opt.h                |  272 ++
>   tools/virtiofsd/fuse_signals.c            |   97 +
>   tools/virtiofsd/fuse_virtio.c             |  985 +++++++
>   tools/virtiofsd/fuse_virtio.h             |   33 +
>   tools/virtiofsd/helper.c                  |  349 +++
>   tools/virtiofsd/passthrough_helpers.h     |   51 +
>   tools/virtiofsd/passthrough_ll.c          | 3006 +++++++++++++++++++++
>   tools/virtiofsd/seccomp.c                 |  164 ++
>   tools/virtiofsd/seccomp.h                 |   16 +
>   tools/virtiofsd/virtiofsd.texi            |  104 +
>   37 files changed, 14242 insertions(+), 15 deletions(-)
>   create mode 100644 docs/tools/conf.py
>   create mode 100644 docs/tools/index.rst
>   create mode 100644 docs/tools/virtiofsd-security.rst
>   create mode 100644 include/standard-headers/linux/fuse.h
>   create mode 100644 tools/virtiofsd/50-qemu-virtiofsd.json.in
>   create mode 100644 tools/virtiofsd/Makefile.objs
>   create mode 100644 tools/virtiofsd/buffer.c
>   create mode 100644 tools/virtiofsd/fuse.h
>   create mode 100644 tools/virtiofsd/fuse_common.h
>   create mode 100644 tools/virtiofsd/fuse_i.h
>   create mode 100644 tools/virtiofsd/fuse_log.c
>   create mode 100644 tools/virtiofsd/fuse_log.h
>   create mode 100644 tools/virtiofsd/fuse_lowlevel.c
>   create mode 100644 tools/virtiofsd/fuse_lowlevel.h
>   create mode 100644 tools/virtiofsd/fuse_misc.h
>   create mode 100644 tools/virtiofsd/fuse_opt.c
>   create mode 100644 tools/virtiofsd/fuse_opt.h
>   create mode 100644 tools/virtiofsd/fuse_signals.c
>   create mode 100644 tools/virtiofsd/fuse_virtio.c
>   create mode 100644 tools/virtiofsd/fuse_virtio.h
>   create mode 100644 tools/virtiofsd/helper.c
>   create mode 100644 tools/virtiofsd/passthrough_helpers.h
>   create mode 100644 tools/virtiofsd/passthrough_ll.c
>   create mode 100644 tools/virtiofsd/seccomp.c
>   create mode 100644 tools/virtiofsd/seccomp.h
>   create mode 100644 tools/virtiofsd/virtiofsd.texi