Re: Attaching 9P VirtFS to PowerNV

[Ryan Burns]

Hi Murilo, thanks for your response.

On 3/3/21 8:37 AM, Murilo Opsfelder Araújo wrote:
> Hi, Ryan.
> 
> On Wednesday, March 3, 2021 4:59:09 AM -03 Ryan Burns wrote:
>> Hi all,
>>
>> I'm trying to port NixOS to run on QEMU's PowerNV platform. The system
>> is able to load the kernel, initrd, and a virtio-blk-pci device without
>> issues, but fails trying to mount the nix store from the host via -virtfs.
>>
>> I noticed that unlike on other platforms, I needed to specify the PCIe
>> bus for the virtio-blk-pci device to appear as /dev/vda. Do I need to
>> do something similar for the 9P virtfs?
>>
>>
>> Here is the relevant snippet from my boot log:
>>
>>
>> /dev/vda: recovering journal
>> /dev/vda: clean, 231/32768 files, 10160/131072 blocks
>> mounting /dev/vda on /...
>> mounting store on /nix/.ro-store...
>> [    5.972587][  T170] 9pnet_virtio: no channels available for device store
>> [    5.972587][  T170] 9pnet_virtio: no channels available for device store
>> mount: mounting store on /mnt-root/nix/.ro-store failed: No such file or
>> directory
> 
> The message suggests that /mnt-root/nix/.ro-store does not exist at the moment
> mount is called.
> 
> Are you 100% sure that the previous mkdir was successful?  Is "/" writable at
> the moment mkdir was called?  Shouldn't the script check for the return code 
> of
> mkdir?
> 
>     https://github.com/NixOS/nixpkgs/blob/
> 545bafa2d49d5184714d94ea4c568316d9ce7b5c/nixos/modules/system/boot/stage-1-
> init.sh#L390
>     mkdir -p "/mnt-root$mountPoint"

The previous mkdir was successful - "/" is always a fresh qcow2 image.
I added some debugging commands to the stage 1 script to be sure -
the mkdir command exits with status 0 and the mountpoint does exist.

> 
> Besides that, I'd also check if filesystem type is missing from mount 
> arguments,
> as per:
> 
>     https://wiki.qemu.org/Documentation/9psetup#Mounting_the_shared_path

The stage 1 script does this by generating an fstab on-the-fly with the
correct options set. Here is what it looks like at mount time:

/dev/vda /mnt-root/ ext4 defaults,
store /mnt-root/nix/.ro-store 9p trans=virtio,version=9p2000.L,cache=loose,

I think this is all correct, right? I might add that these VM scripts are
pretty well-tested on x86 and ARM, which is why I thought I'd ask on this
mailing list - I assume the most likely explanation is that I'm simply
using a faulty QEMU command-line or kernel config.

> 
> Cheers!
> 
>> And here is what my generated qemu command line invocation looks like:
>>
>>
>> exec
>> /nix/store/r2jn54cpvmp2620f6ky8yy2dfd3lglwz-qemu-5.2.0/bin/qemu-system-ppc64
>> -machine powernv \
>>     -name nixos \
>>     -m 2048 \
>>     -smp 1 \
>>     -device virtio-rng-pci \
>>     -net nic,netdev=user.0,model=virtio -netdev
>> user,id=user.0${QEMU_NET_OPTS:+,$QEMU_NET_OPTS} \
>>     -virtfs local,path=/nix/store,security_model=none,mount_tag=store \
>>     -virtfs local,path=$TMPDIR/xchg,security_model=none,mount_tag=xchg \
>>     -virtfs
>> local,path=${SHARED_DIR:-$TMPDIR/xchg},security_model=none,mount_tag=shared
>> \
>>     -drive
>> cache=writeback,file=$NIX_DISK_IMAGE,id=drive1,if=none,index=1,werror=report
>> -device virtio-blk-pci,bus=pcie.0,drive=drive1 \
>>     -kernel
>> /nix/store/4mw3zzh1k4xhlysfmcgvpgx9zjz032c7-nixos-system-nixos-21.03pre-git/
>> kernel -initrd
>> /nix/store/4mw3zzh1k4xhlysfmcgvpgx9zjz032c7-nixos-system-nixos-21.03pre-git/
>> initrd -append "$(cat
>> /nix/store/4mw3zzh1k4xhlysfmcgvpgx9zjz032c7-nixos-system-nixos-21.03pre-git/
>> kernel-params)
>> init=/nix/store/4mw3zzh1k4xhlysfmcgvpgx9zjz032c7-nixos-system-nixos-21.03pr
>> e-git/init
>> regInfo=/nix/store/dnrqa7qar8jakalrz85qn3wslq5k7fmm-closure-info-powerpc64l
>> e-unknown-linux-gnu/registration console=ttyAMA0,115200n8
>> $QEMU_KERNEL_PARAMS" -nographic \
> 
> --
> Murilo
>