Re: qemu-nbd acting as a server or not ?

[Bob Power]

Cheers Hanna,

The more I think about this the more questions I have and the more I realise 1) I need to go and do some reading (I don't know anything about linux system programming) and 2) I need to learn to let things go... (I'll start tomorrow) ;

I was under the impression IPC occurred via /var/lock/qemu-nbd-nbd0 ..... but wondered why it was in /var/lock...in any case...

So then..

1. qemu-nbd is running as process P 

2. P is running a thread PS as a server which accepts data from the kernel on UDS port US which is backed by /var/lock/qemu-nbd-nbd0

3. Having been launched with the -c option, P also launches a client thread PC which accepts data from the kernel on UDS port UC

4. Port UC is not backed by a file (is an "abstract socket address" ?) and this is why ss displays * for local address (?) 

5. Client thread PC sends a connection request to server thread PS on UDS port US

7. The kernel writes the connection request data to US which in reality means writing it to /var/lock/qemu-nbd-nbd0 ?

8. Server thread PS receives the connection request on port US and in response...

9. Server thread PS creates another server thread PSc which accepts requests on port USc which is backed by /var/lock/qemu-nbd-nbd0

10. P  then enlists the kernel nbd client by passing it UC .... or USc ? or both ?

If just UC then PC must be acting as an nbd/PSc go-between ?

If just USc then how does PSc know where to send replies and why would it accept non UC requests ?

If both then presumably PC is no longer involved ..... and shouldn't be around ?

And there are 2 separate threads PS and PSc using /var/lock/qemu-nbd-nbd0 to back 2 different UDS input ports US and USc (as seen in ss output)

tbh I don't really *need* to know.... but ... that's OCD for you.

Bob.

On Wednesday, 4 May 2022, 07:01:52 BST, Hanna Reitz <hreitz@redhat.com> wrote:

On 03.05.22 19:01, Bob Power wrote:

> Yeah, it looks that way...
>
> [root@fedora bob]# qemu-nbd -d /dev/nbd0
> /dev/nbd0 disconnected
>
> All clear :
>
> [root@fedora bob]# netstat -pnxa | head -2 | tail -1 ; netstat -pnxa |
> grep nbd
> Proto RefCnt Flags       Type       State         I-Node PID/Program
> name     Path
> [root@fedora bob]# cat <(ss -pax | head -1 | sed -E 's/(\S+)
> Address:Port/\1_Address Port /g' ) <( ss -pax | grep nbd) | sed -E
> 's/\s+$//' | column -t
> Netid  State  Recv-Q  Send-Q  Local_Address  Port Peer_Address  Port 
> Process
>
> [root@fedora bob]# qemu-nbd -c /dev/nbd0  test.qcow2
>
> Active Unix Domain Sockets:
>
> [root@fedora bob]# cat <(ss -pax | head -1 | sed -E 's/(\S+)
> Address:Port/\1_Address Port /g' ) <( ss -pax | grep nbd) | sed -E
> 's/\s+$//' | column -t
> Netid  State   Recv-Q  Send-Q  Local_Address Port   Peer_Address 
> Port   Process
> u_str  LISTEN  0       1       /var/lock/qemu-nbd-nbd0 42983 
> *             0 users:(("qemu-nbd",pid=5211,fd=3))
> u_str  ESTAB   0       0       * 45022  *             42988
> users:(("qemu-nbd",pid=5211,fd=9))
> u_str  ESTAB   0       0       /var/lock/qemu-nbd-nbd0 42988 
> *             45022 users:(("qemu-nbd",pid=5211,fd=10))
>
> [root@fedora fd]# netstat -pnxa | head -2  ; netstat -pnxa | grep nbd
> Active UNIX domain sockets (servers and established)
> Proto RefCnt Flags       Type       State         I-Node PID/Program
> name     Path
> unix  2      [ ACC ]     STREAM     LISTENING     42983
> 5211/qemu-nbd        /var/lock/qemu-nbd-nbd0
> unix  3      [ ]         STREAM     CONNECTED     45022 5211/qemu-nbd
> unix  3      [ ]         STREAM     CONNECTED     42988
> 5211/qemu-nbd        /var/lock/qemu-nbd-nbd0
>
>
> SS        Netstat
> Netid    =    Proto    : u_str = unix stream
> Port    =    I-Node    : socketfs inode
>         Flags    : ACC= Waiting for a connect request
>
> [root@fedora lock]# lsof +E /var/lock/qemu-nbd-nbd0
> lsof: WARNING: can't stat() fuse.gvfsd-fuse file system
> /run/user/1000/gvfs
>       Output information may be incomplete.
> COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
> qemu-nbd 5211 root    3u  unix 0x00000000b0829fa5      0t0 42983
> /var/lock/qemu-nbd-nbd0 type=STREAM (LISTEN)
> qemu-nbd 5211 root    9u  unix 0x00000000876da850      0t0 45022
> type=STREAM ->INO=42988 5211,qemu-nbd,10u (CONNECTED)
> qemu-nbd 5211 root   10u  unix 0x00000000482ad094      0t0 42988
> /var/lock/qemu-nbd-nbd0 type=STREAM ->INO=45022 5211,qemu-nbd,9u
> (CONNECTED)
>
> So it looks like 45022 is qemu-nbd acting as a client accessing
> qemu-nbd as a server on 42988 having requested that connection (from
> itself) on 42983 (?)
>
> But now I'm more confused - isn't nbd as in "modinfo nbd" =
> /lib/modules/5.16.20-200.fc35.x86_64/kernel/drivers/block/nbd.ko.xz
> the nbd client in the kernel interfacing with an nbd server to provide
> fs access via /dev/nbd0 ? or does nbd.ko.xz just talk to/need an nbd
> client, in this case qemu-nbd (on 45022) ?
>
> And dunno why the netstat RefCnts are 2 and 3 ...
>
> I *would* like to know definitively how this works from an
> overview/architecture perspective eg.
> (qcow2 file) <data/transport> (qemu-nbd as a server) <data/transport>
> (qemu-nbd as a client) <data/transport> (kernel module / filesytem
> interface)
>
> ..but short of an nbd/qemu-nbd developer setting the facts down (or
> getting into the code myself) I think I'll just have to let it go.

AFAIU (just from looking at the code), qemu-nbd always sets up a Unix
socket if you specify `-c` (though you can apparently adjust the path if
you want to):

https://gitlab.com/qemu-project/qemu/-/blob/master/qemu-nbd.c#L982
https://gitlab.com/qemu-project/qemu/-/blob/master/qemu-nbd.c#L387

And then connects to itself by launching a thread:

https://gitlab.com/qemu-project/qemu/-/blob/master/qemu-nbd.c#L1123
https://gitlab.com/qemu-project/qemu/-/blob/master/qemu-nbd.c#L275

nbd_init() is what seems to set up the /dev/nbd* device:

https://gitlab.com/qemu-project/qemu/-/blob/master/nbd/client.c#L1244

The most interesting bit to me there is the ioctl(NBD_SET_SOCK,
sioc->fd) (line 1262).  So apparently qemu-nbd just hands the client
socket over to the Linux kernel, on which the latter will then issue NBD
requests for accesses to the /dev/nbd* device.


So with `-c` the set-up is mostly as normal, except qemu-nbd always
creates a Unix socket; then it launches a thread to connect (as a
client) to the NBD server it has just set up; and once negotiation is
complete, it hands this client socket over to the kernel, so the kernel
then becomes the NBD client.

Maybe I’m wrong, though.

Hanna