Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

Kevin Wolf <address@hidden> writes:

> Am 24.01.2020 um 11:27 hat Daniel P. Berrangé geschrieben:
>> On Fri, Jan 24, 2020 at 08:59:41AM +0100, Markus Armbruster wrote:
>> > John Snow <address@hidden> writes:
>> > 
>> > > On 1/23/20 2:01 PM, Daniel P. Berrangé wrote:
>> > >> So when configuring objects you'll always provide a JSON/YAML doc.
>> > >> They've got some clever stuff for updating objects where you can
>> > >> provide a JSON patch for only the bits which need changing.
>> > >> 
>> > >> When querying/listing objects by default it displays only a small
>> > >> subset of their config information in a human friendly-ish format.
>> > >> If you want to see everything then you ask for it in JSON/YAML
>> > >> format. There's also an expression language that lets you extract
>> > >> particular pieces of information based on requested properties,
>> > >> and you can filter the list of objects based on attributes and so
>> > >> on.
>> > >> 
>> > >> I think it is fair to say the structure of kubernetes object config
>> > >> is on a par with hierarchical complexity of QEMU. The lack of a simple
>> > >> human targetted data input format does not appear to have negatively
>> > >> impacted the adoption of Kubernetes. It is worth questioning why this
>> > >> is the case, while we feel the human CLI syntax for QEMU is so
>> > >> critically important to QEMU's future ?
>> > 
>> > I consider human CLI syntax for QEMU a mostly solved *design* problem:
>> > dotted keys.  It's an unsolved *implementation* problem: the CLI is a
>> > tangled mess of almost two decades' worth of ideas, and only (some of)
>> > the latest strands actually use dotted keys infrastructure.  The
>> > proposed solution is CLI QAPIfication.  Gives us configuration file(s)
>> > and introspection.
>> > 
>> > Dotted keys are merely yet another concrete syntax.  They're designed to
>> > satisfy the CLI requirements we have, which include a measure of
>> > compatibility to what's in the tangled mess.  They're reasonably usable
>> > for simple stuff, but complex stuff can be too verbose to be readable.
>> > They can't express all of the abstract syntax.  Tolerable, since they
>> > provide an escape to JSON.  I recommend programs use the JSON escape
>> > always.  Awkward for humans due to shell quoting.
>> 
>> I agree that the dotted key syntax is our chosen / solved design
>> for expressing JSON on the CLI. I would also say that, in retrospect,
>> this was a incorrect design decision that is one of the key things
>> responsible for QEMU having a bad reputation for complexity.
>
> I doubt this. Whenever I get a bug report with a command line created by
> libvirt, the command line is huge, but basically nothing in it uses
> dotted syntax. Yes, you may have cache.direct=on in it somewhere, but
> that's not actual nesting.
>
> The problem is the amount of options that is specified by management
> tools, and then humans are looking at it and feel it's way too complex.
>
> Command lines written by human users are usually much simpler because
> they just use QEMU's defaults instead of explicitly specifying
> everything.

Yes, machine-generated configuration is more verbose than what humans
produce.  Machines like it explicit.  It's simpler for them than relying
on defaults.  Bonus: immunity to changing defaults.

But no, verbosity is not the core problem, it merely aggravates the core
problem.  Complex configuration is much harder to read in a CLI syntax
than in a half-decent config file.  Yes, it's prohibitively harder just
for lengthy configurations.  That doesn't make it not harder for short
configurations.

>> We should simply never have tried to invent a way to map the full
>> hiearchy of JSON onto the CLI as the result will always be unpleasant.

It's what I had to do to secure a beach head for QAPI on the command
line coast.

>> The dotted notation is the most verbose way to do this type of
>> configuration, because of the string repetition it requires for
>> nested structures.
>
> True. I would have liked a different syntax that used some kind of
> brackets (at least optionally), but Markus didn't like adding another
> character that must be escaped.

Design thread:

    Subject: Non-flat command line option argument syntax
    Date: Thu, 02 Feb 2017 20:42:33 +0100
    Message-ID: <address@hidden>
    https://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg00555.html

A few quick quotes:

* On JSON vs. dotted keys in CLI: "both variants are basically
  illegible.  This is simply something that belongs into a config file
  rather than the command line.  In a config file, JSON would be a
  better choice."

* On dotted keys vs. structured values: "dotted keys are weird and ugly,
  but at least they don't add to the quoting mess.  Structured values
  look better, except when they do add to the quoting mess.  I'm having
  a hard time deciding which one I like less :)"

* Final verdict: "the whole non-flat command line argument design has
  been an exercise in picking what I dislike least."

> I think if we want, we can still evolve our human syntax to be more user
> friendly. My impression was that we don't want to.

Correct.

Quote from the design thread on disambiguating scalars, but I feel the
gist of it applies more widely:

    * When it breaks, fall back to JSON

      Tolerable if it breaks pretty much only in obscure corner cases.

    * More syntax

      Add syntax to disambiguate the type (key sigils?).  Has to be optional
      for backward compatibility reasons.  When omitting type information
      breaks, you have to supply it, or fall back to JSON.

      More syntax for use in obscure cases is about the last thing the QEMU
      command line needs.

    * More magic

      Make the keyval variant of the QObject input visitor shift the
      breakage to hopefully less common cases.

      - Empty list magic

        When the visitor is asked for a list, and the list's key wasn't
        specified, return an empty list instead of failure.  Unbreaks empty
        list, breaks absent optional list.

      - Alternate magic

        When the visitor is asked for an alternate, pick the alternate's
        variant based on the value rather than the type for scalar values
        (the type is always 'str' then).  Unbreaks alternates when this
        picks the variant you want, breaks them when you really want 'str'.

      - Possibly more

      When the magic breaks, fall back to JSON.

      More magic in the QEMU command line feels even worse to me than more
      syntax.

>> Lets consider how libvirt uses blockdev for a LUKS volume stored
>> in iSCSI
>> 
>>   $ qemu-system-x86_64 \
>>   -object secret,id=libvirt-5-storage-secret0,\
>>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>>   -object secret,id=libvirt-5-format-luks-secret0,\
>>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>>   -blockdev '{"driver":"iscsi","portal":"example.org:6000",\
>>     "target":"iqn.1992-01.com.example:storage","lun":1,"transport":"tcp",\
>>     "user":"myname","password-secret":"libvirt-5-storage-secret0",\
>>     
>> "node-name":"libvirt-5-storage","auto-read-only":true,"discard":"unmap"}' \
>>   -blockdev 
>> '{"node-name":"libvirt-5-format","read-only":false,"driver":"qcow2",\
>>     
>> "encrypt":{"format":"luks","key-secret":"libvirt-5-format-luks-secret0"},\
>>     "file":"libvirt-5-storage"}' \
>> 
>> We all know JSON is horrible on the CLI, no surprise. So
>> 
>> Lets use human "friendly" dotted syntax instead:
>> 
>>   $ qemu-system-x86_64 \
>>   -object secret,id=libvirt-5-storage-secret0,\
>>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>>   -object secret,id=libvirt-5-format-luks-secret0,\
>>     data=9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1,\
>>     keyid=masterKey0,iv=AAECAwQFBgcICQoLDA0ODw==,format=base64 \
>>   -blockdev driver=qcow2,node-name=libvirt-5-format,read-only=false,\
>>     encrypt.format=luks,encrypt.key-secret=libvirt-5-format-luks-secret0,\
>>     file.driver=iscsi,file.portal=example.org:6000,\
>>     
>> file.target=iqn.1992-01.com.example:storage,file.lun=1,file.transport=tcp,\
>>     file.user=myname,file.password-secret=libvirt-6-storage-secret0,\
>>     
>> file.node-name=libvirt-5-storage,file.auto-read-only=true,file.dicard=unmap
>> 
>> I don't think that's much of an improvement, aside from not having
>> to worry about matching "}".
>
> I see you merged the two -blockdev arguments into a single one in order
> to get at least some repetition with the file.* prefixes. ;-)
>
>> If we move to JSON in a config file
>> 
>>   $ cat qemu.json
>>   {
>>     "arguments": [
>>       {
>>         "arg": "object",
>>         "data": {
>>           "type": "secret",
>>           "id":"libvirt-5-storage-secret0",
>>           "data": 
>> "9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1",
>>           "keyid": "masterKey0",
>>           "iv": "AAECAwQFBgcICQoLDA0ODw==",
>>           "format": "base64"
>>         }
>>       },
>>       {
>>         "arg": "object",
>>         "data": {
>>           "type": "secret",
>>           "id":"libvirt-5-format-luks-secret0",
>>           "data": 
>> "9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1",
>>           "keyid": "masterKey0",
>>           "iv": "AAECAwQFBgcICQoLDA0ODw==",
>>           "format": "base64"
>>         }
>>       },
>>       {
>>         "arg": "blockdev",
>>         "data": {
>>           "node-name":"libvirt-5-format",
>>           "read-only":false,"driver":"qcow2",
>>           "encrypt":{
>>             "format":"luks","key-secret":
>>             "libvirt-5-format-luks-secret0"
>>           },
>>           "file":{
>>             "driver": "iscsi",
>>             "portal": "example.org:6000",
>>             "target":"iqn.1992-01.com.example:storage",
>>             "lun": 1,
>>             "transport": "tcp",
>>             "user": "myname",
>>             "password-secret": "libvirt-5-storage-secret0",
>>             "node-name":"libvirt-5-storage",
>>             "auto-read-only":"true",
>>             "discard":"unmap"
>>           }
>>         }
>>       }
>>     ]
>>   }
>>   $ qemu-system-x86_64 -f qemu.json
>> 
>> The config file is more volumous than the CLI, but it is also
>> massively more intelligible to humans because you can see the
>> structure of the data.
>> 
>> I still screwed up many times with missing quotes, incorrect
>> commas, etc. All the fun of JSON
>
> JSON is usually easy enough to read, quite reasonable for making minor
> modifications, and a PITA for writing something (like a QMP request)
> from scratch.

JSON is by design a data interchange format.  We (and others) pressed it
into a configuration file role.  PITA ensues.  Crockford shrugs.

Suitable tooling can reduce it.

>> So if we allowed YAML instead of JSON, now we get...
>> 
>>   $ cat qemu.yaml
>>   ---
>>   arguments:
>>   - arg: object
>>     data:
>>       type: secret
>>       id: libvirt-5-storage-secret0
>>       data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
>>       keyid: masterKey0
>>       iv: AAECAwQFBgcICQoLDA0ODw==
>>       format: base64
>>   - arg: object
>>     data:
>>       type: secret
>>       id: libvirt-5-format-luks-secret0
>>       data: 9eao5F8qtkGt+seB1HYivWIxbtwUu6MQtg1zpj/oDtUsPr1q8wBYM91uEHCn6j/1
>>       keyid: masterKey0
>>       iv: AAECAwQFBgcICQoLDA0ODw==
>>       format: base64
>>   - arg: blockdev
>>     data:
>>       node-name: libvirt-5-format
>>       read-only: false
>>       driver: qcow2
>>       encrypt:
>>         format: luks
>>         key-secret: libvirt-5-format-luks-secret0
>>       file:
>>         driver: iscsi
>>         portal: example.org:6000
>>         target: iqn.1992-01.com.example:storage
>>         lun: 1
>>         transport: tcp
>>         user: myname
>>         password-secret: libvirt-5-storage-secret0
>>         node-name: libvirt-5-storage
>>         auto-read-only: true
>>         discard: unmap
>>   $ qemu-system-x86_64 -f qemu.yaml
>> 
>> This is finally something I'd consider to be on a par with the
>> original QEMU syntax, before we added hierarchical data. You
>> have the minimal possible amount of syntax here. No commas,
>> no quotes, no curly brackets, etc.
>
> This seems to have the same problems as the QEMU command line (how do
> you distinguish strings from ints, from bools, from null?).

True: YAML provides only string scalars.

TOML provides strings, integers, floats, booleans, and several flavors
of time.  It lacks null.

>                                                             It's
> basically just a pretty-printed version of it with the consequence that
> it needs to be stored in an external file and there is no reasonable way
> to keep it in my shell history.

There is a reasonable way to keep it in my file system, though.  I find
that decidedly superior.

[...]