Re: Making QEMU easier for management tools and applications

[Markus Armbruster]

John Snow <address@hidden> writes:

> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>>> * scripts/qmp/qmp-shell
>>>
>>>   Half-hearted attempt at a human-friendly wrapper around the JSON
>>>   syntax.  I have no use for this myself.
>> I use this fairly often as its a useful debugging / experimentation
>> / trouble shooting tool. There's similar ish functionality in
>> virsh qemu-monitor-command. I think there's scope of a supported
>> tool here that can talk to libvirt or a UNIX socket for doing
>> QMP commands, with a friendlier syntax & pretty printing. 
>> 
>
> qmp-shell is one of my go-to tools for working through bitmap workflows
> where we don't have convenience commands yet, as some of the setups
> required for fleecing et al involve quite a number of steps.
>
> I can copy-paste raw JSON into a socket, but personally I like seeing my
> commands neatly organized in a format where I can visually reduce them
> to their components at a glance.
>
> (What I mean is: It's hard to remember which QMP commands you've barfed
> into a terminal because JSON is hard to read and looks very visually
> repetitive.)
>
> I tried to rewrite qmp-shell late last year, actually. I wanted to write
> a new REPL that was json-aware in some manner such that you could write
> multi-line commands like this:
>
>> example-command arg={
>   "hello": "world"
> }
>
> This requires, sadly, a streamable JSON parser. Most JSON parsers built
> into Python as-is simply take a file pointer and consume the entirety of
> the rest of the stream -- they don't play very nice with incomplete
> input or input that may have trailing data, e.g.:
>
>> example-command arg={
>   "hello": "world"
> } arg2={
>   "oops!": "more json!"
> }

QMP is in the same boat: it needs to process input that isn't
necessarily full expressions (JSON-text in the RFC's grammar).

Any conventional parser can be made streaming by turning it into a
coroutine.  This is probably the simplest solution for handwritten
streaming LL parsers, because it permits recursive descent.  In Python,
I'd try a generator.

Our actual solution for QMP predates coroutine support in QEMU, and is
rather hamfisted:

* Streaming lexer: it gets fed characters one at a time, and when its
  state machine says "token complete", it feeds the token to the
  "streamer".

* "Streamer": gets fed tokens one at a time, buffers them up counting
  curly and square bracket nesting until the nesting is zero, then
  passes the buffered tokens to the parser.

* Non-streaming parser: it gets fed a sequence of tokens that constitute
  a full expression.

The best I can say about this is that it works.  The streamer's token
buffer eats a lot of memory compared to a real streaming parser, but in
practice, it's a drop in the bucket.

> Also, due to the nature of JSON as being a single discrete object and
> never a stream of objects, no existing JSON parser really supports the
> idea of ever seeing more than one object per buffer.

That plainly sucks.

> ...So I investigated writing a proper grammar for qmp-shell.

Any parser must start with a proper grammar.  If it doesn't, it's a toy,
or a highway to madness.

> Unfortunately, this basically means including the JSON grammar as a
> subset of the shell grammar and writing your own parser for it entirely.

Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
paper, so JSON sticks out left and right.

Scrap and start over.

> I looked into using Python's own lexer; but it's designed to lex
> *python*, not *json*. I got a prototype lexer working for this purpose
> under a grammar that I think reflects JSON, but I got that sinking
> feeling that it was all more trouble than it was worth, and scrapped
> working on it any further.

Parsing JSON is pretty simple.  Data point: QAPISchemaParser parses our
weird derivative of JSON in 239 SLOC.

> I did not find any other flex/yacc-like tools that seemed properly
> idiomatic or otherwise heavily specialized. I gave up on the idea of
> writing a new parser.

While I recommend use of tools for parsing non-trivial grammars (you'll
screw up, they won't), they're massive overkill for JSON.

> I'd love to offer a nice robust QMP shell that is available for use by
> end users, but the syntax of the shell will need some major considerations.

Scrap and start over.

[...]