Re: Making QEMU easier for management tools and applications

[John Snow]

On 1/23/20 2:19 AM, Markus Armbruster wrote:
> John Snow <address@hidden> writes:
> 
>> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>>>> * scripts/qmp/qmp-shell
>>>>
>>>>   Half-hearted attempt at a human-friendly wrapper around the JSON
>>>>   syntax.  I have no use for this myself.
>>> I use this fairly often as its a useful debugging / experimentation
>>> / trouble shooting tool. There's similar ish functionality in
>>> virsh qemu-monitor-command. I think there's scope of a supported
>>> tool here that can talk to libvirt or a UNIX socket for doing
>>> QMP commands, with a friendlier syntax & pretty printing. 
>>>
>>
>> qmp-shell is one of my go-to tools for working through bitmap workflows
>> where we don't have convenience commands yet, as some of the setups
>> required for fleecing et al involve quite a number of steps.
>>
>> I can copy-paste raw JSON into a socket, but personally I like seeing my
>> commands neatly organized in a format where I can visually reduce them
>> to their components at a glance.
>>
>> (What I mean is: It's hard to remember which QMP commands you've barfed
>> into a terminal because JSON is hard to read and looks very visually
>> repetitive.)
>>
>> I tried to rewrite qmp-shell late last year, actually. I wanted to write
>> a new REPL that was json-aware in some manner such that you could write
>> multi-line commands like this:
>>
>>> example-command arg={
>>   "hello": "world"
>> }
>>
>> This requires, sadly, a streamable JSON parser. Most JSON parsers built
>> into Python as-is simply take a file pointer and consume the entirety of
>> the rest of the stream -- they don't play very nice with incomplete
>> input or input that may have trailing data, e.g.:
>>
>>> example-command arg={
>>   "hello": "world"
>> } arg2={
>>   "oops!": "more json!"
>> }
> 
> QMP is in the same boat: it needs to process input that isn't
> necessarily full expressions (JSON-text in the RFC's grammar).
> 
> Any conventional parser can be made streaming by turning it into a
> coroutine.  This is probably the simplest solution for handwritten
> streaming LL parsers, because it permits recursive descent.  In Python,
> I'd try a generator.
> 
> Our actual solution for QMP predates coroutine support in QEMU, and is
> rather hamfisted:
> 
> * Streaming lexer: it gets fed characters one at a time, and when its
>   state machine says "token complete", it feeds the token to the
>   "streamer".
> 
> * "Streamer": gets fed tokens one at a time, buffers them up counting
>   curly and square bracket nesting until the nesting is zero, then
>   passes the buffered tokens to the parser.
> 
> * Non-streaming parser: it gets fed a sequence of tokens that constitute
>   a full expression.
> 
> The best I can say about this is that it works.  The streamer's token
> buffer eats a lot of memory compared to a real streaming parser, but in
> practice, it's a drop in the bucket.
> 

I looked into this at one point. I forget why I didn't like it. I had
some notion that I should replace this one too, but forget exactly why.
Maybe it wasn't that bad, if I've forgotten.

>> Also, due to the nature of JSON as being a single discrete object and
>> never a stream of objects, no existing JSON parser really supports the
>> idea of ever seeing more than one object per buffer.
> 
> That plainly sucks.
> 
>> ...So I investigated writing a proper grammar for qmp-shell.
> 
> Any parser must start with a proper grammar.  If it doesn't, it's a toy,
> or a highway to madness.
> 
>> Unfortunately, this basically means including the JSON grammar as a
>> subset of the shell grammar and writing your own parser for it entirely.
> 
> Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
> paper, so JSON sticks out left and right.
> 
> Scrap and start over.
> 
>> I looked into using Python's own lexer; but it's designed to lex
>> *python*, not *json*. I got a prototype lexer working for this purpose
>> under a grammar that I think reflects JSON, but I got that sinking
>> feeling that it was all more trouble than it was worth, and scrapped
>> working on it any further.
> 
> Parsing JSON is pretty simple.  Data point: QAPISchemaParser parses our
> weird derivative of JSON in 239 SLOC.
> 
>> I did not find any other flex/yacc-like tools that seemed properly
>> idiomatic or otherwise heavily specialized. I gave up on the idea of
>> writing a new parser.
> 
> While I recommend use of tools for parsing non-trivial grammars (you'll
> screw up, they won't), they're massive overkill for JSON.
> 
>> I'd love to offer a nice robust QMP shell that is available for use by
>> end users, but the syntax of the shell will need some major considerations.
> 
> Scrap and start over.
> 
> [...]
> 

Yes, I agree: Scrap and start over.

What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
is that it offers a convenient way to enter the top-level keys of the
arguments dict. This works absolutely fine right up until you need to
start providing nested definitions.

For the nesting, we say: "Go ahead and use JSON, but you have to take
all the spaces out."

This... works, charitably, but is hardly what I would call usable.

For the CLI, we offer a dot syntax notation that resembles nothing in
particular. It often seems the case that it isn't expressive enough to
map losslessly to JSON. I suspect it doesn't handle siblings very well.

A proper HMP-esque TUI would likely have need of coming up with its own
pet syntax for commands that avoid complicated nested JSON definitions,
but for effort:value ratio, having a QMP shorthand shell that works
arbitrarily with any command might be a better win.

Do we still have a general-case problem of how to represent QAPI
structures in plaintext? Will this need to be solved for the CLI, too?

--js