Re: [PATCH v5 03/36] qapi-gen: Separate arg-parsing from generation

[John Snow]

On 10/6/20 7:51 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
>
> > This is a minor re-work of the entrypoint script. It isolates a
> > generate() method from the actual command-line mechanism.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> > Reviewed-by: Cleber Rosa <crosa@redhat.com>
> > Tested-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >   scripts/qapi-gen.py | 85 +++++++++++++++++++++++++++++++++------------
> >   1 file changed, 62 insertions(+), 23 deletions(-)
> >
> > diff --git a/scripts/qapi-gen.py b/scripts/qapi-gen.py
> > index 541e8c1f55d..117b396a595 100644
> > --- a/scripts/qapi-gen.py
> > +++ b/scripts/qapi-gen.py
> > @@ -1,30 +1,77 @@
> >   #!/usr/bin/env python3
> > -# QAPI generator
> > -#
> > +
> >   # This work is licensed under the terms of the GNU GPL, version 2 or later.
> >   # See the COPYING file in the top-level directory.
> >   
> > +"""
> > +QAPI Generator
> > +
> > +This script is the main entry point for generating C code from the QAPI schema.
> > +"""
> >   
> >   import argparse
> >   import re
> >   import sys
> >   
> >   from qapi.commands import gen_commands
> > +from qapi.error import QAPIError
> >   from qapi.events import gen_events
> >   from qapi.introspect import gen_introspect
> > -from qapi.schema import QAPIError, QAPISchema
> > +from qapi.schema import QAPISchema
> >   from qapi.types import gen_types
> >   from qapi.visit import gen_visit
>
> Unrelated cleanup.  Okay.
>
> >   
> >   
> > -def main(argv):
> > +DEFAULT_OUTPUT_DIR = ''
> > +DEFAULT_PREFIX = ''
> > +
> > +
> > +def generate(schema_file: str,
> > +             output_dir: str,
> > +             prefix: str,
> > +             unmask: bool = False,
> > +             builtins: bool = False) -> None:
> > +    """
> > +    generate uses a given schema to produce C code in the target directory.
> > +
> > +    :param schema_file: The primary QAPI schema file.
> > +    :param output_dir: The output directory to store generated code.
> > +    :param prefix: Optional C-code prefix for symbol names.
> > +    :param unmask: Expose non-ABI names through introspection?
> > +    :param builtins: Generate code for built-in types?
> > +
> > +    :raise QAPIError: On failures.
> > +    """
> > +    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix)
> > +    if match.end() != len(prefix):
> > +        msg = "funny character '{:s}' in prefix '{:s}'".format(
> > +            prefix[match.end()], prefix)
> > +        raise QAPIError('', None, msg)
>
> Uh...
>
>      $ python3 scripts/qapi-gen.py --prefix=@ x
>      scripts/qapi-gen.py: : funny character '@' in prefix '@'
>
> Unwanted " :".
>
> This is due to a hack: you pass '' for info (*quack*).  Everything else
> passes QAPISourceInfo (I believe).

Quack indeed - why does our base error class require so much information 
from a specific part of the generation process?

Ah, someone changes this in part 4 so that we have a more generic error 
class to use as a base when we are missing such information.

You are witnessing some more future-bleed.
> Is it really a good idea to do this in generate?  It's not about
> generating code, it's about validating a CLI option.

One might also ask: Is it a good idea to only validate this on a 
frontend, and not in the implementation?

The idea here was to create a function that could be used in a script 
(for tests, debugging interfaces, other python packages) to do all of 
the same things that the CLI tool did, just sans the actual CLI.

Wouldn't make sense to allow garbage to flow in from one interface but 
not the other; so the check is here.

> > +
> > +    schema = QAPISchema(schema_file)
> > +    gen_types(schema, output_dir, prefix, builtins)
> > +    gen_visit(schema, output_dir, prefix, builtins)
> > +    gen_commands(schema, output_dir, prefix)
> > +    gen_events(schema, output_dir, prefix)
> > +    gen_introspect(schema, output_dir, prefix, unmask)
> > +
> > +
> > +def main() -> int:
> > +    """
> > +    gapi-gen shell script entrypoint.
> > +    Expects arguments via sys.argv, see --help for details.
> > +
> > +    :return: int, 0 on success, 1 on failure.
> > +    """
> >       parser = argparse.ArgumentParser(
> >           description='Generate code from a QAPI schema')
> >       parser.add_argument('-b', '--builtins', action='store_true',
> >                           help="generate code for built-in types")
> > -    parser.add_argument('-o', '--output-dir', action='store', default='',
> > +    parser.add_argument('-o', '--output-dir', action='store',
> > +                        default=DEFAULT_OUTPUT_DIR,
> >                           help="write output to directory OUTPUT_DIR")
> > -    parser.add_argument('-p', '--prefix', action='store', default='',
> > +    parser.add_argument('-p', '--prefix', action='store',
> > +                        default=DEFAULT_PREFIX,
> >                           help="prefix for symbols")
>
> I don't like the changes to default=, because:
>
> 1. They are only losely related to the patch's purpose.

Subjective, but OK.

> 2. They split the definition of the CLI: most of it is here, except for
> defaults, which are defined elsewhere.

All of it is in main.py, though! If you were to, say, move generate() 
elsewhere, it'd look pretty compact as just the CLI frontend, no?

> 3. The defaults will not change, and nothing else uses the constants.

But, fine. Cleber had the same comment but I wasn't fully on-board, but 
two folks saying the same thing ...

> >       parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
> >                           dest='unmask',
> > @@ -32,25 +79,17 @@ def main(argv):
> >       parser.add_argument('schema', action='store')
> >       args = parser.parse_args()
> >   
> > -    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
> > -    if match.end() != len(args.prefix):
> > -        print("%s: 'funny character '%s' in argument of --prefix"
> > -              % (sys.argv[0], args.prefix[match.end()]),
> > -              file=sys.stderr)
> > -        sys.exit(1)
> > -
> >       try:
> > -        schema = QAPISchema(args.schema)
> > +        generate(args.schema,
> > +                 output_dir=args.output_dir,
> > +                 prefix=args.prefix,
> > +                 unmask=args.unmask,
> > +                 builtins=args.builtins)
> >       except QAPIError as err:
> > -        print(err, file=sys.stderr)
> > -        exit(1)
> > -
> > -    gen_types(schema, args.output_dir, args.prefix, args.builtins)
> > -    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
> > -    gen_commands(schema, args.output_dir, args.prefix)
> > -    gen_events(schema, args.output_dir, args.prefix)
> > -    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
> > +        print(f"{sys.argv[0]}: {str(err)}", file=sys.stderr)
> > +        return 1
> > +    return 0
>
> Subtle change: you move the gen_FOO() into the try ... except.  Okay;
> they don't raise QAPIError, but perhaps worth a mention in the commit
> message.

Forbidden future knowledge; I intend them to.

> >   
> >   
> >   if __name__ == '__main__':
> > -    main(sys.argv)
> > +    sys.exit(main())
>
> "Python was designed to be easy to understand and fun to use."
> Ha ha ha.

I mean, I'm having fun, aren't you?

--js