Option handling

[Urs Liska]

We have discussed options already on a conceptual level as "package
options", but technically it should be dealt with separately.

Options are not functionally required for the package loading to work,
but they are an integral part of the functionality, and I have found
that their availability has increased my ability to create
infrastructures and "user interfaces" fundamentally/exponentially.
Options are also useful as a low level building block, and I often load
oll-core just for having them available. So I suggest implementing the
option handling as the first step of the extension mechanism so it's in
place when starting on the packages.

The current implementation in oll-core is pretty powerful but it also
has fundamental flaws. Therefore you *can* look up the implementation
at 
https://github.com/openlilylib/oll-core/blob/master/internal/options.ily
but I suggest not to do so and keep the conversation on the conceptual
level like we have so far done with the package mechanism itself.
Eventually it should be coded newly from scratch.


## Interface

The basic interface is to specify an option with

  \registerOption package.component.some.path <default-value>

then retrieve the value with

  \getOption package.component.some.path

or change it with

  \setOption package.component.some.path <new-value>

Today I'm missing one feature: specifying (and enforcing) an optional
type to the value. This is not something I considered undoable, I just
haven't got around giving it a serious shot. An issue with that is that
I'm pretty sure that the underlying data structure is not ideal and
should be redesigned from scratch (see below), which made me less eager
to patch around with the typing.

There are convenience functions to set and get child options, useful
for iterating over a list of (generated) properties (e.g a list of
provided colors or all files in a directory).

There are also variants of the getter functions ...WithFallback in case
an option is "empty". But I think now this would better be handled with
an optional "default" argument to the normal function.

In the current implementation a package has to make sure itself that
the \registerOption is done before possibly given options are
evaluated. There should be a mechanism/interface to make that
straightforward for package authors.


## Command line options

Option should also be settable through the command line invocation so
it is possible to make use of them from outside (i.e. in invocation
scripts or the Frescobaldi/Emacs).

What I have done so far in projects is simply use a -d option, ignore
the warning about the unknown option and look for that option within,
and look for the option when needed. Actually we do this with
Frescobaldi's Layout Control Options as well, and I'm surprised that
noone has complained about the "warning: no such internal option: ..."
yet.

Of course when reading the command line there is no notion of loaded
packages, so what I think would be necessary is a new command line
option that takes a key=value argument and stores it in an independent
flat alist. That might then look like

  #'((scholarly.annotate.use-colors . #t)
     (stylesheets.base . "my-house-style"))

\usepackage (or whatever) will then check whether options for the given
package are present in the command line *and* registered in the
package.

I'm not sure if there are security issues to be considered beyond
executing LilyPond in the first place.


## Option definition

oll-core abuses the ly:context-mod? implementation for its option
interface. For example critical remarks can be entered with
scholarly.annotate like this:

  \criticalRemark "e' in manuscript" es'

or

  \criticalRemark \with {
    message = "e' in manuscript"
    item = Accidental
  } es'

I like this very much as a user interface because it is both powerful
and expressive. However it is clearly an abuse:

 * the predicate ly:context-mod? clearly refers to something
   fundamentally different
 * message = "e' in manuscript" is not represented as a pair but rather
   as '(assign item Accidental)

I have written convenience functions to create an alist from the
expression, and Stefano Troncaro has done an amazing job making the
configuration flexible and robust, for example with enforcing types and
providing defaults. However, when included in LilyPond itself it would
clearly be good to have a native implementation.

>From what I can see this should be a new predicate, say ly:options? and
a parser item that works like the above but with a different keyword.
If this suggestion is agreed upon we should discuss the details
separately.


## Data Structure

Currently oll-core maintains one single nested alist for storing
options which is accessed by the getter and setter functions. The
rationale was to pollute the namespace with only one single variable
oll-options that could be considered "pretty" safe by the prefix. Now I
think the data structure(s) holding the options would better be hidden
inside a Scheme module  which is only visible to the getter and setter
functions themselves. Shouldn't be too hard, I think.

Currently it is sort-of a convention that the top-level entry in that
alist is the package name and below the package is free to do what it
wants. But that convention is not enforced in any way, and I have
already abused that system in my projects by storing arbitrary data in
that alist. I think the data structure should be shielded somewhat more
against the user, but I wouldn't discuss that in too much detail right
now. (Just know that I don't want to push through my current
implementation).

I think the implementation as an alist (with some convenience accessor
functions) is not what is best on the long run. Conceptually it is a
tree, and there are probably better ways to represent that than by
manually travelling over an alist. Jan-Peter has written a tree class (
https://github.com/openlilylib/oll-core/blob/master/scheme/oll-core/tree.scm
) and maybe that's the direction one would want to go. Maybe the
toplevel variable storing the package options should be an alist with
one tree object per package. That would make for a quite good
separation of concerns when accessing with \getOption. It would be even
possible to allow packages to use arbitrary data structures for their
internal storage, in case they don't only want to use usual options but
more complex data (but that's probably already too implementation-
focused).


## Data storage service

I have found myself abusing the oll-options alist for storing data, for
example when collecting stuff to build a score from at a later point. I
feel it's not ideal (from a clarity/design POV) to stuff data into the
option tree, and I suggest to provide an interface for storing
arbitrary data "while we're at it". This is functionally unrelated to
the package handling and should therefore be implemented later in the
process but I think it is a good opportunity to add a significant tool
to the toolkit.

I've several times done things that were loosely inspired by reading
Jan-Peter's descriptions. But I'm by now pretty sure that I've
essentially did *the same thing*, without reusing his code. I think
storing and reusing music expressions in an adressable tree (as
presented in his talk in Salzburg) is a very powerful and versatile
tool for advanced project set-ups. And if we had a vetted and
documented interface for that *inside* LilyPond it can only encourage
other power users to follow in that track *while* reusing code instead 
of reinventing the wheel.

I'm raising it now because I think this should be pretty closely
coupled to the implementation of options.

###

So that's what I noticed should be discussed with regard to an
implementation of options in LilyPond.

Urs