Re: Scheme syntax vs. other languages [was: Re: Appreciation / Financial support]

[David Kastrup]

Joseph Rushton Wakeling <address@hidden> writes:

> On 07/06/12 05:24, David Kastrup wrote:
>> You picked a _Scheme_ function, not a music function.  That does not, I
>> repeat _not_ at all show how you embed this thing into your LilyPond
>> code, and we were talking about using D as an _extension_ language of
>> LilyPond, not about its usefulness as a general purpose programming
>> function.
>
> I don't see any point in simply embedding D as an alternative
> scripting language to Scheme, and I'm not proposing that.
>
> Rather, in my original post I was introducing D as a language that
> could be used to code all levels of LilyPond, from low-level internals
> to high-level stuff all the way up to scripts embedded in LilyPond
> input files.  That seems a pretty reasonable input to a discussion
> where the technology and architectural choices of LilyPond were being
> debated, and one of LilyPond's original authors had written:

That's nice but still does not show how you would embed D for the
"high-level stuff all the way up to scripts embedded in LilyPond input
files".

> I have only a very basic familiarity with Go, but if the intention
> would be to do a rewrite from the ground up, using a language that
> provides the features described here, then D would be a strong
> contender.

You have not shown how this would work at user-level scripting.  The
kind of stuff where cut&paste of Scheme snippets or creating music
functions works.  As I already said: the examples in scm/* don't
interface with the user/LilyPond level: you have to look at ly/* for
that.

> I'm not asking anyone to do that rewrite.  I'm simply pointing out
> that a possibly useful language choice exists, should a bottom-up
> rewrite be considered, and I tried to illustrate the general features
> that I think make it useful.  If that information is useful to anyone
> -- great.  If not -- let's move on.

You are evading the embedding question.  I am not saying it would be
impossible to tackle.  But it is _crucial_ to the success of the whole.
It is the main reason we have a Scheme layer in the first place rather
than just the original C++.

> Then there's a somewhat different discussion, which is the
> accessibility of Scheme as a scripting language.  If you want to argue
> that the accessibility issue is outweighed by the power and
> flexibility of the language, that's fine.

Yes and no.  We don't really utilize Scheme's potential here at the
current point of time.  Music transformations (including transforming
music into displayed form and MusicXML and unfolding repeats and other
things) could be done using tools like
<URL:http://www.gnu.org/software/guile/manual/html_node/Pattern-Matching.html>.
We don't do that right now, and using music terms as non-native C++
constructs rather than accessible through records/Goops does not help.

If you take a look at scm/define-music-display-methods.scm you'll see
that there _is_ some home-brewn pattern matching going on that would not
be easy to do in most other languages, but it is not the same as being
able to use standard facilities, and it is just used for display and
nowhere else in the LilyPond code base.

> I am not sure I disagree with you.  But that doesn't alter the fact
> that its syntax and style is very different from the mainstream
> languages that users and potential new contributors are more likely to
> be familiar with.

New contributors are familiar with music and LilyPond input.

>> The principal source of contributors are musicians, not programmers.
>> C++ is much more mainstream, and will you claim that we have more
>> people working with the C++ parts of LilyPond than with the Scheme
>> parts?
>
> Working with the internals is going to be heavy-duty no matter what
> the programming language.  On the other hand it's a reasonable concern
> (raised by someone other than me) that if too much of the internals is
> rewritten in Scheme, it may be difficult to secure new contributors to
> the codebase.

Straw man.  The principal target is making stuff accessible from Scheme,
not rewriting it.

>> How many people extend LaTeX writing Pascal code rather than TeX
>> code?  Which if Pascal and TeX is the more mainstream language?
>> Which is saner?  That's so far from being even competitive that it
>> isn't funny.
>
> ... and TeX work is confined to a fairly small hardcore group, with
> the mainstream of publishing moving on to other technologies.

ls -R /usr/local/texlive/2011/|wc -l
115785

First let us get there, then worry.

> Besides, TeX has the advantage that when it was introduced it provided
> something that wasn't really available before, so it attracted input
> (both amateur and commercial) by virtue of being the only real show in
> town, and so built up a weight of add-on functionality that kept it
> relevant.

So does LilyPond.  Care to name another scriptable batch typesetting
system for music?

> You can say something similar about your other example, Emacs and its
> extension language.  It was offering new functionality; when it first
> appeared, LISP was a much more mainstream language than it is now;

Not really, because programming was not mainstream.

> and both then and now, its target user base is hard-core programmers,

No, that's the target clientele for working on the C code base.  Most
Emacs "users" accumulate their personal .emacs file with copy&paste from
various other smart sources.  They'd never be able to recompile Emacs.

> for whom LISP and its variants are very likely to be something they
> are either already familiar with or readily able to learn.

Come off it.  The vast majority of Emacs users and most certainly of
Emacs programmers have had their first exposure to Lisp through Emacs.

> LilyPond doesn't have those advantages.  In terms of what it provides
> -- computer-based musical engraving -- it's preceded by multiple
> alternatives, including SCORE, Finale and Sibelius.

I don't know SCORE, but the others are not scriptable batch typesetting
systems.  According to your logic, TeX did not offer anything new since
WordStar was available already.

[...]

>>>      pure this(int o, int n, double a)
>>
>> What is "pure"?  Can I eat it?
>
> The "pure" keyword indicates that a function is pure, and this is
> enforced by the compiler/interpreter, which will throw an error if a
> function marked this way violates purity (e.g. by calling an impure
> function, or trying to modify global state).
>
> I included it simply to illustrate some of the useful features of D,
> not because it's in any way essential to this example.

If a user needs to write it, some understanding is not amiss.  This is a
hard to understand concept without low-level knowledge of computing.

>> How do I know that&&  has lower
>> precedence than<=?
>
> ... then put extra brackets in to ensure the precedence you want,
> which is something easy and familiar to anyone with any experience of
> mainstream programming languages.

Well, "extra brackets" are what people list as a disadvantage of Scheme,
where you don't even _need_ precedence dur to them.

>>> pure auto naturalizePitch(LilyPitch p)
>>
>> What is pure?  What is auto?
>
> Pure I've already explained.  auto indicates that the type (in this
> case, the return type) is to be inferred automatically.

Why do we need a fixed return type?

>> Why do I need to declare the types of p.octave etc, but not of o, n, a?
>
> Because they can be inferred from the type of p.octave etc.  You can
> also specify them if you wish.
>
> You may wish to have a dynamically rather than statically typed
> language for scripting, I do understand if that's a deal-breaker.

It's not a deal-breaker, but it is intended to make things easier for
the computer and harder for the user.

>>>      // First, we disallow anything greater than +1/4-tone on E and B,
>>>      // and anything less than -1/4-tone on C and F.  In other words,
>>>      // no E# or B#, no Cb or Fb.
>>>      if(a>  0.25&&  (n == 6 || n == 2))
>>
>> What are the implications of using an inexact data type?  Shouldn't I be
>> doing my comparisons with some epsilon added?
>
> With a rational data type, this isn't an issue.

Is there one built-in?  With what syntax?

>>>      {
>>>          a -= 0.5;
>>>          n += 1;
>>>      }
>>
>> What's a "-="?  Why do I need that semicolon before "}"?
>
> Again, notational features familiar to anyone with experience of
> mainstream programming languages.

For a definition of "mainstream == C".

>> Is "if" a function call?
>
> Is "if" or "if-else" something that is unfamiliar to anyone with
> programming experience?

if and if-else are just words.  It depends on the language whether they
are reserved words or functions or special forms, whether they take
arguments and what kind.
<URL:http://en.wikipedia.org/wiki/Smalltalk#Control_structures> is one
example.

Again, for you "mainstream == C".

>> For a musician?  Apart from the problems with the inexact numeric type?
>
> A musician without experience of programming is going to struggle
> anyway,

Thanks, but I am not interested in the "programming is hard anyway, so
it does not matter how hard we make it" school of things.  LilyPond has
seen enough of that mind frame.

> but a syntax closer to mainstream languages carries with it a greater
> range of learning opportunities.  A musician with experience of
> programming is probably more likely to have encountered one of the
> mainstream languages (C/C++, Java, PHP, JavaScript, ...) than one of
> the LISP family.

A musician is more likely to have encountered a banana box and a steel
drum in free nature rather than a piano, but that should not restrict
what he should be making music with if it does the job better.

>> Or how would you write something like
>>
>> applyMusic =
>> #(define-music-function (parser location func music) (procedure? ly:music?)
>>     (_i"Apply procedure @var{func} to @var{music}.")
>>     (func music))
>
> Something like this:
>
>     auto applyMusicFunction(alias func)(Parser p, Location l, LilyMusic music)
>     {
>         return func(music);
>     }

"auto" would be a lyric.  We are talking about embedding in LilyPond
still.  I doubt that this order of arguments would magically transform
itself into the right order for embedding in music.

> There's also a method involving function delegates, which is a bit less 
> elegant:
>
>     auto applyMusicFunction(Parser p, location l, LilyMusic
> delegate(LilyMusic) func, LilyMusic music)
>     {
>         return func(music);
>     }

Hm.  Still no view of the lexical embedding angle.  Or of a
documentation string for the sake of interpreter and printed
documentation.

> I don't understand how the parser and location are used in the Scheme
> example you've provided (it doesn't seem to be documented in the CG).

Documented in EG, but mostly magic pixy dust.  Not something that the
music function interface has much reason to celebrate as the height of
good design.  Not particularly Scheme's fault either.

>> It is a bit unfair since you have not even _tried_ creating a
>> LilyPond document where this function is defined and used.  You have
>> created a scrap of source code dangling in the air.  That is not the
>> hard part about an extension language.
>
> It's a bit unfair to expect a simple example of how one notation is
> more accessible than another, should come with a fully-fledged
> implementation!

It's a bit unfair to just deal with the questions for which one has a
convenient answer and skip the rest.

>> Nice if I am writing a packet driver.  So if we use it in LilyPond
>> where C++ is used, we can use almost the same programming style, just
>> with a performance hit for the pains of porting.
>
> What you gain is that, unlike C/C++ you have the possibility to write
> high-level code _in the same language_ rather than having to have a
> split between fast low-level language and high-level scripting
> language; the code syntax is much more elegant than C++, making it
> easier to read and maintain; and a bunch of other useful features
> designed to help maintain code quality (built-in unittests just for
> starters).

I have not seen "much more elegant than C++", and I have not seen a good
match for the document-level scripting.

>>>    * Fast to compile (faster than Go IIRC, _much_ faster than C++).
>>
>> Too bad that compilation times are not a problem in LilyPond.
>
> I can't say that I would object to a cut in the compile times.

make all takes about 4 minutes here, make doc about 1h20.  The 4 minutes
are not the problem.

>> Theproblem would be rather having to compile and link _at_ _all_ just for
>> writing a document.
>
> I wouldn't think of doing a rewrite in D without the scripting issue
> (mentioned above) having been resolved.

You _are_ thinking of a rewrite in D without the scripting issue having
been resolved.  This is what this thread is about.

>>>    * Inbuilt support for functional programming, enforcement of purity etc.
>>>      but with a tolerant approach that allows mutation of internal
>>> functional
>>>      variables as long as it has no side-effects (e.g. my naturalizePitch is
>>>      a pure function, even though the way it's written it has mutation of
>>>      internal variables).
>>
>> Can you please explain this in terms a musician would understand?
>
> Will this explanation of purity do? :-)
> http://lilypond.org/doc/v2.15/Documentation/contributor/understanding-pure-properties

It is documenting quite a different sort of "pure" (being independent
from line breaking decisions), and it is not in user-level
documentation.

>>>    * Thread-safe globals by default, inbuilt support for immutable
>>> variables,
>>>      inbuilt support for both message passing and mutex-based approaches to
>>>      concurrency.
>>
>> Can you please explain this in terms a musician would understand?  Also
>> explain where and why he should acquire the computer science for being
>> able to write LilyPond files?
>
> All of this is relevant almost entirely to the internals of LilyPond.

So why bring it up in the context of Scheme as an extension language?

> From the point of view of a musician: D would probably make it easier
> to write code that would process different parts of the engraving
> process in parallel, which would speed up the production of the score.

Speed is not a worry for the musician.  LilyPond is fast enough to
recompile huge tomes (namely its own documentation) in reasonable
amounts of time.

>>>    * Currently a small but very enthusiastic developer and user
>>>    community, but picking up a growing interest from games
>>>    developers, scientific programmers, and others who have strong
>>>    joint requirements for speed, safety and ease of
>>>    writing/debugging.
>>
>> Yup.  Sounds like a perfect match to the requirements of musicians.
>
> Musicians presumably also need speed, safety and ease of
> writing/debugging? :-)

No.  They need to get work done.  You are listing the needs of
developers.  And even for them, we need frameworks that can withstand
blasts of creativity from Mike without leaving all too many regressions
as side-effects of the architecture being less obsidian than one would
like.  LilyPond is not resilient enough here, and it is not like Mike
actually needs C++ to break things, it just makes it easier to do it by
accident.  We need to narrow the gap between the point of time you
manage to do things, and that where you manage doing them without
repercussions.  The programming language(s) are a marginal concern for
that: it is more the question how they are architectured into a system.
There is a lot of work to do in that area, and language wars detract
from the actual cause of that work.

> If the discussion is not useful, all you have to do is say so.  I've
> no wish to inflict unhelpful remarks on the community.

The discussion is not useful.  It distracts from work needing to get
done, without offering perspectives that are actually feasible since
they are neither thought through nor have the resources for tackling
them _if_ they made sense and were planned out.

> It's just that you keep challenging me to explain or respond to
> various points, so I do feel obliged to give an answer.

Welcome to the dynamics of discussing things "on the internet".

-- 
David Kastrup