Re: Do we really offer the future?

[Urs Liska]

Hi all,

I had to leave this alone for a while, otherwise I wouldn't have been able to reply calmly to a number of the first responses on this thread. It's only lately that the discussion has reached a level of constructivity that may be helpful. In the meantime the discussion has split up to a number of threads and sub-topics, all going in the direction of how to improve LilyPond, and I'm glad this came up. Let's hope it will eventually trigger some real improvements...

However, I think I have been grossly misunderstood by some of the early commenters, as the invocation of certain magic words (where "publishing house", particularly in combination with "big" is one of the foremost, ranking directly after "copyright") triggers near Pavlovian responses that may not really take into account what was originally expressed. As a consequence some of my main points (or not-explicitly-named consequences of them) haven't been part of the discussion at all, and I think these should be picked up too.

It doesn't make any sense to reply to all of the messages individually, so I'll try to do it in a more general reply to my initial post.

First of all:
I have _not_ asked "the LilyPond team" to spend any resources for whatever. What my post was asking for is a discussion in the _user community_. Obviously this discussion arose, and in the meantime it has reached a fruitful direction.

Second:
The intention of my post is _not_ to find ways to help publishers do their business, to help them increase their profit, to "please" them, to sell LilyPond's soul to the "sharks" or whatever some commenters wanted to read from it. I must admit it's somewhat disappointing to be misunderstood that way, and I somehow feel attacked below the belt by such comments.

###

> To whom LilyPond should strive to "offer the future"?
> IMHO, certainly not to the "[...] big house[s] with traditions,
> regulations and limitations".

> What's for the LilyPond team in spending resources trying to work around
> those self-inflicted limitations?

and

> "Can you tell me why we should be interested in helping music publishers exactly?"

I don't inherently care about the fate of publishing houses. But I think there are a few "self-inflicted limitations" in LilyPond's attitude that should be overcome to assist LilyPond to survive or at least to prevent it from eventually being doomed to insignificance.

The main point here is not about helping publishers do their business but (and that is a point that hasn't been discussed at all in this thread) to help LilyPond evolving through an increased user and developer base.

LilyPond's developer base is too thin, to a dramatic extent in my opinion. As it stands, the withdrawal of *any* core developer would have a noticeable impact, and there are a number of persons whose withdrawal (for whatever reason) could trigger an existential crisis.
LilyPond development is much too slow IMO (I don't know if it has decreased or if it was always like this). There are so many brilliant ideas around to make LilyPond even better, and there are so many complaints around about things that LilyPond should do better to be really usable - but much of it just doesn't happen because there are too few man-hours available to do something about it. The current discussion is a very good example for that, and I don't know if the resources are available to do it better this time.

And LilyPond's user base is too small. The results of my recent survey make me believe that one cannot wholeheartedly offer LilyPond services on professional level, as it simply isn't sufficiently clear that there will be enough capable people around for a long time. It is a sad irony that you can claim a fundamental superiority of open standards and plain text file formats, but in reality this doesn't help anyone because you can't rely on the availability of human resources.

Therefore:

• Anything that increases the developer base is an asset for LilyPond as a project.
• Anything that increases the user base raises the chance to increase the developer base.
• Paid projects, particularly big ones with long-term perspectives, do both of the above.
• Such projects are very likely to directly contribute back to LilyPond.

###

The fact that it is practically impossible today to deliver LilyPond files to commercial publishers rules out a whole category of potential users: (highly professional) music engravers who do (want to and/or need to) work for commercial publishers.

I can see that a GNU project is not interested in this kind of work and explicitly won't endorse it. And that it doesn't consider "market share" a goal to be pursued. To some extent I have even accepted that as a fact. But I am sure (and partially know) that a lot of LilyPond users would be happy to work for commercial publishers - and currently they only have the choice to use other programs or just not to get these commissions.
As long as the GPL explicitly permits the use of GPLed software to produce commercial content such use is legitimate, and users pursuing that line of work shouldn't be treated like heretics.
However, if people feel strongly about it, perhaps make the recommendation that LilyPond be placed under a license that prohibits its use for proprietary documents, and the wider community can discuss that proposal.

Opening the door to the publishing business would open the door for existing users to distribute their work in more places. It would open the door for said category of potential users to use LilyPond. As others have stated the fact that LilyPond is non-existent in the publishing business also causes it to remain non-existent in other "markets", namely the educational - and I'm talking of schools and music schools as well as academic musicology. Which in turn would "produce" new users.

Opening the door to the publishing business would also open the door to academic musicology. I think one of the most prominent evolutions of the decade is the exploration of digital edition concepts. This is also interesting for LilyPond because a) engraving tools compiling text are natural choices to process text-encoded content (as is generally the case) and b) it is a strong tendency (if not a completed transition) to think in terms of free software and content there. (In Germany you will only get public money for research projects in the "digital" domain if all resulting material is done with open, accessible standards.) Currently edition projects and institutes don't see using LilyPond as a viable goal because it will require substantial work to be able to do that, and because they can only judge it from the perspective of their individual project - and for an individual project the necessary effort would clearly be inappropriate.
This is the reason why I see it as an important consideration to adopt additional encoding standards like MEI (or even MusicXML) more vigorously than LilyPond does so far (I think one could consider LilyPond a classical case of a vendor lock-in situation).


###

It has been brought up that adoption of LilyPond by publishers would restrict access to culture by "creating" editor's copyright on music that should be in the public domain.

There are (at least) two fundamental flaws in this argument:
First:
When a publisher releases a new edition of an old work its legal status is in no way affected by the tool used. So you can't take that as an argument against endorsing the use of free tools for commercial products. However, when such an edition runs out of copyright, or the publishers should change their minds the editions are already accessible in open formats and for free tools. So using LilyPond for commercial editions can be seen as an (albeit hypothetical) advantage.
Second:
A commercial edition of a public domain work does not restrict access to the work in any way. It just doesn't provide improved free access. So from the perspective of free culture is simply doesn't matter if the publisher produces this edition or not. Consequently this is completely irrelevant to the question whether publishers should be encouraged to use LilyPond.

Additionally, new editions often *do* improve access to the compositions, by means of an improved editorial quality. As a professional musician, I'd nearly always prefer paying for a commercial edition over printing an outdated 19th century edition from the internet.
It has been stated that commercial publishers produce editions that suck for their editorial quality and that therefore these publishers are not worth being convinced of using LilyPond. Sorry, but this really not very helpful. I didn't perform a qualified comparison, but I can't imagine that the average and overall editorial quality of "free" editions comes even close to what commercial publishers have achieved over the centuries. This is also true for projects like IMSLP or Mutopia. These don't provide any inherent mechanism for editorial quality control. The vast majority of scores to be found there are simply copied from existing editions, without any standardized way to guarantee their correctness. And did anybody think about the fact that nearly all of these "model" editions have simply run out of copyright and wouldn't exist if they hadn't been produced by commercial publishers in the first place? Truly Free editions would require people going to the archives and producing new editions from the original sources. Then we can start talking about *their* quality ...

In general I only use these free resources to get a first impression - which is of course an example use case for the free distribution of culture. But edition quality is an aspect that is somewhat at odds with the free distribution, and I would not accept sacrifying this part of the equation.


###

Now to the aspect of "convincing" publishers.

> If they are such corporate dinosaurs that do not recognise the benefits of advanced lilypond technology,
> open source and open systems, of what concern is it to the community of lilypond engravers?

I have outlined above why this should be a concern to the LilyPond community. But the first part of that statement is too short-sighted IMO.

I have talked with a few "corporate dinosaurs", and it quickly became clear that there is no sense in continuing my efforts, presumably until the respective persons have retired. One bluntly told me that plain text is outdated and that he had thought this had been overcome decades ago. Another one only slightly concealed (under politeness) that his company will only consider any technological change when they can't prevent it because everybody else has already jumped the bandwagon.
But most others I have talked with have been seriously interested and grasped quite clearly what I am talking about. They asked exactly the right questions and saw potentials and problems quite clearly too.

But that's not enough to consider a change.
When a business is proposed a new technology I think there are basically three considerations:
a) The suggestion promises solutions for problems the business is actually and significantly suffering from
b) The suggestion promises (mostly financial) benefits that outweigh the investment.
c) The risk of failure seems overseeable.

ad a)
Most people clearly see the advantages.
Notably engraving quality is one of the least in this regard. Most publishing houses are satisfied with their output because they have either invested quite some effort in it or they simply don't care. There is one point that can be made here, and that is the out-of-the-box quality, which can be of interest when it comes to producing performance material on short notice and only later refine that to publication quality.
The most compelling points seem to be the potentials of project management/documentation through version control and the option of distributing work over arbitrary numbers of collaborators. The Grid approach made quite some impression particularly.
As mentioned I have the impression that the overall issues of data integrity and the risks of relying on commercial software have become more present in publishers' minds by now.

b) and c) are the problematic parts. And here I think the "risk" part is more of an issue thant the "investment". People can't really oversee how the new technology can fit into their existing infrastructure, particularly given the fact that they do and will always have contributions by numerous people from outside that have to be integrated. Add to this the question if we can reliably enough convince anybody that "we" will be around in a few years and that there is a sufficient user base to guarantee that they will always find someone to get "tech support" from.

So finally I'm back at the beginning, namely my original post's question, preparing a convincing set of facts, arguments and "promises" that help to overcome the reservations with regard to b) and c) of the above list. I have the impression I'll be more or less alone with that, so I'll try to put my things together on my own. Having triggered quite a bunch of different (and hopefully fruitful) discussions is something too ...

Best
Urs

Am 17.04.2015 um 15:03 schrieb Urs Liska:

Just one more of the fundamental questions I took home from the Musikmesse ...

The question can be asked somewhat less pretentious then in this message's subject line, but I think it actually boils down to no less than that.

You know that I have again been at the Frankfurt Musikmesse this week, and again I had the opportunity to talk with various people from publishing houses (names only privately ...), and I was (unpleasantly) surprised that I didn't always have fully satisfactory answers ready.

The questions came in various variants of "why should a publishing house use LilyPond?" And despite all the reasoning and writing I have produced over the last years I didn't always find "the" striking key features that were convincing in the concrete situation.

I think  I have always taken a perspective that was focused slightly beside the point, namely the perspective of an individual editor or the team of editors. This is perfectly transferable to a publishing house starting from scratch, but not to a big house with traditions, regulations and limitations.

Compared to last year I have the impression that many people have become more aware of the basic questions about longevity of binary and textual data formats and data processing. The question has become much less "why should we consider dropping Finale and Sibelius, it's working, heh?" and more "OK, we see that we need an alternative approach, but how do you convince me that LilyPond has to be it?"

We always say that text based tools are superior because they are much less prone to become unusable. This, and the potentials that come from version control, make quite some impression, but again this is only relevant to new material and doesn't take into account two important issues:

1)
Publishers receive heterogenuous data material from various sources (editors, composers, engravers), and these are mostly done using Finale or Sibelius (and in some cases Score). It is completely out of question to requre all these people to switch to LilyPond.

2)
One major question publishing houses consider today is how to carry their (digitally) existing material to the future. By now they have realized that simply using the latest version of the mainstream programs is calling for disaster, that's good for us. But again, the question is: Can we really offer "the" solution for this?

The immediate idea would be to go some route via MusicXML and offer hassle-free workflows to convert existing editions or said received data material) to LilyPond. But firstly I don't think we can already guarantee this. And secondly, the question is natural to pop up: If one already uses MusicXML as the permanent and future-oriented storage format, where is then the need to consider LilyPond for this?
I think it is reasonable to expect that there will always be mainstream programs that can work with MusicXML files, and their user base will probably always be larger than ours.

###
So, now I've somewhat laid out why we are actually facing this question about "offering the future".
I can't give convincing answers to that, but of course I have a few ideas, and I would be happy about a constructive discussion. This is not a pipe dream discussion as I will (have to) pick up the communication with the publishers soon. And it will probably make a good impression if I can show that we have taken their concerns seriously, especially if I can come with some promising suggestions.

The first asset is the fact that plain text tools allow highly sophisticated workflows and adaptation of the programs' functionality. The biggest impression I could make was probably our "grid" approach (of course only backed up by the fact that we have successfully realized it in a real project), and - to some extent - the prospect of maintaining the whole edition process within one single context (the \annotation functionality in the ScholarLY library). But the latter was somewhat less striking because it seems most publishing houses don't really care anymore about the editorial process, and they have the impression that this could actually create more overhead than they have currently.

The second asset I see is that we can (principally, in the real-world it isn't completely mature yet) completely separate content from representation, which should be stressed very much when it comes to the questions of long-term data storage and of repurposing content.

In theory, LilyPond (as LaTeX) is better suited to process textual data than binary tools, simply because it's their natural appraoch. But I don't know what I would answer to the question "well, yes, I see, but what *impact* does this really have?"

There is one road that I could see as the "golden bridge".
I think MusicXML isn't the best solution for long-term management of editions. It's just too much focused on data exchanged between music software - and mostly pop music oriented tools. Looking for a fundamental solution to migrate the whole data base of a publishing house I would think that MEI is a much better solution because it's inherently more comprehensive, and because it has been originally conceived from an editor's perspective rather than music production tools.
Currently there doesn't exist *any* straightforward way to render MEI encoded data with a professional engraving program, so this could be the key feature for avoiding the question "so why should we prefer LilyPond over Sibelius or the new Steinberg app?".
IF we could come up with a promising path to let LilyPond work with MEI data (that is firstly: use MEI as input to LilyPond and/or convert MEI data to LilyPond files, and secondly: Be able to convert to both directions so one can also edit scores as LilyPond and convert them back to MEI for storage) that _could_ be the satisfactory answer I claimed as missing above. Publishers have more than once thought about a concerted effort with regard to data management. Of course that would also imply a platform for _distributing_ music, so the option/request to provide *digital* scores is also ubiquituous. Actually this was again suggested this week in one of my meetings, and I would love to pick that up with a more or less concrete but at least convincing outline.
At the same time the prospect of publishers (or "the" publishers in a common effort) would consider MEI this would motivate the MEI community, and if it would be somehow connected with LilyPond it would raise the chance that some of them would actually step out and start something worthwile (I know there is a latent interst in LilyPond within the MEI community, but there's no sufficient "market" for it so nobody actually came up with a solution so far.


####
####
To conclude:

- most people in the business have moved away from taken the status quo
  with Finale and Sibelius for granted.
- they know that they *have* to find new answers.
- many (except a few die-hard reactionists) see that LilyPond and friends *can* offer answers to their questions
- but they also see that these are maybe not the only possible answers and
- that we (currently) can't guarantee straightforward migration paths.

Market is hard, and everything is moving quite slowly, of course.
But IF we should be able to come up with convincing solutions or at least roadmaps I see that we now have better chances than ever to get LilyPond a foot in the door with the publishing business in general.

Sorry for that elaborate text, but I think it is important and hopefully fruitful.

Best
Urs



_______________________________________________
lilypond-user mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/lilypond-user