macOS 64-bit

[Jahrme Risner]

I want to preemptively apologize for the length of this email and give a TL;DR.

-----------------------------------------------------------------------------

TL;DR

I have been working on building a 64-bit macOS (x86_64-apple-darwin) release.

One option for build LilyPond for 64-bit macOS is Homebrew. Building LilyPond
with Homebrew has been met with partial success, but it is unclear whether the
ongoing work to make that method production ready would be worth the effort. My
full comments about working on Homebrew are at the bottom of this email.

In addition to Homebrew, I have also been looking at GUB and what I am
currently calling "macGUB". My explorations in GUB have already resulted in a
couple of patches, but there are further issues to resolve. "macGUB" is a very
early look at creating a macOS-specific build system that has as few
requirements beyond a clean install of macOS as possible. Comments on GUB and
macGUB follow below.

One final option which (for me) has met with failure is Nix. Nix is a
functional package manager (similar to GNU Guix) which allegedly works on
macOS. I was unable to get Nix to install correctly without extensive
troubleshooting (and even then it was rather brittle), so at this time, I
cannot suggest considering Nix as an official method for building or releasing
LilyPond for macOS. Further, the official Nix build farm has been unable to
successfully build LilyPond for macOS since 2017-08-15 so even with a smoother
installation there would be work necessary to make Nix viable.

-----------------------------------------------------------------------------

GUB

Currently, GUB has no knowledge of the x86_64-apple-darwin target. I have been
working on creating such a target (referred to as darwin-64 in GUB parlance).
This work has already resulted in two pull-requests to GUB so far with further
work yet to be submitted. Here are the two issues already resolved:

PR #61: Use generic Python database modules.
        https://github.com/gperciva/gub/pull/61
        This resolved an issue in the way the db python module was called which
        made GUB unusable on macOS as well as some Linux systems (Alpine Linux).

PR #62: Enable compiling librestrict on macOS.
        https://github.com/gperciva/gub/pull/62
        This change altered the way in which aliasing in librestrict was
        handled which made compiling librestrict possible on macOS

In addition to the pull request, I have also have work sitting on a branch that
is not yet ready for formal review, but if anyone else is interested can be
seen here:
    https://github.com/Jahrme/gub/tree/add_darwin-64

The current roadblock is Perl which will not compile on macOS. The log states
that the configuration script cannot find the standard c library and several
attempts to "point" the script to the library have been met with disappointment.

My current goal is to make GUB able to build the darwin-64 target on macOS, but
without the intent (yet) of enabling cross-compilation to darwin-64 from other
systems due to macOS SDK licensing issues. Once this has been achieved, the
next step would be to examine the offerings on
    https://opensource.apple.com
and hopefully be able to create the necessary portions of the SDK exclusively
from components explicitly release as open source by Apple.

If that is not possible, it may be that when releasing a new version of
LilyPond, either a contributor builds all release artifacts by running GUB on a
mac, or one contributor runs GUB on a mac to produce the mac-specific release
artifacts while a second contributor runs GUB on some other system to produce
all other targets.

In addition to this email I have some other thoughts more generally on GUB that
I am planning on sending shortly.

-----------------------------------------------------------------------------

macGUB

Right now I am (ball-parking it) perhaps 75% of the way to building LilyPond
from source on macOS with the only non-standard dependency being the Xcode
command line tools which are required by almost every other development tool
on macOS.

Currently I have two parallel implementations, one as a collection of
makefiles, the other as a python script compatible with macOS's system python.

In either case, these scripts recursively build the dependencies of LilyPond
from source which they download from the official release pages. This is slow
due to the time it takes some dependencies (e.g., gcc and glib) to compile.
While this is a non-optimal solution, it seems that as long as it was set up
carefully it may be a consistent way in which to specifically build on macOS.

My hope is that once macGUB can build LilyPond, it will at worst serve as one
way macOS users can get a 64-bit version of LilyPond, and hopefully would be an
option (barring getting GUB to work) to produce a (potentially unofficial) mac
release of LilyPond. Further, once the necessary flags/options for building on
macOS are understood, it may serve simply as a case-study on how to resolve
some of the GUB issues that currently prevent GUB from running on macOS.

-----------------------------------------------------------------------------

Homebrew (https://brew.sh) is a very commonly used package manager on macOS.

Within the last couple of years, Homebrew has chosen to completely remove all
tex-related content from their available packages. Along with the dependency on
address@hidden, the dependency on programs provided through texlive was grounds 
for
the removal of LilyPond from Homebrew's packages.

The dependence on address@hidden may not be an issue anymore; I say may as 
Homebrew
requires ongoing security patch support from upstream for versioned packages,
and states that versioned packages should be "expected to be used by a large
number of people", see https://docs.brew.sh/Versions for more information.

The dependence on programs from texlive IS still an issue that requires awkward
workarounds that would definitely prevent LilyPond from being accepted into the
core set of packages.

For anyone who is interested in examining what I have done so far (or better-yet
would like to collaborate on finding a solution) I have uploaded my work to
    https://github.com/Jahrme/homebrew-lilypond.git
which includes instructions on using the Homebrew tap to install LilyPond.

I can currently build the stable release (2.18.2), but the development release
(2.19.83) fails out due to (I believe) a previously discovered issue with clang
involving template resolution. I have begun examining the possibility of having
gcc as a dependency to remove the clang issue, but I have not yet had success.

One further issue with the Homebrew build is that when running the Homebrew
audit tool, it complains about LilyPond's placement of Emacs Lisp files. The
full error message can be seen in the section below. I don't think this is a
pressing issue, but if Homebrew was used as a method of distributing LilyPond
it may be that providing an option to configure where the Emacs Lisp files are
placed would be a low-effort solution that would remove the error.

At this point I am unsure whether it is worth the effort to continue pursuing
Homebrew as a possible method of distributing LilyPond in its current form. As
I currently understand the dependencies, the texlive related dependencies are
necessary for building the font files used by LilyPond, so it may be possible
that if pre-compiled versions of the font files were available the texlive
issue could become a non-issue. In regards to address@hidden, that might be 
solvable
by making it an internal dependency not (easily) accessible from outside of
LilyPond, however that may still be too much of a stretch for getting LilyPond
into mainline Homebrew. Either way, it is also possible (as I did) to provide
LilyPond through Homebrew without it having to be added into the main repo; the
main issue with creating a tap instead of having LilyPond officially added is
that when users use the brew search function the tap will not be shown unless
the user has previously added the tap.

Any comments for, against, or otherwise related to pursuing a Homebrew release
are welcome.

-----------------------------------------------------------------------------

Homebrew Audit Error

lilypond:
  * Emacs Lisp files were linked directly to /usr/local/share/emacs/site-lisp
    This may cause conflicts with other packages.
    They should instead be installed into:
    /usr/local/opt/lilypond/share/emacs/site-lisp/lilypond

    The offending files are:
        /usr/local/opt/lilypond/share/emacs/site-lisp/lilypond-what-beat.el
        /usr/local/opt/lilypond/share/emacs/site-lisp/lilypond-words.el
        /usr/local/opt/lilypond/share/emacs/site-lisp/lilypond-song.el
        /usr/local/opt/lilypond/share/emacs/site-lisp/lilypond-font-lock.el
        /usr/local/opt/lilypond/share/emacs/site-lisp/lilypond-init.el
        /usr/local/opt/lilypond/share/emacs/site-lisp/lilypond-indent.el
        /usr/local/opt/lilypond/share/emacs/site-lisp/lilypond-mode.el
Error: 1 problem in 1 formula detected

-----------------------------------------------------------------------------

Again, sorry for the length of the email!

Best wishes,

Jahrme