Re: [GSoC] Integrating npm packages into the Guix ecosystem

[David Thompson]

Jelle Licht <address@hidden> writes:

> Hello Guix,
>
> Seeing as this time next year I'll be finishing up my studies, this was a
> now-or-never moment for me. I would love to spend this summer hacking on
> both guix, and reading up on npm.

I think this project is a great idea!  Feedback inline below.

> #+AUTHOR: Jelle Licht,  address@hidden
> #+DATE: 21-03-2016
> #+OPTIONS: toc:nil
> 
> * Overview
> 
> This project will allow Guix hackers to more easily package software that is
> distributed through the Node Package Manager (npm), as well as allowing Node
> developers on Guix to make use of the reproducible builds guarantee of Guix.
> 
> After completing this project, it should be possible to easily make use of the
> less-problematic packages in the npm registry on the Guix Software 
> Distribution.

I assume that by “less-problematic” you are referring to the packages
that actually upload their corresponding source code and not some
minified blob.  Maybe also the ones with the least insane dependency
graph.  Is this correct?

> * Project structure
> 
> Depending on findings in the early stages of the project, I foresee the
> following distinct parts:
> 
> 1. Extend Guix so it can 'simulate' the dependency graph generation of both 
> the
>    old and new npm [fn:6].

I don’t quite get this part.  Why does it matter how old and new
versions of NPM store dependencies?  IIRC, there’s a NODE_PATH
environment variable that we could set to specify where to find each
library.

Even if we have to make a similar directory structure as NPM for some
reason, shouldn’t we just do what NPM 3 does and forget about older
versions?

> 2. Extend guix with an algorithm that matches npm's package.json flexible
>    version specification to a specific version.

This sounds like it could be part of ’guix import npm’, but not in the
Guix package recipes themselves.  To date, we do not have a single
importer that recursively imports packages, so this would be somewhat
new territory.

> 3. Add a ~guix import~ backend for the npm registry

Yup. :)

> 4. Package npm modules in guix

Could you perhaps come up with an application or library to target for
packaging that will exercise all of the code you will be adding to Guix?
As you probably know, many non-trivial Node modules typically have
*massive* dependency graphs (hundreds of nodes), so I think we need to
be strategic in what we pick to package.

I should also note that NPM, like other language package management
systems, obscures the full dependency graph of a module by
differentiating between “dependencies” and “development dependencies”.
In Guix, we build from source, so we need those “development” (misnomer:
they are really build-time dependencies, but apparently only a developer
would want to build from source) dependencies, too.

Ideally, we would like to run test suites for node modules as well.
Unfortunately, npm packages are viewed more like binaries, so they strip
out things like tests.  It’s an open question if the tarballs available
on npmjs.com are worth anything at all, or if we should instead build
from the upstream source code releases.  Unfortunately again, those
upstream releases are usually just auto-generated tarballs from git tags
on GitHub, which may prove problematic because no one actually uses
them.

Long story short, we need to be diligent about ensuring that we are
truly building from source. :)

> 5. Interface
> 
> Right now, dependency resolution in npm is as stateful as can be, with even 
> the
> installation order [fn:6] making a difference for where npm expects to find a
> certain dependency. As two different dependency resolution mechanism are in 
> use,
> of which especially the newer one is problematic because of its habit of
> propagating dependencies upwards in the folder structure, both have to be
> supported by a guix module.

As I mentioned earlier, I don’t think that we need to support this at
all.  We should just arrange that NODE_PATH be configured correctly.
We’re *replacing* NPM, after all, so we should use a more sane
architecture if we can.  Does that sound doable or are there technical
reasons why it wouldn’t work?
 
> npm uses SemanticVersioning range patterns [fn:7] to declare dependencies
> between packages. A problem with this approach is that the same package
> declaration can lead to an entirely different dependency graph, which defeats
> the purpose of having a system with a focus on reproducible builds. If npm
> packages are to be used, these version numbers need to be locked down to a
> specific version, corresponding to the version that npm would install if left 
> to
> its devices. A potential problem is that the entire dependency graph has to be
> known ahead of time in order to pinpoint a 'correct' version of the 
> dependency.
> A consistent ordering for 'installing' dependencies also has to be decided 
> upon.

This is indeed a problem that needs solving *if* we consider recursive
package imports a requirement for this project.  Recursive imports
haven’t been done before, so maybe this could be a “nice to have” part
of the project rather than a “must have”.  This functionality would be
part of ’guix import npm’.

> After the dependency resolution has been worked out, it should be possible to
> create a guix import backend to leverage the code that has been produced up 
> till
> now to allow the packaging of npm modules. The last part of the project
> essentially serves as a starting point for packaging up useful npm packages.
> 
> A stretch goal for the summer would be to create a guix build system for a
> subset of npm packages, such as gulp.

I would promote this from a stretch goal to a “must have”.  One of the
first steps of this project should be to create (guix build-system node)
and (guix build node-build-system) modules that will do whatever the
standard build+test+install process is for node modules.  From what I’ve
seen, a lot of node modules need gulp or grunt in order to build, so
boostrapping one or both of those tools will probably be necessary in
order to package anything substantial.
 
> * Planning
> 
> As I am currently a novice with regards to the internals of guix and the guile
> programming language, up to the start of the actual project I will mostly be
> reading up and hacking on guix. As such, the planning becomes:
> 
> April 22 -  May 22:
>   - Getting to know Guix(SD)
>   - Package programs using the guix import module
>   - An informal specification of the npm dependency resolution mechanism
>   - Getting the know the guix community and what everyone is working on
>   - Getting familiar with the contributing work flows
> 
> May 23 - Jun 5:
>   - Formal specification to allow guix to simulate npm dependency resolution 
> (1)
> 
> Jun 6 - Jun 19:
>   - npm Version pinning should be working (2)
>   - start working on the guix import backend
> 
> Jun 20 - Jul 3: 
>   - Guix import backend should be finished by now (3)
>   - Start testing npm packages 
> 
> Jul 4 - Jul 10:
>   - Holidays! 
> 
> Jul 11 - Aug 7:
>   - Solve any problems and corner cases with building and installing npm
>     packages (4)
> 
> Aug 8 - Aug 23:
>   - If life goes a planned, Get all contributions ready to be merged back in 
> the
>     main Guix codebase.

Please add to the timeline when you would expect to have
node-build-system implemented.  I may have some code laying around that
could accelerate this a bit.

> * About me
> 
> My name is Jelle Licht, and I am currently studying Data Science at the
> University of Technology Delft. I finished my BSc in Computer Science in 2015,
> at the University of Technology Delft as well. Somewhere in the second 
> semester
> of my studies, my roommate was using Vi in front of me, and since then I've 
> been
> falling into the rabbit hole that is GNU/Linux, free software and the 'open
> source' community. Somewhere along the line I became a huge fan of Clojure, 
> and
> for a short while was a member of the very much unofficial Lisp Community 
> Delft.
> 
> I have been a small-time contributor to some free software projects on github,
> as well as having a day job to make ends meet using mostly Node. My online
> handles include 'wordempire', as I am quite fond of reading, and 'jlicht'. For
> the past month, I have been lurking on-and-off again in the #guix irc channel,
> as well as reading up on some of the motivations behind reproducible research
> and reproducible builds.
> 
> Besides one week for visiting family and general holidays, this project would 
> be
> my full time focus during the summer.

Thank you for your proposal!  Making JavaScript software packagable in
Guix would be a *huge* advance in our quest for reproducible web
applications.

- Dave