Re: packaging a golang package

[Timmy Douglas]

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Hello again, François! I've redirected this thread to guix-devel, and
> the bug since we've begun discussing implementation details.
>
> JOULAUD François <Francois.JOULAUD@radiofrance.com> writes:
>
>> First is to use vendored dependencies (when upstream provides them). This
>> one has the merits of simplicity (as we just have to download the
>> source and launch `go build` or whatever is needed) and less risks of
>> bugs. The downsides are known: difficult to audit Licences, duplication
>> of code source, difficulty to follow security bugs, etc.
>
> -1 to this approach (explanation below).

Seems like there could be two sub-scenarios here: the code could use go
modules or not.

>> Second is to re-vendor package from go.mod (coming directly from code
>> source or synthetized from source) by creating a source derivation
>> including all dependencies. This is the strategy followed by Nixpkgs
>> as explained in [1]. (It seems to me this is the route followed in
>> the patches to  from Helio in [2] but I did not take the time to read
>> them.) With this approach we still have some of the downsides of using
>> vendored upstream but it is at least easier to verify the existence
>> of upstream.
>
> I don't fully understand this approach, so I don't understand how this
> is different to approach three? Does this mean we create a pseudo,
> non-public package which states all the dependencies as inputs? If so,
> why wouldn't we just go with option three?

I think this approach is like saying you git clone the upstream repo,
run go mod vendor or go mod download, then go build. Or whatever
buildGoModule does in nix (https://github.com/NixOS/nixpkgs/issues/84826)

I read through that issue and would personally vote for this approach of
using `go` to restore the code.

I think trying to reimplement go module restore process with Guix
packages is bordering on Not Invented Here. There would be a never
ending battle of trying to reimplement the go module restore process and
the amount of source packages would really clutter things up. I think
some of the issues with the distro wanting to change a package could be
solved with a feature to patch go.mod before calling `go` to restore.

>> Third is to package every dependencies. This is the most transparent way
>> and the one that leads to less code duplication but the downside is the
>> difficulty to scale with the number of packages (even if the objective of
>> patch at [3] is to automate it) and the need to have only one reference
>> version for each package in the distribution which have its own share of
>> problems (one of them being that we don't use those tested by upstream
>> as stated in [4]).
>
> I think this is the eternal conflict between distributions and
> code-bases. As a software engineer, I am a fan of vendoring. It removes
> entire classes of issues:
>
> - Dependency on remote servers to be up to fetch dependencies and
>   perform builds
> - Requiring some kind of system to pin versions
> - Needing to stay on top of releases even if that doesn't meet your
>   schedule or needs
>
> As a packager for a distribution, I dislike vendoring because of the
> reasons you outlined above, _but_ I also dislike building upstream
> software with versions of dependencies that weren't approved, tested,
> and verified, upstream. It seems to me like that's a recipe for
> unstable, maybe even insecure, software.
>
> Normally, distributions are forced to do this because their packaging
> and build systems are only set up to support one version at a time. Guix
> can theoretically support all versions, but doesn't want to take on the
> dependency graph explosion this approach would cause.
>
> If we go on historical precedent, and how Guix handles the other
> language ecosystems, we should only have one version of each dependency,
> and build all software which relies on that dependency on the version we
> have packaged. Guix has already begun taking on the difficult challenges
> with this approach, including patching upstream to work with versions of
> dependencies upstream was not built for.

That sounds like a pretty difficult challenge. Seems like it could
quickly become untenable if a commonly used library had some sort of
breaking change between versions and different versions of it were used
by different packages.

I don't think anything is stopping Guix from having multiple versions,
but hand-assigning them would feel pretty manual and error-prone. On the
other hand, like you said, the dependency graph explosion from importing
everything would be overwhelming.

> I dislike it, but I also don't think we should try to solve the broader
> class of issues while trying to implement an importer. It should be a
> larger discussion within the Guix community across _all_ language
> packages about how we might achieve upstream parity while still
> maintaining our goals as a distribution, all while not crippling our
> infrastructure and people :)
>
> That's my viewpoint, but I think we should all defer to some of the
> longer-term maintainers who have helped guide Guix.

You wrote up both sides pretty well, but I couldn't tell if you had a
strong opinion on having go restore dependencies vs Guix packaging
dependencies. You had a lot of strong points for vendoring, but you're
writing an importer...

> Thanks for writing up these options.

+1