Re: Go importer and packages with version flags

[Sarah Morgensen]

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> Sarah Morgensen <iskarian@mgsn.dev> writes:
>
>> Hi Katherine, Jack,
>>
>> Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:
>>
>>> Jack Hill <jackhill@jackhill.us> writes:
>>>
>>>> Hi Guix,
>>>
>>> Hey, Jack, a few thoughts.
>>>
>>>> While I was was working with the go importer today, it suggested I package
>>>> go-github-com-russross-blackfriday-v2. Fair enough, except we already have 
>>>> a
>>>> package for go-github-com-russross-blackfriday.
>>>
>>> I was poking around a rust code-base the other day and I noticed our crate
>>> importer, and thus a lot of crate packages, have major-version suffixes. I
>>> think one of the unique benefits of Guix is that it can simultaneously have
>>> multiple versions of libraries installed, and I think we should allow this
>>> for library packages.
>>>
>>> I know that leads to dependency graph explosion, but perhaps we only commit
>>> to substitutes for the latest version, and thus any packages using old
>>> versions. It should converge over time unless packages go unmaintained.
>>>
>>> I thought our current stance was to only allow one version at a time, but
>>> the crate packages made me question this. I'd like clarity too.
>>
>> I think there's a bit of a difference between (our packages for) the Rust and
>> Go ecosystems here.
>>
>> In the Go ecosystem, a module is uniquely identified by its module path,
>> e.g. "github.com/russross/blackfriday/v2".  According to Go's major version
>> suffix rules [0], "[s]tarting with major version 2, module paths must have a
>> major version suffix like /v2 that matches the major version."  Therefore,
>> each major version is logically a different module according to Go, and so I
>> think we should treat them as separate packages as well.  (Note that in many
>> cases we can use 'inherit' for brevity.)
>
> That's a great point! I hadn't considered that we could leverage this to
> consider major versioned Go modules as separate packages. That's great!
>
>> Additionally, the major version suffix rules dictate that "[i]f an old 
>> package
>> and a new package have the same import path, the new package must be 
>> backwards
>> compatible with the old package."  Assuming upstream sources follow the 
>> rules,
>> we should be able to update each Go package within each major version without
>> breaking dependencies.
>>
>> (A corollary to that is that packages often break if you try to use a v2 when
>> it is expecting a v1.)
>>
>> I think this differs from Rust, where we have, for example, package-0.1 and
>> package-0.2 coexisting.  This difference should prevent dependency graph
>> explosion for Go.
>
> It's nice that our Rust packages are enjoying the same stance, but I'm still
> not clear on why? Does Rust have the same guarantees?

I'll leave this for someone who actually knows the Rust ecosystem to answer :)

>
>> There are some caveats with "major version suffixes":
>>
>> * Major versions 0 and 1 don't get a version suffix (so no /v1)...
>>
>>     * ...except for module paths starting with "gopkg.in/", which always have
>>       a major version suffix, but theirs is styled ".v1" rather than "/v1".
>>
>> * Modules may either be located in the repository root, or in a "/v2"
>>   subdirectory (for major version 2).  This makes things difficult for our
>>   importer, because we can't know whether the unpack path should include 
>> "/v2"
>>   without looking at the repository contents.
>>
>> This is why Jack had to manually add "/v2" to the unpack path.  I recently
>> made some changes to the importer to better handle, for example,
>> "github.com/example/repository/subproject", but it doesn't yet discriminate
>> between "/subproject" and "/v2", so it treated "/v2" like a subdirectory of
>> the repository.  (Until we fix this properly, the importer should probably 
>> not
>> count major version suffixes when calculating the unpack path, since most
>> projects don't use a "/v2" subdirectory.)
>
> As an aside, when I started writing the importer, I didn't know it was a
> possibility to just use the Go toolchain to help us generate packages. I
> thought "the Guix way" was to do it all in scheme. It's nice that it's in
> scheme, but I would love to leverage the Go toolchain instead.
>
> IMO, module resolution and graph dependencies in Go are complicated enough
> that I'd much rather take the effort we put in trying to replicate and keep
> up with the Go toolchain's behavior, and spend that effort elsewhere.
>
> Does anyone have opinions on this?

Hmmm.  Setting aside whether or not we want to use external tools in
general...

If we use the Go tool, we shift the problem domain into "translating between
`go' and Guix."

For example: when Go downloads a module, it only saves the module, not the
whole repository*, so we can't use that to generate the hash.  (* Except it
does if you used GOPROXY=direct, but that's in the module cache, and only a
bare repository.)

Or, the fact that we import the latest available version of packages (unless
--pin-versions is used), but Go uses exact versions selected with MVS [0].

You might also be interested in taking a look at Gentoo dealing with this
translation problem [1].

I'm not saying that this would necessarily be a bad tradeoff either, but it's
definitely a tradeoff.

Did you have something particular in mind as far as leveraging the Go tooling?

>
>> All that to say... I think we should definitely maintain coexisting Go v2, 
>> v3,
>> etc. package definitions.  We should probably go the way of Rust though, so 
>> we
>> have them all in the same package, at different versions:
>>
>> (define-public go-github-com-russross-blackfriday-v2
>>   (package
>>     (name "go-github-com-russross-blackfriday")
>>     (version "2.1.0")
>>
>> instead of as different packages:
>>
>> (define-public "go-github-com-russross-blackfriday-v2"
>>   (package
>>     (name "go-github-com-russross-blackfriday-v2")
>>     (version "2.1.0")
>>
>> And of course, it should be policy to remove dependency packages with no
>> dependents.  (Perhaps we could write a new linter to warn if a "go-" package
>> has no inheriters and no dependents?)
>
> I disagree with this part, only because it's possible the packages are
> dependent on non-public (or at least not Guix mainstream) Guix packages. We
> get the wealth of the commons if we maintain this package in Guix
> proper. However, I think this is definitely an edge case.

Good point.

>> Does that sound reasonable?
>
> Reasonable? No, incredible! :)

Thank you.  I think I've just spent too much time reading Go docs in the past
couple months... not that I even write Go! :)

[0] <https://golang.org/ref/mod#minimal-version-selection>

[1] <https://github.com/golang/go/issues/35922> cmd/go: allow extraction of
urls used to download dependencies

--
Sarah