go-build-system possible improvments

[François]

Hello,

I come back to public guix-devel as I think it can be of interest to
others too.

On Sat, Apr 17, 2021 at 08:57:03PM -0400, Maxim Cournoyer wrote:
> JOULAUD François writes:
> 
> > I am still unsure of what to do with versions and Go but the ability to
> > pin version will surely be useful in some way.
> 
> Yeah, I was expecting it might unlock building the new protobuf package
> especially a couple version backs, it had a really messy dependency
> chain including past versions of itself, but it turns out that for now
> the bigger problem is the lack of support for Go modules.

Yes. This is the biggest lock. Supporting go modules will really unlock
a lot of things.

> I'm glad to hear it!  I had read lots about Go and had some kind of plan
> for the GOPROXY (the project doing best in this regard is Gentoo); in
> theory it's possible to populate a cache with the source artifacts in a
> given directory (probably via a union of the individual packages
> contribution to the cache), then setup GOPROXY as a file server to
> provide these at build time.

I just made a lot of reading this week-end and I was more oriented towards
using GOPROXY=off and populating on-disk [module cache] which have the
exact same organisation as GOPROXY as it is indeed the last layer of
proxying. If it works it will avoid the need for a specific local server.

Concerning the former dependeency loop with google-protobuf it was
(before 1.26) indeed necessary to have all go.mod files for all versions
referenced recursively. Go downloads (or need to have in cache) only
the go.mod files though because once it have the go.mod the [Minimal
Version Selection] kicks in and keep only the higher version
requested[^1].

So we could cheat by putting only go.mod files for older versions in
out local cache without the need for anything else and it should work.

The same thing applies to conditional compilation. Go needs the go.mod
files but don't need the code itself until it tries to compiles it.

So we have probably a safe way to import those packages without too
many useless packages in our dependency graph even if it can require a
solution to easily add specific go.mod files in our outputs.

They, at Go, are nevertheless [working][lazy-loading] to modify this
behaviour in order to avoid useless network calls to GOPROXY but it
could not land in Go 1.15. It should be there in next release.


[module cache]: https://golang.org/ref/mod#module-cache
[lazy-loading]: 
https://go.googlesource.com/proposal/+/master/design/36460-lazy-module-loading.md
[Minimal Version Selection]: https://research.swtch.com/vgo-mvs
[^1]: Which seems strange until you understand that you always require a
"minimal" version in Go. The minimal version needed to build a project
is thus the maximum of all minimal required versions. The link explains
this a lot better than me ;-)

François

P.S. just as writing this I found that if we want to limit the number
of versions of Guix-packaged Go modules we could devise a mechanism to
force upgrade (go get -u) to the latest in-Guix version. This open a
lots more question to me than it resolves.

P.P.S. other solution, perhaps simpler is to not support go.mod at all
(GO11MODULE=off) and to populate the local GOPATH as we see fit. I think
(but I am not sure) that Debian took this route. Not found of this
solution, I think we have the means to do better in Guix and to keep
more in touch with upstream methodologies.