Re: go-build-system possible improvments

[Maxim Cournoyer]

Hi François,

François <francois-oss@avalenn.eu> writes:

> Hello,
>
> I come back to public guix-devel as I think it can be of interest to
> others too.
>
> On Sat, Apr 17, 2021 at 08:57:03PM -0400, Maxim Cournoyer wrote:
>> JOULAUD François writes:
>>
>> > I am still unsure of what to do with versions and Go but the ability to
>> > pin version will surely be useful in some way.
>>
>> Yeah, I was expecting it might unlock building the new protobuf package
>> especially a couple version backs, it had a really messy dependency
>> chain including past versions of itself, but it turns out that for now
>> the bigger problem is the lack of support for Go modules.
>
> Yes. This is the biggest lock. Supporting go modules will really unlock
> a lot of things.

[...]

> I just made a lot of reading this week-end and I was more oriented towards
> using GOPROXY=off and populating on-disk [module cache] which have the
> exact same organisation as GOPROXY as it is indeed the last layer of
> proxying. If it works it will avoid the need for a specific local server.

That'd be nice!

> Concerning the former dependeency loop with google-protobuf it was
> (before 1.26) indeed necessary to have all go.mod files for all versions
> referenced recursively. Go downloads (or need to have in cache) only
> the go.mod files though because once it have the go.mod the [Minimal
> Version Selection] kicks in and keep only the higher version
> requested[^1].
>
> So we could cheat by putting only go.mod files for older versions in
> out local cache without the need for anything else and it should work.
>
> The same thing applies to conditional compilation. Go needs the go.mod
> files but don't need the code itself until it tries to compiles it.
>
> So we have probably a safe way to import those packages without too
> many useless packages in our dependency graph even if it can require a
> solution to easily add specific go.mod files in our outputs.

Interesting ideas!  I didn't know that.  My high level idea was to mimic
GOPATH but with GOPROXY; instead of source files, we'd have the metadata
+ zip (sources) populated in a given directory (say, the gocache/
subdirectory) of every Go package in Guix, and then could build a union
of these various directories into one (matching the expected GOPROXY
file layout) in the Go build system and profile hook.

> They, at Go, are nevertheless [working][lazy-loading] to modify this
> behaviour in order to avoid useless network calls to GOPROXY but it
> could not land in Go 1.15. It should be there in next release.

You meant Go 1.16, right?  I'm happy if they're working to simplify
this.

> P.P.S. other solution, perhaps simpler is to not support go.mod at all
> (GO11MODULE=off) and to populate the local GOPATH as we see fit. I think
> (but I am not sure) that Debian took this route. Not found of this
> solution, I think we have the means to do better in Guix and to keep
> more in touch with upstream methodologies.

That's exactly what we're doing now (disable Go module with
GO11MODULE=off and use GOPATH to find the sources).  Debian does
something lazy like just calling 'go module vendor', which builds a huge
directory filled with all the sources needed.  So does Nix.  Only Gentoo
goes to some length, from what I had seen last I check.

Thank you,

Maxim