Re: Overhauling the cargo-build-system

[Ludovic Courtès]

Hi Chris,

Chris Marusich <address@hidden> skribis:

> Ludovic Courtès <address@hidden> writes:
>
>> What I would have liked is to somehow replace the #:cargo-inputs
>> argument (which is build-system-specific and thus “opaque”) with regular
>> ‘native-inputs’ or ‘inputs’ field.
>
> That would be nice.  However, it doesn't seem possible to express
> Cargo's "dependencies" and "dev-dependencies" concepts using Guix's
> current package DSL.
>
> Consider the proc-macro2 and quote crates.  We added these two crates in
> commit 2444abd9c124cc55f8f19a0462e06a2094f25a9d, in the same patch
> series where we added #:cargo-inputs and #:cargo-development-inputs:
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35318
>
> Here is the Cargo.toml file for proc-macro2:
>
>   https://github.com/alexcrichton/proc-macro2/blob/master/Cargo.toml
>
>   [dev-dependencies]
>   quote = { version = "1.0", default_features = false }
>
> And here is the Cargo.toml file for quote:
>
>   https://github.com/dtolnay/quote/blob/master/Cargo.toml
>
>   [dependencies]
>   proc-macro2 = { version = "1.0", default-features = false }
>
> Here is a diagram of their dependency relationship:
>
>   +---------------+
>   |     quote     | <+
>   +---------------+  |
>     |                |
>     | dependencies   | dev-dependencies
>     v                |
>   +---------------+  |
>   |  proc-macro2  | -+
>   +---------------+
>
> To Cargo, this cycle is not a problem, since "dev-dependencies" are
> treated differently from "dependencies":
>
>   https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html
>
>   "Dev-dependencies are not used when compiling a package for building,
>   but are used for compiling tests, examples, and benchmarks.
>
>   These dependencies are not propagated to other packages which depend
>   on this package."
>
> The reason proc-macro2 declares a "dev-dependency" on quote is because
> proc-macro2 uses quote in its doc tests:
>
>   
> https://github.com/alexcrichton/proc-macro2/blob/e82e8571460a0a0e00f52f011a74a5e0359acf3e/src/lib.rs#L785

I see.

> This relationship between proc-macro2 and quote cannot be readily
> expressed using the current package DSL in Guix.  If you try to model
> "dependencies" and "dev-dependencies" as "inputs" (or "native-inputs",
> or some combination of the two), Guix will fail due to the cycle.

True, but we have the same problem with many non-Rust packages, which we
address in various way—e.g., via an intermediate “-minimal” variant.

> Presumably, proc-macro2 just needs the source of quote (and the source
> of proc-macro2's other dependency, unicode-xid).  When Cargo builds
> proc-macro2, it will take care of building quote and making it available
> during proc-macro2's tests.  Guix "just" needs to provide proc-macro2
> with the quote source.  You might think this poses a bootstrapping
> problem for Cargo, but I guess it doesn't.  As long as Cargo has the
> source for proc-macro2, quote, and unicode-xid, I guess it can build
> proc-macro2 and quote in any order.

In that case, would it work to turn “dev-dependencies” into dependencies
on the source rather than on the package?

> Unless we missed something in our discussion of patch 35318, there is no
> easy way to express the relationship between proc-macro2 and quote
> without changing (or mis-using) the existing package DSL.  In the same
> way that the package DSL introduced "native-inputs" and "inputs" as
> concepts to facilitate cross-compilation, one way to solve this problem
> might be to introduce a new concept to the package DSL that would make
> it possible for Guix to express this kind of relationship correctly.
>
> However, in the discussion of patch 35318, everyone (myself included)
> seemed opposed to changing the package DSL if we didn't have to.  For
> example, in response to an earlier version of the patch series in which
> we tried to map "dependencies" and "dev-dependencies" onto the "inputs"
> and "native-inputs" concepts (which was probably an abuse of the package
> DSL, since "native-inputs" is a cross-compilation concept), you said: "I
> don't understand yet why you change the role of 'inputs' compared to how
> it is in the rest of Guix."  Ultimately, we decided not to modify the
> package DSL or the meaning of "inputs".  Instead, we decided to encode
> the necessary information about dependencies in the cargo-build-system's
> arguments.  That is how we arrived at #:cargo-inputs and
> #:cargo-development-inputs.

OK, thanks for the reminder.  :-)

I’d be curious to hear what Ivan and others think of
<https://issues.guix.gnu.org/issue/35318> in hindsight.

> By introducing #:cargo-inputs and #:cargo-development-inputs as package
> arguments to the cargo-build-system, we were able to solve the cyclic
> dependency problem in one specific way.  Perhaps there are better ways.
> I agree it would be nice if it were integrated into the package DSL.  I
> think that changing the package DSL to suit our needs might work, but
> I'm not sure how to change it without making it too Cargo-specific.

Still, the notion of dependency definitely exists in Cargo, so it would
seem natural to use ‘inputs’ or ‘native-inputs’ to express that.

I realize my understanding of Crates is still too limited, but I think
our goal should be to somehow have something that’s closer to normal
<package> objects.  We’ll probably still need Cargo-specific extensions,
but it’d be nice if the <package> graph for Crates was meaningful.

Also, ‘guix import crates’ has all the info so we can always tweak it to
generate something different.

Thanks!

Ludo’.