Re: ‘cargo-build-system’ makes everything slow

[Efraim Flashner]

On Mon, Dec 02, 2024 at 05:44:16PM +0100, Ludovic Courtès wrote:
> Hi,
> 
> I was profiling the ‘dependents’ procedure use by the new ‘guix build
> --dependents’ option in (guix scripts build).  Like ‘guix refresh -l’,
> it lowers all the packages to a bag, constructs the graph, and traverses
> all the nodes backward.  And it’s slow (~25s).
> 
> The profile is clear (test program below):
> 
> --8<---------------cut here---------------start------------->8---
> 100.0% with-exception-handler at ice-9/boot-9.scm:1689:4
>   100.0% thunk at guix/store.scm:686:4
>     100.0% run-with-store at guix/store.scm:2196:0
>       100.0% #x7f1cb0ccbeb0 at guix/scripts/build.scm:596:6
>         100.0% #x7f1c9c353ef0 at guix/graph.scm:98:7
>           99.6% #x7f1cb1b61e00 at guix/store.scm:1925:0
>             99.6% loop at guix/store.scm:1925:0
>               99.6% add-edge at guix/graph.scm:92:7
>                 99.6% #x7f1c9c3340bc at guix/scripts/graph.scm:187:10
>                   96.3% #x15d3b3bc
>                     95.8% bag-node-edges at guix/scripts/graph.scm:159:0
>                       93.1% package->bag at guix/packages.scm:1755:0
>                         92.5% thunk at guix/packages.scm:1763:4
>                           89.3% lower at guix/build-system/cargo.scm:305:0
>                             33.5% vhash-assq at ice-9/vlist.scm:539:0
>                               9.4% #x15d3cf98
>                             21.0% #x15d3de90
>                             3.6% vhash-cons at ice-9/vlist.scm:449:0
>                               0.9% #x15d3cf98
>                               0.1% #x15d3b430
>                                 0.1% #x15d3daf4
>                                   0.1% reap-pipes at ice-9/popen.scm:183:0
>                             2.5% filter-map at srfi/srfi-1.scm:681:0
>                               2.0% #x7f1cb0a997f4 at 
> guix/build-system/cargo.scm:298:4
> --8<---------------cut here---------------end--------------->8---
> 
> 89% of the time is spent lowering ‘cargo-build-system’ packages,
> specifically in ‘expand-crate-sources’, which computes more or less the
> same thing over and over again.  (It also returns more than just
> transitive sources of Crates, AFAICS.)
> 
> Problem is little can be done about it, at least not without triggering
> a world rebuild.
> 
> But even then, it would help a lot to just not do that.
> 
> Thoughts?
> 
> Ludo’.

If its a world rebuild through the cargo-build-system its not so bad, CI
and QA build it out pretty quickly.  If it's everything, then yeah,
that's going to be slow.

Can we cache it after the first time like with guix shell?  Or a
hash-table of some sort? (I'm just using big words, I don't remember the
specifics of how the different hashing and caching bits work). Chris
Baines sped up the time to compute derivations for cross-compiling with
something with the make-rust-sysroot.

> (use-modules (statprof)
>              (ice-9 time)
>              (guix)
>              ((gnu packages) #:select (all-packages))
>              ((guix scripts build) #:select (dependents))
>              (gnu packages version-control))
> 
> (time (->bool (all-packages)))                    ;warm up
> 
> (with-store store
>   (statprof
>    (lambda ()
>      (dependents store (list libgit2) 1))
>    #:full-stacks? #t
>    #:display-style 'tree))


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

signature.asc
Description: PGP signature