[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Parallelization of shell scripts for 'configure' etc.
From: |
madmurphy |
Subject: |
Re: Parallelization of shell scripts for 'configure' etc. |
Date: |
Thu, 16 Jun 2022 20:46:15 +0100 |
Probably my previous email
<https://lists.gnu.org/archive/html/autoconf/2022-06/msg00005.html> went
unnoticed… But as I was saying I have published
<https://github.com/madmurphy/not-autotools/blob/master/m4/not-parallel-configure.m4>
a couple of m4 macros for addressing exactly this point raised
<https://lists.gnu.org/archive/html/autoconf/2022-06/msg00000.html> by Paul
Eggert:
In many Gnu projects, the 'configure' script is the biggest barrier to
building because it takes soooo long to run. Is there some way that we
could improve its performance without completely reengineering it, by
improving Bash so that it can parallelize 'configure' scripts?
Example attached.
--madmurphy
On Thu, Jun 16, 2022 at 4:06 PM Zack Weinberg <zack@owlfolio.org> wrote:
> On Thu, Jun 16, 2022, at 12:13 AM, Warren Young wrote:
> > A Makefile can be viewed as a series of shell scripts interconnected by
> > dependency rules that dictate which elements can be safely skipped, which
> > must be run, and in what order.
> >
> > I believe it's possible to recast Autoconf so that it generates a
> Makefile
> > that produces the same outputs as a configure script, but which can be
> run
> > under "make -jN" to run in parallel, safely.
>
> Probably, yeah. I like the idea of using a Makefile, that means we can
> outsource the "do up to N things in parallel" part to the Make
> implementation. I have about half of a patch on my local HDD that converts
> auto*test* to that style of parallelism, and if I ever get the time to
> finish it, it might be adaptable to autoconf.
>
> Potential lower-hanging fruit, though, is to add hand-coded parallelism
> inside AC_CHECK_{DECLS,FILES,HEADERS,FUNCS,MEMBERS,PROGS,TOOLS,TYPES}. I
> wouldn't be surprised if many existing configure scripts spend the bulk of
> their time inside these, and they are already looping over lists, where we
> know the answer to each iteration is independent of all the others.
>
> Whatever we do needs to be carefully designed to avoid introducing
> nondeterminism in the contents of config.status and everything downstream
> of it, and each check's log output should be appended atomically to
> config.log.
>
> zw
>
>
not-parallel-configure.tar.xz
Description: application/xz
- Re: Parallelization of shell scripts for 'configure' etc., (continued)
- Re: Parallelization of shell scripts for 'configure' etc., Nick Bowler, 2022/06/14
- Re: Parallelization of shell scripts for 'configure' etc., Nick Bowler, 2022/06/14
- Re: Parallelization of shell scripts for 'configure' etc., Richard Purdie, 2022/06/14
- Re: Parallelization of shell scripts for 'configure' etc., Demi Marie Obenour, 2022/06/16
- Re: Parallelization of shell scripts for 'configure' etc., Alex Ameen, 2022/06/16
- Re: Parallelization of shell scripts for 'configure' etc., Bob Friesenhahn, 2022/06/17
- Re: Parallelization of shell scripts for 'configure' etc., Paul Eggert, 2022/06/14
- Re: Parallelization of shell scripts for 'configure' etc., Bruno Haible, 2022/06/15
- Re: Parallelization of shell scripts for 'configure' etc., Warren Young, 2022/06/16
- Re: Parallelization of shell scripts for 'configure' etc., Zack Weinberg, 2022/06/16
- Re: Parallelization of shell scripts for 'configure' etc.,
madmurphy <=
Re: Parallelization of shell scripts for 'configure' etc., Chet Ramey, 2022/06/14
Re: Parallelization of shell scripts for 'configure' etc., L A Walsh, 2022/06/18
Re: Parallelization of shell scripts for 'configure' etc., Tim Rühsen, 2022/06/18
Re: Parallelization of shell scripts for 'configure' etc., Alexandre Oliva, 2022/06/21