Re: 4.4.1 breaks recursive invocation with --print-directory [when adding to MAKEFLAGS]

[Satish Balay]

Thanks for the responses - I'm yet to try some of the suggestions [and
check if they are appropriate for my use case].

On Wed, 1 Mar 2023, Paul Smith wrote:

> On Tue, 2023-02-28 at 22:34 -0600, Satish Balay via Bug reports and
> discussion for GNU make wrote:
> > And the reason we use the order '-j8 ${MAKEFLAGS}' is so that we have
> > the following behavior':
> > 
> > - invoking 'make' defaults to using a default of '-j8'
> > - invoking 'make -j4' gives: '-j8 -j4' - i.e ability to override this
> > (pre-set) default [when needed].
> 
> I'm pretty confident that this has never worked.

Perhaps my minimal test code is not an exact representation. However
this usage does work in our code-base.

>>>
[balay@pj01 petsc]$ make --version |head -1
GNU Make 4.3
[balay@pj01 petsc]$ grep ^MAKE_NP configure.log 
MAKE_NP = 6
[balay@pj01 petsc]$  (ccache -c -C && make clean) > /dev/null && time (make  
all) > /dev/null

real    1m15.591s
user    6m4.442s
sys     1m9.179s
[balay@pj01 petsc]$  (ccache -c -C && make clean) > /dev/null && time (make -j1 
all) > /dev/null

real    6m5.378s
user    5m5.263s
sys     0m59.175s
[balay@pj01 petsc]$  (ccache -c -C && make clean) > /dev/null && time (make 
-j12 all) > /dev/null

real    1m8.758s
user    10m58.865s
sys     1m41.094s
<<<

Note: '-j6' is the value set in the sub-makefile for all the above runs.

I guess something that might have enabled this usage is - the toplevel
makefile is set as .NOTPARALLEL

Additional note: one of the reasons for the 2 level makefiles is - the
top level can be invoked with "make" binary available in user PATH
(i.e could be bsd-make, solaris-make, or older gnumake aka MacOS
etc..) - and the sub-makefile is setup to use gnu-make (could be a
newer make on MacOS).

[and sure - with some make implementation combination - some MAKEFLAGS
don't get passed in appropriately from top-level to gnu-make in this
setup]

Satish


> 
> When the initial, top-level make is invoked with -jN it creates the
> jobserver and it seeds the jobserver with N tokens to be used across
> all the sub-makes.  It then passes details about the jobserver, in
> MAKEFLAGS, to all sub-makes.
> 
> Any sub-make that sees the jobserver details in MAKEFLAGS will use that
> same jobserver and when they want to run a job they will retrieve a
> token from the jobserver.  If they get one they run the job.  Otherwise
> they wait until one is available.  They pay no attention whatsoever to
> the value of N in any -jN argument that they receive and they don't
> keep track of the number of jobs they are currently running and compare
> it to this value.
> 
> It is definitely not the case that you can limit (or increase) a
> specific sub-make to have a different number of jobserver tokens.  The
> only way you can change this is to create a NEW jobserver, with its own
> set of tokens that are completely distinct from the original jobserver.
> And the only way to do that is to _remove_ the jobserver details from
> MAKEFLAGS; here you are explicitly _preserving_ the jobserver details
> in MAKEFLAGS.
> 
> In fact, in earlier versions of GNU make we didn't even pass the -j
> option to sub-makes at all; now we do but it's really only there so
> people can query the MAKEFLAGS variable to see how many tokens were
> available.  GNU Make doesn't pay any attention to this value, in a sub-
> make.
> 
> I'm not saying that there isn't an issue here related to how we are now
> handling MAKEFLAGS.  But, the particular use case, at least as you've
> described it here, has never worked the way you expect.
>