Re: GNU make 3.81beta4 released

[Markus Mauhart]

"Eli Zaretskii" <address@hidden> wrote ...
>
>> From: "Markus Mauhart" <address@hidden>
>> Date: Fri, 20 Jan 2006 00:49:27 +0100
>>
>> "Eli Zaretskii" <address@hidden> wrote ...
>> >
>> >> From: "Markus Mauhart" <address@hidden>
>> >> Date: Thu, 19 Jan 2006 00:55:46 +0100
>> >>
>> >> Now I continued with your suggestion "-j 64" -- it run AFAICS allmost
>> >> 1m without errors until I got ...
>> >>     Assertion failed: a == g->changed, file .\remake.c, line 169
>> >> ... this comes from an assertion I had inserted around this bug:
>> >>     g->changed += commands_started - ocommands_started;
>> >> (g->changed is only 8 bits wide).
>> >
>> > What is the value of commands_started and ocommands_started at that
>> > point, and what is the value of g->changed?
>>
>> see today's 3 traces below
>>
>> > What exactly is that assertion testing?  I mean, what is `a' to which
>> > it compares the value of g->changed?
>>
>> yesterday it was ...
>>
>> {
>> int a = (int)commands_started - (int)ocommands_started;
>> a += g->changed;
>> g->changed += commands_started - ocommands_started;
>> assert (a == g->changed);
>> }
>>
>> Today I changed it to ...
>>
>> {
>> unsigned int const g_changed_old = g->changed;
>> int a = (int)commands_started - (int)ocommands_started;
>> a += g->changed;
>> g->changed += commands_started - ocommands_started;
>> if (a != g->changed)
>>     {
>>     fprintf (stderr, "\n(g_changed_old, commands_started, ocommands_started) 
>> = (%u ,%u ,%u)\n"
>>             ,g_changed_old ,commands_started ,ocommands_started);
>>     g->changed = (commands_started != ocommands_started);
>>     }
>> }
>>
>> 3 traces from 3 successfull(!) "-j 64" runs:
>>
>> (g_changed_old, commands_started, ocommands_started) = (0 ,419) //forgot one 
>> %u
>> (g_changed_old, commands_started, ocommands_started) = (0 ,436) //forgot to 
>> rebuild
>> (g_changed_old, commands_started, ocommands_started) = (0 ,466 ,0)
>
> ...
>
>> Btw, such successfull "-j 64"-build then sometimes uses ~50 compilers
>> in parallel - with my 1GB this is no problem with msvc71, while yet "-j 3" is
>> a big problem with mingw-gcc344 (both -O2).
>
> What do you mean by ``a big problem''?  Could you tell the details?

consuming (allmost) too much memory - with mingw-gcc344 I had to stop it
cause it would take at least 100 times the "-j 1" time, and even with "-j 4"
I stopped it after ~1m cause 4 compilers sometimes concurrently touching
phys-RAM * 1.2 (1.5 including the rest of the system) is a significant
performance problem, at least with winXP.


>> >    g->changed = commands_started - ocommands_started;
>>
>> or ... = (commands_started != ocommands_started)
>
> Yes.  Did you try such a change, and if so, did it allow your build
> procedure to come to a successful completion?

yes, 100% success for all 4 or 5 "-j 64"-msvc-builds I did.


> I don't think load balancing is disabled in recursive Make's, it is
> disabled because you didn't tell Make (via the -l switch) that you
> want it to pay attention to the load.

I meant that recursive sub-makes not knowing the "trick" would get -j=1,
while ALL others would use the constant value the "trick" stores in some
environment variable -- with gmake's builtin "-j n" AFAIK sub-makes
get dynamically adjusted smaller "n".

>> 2) "-j noNumber" (btw, array sizes surely reduced to MAXIMUM_WAIT_OBJECTS, 
>> and
>>                  overflows handeled correctly for some levels of callers)
>> As I said, this has the advantage of propagating to recursive $(MAKE) 
>> whithout
>> tricks, but the disadvantage of looping forever in all my tests:
>> Initially ~30 trivial cmds (.bat files) are executed successfully 
>> (containing echo ...).
>> Then in my last test I get 89 traces ...
>>     process_easy() failed failed to launch process (e=-1)
>
> I'd be interested to know what part of process_easy failed and why.

As I said that I added some (64-)overflow handling "at the ends" of all (few)
code-paths that would end in process_register(): process_register(..)
itself wont overflow but return false, and its callers would either
reasonably use this result (if possible) or more likely would use my
new function "bool can_process_register()" sometimes before - in case of
process_easy() I inserted "! can_process_register() ||" before "process_begin"
to make use of process_easy's existing error handler.

The caller of this failed process_easy() is start_job_command() and it
emmits the trace from above. But remember that some minutes later
we end up with 89 zombie-childs - IMO the 1st step to solve this would
be, after emitting the trace, to proceed with the same kind of cleanup
that other parties inside start_job_command() do: "goto error".
-->
Ok, I tested this one:
1) run "-j noNumber" with old code -> ~90 "process_easy() failed", then
   some minutes successfull build, then enter loop forever, must kill it,
   ~90 .bat files remaining --> reproduced the problem we want to solve.
2) same after rebuild with added "goto error":
  Only 2 "process_easy() failed", then some minutes successfull build, make
  stops with "make_msvc.net2003: *** [all] Error 2", 2 .bat files remaining.
  ... that is allthough it was not handeled like desired ("load too high,
  retry later"), the cleanup was much better - ideal cleanup would at least
  delete the bat files and probably stop immediately (allthough late stopping
  may be by design).
  No start make again (whithout "make clean" as prefix, which I do before
  every other test), make proceeds, and after 18 seconds I see my makefile's
  trace 'Done making target "all"', make stops whithout error.

As Paul said, your suggestion to handle every surly temporary resource
allocation problem like (existing) "load too high handling" is the way
to go.


Please see
http://members.chello.at/ejg8dk49/make.cvs.20060114.j_patch.zip (1.6MB)
for the sources of my last test, then built with make_msvc_net2003.vcproj
and used the "Release" build.
Most differences allready reported, otherwise maybe you can incorporate
them into your next patch.


Best Regards,
Markus.