Re: Parallelism a la make -j <n> / GNU parallel

[John Kearney]

Am 05.05.2012 06:35, schrieb Mike Frysinger:
> On Friday 04 May 2012 15:25:25 John Kearney wrote:
>> Am 04.05.2012 21:13, schrieb Mike Frysinger:
>>> On Friday 04 May 2012 15:02:27 John Kearney wrote:
>>>> Am 04.05.2012 20:53, schrieb Mike Frysinger:
>>>>> On Friday 04 May 2012 13:46:32 Andreas Schwab wrote:
>>>>>> Mike Frysinger <vapier@gentoo.org> writes:
>>>>>>> i wish there was a way to use `wait` that didn't block until all the
>>>>>>> pids returned.  maybe a dedicated option, or a shopt to enable this,
>>>>>>> or a new command.
>>>>>>>
>>>>>>> for example, if i launched 10 jobs in the background, i usually want
>>>>>>> to wait for the first one to exit so i can queue up another one, not
>>>>>>> wait for all of them.
>>>>>> If you set -m you can trap on SIGCHLD while waiting.
>>>>> awesome, that's a good mitigation
>>>>>
>>>>> #!/bin/bash
>>>>> set -m
>>>>> cnt=0
>>>>> trap ': $(( --cnt ))' SIGCHLD
>>>>> for n in {0..20} ; do
>>>>>   (
>>>>>           d=$(( RANDOM % 10 ))
>>>>>           echo $n sleeping $d
>>>>>           sleep $d
>>>>>   ) &
>>>>>   : $(( ++cnt ))
>>>>>   if [[ ${cnt} -ge 10 ]] ; then
>>>>>           echo going to wait
>>>>>           wait
>>>>>   fi
>>>>> done
>>>>> trap - SIGCHLD
>>>>> wait
>>>>>
>>>>> it might be a little racy (wrt checking cnt >= 10 and then doing a
>>>>> wait), but this is good enough for some things.  it does lose
>>>>> visibility into which pids are live vs reaped, and their exit status,
>>>>> but i more often don't care about that ...
>>>> That won't work I don't think.
>>> seemed to work fine for me
>>>
>>>> I think you meant something more like this?
>>> no.  i want to sleep the parent indefinitely and fork a child asap (hence
>>> the `wait`), not busy wait with a one second delay.  the `set -m` +
>>> SIGCHLD interrupted the `wait` and allowed it to return.
>> The functionality of the code doesn't need SIGCHLD, it still waits till
>> all the 10 processes are finished before starting the next lot.
> not on my system it doesn't.  maybe a difference in bash versions.  as soon 
> as 
> one process quits, the `wait` is interrupted, a new one is forked, and the 
> parent goes back to sleep until another child exits.  if i don't `set -m`, 
> then i see what you describe -- the wait doesn't return until all 10 children 
> exit.
> -mike
Just to clarify what I see with your code, with the extra echos from me
and less threads so its shorter.
set -m
cnt=0
trap ': $(( --cnt )); echo "SIGCHLD"' SIGCHLD
for n in {0..10} ; do
    (
        d=$(( RANDOM % 10 ))
        echo $n sleeping $d
        sleep $d
        echo $n exiting $d
    ) &
    : $(( ++cnt ))
    if [[ ${cnt} -ge 5 ]] ; then
        echo going to wait
        wait
        echo Back from wait
    fi
done
trap - SIGCHLD
wait
   
gives
0 sleeping 9
2 sleeping 4
going to wait
4 sleeping 7
3 sleeping 4
1 sleeping 6
2 exiting 4
SIGCHLD
3 exiting 4
SIGCHLD
1 exiting 6
SIGCHLD
4 exiting 7
SIGCHLD
0 exiting 9
SIGCHLD
Back from wait
5 sleeping 5
6 sleeping 5
going to wait
8 sleeping 1
9 sleeping 1
7 sleeping 3
9 exiting 1
8 exiting 1
SIGCHLD
SIGCHLD
7 exiting 3
SIGCHLD
6 exiting 5
SIGCHLD
5 exiting 5




now
this code
function TestProcess_22 {
        local d=$(( RANDOM % 10 ))
        echo $1 sleeping $d
        sleep $d
        echo $1 exiting $d
}
function trap_SIGCHLD {
        echo "SIGCHLD";
        if [ $cnt -gt 0 ]; then
            : $(( --cnt ))
            TestProcess_22 $cnt  &
        fi
}
set -m
cnt=10
maxJobCnt=5
trap 'trap_SIGCHLD' SIGCHLD
for (( x=0; x<maxJobCnt ; x++ )); do
            : $(( --cnt ))
            TestProcess_22 $cnt  &
done
wait
trap - SIGCHLD

9 sleeping 5
7 sleeping 0
5 sleeping 8
6 sleeping 5
8 sleeping 6
7 exiting 0
SIGCHLD
4 sleeping 9
9 exiting 5
SIGCHLD
3 sleeping 5
6 exiting 5
SIGCHLD
2 sleeping 8
8 exiting 6
SIGCHLD
1 sleeping 8
5 exiting 8
SIGCHLD
0 sleeping 6
4 exiting 9
SIGCHLD
3 exiting 5
SIGCHLD
2 exiting 8
SIGCHLD
0 exiting 6
SIGCHLD
1 exiting 8
SIGCHLD




bash --version
GNU bash, version 4.2.24(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

uname -a
Linux DETH00 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC
2012 x86_64 x86_64 x86_64 GNU/Linux