Re: Line buffering doesn't work with --shard

[David King]

On the https://www.osboxes.org/ubuntu/ boxes for 18 and 14 both (after
running system updates, in case that matters), this sequence:

wget https://ftp.gnu.org/gnu/parallel/parallel-20191122.tar.bz2
tar -jxvf parallel-20191122.tar.bz2
cd parallel-20191122
./configure
cd src
# then the command from my initial report:
for x in `seq 1 10`; do cat /usr/share/dict/words; done | awk -F'\t'
'{print $1 "\t" $1 "\t" $1 "\t" $1}' | ./parallel --line-buffer -j16
--pipe --colsep '\t' --shard 1 -- cat | awk -F'\t' 'NF!=4'

just hangs with no output. Removing `--shard 1` causes it to start
producing output normally. This much shorter version hangs the same way:

cat /usr/share/dict/words | ./parallel -j2 --pipe --shard 1 -- cat

I observe the same behaviour on FreeBSD 12.1-RELEASE-p1. In all cases,
top shows a bunch of processes spawn and then fall to 0% CPU and stay there.

All of these work on those same Ubuntu 14/macOS 10.14 boxes from the
initial report but of course have the initial bug behaviour where
records are interleaved. And the hangs of course also mean that I can't
reproduce the interleaving problem on the osboxes.org VMs :)

I listed out the wget/configure bits above because I feel like I must be
doing something wrong somewhere for this to just not work at all in a
fresh VM on a popular OS. Is anything there obviously incorrect? I
scanned `./configure --help` for things that seemed relevant but didn't
find anything



On 2019.12.15 17:32, Ole Tange wrote:
> On Fri, Dec 13, 2019 at 3:15 AM David King <address@hidden> wrote:
>>
> 
> Thanks for a good error report. Really good example code.
> 
>> This happens using version 20191122 on macOS 10.14 and Ubuntu 14.04. I
> 
> No repro on Ubuntu 16.04 and 18.04. I do not have an Ubuntu 14.04 on
> hand, but I can set that up, if you can you verify that it breaks on:
> 14.04 from https://www.osboxes.org/ubuntu/
> 
> No repro on System Version: Mac OS X 10.7.5 (11G63). I do not have
> access a different version of MacOS.
> 
>> Separately, I've also experienced intermittent hangs with --shard. The
>> symptom is that the background processes have all finished and output
>> has stopped but parallel doesn't exit. I haven't found a minimal way to
>> reproduce it, but the second and third test cases above do for me.
> 
> Good to know. I also cannot reproduce this.
> 
> /Ole
>