[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
GNU Parallel Bug Reports Distribution of args skewed on input run with -
From: |
Henrik Mühe |
Subject: |
GNU Parallel Bug Reports Distribution of args skewed on input run with -X -j 4. |
Date: |
Tue, 27 Jun 2017 12:51:35 -0700 |
This was observed both in
GNU parallel 20130922
as well as
GNU parallel 20170622
Repro is for the latter (2017 version), the only difference is the
count given in tail that triggers the behavior between the two
versions. For brevity, let me give the repro without the data and then
at the end of the email show the full repro with the data. I tried to
find a repro with just seq but failed.
echo "Qlpo...more data see below" | base64 -d | bunzip2 - | head -n
1500 | parallel -X -j 4 "echo {} | wc -w"
output
375
375
375
375
echo "Qlpo...more data see below" | base64 -d | bunzip2 - | head -n
1550 | parallel -X -j 4 "echo {} | wc -w"
output:
7
7
7
1523
6
I would have expected for both to be evenly distributed among 4 tasks.
I glanced at the code and it looks like the arg_list variable is a lot
shorter than expected (the total of the smaller 4 numbers) but we
weirdly spawn and additional process which gets the bulk of the work
and then got lost.
Please help me understand if this is a feature that I fail to
understand or a bug.
Full commands to repro:
Good case:
echo
"QlpoOTFBWSZTWbfZQ+wAI11/gABf//////////////EAAEBgCByn1rKFs0rShSqV7s3bajtGuIp4hMjI0CaYARR6NJkDKAwDUGBGAAcZME0MhkZGTQ0AaDIwgGg0aZDENACKeKE9U9PTU20kyGSGmjTTTQaAAAAGgAARU/QAJkMgYlNlKe1TeknqemggaBoABoyBo9DjJgmhkMjIyaGgDQZGEA0GjTIYhoANNIIaAACNBqnpok8k2oAAAAAAGTvwUwauqmTpmggdspnda8maZZvmc/VmyxNr40zqg+BzVRX4bXC9UcSYZv3gGPUvPTG40rk7uutvIOD9iJQ4WZFitKZPlco53aSPkpdC+PLA2LJjhr3KdtKzhEHPREC+fUDELcCIUDOl5B1prGKvESVIb4MbTjw6LyY0YUBuJdCIkMVcVtVNruhdjrGRc1jWPATEGJbEvkgAx4MFqKi2A54JzlJiqytpmzdayNqvbweKqcIrJK5HKNzyKs1AnIy9Sx7b5odMhcZvNcEVSuMOu+DCtQq0/HCApgqonb2b9xqBuLyD0kALxA4ybhsSFES5tk1Yqae/FOsuwyNyNZLXk1FLJmtUXOkHWkcCqIoTC1ZDVNhDoutUtGqhNxNhiGzJgvK3G98prDmbdLZ1t0n0ipzjv42FYJe1UgbSAwpiXGG69mZGET9W2TrVVW2rGRuBWK+fCK2zlpTYXazsCtCrWR0OjVQcCBoQIgCEwxNbJ6pzTkaCVYLvZehg4Mg4pNXZcZVQ+DImmsk65OxjfrblM5MyBPconL+xUmMRAiIBxxbhSrZjVFrR9edpM7rBtbRU3XG/TOWw0O3nTkCBIWyqKTaWkmQoMRmqUFJuML2TFyYuMVM72unbJpRFdFIaNKC6oeOLhRWIc6Y+MTsQNxAkKbijCE0tVL6sWC4jzkdXm/ZtK04ivbU3hlrZavuLMtgMJN7D4zvAcDtdEIXUAIZcs8juGQ8FC7UgQIYuLBDDBYsqIMjJzttFMOe0KogIYbe7l+/UrN9F7dGqU0y2QDEoSN5QCuNRoRfG5iBhGZZHXlHq9dFky5PdO7lLYyeHbdd3ayRmOPHJZZ1EvOwC0oOBAvPeeY7bkcIBWLiFm7ptMsbML7WijngyqmPQvTIkY5purz7Yq3oi334eJponk52ZB9CieRQkpEMlAERE5AuqXD2VY593dGLtQzJcYsnvtkplkmmx+VYU8ZV8MAlBTEgOKRmMjtPC0QWogBkIa5oLILmpNyGZTa2l9Fh0UmbWwyvPOZmKJ1bbOZ4NueHLQ3aLgy+14jKWb1AVVdaoZTcwZEHLFYNu0tx7BxA51QVy5XptR0UdYoZyZyGAtZHsKniOYrG/yFt/mrnangdxEsagTtWY7FGESFE6cIMTlKrUdx4Ol6P76IXzuc9uPW/fL8ldhzI5/ParMi9ogQnIjHoFECOBGkFA7OSwXXF4RC2mVYigOZDp+TtMxB/LJqx205sTTN/G7FxT+LpfeysMTnRtaxma+EUjOwgVVLaeBQKQl/NXm8heNtVWHOV09OnGV9+6uLu8tcsXrMhFTz0Ke/zVBcramkhrFAHjI7vbESKloNzamKRjTyvhRpnjo+ExFGipl3iK4OMZA3UKl5CPqUDMjIi71amCvg/k2TLlhYKBZ4P6e3JLTZOLfJ1i4HhUgWPBaz7ccpwWBGUcGhUt0SSuhtK4r69Ql4m3XWaJm8zgDuD5uqoLcim8p1IzppYGFHkRlkV7W9nJcGdWPeskd0eEu3l3O26deQbdFCh1nVmro2c3Llg23NQMiOE6tnXMoSkLFtPofUlQhjZQ66uQvPIeMQOOKLOLu9tIacxegYFtKMYxAn41vtXG9l5blE3d4876GT8mQfPaVbKypvm1oYA0ERl58xSmkHTsS6ic4M+ac88LtJgXWsZWuKDjJ10QoR5sOSfeAbxzHaWSa8ZnAeO5YGs2nfcqTY+ozRtWTIRy0s5eMFxWWA6FxQkNKE5kaZrmW6q/z32sfX24pq5zJL6anyhrtVrqbOYv1buFW/MTDcISkORhGcxvCUoR0HZhDbUXoG1ld2PXhlfVwKKaS6KRllKjeVXT6q3I/3llslz8ndv6S0abMtyHZ0wtoERDLEJBEbAuEIbV43qsWvQvPuqYtD22oL/k3E5CRan0vMhlNOXDzCHB2TuJgSjkrPXZb9WOWdVmjAkqV9pG9Xi8OYACbiTYGDLTUQOpRLc1tYqOpO1cPIxu6bQ0tpm+LQUTPqz7WySkWlKzz7YBiYRM1i36Rtpso4tsu2zaatt5MpQYlIBVcNdx03yiXKkHELN2d0mjinZDhobbIggeEAMQMbooLSjEOCkhahJZ30nkcTyY0XVDB2hyuadUtHIYXJUJz1rdqinBwspnZtZX7hEpTAF6qGFNsFbB1OfBa9LL5/6w/B0JhA6kLzaCYILpcaWicONb6DoV95UEsq7eKOqK1YTcAMtRGVQW4JyVy6ECOO6K/ORS4aK7I349RusRBnWyNEIoERTS5CPWY3LR+FPS2eOSNd8qBrP81k8chTLiJ6oINGa2cz7BVUbZKDASvECeGUvSj2DA0QtGC55NjIEtUpDsqE9nZLsnMe/ebmEBZEQIw5Ue2wkLPhSfzRSBwx8IE8SUOaCUP91aJCYbeknLYDyYIQ0zvCvSO6LNssLUX/FGGeSZUJlk7NT0Jhq7/HluRAltp5FGCL1u66HV1+vH50FrQTmsGDqZ21NPsoKaN0LLRqAqQgdbqwL9fliZMMommC5B710RonI+hhOl+dqphCYV5hKhhsUtCTjdVWTrs2STqkQKsjOSi6v2VdhL0LU/EdFz7dOcdDg3vncXgLE2AUqpYe4dQk+wvMPJBi+7lvkug7YsHS1yHnRFXumbNbBCladLH3ce0TfHb55GkFvYJA22unjaAGEKCpiTcqeS8UZuhhs1RFdZ2AS4J5DkfwhCHvKjPJJT32c1Gis3WUz/dvTWOqX7EQ3xNKraXvyJ7NL+lzN24+wdIH4yCyVgvmWLXAaJD95FP508pPR6cvQiIAiIiAJ6QgRERAE/8XckU4UJC32UPsA="
| base64 -d | bunzip2 - | head -n 1500 | ./parallel -X -j 4 "echo {} |
wc -w"
Not so good case:
echo
"QlpoOTFBWSZTWbfZQ+wAI11/gABf//////////////EAAEBgCByn1rKFs0rShSqV7s3bajtGuIp4hMjI0CaYARR6NJkDKAwDUGBGAAcZME0MhkZGTQ0AaDIwgGg0aZDENACKeKE9U9PTU20kyGSGmjTTTQaAAAAGgAARU/QAJkMgYlNlKe1TeknqemggaBoABoyBo9DjJgmhkMjIyaGgDQZGEA0GjTIYhoANNIIaAACNBqnpok8k2oAAAAAAGTvwUwauqmTpmggdspnda8maZZvmc/VmyxNr40zqg+BzVRX4bXC9UcSYZv3gGPUvPTG40rk7uutvIOD9iJQ4WZFitKZPlco53aSPkpdC+PLA2LJjhr3KdtKzhEHPREC+fUDELcCIUDOl5B1prGKvESVIb4MbTjw6LyY0YUBuJdCIkMVcVtVNruhdjrGRc1jWPATEGJbEvkgAx4MFqKi2A54JzlJiqytpmzdayNqvbweKqcIrJK5HKNzyKs1AnIy9Sx7b5odMhcZvNcEVSuMOu+DCtQq0/HCApgqonb2b9xqBuLyD0kALxA4ybhsSFES5tk1Yqae/FOsuwyNyNZLXk1FLJmtUXOkHWkcCqIoTC1ZDVNhDoutUtGqhNxNhiGzJgvK3G98prDmbdLZ1t0n0ipzjv42FYJe1UgbSAwpiXGG69mZGET9W2TrVVW2rGRuBWK+fCK2zlpTYXazsCtCrWR0OjVQcCBoQIgCEwxNbJ6pzTkaCVYLvZehg4Mg4pNXZcZVQ+DImmsk65OxjfrblM5MyBPconL+xUmMRAiIBxxbhSrZjVFrR9edpM7rBtbRU3XG/TOWw0O3nTkCBIWyqKTaWkmQoMRmqUFJuML2TFyYuMVM72unbJpRFdFIaNKC6oeOLhRWIc6Y+MTsQNxAkKbijCE0tVL6sWC4jzkdXm/ZtK04ivbU3hlrZavuLMtgMJN7D4zvAcDtdEIXUAIZcs8juGQ8FC7UgQIYuLBDDBYsqIMjJzttFMOe0KogIYbe7l+/UrN9F7dGqU0y2QDEoSN5QCuNRoRfG5iBhGZZHXlHq9dFky5PdO7lLYyeHbdd3ayRmOPHJZZ1EvOwC0oOBAvPeeY7bkcIBWLiFm7ptMsbML7WijngyqmPQvTIkY5purz7Yq3oi334eJponk52ZB9CieRQkpEMlAERE5AuqXD2VY593dGLtQzJcYsnvtkplkmmx+VYU8ZV8MAlBTEgOKRmMjtPC0QWogBkIa5oLILmpNyGZTa2l9Fh0UmbWwyvPOZmKJ1bbOZ4NueHLQ3aLgy+14jKWb1AVVdaoZTcwZEHLFYNu0tx7BxA51QVy5XptR0UdYoZyZyGAtZHsKniOYrG/yFt/mrnangdxEsagTtWY7FGESFE6cIMTlKrUdx4Ol6P76IXzuc9uPW/fL8ldhzI5/ParMi9ogQnIjHoFECOBGkFA7OSwXXF4RC2mVYigOZDp+TtMxB/LJqx205sTTN/G7FxT+LpfeysMTnRtaxma+EUjOwgVVLaeBQKQl/NXm8heNtVWHOV09OnGV9+6uLu8tcsXrMhFTz0Ke/zVBcramkhrFAHjI7vbESKloNzamKRjTyvhRpnjo+ExFGipl3iK4OMZA3UKl5CPqUDMjIi71amCvg/k2TLlhYKBZ4P6e3JLTZOLfJ1i4HhUgWPBaz7ccpwWBGUcGhUt0SSuhtK4r69Ql4m3XWaJm8zgDuD5uqoLcim8p1IzppYGFHkRlkV7W9nJcGdWPeskd0eEu3l3O26deQbdFCh1nVmro2c3Llg23NQMiOE6tnXMoSkLFtPofUlQhjZQ66uQvPIeMQOOKLOLu9tIacxegYFtKMYxAn41vtXG9l5blE3d4876GT8mQfPaVbKypvm1oYA0ERl58xSmkHTsS6ic4M+ac88LtJgXWsZWuKDjJ10QoR5sOSfeAbxzHaWSa8ZnAeO5YGs2nfcqTY+ozRtWTIRy0s5eMFxWWA6FxQkNKE5kaZrmW6q/z32sfX24pq5zJL6anyhrtVrqbOYv1buFW/MTDcISkORhGcxvCUoR0HZhDbUXoG1ld2PXhlfVwKKaS6KRllKjeVXT6q3I/3llslz8ndv6S0abMtyHZ0wtoERDLEJBEbAuEIbV43qsWvQvPuqYtD22oL/k3E5CRan0vMhlNOXDzCHB2TuJgSjkrPXZb9WOWdVmjAkqV9pG9Xi8OYACbiTYGDLTUQOpRLc1tYqOpO1cPIxu6bQ0tpm+LQUTPqz7WySkWlKzz7YBiYRM1i36Rtpso4tsu2zaatt5MpQYlIBVcNdx03yiXKkHELN2d0mjinZDhobbIggeEAMQMbooLSjEOCkhahJZ30nkcTyY0XVDB2hyuadUtHIYXJUJz1rdqinBwspnZtZX7hEpTAF6qGFNsFbB1OfBa9LL5/6w/B0JhA6kLzaCYILpcaWicONb6DoV95UEsq7eKOqK1YTcAMtRGVQW4JyVy6ECOO6K/ORS4aK7I349RusRBnWyNEIoERTS5CPWY3LR+FPS2eOSNd8qBrP81k8chTLiJ6oINGa2cz7BVUbZKDASvECeGUvSj2DA0QtGC55NjIEtUpDsqE9nZLsnMe/ebmEBZEQIw5Ue2wkLPhSfzRSBwx8IE8SUOaCUP91aJCYbeknLYDyYIQ0zvCvSO6LNssLUX/FGGeSZUJlk7NT0Jhq7/HluRAltp5FGCL1u66HV1+vH50FrQTmsGDqZ21NPsoKaN0LLRqAqQgdbqwL9fliZMMommC5B710RonI+hhOl+dqphCYV5hKhhsUtCTjdVWTrs2STqkQKsjOSi6v2VdhL0LU/EdFz7dOcdDg3vncXgLE2AUqpYe4dQk+wvMPJBi+7lvkug7YsHS1yHnRFXumbNbBCladLH3ce0TfHb55GkFvYJA22unjaAGEKCpiTcqeS8UZuhhs1RFdZ2AS4J5DkfwhCHvKjPJJT32c1Gis3WUz/dvTWOqX7EQ3xNKraXvyJ7NL+lzN24+wdIH4yCyVgvmWLXAaJD95FP508pPR6cvQiIAiIiAJ6QgRERAE/8XckU4UJC32UPsA="
| base64 -d | bunzip2 - | head -n 1550 | ./parallel -X -j 4 "echo {} |
wc -w"
Thanks a lot for this fantastic tool and for all helpful input!
Henrik
- GNU Parallel Bug Reports Distribution of args skewed on input run with -X -j 4.,
Henrik Mühe <=