[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
GNU Parallel Bug Reports A suggestion: --shuf and -k
From: |
paralleluser |
Subject: |
GNU Parallel Bug Reports A suggestion: --shuf and -k |
Date: |
Fri, 30 Jun 2017 17:11:12 -0400 |
Friends
A suggestion that merits your comments and review:
--shuf does exactly what the man page says it does, but when you combine --shuf
and -k, the -k does nothing, --shuf rules over -k
I'm going to propose that combining --shuf and -k that this happens:
the jobs are still processed randomly
but the output be in order as the true input
When do you use this? Assume your input to parallel process is:
server1/resource1
server1/resource2
server1/resource3
...etc...
server2/resource1
server2/resource2
...etc...
...up to server50
For human processing reasons, it is easier to keep all the server/resource
input lines in ASCII sort order
For computer processing reasons, server 1 is going to hate you if you are
hitting it with a lot of requests all at the same time
Thus with the "--shuf -k" combo, the sever loads will be spread around, but you
will get your data back in the same order.
Comments welcome........thanks
- GNU Parallel Bug Reports A suggestion: --shuf and -k,
paralleluser <=