Re: Unexpected behavior of -m/-X relating to -j/derived cores

[Ole Tange]

On Wed, May 20, 2020 at 5:33 PM Josef wells
<Josefwells@alumni.utexas.net> wrote:

> This will fail, since 'dd' doesn't allow bare arguments and the -j1
> forces parallel to try to run it all at once.
> parallel --tag -j1  -m 'dd if=/dev/zero of=/dev/null count={}' ::: 1 2
>
> This doesn't fail.
> parallel --tag -j2  -m 'dd if=/dev/zero of=/dev/null count={}' ::: 1 2
:
> I would have liked for the -m/-X to fail regardless of the number of
> cores, since it was really inappropriate to run the tool with options
> in that way. To me the -m/-X is more important than the -j but I can
> see it both ways.

The reason for the current design is this situation: You have 1000
file names and 4 cores. Your_program takes multiple inputs. 1000 file
names fit on a single command line. So GNU Parallel _could_ run:

  Your_program file1 ... file1000

But to use all cores GNU Parallel spreads the inputs over all cores:

  Your_program file1 ... file250
  Your_program file251 ... file500
  Your_program file501 ... file750
  Your_program file751 ... file1000

and runs those in parallel.

You have seen the extreme situation where you have more core than arguments.

In this case the arguments will be spread with 1 argument for each job.

I am not sure if it is possible to make a design, that would both suit
you and the mentioned situation.

> Also, Thanks for parallel, it is fantastic and I love it.

You can pay back by spreading the word and help others use the tool.


/Ole