Re: problem with -S with -a?

[Ole Tange]

On Fri, Aug 20, 2010 at 11:28 PM, Max Birkoff <address@hidden> wrote:
> Hello,
>
> First and foremost, thank you so much for gnu parallel! it's allowed me to 
> get a lot done. Thank you!

I would love to hear, what you have done with GNU Parallel to get
inspiration for new examples.

> I've been trying to use parallel in this way (on host1.here.com):
>  parallel -a /home/max/commands -j 4 -S 4/host2.here.com
>
> and I expect the set of commands to execute on both host1.here.com and 
> host2.here.com.

In your command you have only told GNU Parallel to run on a single
host: host2. So it is surprising to me if it also runs on your local
machine. You will need to add the special host ':' to do that.

The 4/ means the number of cores - not the number of jobs to run. So
in this case it will not have any effect as you do not use neither +,
-, nor % in your -j.

Try running:

parallel --progress -a /home/max/commands -j 4 -S host2.here.com,:

It shows you how many jobs are running on each machine.

> While they do, I observe two things that I did not expect:
>  (1) -j does not appear to be respected on host2 (hence the 4/)

It should. I cannot reproduce this. Can you test --progress and see if
that is showing the expected?

If the jobs to run are very fast (like 'echo a') then the ssh to the
machine will take up most of the time and thus it may look as if there
are not 4 jobs running on the machine. Can you try this:

seq 1 100 | parallel -j4 -S host2,: "sleep 5; echo {}"

On both hosts you should see 4 sleeps running most of the time.

A workaround if the jobs are fast is to do something like:

-S 1/:,10/host2 -j400%

That should run 40 jobs on host2 but only 4 on local. You may also try
using -M - though that is experimental as it seems to fail now and
then.

>  (2) the commands from the commands file are executed on both host1 and host2.

That should never happen and is clearly a bug if it does. I tried with
your example and cannot reproduce that a job is run twice.

> To workaround this issue, my plan is to split the commands file and invoke 
> parallel on both hosts, but I wondered if anyone else has seen this behavior.

That should not be needed.


/Ole