[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Feature Request: Special input value to create a barrier
From: |
Achim Gratz |
Subject: |
Re: Feature Request: Special input value to create a barrier |
Date: |
Sun, 23 Jun 2019 18:21:50 +0200 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) |
Ole Tange writes:
> I can see how this would make sense from a user perspective, so I am
> not opposed to the idea.
OK.
> You _could_ just do:
>
> parallel --barrier + do_something '{}' ::: job1 job2 job3
> parallel --barrier + do_something '{}' ::: job4 job5
> parallel --barrier + do_something '{}' ::: job6 job7
>
> but I can see how the syntactic sugar could make some tasks easier.
I know that I can already do what I asked by running multiple
invocations of parallel (or sem, as I already do for other stuff).
However, that'd require to split things someplace else in the particular
piepeline I am using. So far I've had everything just serialized in
dependency order, but since I have a much more powerful machine to run
things by now it'd be useful to extract the available parallelism
inherent in the dependency ordering.
> What should this do:
>
> parallel --barrier + dostuff ::: 1 2 3 + 4 5 + 6 7 ::: a + b c + d e
> f + g + h :::+ A B C D E + F G H
Well, that doesn't really make sense unless the resulting sequence has
the barrier in all arguments simultaneously. Combinatorial
parallelization is not likely to have a dependency graph leading to a
rank ordering anyway.
> We already have -E to make GNU Parallel stop reading when hitting a
> special value. This gets us partly there.
Interesting idea. That might actually be workable if I can run the next
invocation(s) of parallel by telling to skip the first (and following)
"virtual" EOF until it finally hits the real one. At least in the
application I have in mind, reading the input multiple times is no
problem of any kind.
> I think, however, it will be quite tricky make GNU Parallel resume
> after it finished all running jobs. Nothing in the design is made for
> resuming. I am not saying it cannot be done, but I will be surprised
> if it is an easy change.
As I outlined above, that might not be quite as tricky as you seem to
think as long as we can have two kinds of EOF.
> So patch welcome (but I have a feeling it will be a rather big change).
I'll put it on my todo list. No promises of when I'll have time to look
at it.
Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra