Re: Bug in GNU Parallel 20200622

[Ole Tange]

On Wed, Aug 12, 2020 at 2:51 AM Ole Tange <ole@tange.dk> wrote:
> On Fri, Aug 7, 2020 at 10:22 PM Mark Setchell <marksetchell@icloud.com> wrote:
>
> > I run the following command on macOS Catalina 10.15
> >
> > seq -w 270000 | parallel -X touch
:
> Using '-s 27000' will work in all the cases I have found so far. But
> to me that is a highly unsatisfactory workaround, and it is likely
> that it will break in some situations. I would much rather understand
> why this happens, and how to compute the maximal size.
>
> If you find some logic in that, please submit a patch.

I think I have found some logic.

The maximal command length with a single arg = 5 * maximal command
length with 1 chars args
The limit is lowered by the number of environment names (i.e.
variables and functions) and the size of the environment names and
bodies.

So:

actual_max = max_single / 5 - count(env) - length(env)

(minus a small constant around 1k that I cannot explain).

On the MacOS X I had access to yesterday this seems to work.

But I would love to see this confirmed on other MacOS systems.

If it works reliably with different size environments and different
size commands, we need to discuss where it should be put to adhere to
POLS:

Should it be part of populate()? This way the user will never see this
limit and will be surprised why his command is not longer.

Should it instead be part of real_max_length? Then the user will be
surprised that `parallel --max-line-length` is not what he gets using
getconf.

I have put it into parallel real_max_length() but am open to moving it.

Please test the git version:

* with different environments (number of exported variables/function,
size of same)
* with different arguments (tiny one-char arguments, huge 10-kb arguments)

If you break it, see if you can fix it by changing the numbers in
real_max_length().


/Ole