Re: Whether the parallel tool has the similar size limit problem as xargs?

[Joe Sapp]

If you're trying to get around the maximum argument limit, maybe this will help:
https://www.gnu.org/software/parallel/man.html#example-inserting-multiple-arguments
Or this: 
https://www.gnu.org/software/parallel/man.html#example-processing-a-big-file-using-more-cpus

Parallel can break up the calls to a command, limiting the number of
arguments to the maximum allowed on the command line.  But then you
won't have one sorted file in the end.  Try the examples, use "-m -j1"
or "-X -j1", and do a final `sort -u` on the output file.

-- 
Joe

On Tue, Nov 30, 2021 at 3:25 AM Hongyi Zhao <hongyi.zhao@gmail.com> wrote:
>
> As far as the following command is concerned:
>
> $ find ./source -type f | xargs sort -o bbb -u
>
> Lew Pitcher told me that it has the following limitation [1]:
>
> The above command depends on xargs(1) to provide sort(1) with a list
> of input files. As the size of this list depends on both the number of
> files found, /and/ the maximum size of an argument vector (argv[]), there
> is a chance that the file bbb will only contain the sorted contents of
> a subset of the files found by find(1).
>
> [1] https://groups.google.com/g/comp.unix.shell/c/ha5t3U54GmY/m/M3tIUPtPBAAJ
>
> So, I want to know whether the parallel tool still has the similar problem.
>
> Regards,
> HZ
>