[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
GNU Parallel and max jobs
From: |
paralleluser |
Subject: |
GNU Parallel and max jobs |
Date: |
Sun, 07 Oct 2018 19:24:11 -0400 |
Friends
I decided to re-read the GNU parallel man page in full and found two
interesting paragraphs. Let us assume a single user machine where the user has
root access. It is the goal of the user to not limit himself in terms of the
number of jobs parallel can spawn (ie: full power to parallel).
-------------------------------------------------------------------------------------------------------------
Paragraph 1 from man page:
If -j0 normally spawns 252 jobs, then the above will try to spawn 2520
jobs. On a normal GNU/Linux system you can spawn 32000 jobs using this
technique with no problems. To raise the 32000 jobs limit raise
/proc/sys/kernel/pid_max to 4194303.
Analysis: OK I can make that change.
-------------------------------------------------------------------------------------------------------------
Paragraph 2 from man page:
EXAMPLE: Running more than 250 jobs workaround
If you need to run a massive amount of jobs in parallel, then you will
likely hit the filehandle limit which is often around 250 jobs. If you
are super user you can raise the limit in /etc/security/limits.conf
Analysis: How is this different from the change above? What number to I set
it to? What is the syntax for that file (it is not as clear as pid_max)
-------------------------------------------------------------------------------------------------------------
Overall Analysis: Do I care about anything with "ulimit" if I make the changes
above? Is there anything else to do to give parallel freedom to go to the max?
Thanks!
- GNU Parallel and max jobs,
paralleluser <=