Re: [Patch] Xargs: vary parallelism with SIGUSR1/2

[John Gilmore]

> I would like to apply your patch.  However, since this is a
> significant change, I need to ask you to assign your copyright
> interest in the change to the Free Software Foundation.  I'm obliged
> to do this before applying the patch.  Would you be willing to do
> this?  

It's been almost nine months, but I have (finally!) completed the
paperwork with the FSF to contribute my changes to the findutils to
the FSF.

I've also ported the patch forward into version 4.3.0 for your
convenience.

I also fixed a thinko in the "Unusual Characters in File Names" section
of the doc, where it said backslash when it meant slash.

> If so, I'll put a copy of your patch at
> http://savannah.gnu.org/patch/?group=findutils to make sure I don't
> lose track of it while we wait for the copyright assignment process to
> complete.
> 
> Two small nits with the patch:
> 
> 1. It's probably better to reference signal(7) (which most Unix
>    systems have) than signal(2) (which I suspect many don't).

Fixed.

> 2. There's no need to say "+ blah patch" in the version information - 
>    if your patch is applied to findutils, it becomes part of the mainstream
>    findutils release, and users can see the feature exists by reading the 
>    documentation or the NEWS file.  I realise that putting that
>    information into the version information would be helpful to you
>    locally, of course.

The GPL license (paragraph 2a) requires me to mark my changed versions
as different from FSF's.  But of course, when you apply the patch to
the official version, you would skip that part.  I've removed that
change from the diff below.

        John

diff -r -u findutils-4.3.0/NEWS findutils-4.3.0+sigs/NEWS
--- findutils-4.3.0/NEWS        Mon Dec 12 00:15:25 2005
+++ findutils-4.3.0+sigs/NEWS   Mon Mar 13 22:14:18 2006
@@ -29,6 +29,9 @@
 New tests, -readable, -writable, -executable.  These check that a file
 can be read, written or executed respectively.
 
+You can now increase the parallelism of xargs in mid-run by sending
+it SIGUSR1, and decrease the parallelism with SIGUSR2.
+
 * Major changes in release 4.2.27
 
 ** Warnings of Future Changes
diff -r -u findutils-4.3.0/doc/find.texi findutils-4.3.0+sigs/doc/find.texi
--- findutils-4.3.0/doc/find.texi       Sun Dec 11 23:08:08 2005
+++ findutils-4.3.0+sigs/doc/find.texi  Mon Mar 13 22:25:16 2006
@@ -2029,6 +2029,7 @@
 * Safe File Name Handling::
 * Unusual Characters in File Names::
 * Limiting Command Size::
+* Controlling Parallelism::
 * Interspersing File Names::
 @end menu
 
@@ -2244,16 +2245,111 @@
 or lower limit is used instead.  You can use @samp{--show-limits}
 option to understand the command-line limits applying to @code{xargs}
 and how this is affected by any other options.
address@hidden table
+
address@hidden Controlling Parallelism
address@hidden Controlling Parallelism
+
+Normally, @code{xargs} runs one command at a time.  This is called
+"serial" execution; the commands happen in a series, one after another.
+If you'd like @code{xargs} to do things in "parallel", you can ask it
+to do so, either when you invoke it, or later while it is running.
+Running several commands at one time can make the entire operation
+go more quickly, if the commands are independent, and if your system
+has enough resources to handle the load.  When parallelism works in
+your application, @code{xargs} provides an easy way to get your work
+done faster.
 
address@hidden @code
 @item address@hidden
 @itemx -P @var{max-procs}
 Run up to @var{max-procs} processes at a time; the default is 1.  If
 @var{max-procs} is 0, @code{xargs} will run as many processes as
-possible at a time.  Use the @samp{-n}, @samp{-s}, or @samp{-L} option
+possible at a time.  This parameter can be incremented in mid-run
+by sending @code{SIGUSR1}, or decremented with @code{SIGUSR2}.
+(It will not decrement to zero with a signal.)
+Use the @samp{-n}, @samp{-s}, or @samp{-L} option
 with @samp{-P}; otherwise chances are that the command will be run
 only once.
 @end table
 
+For example, suppose you have a directory tree of large image files
+and a @code{makeallsizes} script that takes a single file name and
+creates various sized images from it (thumbnail-sized, web-page-sized,
+printer-sized, and the original large file).  The script is doing enough
+work that it takes significant time to run, even on a single image.
+You could run:
+
address@hidden
+find originals -name '*.jpg' | xargs -n 1 makeallsizes
address@hidden example
+
+This will run @code{makeallsizes @var{filename}} once for each @code{.jpg}
+file in the @code{originals} directory.  However, if your system has
+two central processors, this script will only keep one of them busy.
+Instead, you could probably finish in about half the time by running:
+
address@hidden
+find originals -name '*.jpg' | xargs -n 1 -P 2 makeallsizes
address@hidden example
+
address@hidden will run the first two commands in parallel, and then
+whenever one of them terminates, it will start another one, until
+the entire job is done.
+
+The same idea can be generalized to as many processors as you have handy.
+It also generalizes to other resources besides processors.  For example,
+if xargs is running commands that are waiting for a response from a
+distant network connection, running a few in parallel may reduce the
+overall latency by overlapping their waiting time.
+
address@hidden also allows you to "turn up" or "turn down" its parallelism
+in the middle of a run.  Suppose you are keeping your four-processor
+system busy for hours, processing thousands of images using @code{-P 4}.
+Now, in the middle of the run, you or someone else wants you to reduce
+your load on the system, so that something else will run faster.
+If you interrupt @code{xargs}, your job will be half-done, and it
+may take significant manual work to resume it only for the remaining
+images.  If you suspend @code{xargs} using your shell's job controls
+(e.g. @code{control-Z}), then it will get no work done while suspended.
+
+Find out the process ID of the @code{xargs} process, either from your
+shell or with the @code{ps} command.  After you send it the signal
address@hidden, @code{xargs} will run one fewer command in parallel.
+If you send it the signal @code{SIGUSR1}, it will run one more command
+in parallel.  For example:
+
address@hidden
+shell$ xargs <allimages -n 1 -P 4 makeallsizes &
+[4] 27643
+   ... at some later point ...
+shell$ kill -USR2 27643
+shell$ kill -USR2 %4
address@hidden example
+
+The first @code{kill} command will cause @code{xargs} to wait for
+two commands to terminate before starting the next command (reducing
+the parallelism from 4 to 3).  The second @code{kill} will reduce it from
+3 to 2.  (@code{%4} works in some shells as a shorthand for the process
+ID of the background job labeled @code{[4]}.)
+
+Similarly, if you started a long xargs job without parallelism, you
+can easily switch it to start running two commands in parallel by sending
+it a @code{SIGUSR1}.
+
address@hidden will never terminate any existing commands when you ask it
+to run fewer processes.  It merely waits for the excess commands to
+finish.  If you ask it to run more commands, it will start the next
+one immediately (if it has more work to do).
+
+If you send several identical signals quickly, the operating system
+does not guarantee that each of them will be delivered to @code{xargs}.
+This means that you can't rapidly increase or decrease the parallelism by
+more than one command at a time.  You can avoid this problem by sending
+a signal, observing the result, then sending the next one; or merely by
+delaying for a few seconds between signals (unless your system is very
+heavily loaded).
+
 @node Interspersing File Names
 @subsubsection Interspersing File Names
 
@@ -3167,7 +3263,9 @@
 @itemx -P @var{max-procs}
 Run simultaneously up to @var{max-procs} processes at once; the default is 1.  
If
 @var{max-procs} is 0, @code{xargs} will run as many processes as
-possible simultaneously.
+possible simultaneously.  This parameter can be incremented in mid-run
+by sending @code{SIGUSR1}, or decremented with @code{SIGUSR2}.
+(It will not decrement to zero with a signal.)
 @end table
 
 
@@ -3606,8 +3704,8 @@
 @subsection Unusual characters in filenames
 
 Unix-like systems allow any characters to appear in file names with
-the exception of the ASCII NUL character and the backslash.
-Backslashes can occur in path names (as the directory separator) but
+the exception of the ASCII NUL character and the slash.
+Slashes can occur in path names (as the directory separator) but
 not in the names of actual directory entries.  This means that the
 list of files that @code{xargs} reads could in fact contain white space
 characters --- spaces, tabs and newline characters.  Since by default,
@@ -4587,6 +4685,13 @@
 
 @item <program>: terminated by signal 99
 See the description of the similar message for @code{find}.
+
address@hidden cannot set SIGUSR1 signal handler
address@hidden is having trouble preparing for you to be able to send it
+signals to increase or decrease the parallelism of its processing.
+If you don't plan to send it those signals, this warning can be ignored
+(though if you're a programmer, you may want to help us figure out
+why @code{xargs} is confused by your operating system).
 @end table
 
 @node Error Messages From locate, Error Messages From updatedb, Error Messages 
From xargs, Error Messages
diff -r -u findutils-4.3.0/xargs/xargs.1 findutils-4.3.0+sigs/xargs/xargs.1
--- findutils-4.3.0/xargs/xargs.1       Mon Dec  5 23:52:33 2005
+++ findutils-4.3.0+sigs/xargs/xargs.1  Mon Mar 13 22:31:09 2006
@@ -159,6 +159,16 @@
 \fImax-procs\fR is 0, \fBxargs\fR will run as many processes as
 possible at a time.  Use the \fI\-n\fR option with \fI\-P\fR;
 otherwise chances are that only one exec will be done.
+While
+.B xargs
+is running, you can
+send its process 
+a SIGUSR1 signal to increase the number of commands to run simultaneously,
+or a SIGUSR2 to decrease the number.  You cannot decrease it below 1.
+.B xargs
+never terminates its commands; when asked to decrease, it merely
+waits for more than one existing command to terminate before starting
+another.
 .SH "EXAMPLES"
 .nf
 .B find /tmp \-name core \-type f \-print | xargs /bin/rm \-f
@@ -213,6 +223,7 @@
 
 .SH "SEE ALSO"
 \fBfind\fP(1), \fBlocate\fP(1), \fBlocatedb\fP(5), \fBupdatedb\fP(1),
+\fBkill\fP(1), \fBsignal\fP(7),
 \fBFinding Files\fP (on-line in Info, or printed)
 .SH "BUGS"
 The \-L option is incompatible with the \-I option, but perhaps should not be.
diff -r -u findutils-4.3.0/xargs/xargs.c findutils-4.3.0+sigs/xargs/xargs.c
--- findutils-4.3.0/xargs/xargs.c       Sun Dec 11 09:54:12 2005
+++ findutils-4.3.0+sigs/xargs/xargs.c  Mon Mar 13 22:34:34 2006
@@ -207,7 +207,7 @@
 
 /* If nonzero, the maximum number of child processes that can be running
    at once.  */
-static int proc_max = 1;
+static volatile sig_atomic_t proc_max = 1;
 
 /* Total number of child processes that have been executed.  */
 static int procs_executed = 0;
@@ -221,6 +221,9 @@
 /* The number of allocated elements in `pids'. */
 static int pids_alloc = 0;
 
+/* If nonzero, we've been signaled that we can start more child processes. */
+static volatile sig_atomic_t stop_waiting = 0;
+
 /* Exit status; nonzero if any child process exited with a
    status of 1-125.  */
 volatile static int child_error = 0;
@@ -270,6 +273,8 @@
 static void add_proc PARAMS ((pid_t pid));
 static void wait_for_proc PARAMS ((boolean all));
 static void wait_for_proc_all PARAMS ((void));
+static void increment_proc_max PARAMS ((int));
+static void decrement_proc_max PARAMS ((int));
 static long parse_num PARAMS ((char *str, int option, long min, long max, int 
fatal));
 static long env_size PARAMS ((char **envp));
 static void usage PARAMS ((FILE * stream));
@@ -428,6 +433,7 @@
   long size_of_environment = env_size(environ);
   char *default_cmd = "/bin/echo";
   int (*read_args) PARAMS ((void)) = read_line;
+  struct sigaction sigact;
   int env_too_big = 0;
 
   program_name = argv[0];
@@ -607,6 +613,25 @@
     error (1, 0, _("environment is too large for exec"));
   }
 
+#ifdef SIGUSR1
+#ifdef SIGUSR2
+  /* Accept signals to increase or decrease the number of running
+     child processes.  Do this as early as possible after setting
+     proc_max.  */
+  sigact.sa_handler = increment_proc_max;
+  sigemptyset(&sigact.sa_mask);
+  sigact.sa_flags = 0;
+  if (0 != sigaction (SIGUSR1, &sigact, (struct sigaction *)NULL))
+         error (0, errno, _("Cannot set SIGUSR1 signal handler"));
+
+  sigact.sa_handler = decrement_proc_max;
+  sigemptyset(&sigact.sa_mask);
+  sigact.sa_flags = 0;
+  if (0 != sigaction (SIGUSR2, &sigact, (struct sigaction *)NULL))
+         error (0, errno, _("Cannot set SIGUSR2 signal handler"));
+#endif /* SIGUSR2 */
+#endif /* SIGUSR1 */
+
   if (0 == strcmp (input_file, "-"))
     {
       input_stream = stdin;
@@ -1026,7 +1051,7 @@
   
   if (!query_before_executing || print_args (true))
     {
-      if (proc_max && procs_executing >= proc_max)
+      while (proc_max && procs_executing >= proc_max)
        wait_for_proc (false);
       if (!query_before_executing && print_command)
        print_args (false);
@@ -1096,7 +1121,8 @@
 }
 
 /* If ALL is true, wait for all child processes to finish;
-   otherwise, wait for one child process to finish.
+   otherwise, wait for one child process to finish, or for another signal
+   that tells us that we can run more child processes.
    Remove the processes that finish from the list of executing processes.  */
 
 static void
@@ -1110,9 +1136,16 @@
        {
          pid_t pid;
 
+         stop_waiting = 0;
          while ((pid = wait (&status)) == (pid_t) -1)
-           if (errno != EINTR)
-             error (1, errno, _("error waiting for child process"));
+           {
+             if (errno != EINTR)
+               error (1, errno, _("error waiting for child process"));
+             /* If a signal handler has given us more processes, no
+                need to keep waiting (unless waiting for all). */
+             if (stop_waiting && !all)
+               return;
+           }
 
          /* Find the entry in `pids' for the child process
             that exited.  */
@@ -1170,6 +1203,30 @@
   
 }
 
+
+/* Increment or decrement the number of processes we can start simultaneously,
+   when we receive a signal from the outside world.
+   
+   We must take special care around proc_max == 0 (unlimited children),
+   proc_max == 1 (don't decrement to zero).  */
+static void
+increment_proc_max (int ignore)
+{
+  /* If user increments from 0 to 1, we'll take it and serialize. */
+  proc_max++;
+  /* If we're waiting for a process to die before doing something,
+     no need to wait any more. */
+  stop_waiting = 1;
+}
+
+static void
+decrement_proc_max (int ignore)
+{
+  if (proc_max > 1)
+         proc_max--;
+}
+
+
 /* Return the value of the number represented in STR.
    OPTION is the command line option to which STR is the argument.
    If the value does not fall within the boundaries MIN and MAX,