[task #16251] Using Make's jobserver for parallelization

[Mohammad Akhlaghi]

URL:
  <https://savannah.gnu.org/task/?16251>

                 Summary: Using Make's jobserver for parallelization
                 Project: GNU Astronomy Utilities
               Submitter: makhlaghi
               Submitted: Sun 04 Sep 2022 01:50:05 PM BST
         Should Start On: Sun 04 Sep 2022 12:00:00 AM BST
   Should be Finished on: Sun 04 Sep 2022 12:00:00 AM BST
                Category: All Gnuastro
                Priority: 5 - Normal
              Item Group: Enhancement
                  Status: None
                 Privacy: Public
        Percent Complete: 0%
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
                  Effort: 0.00


    _______________________________________________________

Follow-up Comments:


-------------------------------------------------------
Date: Sun 04 Sep 2022 01:50:05 PM BST By: Mohammad Akhlaghi <makhlaghi>
One of the most common high-level frameworks where Gnuastro is run is GNU Make
since it manages dependencies and can run in Parallel. 

When running in Parallel, by default, each program counts the total number of
CPU threads and uses all of them. So if a CPU has 8 cores, and a user runs
Make with '-j8' (to use all those 8 threads), and each rule has one
NoiseChisel call within it (which will use all 8 cores by default), the system
will have a queue of 8x8=64 jobs! It is possible to manage this by setting all
Gnuastro programs to use one thread in a 'gnuastro.conf' configuration file,
but that is not usually the most effective way either (there might be times
that more than one thread is available).

For such situations, Make has the "jobserver" concept, which is described in
the Sharing Job Slots with GNU make
<https://www.gnu.org/software/make/manual/html_node/Job-Slots.html>. Since all
Gnuastro programs use a single function to spin-off threads, it should be
relatively easy to implement this in Gnuastro; so it integrates with GNU Make
and doesn't add too many jobs to the queue.







    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/task/?16251>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/