Re: [lmi] Continuing deboostification with removing dependency on Boost.Regex

[Greg Chicares]

On 5/28/21 11:36 PM, Vadim Zeitlin wrote:
> On Fri, 28 May 2021 20:37:39 +0000 Greg Chicares <gchicares@sbcglobal.net> 
> wrote:

[...simple shell commands suffice for the summary statistics
printed by 'test_coding_rules', so factor that part out and...]

> GC> use the builtin parallelism of 'gnu make -jN' to run a new
> GC> 'test_coding_rules_without_summary_statistics' against an arbitrary
> GC> list of files independently, without threading.
> 
>  But how would you share the list of files among all copies of this
> program? I don't know how to do it from make and I don't think it can be
> done optimally from outside the program (inside the program we'd have a
> simple work queue that would keep as many threads as we have as fully
> occupied as possible).

Right now, we have this, in 'GNUmakefile':

  @-cd $(prefascicle_dir) && $(PERFORM) $(TEST_CODING_RULES) *

so the program receives the entire file list [which seems to be
15000 bytes, so invoking that makefile target from a remote
directory seems like it might exceed some system limit, BTW].

Let's approach this in two steps. First, couldn't we just do
something like:

  find $(prefascicle_dir) -print0 | xargs -0 $(PERFORM) $(TEST_CODING_RULES)

instead, to achieve the same (non-parallelized) outcome as today?
Second, couldn't we parallelize that with a patch like [untested]...

+concinnity_check_files := $(addsuffix --concinnity-check,$(prefascicle_dir)/.)

+.PHONY: %-concinnity-check
+%-concinnity-check:
+       @-$(PERFORM) $(TEST_CODING_RULES) $*

 .PHONY: check_concinnity
-check_concinnity: source_clean custom_tools
+check_concinnity: source_clean custom_tools $(concinnity_check_files)

...thereby using 'make' to take care of parallelism, without
making 'test_coding_rules$(EXEEXT)' multithreaded?

Then the list of all files is known to 'make', and to the commands
it uses to display summary statistics--whereas each running
instance of 'test_coding_rules$(EXEEXT)' knows only the single
file it's testing, but isn't that perfectly okay?

>  Please correct me if I'm wrong, but you still seem to be against actually
> changing test_coding_rules to use threads, and I don't know how to do it
> otherwise, so this still needs to be clarified, ideally.

I've never used threads myself, so it's always been my habit to
avoid threading. Old habit isn't a good reason not to make this
multithreaded. A complete lack of experience makes it seem to
require a lot of effort. OTOH, having one threaded example in
the codebase makes it much easier to add more, so using threads
here could be a good thing, educationally.

But would it be technically superior to using 'make' as above?
Maybe: we'd be running a single program rather than hundreds of
instances of the same program, and there might be a measurable
speed difference. Or maybe not--in the makefile sketch above, if
we choose to omit this line:
  .PHONY: %-concinnity-check
then, as an automatic byproduct, we get a sentinel file for each
source file whose concinnity has been tested since it was last
changed [which we'd probably want to put in some subdirectory];
and then when we modify a few files, only their concinnity needs
to be rechecked.

>  But any parallelization will only come later, i.e. my order of sub-tasks
> here is:
> 
> 1. Rebase std::regex patch on master and benchmark it.
> 2. Try using CTRE.
> 3. Parallelize test_coding_rules.
> 
>  I think this makes sense because if you're interested in using CTRE
> anyhow, there is no need to try (3) before (2) to see if it's going to be
> fast enough -- we don't know by how much exactly, but we can be quite sure
> that CTRE will be still faster anyhow, and it might even be fast enough to
> not require (3) in practice. But please correct me if I'm wrong.

I certainly agree with (1) and (2). As for (3), maybe I should
test the sketch above and share a sentinel-based implementation
that we could experiment with, measure, and discuss.