Re: GNU Make 4.2 Query

[David Boyce]

I did not suggest using lsmake, I simply mentioned that we use it.

On Mon, Sep 2, 2019 at 11:04 AM nikhil jain <address@hidden> wrote:

> Thanks for detailed information.
>
> I will see if I can use shell wrapper program as mentioned by you.
>
> I had used LSF a lot like for 5 years. I still use it.  bsub, bjobs.
> bkill, lim, sbatchd, mbatchd etc. it is easy to understand and use
>
> lsmake - I do not want to use IBM's proprietary stuff.
>
> Thanks for your suggestions.
>
> Nikhil
>
> On Mon, Sep 2, 2019 at 11:10 PM David Boyce <address@hidden>
> wrote:
>
>> I'm not going to address the remote execution topic since it sounds like
>> you already have the solution and are not looking for help. However, I do
>> have fairly extensive experience with the NFS/retry area so will try to
>> contribute there.
>>
>> First, I don't think what Paul says:
>>
>> > As for your NFS issue, another option would be to enable the .ONESHELL
>> > feature available in newer versions of GNU make: that will ensure that
>> > all lines in a recipe are invoked in a single shell, which means that
>> > they should all be invoked on the same remote host.
>>
>> Is sufficient. Consider the typical case of compiling foo.c to foo.o and
>> linking it into foo.exe. Typically, and correctly, those actions would be
>> in two separate recipes which in a distributed-build scenario could run on
>> different hosts so the linker may not find the .o file from a previous
>> recipe. Here .ONSHELL cannot help since they're different recipes.
>>
>> In my day job we use a product from IBM called LSF (Load Sharing
>> F-something,
>> https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.3/lsf_welcome.html)
>> which exists to distribute jobs over a server farm (typically using NFS)
>> according to various metrics like load and free memory and so on. Part of
>> the LSF package is a program called lsmake (
>> https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.3/lsf_command_ref/lsmake.1.html)
>> which under the covers is a version of GNU make with enhancement to enable
>> remote/distributed recipes and also adds retry-with-delay feature Nikhil
>> requested). Since GNU make is GPL, IBM is required to make its package of
>> enhancements available under GPL as well. Much of it is not of direct
>> interest to the open source community because it's all about communicating
>> with IBM's proprietary daemons but their retry logic could probably be
>> taken directly from the patch. At the very least, if retries were to be
>> added to GNU make per se it would be nice if the flags were compatible with
>> lsmake.
>>
>> However, my personal belief is that retries are God's way of telling us
>> to think harder and better. Retrying (and worse, delay-and-retry) is a form
>> of defeatism which I call "sleep and hope". Computers are deterministic,
>> there's always a root cause which can usually be found and addressed with
>> sufficient analysis, etc. Granted there are cases where you understand the
>> problem but can't address it for administrative/permissions/business
>> reasons but that can't be known until the problem is understood.
>>
>> NFS caching is the root cause of unreliable distributed builds, as you've
>> already described, but most or all of these issues can be addressed with a
>> less blunt instrument than sleep-and-retry. Even LSF engineers threw up
>> their hands and did retries but what we did here was take their patch,
>> which at last check was still targeted to 3.81, and while porting it to 4.1
>> added some of the cache-flushing strategies detailed below. This has solved
>> most if not all of our NFS sync problems. Caveat: most of our people still
>> use the LSF retry logic in addition, because they're not as absolutist as I
>> am and just want to get their jobs done (go figure), which makes it harder
>> to determine what percentage of problems are solved by cache flushing vs
>> retries but I'm pretty sure flushing has resolved the great majority of
>> problems.
>>
>> One problem with sleep-and-hope is that there's no amount of time
>> guaranteed to be enough so you're just driving the incidence rate down, not
>> fixing it.
>>
>> Since we were already working with a hacked version of GNU make we found
>> it most convenient to implement flushing directly in the make program but
>> it can also be done within recipes. In fact we have 3 different
>> implementations of the same NFS cache flushing logic:
>>
>> 1. Directly within our enhanced version of lsmake.
>> 2. In a standalone binary called "nfsflush".
>> 3. In a Python script called nfsflush.py.
>>
>> The Python script is a lab for trying out new strategies but it's too
>> slow for production use. The binary is a faster version of the same
>> techniques for direct use in recipes, and that same C code is linked
>> directly into lsmake as well. Here's the usage message of our Python script:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *$ nfsflush.py --helpusage: nfsflush.py [-h] [-f] [-l] [-r] [-t] [-u]
>> [-V] path [path ...]positional arguments:  path             directory paths
>> to flushoptional arguments:  -h, --help       show this help message and
>> exit  -f, --fsync      Use fsync() on named files  -l, --lock       Lock
>> and unlock the named files  -r, --recursive  flush recursively  -t, --touch
>>      additional flush action - touch and remove a temp file  -u, --upflush
>>    flush parent directories back to the root  -V, --verbose    increment
>> verbosity levelFlush the NFS filehandle caches of NFS directories.Newly
>> created files are sometimes unavailable via NFS for a periodof time due to
>> filehandle caching, leading to apparent race problems.See
>> http://tss.iki.fi/nfs-coding-howto.html#fhcache
>> <http://tss.iki.fi/nfs-coding-howto.html#fhcache> for details.This script
>> forces a flush using techniques mentioned in the URL. Itcan optionally do
>> so recursively.This always does an opendir/closedir sequence on each
>> directoryvisited, as described in the URL, because that's cheap and safe
>> andoften sufficient. Other strategies, such as creating and removing atemp
>> file, are optional.EXAMPLES:    nfsflush.py /nfs/path/...*
>>
>> The most important thing is to read the URL given above and/or to google
>> for similar resource of which there are many. While I'm not an NFS guru
>> myself, the summary of my understanding is that NFS caches all sorts of
>> things (metadata like atime/mtime, directory updates, etc) with various
>> degrees of aggression according to NFS vendor and internal configuration.
>> We've seen substantial variation between NAS providers such as NetApp, EMC,
>> etc, so much depends on whose NFS server you're using. However, the NFS
>> spec _requires_ that caches be flushed on a write operation so all
>> implementations will do this.
>>
>> Bottom line, the most common failure case is as mentioned above: foo.o is
>> compiled on host A and immediately linked on host B. The close() system
>> call following the final write() of foo.o on host A will cause its data to
>> be flushed. Similarly I *believe* the directory write (assuming foo.o is
>> newly created and not just updated) will cause the filehandle cache to be
>> flushed. Thus, after these two write ops (directory and file) the NFS
>> server will know about the new foo.o as soon as it's created.
>>
>> The problem typically arises on host B because no write operation has
>> taken place there after foo.o was created on A so no one has told it to
>> update its caches and as a result it doesn't know foo.o exists and the link
>> fails with ENOENT. All the flushing techniques in the script above are
>> attempts to address this. One takeaway from all this is that even if you do
>> retries, a "dumb" retry is immeasurably enhanced by adding a flush. In
>> other words the most efficient retry formula in a distributed build
>> scenario would be:
>>
>> <recipe> || flush || <recipe>
>>
>> This never flushes a cache unless the first attempt fails. It presumes
>> that NFS implementors and admins know what they're doing and thus caching
>> helps with performance so it's not done unless needed. This is what we
>> built into our variant of lsmake. However, the same can also be done in the
>> shell.
>>
>> Details about implemented cache flushing techniques: the filehandle cache
>> is the biggest source of problems in distributed builds and the simplest
>> solution for it seems to be opening and reading the directory entry. Thus
>> our script and its parallel C implementation always do that. We've also
>> seen cases where forcing a directory write operation is required which the
>> -t, --touch option does. Sometimes you can't easily enumerate all
>> directories involved (vpath etc) so the recurse-downward (-r) and recurse
>> upward (-u) flags may be helpful though they (especially -u) may also be
>> overkill. The -f and -l options were added based on advice found on the net
>> but have not been shown to be helpful in our environment.
>>
>> Some techniques may be of limited utility because they require write
>> and/or ownership privileges. For instance I've seen statements that
>> umounts, even failed umounts, will force flushes. Thus a command like "cd
>> <dir> && umount $(pwd)" would have to fail since the moount is busy but
>> would flush as a side effect. However I believe this requires root
>> privileges so is not helpful in the normal case.
>>
>> In summary: although I don't believe in retries, if they're going to be
>> used I think they should be implemented in a shell wrapper program which
>> could be passed to make as SHELL=<wrapper> and the wrapper should use
>> flushing in addition to, or instead of, retries. We didn't do it that way
>> but I think our nfsflush program could just as well have been implemented
>> as say "nfsshell" such that "nfsshell [other-options] -c <recipe>" would
>> run the recipe along with added flushing and retrying options. I agree with
>> Paul that I see no reason to implement any of these features, retry and/or
>> flush, directly in make.
>>
>> David
>>
>> On Mon, Sep 2, 2019 at 6:05 AM Paul Smith <address@hidden> wrote:
>>
>>> On Sun, 2019-09-01 at 23:23 -0700, Kaz Kylheku (gmake) wrote:
>>> > If your R&D team would allow you to add just one line to the
>>> > legacy GNU Makefile to assign the SHELL variable, you can assign that
>>> > to a shell wrapper program which performs command re-trying.
>>>
>>> You don't have to add any lines to the makefile.  You can reset SHELL
>>> on the command line, just like any other make variable:
>>>
>>>     make SHELL=/my/special/sh
>>>
>>> You can even override it only for specific targets using the --eval
>>> command line option:
>>>
>>>     make --eval 'somerule: SHELL := /my/special/sh'
>>>
>>> Or, you can add '-f mymakefile.mk -f Makefile' options to the command
>>> line to force reading of a personal makefile before the standard
>>> makefile.
>>>
>>> Clearly you can modify the command line, otherwise adding new options
>>> to control a putative retry on error option would not be possible.
>>>
>>> As for your NFS issue, another option would be to enable the .ONESHELL
>>> feature available in newer versions of GNU make: that will ensure that
>>> all lines in a recipe are invoked in a single shell, which means that
>>> they should all be invoked on the same remote host.  This can also be
>>> done from the command line, as above.  If your recipes are written well
>>> it should Just Work.  If they aren't, and you can't fix them, then
>>> obviously this solution won't work for you.
>>>
>>> Regarding changes to set re-invocation on failure, at this time I don't
>>> believe it's something I'd be willing to add to GNU make directly,
>>> especially not an option that simply retries every failed job.  This is
>>> almost never useful (why would you want to retry a compile, or link, or
>>> similar?  It will always just fail again, take longer, and generate
>>> confusing duplicate output--at best).
>>>
>>> The right answer for this problem is to modify the makefile to properly
>>> retry those specific rules which need it.
>>>
>>> I commiserate with you that your environment is static and you're not
>>> permitted to modify it, however adding new specialized capabilities to
>>> GNU make so that makefiles don't have to be modified isn't a design
>>> philosophy I want to adopt.
>>>
>>>
>>> _______________________________________________
>>> Help-make mailing list
>>> address@hidden
>>> https://lists.gnu.org/mailman/listinfo/help-make
>>>
>>