Re: [lmi] Nychthemeral test requirements

[Greg Chicares]

[...moving back to the public list now that we've dispensed with
any need to share files not in the repository...]

On 2018-08-30 11:51, Vadim Zeitlin wrote:
> On Tue, 28 Aug 2018 12:49:37 +0000 Greg Chicares <address@hidden> wrote:

[You might want to scan forward to the string
  Success at last
and then read backward only until your interest (rapidly) diminishes,
then resume from the "success" point.]

[...]
> GC> Please copy 'sample.ill' and 'sample.cns' from the lmi repository's
> GC> main (source) directory to /opt/lmi/test/ and then run the following
> GC> command in that main (source) directory (where the makefiles live):
> GC> 
> GC>   make $coefficiency system_test 2>&1 | less -S
> GC> 
> GC> I'm hoping that won't fail, although it will give warnings (at least)

Okay, at this point, the tests should run, and there should be a
complaint that there's no saved touchstone to compare against...

> GC> that you can then work around thus:
> GC> 
> GC>   mkdir /opt/lmi/touchstone
> GC>   cp -a /opt/lmi/test/* /opt/lmi/touchstone/
> GC> 
> GC> After that, I think the test should pass.

At this point, those two directories should have identical contents.
Now, when you repeat the system test, it should
 - regenerate all the '*.test' files in /opt/lmi/test/ , which should
   be identical to those created above, because running the same program
   with the same input should produce the same output; and then
 - compare all files (i.e., '*') in /opt/lmi/test/ to their preceding
   versions that had been copied to /opt/lmi/touchstone/ , expecting
   them all to be identical

>  Thank you for preempting my next question! After doing this, the test does
> pass, but outputs an error:
> 
> ---------------------------------- >8 --------------------------------------
> % make system_test && echo PASSED
> Generating product files.
> 
> All product files written.
> 
> System test:
> Files /opt/lmi/test/md5sums-20180830T1143Z and /opt/lmi/touchstone/md5sums 
> differ
> *** System test failed ***
>   9 system-test files compared
>   5 system-test files match
>   4 system-test files differ
>   0 system-test files missing

Zero files missing--so you must have copied all files to touchstone/ .
Four of nine files unexpectedly differ. Which ones? There should be a
file named 'md5sums' in both directories, which summarizes all the
files; clearly it's present, but differs, so
  diff -U0 /opt/lmi/test/md5sums /opt/lmi/touchstone/md5sums
should show the three other files that differ; those would probably
be the generated '.test' files, i.e.
  sample.test
  sample.000000000.test
  sample.000000001.test
contradicting the assumption that they would all match perfectly.

Let's see what happens if I try to reproduce that.

mkdir /opt/lmi/vz-test
cp -a sample.??? /opt/lmi/vz-test
mkdir /opt/lmi/vz-touchstone
echo "Run it once to generate files to use as a touchstone"
make test_dir='/opt/lmi/vz-test' touchstone_dir='/opt/lmi/vz-touchstone/' 
$coefficiency system_test >/dev/null 2>&1
cp -a /opt/lmi/vz-test/* /opt/lmi/vz-touchstone
echo "Now run it for real"
make test_dir='/opt/lmi/vz-test' touchstone_dir='/opt/lmi/vz-touchstone/' 
$coefficiency system_test
...
  9 system-test files compared
  5 system-test files match
  4 system-test files differ
  0 system-test files missing

Okay, reproduced. Sorry about that--the second 'cp -a' command was wrong:
it copied all files instead of the subset we truly want. Revised commands:

rm -rf /opt/lmi/vz-test
rm -rf /opt/lmi/vz-touchstone
mkdir /opt/lmi/vz-test
cp -a sample.??? /opt/lmi/vz-test
mkdir /opt/lmi/vz-touchstone
echo "Run it once to generate files to use as a touchstone"
make test_dir='/opt/lmi/vz-test' touchstone_dir='/opt/lmi/vz-touchstone/' 
$coefficiency system_test >/dev/null 2>&1
cp -a /opt/lmi/vz-test/*.{test,cns,ill} /opt/lmi/vz-touchstone
echo "Now run it for real"
make test_dir='/opt/lmi/vz-test' touchstone_dir='/opt/lmi/vz-touchstone/' 
$coefficiency system_test
...
Files /opt/lmi/vz-test/md5sums-20180830T1310Z and 
/opt/lmi/vz-touchstone//md5sums differ
*** System test failed ***
  4 system-test files compared
  3 system-test files match
  1 system-test files differ
  0 system-test files missing
...system test completed.

Oh. The problem now is that, although the md5sum of each file except 'md5sums'
is identical, the 'md5sums' file contains duplicates. The problem is:

# Naming the output files would be more natural, but that's infeasible
# because $(test_emission) can be overridden implicitly in ways that a
# makefile cannot readily discern.
...
        @$(MD5SUM) --binary $(basename $(notdir $@)).* >> $(system_test_md5sums)

because '$(basename $(notdir $@)).*' is 'sample.*' for both of the
input files copied above from the main repository. Fixing that is
somewhere between hard and infeasible, and it's easy to avoid, so
let's try one more time with a complete set of improved instructions:

rm -rf /opt/lmi/vz-test
rm -rf /opt/lmi/vz-touchstone
mkdir /opt/lmi/vz-test
cp -a sample.ill /opt/lmi/vz-test/sample_i.ill
cp -a sample.cns /opt/lmi/vz-test/sample_c.cns
mkdir /opt/lmi/vz-touchstone
echo "Run it once to generate files to use as a touchstone"
make test_dir='/opt/lmi/vz-test' touchstone_dir='/opt/lmi/vz-touchstone/' 
$coefficiency system_test >/dev/null 2>&1
cp -a /opt/lmi/vz-test/*.{test,cns,ill} /opt/lmi/vz-touchstone
echo "Wait for the minute to increment..."
sleep 1m
echo "Now run it for real"
make test_dir='/opt/lmi/vz-test' touchstone_dir='/opt/lmi/vz-touchstone/' 
$coefficiency system_test
...
All 5 files match.

Success at last. Sorry, I should have done all of this myself beforehand
instead of wasting your time.

Why wait one minute? Because in real practice we don't run the system test
more than once in the same minute, and therefore names like this:
  system_test_md5sums  := $(test_dir)/md5sums-$(yyyymmddhhmm)
provide all the resolution we need.

Perhaps we should change that. In the distant past, a full system test
took forty-five minutes, but now:

$time make $coefficiency system_test                             
...
All 1510 files match.
make $coefficiency system_test  126.21s user 53.77s system 614% cpu 29.309 total

and, of course, with fewer than 1510 test files, it'll be even faster
for you:

make test_dir='/opt/lmi/vz-test' touchstone_dir='/opt/lmi/vz-touchstone/'
2.35s user 0.34s system 98% cpu 2.725 total

I'd like to keep the strict ISO8601 "basic" format, and keep it human
readable, so I can't just add seconds:
  20180830T1200Z is recognizably noon today, but
  20180830T120001Z is harder to parse as a time one second later
I'd use fractional minutes:
  20180830T1200.01Z readable, but...
except gnu 'date' doesn't seem to offer that.

Maybe the least bad option is to change nothing and ask you not to run
more than one system test per minute.

> ..system test completed.
> PASSED
> ---------------------------------- >8 --------------------------------------
> 
> I.e. it says "failed" but make exits with 0 error code. Is this normal?

It is intentional. If the test ran to completion, that constitutes
"success", and a comparison of test vs. touchstone determines
whether it's "success squared". Exit code zero means only that
test results are ready to compare.

>  FWIW, nychthemeral_test.sh compensates this by failing, i.e. existing with
> error code 141,

That's intentional, too. That script ascertains whether or not any
slightest imperfection has been detected. Exit code zero means it's
safe to run 'git push'.

> but not showing any error messages other than the
> presumably non-fatal ones from grep.

That's not intentional. In my chroot I see only:

$./nychthemeral_test.sh 2>&1 | tee /tmp/lmi/logs/log | sed -f errors.sed
Is it time to 'make release_candidate'?
Production system built--ready to start GUI test in another session.
Do not forget to run wx_test.

and anything more is undesirable for a successful run. Kim reports
that with cygwin she sees other stuff such as you describe, and I'd
like to know how to prevent that from happening.

> It also outputs "0 tests succeeded"
> at the end which I don't yet understand -- but I'll debug it later, after
> (finally) committing the uniform initialization patch.

That message sounds like it might be coming from 'make unit_tests'.
Either I've made a mistake with shell redirection, or cygwin
behaves differently than debian. But we want the debian behavior.