[lmi] Problems on corporate server--live stream

[Greg Chicares]

Vadim--Feel free to jump in if you feel like it, or to ignore this if
you don't. I'm just using this email to record the problems I'm seeing
and the fixes I'm trying. In addition to the 'nemo' issues previously
discussed, I'm seeing a failure in schroot's script that is supposed
to unmount a chroot's dynamic mounts, and I've removed an installation
of 'wine' on the host.

/opt/lmi/src/lmi[0]$git pull
/opt/lmi/src/lmi[0]$pushd /tmp
/tmp /opt/lmi/src/lmi ~
/tmp[0]$/opt/lmi/src/lmi/gwc/redhat_rebuild.sh
Last-modified header missing -- time-stamps turned off.
2020-10-12 19:50:05 
URL:https://raw.githubusercontent.com/let-me-illustrate/lmi/master/lmi_setup_00.sh
 [6116/6116] -> "lmi_setup_00.sh" [1]
20201012T235103Z ./lmi_setup_02.sh: Removed old chroot.
20201012T235118Z ./lmi_setup_05r.sh: Reconfigured redhat chroot.
20201012T235124Z ./lmi_setup_07r.sh: Installed EPEL.
20201012T235125Z ./lmi_setup_10r.sh: Installed 'debootstrap' [redhat].
20201012T235531Z ./lmi_setup_11.sh: Ran 'debootstrap'.
20201013T003102Z ./lmi_setup_20.sh: Installed debian packages.
20201013T003104Z ./lmi_setup_21.sh: Updated debian chroot.
20201013T003109Z ./lmi_setup_24.sh: Configured users.
20201013T003109Z ./lmi_setup_25.sh: Configured {zsh,vim,git} for user 'root'.
20201013T003110Z ./lmi_setup_25.sh: Configured {zsh,vim,git} for user 
'[REDACTED]'.
20201013T003111Z ./lmi_setup_25.sh: Configured {zsh,vim,git} for user 
'[REDACTED2]'.
E: You are required to change your password immediately (password aged)
20201013T003111Z ./lmi_setup_29.sh: Created lmi directories.
20201013T003112Z ./lmi_setup_30.sh: Copied optional files.
20201013T003137Z ./lmi_setup_40.sh: Configured 'wine' for user '[REDACTED]'.
20201013T003156Z ./lmi_setup_40.sh: Configured 'wine' for user '[REDACTED2]'.
E: You are required to change your password immediately (password aged)
Finished building lmi.
20201013T011805Z ./lmi_setup_42.sh: Installed lmi for '[REDACTED]'.

Installing debian packages took about thirty-five minutes:
  20201012T235531Z ./lmi_setup_11.sh: Ran 'debootstrap'.
  20201013T003102Z ./lmi_setup_20.sh: Installed debian packages.
whereas it takes about seven on my personal machine:
  20201011T123938Z ./lmi_setup_11.sh: Ran 'debootstrap'.
  20201011T124616Z ./lmi_setup_20.sh: Installed debian packages.
but the server's apt-get log says
  Need to get 23.1 MB/811 MB of archives.
so it's using cache and downloading rather little. Weird. I don't
recall it taking that long before.

Anyway, it ended there with return code 1. Instead, it was supposed to
continue and run 'lmi_setup_43.sh' next. Examining the logs:

20201013T011805Z ./lmi_setup_42.sh: Installed lmi for '[REDACTED]'.
E: 10mount: umount: 
/run/schroot/mount/lmi_bullseye_3-9fc3415f-d9ef-4c67-be6c-0cb6ad47e2cb: target 
is busy.
E: 10mount:         (In some cases useful info about processes that use
E: 10mount:          the device is found by lsof(8) or fuser(1))
E: 10mount: rmdir: failed to remove 
'/var/run/schroot/mount/lmi_bullseye_3-9fc3415f-d9ef-4c67-be6c-0cb6ad47e2cb': 
Device or resource busy
E: lmi_bullseye_3-9fc3415f-d9ef-4c67-be6c-0cb6ad47e2cb: Chroot setup failed: 
stage=setup-stop

https://wiki.debian.org/Schroot
| to retrive the PID you can take a piece of the name of the directory,
| says "d2c072e7" and look for in the /proc filesystem:
|
| ~$ grep -r d2c072e7 /proc/*/mountinfo

I tried that, with '0cb6ad', but it was no help--way too many PIDs are shown.

I can "run" the session with:
  sudo schroot --chroot=session:<Tab> --run-session
but that doesn't seem helpful.

These commands:
  lsof  /var/run/schroot/mount/lmi_bullseye_3-<Tab>
  fuser /var/run/schroot/mount/lmi_bullseye_3-<Tab>
return nothing, so maybe it was a transient problem?
No, it persists--this command:
  sudo schroot --end-session --all-sessions
fails, repeating the "busy" message above. So does
  sudo schroot --recover-session --chroot=session:<Tab>

This seemed worth looking into:
  https://bugzilla.redhat.com/show_bug.cgi?id=726025
| please edit the /etc/schroot/default/fstab file,
| change all entries from rbind to bind,
| and see if the problem goes away?
as well as this URI that it links to:
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=622756
but they're from 2011, while the server has:
  schroot-1.6.10 (04 May 2014)
and both of these filesystem tables:
  /etc/schroot/default/fstab
  /etc/schroot/lmi_profile/fstab
use only 'bind', not 'rbind'.

So I used 'umount -l', the lazy-but-forcible option, and then
  $mount | grep cb6ad
showed nothing.

Skipping 'lmi_setup_43.sh' (set up proprietary product repository),
I tried running 'lmi_setup_44.sh' (nychthemeral test) directly:

$schroot --chroot=chroot:lmi_bullseye_3 --user=`whoami` --directory=tmp 
./lmi_setup_44.sh 2>1 |less -S

0022
0002
LMI_TRIPLET = "x86_64-w64-mingw32"
  Production system built--ready to start GUI test in another session.
wine: a wine server seems to be running, but I cannot connect to it.
   You probably need to kill that process (it might be pid 29580).
make[2]: *** [/opt/lmi/src/lmi/workhorse.make:1331: cli_selftest] Error 1
make[1]: *** [/opt/lmi/src/lmi/workhorse.make:1316: cli_tests] Error 2
make: *** [GNUmakefile:241: /opt/lmi/gcc_x86_64-w64-mingw32/build/ship] Error 2
make: Target 'cgi_tests' not remade because of errors.
make: Target 'cli_tests' not remade because of errors.
20201013T124435Z ./lmi_setup_44.sh: Ran nychthemeral test for '[REDACTED]'.
20201013T124435Z ./lmi_setup_44.sh: Ran nychthemeral test for '[REDACTED]'.

Searching for that exact error message:
  
https://www.google.com/search?q=%22wine:+a+wine+server+seems+to+be+running,+but+I+cannot+connect+to+it.%22&filter=0
finds one unanswered question at winehq.org, an apparent copy
of the same at some domain whose name looks like leetspeak,
and several Cyrillic instances of a "Lenux Game Tread", mostly
at some Hong Kong domain, which seem to suggest using 'proton'
instead of 'wine'. That's not a path I want to follow.

Stepping back and wondering how a wine server could possibly be
running, I noticed that wine is installed not only in chroots,
but also on the redhat host (probably from a long time ago when
I was trying to build lmi on that host), so I purge it:

#yum autoremove wine

and repeat this command:

$schroot --chroot=chroot:lmi_bullseye_3 --user=`whoami` --directory=tmp 
./lmi_setup_44.sh 2>1 |less -S

0022
0002
LMI_TRIPLET = "x86_64-w64-mingw32"
  Production system built--ready to start GUI test in another session.
0009:err:module:__wine_process_init failed to load L"Z:\\opt\\lmi\\bin\\lmi_cli_
20201013T135715Z ./lmi_setup_44.sh: Ran nychthemeral test for '[REDACTED]'.
20201013T135715Z ./lmi_setup_44.sh: Ran nychthemeral test for '[REDACTED]'.

This seems to represent progress--at least the 'schroot' command
returned zero. However:

ls /srv/chroot/lmi_bullseye_3/opt/lmi/bin/cli*

returns "no matches found".

It looks like the unwanted schroot session still existed, even after
I used 'umount -l' above, but now this command works:

sudo schroot --end-session --all-sessions

Repeating the above command yet again, it seems close to succeeding.
I see no errors obviously attributable to the OS or to 'wine'.
Somehow, this file exists:
  /opt/lmi/bin/data/configurable_settings.xml
instead of the intended one:
  /opt/lmi/data/configurable_settings.xml
but at least that appears to be a problem under my control.

$schroot --chroot=chroot:lmi_bullseye_3 --user=`whoami` --directory=tmp 
./lmi_setup_44.sh 2>1 |less -S

CODENAME=bullseye
+ CODENAME=bullseye
CHRTVER=3
+ CHRTVER=3
CHRTNAME=lmi_${CODENAME}_${CHRTVER}
+ CHRTNAME=lmi_bullseye_3

umask
+ umask
0022
umask g=rwx
+ umask g=rwx
umask
+ umask
0002

set +vx
+ set +vx
     CHROOT_UIDS=65345,51308,21135
    CHROOT_USERS=[REDACTED],[REDACTED2],nemo
    GIT_URL_BASE=https://github.com/let-me-illustrate/lmi/raw/master
    NORMAL_GROUP=lmi
NORMAL_GROUP_GID=19504
     NORMAL_USER=[REDACTED]
set +v

assert_not_su
+ assert_not_su
+ id -u
+ [ 65345 -eq 0 ]
assert_chrooted
+ assert_chrooted
+ [ lmi_bullseye_3 != lmi_bullseye_3 ]

# Allow script to continue even if some test fails.
/opt/lmi/src/lmi/nychthemeral_test.sh || true
+ /opt/lmi/src/lmi/nychthemeral_test.sh
LMI_TRIPLET = "x86_64-w64-mingw32"
  Production system built--ready to start GUI test in another session.
No readable file 'configurable_settings.xml' exists.
[configurable_settings.cpp : 108]
No readable file 'configurable_settings.xml' exists.
[configurable_settings.cpp : 108]
  184 errors
make[2]: *** No rule to make target '/opt/lmi/data/configurable_settings.xml', 
needed by 'cli_tests_init'.
make[2]: Target 'cli_tests_init' not remade because of errors.
make[2]: Target 'cli_selftest' not remade because of errors.
make[2]: Target 'cli_test-sample.cns' not remade because of errors.
make[2]: Target 'cli_test-sample.ill' not remade because of errors.
make[1]: *** [/opt/lmi/src/lmi/workhorse.make:1316: cli_tests] Error 2
make: *** [GNUmakefile:241: /opt/lmi/gcc_x86_64-w64-mingw32/build/ship] Error 2
make: Target 'cgi_tests' not remade because of errors.
make: Target 'cli_tests' not remade because of errors.
+ true

stamp=$(date -u +'%Y%m%dT%H%M%SZ')
+ date -u +%Y%m%dT%H%M%SZ
+ stamp=20201013T145732Z
echo "$stamp $0: Ran nychthemeral test for '$(whoami)'."  | tee /dev/tty
+ tee /dev/tty
+ whoami
+ echo 20201013T145732Z ./lmi_setup_44.sh: Ran nychthemeral test for 
'[REDACTED]'.
20201013T145732Z ./lmi_setup_44.sh: Ran nychthemeral test for '[REDACTED]'.