[lmi] Sporadic segfault during OnInit()

[Greg Chicares]

Here is a sporadic segfault that occurs about twice in every ten trials.
To reproduce:

  cd /opt/lmi/data
  mkdir bogus
  cd bogus
  echo -n '2450449 2458849' > expiry
  echo -n '8fa614e38dde6f7ab0f9fade87dfa2e3' > passkey
  md5sum expiry passkey > validated.md5
  echo -n 'foo' > foo
  echo -n 'eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee  foo' >> validated.md5

  cd /opt/lmi/bin
  ./lmi_wx_shared --data_path=/opt/lmi/data/bogus

Just keep repeating the last command above until it segfaults.
(Otherwise, two messageboxes should appear, saying:
  Exit code 1 from command 'md5sum --check --status validated.md5'.
and:
  Untimely warning:
  At least one required file is missing, altered, or invalid. Try reinstalling.
  [file /lmi/src/lmi/authenticity.cpp, line 287]
because the 'bogus' directory's files are deliberately invalid.)

By inserting temporary print statements, I determined where the
segfault arises--working from outer through inner frames:
  in 'main_wx.cpp':
    authenticate_system();
  in 'authenticity.cpp':
    system_command("md5sum --check --status " + std::string(md5sum_file()));
  in 'system_command_wx.cpp':
    fatal_error() << std::flush;
  in 'alert.cpp':
    fatal_error_alert_function(alert_string());

The outermost frame in which the segfault arises is in 'main_wx.cpp',
inside Skeleton::OnInit(). If I move the offending line in that function:
    authenticate_system();
down, to follow this line:
    frame_->CreateStatusBar();
then no segfault occurs. Could this possibly suggest that there's some
instability in wx while the application is initializing?

[Unfortunately, moving that line also requires that I copy all xml
resource files to /opt/lmi/data/bogus/ ; and that also means that
lmi must load those files before it can validate their md5sums, so
it's not a really tidy way to solve the problem.]

I found it hard to reproduce the segfault in gdb, but eventually did
stumble on a rare success:

gdb ./lmi_wx_shared
set arg --data_path=/opt/lmi/data/bogus
handle SIGTRAP nostop noprint
run
Program received signal SIGSEGV, Segmentation fault.
0x0022edf7 in ?? ()
(gdb) bt
#0  0x0022edf7 in ?? ()
#1  0x004ba4a5 in __cxa_throw ()
#2  0x00404caf in fatal_error_alert (s=...) at /lmi/src/lmi/alert_wx.cpp:138
#3  0x01adff06 in fatal_error_buf::raise_alert (this=0x0)
    at 
C:/opt/lmi/MinGW-20090203/bin/../lib/gcc/mingw32/3.4.5/../../../../include/c++/3.4.5/bits/basic_string.h:1456
#4  0x01b7e421 in alert_buf::sync (this=0x0) at /lmi/src/lmi/alert.cpp:144
#5  0x00531277 in std::ostream::flush() ()
#6  0x004a34dd in (anonymous namespace)::concrete_system_command (
    command_line=...)
    at 
C:/opt/lmi/MinGW-20090203/bin/../lib/gcc/mingw32/3.4.5/../../../../include/c++/3.4.5/bits/ostream.tcc:63
#7  0x0199938e in Authenticity::Assay (candidate=..., data_path=...)
    at /lmi/src/lmi/authenticity.hpp:89
#8  0x019999c3 in authenticate_system () at /lmi/src/lmi/authenticity.cpp:283
#9  0x0045d523 in Skeleton::OnInit (this=0x3a4cb80)
    at /lmi/src/lmi/main_wx.cpp:646
#10 0x004f8c41 in wxAppConsoleBase::CallOnInit (this=0x0)
    at /opt/lmi/local/include/wx-2.9/wx/app.h:94
#11 0x6c2d4a56 in wxEntryReal(int&, wchar_t**) ()
   from 
/opt/lmi/local/lib/wxmsw295u_gcc_gcc-345-e98c5f92805493f150656403ffef3bb0.dll
#12 0x6c45b613 in wxEntry(HINSTANCE__*, HINSTANCE__*, char*, int) ()
---Type <return> to continue, or q <return> to quit---
   from 
/opt/lmi/local/lib/wxmsw295u_gcc_gcc-345-e98c5f92805493f150656403ffef3bb0.dll
#13 0x00450e9d in WinMain (hInstance=0x0, hPrevInstance=0x0, lpCmdLine=0x0,
    nCmdShow=0) at /lmi/src/lmi/main_wx.cpp:215
#14 0x004c3028 in main ()

Studying lmi's 'fatal_error' code, I was unable to determine how
this is possible. Initially I wondered whether the cause might lie
somewhere in the rather old version of libstdc++ we're using; but
the 'fatal_error' stuff always works everywhere else...and it works
here, too, if I rearrange the code as above, so that the code leading
to the crash is invoked later in OnInit(). Do you have any idea what's
really happening here?