Re: [lmi] Inchoate support for GNU/Linux builds with lmi makefiles

[Greg Chicares]

[TL;DR: export LD_LIBRARY_PATH and use '-fPIC'; then everything works]

On 2018-11-16 01:44, Vadim Zeitlin wrote:
> On Thu, 15 Nov 2018 23:52:44 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> On 2017-04-12 00:07, Greg Chicares wrote:
[...]
> GC> > /opt/lmi/src/lmi[0]$make $coefficiency build_type=native CXXFLAGS='-O2 
> -ggdb -m64 -Wno-pedantic' CFLAGS='-m64' LDFLAGS='-m64' unit_tests 
> unit_test_targets=bourn_cast_test
> GC> 
> GC> This updated command:
> GC> 
> GC> make $coefficiency build_type=native CXXFLAGS='-O2 -ggdb -m64 -fno-pie
> GC> -Wno-pedantic -Wno-useless-cast' CFLAGS='-m64 -fno-pie' LDFLAGS='-m64
> GC> -no-pie' unit_tests cli_tests >../log 2>&1
> GC> 
> GC> adds '-Wno-useless-cast' to ignore a pervasive boost-1.33.1 problem,
> 
>  I wonder why was it necessary when we already have a pragma disabling this
> warning in boost_regex.hpp.

Because the boost filesystem library also needs it with gcc-8.2.0
for x86_64-linux-gnu, e.g.:

make $coefficiency build_type=native CXXFLAGS='-O2 -ggdb -m64 -Wno-pedantic' 
CFLAGS='-m64' LDFLAGS='-m64' unit_tests unit_test_targets=global_settings_test
...
In file included from /opt/lmi/third_party/include/boost/mpl/int.hpp:20,
                 from /opt/lmi/third_party/include/boost/mpl/lambda_fwd.hpp:23,
                 from 
/opt/lmi/third_party/include/boost/mpl/aux_/na_spec.hpp:18,
                 from /opt/lmi/third_party/include/boost/mpl/or.hpp:24,
                 from 
/opt/lmi/third_party/include/boost/iterator/interoperable.hpp:11,
                 from 
/opt/lmi/third_party/include/boost/iterator/iterator_facade.hpp:11,
                 from /opt/lmi/third_party/include/boost/filesystem/path.hpp:16,
                 from /opt/lmi/src/lmi/path_utility.hpp:29,
                 from /opt/lmi/src/lmi/path_utility.cpp:24:
/opt/lmi/third_party/include/boost/mpl/aux_/static_cast.hpp:24:66: error: 
useless cast to type 'int
 #   define BOOST_MPL_AUX_STATIC_CAST(T, expr) static_cast<T>(expr)
                                                                  ^

We use boost::regex in so few cases that we created a 'boost_regex.hpp'
header with pragmata to suppress warnings. But we use boost::filesystem
so widely as to render such a technique far less attractive. And, while
I've tried to compile only the unit tests and the lmi CLI binary so far,
this warning is triggered only by boost:

/opt/lmi/src/lmi[0]$<../log grep 'error: useless' |wc -l
224
/opt/lmi/src/lmi[0]$<../log grep 'error: useless' |sort -u
/opt/lmi/third_party/include/boost/mpl/aux_/static_cast.hpp:24:66: error: 
useless cast to type 'int' [-Werror=useless-cast]
/opt/lmi/third_party/include/boost/mpl/aux_/static_cast.hpp:24:66: error: 
useless cast to type 'long int' [-Werror=useless-cast]

so we aren't missing any useful diagnostic.

> Do you use a version of gcc older than 7 or is
> there something else going on here that I'm missing?

TL;DR: 8.2.0 .

/opt/lmi/src/lmi[0]$whence gcc
/usr/bin/gcc
/opt/lmi/src/lmi[0]$gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 8.2.0-9' 
--with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs 
--enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr 
--with-gcc-major-version-only --program-suffix=-8 
--program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id 
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix 
--libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu 
--enable-libstdcxx-debug --enable-libstdcxx-time=yes 
--with-default-libstdcxx-abi=new --enable-gnu-unique-object 
--disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie 
--with-system-zlib --with-target-system-zlib --enable-objc-gc=auto 
--enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 
--with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic 
--enable-offload-targets=nvptx-none --without-cuda-driver 
--enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu 
--target=x86_64-linux-gnu
Thread model: posix
gcc version 8.2.0 (Debian 8.2.0-9) 

> GC> and '-fno-pie' and '-no-pie' to prevent numerous diagnostics like:
> GC> 
> GC> relocation R_X86_64_32 against `.rodata.str1.1' can not be used when
> GC> making a shared object; recompile with -fPIC
> 
>  I really don't think -no-pie is the right flag to use. Why not add -fPIC
> and compile all code as position-independent instead? It does impose a
> slight overhead for the code which never ends up in a shared library, but I
> don't think it's really noticeable.

That does seem to work. In earlier experiments, I had tried the
all-lowercase '-fpic'; those experiments failed, perhaps because
I was also trying some "pie" options. But this does work:

rm -rf ../build/lmi/Linux/gcc/native 
export LD_LIBRARY_PATH="/opt/lmi/src/build/lmi/Linux/gcc/native"
make $coefficiency build_type=native CXXFLAGS='-O2 -ggdb -m64 -fPIC 
-Wno-pedantic -Wno-useless-cast' CFLAGS='-m64 -fPIC' LDFLAGS='-m64 -fPIC' 
cli_tests >../log 2>&1

BTW, 32- vs. 64-bit speed comparison:

* 32-bit:
Test command line interface:
Test solve speed: 1.451e-002 s mean;      14411 us least of   7 runs
Test solve speed: 1.253e-001 s: first trial took longer than 1.000e-001 s 
desired limit
Test sample.cns:
    Input:        30 milliseconds
    Calculations: 72 milliseconds
    Output:       1 milliseconds
  0 errors
Test sample.ill:
    Input:        20 milliseconds
    Calculations: 71 milliseconds
    Output:       1 milliseconds
  0 errors

* 64-bit [in square brackets: 32-bit / 64-bit ratios where meaningful]
Test command line interface:
Test solve speed: 8.445e-03 s mean;  [1.7181764357608054] 7103 us least of  12 
runs
Test solve speed: 3.964e-02 s mean;                      39447 us least of   3 
runs
Test sample.cns:
    Input:        22 milliseconds [1.3636363636363635]
    Calculations: 36 milliseconds [2]
    Output:       1 milliseconds
  0 errors
Test sample.ill:
    Input:        15 milliseconds [1.3333333333333333]
    Calculations: 38 milliseconds [1.868421052631579]
    Output:       1 milliseconds
  0 errors

> GC> but perhaps $LDFLAGS needs to vary between $EXEEXT and $SHREXT,
> GC> because for '.so' targets I still get:
> GC> 
> GC> /usr/bin/ld: 
> /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crt1.o: in function 
> `_start':
> GC> (.text+0x20): undefined reference to `main'
> GC> collect2: error: ld returned 1 exit status
> 
>  This looks like a wrong linking command to me, i.e. is there a -shared in
> it? It seems that the linker is trying to create an executable instead of a
> shared library here.

Yes, '-shared' is specified for $SHREXT and not for $EXEEXT,
exactly the same way it's specified for 32-bit builds.
Specifying [capitalized] '-fPIC' in both $CXXFLAGS and $LDFLAGS
seems to have fixed that.

> GC> I also get:
> GC> 
> GC> /usr/bin/ld: skipping incompatible /opt/lmi/local/lib/liblzma.a when 
> searching for -llzma
> GC> 
> GC> which is apparently just a nuisance stemming from
> GC>   $(shell xml2-config --libs)
> GC> in 'posix_fhs.make': it finds the 32-bit msw xml2-config instead of the
> GC> debian native one in /usr/bin/ , which was built without lzma support;
> GC> this happens even though I've installed a native liblzma-dev in my chroot.
> 
>  I'm not sure how are things expected to work exactly in this case, but I'd
> strongly advise avoiding mixing MSW and native binaries under the same
> directory. I.e. it seems like your native build is using xml2-config from
> /opt/lmi/local/bin which it really shouldn't see at all in the first place/

Absolutely. Just to be clear, the binaries aren't mixed; the only
problem is that the msw 'xml2-config' script is accidentally found.
I can certainly fix that; but I'll do that last, because it only
causes a warning that is gracefully ignored.

> GC> And yet another related message from the past said:
> GC> 
> GC>   https://lists.nongnu.org/archive/html/lmi/2017-05/msg00018.html
> GC> | make $coefficiency build_type=native CXXFLAGS='-O2 -ggdb -m64 -fpic 
> GC> | -Wno-pedantic' CFLAGS='-m64 -fpic' LDFLAGS='-m64' cli_tests
> GC> |
> GC> | and everything built, but to run the tests I needed to do this:
> GC> |
> GC> | 
> LD_LIBRARY_PATH='$LD_LIBRARY_PATH:/opt/lmi/src/build/lmi/Linux/gcc/native'
> GC> 
> GC> but, if I obliterate the build directory:
> [...]
> GC> but still I get (e.g.):
> GC> 
> GC> ./lmi_cli_shared: error while loading shared libraries: liblmi.so: cannot 
> open shared object file: No such file or directory
> GC> 
> GC> Vadim, do you see anything obviously wrong in my approach?
> 
>  The only thing I can think of is that your LD_LIBRARY_PATH is not
> exported, which could well be the case if you hadn't used it before. So
> please try either
> 
>       $ export LD_LIBRARY_PATH
>       $ ./lmi_cli_shared
> 
> or, as I typically do
> 
>       $ LD_LIBRARY_PATH=/opt/lmi/src/build/lmi/Linux/gcc/native 
> ./lmi_cli_shared

Thanks very much: using 'export' made it work.

> GC> I don't really understand '-fpic' and '-f[no-]pie'
> 
>  With x86-64 Linux, all code used in shared libraries, including code in
> the static libraries which are linked into a shared one, must be
> position-independent (in 32 bits this was strongly recommended but, due to
> the peculiarities of this architecture, not actually necessary and so not
> imposed, while in 64 bits things just can't work without it), i.e. be
> compiled with -fPIC (or -fpic, but it's the same for x86-64, I think).

Yes, apparently the same for x86-64:

  https://linux.die.net/man/1/gcc
| When generating code for shared libraries,
| -fpic implies -msmall-data and -fPIC implies -mlarge-data.
but it also says
| -fPIC  This option makes a difference on the m68k, PowerPC and SPARC .
| -fpic  If the GOT size for the linked executable exceeds a machine-
| specific maximum size, you get an error message from the linker
| indicating that -fpic does not work; in that case, recompile with
| -fPIC instead. (These maximums are 8k on the SPARC and 32k on the
| m68k and RS/6000 . The 386 has no such limit.)

>  For the executables, using position-independent code is required to
> support ASLR (address space layout randomization) which complicates the
> exploitation of security holes in C and C++ code. Whatever you think about
> ASLR (and if you think it's a hack, I won't argue with you, but it's a
> pretty effective one), it's better to not disable PIE (position independent
> executable) without some good reason. BTW, -fpie results in a smaller
> overhead than -fPIC/-fpic, so it's a good idea to use it for the code which
> only ends up in the executables and not a library.

AIUI, debian enabled it by default ("--enable-default-pie" in gcc
configuration above), so it's automatically used whenever possible.