bug#35245: libltdl.so uses 32-bit search path in 64-bit mode in Solaris 11.4

[Vita Batrla]

Hi all,

I noticed that 64-bit version of libtool's libltdl.so shipped in Solaris 11.4
is compiled with 32-bit search path. This mismatch leads causes a failure to
dynamically load a module using lt_dlopen() interface.

Here's an example of a 64-bit executable trying to dynamically load
"libcrypt.so":

$ truss -t open,mmapobj ./ltdldemo
openat(AT_FDCWD, "/var/ld/64/ld.config", O_RDONLY) Err#2 ENOENT
openat(AT_FDCWD, "/usr/lib/64/libltdl.so.7", O_RDONLY) = 3
openat(AT_FDCWD, "/lib/64/libc.so.1", O_RDONLY) = 3
calling lt_dlinit
calling lt_dlopen
openat(AT_FDCWD, "/usr/lib/libcrypt.so", O_RDONLY) = 3

^ found "libcrypt.so" in 32-bit path

mmapobj(3, MMOBJ_INTERPRET, 0x7FFFBF6ED210, 0x7FFFBFFFEA0C, 0x00000000) Err#48 
ENOTSUP

^ failed to map it

libcrypt.so: file not found

I assume the failure is obvious. A 64-bit program doesn't like 32-bit
library (/usr/lib/libcrypt.so) and so mmapobj(2) syscall returns ENOTSUP
per its man page:

       ENOTSUP       The current user data model does not match the fd  to  be
                     interpreted.  For example, a 32-bit process that tried to
                     use mmapobj() to interpret a 64-bit object  would  return

The fix is to use 64-bit default search path in 64-bit mode (/lib/64
/usr/lib/64). There are two workarounds:

1. LTDL_LIBRARY_PATH=/lib/64:/usr/lib/64
2. re-compile 64-bit libtool:
     ./configure lt_cv_sys_dlsearch_path="/lib/64 /usr/lib/64"

Perhaps the best solution would be to auto-detect whether to use 32-bit or
64-bit library search paths [1][2][3] on Solaris OS in a configure script.

Please see a suggested fix in an attached patch.

Thanks,

Vita

[1] Search paths per ld(1) man page:
       LIBPATH        For  32-bit  libraries, the default search path is /lib,
                      followed by /usr/lib. For 64-bit libraries, the  default
                      search path is /lib/64, followed by /usr/lib/64.

[2] Search paths per ld.so.1(1) man page:
       The runtime linker uses a  prescribed  search  path  for  locating  the
       dynamic  dependencies  of  an  object. The default search paths are the
       runpath recorded in the object, followed by a series of  defaults.  For
       32-bit  objects, the defaults are /lib followed by /usr/lib. For 64-bit
       objects, the  defaults  are  /lib/64  followed  by  /usr/lib/64.

[3] Search paths per crle(1) man page:
           The  default  search paths for 32-bit ELF objects are /lib followed
           by /usr/lib. For 64-bit ELF objects, the default search  paths  are
           /lib/64 followed by /usr/lib/64.

000-64-bit-sys_dlsearch_path.patch
Description: Text document