bug#42740: Segfault in libssh during ‘guix copy’

[Ludovic Courtès]

Hi Artyom!

Artyom Poptsov <poptsov.artyom@gmail.com> skribis:

> please check if this branch will work without segfaults in Guix:
>   https://github.com/artyom-poptsov/guile-ssh/tree/wip-fix-segfaults-on-gc
>
> Key changes:
>
> - Channels are now protecting the parent session from GC'ing.
>
> - Every channel procedure now ensures that the parent session is
>   connected before calling any libssh procedures upon a channel
>   instance.  The idea is that a channel cannot be created when a session
>   is disconnected and when channel is present and the session is closed,
>   it means that the session is disconnected and freed.

Looks like the problem is still there, after all:

--8<---------------cut here---------------start------------->8---
$ guix describe
Generacio 154   Aug 29 2020 14:49:14    (nuna)
  guix 0ec6b8a
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 0ec6b8afd7e7a6c288fbf48c5779f2e0bdaffb55
$  guix copy --to=olimex coreutils-minimal
sending 3 store items (86 MiB) to 'A20-OLinuXino.local'...
Adres-eraro(nekropsio elŝutita)
$ gdb $(type -P guile) core

[...]

Core was generated by 
`/gnu/store/0w76khfspfy8qmcpjya41chj3bgfcy0k-guile-3.0.4/bin/guile 
--no-auto-com'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section `.reg-xstate/25533' in core file.
#0  0x00007f1ba90e4185 in deflate_fast ()
   from /gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1
[Current thread is 1 (Thread 0x7f1baefb9b80 (LWP 25533))]
(gdb) bt
#0  0x00007f1ba90e4185 in deflate_fast ()
   from /gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1
#1  0x00007f1ba90e653d in deflate () from 
/gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1
#2  0x00007f1ba89b1b4a in gzip_compress (session=session@entry=0x12a4b20, 
source=source@entry=0x12a5580, 
    level=<optimized out>) at 
/tmp/guix-build-libssh-0.9.4.drv-0/source/src/gzip.c:91
#3  0x00007f1ba89b1e83 in compress_buffer (session=session@entry=0x12a4b20, 
buf=0x12a5580)
    at /tmp/guix-build-libssh-0.9.4.drv-0/source/src/gzip.c:112
#4  0x00007f1ba898eb5f in packet_send2 (session=session@entry=0x12a4b20)
    at /tmp/guix-build-libssh-0.9.4.drv-0/source/src/packet.c:1632
#5  0x00007f1ba898ec32 in ssh_packet_send (session=session@entry=0x12a4b20)
    at /tmp/guix-build-libssh-0.9.4.drv-0/source/src/packet.c:1810
#6  0x00007f1ba8978639 in channel_write_common (channel=0x12b0e90, 
data=0x7f1b9dba7020, len=65536, is_stderr=0)
    at /tmp/guix-build-libssh-0.9.4.drv-0/source/src/channels.c:1488
#7  0x00007f1ba89fce7a in write_to_channel_port ()
   from 
/gnu/store/hw2wb78q8zxza1p1kdi8bffdbi1hb19n-guile-ssh-0.13.1/lib/libguile-ssh.so.13
#8  0x00007f1baf67eedc in scm_i_write_bytes (port=#<port #<port-type channel 
7f1baaa1c6c0> 7f1ba7f25300>, 
    src="#<vu8vector>" = {...}, start=0, count=65536) at ports.c:2865
#9  0x00007f1baf68686f in scm_put_bytevector (port=#<port #<port-type channel 
7f1baaa1c6c0> 7f1ba7f25300>, 
    bv="#<vu8vector>" = {...}, start=<optimized out>, count=<optimized out>) at 
r6rs-ports.c:676

[...]

(gdb) info threads
  Id   Target Id                         Frame 
* 1    Thread 0x7f1baefb9b80 (LWP 25533) 0x00007f1ba90e4185 in deflate_fast ()
   from /gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1
  2    Thread 0x7f1baec93700 (LWP 25534) warning: Unexpected size of section 
`.reg-xstate/25534' in core file.
0x00007f1baf56094c in futex_wait_cancelable (private=<optimized out>, 
    expected=0, futex_word=0x7f1baf5b86e8 <mark_cv+40>) at 
../sysdeps/nptl/futex-internal.h:183
  3    Thread 0x7f1bac9d0700 (LWP 25537) warning: Unexpected size of section 
`.reg-xstate/25537' in core file.
0x00007f1ba90e479f in deflate_fast ()
   from /gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1
  4    Thread 0x7f1bae302700 (LWP 25535) warning: Unexpected size of section 
`.reg-xstate/25535' in core file.
0x00007f1baf56094c in futex_wait_cancelable (private=<optimized out>, 
    expected=0, futex_word=0x7f1baf5b86e8 <mark_cv+40>) at 
../sysdeps/nptl/futex-internal.h:183
  5    Thread 0x7f1baa6f9700 (LWP 25538) warning: Unexpected size of section 
`.reg-xstate/25538' in core file.
0x00007f1baf5640a4 in __libc_read (fd=10, buf=buf@entry=0x7f1baa6f8660, 
    nbytes=nbytes@entry=1) at ../sysdeps/unix/sysv/linux/read.c:26
  6    Thread 0x7f1bad971700 (LWP 25536) warning: Unexpected size of section 
`.reg-xstate/25536' in core file.
0x00007f1baf56094c in futex_wait_cancelable (private=<optimized out>, 
    expected=0, futex_word=0x7f1baf5b86e8 <mark_cv+40>) at 
../sysdeps/nptl/futex-internal.h:183
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f1bac9d0700 (LWP 25537))]
#0  0x00007f1ba90e479f in deflate_fast ()
   from /gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1
(gdb) bt
#0  0x00007f1ba90e479f in deflate_fast ()
   from /gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1
#1  0x00007f1ba90e653d in deflate () from 
/gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1
#2  0x00007f1ba89b1b4a in gzip_compress (session=session@entry=0x12a4b20, 
source=source@entry=0x12a5580, 
    level=<optimized out>) at 
/tmp/guix-build-libssh-0.9.4.drv-0/source/src/gzip.c:91
#3  0x00007f1ba89b1e83 in compress_buffer (session=session@entry=0x12a4b20, 
buf=0x12a5580)
    at /tmp/guix-build-libssh-0.9.4.drv-0/source/src/gzip.c:112
#4  0x00007f1ba898eb5f in packet_send2 (session=session@entry=0x12a4b20)
    at /tmp/guix-build-libssh-0.9.4.drv-0/source/src/packet.c:1632
#5  0x00007f1ba898ec32 in ssh_packet_send (session=session@entry=0x12a4b20)
    at /tmp/guix-build-libssh-0.9.4.drv-0/source/src/packet.c:1810
#6  0x00007f1ba897a178 in ssh_channel_send_eof (channel=channel@entry=0x12b0930)
    at /tmp/guix-build-libssh-0.9.4.drv-0/source/src/channels.c:1250
#7  0x00007f1ba897a23b in ssh_channel_close (channel=0x12b0930)
    at /tmp/guix-build-libssh-0.9.4.drv-0/source/src/channels.c:1301
#8  0x00007f1ba89fcc36 in ptob_close ()
   from 
/gnu/store/hw2wb78q8zxza1p1kdi8bffdbi1hb19n-guile-ssh-0.13.1/lib/libguile-ssh.so.13
#9  0x00007f1baf67c153 in release_port (port=#<port #<port-type channel 
7f1baaa1c6c0> 7f1ba8e73400>)
    at ports.c:165
#10 0x00007f1baf67f19b in close_port (port=#<port #<port-type channel 
7f1baaa1c6c0> 7f1ba8e73400>, 
    explicit=<optimized out>) at ports.c:893
#11 0x00007f1baf63632a in scm_c_with_exception_handler (type=type@entry=#t, 
    handler=handler@entry=0x7f1baf6ad7e0 <catch_post_unwind_handler>, 
    handler_data=handler_data@entry=0x7f1bac9cf970, 
thunk=thunk@entry=0x7f1baf6ad920 <catch_body>, 
    thunk_data=thunk_data@entry=0x7f1bac9cf970) at exceptions.c:170
#12 0x00007f1baf6adb1d in scm_c_catch (tag=tag@entry=#t, 
body=body@entry=0x7f1baf67f200 <do_close>, 
    body_data=<optimized out>, handler=<optimized out>, 
handler_data=handler_data@entry=0x0, 
    pre_unwind_handler=pre_unwind_handler@entry=0x0, 
pre_unwind_handler_data=0x0) at throw.c:168
#13 0x00007f1baf6adb3e in scm_internal_catch (tag=tag@entry=#t, 
body=body@entry=0x7f1baf67f200 <do_close>, 
    body_data=<optimized out>, handler=<optimized out>, 
handler_data=handler_data@entry=0x0) at throw.c:177
#14 0x00007f1baf67ad84 in finalize_port (ptr=<optimized out>, data=<optimized 
out>) at ports.c:710
#15 0x00007f1baf58a6ef in GC_invoke_finalizers ()
   from /gnu/store/iycnpxxrg8m9wf9w58d6zvp9sdby6m9d-libgc-7.6.12/lib/libgc.so.1
#16 0x00007f1baf63ee79 in scm_run_finalizers () at finalizers.c:399
#17 0x00007f1baf63eefd in finalization_thread_proc (unused=<optimized out>) at 
finalizers.c:234
--8<---------------cut here---------------end--------------->8---

So we have the finalization thread closing a channel of session
0x12a4b20 (which causes a write on the channel), and the main thread
writing to a channel of that same session.  This is exactly what I
described at <https://issues.guix.gnu.org/26976#11>:

  AIUI, that means there’s one output compression buffer per session,
  and it’s not thread-safe (in Guile 2.2 finalizers are called from a
  separate thread.)

  I think the fix, in Guile-SSH, is to associate each libssh object
  (session, channel, etc.) with a mutex, and to protect all uses of the
  libssh object by that mutex.

Artyom, WDYT?  Do you think you could take a look into that?

In the meantime, I’ll look for the origin of the channel port that’s not
explicitly closed and see if we can work around it.

Ludo’.