Hi,

I tried to optimize the build of apt-cacher-ng a little bit, and I found
that I have not used -Wl,threads with gold. So I "fixed" this and then
it started crashing. Apparently linking shared libraries works, but
executables segfaults. Tried with current sid (version 2.31, I think)
and experimental, same problem.

While trying to reproduce, I run into the following (and not sure why
omiting -Wl,threads makes this go away, but something with -lug looks
fishy).

gcc is latest from Sid:
ii  gcc-8          8.3.0-2      amd64        GNU C compiler

dev/apt-cacher-ng/dev/relbuild $ gdb /usr/bin/ld.gold
GNU gdb (Debian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/ld.gold...(no debugging symbols found)...done.
(gdb) run -plugin /usr/lib/gcc/x86_64-linux-gnu/8/liblto_plugin.so -plug
Starting program: /usr/bin/ld.gold -plugin /usr/lib/gcc/x86_64-linux-gnu/8/liblto_plugin.so -plug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
/usr/bin/ld.gold: error: cannot find -lug

Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
65      ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory.
(gdb) bt
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
#1  0x000055555569ab8d in ?? ()
#2  0x000055555569fe1d in ?? ()
#3  0x000055555566edb9 in ?? ()
#4  0x00005555556b0448 in ?? ()
#5  0x00005555556b066a in ?? ()
#6  0x00005555555a148b in ?? ()
#7  0x00007ffff7a2009b in __libc_start_main (main=0x5555555a0ef0, argc=4, argv=0x7fffffffdf08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdef8) at ../csu/libc-start.c:308
#8  0x00005555555a235a in ?? ()

Best regards,
Eduard.

Control: tags -1 + moreinfo

what is -Wl,threads supposed to do? Should be -Wl,--threads

I suspect the run line is incomplete.

Hallo,
* Matthias Klose [Wed, Mar 13 2019, 02:23:55PM]:

Yes, it was -Wl,--threads.

That's what "file" told me checking the core dump.

However, I cannot reproduce it anymore. After updating the toolchain,
the issue seems to have disappeared.

Best regards,
Eduard.

Hallo,
* Eduard Bloch [Fri, Mar 15 2019, 09:39:26AM]:
reproducible. It seems to be a combination of -Wl,--threads and LTO
(and maybe a machine with 4+ cores) which triggers segfaulting. It
happens on Intel and AMD machines, hardly a hardware problem.

The issue was only temporarily covered by local problems (maybe some
cmake issue, garbling parameters after updating cmake without clearing
the local cache and then by incorrect detection of LTO in the build
script).

Repro:

git clone https://salsa.debian.org/blade/apt-cacher-ng.git
cd apt-cacher-ng
git checkout d204340833d88424792a42013e310f21aecef39b
there:
cmake .
(that should include: "-- LTO use: ON" and "Performing Test LD_MULTITHREADED - Success")
make -j$(nproc)

Result:
...
collect2: fatal error: ld terminated with signal 11 [Speicherzugriffsfehler], core dumped
compilation terminated.
make[2]: *** [client/CMakeFiles/in.acng.dir/build.make:84: in.acng] Fehler 1
make[1]: *** [CMakeFiles/Makefile2:91: client/CMakeFiles/in.acng.dir/all] Fehler 2

Best regards,
Eduard.

Dear Maintainer,
just tried to collect some more information.

I could reproduce the issue inside a qemu amd64 VM,
running current Buster.
Not every attempt resulted in a crash.

Indeed "file" seems to report just the first 80
character of the command line.
Like gdb does too when it opens a core file.
The full command line is shown by "coredumpctl gdb"
in my case, as I have installed systemd-coredump.


The actual crash looks like we expect there 320 entries in
the array syms, but that is now pointing to 0x0.

    (gdb) print *symtab
    $12 = {nsyms = 320, aux = 0x7f1370010a80, syms = 0x0, id = 5375926499655753679}

See more details in attached file.

Kind regards,
Bernhard


# coredumpctl gdb 20585
           PID: 20585 (ld.gold)
           UID: 1000 (benutzer)
           GID: 1000 (benutzer)
        Signal: 11 (SEGV)
     Timestamp: Fri 2019-03-22 21:29:24 CET (2min 24s ago)
  Command Line: /usr/bin/ld.gold -plugin /usr/lib/gcc/x86_64-linux-gnu/8/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/8/lto-wrapper -plugin-opt=-fresolution=/tmp/ccfAic5u.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -o ../apt-cacher-ng /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/8 -L/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/8/../../.. --as-needed -O1 --discard-all --no-undefined --build-id=sha1 --threads --gc-sections CMakeFiles/apt-cacher-ng.dir/apt-cacher.cc.o ../libsupacng.so -latomic -levent -lwrap -lz -lbz2 -llzma -lssl -lcrypto -lsystemd -lpthread -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/8/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o


(gdb) bt
#0  gold::Pluginobj::get_symbol_resolution_info (this=0x7f1370001040, symtab=0x7ffe8567e0a0, nsyms=<optimized out>, syms=<optimized out>, version=<optimized out>) at ../../gold/plugin.cc:1279
#1  0x00007f13ed5be0cf in write_resolution () at ../../src/lto-plugin/lto-plugin.c:490
#2  all_symbols_read_handler () at ../../src/lto-plugin/lto-plugin.c:657
#3  0x000055794b4cdbaf in gold::Plugin::all_symbols_read (this=<optimized out>) at ../../gold/plugin.cc:403
#4  gold::Plugin_manager::all_symbols_read (task=<optimized out>, input_objects=<optimized out>, symtab=<optimized out>, dirpath=<optimized out>, mapfile=<optimized out>, last_blocker=<optimized out>, workqueue=0x7ffe8567dd70, this=0x55794be1e760) at ../../gold/plugin.cc:842
#5  gold::Plugin_manager::all_symbols_read (this=0x55794be1e760, workqueue=0x7ffe8567dd70, task=<optimized out>, input_objects=<optimized out>, symtab=<optimized out>, dirpath=<optimized out>, mapfile=<optimized out>, last_blocker=<optimized out>) at ../../gold/plugin.cc:819
#6  0x000055794b4cdcfc in gold::Plugin_hook::run (this=0x55794be54680, workqueue=0x7ffe8567dd70) at ../../gold/options.h:1635
#7  0x000055794b50f338 in gold::Workqueue::find_and_run_task (this=0x7ffe8567dd70, thread_number=7) at ../../gold/token.h:290
#8  0x000055794b50f55a in gold::Workqueue::process (this=0x7ffe8567dd70, thread_number=7) at ../../gold/workqueue.cc:495
#9  0x000055794b50f5d3 in gold::Workqueue_threader_threadpool::process (thread_number=<optimized out>, this=<optimized out>) at ../../gold/workqueue-internal.h:92
#10 gold::Workqueue_thread::thread_body (arg=0x55794be51c80) at ../../gold/workqueue-threads.cc:117
#11 0x00007f13edae5fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#12 0x00007f13eda1682f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95