Dear Maintainer,

During a recent apt run, my system became almost completely
unresponsive.

Luckily I was able to get to another terminal, find the offender with
`top`, and neutralize it.

/usr/lib/tracker/tracker-extract is dying with SIGSYS, backtrace below.

According to `coredumpctl list`, it restarted approximately 6000 times
before I replaced it with a symlink to /bin/true, which finally allowed
aptitude to finish. That auto-restart behavior, in itself, deserves a grave
bug at least.

I haven't got debug symbols, but I will follow-up after installing them.

# coredumpctl gdb
           PID: 23962 (tracker-extract)
           UID: 1000 (ben)
           GID: 1000 (ben)
        Signal: 31 (SYS)
     Timestamp: Mon 2016-12-19 21:16:42 PST (12min ago)
  Command Line: /usr/lib/tracker/tracker-extract
    Executable: /usr/lib/tracker/tracker-extract
 Control Group: /
         Slice: -.slice
       Boot ID: db830d9ceb0441bd97c414fa76830061
    Machine ID: a9f5005691f11289cd92098b52b4f3f9
      Hostname: joyplim
       Storage: /var/lib/systemd/coredump/core.tracker-extract.1000.db830d9ceb0441bd97c414fa76830061.23962.1482211002000000000000.lz4
       Message: Process 23962 (tracker-extract) of user 1000 dumped core.

                Stack trace of thread 23990:
                #0  0x00007fe59d43e4d7 mlock (libc.so.6)
                #1  0x00007fe5766da6f1 fluid_defsfont_load_sampledata (libfluidsynth.so.1)
                #2  0x00007fe5766de78a fluid_defsfont_load (libfluidsynth.so.1)
                #3  0x00007fe5766de8c1 fluid_defsfloader_load (libfluidsynth.so.1)
                #4  0x00007fe5766ebdc9 fluid_synth_sfload (libfluidsynth.so.1)
                #5  0x00007fe57698d276 n/a (libgstfluidsynthmidi.so)
                #6  0x00007fe58647d6de gst_element_change_state (libgstreamer-1.0.so.0)
                #7  0x00007fe58647de4f n/a (libgstreamer-1.0.so.0)
                #8  0x00007fe584266edd n/a (libgstplayback.so)
                #9  0x00007fe5842725c4 n/a (libgstplayback.so)
                #10 0x00007fe584272fe2 n/a (libgstplayback.so)
                #11 0x00007fe58427322f n/a (libgstplayback.so)
                #12 0x00007fe59dc41f75 g_closure_invoke (libgobject-2.0.so.0)
                #13 0x00007fe59dc53f82 signal_emit_unlocked_R (libgobject-2.0.so.0)
                #14 0x00007fe59dc5cbcc g_signal_emit_valist (libgobject-2.0.so.0)
                #15 0x00007fe59dc5cfaf g_signal_emit (libgobject-2.0.so.0)
                #16 0x00007fe59dc463a4 g_object_dispatch_properties_changed (libgobject-2.0.so.0)
                #17 0x00007fe586451634 n/a (libgstreamer-1.0.so.0)
                #18 0x00007fe59dc48979 g_object_notify_by_spec_internal (libgobject-2.0.so.0)
                #19 0x00007fe586492434 n/a (libgstreamer-1.0.so.0)
                #20 0x00007fe58649ded0 gst_pad_push_event (libgstreamer-1.0.so.0)
                #21 0x00007fe577397829 n/a (libgstmidi.so)
                #22 0x00007fe5864c8a11 n/a (libgstreamer-1.0.so.0)
                #23 0x00007fe59d98cd3e g_thread_pool_thread_proxy (libglib-2.0.so.0)
                #24 0x00007fe59d98c345 g_thread_proxy (libglib-2.0.so.0)
                #25 0x00007fe59d701464 start_thread (libpthread.so.0)
                #26 0x00007fe59d4429df __clone (libc.so.6)

                Stack trace of thread 23964:
                #0  0x00007fe59d43e119 syscall (libc.so.6)
                #1  0x00007fe59d9aa14f g_cond_wait (libglib-2.0.so.0)
                #2  0x00007fe59d938ecb g_async_queue_pop_intern_unlocked (libglib-2.0.so.0)
                #3  0x00007fe59d98ce8d g_thread_pool_wait_for_new_task (libglib-2.0.so.0)
                #4  0x00007fe59d98c345 g_thread_proxy (libglib-2.0.so.0)
                #5  0x00007fe59d701464 start_thread (libpthread.so.0)
                #6  0x00007fe59d4429df __clone (libc.so.6)

                Stack trace of thread 23962:
                #0  0x00007fe59d43956d poll (libc.so.6)
                #1  0x00007fe59d9649f6 g_main_context_poll (libglib-2.0.so.0)
                #2  0x00007fe59d964d82 g_main_loop_run (libglib-2.0.so.0)
                #3  0x000056523e970d47 n/a (/usr/lib/tracker/tracker-extract)

GNU gdb (Debian 7.12-4) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
"/usr/lib/tracker/tracker-extract": not in executable format: File format not recognized
[New LWP 23990]
[New LWP 23964]
[New LWP 23962]
[New LWP 23975]
[New LWP 23970]
[New LWP 23967]
[New LWP 23971]
[New LWP 23968]
[New LWP 23972]
[New LWP 23973]
[New LWP 23974]
[New LWP 23963]
[New LWP 23976]
[New LWP 23969]
[New LWP 23989]
[New LWP 23985]
[New LWP 23965]
[New LWP 23966]
Core was generated by `/usr/lib/tracker/tracker-extract'.
Program terminated with signal SIGSYS, Bad system call.
#0  0x00007fe59d43e4d7 in ?? ()
[Current thread is 1 (LWP 23990)]
(gdb) quit

Am 20.12.2016 um 06:42 schrieb Ben Longbons:

It's most certainly related to the new seccomp based sandbox.
Might be https://bugzilla.gnome.org/show_bug.cgi?id=776117

Would be great if you can try the patch that was committed as
https://git.gnome.org/browse/tracker/commit/?id=88cfdc5

Dear Maintainer,
until the rest of the OS was swapped out so that I *couldn't* kill it?

Just because it's `Priority: optional` doesn't mean it's not an RC bug
within that package.


Anyway, I got the debug symbols, but it looks like I just need the
syscall numbers ($orig_rax in all threads):

/usr/include/x86_64-linux-gnu/asm/unistd_64.h:#define __NR_poll 7
/usr/include/x86_64-linux-gnu/asm/unistd_64.h:#define __NR_mlock 149
/usr/include/x86_64-linux-gnu/asm/unistd_64.h:#define __NR_futex 202
/usr/include/x86_64-linux-gnu/asm/unistd_64.h:#define __NR_ppoll 271

Remember, SIGSYS is catchable, and the siginfo_t contains details. If a
program is going to use `seccomp`, it should log those details, before
exiting with a value that tells systemd not to restart it.

Michael, the three latest commits on tracker master from December 17
(not commited to the 1.10 branch) look significant too. On Ubuntu,
there were reports of crashes from dup2 (LP: #1649035). We might want
to use --disable-libmediaart

On Ubuntu there were also crashes from getrusage()
https://launchpad.net/bugs/1649004

Thanks,
Jeremy

Control: clone 848842 -2
Control: retitle 848842 tracker-extract: crashes with SIGSYS in mlock
Control: retitle -2 tracker-extract: dumps core repeatedly if seccomp raises SIGSYS

I've cloned a new bug for this. Please send any further messages
on this topic to the new bug (only), and reserve #848842 for the specific
crash that you reported (involving mlock()), for which there is an upstream
patch that I'm testing now.

I'll respond to the new bug when it has a bug number.

    S

(It is #851148.)

This restart behaviour is less straightforward than you might think.
There are two things that restart tracker-extract:

* On systems with dbus-user-session installed, `systemd --user` restarts
  tracker-extract on failure, as requested by the systemd unit.
  This is easy to avoid by adding SIGSYS to RestartPreventExitStatus,
  and I'm trying a patch that does that.

* On all systems, tracker-miner-fs watches the tracker-extract process
  and sends a D-Bus message to restart it, either started by
  `systemd --user` (with dbus-user-session) or by dbus-daemon (without).
  This is deliberate: tracker-extract only tries indexing any given
  file a limited number of times, and will eventually skip over it.

Processing each file n times doesn't do much to mitigate missing benign
syscalls in the seccomp sandbox, because n tries * m files is still a
large number: the assumption is that the typical failure mode is a
particular file that trips a bug in the extractor libraries.

The introduction of the seccomp sandbox is an unusual situation that
caused extraction of broad equivalence classes of files to fail,
breaking that assumption. There are several patches upstream that
hopefully resolve this, which I'm testing now.

I agree with the maintainer that this is not release-critical,
although I personally think the actual crashes might be.
to use anything much more complicated than a syscall in a signal
handler (in particular, stdio or GLib logging is a bad idea and
could deadlock). I think the correct thing here is for tracker-extract
to just crash, and let system-level diagnostic tools like
systemd-coredump work out why.

I suspect that the thing stalling your system here is actually
the repeated core-dump processing, not the repeated
tracker-extract startup - at least, that has been my experience when
dealing with repeatedly crashing software. systemd-coredump uses
relatively expensive compression.

    S

See attached. On my system, this delays the restart by a couple of
seconds, which hopefully mitigates the load issue.

    S

FYI, your mailer is not using TLS, so it's getting marked a spam.

It's quite true that doing anything *complicated* is forbidden. But
there's no need to be complicated.

Just write(2) a newline, a handful of fixed strings, and the hex value
of the syscall and instruction. snprintf(3) isn't safe, but converting
an integer to hex by hand is trivial anyway, compared to using a
seccomp sandbox. Personally, I would've just chrooted (having userns
these days makes sandboxing *so* much easier since you don't need
permissions to become a new "root").

True, but the crashing application is still responsible for the
consequences, especially when it's something that nobody ever asked
for anyway.

Dear Maintainer(s),

I just wanted to add in that on Mobian (=Debian ARM64 repos with
phone-specific package additions) on the PinePhone this seems to
happen to multiple users out of the box, with tracker continuously
crashing and restarting generating a large amount of dumps in the
default configuration. Especially on a phone I think this is
a bit of an issue from a battery runtime angle. Is there a
reliable work-around?

For what it's worth, here is the mobian ticket:

https://gitlab.com/mobian1/issues/-/issues/183

Looks like Debian bullseye is missing this patch:
https://gitlab.gnome.org/GNOME/tracker-miners/-/commit/30b24e9d379458b66f2465422821a66bec3a749b

strace of tracker-extract shows this:
    ...
    [pid 42781] openat(AT_FDCWD, "/sys/devices/system/cpu",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 21
    [pid 42781] fstat(21, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
    [pid 42781] getdents64(21, 0xffff5005a950 /* 32 entries */, 32768) = 928
    [pid 42781] getdents64(21, 0xffff5005a950 /* 0 entries */, 32768) = 0
    [pid 42781] close(21)                   = 0
    [pid 42781] sched_getaffinity(42781, 8, [0, 1, 2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12, 13, 14, 15]) = 8
    [pid 42781] faccessat(AT_FDCWD, "/etc/gcrypt/fips_enabled", F_OK)
= -1 ENETDOWN (Network is down)
    [pid 42781] --- SIGSYS {si_signo=SIGSYS, si_code=SYS_SECCOMP,
si_call_addr=0xffffa5a7c014, si_syscall=__NR_faccessat,
si_arch=AUDIT_ARCH_AARCH64} ---

For reference, my machine is a Honeycomb LX2K (LX2160A).

dana@honeycomb:~$ apt-cache policy tracker-miner-fs
tracker-miner-fs:
  Installed: 2.3.5-2.1
  Candidate: 2.3.5-2.1
  Version table:
 *** 2.3.5-2.1 500
        500 http://mirrors.edge.kernel.org/debian bullseye/main arm64 Packages
        100 /var/lib/dpkg/status