#977043 fwupdmgr[1191] trap invalid opcode ip:4247d0 sp:bf98504c error:0 in fwupdmgr[423000+15000]

Package:
fwupd
Source:
fwupd
Description:
Firmware update daemon
Submitter:
Martin-Éric Racine
Date:
2021-03-10 18:03:03 UTC
Severity:
important
#977043#5
Date:
2020-12-10 14:35:00 UTC
From:
To:
$ sudo invoke-rc.d fwupd restart
Job for fwupd.service failed because a fatal signal was delivered causing the control process to dump core.
See "systemctl status fwupd.service" and "journalctl -xe" for details.
invoke-rc.d: initscript fwupd, action "restart" failed.
● fwupd.service - Firmware update daemon
     Loaded: loaded (/lib/systemd/system/fwupd.service; static)
     Active: failed (Result: core-dump) since Thu 2020-12-10 16:29:46 EET; 271ms ago
       Docs: https://fwupd.org/
    Process: 1404 ExecStart=/usr/libexec/fwupd/fwupd (code=dumped, signal=ILL)
   Main PID: 1404 (code=dumped, signal=ILL)

joulu 10 16:29:46 geode systemd[1]: fwupd.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
joulu 10 16:29:46 geode systemd[1]: fwupd.service: Child 1404 belongs to fwupd.service.
joulu 10 16:29:46 geode systemd[1]: fwupd.service: Main process exited, code=dumped, status=4/ILL
joulu 10 16:29:46 geode systemd[1]: fwupd.service: Failed with result 'core-dump'.
joulu 10 16:29:46 geode systemd[1]: fwupd.service: Service will not restart (restart setting)
joulu 10 16:29:46 geode systemd[1]: fwupd.service: Changed start -> failed
joulu 10 16:29:46 geode systemd[1]: fwupd.service: Job 992 fwupd.service/start finished, result=failed
joulu 10 16:29:46 geode systemd[1]: Failed to start Firmware update daemon.
joulu 10 16:29:46 geode systemd[1]: fwupd.service: Unit entered failed state.
joulu 10 16:29:46 geode systemd[1]: fwupd.service: Control group is empty.

- -- System Information:
Debian Release: bullseye/sid
  APT prefers testing-debug
  APT policy: (1000, 'testing-debug'), (1000, 'testing'), (500, 'stable')
Architecture: i386 (i586)

Kernel: Linux 5.9.0-4-686 (SMP w/1 CPU thread)
Locale: LANG=fi_FI.UTF-8, LC_CTYPE=fi_FI.UTF-8 (charmap=UTF-8), LANGUAGE=fi:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages fwupd depends on:
ii  libc6                  2.31-5
ii  libefiboot1            37-6
ii  libelf1                0.182-1
ii  libflashrom1           1.2-5
ii  libfwupd2              1.5.1-5
ii  libfwupdplugin1        1.5.1-5
ii  libglib2.0-0           2.66.3-2
ii  libgudev-1.0-0         234-1
ii  libgusb2               0.3.5-1
ii  libjcat1               0.1.3-2
ii  libjson-glib-1.0-0     1.6.0-1
ii  libpolkit-gobject-1-0  0.105-29
ii  libsmbios-c2           2.4.3-1
ii  libsoup2.4-1           2.72.0-2
ii  libsqlite3-0           3.34.0-1
ii  libtss2-esys0          3.0.1-1
ii  libxmlb1               0.1.15-2
ii  shared-mime-info       2.0-1

Versions of packages fwupd recommends:
ii  bolt                              0.9-1
ii  fwupd-i386-signed [fwupd-signed]  1.5.1+5
ii  python3                           3.9.0-4
pn  secureboot-db                     <none>
ii  udisks2                           2.9.1-2

fwupd suggests no packages.

- -- no debconf information
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEyJACx3qL7GpObXOQrh+Cd8S017YFAl/SMgwACgkQrh+Cd8S0
17asyg//aO2/vcgq8DAwqAnmK0SX8tTCeWYb3sJ0ISqUf+yy3i8z+rlbxs/RXIGy
CT7NLsXGn1bK+RkjYRiQn7whCLl2dOlsO2zpWAn2JBocMyfud+L/M3N4JMUz0Sdk
kTAmzXQ4/pc2dYLjnKBd7PPox2JezvE0hfdWn79ugVOiBZmOES7nFoFXJ8r3L6RD
qLvxg5H0YrZnxn0LQ7PBqX7XiP4WsCpmbcKhd4/odxKsjlEuUa4nguEpog4TTWqK
RW6q4Yg9tgSUs+0qNRYSagDMs/9/T9Sdjy4usN2D4HHlCt/NrKWU4Z/bZVSHZwo3
ohzTsqY6NwtqosZ/IZdqujnMW74dy19w4scrX+n8CtT3cDQv4rS8w6DWzdp/EKZQ
T1Vf0lEbYXv0EHvejVecpTpE76VqlOXrENSI8vobLMgSXwo1UqlPXn1a49E1IpU6
j/jGxtgBngfYDUwKSANAoo0uqdAtbkvxLS8gfaRj5rcDLG0cedVKtW9eg/haCK1x
c/ZcF9L37XTrc/lmkZ7frzGmGeCtSlGXMVS9JOPlDRk+9ULvfMhQIOPuPrXVjuf6
pkWZCc05FtyXTv1j/R3hLhtCWJQTL6z3MaU2y/sdt5G/0GyrLWfJu15dgGgfquAY
VAESpx8C+hnrsrhsYprdHU2bdnO8ep7pLUhgk8fjTiY1tfiTWzc=
=oVJ+
-----END PGP SIGNATURE-----

#977043#10
Date:
2020-12-10 14:43:41 UTC
From:
To:
$ sudo coredumpctl debug 1191 --output /tmp/coredump_fwupd
           PID: 1191 (fwupdmgr)
           UID: 62803 (62803)
           GID: 62803 (62803)
        Signal: 4 (ILL)
     Timestamp: Thu 2020-12-10 16:25:33 EET (15min ago)
  Command Line: /usr/bin/fwupdmgr refresh --no-metadata-check
    Executable: /usr/bin/fwupdmgr
 Control Group: /system.slice/fwupd-refresh.service
          Unit: fwupd-refresh.service
         Slice: system.slice
       Boot ID: 3381b144358645a48346c772bd0a2b8f
    Machine ID: 1063a9d1fb9df6e371ea9f94491345ed
      Hostname: geode
       Storage:
/var/lib/systemd/coredump/core.fwupdmgr.62803.3381b144358645a48346c772bd0a2b8f.1191.1607610333000000.zst
       Message: Process 1191 (fwupdmgr) of user 62803 dumped core.

                Stack trace of thread 1191:
                #0  0x00000000004247d0 n/a (fwupdmgr + 0x77d0)
                #1  0x00000000004274b1 _start (fwupdmgr + 0xa4b1)

GNU gdb (Debian 10.1-1+b1) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/fwupdmgr...
Reading symbols from
/usr/lib/debug/.build-id/94/55f3483d573ccb3fe3629c46880ba8b46a9b02.debug...
[New LWP 1191]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/fwupdmgr refresh --no-metadata-check'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x004247d0 in __libc_start_main@plt ()
(gdb) bt full
#0  0x004247d0 in __libc_start_main@plt ()
No symbol table info available.
#1  0x004274b1 in _start ()
No symbol table info available.
(gdb) thread apply all bt full

Thread 1 (Thread 0xb47228c0 (LWP 1191)):
#0  0x004247d0 in __libc_start_main@plt ()
No symbol table info available.
#1  0x004274b1 in _start ()
No symbol table info available.
(gdb)

#977043#15
Date:
2020-12-10 14:53:30 UTC
From:
To:
Reading symbols from /usr/bin/fwupdmgr...
Reading symbols from
/usr/lib/debug/.build-id/94/55f3483d573ccb3fe3629c46880ba8b46a9b02.debug...
[New LWP 1191]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/fwupdmgr refresh --no-metadata-check'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x004247d0 in __libc_start_main@plt ()
(gdb) backtrace full
#0  0x004247d0 in __libc_start_main@plt ()
No symbol table info available.
#1  0x004274b1 in _start ()
No symbol table info available.
(gdb) info registers
eax            0x437690            4421264
ecx            0xbf985074          -1080536972
edx            0xb7f8c080          -1208434560
ebx            0x4469f8            4483576
esp            0xbf98504c          0xbf98504c
ebp            0x0                 0x0
esi            0x3                 3
edi            0x427480            4355200
eip            0x4247d0            0x4247d0 <__libc_start_main@plt>
eflags         0x10202             [ IF RF ]
cs             0x73                115
ss             0x7b                123
ds             0x7b                123
es             0x7b                123
fs             0x0                 0
gs             0x33                51
(gdb) x/16i $pc
=> 0x4247d0 <__libc_start_main@plt>:    endbr32
   0x4247d4 <__libc_start_main@plt+4>:    jmp    *0x20(%ebx)
   0x4247da <__libc_start_main@plt+10>:    nopw   0x0(%eax,%eax,1)
   0x4247e0 <fwupd_device_set_created@plt>:    endbr32
   0x4247e4 <fwupd_device_set_created@plt+4>:    jmp    *0x24(%ebx)
   0x4247ea <fwupd_device_set_created@plt+10>:    nopw   0x0(%eax,%eax,1)
   0x4247f0 <fwupd_client_get_host_product@plt>:    endbr32
   0x4247f4 <fwupd_client_get_host_product@plt+4>:    jmp    *0x28(%ebx)
   0x4247fa <fwupd_client_get_host_product@plt+10>:    nopw   0x0(%eax,%eax,1)
   0x424800 <sqlite3_column_int64@plt>:    endbr32
   0x424804 <sqlite3_column_int64@plt+4>:    jmp    *0x2c(%ebx)
   0x42480a <sqlite3_column_int64@plt+10>:    nopw   0x0(%eax,%eax,1)
   0x424810 <sqlite3_exec@plt>:    endbr32
   0x424814 <sqlite3_exec@plt+4>:    jmp    *0x30(%ebx)
   0x42481a <sqlite3_exec@plt+10>:    nopw   0x0(%eax,%eax,1)
   0x424820 <fwupd_remote_get_firmware_base_uri@plt>:    endbr32
(gdb) thread apply all backtrace

Thread 1 (Thread 0xb47228c0 (LWP 1191)):
#0  0x004247d0 in __libc_start_main@plt ()
#1  0x004274b1 in _start ()
(gdb) quit

#977043#24
Date:
2021-03-10 10:20:25 UTC
From:
To:
Hello Martin-Éric,
without being involved in packaging fwupd I tried to
have a look at this issue.

I could not reproduce it inside a i386 qemu VM (not even
with "-cpu pentium"). Have not tested on real hardware.


Looking up the endbr32 instruction, it seems it belongs to something
called "Control-flow Enforcement Technology" (CET, indirect branch) [1].

The opcode for this instruction got selected to run on old
CPUs as NOP, but it looks like your CPU handles it differently.
 From the system name it is some "geode" CPU?

(In [2] someone mentions also a illegal instruction
for a geode CPU with the endbr32 instruction.)


Maybe you could add to this bug report the output of
'lscpu' or 'cat /proc/cpuinfo' ?

Then the maintainer might be able to tell if this CPU
meets the Debian baseline requirements for bullseye.
([3], maybe outdated? Is there a better "baseline" description?)


Kind regards,
Bernhard

[1] https://www.linuxplumbersconf.org/event/2/contributions/147/attachments/72/83/CET-LPC-2018.pdf
[2] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84148#c3
[3] https://wiki.debian.org/ArchitectureSpecificsMemo#i386-1

#977043#29
Date:
2021-03-10 10:49:11 UTC
From:
To:
Cheers,
Julien

#977043#34
Date:
2021-03-10 11:01:30 UTC
From:
To:
This could indeed be it. The baseline for Bullseye still is a basic
686. Additionally, the lowest Linux kernel for i386 is linux-image-686
which is configured for Geode. Still, the bug just appeared a few days
ago when the package in Testing was updated. This seems like a
regression.

$ lscpu
Architecture:                    i586
CPU op-mode(s):                  32-bit
Byte Order:                      Little Endian
Address sizes:                   32 bits physical, 32 bits virtual
CPU(s):                          1
On-line CPU(s) list:             0
Thread(s) per core:              1
Core(s) per socket:              1
Socket(s):                       1
Vendor ID:                       AuthenticAMD
CPU family:                      5
Model:                           10
Model name:                      Geode(TM) Integrated Processor by AMD PCS
Stepping:                        2
CPU MHz:                         498.044
BogoMIPS:                        996.08
L1d cache:                       57 KiB
L1i cache:                       57 KiB
L2 cache:                        128 KiB
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers
and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline,
STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu de pse tsc msr cx8 sep pge cmov
clflush mmx mmxext 3dnowext 3dnow cpuid 3dnowprefetch vmmcall


ke 10. maalisk. 2021 klo 12.49 Julien Cristau (jcristau@debian.org) kirjoitti:

#977043#39
Date:
2021-03-10 11:08:49 UTC
From:
To:
You filed this 4 months ago, so it seems rather older?

Cheers,
Julien

#977043#44
Date:
2021-03-10 11:13:35 UTC
From:
To:
ke 10. maalisk. 2021 klo 13.08 Julien Cristau (jcristau@debian.org) kirjoitti:

Apparently, it comes and goes.  It started showing up in 'coredumpctl
list' just a few days ago. That's when I sent the output to the bug.

Martin-Éric

#977043#49
Date:
2021-03-10 16:24:56 UTC
From:
To:
You probably meant this:

DisabledPlugins=test;test_ble;invalid;cpu

Still signal 4.

TIME                            PID   UID   GID SIG COREFILE  EXE
Wed 2021-03-10 18:23:39 EET    1322     0     0   4 error     /usr/bin/fwupdmgr

ke 10. maalisk. 2021 klo 17.49 Limonciello, Mario
(Mario.Limonciello@dell.com) kirjoitti:

#977043#54
Date:
2021-03-10 16:30:34 UTC
From:
To:
Yes, I forgot it renamed - that's right.
Also I didn't properly acknowledge the crash was in the client not the daemon, so this wouldn't have done anything anyway.

I tend to think this is a compiler issue, not a fwupd upstream or packaging issue.

#977043#59
Date:
2021-03-10 15:49:17 UTC
From:
To:
As an experiment, can you please try to disable the "cpu" plugin in /etc/fwupd/daemon.conf?
Add to "BlacklistPlugins" list.