#1106879 libmimalloc3: Illegal instruction Running mtxrun --generate.

Package:
libmimalloc3
Source:
libmimalloc3
Description:
compact general purpose allocator with excellent performance
Submitter:
Date:
2025-09-03 08:55:01 UTC
Severity:
normal
#1106879#5
Date:
2025-05-31 00:48:55 UTC
From:
To:
~# apt-get install --purge context
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
 libmimalloc3 luametatex
The following NEW packages will be installed:
 libmimalloc3 luametatex
The following held packages will be changed:
 context
The following packages will be upgraded:
 context
1 upgraded, 2 newly installed, 0 to remove and 1 not upgraded.
Need to get 99.2 MB of archives.
After this operation, 6,011 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 https://deb.debian.org/debian unstable/main amd64 libmimalloc3
amd64 3.0.3+ds-1 [58.1 kB]
Get:2 https://deb.debian.org/debian trixie/main amd64 context all
2024.04.01.20240428+dfsg-2 [97.8 MB]
Get:3 https://deb.debian.org/debian trixie/main amd64 luametatex amd64
2.11.02+really2.11.01+ds-1+b1 [1,344 kB]
Fetched 99.2 MB in 10s (9,876 kB/s)
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
Reading changelogs... Done
apt-listchanges: Do you want to continue? [Y/n]
Selecting previously unselected package libmimalloc3:amd64.
(Reading database ... 470453 files and directories currently installed.)
Preparing to unpack .../libmimalloc3_3.0.3+ds-1_amd64.deb ...
Unpacking libmimalloc3:amd64 (3.0.3+ds-1) ...
Preparing to unpack .../context_2024.04.01.20240428+dfsg-2_all.deb ...
Unpacking context (2024.04.01.20240428+dfsg-2) over
(2021.03.05.20230120+dfsg-1+deb12u1) ...
Selecting previously unselected package luametatex.
Preparing to unpack .../luametatex_2.11.02+really2.11.01+ds-1+b1_amd64.deb ...
Unpacking luametatex (2.11.02+really2.11.01+ds-1+b1) ...
Setting up libmimalloc3:amd64 (3.0.3+ds-1) ...
Setting up luametatex (2.11.02+really2.11.01+ds-1+b1) ...
Setting up context (2024.04.01.20240428+dfsg-2) ...
Processing triggers for auctex (13.2-1) ...
Processing triggers for tex-common (6.19) ...
Running mktexlsr. This may take some time... done.
Running mtxrun --generate. This may take some time...
mtxrun --generate failed. Output has been stored in
/tmp/mtxrun.p2dVjw0e
Please include this file if you report a bug.

dpkg: error processing package tex-common (--configure):
installed tex-common package post-installation script subprocess
returned error exit status 1
Processing triggers for fontconfig (2.15.0-2.3) ...
Processing triggers for libc-bin (2.41-8) ...
Processing triggers for man-db (2.13.1-1) ...
Errors were encountered while processing:
tex-common
E: Sub-process /usr/bin/dpkg returned an error code (1)


The contents of /tmp/mtxrun.p2dVjw0e are:
~# cat /tmp/mtxrun.p2dVjw0e
Illegal instruction


I've taken a look at the mimalloc-3.0.3 source. The setting
MI_OPT_ARCH=OFF in debian/rules appears to
invoke a falsehood if my interpretation of
mimalloc-3.0.3+ds/CMakeLists.txt is correct:

# Compiler and architecture specific flags
if(CMAKE_C_COMPILER_ID MATCHES "AppleClang|Clang|GNU|Intel" AND NOT
CMAKE_SYSTEM_NAME MATCHES "Haiku")
# ^true
 if(MI_OPT_ARCH)
   # ^false - go to continue...
   if(APPLE AND CMAKE_C_COMPILER_ID MATCHES "AppleClang|Clang" AND
CMAKE_OSX_ARCHITECTURES)   # to support multi-arch binaries (#999)
     if("arm64" IN_LIST CMAKE_OSX_ARCHITECTURES)
       list(APPEND MI_OPT_ARCH_FLAGS "-Xarch_arm64;-march=armv8.1-a")
     endif()
     if("x86_64" IN_LIST CMAKE_OSX_ARCHITECTURES)
       list(APPEND MI_OPT_ARCH_FLAGS
"-Xarch_x86_64;-march=haswell;-Xarch_x86_64;-mavx2")
     endif()
   # continue...
   elseif(MI_ARCH STREQUAL "x64")
   # ^true. Architecture is x86-64
     set(MI_OPT_ARCH_FLAGS "-march=haswell;-mavx2")    # fast bit scan
(since 2013)
     # ^These flags force Haswell architecture or above and allow the
generation of AVX2 code
   elseif(MI_ARCH STREQUAL "arm64")
     set(MI_OPT_ARCH_FLAGS "-march=armv8.1-a")         # fast atomics
(since 2016)
   endif()
 endif()
endif()

#1106879#10
Date:
2025-05-31 16:40:00 UTC
From:
To:
arch baselines violations are serious.

C.

#1106879#17
Date:
2025-08-12 23:34:09 UTC
From:
To:
[...]

I don't think this diagnosis can be the right one, because
https://buildd.debian.org/status/fetch.php?pkg=mimalloc&arch=amd64&ver=3.0.3%2Bds-1&stamp=1743673297&raw=0
is verbose enough to show compiler command lines and they don't include
-march.  Something else must be going on ...

Thanks,

#1106879#22
Date:
2025-08-13 11:05:57 UTC
From:
To:
Control: severity -1 important

I'm downgrading this for the moment as I can't currently find evidence
that it's a baseline violation.  I've tried this in various ancient qemu
CPU models ("-cpu Conroe", "-cpu qemu64", "-cpu core2duo"), and it seems
fine there.  I'm prepared to believe that I've missed something, but
figuring it out seems like a bit of a fishing expedition.

Adam, could you please give some details about your machine, and attach
your /proc/cpuinfo file?  That might help narrow things down.

Thanks,

#1106879#29
Date:
2025-08-14 12:47:35 UTC
From:
To:
On Wed, 13 Aug 2025 at 23:06, Colin Watson <cjwatson@debian.org> wrote:
...

Hi Colin! It's a baseline violation. Your analysis of the build files
was helpful but ultimately I just had to check the dmesg log for the
segfault and look up the offset in the shared library:

traps: mtxrun[62011] trap invalid opcode ip:7fe6e4f64988
sp:7ffe42301c80 error:0 in libmimalloc.so.3.0[c988,7fe6e4f5e000+15000]

c988:       f3 48 0f b8 c2          popcnt rax,rdx

Here are the additional occurrences of popcnt in the binary:

 $ objdump -d -m i386:x86-64:intel
./x86_64-linux-gnu/libmimalloc.so.3.0|grep popcnt
    c988:       f3 48 0f b8 c2          popcnt rax,rdx
    cbec:       f3 49 0f b8 c1          popcnt rax,r9
    cd56:       f3 4c 0f b8 d6          popcnt r10,rsi
    d4e3:       f3 48 0f b8 c2          popcnt rax,rdx
    d585:       f3 4d 0f b8 f5          popcnt r14,r13
    d5ca:       f3 48 0f b8 c2          popcnt rax,rdx
    def4:       f3 4c 0f b8 df          popcnt r11,rdi
    e273:       f3 48 0f b8 c7          popcnt rax,rdi
    ecd2:       f3 49 0f b8 c2          popcnt rax,r10
    ee21:       f3 49 0f b8 e8          popcnt rbp,r8
    f095:       f3 48 0f b8 c3          popcnt rax,rbx
    f1f9:       f3 4d 0f b8 d8          popcnt r11,r8

This library is going to segfault on hardware pre-Sandy Bridge.

The primary source uses a generic implementation of popcount or
built-in intrinsics, which should be OK if baseline code is in fact
being generated:

https://github.com/microsoft/mimalloc/blob/09a27098aa6e9286518bd9c74e6ffa7199c3f04e/include/mimalloc/internal.h#L1069

Regards,
Adam

#1106879#34
Date:
2025-08-14 13:03:34 UTC
From:
To:
OK.  I think this is coming from supposedly CPUID-guarded code:

   $ git grep -i popcnt
   include/mimalloc/bits.h:extern bool _mi_cpu_has_popcnt;
   include/mimalloc/bits.h:    if mi_unlikely(!_mi_cpu_has_popcnt) { return _mi_popcount_generic(x); }
   include/mimalloc/bits.h:    __asm ("popcnt\t%1,%0" : "=r"(r) : "r"(x) : "cc");
   include/mimalloc/bits.h:    if mi_unlikely(!_mi_cpu_has_popcnt) { return _mi_popcount_generic(x); }
   include/mimalloc/bits.h:    return (size_t)mi_msc_builtinz(__popcnt)(x);
   include/mimalloc/bits.h:    return (size_t)mi_msc_builtinz(__popcnt)(x);
   src/init.c:mi_decl_cache_align bool _mi_cpu_has_popcnt = false;
   src/init.c:    _mi_cpu_has_popcnt = ((cpu_info[2] & (1 << 23)) != 0); // bit 23 of ECX : see <https://en.wikipedia.org/wiki/CPUID#EAX=1:_Processor_Info_and_Feature_Bits>
   src/init.c:  _mi_cpu_has_popcnt = true;

Here's the relevant code (for GCC/amd64):

     #if !defined(__BMI1__)
     if mi_unlikely(!_mi_cpu_has_popcnt) { return _mi_popcount_generic(x); }
     #endif
     size_t r;
     __asm ("popcnt\t%1,%0" : "=r"(r) : "r"(x) : "cc");
     return r;

And:

   static void mi_detect_cpu_features(void) {
     // FSRM for fast short rep movsb/stosb support (AMD Zen3+ (~2020) or Intel Ice Lake+ (~2017))
     // EMRS for fast enhanced rep movsb/stosb support
     uint32_t cpu_info[4];
     if (mi_cpuid(cpu_info, 7)) {
       _mi_cpu_has_fsrm = ((cpu_info[3] & (1 << 4)) != 0); // bit 4 of EDX : see <https://en.wikipedia.org/wiki/CPUID#EAX=7,_ECX=0:_Extended_Features>
       _mi_cpu_has_erms = ((cpu_info[1] & (1 << 9)) != 0); // bit 9 of EBX : see <https://en.wikipedia.org/wiki/CPUID#EAX=7,_ECX=0:_Extended_Features>
     }
     if (mi_cpuid(cpu_info, 1)) {
       _mi_cpu_has_popcnt = ((cpu_info[2] & (1 << 23)) != 0); // bit 23 of ECX : see <https://en.wikipedia.org/wiki/CPUID#EAX=1:_Processor_Info_and_Feature_Bits>
     }
   }

In principle this sort of thing should be OK.  But maybe __BMI1__ is
defined and so the generic fallback isn't present, or maybe the CPUID
check is incorrect for your CPU?  I'll check the former when I have a
little more time, but if you could check the latter that would be
helpful.

(Note that I don't know this library well.  I'm just trying to figure
this out since it's been blocking some other things I work on.)

#1106879#39
Date:
2025-08-14 14:47:01 UTC
From:
To:
This observation may help. The relevant code (for GCC/amd64) should
include a runtime check before executing the popcnt instruction if
__BMI1__ is not defined.

If we look at the surrounding assembly before popcnt is executed there
should be a simple compare for the boolean value _mi_cpu_has_popcnt
and a jump skipping the popcnt instruction if the result is false:

    c976:       f0 49 0f b1 0c 24       lock cmpxchg QWORD PTR [r12],rcx
    c97c:       75 f2                   jne    c970
<mi_arena_reload@@Base+0x7a0>
    c97e:       48 8d 0d bb ce 01 00    lea    rcx,[rip+0x1cebb]
 # 29840 <mi_stats_get_json@@Base+0x10a70>
    c985:       48 21 c2                and    rdx,rax
    c988:       f3 48 0f b8 c2          popcnt rax,rdx

Here's another snippet from the binary (no check is apparent):

    ded9:       b8 01 00 00 00          mov    eax,0x1
    dede:       48 d3 e0                shl    rax,cl
    dee1:       48 83 e8 01             sub    rax,0x1
    dee5:       89 d1                   mov    ecx,edx
    dee7:       48 d3 e0                shl    rax,cl
    deea:       49 89 c7                mov    r15,rax
    deed:       48 8d 05 4c b9 01 00    lea    rax,[rip+0x1b94c]
 # 29840 <mi_stats_get_json@@Base+0x10a70>
    def4:       f3 4c 0f b8 df          popcnt r11,rdi

Locally there is no indication these simple checks are being performed.

This indicates __BMI1__ is incorrectly defined.

(Or perhaps there is no runtime detection going on and what we are
seeing is output from intrinsics compiled for a platform that supports
the popcnt instruction.)

Regards,
Adam

#1106879#46
Date:
2025-09-01 19:33:40 UTC
From:
To:
I have two 64-bit laptops, both running Debian unstable.  On the newish
one, which has an Intel(R) Core(TM) i7-5600U CPU, the mtxrun works fine
(no "illegal instruction").

On the ancient one, which has an Intel(R) Core(TM)2 Duo CPU T9300, it
fails right away.  According to x86-64-level
<https://github.com/HenrikBengtsson/x86-64-level>, it's a level-1 CPU.

Here is its cpuinfo, in case it's useful:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Core(TM)2 Duo CPU     T9300  @ 2.50GHz
stepping	: 6
microcode	: 0x60f
cpu MHz		: 1197.124
cache size	: 6144 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm pti tpr_shadow flexpriority vpid dtherm ida vnmi
vmx flags	: vnmi flexpriority tsc_offset vtpr vapic
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips	: 4988.01
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Core(TM)2 Duo CPU     T9300  @ 2.50GHz
stepping	: 6
microcode	: 0x60f
cpu MHz		: 1197.124
cache size	: 6144 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm pti tpr_shadow flexpriority vpid dtherm ida vnmi
vmx flags	: vnmi flexpriority tsc_offset vtpr vapic
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips	: 4988.01
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

#1106879#51
Date:
2025-09-03 08:52:43 UTC
From:
To:
On the ancient laptop, I applied the attached patch and rebuilt
libmimalloc3.  The patch completely avoids the popcnt instruction and
forces the use of _mi_popcount_generic().  Now mtxrun works (so the
long-missing formats have finally been generated).

Of course, this patch is not a viable solution in general.  But it
indicates a problem in the test for whether the CPU has popcnt.  So, it
might point someone who understands how this test should work toward a
proper patch.