- Package:
- src:llvm-toolchain-19
- Source:
- src:llvm-toolchain-19
- Submitter:
- Paul Gevers
- Date:
- 2025-04-13 11:03:01 UTC
- Severity:
- normal
- Tags:
Hi, I happened to look at the llvm-toolchain-18/19 reproducibility results because they seem to be flaky. I noticed that the diffoscope output [1] showed differences in assembly code that to my untrained eyes suggest that the binaries of llvm-toolchain-18/19 might depend on the CPU capabilities of the build cpu, which might be a baseline violation. Can you please check? It shows calls like pand vs vpand and movd vs vmovd. A quick query on the internets seems to indicate that compiling with -mno-avx could prevent this delta. Paul https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/llvm-toolchain-19.html objdump --line-numbers --disassemble --demangle --reloc --no-show-raw-insn --section=.text.acosf {} Offset 7, 200 lines modified Offset 7, 225 lines modified 7 acosf(): 7 acosf(): 8 » endbr64 8 » endbr64 9 » push···%rbp 9 » push···%rbp 10 » mov····%rsp,%rbp 10 » mov····%rsp,%rbp 11 » sub····$0x30,%rsp 11 » sub····$0x30,%rsp 12 » mov····%fs:0x28,%rax 12 » mov····%fs:0x28,%rax 13 » mov····%rax,-0x8(%rbp) 13 » mov····%rax,-0x8(%rbp) 14 » movd···%xmm0,%eax 14 » vpbroadcastd·0x0(%rip),%xmm1········ 15 » movdqa·0x0(%rip),%xmm1········ 15 ·R_X86_64_PC32» .LCPI0_0-0x4 16 ·R_X86_64_PC32» .LCPI0_0-0x4 16 » vmovd··%xmm0,%eax 17 » vpand··%xmm1,%xmm0,%xmm1 17 » pand···%xmm0,%xmm1 18 » vmovd··%xmm1,%ecx 18 » movd···%xmm1,%ecx 19 » cmp····$0x3f000000,%ecx 19 » cmp····$0x3f000000,%ecx 20 » ja·····6f·<__llvm_libc_19_1_1_::acosf(float)+0x6f> 20 » ja·····6a·<__llvm_libc_19_1_1_::acosf(float)+0x6a> 21 » cmp····$0x3a7fffff,%ecx 21 » cmp····$0x3a7fffff,%ecx 22 » ja·····c3·<__llvm_libc_19_1_1_::acosf(float)+0xc3> 22 » ja·····ab·<__llvm_libc_19_1_1_::acosf(float)+0xab> 23 » cmp····$0x328885a2,%eax 23 » cmp····$0x328885a2,%eax 24 » jg·····1e6·<__llvm_libc_19_1_1_::acosf(float)+0x1e6> 24 » jg·····225·<__llvm_libc_19_1_1_::acosf(float)+0x225> 25 » cmp····$0xb28885a3,%eax 25 » cmp····$0xb28885a3,%eax 26 » je·····2c9·<__llvm_libc_19_1_1_::acosf(float)+0x2c9> 26 » je·····2ed·<__llvm_libc_19_1_1_::acosf(float)+0x2ed> 27 » cmp····$0xb9826222,%eax 27 » cmp····$0xb9826222,%eax 28 » jne····28f·<__llvm_libc_19_1_1_::acosf(float)+0x28f> 28 » jne····2bc·<__llvm_libc_19_1_1_::acosf(float)+0x2bc> 29 » lea····0x0(%rip),%rax········ 29 » lea····0x0(%rip),%rax········ 30 ·R_X86_64_PC32» .rodata._ZN19__llvm_libc_19_1_1_L13ACOSF_EXCEPTSE+0x38 30 ·R_X86_64_PC32» .rodata._ZN19__llvm_libc_19_1_1_L13ACOSF_EXCEPTSE+0x38 31 » jmp····2d9·<__llvm_libc_19_1_1_::acosf(float)+0x2d9> 31 » jmp····2fd·<__llvm_libc_19_1_1_::acosf(float)+0x2fd> 32 » cmp····$0x3f800000,%ecx 32 » cmp····$0x3f800000,%ecx 33 » jb·····146·<__llvm_libc_19_1_1_::acosf(float)+0x146> 33 » jb·····15d·<__llvm_libc_19_1_1_::acosf(float)+0x15d> 34 » jne····208·<__llvm_libc_19_1_1_::acosf(float)+0x208> 34 » jne····247·<__llvm_libc_19_1_1_::acosf(float)+0x247> 35 » vxorps·%xmm0,%xmm0,%xmm0 35 » pxor···%xmm0,%xmm0 36 » test···%eax,%eax 36 » test···%eax,%eax 37 » jns····2b4·<__llvm_libc_19_1_1_::acosf(float)+0x2b4> 37 » jns····347·<__llvm_libc_19_1_1_::acosf(float)+0x347> 38 » movl···$0x40490fdb,-0x28(%rbp) 38 » movl···$0x40490fdb,-0x28(%rbp) 39 » vmovss·-0x28(%rbp),%xmm0 39 » movss··-0x28(%rbp),%xmm0 40 » vaddss·0x0(%rip),%xmm0,%xmm0········ 40 » addss··0x0(%rip),%xmm0········
clang doesn't have a good way to set the baseline for a build, nothing as easy as in GCC with --with-arch=<baseline>. I did that for both the armel and armhf architectures (see the arm32-defaults.diff patch), but this is unchecked for any other release architectures, and obviously wrong for some architectures. Please could you run (at least on amd64) /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --help and look for the last lines like Subdirectories of glibc-hwcaps directories, in priority order: x86-64-v4 (supported, searched) x86-64-v3 (supported, searched) x86-64-v2 (supported, searched) on the machines where you saw that?
Hi Paul, sorry for the delay in replying...! shall I (still) run this on all our amd64 build nodes (which are 4) and just attach the results to this bug report?
Hi h01ger, I assume it is still valid to do this. At least without information this bug might remain RC and I understand is hardly actionable. I guess you only need to inspect the two nodes that were used for one of the failing llvm-toolchain-18|19 runs e.g. ionos1-amd64 vs ionos15-amd64 Paul
Hi Paul, Usage: /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...] You have invoked 'ld.so', the program interpreter for dynamically-linked ELF programs. Usually, the program interpreter is invoked automatically when a dynamically-linked executable is started. [...] This program interpreter self-identifies as: /lib64/ld-linux-x86-64.so.2 Shared library search path: (libraries located via /etc/ld.so.cache) /lib/x86_64-linux-gnu (system search path) /usr/lib/x86_64-linux-gnu (system search path) /lib (system search path) /usr/lib (system search path) Subdirectories of glibc-hwcaps directories, in priority order: x86-64-v4 x86-64-v3 (supported, searched) x86-64-v2 (supported, searched) Legacy HWCAP subdirectories under library search path directories: haswell (AT_PLATFORM; supported, searched) tls (supported, searched) avx512_1 x86_64 (supported, searched) holger@ionos15-amd64:~ $ holger@ionos15-amd64:~ $ cat /proc/cpuinfo |head -28 processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 60 model name : Intel Core Processor (Haswell) stepping : 1 microcode : 0x1 cpu MHz : 2394.454 cache size : 4096 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt arat vnmi md_clear vmx flags : vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid shadow_vmcs bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit srbds mmio_unknown bhi bogomips : 4788.90 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: I don't have access to ionos1 running that CPU anymore, as IONOS asked me to migrate machines away from precicely that CPU type, so here is the same command on ionos11: holger@ionos11-amd64:~ $ /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --help Usage: /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...] [...] This program interpreter self-identifies as: /lib64/ld-linux-x86-64.so.2 Shared library search path: (libraries located via /etc/ld.so.cache) /lib/x86_64-linux-gnu (system search path) /usr/lib/x86_64-linux-gnu (system search path) /lib (system search path) /usr/lib (system search path) Subdirectories of glibc-hwcaps directories, in priority order: x86-64-v4 x86-64-v3 x86-64-v2 (supported, searched) Legacy HWCAP subdirectories under library search path directories: x86_64 (AT_PLATFORM; supported, searched) tls (supported, searched) avx512_1 x86_64 (supported, searched) holger@ionos11-amd64:~ $ holger@ionos11-amd64:~ $ cat /proc/cpuinfo |head -28 processor : 0 vendor_id : AuthenticAMD cpu family : 21 model : 1 model name : AMD Opteron 62xx class CPU stepping : 2 microcode : 0x1000065 cpu MHz : 2600.000 cache size : 512 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm cmp_legacy svm abm sse4a misalignsse 3dnowprefetch osvw ssbd ibpb vmmcall virt_ssbd arat npt nrip_save bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed ibpb_no_ret bogomips : 5200.00 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: Please note that both CPUs are vitualised (running on that actual CPU hardware).
Just making sure everybody, esp. both bugs, got the message.
just a note that if you need me to run something on those opterons, then please hurry, I converted all our opterons except one to xeon yesterday and now only have access to one opteron machine left, which I might loose next week and will quite definitly loose within the next 30 days.