Source: faiss
Version: 1.13.2-1
Severity: serious
X-Debbugs-CC: gcc-defaults@packages.debian.org debian-riscv@lists.debian.org
Tags: sid forky
User: debian-ci@lists.debian.org
Usertags: needs-update
Control: affects -1 src:gcc-defaults
User: debian-riscv@lists.debian.org
Usertags: riscv64

Dear maintainer(s),

With a recent upload of gcc-defaults the autopkgtest of faiss fails in
testing when that autopkgtest is run with the binary packages of
gcc-defaults from unstable. It passes when run with only packages from
testing. In tabular form:

                        pass            fail
gcc-defaults           from testing    1.233
faiss                  from testing    1.13.2-1
all others             from testing    from testing

I copied some of the output at the bottom of this report.

Currently this regression is blocking the migration of gcc-defaults to
testing [1]. Of course, gcc-defaults shouldn't just break your
autopkgtest (or even worse, your package), but it seems to me that the
change in gcc-defaults was intended and your package needs to update to
the new situation.

If this is a real problem in your package (and not only in your
autopkgtest), the right binary package(s) from gcc-defaults should
really add a versioned Breaks on the unfixed version of (one of your)
package(s). Note: the Breaks is nice even if the issue is only in the
autopkgtest as it helps the migration software to figure out the right
versions to combine in the tests.

More information about this bug and the reason for filing it can be found on
https://wiki.debian.org/ContinuousIntegration/RegressionEmailInformation

Paul

[1] https://qa.debian.org/excuses.php?package=gcc-defaults

https://ci.debian.net/data/autopkgtest/testing/riscv64/f/faiss/74133117/log.gz

7572s test 119
7572s         Start 119: HNSWTest.TEST_search_from_candidate_unbounded
7572s 7572s 119: Test command:
/tmp/autopkgtest.Eu4j7u/autopkgtest_tmp/tests/build/faiss_test
"--gtest_filter=HNSWTest.TEST_search_from_candidate_unbounded"
"--gtest_also_run_disabled_tests"
7572s 119: Working Directory:
/tmp/autopkgtest.Eu4j7u/autopkgtest_tmp/tests/build
7572s 119: Test timeout computed to be: 10000000
7572s 119: WARNING clustering 1000 points to 40 centroids: please
provide at least 1560 training points
7572s 119: Running main() from ./googletest/src/gtest_main.cc
7572s 119: Note: Google Test filter =
HNSWTest.TEST_search_from_candidate_unbounded
7572s 119: [==========] Running 1 test from 1 test suite.
7572s 119: [----------] Global test environment set-up.
7572s 119: [----------] 1 test from HNSWTest
7572s 119: [ RUN      ] HNSWTest.TEST_search_from_candidate_unbounded
7572s 119/133 Test #119: HNSWTest.TEST_search_from_candidate_unbounded
.....................***Exception: SegFault  0.14 sec


7577s 7577s 99% tests passed, 1 tests failed out of 133
7577s 7577s Total Test time (real) = 444.00 sec
7577s 7577s The following tests FAILED:
7577s 	119 - HNSWTest.TEST_search_from_candidate_unbounded (SEGFAULT)
7577s Errors while running CTest

Hmm
I forgot to explicitly mention: on riscv64.

Paul

Hi,

While the test seemed to fail reliably on the debci host, I wasn't able
to reproduce the problem. After giving it back, it succeeded [1] with
gcc-16, so it's not a regression, but rather an unreliable test.

I also noticed that the autopkgtest time varies quite a lot (up to a
factor 3 for a given debci runner), so I guess the high level of
concurrent autopkgtests job following the gcc-default upload explains
that.

I have therefore tried to reproduce the issue under high load, and I
have been successful. Here is the corresponding backtrace:

| GNU gdb (Debian 17.2-1+b1) 17.2
| Copyright (C) 2025 Free Software Foundation, Inc.
| License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
| This is free software: you are free to change and redistribute it.
| There is NO WARRANTY, to the extent permitted by law.
| Type "show copying" and "show warranty" for details.
| This GDB was configured as "riscv64-linux-gnu".
| Type "show configuration" for configuration details.
| For bug reporting instructions, please see:
| <https://www.gnu.org/software/gdb/bugs/>.
| Find the GDB manual and other documentation resources online at:
|     <http://www.gnu.org/software/gdb/documentation/>.
|
| For help, type "help".
| Type "apropos word" to search for commands related to "word"...
| Reading symbols from /home/aurel32/faiss/autopkgtest_tmp/tests/build/faiss_test...
| (No debugging symbols found in /home/aurel32/faiss/autopkgtest_tmp/tests/build/faiss_test)
| [New LWP 16849]
| [New LWP 16848]
| [New LWP 16850]
| [New LWP 16851]
| [Thread debugging using libthread_db enabled]
| Using host libthread_db library "/usr/lib/riscv64-linux-gnu/libthread_db.so.1".
| Core was generated by `/home/aurel32/faiss/autopkgtest_tmp/tests/build/faiss_test --gtest_filter=HNSWT'.
| Program terminated with signal SIGSEGV, Segmentation fault.
| #0  0x0000002ad7aeb6c2 in faiss::fvec_L2sqr_batch_4(float const*, float const*, float const*, float const*, float const*, unsigned long, float&, float&, float&, float&) ()
| [Current thread is 1 (Thread 0x3faf4420e0 (LWP 16849))]
| (gdb) bt
| #0  0x0000002ad7aeb6c2 in faiss::fvec_L2sqr_batch_4(float const*, float const*, float const*, float const*, float const*, unsigned long, float&, float&, float&, float&) ()
| #1  0x0000002ad7823f0c in faiss::(anonymous namespace)::FlatL2Dis::distances_batch_4(long, long, long, long, float&, float&, float&, float&) ()
| #2  0x0000002ad79014f6 in faiss::search_neighbors_to_add(faiss::HNSW&, faiss::DistanceComputer&, std::priority_queue<faiss::HNSW::NodeDistCloser, std::vector<faiss::HNSW::NodeDistCloser, std::allocator<faiss::HNSW::NodeDistCloser> >, std::
| less<faiss::HNSW::NodeDistCloser> >&, int, float, int, faiss::VisitedTable&, bool) ()
| #3  0x0000002ad7902f94 in faiss::HNSW::add_links_starting_from(faiss::DistanceComputer&, int, int, float, int, omp_lock_t*, faiss::VisitedTable&, bool) ()
| #4  0x0000002ad79033d2 in faiss::HNSW::add_with_locks(faiss::DistanceComputer&, int, int, std::vector<omp_lock_t, std::allocator<omp_lock_t> >&, faiss::VisitedTable&, bool) ()
| #5  0x0000002ad78498b0 in faiss::(anonymous namespace)::hnsw_add_vertices(faiss::IndexHNSW&, unsigned long, unsigned long, float const*, bool, bool) [clone ._omp_fn.0] ()
| #6  0x0000003fafeeff3e in ?? () from /usr/lib/riscv64-linux-gnu/libgomp.so.1
| #7  0x0000003faf64e330 in ?? () from /usr/lib/riscv64-linux-gnu/libc.so.6
| #8  0x0000003faf69b670 in ?? () from /usr/lib/riscv64-linux-gnu/libc.so.6
| (gdb) disassemble
| Dump of assembler code for function _ZN5faiss18fvec_L2sqr_batch_4EPKfS1_S1_S1_S1_mRfS2_S2_S2_:
|    0x0000002ad7aeb59e <+0>:     ld      t2,0(sp)
|    0x0000002ad7aeb5a0 <+2>:     beqz    a5,0x2ad7aeb7e0 <_ZN5faiss18fvec_L2sqr_batch_4EPKfS1_S1_S1_S1_mRfS2_S2_S2_+578>
|    0x0000002ad7aeb5a4 <+6>:     slli    a5,a5,0x2
|    0x0000002ad7aeb5a6 <+8>:     addi    t3,a5,-4
|    0x0000002ad7aeb5aa <+12>:    fmv.w.x fa5,zero
|    0x0000002ad7aeb5ae <+16>:    srli    t0,t3,0x2
|    0x0000002ad7aeb5b2 <+20>:    addi    t1,t0,1
|    0x0000002ad7aeb5b6 <+24>:    andi    t5,t1,3
|    0x0000002ad7aeb5ba <+28>:    fmv.s   ft5,fa5
|    0x0000002ad7aeb5be <+32>:    fmv.s   ft3,fa5
|    0x0000002ad7aeb5c2 <+36>:    fmv.s   fa4,fa5
|    0x0000002ad7aeb5c6 <+40>:    li      t1,0
|    0x0000002ad7aeb5c8 <+42>:    beqz    t5,0x2ad7aeb6a6 <_ZN5faiss18fvec_L2sqr_batch_4EPKfS1_S1_S1_S1_mRfS2_S2_S2_+264>
|    0x0000002ad7aeb5cc <+46>:    li      t4,1
|    0x0000002ad7aeb5ce <+48>:    beq     t5,t4,0x2ad7aeb658 <_ZN5faiss18fvec_L2sqr_batch_4EPKfS1_S1_S1_S1_mRfS2_S2_S2_+186>
|    0x0000002ad7aeb5d2 <+52>:    li      t6,2
|    0x0000002ad7aeb5d4 <+54>:    beq     t5,t6,0x2ad7aeb60e <_ZN5faiss18fvec_L2sqr_batch_4EPKfS1_S1_S1_S1_mRfS2_S2_S2_+112>
|    0x0000002ad7aeb5d8 <+58>:    flw     fa3,0(a0)
|    0x0000002ad7aeb5dc <+62>:    flw     fa4,0(a1)
|    0x0000002ad7aeb5e0 <+66>:    flw     ft1,0(a2)
|    0x0000002ad7aeb5e4 <+70>:    flw     ft0,0(a3)
|    0x0000002ad7aeb5e8 <+74>:    flw     fa2,0(a4)
|    0x0000002ad7aeb5ec <+78>:    fsub.s  ft2,fa3,fa4
|    0x0000002ad7aeb5f0 <+82>:    fsub.s  ft3,fa3,ft1
|    0x0000002ad7aeb5f4 <+86>:    fsub.s  ft4,fa3,ft0
|    0x0000002ad7aeb5f8 <+90>:    fsub.s  ft6,fa3,fa2
|    0x0000002ad7aeb5fc <+94>:    li      t1,4
|    0x0000002ad7aeb5fe <+96>:    fmadd.s fa4,ft2,ft2,fa5
|    0x0000002ad7aeb602 <+100>:   fmadd.s ft3,ft3,ft3,fa5
|    0x0000002ad7aeb606 <+104>:   fmadd.s ft5,ft4,ft4,fa5
|    0x0000002ad7aeb60a <+108>:   fmadd.s fa5,ft6,ft6,fa5
|    0x0000002ad7aeb60e <+112>:   add     t0,a0,t1
|    0x0000002ad7aeb612 <+116>:   add     t3,a1,t1
|    0x0000002ad7aeb616 <+120>:   add     t5,a2,t1
|    0x0000002ad7aeb61a <+124>:   add     t4,a3,t1
|    0x0000002ad7aeb61e <+128>:   add     t6,a4,t1
|    0x0000002ad7aeb622 <+132>:   flw     ft7,0(t0)
|    0x0000002ad7aeb626 <+136>:   flw     fa0,0(t3)
|    0x0000002ad7aeb62a <+140>:   flw     fa1,0(t5)
|    0x0000002ad7aeb62e <+144>:   flw     fa6,0(t4)
|    0x0000002ad7aeb632 <+148>:   flw     fa7,0(t6)
|    0x0000002ad7aeb636 <+152>:   fsub.s  ft8,ft7,fa0
|    0x0000002ad7aeb63a <+156>:   fsub.s  ft9,ft7,fa1
|    0x0000002ad7aeb63e <+160>:   fsub.s  ft10,ft7,fa6
|    0x0000002ad7aeb642 <+164>:   fsub.s  ft11,ft7,fa7
|    0x0000002ad7aeb646 <+168>:   addi    t1,t1,4
|    0x0000002ad7aeb648 <+170>:   fmadd.s fa4,ft8,ft8,fa4
|    0x0000002ad7aeb64c <+174>:   fmadd.s ft3,ft9,ft9,ft3
|    0x0000002ad7aeb650 <+178>:   fmadd.s ft5,ft10,ft10,ft5
|    0x0000002ad7aeb654 <+182>:   fmadd.s fa5,ft11,ft11,fa5
|    0x0000002ad7aeb658 <+186>:   add     t0,a0,t1
|    0x0000002ad7aeb65c <+190>:   add     t3,a1,t1
|    0x0000002ad7aeb660 <+194>:   add     t5,a2,t1
|    0x0000002ad7aeb664 <+198>:   add     t4,a3,t1
|    0x0000002ad7aeb668 <+202>:   add     t6,a4,t1
|    0x0000002ad7aeb66c <+206>:   flw     fa3,0(t0)
|    0x0000002ad7aeb670 <+210>:   flw     ft1,0(t3)
|    0x0000002ad7aeb674 <+214>:   flw     ft2,0(t5)
|    0x0000002ad7aeb678 <+218>:   flw     fa2,0(t4)
|    0x0000002ad7aeb67c <+222>:   flw     ft0,0(t6)
|    0x0000002ad7aeb680 <+226>:   fsub.s  ft4,fa3,ft1
|    0x0000002ad7aeb684 <+230>:   fsub.s  ft6,fa3,ft2
|    0x0000002ad7aeb688 <+234>:   fsub.s  ft7,fa3,fa2
|    0x0000002ad7aeb68c <+238>:   fsub.s  fa0,fa3,ft0
|    0x0000002ad7aeb690 <+242>:   addi    t1,t1,4
|    0x0000002ad7aeb692 <+244>:   fmadd.s fa4,ft4,ft4,fa4
|    0x0000002ad7aeb696 <+248>:   fmadd.s ft3,ft6,ft6,ft3
|    0x0000002ad7aeb69a <+252>:   fmadd.s ft5,ft7,ft7,ft5
|    0x0000002ad7aeb69e <+256>:   fmadd.s fa5,fa0,fa0,fa5
|    0x0000002ad7aeb6a2 <+260>:   beq     a5,t1,0x2ad7aeb7cc <_ZN5faiss18fvec_L2sqr_batch_4EPKfS1_S1_S1_S1_mRfS2_S2_S2_+558>
|    0x0000002ad7aeb6a6 <+264>:   add     t0,a0,t1
|    0x0000002ad7aeb6aa <+268>:   add     t3,a1,t1
|    0x0000002ad7aeb6ae <+272>:   flw     fa6,0(t0)
|    0x0000002ad7aeb6b2 <+276>:   flw     fa7,0(t3)
|    0x0000002ad7aeb6b6 <+280>:   add     t5,a2,t1
|    0x0000002ad7aeb6ba <+284>:   add     t6,a4,t1
|    0x0000002ad7aeb6be <+288>:   flw     ft8,0(t5)
| => 0x0000002ad7aeb6c2 <+292>:   flw     ft9,0(t6)
|    0x0000002ad7aeb6c6 <+296>:   add     t4,a3,t1
|    0x0000002ad7aeb6ca <+300>:   fsub.s  ft10,fa6,fa7
|    0x0000002ad7aeb6ce <+304>:   flw     fa1,0(t4)
|    0x0000002ad7aeb6d2 <+308>:   add     t3,a0,t1
|    0x0000002ad7aeb6d6 <+312>:   add     t5,a1,t1
|    0x0000002ad7aeb6da <+316>:   flw     fa0,4(t3)
|    0x0000002ad7aeb6de <+320>:   fsub.s  ft11,fa6,ft8
|    0x0000002ad7aeb6e2 <+324>:   fsub.s  fa3,fa6,ft9
|    0x0000002ad7aeb6e6 <+328>:   flw     fa2,4(t5)
|    0x0000002ad7aeb6ea <+332>:   fsub.s  ft7,fa6,fa1
|    0x0000002ad7aeb6ee <+336>:   add     t6,a2,t1
|    0x0000002ad7aeb6f2 <+340>:   add     t4,a3,t1
|    0x0000002ad7aeb6f6 <+344>:   add     t0,a4,t1
|    0x0000002ad7aeb6fa <+348>:   fmadd.s ft2,ft10,ft10,fa4
|    0x0000002ad7aeb6fe <+352>:   flw     ft1,4(t6)
|    0x0000002ad7aeb702 <+356>:   fsub.s  fa4,fa0,fa2
|    0x0000002ad7aeb706 <+360>:   flw     ft0,4(t4)
|    0x0000002ad7aeb70a <+364>:   flw     ft6,4(t0)
|    0x0000002ad7aeb70e <+368>:   add     t5,a0,t1
|    0x0000002ad7aeb712 <+372>:   add     t6,a1,t1
|    0x0000002ad7aeb716 <+376>:   fmadd.s ft4,ft11,ft11,ft3
|    0x0000002ad7aeb71a <+380>:   fmadd.s ft10,fa3,fa3,fa5
|    0x0000002ad7aeb71e <+384>:   flw     ft11,8(t6)
|    0x0000002ad7aeb722 <+388>:   flw     fa5,8(t5)
|    0x0000002ad7aeb726 <+392>:   fmadd.s ft3,ft7,ft7,ft5
|    0x0000002ad7aeb72a <+396>:   fsub.s  ft9,fa0,ft6
|    0x0000002ad7aeb72e <+400>:   fsub.s  fa6,fa0,ft1
|    0x0000002ad7aeb732 <+404>:   fsub.s  fa7,fa0,ft0
|    0x0000002ad7aeb736 <+408>:   add     t0,a2,t1
|    0x0000002ad7aeb73a <+412>:   add     t4,a3,t1
|    0x0000002ad7aeb73e <+416>:   add     t3,a4,t1
|    0x0000002ad7aeb742 <+420>:   fmadd.s ft8,fa4,fa4,ft2
|    0x0000002ad7aeb746 <+424>:   fsub.s  fa3,fa5,ft11
|    0x0000002ad7aeb74a <+428>:   flw     fa1,8(t0)
|    0x0000002ad7aeb74e <+432>:   flw     ft7,8(t4)
|    0x0000002ad7aeb752 <+436>:   flw     ft5,8(t3)
|    0x0000002ad7aeb756 <+440>:   fmadd.s fa4,ft9,ft9,ft10
|    0x0000002ad7aeb75a <+444>:   fmadd.s ft1,fa6,fa6,ft4
|    0x0000002ad7aeb75e <+448>:   fsub.s  ft6,fa5,fa1
|    0x0000002ad7aeb762 <+452>:   fmadd.s ft0,fa7,fa7,ft3
|    0x0000002ad7aeb766 <+456>:   fsub.s  fa2,fa5,ft7
|    0x0000002ad7aeb76a <+460>:   fsub.s  fa6,fa5,ft5
|    0x0000002ad7aeb76e <+464>:   add     t0,a0,t1
|    0x0000002ad7aeb772 <+468>:   add     t6,a1,t1
|    0x0000002ad7aeb776 <+472>:   add     t5,a2,t1
|    0x0000002ad7aeb77a <+476>:   add     t4,a3,t1
|    0x0000002ad7aeb77e <+480>:   add     t3,a4,t1
|    0x0000002ad7aeb782 <+484>:   flw     fa7,12(t0)
|    0x0000002ad7aeb786 <+488>:   fmadd.s fa0,fa3,fa3,ft8
|    0x0000002ad7aeb78a <+492>:   flw     ft3,12(t3)
|    0x0000002ad7aeb78e <+496>:   flw     ft4,12(t6)
|    0x0000002ad7aeb792 <+500>:   flw     ft8,12(t5)
|    0x0000002ad7aeb796 <+504>:   flw     ft2,12(t4)
|    0x0000002ad7aeb79a <+508>:   fmadd.s fa5,fa6,fa6,fa4
|    0x0000002ad7aeb79e <+512>:   fsub.s  fa3,fa7,ft3
|    0x0000002ad7aeb7a2 <+516>:   fsub.s  ft9,fa7,ft4
|    0x0000002ad7aeb7a6 <+520>:   fmadd.s ft10,ft6,ft6,ft1
|    0x0000002ad7aeb7aa <+524>:   fsub.s  ft11,fa7,ft8
|    0x0000002ad7aeb7ae <+528>:   fmadd.s fa1,fa2,fa2,ft0
|    0x0000002ad7aeb7b2 <+532>:   fsub.s  ft7,fa7,ft2
|    0x0000002ad7aeb7b6 <+536>:   addi    t1,t1,16
|    0x0000002ad7aeb7b8 <+538>:   fmadd.s fa4,ft9,ft9,fa0
|    0x0000002ad7aeb7bc <+542>:   fmadd.s ft3,ft11,ft11,ft10
|    0x0000002ad7aeb7c0 <+546>:   fmadd.s fa5,fa3,fa3,fa5
|    0x0000002ad7aeb7c4 <+550>:   fmadd.s ft5,ft7,ft7,fa1
|    0x0000002ad7aeb7c8 <+554>:   bne     a5,t1,0x2ad7aeb6a6 <_ZN5faiss18fvec_L2sqr_batch_4EPKfS1_S1_S1_S1_mRfS2_S2_S2_+264>
|    0x0000002ad7aeb7cc <+558>:   fsw     fa4,0(a6)
|    0x0000002ad7aeb7d0 <+562>:   ld      a0,8(sp)
|    0x0000002ad7aeb7d2 <+564>:   fsw     ft3,0(a7)
|    0x0000002ad7aeb7d6 <+568>:   fsw     ft5,0(t2)
|    0x0000002ad7aeb7da <+572>:   fsw     fa5,0(a0)
|    0x0000002ad7aeb7de <+576>:   ret
|    0x0000002ad7aeb7e0 <+578>:   fmv.w.x fa5,zero
|    0x0000002ad7aeb7e4 <+582>:   ld      a0,8(sp)
|    0x0000002ad7aeb7e6 <+584>:   fmv.s   fa4,fa5
|    0x0000002ad7aeb7ea <+588>:   fmv.s   ft3,fa5
|    0x0000002ad7aeb7ee <+592>:   fmv.s   ft5,fa5
|    0x0000002ad7aeb7f2 <+596>:   fsw     fa4,0(a6)
|    0x0000002ad7aeb7f6 <+600>:   fsw     ft3,0(a7)
|    0x0000002ad7aeb7fa <+604>:   fsw     ft5,0(t2)
|    0x0000002ad7aeb7fe <+608>:   fsw     fa5,0(a0)
|    0x0000002ad7aeb802 <+612>:   ret

At this stage, I haven't debugged it further. I'll try to see if I can
reproduce the issue on another architecture. Given it's not a
regression, I guess the severity can be decreased.

Regards
Aurelien

[1] https://ci.debian.net/packages/f/faiss/testing/riscv64/74151428/