Dear Maintainer,
*** Reporter, please consider answering these questions, where appropriate ***
* What led up to the situation?
Upgraded to mariadb (1:10.11.11-0+deb12u1) bookworm;
after that so did these crashes start to happen.
Previews version of mariadb in debian bookworm worked fine.
* What exactly did you do (or not do) that was effective (or
ineffective)?
No workaround so far.
This Mariadb crash happens on multiple servers with different hardware.
Uppstream repport:
https://jira.mariadb.org/projects/MDEV/issues/MDEV-35424?filter=allopenissues
The maraidb settings for the servers in this report.
thread_stack = 192K
thread_cache_size = 8
max_connections = 600
query_cache_size = 0
query_cache_type = 0
max_heap_table_size = 128M
tmp_table_size = 128M
read_buffer_size = 512K
innodb_buffer_pool_size = 12G
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 4M
innodb_flush_method = O_DIRECT
innodb_log_file_size = 1024M
innodb_log_files_in_group = 2
join_buffer_size = 2M
table_definition_cache = 10000
* What was the outcome of this action?
Crash of mariadb on first web9 server.
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61628 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61462 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61447 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61602 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61351 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61342 [Warning] InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:40:12 web9 mariadbd[3888776]: 2025-03-24 7:40:12 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 24 07:40:12 web9 mariadbd[3888776]: 250324 7:40:12 [ERROR] /usr/sbin/mariadbd got signal 6 ;
Mar 24 07:40:12 web9 mariadbd[3888776]: Sorry, we probably made a mistake, and this is a bug.
Mar 24 07:40:12 web9 mariadbd[3888776]: Your assistance in bug reporting will enable us to fix this for the next release.
Mar 24 07:40:12 web9 mariadbd[3888776]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
Mar 24 07:40:12 web9 mariadbd[3888776]: a bug on https://jira.mariadb.org/.
Mar 24 07:40:12 web9 mariadbd[3888776]: Please include the information from the server start above, to the end of the
Mar 24 07:40:12 web9 mariadbd[3888776]: information below.
Mar 24 07:40:12 web9 mariadbd[3888776]: Server version: 10.11.11-MariaDB-0+deb12u1 source revision: e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4
Mar 24 07:40:12 web9 mariadbd[3888776]: The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Mar 24 07:40:12 web9 mariadbd[3888776]: contains instructions to obtain a better version of the backtrace below.
Mar 24 07:40:12 web9 mariadbd[3888776]: Following these instructions will help MariaDB developers provide a fix quicker.
Mar 24 07:40:12 web9 mariadbd[3888776]: Attempting backtrace. Include this in the bug report.
Mar 24 07:40:12 web9 mariadbd[3888776]: (note: Retrieving this information may fail)
Mar 24 07:40:12 web9 mariadbd[3888776]: Thread pointer: 0x0
Mar 24 07:40:12 web9 mariadbd[3888776]: stack_bottom = 0x0 thread_stack 0x30000
Mar 24 07:40:13 web9 mariadbd[3888776]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5576c94642ee]
Mar 24 07:40:13 web9 mariadbd[3888776]: /usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x5576c8f42e29]
Mar 24 07:40:14 web9 mariadbd[3888776]: 2025-03-24 7:40:14 0 [Note] InnoDB: Memory pressure event freed 771915 pages
Mar 24 07:40:14 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f52b305b050]
Mar 24 07:40:14 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(+0x8aeec)[0x7f52b30a9eec]
Mar 24 07:40:14 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f52b305afb2]
Mar 24 07:40:14 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f52b3045472]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(+0x6c323b)[0x5576c8b5f23b]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(+0x6bb7fa)[0x5576c8b577fa]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic13timer_generic7executeEPv+0x38)[0x5576c93fa378]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(_ZN5tpool4task7executeEv+0x2f)[0x5576c93fac6f]
Mar 24 07:40:15 web9 mariadbd[3888776]: /usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x4f)[0x5576c93f967f]
Mar 24 07:40:16 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd44a3)[0x7f52b32d44a3]
Mar 24 07:40:17 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7f52b30a81f5]
Mar 24 07:40:17 web9 mariadbd[3888776]: /lib/x86_64-linux-gnu/libc.so.6(+0x10989c)[0x7f52b312889c]
Mar 24 07:40:17 web9 mariadbd[3888776]: Writing a core file...
Mar 24 07:40:17 web9 mariadbd[3888776]: Working directory at /var/lib/mysql
Mar 24 07:40:17 web9 mariadbd[3888776]: Resource Limits (excludes unlimited resources):
Mar 24 07:40:17 web9 mariadbd[3888776]: Limit Soft Limit Hard Limit Units
Mar 24 07:40:17 web9 mariadbd[3888776]: Max stack size 8388608 unlimited bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max core file size 0 unlimited bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max processes 256997 256997 processes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max open files 32768 32768 files
Mar 24 07:40:17 web9 mariadbd[3888776]: Max locked memory 524288 524288 bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max pending signals 256997 256997 signals
Mar 24 07:40:17 web9 mariadbd[3888776]: Max msgqueue size 819200 819200 bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max nice priority 0 0
Mar 24 07:40:17 web9 mariadbd[3888776]: Max realtime priority 0 0
Mar 24 07:40:17 web9 mariadbd[3888776]: Core pattern: core
Mar 24 07:40:17 web9 mariadbd[3888776]: Kernel version: Linux version 6.1.0-32-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.129-1 (2025-03-06)
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Failed with result 'signal'.
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Consumed 18h 2min 38.872s CPU time.
Mar 24 07:40:24 web9 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
Mar 24 07:40:24 web9 systemd[1]: Stopped mariadb.service - MariaDB 10.11.11 database server.
Mar 24 07:40:24 web9 systemd[1]: mariadb.service: Consumed 18h 2min 38.872s CPU time.
Crash of mariadb on second web8 server.
Mar 22 10:55:13 web8 mariadbd[3951656]: 2025-03-22 10:55:13 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 22 10:55:13 web8 mariadbd[3951656]: 250322 10:55:13 [ERROR] /usr/sbin/mariadbd got signal 6 ;
Mar 22 10:55:13 web8 mariadbd[3951656]: Sorry, we probably made a mistake, and this is a bug.
Mar 22 10:55:13 web8 mariadbd[3951656]: Your assistance in bug reporting will enable us to fix this for the next release.
Mar 22 10:55:13 web8 mariadbd[3951656]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
Mar 22 10:55:13 web8 mariadbd[3951656]: a bug on https://jira.mariadb.org/.
Mar 22 10:55:13 web8 mariadbd[3951656]: Please include the information from the server start above, to the end of the
Mar 22 10:55:13 web8 mariadbd[3951656]: information below.
Mar 22 10:55:13 web8 mariadbd[3951656]: Server version: 10.11.11-MariaDB-0+deb12u1 source revision: e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4
Mar 22 10:55:13 web8 mariadbd[3951656]: The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Mar 22 10:55:13 web8 mariadbd[3951656]: contains instructions to obtain a better version of the backtrace below.
Mar 22 10:55:13 web8 mariadbd[3951656]: Following these instructions will help MariaDB developers provide a fix quicker.
Mar 22 10:55:13 web8 mariadbd[3951656]: Attempting backtrace. Include this in the bug report.
Mar 22 10:55:13 web8 mariadbd[3951656]: (note: Retrieving this information may fail)
Mar 22 10:55:13 web8 mariadbd[3951656]: Thread pointer: 0x0
Mar 22 10:55:13 web8 mariadbd[3951656]: stack_bottom = 0x0 thread_stack 0x30000
Mar 22 10:55:16 web8 mariadbd[3951656]: 2025-03-22 10:55:16 0 [Note] InnoDB: Memory pressure event freed 387350 pages
Mar 22 10:55:21 web8 mariadbd[3951656]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55ed836892ee]
Mar 22 10:55:21 web8 mariadbd[3951656]: /usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x55ed83167e29]
Mar 22 10:55:25 web8 mariadbd[3951656]: 2025-03-22 10:55:25 0 [Note] InnoDB: Memory pressure event freed 90 pages
Mar 22 10:55:25 web8 mariadbd[3951656]: 2025-03-22 10:55:25 418105 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Failed with result 'signal'.
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Consumed 1d 20h 49min 7.743s CPU time.
Mar 22 10:55:32 web8 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
Mar 22 10:55:32 web8 systemd[1]: Stopped mariadb.service - MariaDB 10.11.11 database server.
Mar 22 10:55:32 web8 systemd[1]: mariadb.service: Consumed 1d 20h 49min 7.743s CPU time.
Mar 22 10:55:32 web8 systemd[1]: Starting mariadb.service - MariaDB 10.11.11 database server...
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] Starting MariaDB 10.11.11-MariaDB-0+deb12u1 source revision e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4 server_uid Xmmv8P9Fsl7fwijTPqiP1aMaxa8= as process 200075
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Number of transaction pools: 1
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Using liburing
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Initializing buffer pool, total size = 6.000GiB, chunk size = 96.000MiB
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Initialized memory pressure event listener
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Completed initialization of buffer pool
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2376864813498
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: End of log at LSN=2376872752547
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: To recover: 1923 pages
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: 128 rollback segments are active.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: log sequence number 2376872752547; transaction id 7237064055
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] Server socket created on IP: '127.0.0.1'.
Mar 22 10:55:34 web8 mariadbd[200075]: 2025-03-22 10:55:34 0 [Note] /usr/sbin/mariadbd: ready for connections.
Mar 22 10:55:34 web8 mariadbd[200075]: Version: '10.11.11-MariaDB-0+deb12u1' socket: '/run/mysqld/mysqld.sock' port: 3306 Debian 12
Mar 22 10:55:34 web8 systemd[1]: Started mariadb.service - MariaDB 10.11.11 database server.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200167]: Upgrading MySQL tables if necessary.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200170]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200190]: Checking for insecure root accounts.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200196]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Mar 22 10:55:38 web8 mariadbd[200075]: 2025-03-22 10:55:38 0 [Note] InnoDB: Memory pressure event freed 62433 pages
Mar 22 10:55:39 web8 mariadbd[200075]: 2025-03-22 10:55:39 0 [Note] InnoDB: Buffer pool(s) load completed at 250322 10:55:39
* What outcome did you expect instead?
That Mariadb did not crash.
*** End of the template - remove these template lines ***