- Package:
- mariadb-server-10.6
- Source:
- mariadb-10.6
- Description:
- MariaDB database server binaries
- Submitter:
- Daniel Lewart
- Date:
- 2025-05-08 08:15:01 UTC
- Severity:
- important
Debian MariaDB Maintainers, Live Build (--distribution testing) image from April 2022. 8 GB RAM, so should be plenty of disk space available. $ sudo apt install mariadb-server-10.6-dbgsym ... Setting up mariadb-server-10.6 (1:10.6.7-3) ... Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /lib/systemd/system/mariadb.service. Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. ... $ sudo journalctl | grep -i mariadb | cut -c58- [See below] Perhaps this is similar to: #983002 - plocate: does not work with /var on overlayfs https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983002 Please let me know any additional information that you could use. Thank you! Daniel Lewart Urbana, Illinois --- 2022-04-10 22:53:02 0 [Warning] InnoDB: Retry attempts for writing partial data failed. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Write to file ./ibdata1 failed at offset 0, 1048576 bytes should have been written, only 0 were written. Operating system error number 22. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument' 2022-04-10 22:53:02 0 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space 2022-04-10 22:53:02 0 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 22 in a file operation. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument' 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation 220410 22:53:02 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.6.7-MariaDB-3 key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467957 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation Printing to addr2line failed /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x555d0aa77f0e] /usr/sbin/mariadbd(handle_fatal_signal+0x478)[0x555d0a5c5268] 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation /lib/x86_64-linux-gnu/libpthread.so.0(+0x12200)[0x7f606ba96200] 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f606b57a8a1] /lib/x86_64-linux-gnu/libc.so.6(abort+0x112)[0x7f606b564546] 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation /usr/sbin/mariadbd(+0x63b8e9)[0x555d0a2638e9] /usr/sbin/mariadbd(+0xcb66e5)[0x555d0a8de6e5] /usr/sbin/mariadbd(+0xdbc308)[0x555d0a9e4308] /usr/sbin/mariadbd(+0xdbc371)[0x555d0a9e4371] /usr/sbin/mariadbd(+0xdbea20)[0x555d0a9e6a20] /usr/sbin/mariadbd(+0xdbfb41)[0x555d0a9e7b41] /usr/sbin/mariadbd(+0xd2585f)[0x555d0a94d85f] /usr/sbin/mariadbd(+0xc64245)[0x555d0a88c245] /usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x555d0a5c839e] /usr/sbin/mariadbd(+0x797a76)[0x555d0a3bfa76] /usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x828)[0x555d0a3c0a38] /usr/sbin/mariadbd(+0x6c367a)[0x555d0a2eb67a] /usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x404)[0x555d0a2f1224] 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f606b5657fd] 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation. 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation /usr/sbin/mariadbd(_start+0x2a)[0x555d0a2e680a] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 0 bytes Max resident set unlimited unlimited bytes Max processes 29195 29195 processes Max open files 32190 32190 files Max locked memory 976257024 976257024 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 29195 29195 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core /usr/bin/mysql_install_db: line 541: 7508 Aborted "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup "--basedir=$basedir" "--datadir=$ldata" --log-warnings=0 --enforce-storage-engine="" "--plugin-dir=${plugindir}" $args --max_allowed_packet=8M --net_buffer_length=16K Installation of system tables failed! Examine the logs in /var/lib/mysql for more information. The problem could be conflicting information in an external my.cnf files. You can ignore these by doing: shell> /usr/bin/mysql_install_db --defaults-file=~/.my.cnf You can also try to start the mysqld daemon with: shell> /usr/sbin/mariadbd --skip-grant-tables --general-log & and use the command line tool /usr/bin/mariadb to connect to the mysql database and look at the grant tables: shell> /usr/bin/mysql -u root mysql mysql> show tables; Try 'mysqld --help' if you have problems with paths. Using --general-log gives you a log in /var/lib/mysql that may be helpful. The latest information about mysql_install_db is available at https://mariadb.com/kb/en/installing-system-tables-mysql_install_db You can find the latest source at https://downloads.mariadb.org and the maria-discuss email list at https://launchpad.net/~maria-discuss Please check all of the above before submitting a bug report at https://mariadb.org/jira ###
Hello. The apt install of mariadb-server-10.6 fails for me too, with the same line "Could not execute systemctl" but different journal messages. I removed the existing setup in /etc and /var, purged dependencies, but install still fails. How do I debug? systemd[1]: Starting MariaDB 10.6.7 database server... [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-3+b1) starting as process 98614 ... [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created! [Note] InnoDB: Compressed tables use zlib 1.2.11 [Note] InnoDB: Number of pools: 1 [Note] InnoDB: Using crc32 + pclmulqdq instructions [Note] InnoDB: Using liburing [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 [Note] InnoDB: Completed initialization of buffer pool [Note] InnoDB: Setting file './ibdata1' size to 12 MB. Physically writing the file full; Please wait ... [Note] InnoDB: File './ibdata1' size is now 12 MB. [Note] InnoDB: Setting log file ./ib_logfile101 size to 100663296 bytes [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 [Note] InnoDB: New log file created, LSN=10329 [Note] InnoDB: Doublewrite buffer not found: creating new [Note] InnoDB: Doublewrite buffer created [Note] InnoDB: 128 rollback segments are active. [Note] InnoDB: Creating shared tablespace for temporary tables [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... [Note] InnoDB: File './ibtmp1' size is now 12 MB. [Note] InnoDB: 10.6.7 started; log sequence number 0; transaction id 3 [Note] Plugin 'FEEDBACK' is disabled. [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work. [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist [Note] Server socket created on IP: '127.0.0.1'. [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist [ERROR] Aborting Warning: Memory not freed: 280 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE systemd[1]: mariadb.service: Failed with result 'exit-code'. systemd[1]: Failed to start MariaDB 10.6.7 database server.
Hi Daniel! Thanks for your detailed report. I am not able to reproduce this with the latest mariadb-server-10.6-dbgsym version (10.6.8). Can you try with the latest version and tell me? Also, can you describe a bit better your setup? If this is a filesystem related issue, I would like to try to reproduce it myself. I have tested with a systemd container that I have created for these kind of testing, here are the commands (using podman): | podman run --name sys-test --rm -d fauust/docker-systemd:debian-sid | podman exec -it sys-test bash -c "echo \"deb http://deb.debian.org/debian-debug/ sid-debug main\" >> /etc/apt/sources.list && apt update && apt install -y mariadb-server-10.6-dbgsym" Cheers! Daniel Lewart <lewart3@gmail.com>, 11/04/2022 – 00:20:00 (-0500):
Hi Denys, I am not sure that the problem is the same, in your case it seems that the datadir does not exist or need to be recreated. If you are sure that nothing can be lost (or that you have backups), you should try the following: | sudo systemctl stop mariadb | sudo rm -rf /var/lib/mysql/* | sudo -u mysql mariadb-install-db | Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... | 2022-05-23 14:18:53 0 [Warning] mariadbd: io_uring_queue_init() failed with errno 1 | 2022-05-23 14:18:53 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF | 2022-05-23 14:18:53 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work. | OK | ... | sudo systemctl start mariadb Regards, Denys Nykula <vegan@libre.net.ua>, 18/05/2022 – 17:16:00 (+0300):
Thanks Faustin, something was fixed between 10.6.7-3 and 10.6.8-1 that made mariadb-install-db succeed for me since today morning. With the March packages it kept failing every time during or after the InnoDB initialization (as I mentioned, I started clean every time, removing the mysql directories in /etc and /var).
Faustin,
Yes, I tried 10.6.8-1 with the same result.
Yes. This is a simplified version of my setup ...
1) sudo apt -y install apt-cacher-ng live-build
2) I used Live Build to create a live sid image:
#!/bin/sh
cd /dev/shm
sudo mount /dev/shm -odev,exec,remount,size=3G
lb config --apt-http-proxy http://localhost:3142 -d sid
echo '! Packages Priority standard' \
> config/package-lists/standard.list.chroot
echo 'deb http://deb.debian.org/debian-debug sid-debug main' \
> config/archives/live.list.binary
sudo lb build
3) qemu-system-x86_64 -enable-kvm -cdrom /dev/shm/live-image-amd64.hybrid.iso \
-cpu host -m 4G -smp 2
4) In the guest, it didn't matter what I tried:
sudo apt -y install mariadb-server
sudo apt -y install mariadb-server-10.6-dbgsym
mariadb-server-core-10.6-dbgsym
I'm sure it is an overlay filesystem issue:
$ sudo ls -l /var/lib/mysql/ibdata1
-rw-rw---- 1 mysql mysql 0 May 25 12:34 /var/lib/mysql/ibdata1
$ df -hT /var/lib/mysql
Filesystem Type Size Used Avail Use% Mounted on
overlay overlay 2.0G 355M 1.6G 19% /
$ findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "overlay",
"fstype": "overlay",
"options":
"rw,noatime,lowerdir=/run/live/rootfs/filesystem.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work"
}
]
}
I will give this a try. I'm curious what filesystem the container uses.
Thank you!
Dan
Urbana, IL
Faustin,
DL> I will give this a try. I'm curious what filesystem the container uses.
$ podman exec -it sys-test findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "fuse-overlayfs",
"fstype": "fuse.fuse-overlayfs",
"options":
"rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other"
}
]
}
Dan
Urbana, Illinois
tags + upstream
Faustin,
I found a much simpler way to demonstrate my problem with
a Debian Bullseye Live image and MariaDB generic Linux server:
* debian-live-11.3.0-amd64-standard.iso
* mariadb-10.6.8-linux-systemd-x86_64.tar.gz
I decided to file a new bug upstream:
[DEV-28751] mariadb-install-db fails writing ibdata1 on overlayfs
https://jira.mariadb.org/browse/MDEV-28751
Also, strace shows a difference between 10.5.16 (good) and 10.6.8 (fails).
Merci beaucoup,
Dan
Urbana, Illinois
Hi Daniel! Thank you very much for your detailed report, your tests and to have filled a jira issue! I am now also subscribed to MDEV-28751 and will talk with Daniel Black to see how we can debug this and maybe check if this is an interesting step to add in our CI system. Cheers!
Hi! I don't see anything new in upstream https://jira.mariadb.org/browse/MDEV-28751 about this. However we do have MariaDB 10.11.1 in Debian now. Maybe you Daniel can for the sake of it check if the behaviour is still same on 10.11? I would personally also be keen to figure out how to make the MariaDB datadir be able to run off overlayfs or something similar so I can implement my long time idea to have shadow upgrades where the old mariadbd and old data is kept running and intact, and every new version mariadbd would first run off overlayfs with real data to simulate the upgrade before it is done for real.
Otto, et al,
I created a live standard image with CODENAME=unstable:
$ uname -a
Linux debian 6.1.0-2-amd64 #1 SMP PREEMPT_DYNAMIC
Debian 6.1.7-1 (2023-01-18) x86_64 GNU/Linux
Then I did the following:
$ sudo apt -y install mariadb-server
Output included:
Setting up mariadb-server (1:10.11.1-1) ...
Created symlink
/etc/systemd/system/multi-user.target.wants/mariadb.service →
/lib/systemd/system/mariadb.service.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Below is the output from the systemd journal.
In the upstream bug:
Daniel Black added a comment - 2022-06-07 04:50
I'll be just pushing this to the kernel (overlayfs) developers.
I don't know what happened with that.
The good news is that Arnaud R's upstream "innodb_flush_method = fsync"
workaround still works.
Sorry!
Daniel Lewart
Urbana, Illinois
---
$ sudo journalctl -t mariadb-server.postinst | cut -c55-
2023-01-30 0:54:27 0 [Warning] InnoDB: Retry attempts for writing
partial data failed.
2023-01-30 0:54:27 0 [ERROR] InnoDB: Write to file ./ibdata1 failed
at offset 0, 1048576 bytes should have been written, only 0 were
written. Operating system error number 22. Check that your OS and file
system support files of this size. Check also that the disk is not
full or a disk quota exceeded.
2023-01-30 0:54:27 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2023-01-30 0:54:27 0 [ERROR] InnoDB: Could not set the file size of
'./ibdata1'. Probably out of disk space
2023-01-30 0:54:27 0 [ERROR] InnoDB: Database creation was aborted
with error Generic error. You may need to delete the ibdata1 file
before trying to start up again.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 22
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 222. Cannot continue operation
230130 0:54:28 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.11.1-MariaDB-1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
468018 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
Printing to addr2line failed
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55ba61bd403e]
/usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x55ba61741c69]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libc.so.6(+0x3bf90)[0x7f028b392f90]
/lib/x86_64-linux-gnu/libc.so.6(+0x8accc)[0x7f028b3e1ccc]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f028b392ef2]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f028b37d472]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
/usr/sbin/mariadbd(+0x6692e1)[0x55ba613972e1]
/usr/sbin/mariadbd(+0xd0a775)[0x55ba61a38775]
/usr/sbin/mariadbd(+0xe140c8)[0x55ba61b420c8]
/usr/sbin/mariadbd(+0xe14131)[0x55ba61b42131]
/usr/sbin/mariadbd(+0xe16060)[0x55ba61b44060]
/usr/sbin/mariadbd(+0xe17041)[0x55ba61b45041]
/usr/sbin/mariadbd(+0xd7bbae)[0x55ba61aa9bae]
/usr/sbin/mariadbd(+0xcb7545)[0x55ba619e5545]
/usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x55ba61744e6e]
/usr/sbin/mariadbd(+0x802167)[0x55ba61530167]
/usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x85a)[0x55ba6153112a]
/usr/sbin/mariadbd(+0x6f42e0)[0x55ba614222e0]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x41d)[0x55ba61427e7d]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libc.so.6(+0x2718a)[0x7f028b37e18a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f028b37e245]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
/usr/sbin/mariadbd(_start+0x21)[0x55ba6141ceb1]
The manual page at
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 7693 7693 processes
Max open files 32184 32184 files
Max locked memory 258789376 258789376 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 7693 7693 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
Kernel version: Linux version 6.1.0-2-amd64
(debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0,
GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian
6.1.7-1 (2023-01-18)
/usr/bin/mysql_install_db: line 554: 2096 Aborted
"$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt"
--bootstrap $silent_startup "--basedir=$basedir" "--datadir=$ldata"
--log-warnings=0 --enforce-storage-engine=""
"--plugin-dir=${plugindir}" $args --max_allowed_packet=8M
--net_buffer_length=16K
Installation of system tables failed! Examine the logs in
/var/lib/mysql for more information.
The problem could be conflicting information in an external
my.cnf files. You can ignore these by doing:
shell> /usr/bin/mysql_install_db --defaults-file=~/.my.cnf
You can also try to start the mysqld daemon with:
shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &
and use the command line tool /usr/bin/mariadb
to connect to the mysql database and look at the grant tables:
shell> /usr/bin/mysql -u root mysql
mysql> show tables;
Try 'mysqld --help' if you have problems with paths. Using
--general-log gives you a log in /var/lib/mysql that may be helpful.
The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
You can find the latest source at https://downloads.mariadb.org and
the maria-discuss email list at https://launchpad.net/~maria-discuss
Please check all of the above before submitting a bug report
at https://mariadb.org/jira
###
Otto, et al,
How about temporarily inserting something like the implementation below of
Arnaud R's workaround somewhere (where?) in mariadb-server(-10.6).postinst?
Thank you!
Daniel Lewart
Urbana, Illinois
---
# Temporary workaround which should be removed after upstream fixes
# https://jira.mariadb.org/browse/MDEV-28751
fstype=$(findmnt -n -o FSTYPE -T /var/lib/mysql)
if [ "$fstype" = overlay ]; then
cat <<- EOF > "${mysql_cfgdir:-/etc/mysql}/mariadb.conf.d/90-live.cnf"
[mysqld]
innodb_flush_method = fsync
EOF
fi
Maybe just call it 90-overlayfs.cnf? But I don't think this should be part of the maintainer scripts, maybe something else? Feel free to open MR if you have a concrete suggestion. https://salsa.debian.org/mariadb-team/mariadb-server/-/wikis/Contributing-to-MariaDB-packaging-in-Debian It would also be cool to have in the salsa-ci.yml a job that attempts to use and run off overlayfs.
Dear submitter, as the package mariadb-10.6 has just been removed from the Debian archive unstable we hereby close the associated bug reports. We are sorry that we couldn't deal with your issue properly. For details on the removal, please see https://bugs.debian.org/1033485 The version of this package that was in Debian prior to this removal can still be found using https://snapshot.debian.org/. Please note that the changes have been done on the master archive and will not propagate to any mirrors until the next dinstall run at the earliest. This message was generated automatically; if you believe that there is a problem with it please contact the archive administrators by mailing ftpmaster@ftp-master.debian.org. Debian distribution maintenance software pp. Scott Kitterman (the ftpmaster behind the curtain)
Hi, I've faced same problem on a fresh Bookworm live system (mariadb-server 1:10.11.2-1). As a workaround I've mounted /var/lib/mysql as a ext4 filesystem. FYI: It works fine on Bullseye. Kind regards,
Still broken in 10.11.3. This is fresh installation, zero modifications. Jun 24 19:14:24 debian systemd[1]: Starting mariadb.service - MariaDB 10.11.3 database server... Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] Starting MariaDB 10.11.3-MariaDB-1 source revision as process 2685 Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.13 Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Number of transaction pools: 1 Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Using liburing Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Completed initialization of buffer pool Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: The Auto-extending data file './ibdata1' is of a different size 0 pages than specified by innodb_data_file_path Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: Operating system error number 22 in a file operation. Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument' Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/ Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation Jun 24 19:14:25 debian mariadbd[2685]: 230624 19:14:25 [ERROR] mysqld got signal 6 ; Jun 24 19:14:25 debian mariadbd[2685]: This could be because you hit a bug. It is also possible that this binary Jun 24 19:14:25 debian mariadbd[2685]: or one of the libraries it was linked against is corrupt, improperly built, Jun 24 19:14:25 debian mariadbd[2685]: or misconfigured. This error can also be caused by malfunctioning hardware. Jun 24 19:14:25 debian mariadbd[2685]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Jun 24 19:14:25 debian mariadbd[2685]: We will try our best to scrape up some info that will hopefully help Jun 24 19:14:25 debian mariadbd[2685]: diagnose the problem, but since we have already crashed, Jun 24 19:14:25 debian mariadbd[2685]: something is definitely wrong and this may fail. Jun 24 19:14:25 debian mariadbd[2685]: Server version: 10.11.3-MariaDB-1 source revision: Jun 24 19:14:25 debian mariadbd[2685]: key_buffer_size=134217728 Jun 24 19:14:25 debian mariadbd[2685]: read_buffer_size=131072 Jun 24 19:14:25 debian mariadbd[2685]: max_used_connections=0 Jun 24 19:14:25 debian mariadbd[2685]: max_threads=153 Jun 24 19:14:25 debian mariadbd[2685]: thread_count=0 Jun 24 19:14:25 debian mariadbd[2685]: It is possible that mysqld could use up to Jun 24 19:14:25 debian mariadbd[2685]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468022 K bytes of memory Jun 24 19:14:25 debian mariadbd[2685]: Hope that's ok; if not, decrease some variables in the equation. Jun 24 19:14:25 debian mariadbd[2685]: Thread pointer: 0x0 Jun 24 19:14:25 debian mariadbd[2685]: Attempting backtrace. You can use the following information to find out Jun 24 19:14:25 debian mariadbd[2685]: where mysqld died. If you see no messages after this, something went Jun 24 19:14:25 debian mariadbd[2685]: terribly wrong... Jun 24 19:14:25 debian mariadbd[2685]: stack_bottom = 0x0 thread_stack 0x49000 Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x559932b6ecfe] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x5599326dc7c9] Jun 24 19:14:25 debian mariadbd[2685]: libc_sigaction.c:0(__restore_rt)[0x7efc03891f90] Jun 24 19:14:25 debian mariadbd[2685]: nptl/pthread_kill.c:44(__pthread_kill_implementation)[0x7efc038e0ccc] Jun 24 19:14:25 debian mariadbd[2685]: posix/raise.c:27(__GI_raise)[0x7efc03891ef2] Jun 24 19:14:25 debian mariadbd[2685]: stdlib/abort.c:81(__GI_abort)[0x7efc0387c472] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x67483c)[0x55993232883c] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xd2a515)[0x5599329de515] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe29078)[0x559932add078] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe2c5cf)[0x559932ae05cf] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe2c6bd)[0x559932ae06bd] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x685d85)[0x559932339d85] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xcd9051)[0x55993298d051] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x5599326dfafe] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x80f257)[0x5599324c3257] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x85a)[0x5599324c421a] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x7008f7)[0x5599323b48f7] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x41d)[0x5599323ba31d] Jun 24 19:14:25 debian mariadbd[2685]: x86/libc-start.c:74(__libc_start_call_main)[0x7efc0387d18a] Jun 24 19:14:25 debian mariadbd[2685]: csu/libc-start.c:128(call_init)[0x7efc0387d245] Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_start+0x21)[0x5599323af541] Jun 24 19:14:25 debian mariadbd[2685]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains Jun 24 19:14:25 debian mariadbd[2685]: information that should help you find out what is causing the crash. Jun 24 19:14:25 debian mariadbd[2685]: Writing a core file... Jun 24 19:14:25 debian mariadbd[2685]: Working directory at /var/lib/mysql Jun 24 19:14:25 debian mariadbd[2685]: Resource Limits: Jun 24 19:14:25 debian mariadbd[2685]: Limit Soft Limit Hard Limit Units Jun 24 19:14:25 debian mariadbd[2685]: Max cpu time unlimited unlimited seconds Jun 24 19:14:25 debian mariadbd[2685]: Max file size unlimited unlimited bytes Jun 24 19:14:25 debian mariadbd[2685]: Max data size unlimited unlimited bytes Jun 24 19:14:25 debian mariadbd[2685]: Max stack size 8388608 unlimited bytes Jun 24 19:14:25 debian mariadbd[2685]: Max core file size 0 unlimited bytes Jun 24 19:14:25 debian mariadbd[2685]: Max resident set unlimited unlimited bytes Jun 24 19:14:25 debian mariadbd[2685]: Max processes 1030035 1030035 processes Jun 24 19:14:25 debian mariadbd[2685]: Max open files 32768 32768 files Jun 24 19:14:25 debian mariadbd[2685]: Max locked memory 524288 524288 bytes Jun 24 19:14:25 debian mariadbd[2685]: Max address space unlimited unlimited bytes Jun 24 19:14:25 debian mariadbd[2685]: Max file locks unlimited unlimited locks Jun 24 19:14:25 debian mariadbd[2685]: Max pending signals 1030035 1030035 signals Jun 24 19:14:25 debian mariadbd[2685]: Max msgqueue size 819200 819200 bytes Jun 24 19:14:25 debian mariadbd[2685]: Max nice priority 0 0 Jun 24 19:14:25 debian mariadbd[2685]: Max realtime priority 0 0 Jun 24 19:14:25 debian mariadbd[2685]: Max realtime timeout unlimited unlimited us Jun 24 19:14:25 debian mariadbd[2685]: Core pattern: |/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h Jun 24 19:14:25 debian mariadbd[2685]: Kernel version: Linux version 6.1.0-9-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08) Jun 24 19:14:25 debian systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT Jun 24 19:14:25 debian systemd[1]: mariadb.service: Failed with result 'core-dump'. Jun 24 19:14:25 debian systemd[1]: Failed to start mariadb.service - MariaDB 10.11.3 database server. Maybe mariadb does not like it being on overlayfs. Here are my mounts: user@debian:~$ mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,relatime,size=131844608k,nr_inodes=32961152,mode=755,inode64) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26378644k,mode=755,inode64) /dev/sda1 on /run/live/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8) /dev/loop0 on /run/live/rootfs/filesystem.squashfs type squashfs (ro,noatime,errors=continue) tmpfs on /run/live/overlay type tmpfs (rw,noatime,size=131893220k,mode=755,inode64) overlay on / type overlay (rw,noatime,lowerdir=/run/live/rootfs/filesystem.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work,redirect_dir=on) tmpfs on /usr/lib/live/mount type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26378644k,mode=755,inode64) /dev/sda1 on /usr/lib/live/mount/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8) /dev/loop0 on /usr/lib/live/mount/rootfs/filesystem.squashfs type squashfs (ro,noatime,errors=continue) tmpfs on /usr/lib/live/mount/overlay type tmpfs (rw,noatime,size=131893220k,mode=755,inode64) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=29782) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,inode64) ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime) sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime) lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=26378640k,nr_inodes=6594660,mode=700,uid=1000,gid=1000,inode64) gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000) portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
Just wwitching to fsync, does not appear to solve the problem either: root@debian:~# cat /etc/mysql/mariadb.conf.d/90-live.cnf [mysqld] innodb_flush_method = fsync root@debian:~# Jun 25 23:30:03 debian systemd[1]: Starting mariadb.service - MariaDB 10.11.3 database server... Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] Starting MariaDB 10.11.3-MariaDB-1 source revision as process 1395434 Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.13 Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Number of transaction pools: 1 Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Using liburing Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Completed initialization of buffer pool Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] InnoDB: The Auto-extending data file './ibdata1' is of a different size 0 pages than specified by innodb_data_file_path Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Starting shutdown... Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Plugin 'InnoDB' init function returned error. Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] Plugin 'FEEDBACK' is disabled. Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Unknown/unsupported storage engine: InnoDB Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Aborting Jun 25 23:30:03 debian systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE Jun 25 23:30:03 debian systemd[1]: mariadb.service: Failed with result 'exit-code'. Jun 25 23:30:03 debian systemd[1]: Failed to start mariadb.service - MariaDB 10.11.3 database server. This is probably, becuase postinst script probably (silently) failed, and something was not initialized properly during install. Adding runtime check in that script to detect overlay fs will not be enough, because it is possible to install mariadb-server in chroot for live image, which could be on non-overlay fs, but when actually started it might be on overlay fs. It would be preferable to do this in startup script instead, but then a lot of logic would probably need to move from postinst, to startup script.
Otto, et al, I took a fresh look at this bug and it is fixed in the following live images now: * debian-live-12.10.0-amd64-gnome.iso * debian-live-testing-amd64-gnome.iso (May 5, 2025) Unfortunately, I do not know when it was fixed. I think this bug was not in mariadb, but actually in linux-image-amd64; specifically, in the overlay filesystem. Thank you! Daniel Lewart Urbana, Illinois