Dear Maintainer, I am running ceph-fs across at least five different machines. After a whole-house power outage, two of the machines on debian stable can no longer access the ceph file system, they get kernel null pointer derefs. Other machines (on different kernel versions) work. The working kernels include: $ cat /proc/version Linux version 6.12.57+deb13-amd64 (debian-kernel@lists.debian.org) (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT_DYNAMIC Debian 6.12.57-1 (2025-11-05) $ cat /proc/version Linux version 6.18.5+deb14-amd64 (debian-kernel@lists.debian.org) (x86_64-linux-gnu-gcc-15 (Debian 15.2.0-12) 15.2.0, GNU ld (GNU Binutils for Debian) 2.45.50.20251209) #1 SMP PREEMPT_DYNAMIC Debian 6.18.5-1 (2026-01-16) The non-working kernels are both: # cat /proc/version Linux version 6.12.63+deb13-amd64 (debian-kernel@lists.debian.org) (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT_DYNAMIC Debian 6.12.63-1 (2025-12-30) I include several stack traces below, one for each machine. The system information here is for one of the machines; it should be the same on the other non-working machine as well. (The config has been working great for over a year; the issue occurred only after reboot to the latest kernels.)
I rebooted one of the failing machines to an earlier kernel, and ceph-fs works fine there. i.e. works great for this kernel; with no other changes: # cat /proc/version Linux version 6.12.57+deb13-amd64 (debian-kernel@lists.debian.org) (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT_DYNAMIC Debian 6.12.57-1 (2025-11-05)
severity 1127056 normal reassign 1127056 linux thanks Hi, thank you for your report - based on your description it seems more like there's an issue in the kernel since that was the only moving part in you rsetup that triggert the error. Hence reassigning to the kernel. Regards, Daniel
severity 1127056 normal reassign 1127056 linux thanks Hi, thank you for your report - based on your description it seems more like there's an issue in the kernel since that was the only moving part in you rsetup that triggert the error. Hence reassigning to the kernel. Regards, Daniel
Hi, A quick check with the google LLM suggests this is a Ceph bug exposed by recent changes to the Linux kernel, and not a kernel bug. A selective quote: Technical Root Cause: The issue stems from how the kernel handles the fs_name field. In certain versions (notably regressions starting around kernel 6.18-rc1), if an MDS namespace is not explicitly specified during mount, the fs_name variable can remain NULL. When the kernel later attempts a strict authorization check via strcmp(auth->match.fs_name, fs_name), it triggers the NULL pointer dereference. Fix Implementation: Rework ceph_mdsmap_decode() to ensure m_fs_name is always populated (using "cephfs" as a default for older systems) to prevent the NULL comparison. The LLM notes the related bug CVE-2026-23189 https://nvd.nist.gov/vuln/detail/cve-2026-23189 This explanation sounds entirely plausible to me, but without several days of code review and experimentation, I cannot confirm this. FWIW, my ceph cluster has a non-default name (and so I have to use `--cluster foobar` for many commands.) I have found that this non-default name has created a significant number of "minor" issues, which I had to patch via systemd config file settings. They were certainly annoyances, not quite rising to the level of "bugs". I suspect that my non-default config might lead to a non-default MDS namespace, which, on some of the machines, results in `fs_name` being set to null. This is a wild guess, but would explain why I see this, but few others do.
this is actively discuraged by upstream and support for this has been removed in various places since the (iirc) luminous release (ceph 12, 2017). I recommend not setting a custom cluster-name at all. Regards, Daniel
Gahh! I set up the cluster in Nov/Dec 2024 and at that time, the Ceph documentation seemed to encourage setting a cluster name (giving explicit examples how to do this) Initially, I set this up on reef, but switched to squid quite soon. Using a non-default cluster name "works", other than occasional sharp edges in various config files. I fear reverting to the default; there are just enough config settings to make this a weeklong task.