Upon boot I saw that a large file system was up for periodic check at
the next boot. To shorten the next boot time, I umounted the file
system and wanted to run e2fsck on it as I've done in the past.
However, e2fsck refuses to do anything, stating:

e2fsck 1.42.12 (29-Aug-2014)
… is in use.
e2fsck: Cannot continue, aborting.

To verify that this was not a problem with this specific file system,
I tried another one:

root@sneo:~# mount | grep video
/dev/mapper/system_vg-video_lv on /scr/media/video type ext3 (rw,nosuid,nodev,noexec,relatime,errors=continue,barrier=1,data=ordered)

root@sneo:~# fuser /scr/media/video/
root@sneo:~#

root@sneo:~# umount /dev/mapper/system_vg-video_lv
root@sneo:~#

root@sneo:~# mount | grep video
root@sneo:~#

root@sneo:~# LC_ALL=C e2fsck /dev/mapper/system_vg-video_lv
e2fsck 1.42.12 (29-Aug-2014)
/dev/mapper/system_vg-video_lv is in use.
e2fsck: Cannot continue, aborting.

root@sneo:~# mount /dev/mapper/system_vg-video_lv
root@sneo:~# mount | grep video
/dev/mapper/system_vg-video_lv on /scr/media/video type ext3
(rw,nosuid,nodev,noexec,relatime,errors=continue,barrier=1,data=ordered)
root@sneo:~#

I recently did this to enlarge file systems / lvm volumes, so it just
recently broke. (Sorry, I cannot give you exact dates, I don't log
this for my private machine).


P.S. Upon boot another (large) file system was checked, so checking in
     principle seems to work stilll

tries to open the device using the O_EXCL flag.  If the kernel
believes that the kernel is in use somehow (i.e., mounted, or is being
used as half of a /dev/mdX RAID array, or some other userspace progam
has the device open with O_EXCL), then it will return EBUSY.

Try stracing e2fsck, and I think you will see it is the open(2) system
call which is failing.  It doesn't fail for me:

<tytso.root@closure> {/}, level 2
577# umount /build
<tytso.root@closure> {/}, level 2
578# e2fsck -f /dev/heap/build
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/heap/build: 128865/2097152 files (2.6% non-contiguous), 4600740/8388608 blocks

So there is something specific with your system that is holding the
device open.  This is a feature not a bug; for example, if while the
e2fsck is running in one window, someone else (perhaps another
well-meaning system administrator) tries running e2fsck on the same
device, they will get:

# e2fsck -f /dev/heap/build
e2fsck 1.42.12 (29-Aug-2014)
/dev/heap/build is in use.
e2fsck: Cannot continue, aborting.

If two system administrators were to try running e2fsck on the same
device at the same time, it could result in nasty file system
corruption.

This mechanism (using O_EXCL when opening the device) is also what is
used to prevent a failure mode where a junior intern accidentally ran
mke2fs on half of a RAID1 mirror on an early version of SourceForge.
This was obviously a mistake, although the bigger mistake was made by
the sysadmin who gave root privs to said well-meaning but clueless
intern.  :-)

Cheers,

						- Ted

Hello Theodore,

That is good to read.

Until recently, I never experienced this bug, and I run e2fsck around
once or twice a month, either because it it be upcoming after next
boot or because I enlarge LVM.

I retried the test case, and it fails again. I attached the strace, I
cannot make anything meaningful out of it (except, that it actually
fails in the open call:
open("/dev/system_vg/video_lv", O_RDONLY|O_EXCL) = -1 EBUSY (Device or resource busy)

There is no other system admin, I'm only logged in as normal user and
on one single VT as admin. I never had this problem before. And it is
repeatible across reboots. (And I did not change my kernel recently).

…

Thanks for the explanation, but would there be a way to figure out
this second process? I tried fuser and lsopen, both to no avail (but
I've might have missed the right set of options, of course).

I'm willing to experiement further, however, I've no clue what to try
(and at some time I will definitly want to enlarge my LVM volumes).

Thanks for your help,

            Helge


Here is the strace output:

execve("/sbin/e2fsck", ["e2fsck", "/dev/system_vg/video_lv"], [/* 14 vars */]) = 0
brk(0)                                  = 0x22ad290
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3422af81000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=194016, ...}) = 0
mmap(NULL, 194016, PROT_READ, MAP_PRIVATE, 3, 0) = 0x3422af51000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libext2fs.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\336\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=285248, ...}) = 0
mmap(NULL, 2381192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3422ab1f000
mprotect(0x3422ab63000, 2093056, PROT_NONE) = 0
mmap(0x3422ad62000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x43000) = 0x3422ad62000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\25\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14640, ...}) = 0
mmap(NULL, 2109960, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3422a91b000
mprotect(0x3422a91e000, 2093056, PROT_NONE) = 0
mmap(0x3422ab1d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x3422ab1d000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\210\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=258688, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3422af50000
mmap(NULL, 2358248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3422a6db000
mprotect(0x3422a716000, 2097152, PROT_NONE) = 0
mmap(0x3422a916000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x3422a916000
mmap(0x3422a91a000, 3048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3422a91a000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \26\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18904, ...}) = 0
mmap(NULL, 2113952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3422a4d6000
mprotect(0x3422a4da000, 2093056, PROT_NONE) = 0
mmap(0x3422a6d9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x3422a6d9000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libe2p.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20 \0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=32592, ...}) = 0
mmap(NULL, 2128528, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3422a2ce000
mprotect(0x3422a2d5000, 2093056, PROT_NONE) = 0
mmap(0x3422a4d4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x3422a4d4000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1729984, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3422af4f000
mmap(NULL, 3836448, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34229f25000
mprotect(0x3422a0c4000, 2097152, PROT_NONE) = 0
mmap(0x3422a2c4000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19f000) = 0x3422a2c4000
mmap(0x3422a2ca000, 14880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3422a2ca000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20o\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=137440, ...}) = 0
mmap(NULL, 2213008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x34229d08000
mprotect(0x34229d20000, 2093056, PROT_NONE) = 0
mmap(0x34229f1f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x34229f1f000
mmap(0x34229f21000, 13456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x34229f21000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3422af4e000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3422af4c000
arch_prctl(ARCH_SET_FS, 0x3422af4c780)  = 0
mprotect(0x3422a2c4000, 16384, PROT_READ) = 0
mprotect(0x34229f1f000, 4096, PROT_READ) = 0
mprotect(0x3422a4d4000, 4096, PROT_READ) = 0
mprotect(0x3422a6d9000, 4096, PROT_READ) = 0
mprotect(0x3422a916000, 12288, PROT_READ) = 0
mprotect(0x3422ab1d000, 4096, PROT_READ) = 0
mprotect(0x3422ad62000, 4096, PROT_READ) = 0
mprotect(0x639000, 4096, PROT_READ)     = 0
mprotect(0x3422af85000, 4096, PROT_READ) = 0
munmap(0x3422af51000, 194016)           = 0
set_tid_address(0x3422af4ca50)          = 3961
set_robust_list(0x3422af4ca60, 24)      = 0
rt_sigaction(SIGRTMIN, {0x34229d0e9f0, [], SA_RESTORER|SA_SIGINFO, 0x34229d178d0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x34229d0ea80, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x34229d178d0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
rt_sigaction(SIGFPE, {0x427d30, [], SA_RESTORER|SA_SIGINFO, 0x34229f5a180}, NULL, 8) = 0
rt_sigaction(SIGILL, {0x427d30, [], SA_RESTORER|SA_SIGINFO, 0x34229f5a180}, NULL, 8) = 0
rt_sigaction(SIGBUS, {0x427d30, [], SA_RESTORER|SA_SIGINFO, 0x34229f5a180}, NULL, 8) = 0
rt_sigaction(SIGSEGV, {0x427d30, [], SA_RESTORER|SA_SIGINFO, 0x34229f5a180}, NULL, 8) = 0
brk(0)                                  = 0x22ad290
brk(0x22ce290)                          = 0x22ce290
brk(0x22cf000)                          = 0x22cf000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3149360, ...}) = 0
mmap(NULL, 3149360, PROT_READ, MAP_PRIVATE, 3, 0) = 0x34229a07000
close(3)                                = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x3d0446aa400) = -1 ENOTTY (Inappropriate ioctl for device)
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1
open("/etc/blkid.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/run/blkid/blkid.tab", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/e2fsck.conf", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
rt_sigaction(SIGINT, {0x40c720, [], SA_RESTORER, 0x34229f5a180}, NULL, 8) = 0
rt_sigaction(SIGTERM, {0x40c720, [], SA_RESTORER, 0x34229f5a180}, NULL, 8) = 0
rt_sigaction(SIGUSR1, {0x40c700, [], SA_RESTORER|SA_RESTART, 0x34229f5a180}, NULL, 8) = 0
rt_sigaction(SIGUSR2, {0x40d760, [], SA_RESTORER|SA_RESTART, 0x34229f5a180}, NULL, 8) = 0
open("/dev/null", O_RDWR)               = 3
getrusage(RUSAGE_SELF, {ru_utime={0, 0}, ru_stime={0, 4000}, ...}) = 0
write(2, "e2fsck 1.42.12 (29-Aug-2014)\n", 29e2fsck 1.42.12 (29-Aug-2014)
) = 29
stat("/dev/system_vg/video_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 7), ...}) = 0
open("/proc/swaps", O_RDONLY)           = 4
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3422af80000
read(4, "Filename\t\t\t\tType\t\tSize\tUsed\tPrio"..., 1024) = 101
stat("/dev/dm-0", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 0), ...}) = 0
read(4, "", 1024)                       = 0
close(4)                                = 0
munmap(0x3422af80000, 4096)             = 0
open("/proc/mounts", O_RDONLY|O_CLOEXEC) = 4
stat("/dev/system_vg/video_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 7), ...}) = 0
futex(0x3422a2cb088, FUTEX_WAKE_PRIVATE, 2147483647) = 0
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3422af80000
read(4, "rootfs / rootfs rw 0 0\n/dev/root"..., 1024) = 1024
stat("/dev/root", 0x3d0446aa310)        = -1 ENOENT (No such file or directory)
stat("/dev/sda2", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 2), ...}) = 0
stat("/dev/mapper/system_vg-home_vg", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 3), ...}) = 0
stat("/dev/mapper/system_vg-tmp_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 4), ...}) = 0
read(4, "=ordered 0 0\n/dev/mapper/system_"..., 1024) = 1003
stat("/dev/mapper/system_vg-usr_vg", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 1), ...}) = 0
stat("/dev/mapper/system_vg-var_vg", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 2), ...}) = 0
stat("/dev/mapper/system_vg-build_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 5), ...}) = 0
stat("/dev/mapper/system_vg-mirror_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 8), ...}) = 0
stat("/dev/mapper/system_vg-cache_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 9), ...}) = 0
stat("/dev/mapper/system_vg-media_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 6), ...}) = 0
stat("/dev/mapper/system_vg-aptcache_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 10), ...}) = 0
stat("/dev/mapper/backup_vg-mediabackup", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 12), ...}) = 0
read(4, "", 1024)                       = 0
stat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
close(4)                                = 0
munmap(0x3422af80000, 4096)             = 0
open("/etc/mtab", O_RDONLY|O_CLOEXEC)   = 4
stat("/dev/system_vg/video_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 7), ...}) = 0
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3422af80000
read(4, "rootfs / rootfs rw 0 0\n/dev/root"..., 1024) = 1024
stat("/dev/root", 0x3d0446aa310)        = -1 ENOENT (No such file or directory)
stat("/dev/sda2", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 2), ...}) = 0
stat("/dev/mapper/system_vg-home_vg", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 3), ...}) = 0
stat("/dev/mapper/system_vg-tmp_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 4), ...}) = 0
read(4, "=ordered 0 0\n/dev/mapper/system_"..., 1024) = 1003
stat("/dev/mapper/system_vg-usr_vg", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 1), ...}) = 0
stat("/dev/mapper/system_vg-var_vg", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 2), ...}) = 0
stat("/dev/mapper/system_vg-build_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 5), ...}) = 0
stat("/dev/mapper/system_vg-mirror_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 8), ...}) = 0
stat("/dev/mapper/system_vg-cache_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 9), ...}) = 0
stat("/dev/mapper/system_vg-media_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 6), ...}) = 0
stat("/dev/mapper/system_vg-aptcache_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 10), ...}) = 0
stat("/dev/mapper/backup_vg-mediabackup", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 12), ...}) = 0
read(4, "", 1024)                       = 0
stat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
close(4)                                = 0
munmap(0x3422af80000, 4096)             = 0
stat("/dev/system_vg/video_lv", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 7), ...}) = 0
open("/dev/system_vg/video_lv", O_RDONLY|O_EXCL) = -1 EBUSY (Device or resource busy)
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=2492, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3422af80000
read(4, "# Locale name alias data base.\n#"..., 4096) = 2492
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x3422af80000, 4096)             = 0
open("/usr/share/locale/de_DE.UTF-8/LC_MESSAGES/e2fsprogs.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de_DE.utf8/LC_MESSAGES/e2fsprogs.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de_DE/LC_MESSAGES/e2fsprogs.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de.UTF-8/LC_MESSAGES/e2fsprogs.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de.utf8/LC_MESSAGES/e2fsprogs.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/de/LC_MESSAGES/e2fsprogs.mo", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=152801, ...}) = 0
mmap(NULL, 152801, PROT_READ, MAP_PRIVATE, 4, 0) = 0x3422af5b000
close(4)                                = 0
open("/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=26258, ...}) = 0
mmap(NULL, 26258, PROT_READ, MAP_SHARED, 4, 0) = 0x3422af54000
close(4)                                = 0
futex(0x3422a2c98f8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(1, "/dev/system_vg/video_lv wird ver"..., 40/dev/system_vg/video_lv wird verwendet.
) = 40
write(2, "e2fsck: Fortsetzung nicht m\303\266gli"..., 56e2fsck: Fortsetzung nicht möglich, wird abgebrochen.


) = 56
exit_group(8)                           = ?
+++ exited with 8 +++

Is there any chance that you might be using mount namespaces or bind
mounts?  Are you using LXC, or something like schroot which might be
doing this on your behalf without knowing it?  If so, it might be that
although you *think* you've unmounted the file system, but in fact
it's still mounted.  Try checking /proc/fs/ext4 and see if you see the
device-mapper node corresponding to /dev/system_vg/video_lv.  If it's
there, then you probably have some mount namespace where the file
system is mounted.

Unfortunately depending on which namespaces are in use, trying to
figure out what still has the file system mounted can be a bit of a
mess.  If you're not using pid namespaces, you coulld look at
/proc/<pid>/mounts for every single pid in the system, and see if one
or more processes has a /proc/<pid>/mounts file that has the file
system mounted.

As you'll see, /proc/mounts is symlink of /proc/self/mounts, so that
just shows you the file systems mounted in your current shell's mount
namespace.  So you should of course check that, but just because you
don't see the device there, it doesn't guarantee that the file system
isn't mounted.

And of course, if you have pid namespaces enabled, and something has
set up separate pid namespaces, then /proc/* won't enumerate all of
the processes in the system, and for the same reason, fuser and lsof,
which works by examining all of the open file descriptors using
/proc/<pid>/fds/*, will also not work.

The other thing to consider is if you've created any LVM snapshots;
that will also pin a block device:

# lvcreate -s -L 5G -n build-snap heap/build
# dmsetup ls --tree
    ...
heap-build--snap (252:15)
 ├─heap-build--snap-cow (252:17)
 │  └─ (8:16)
 └─heap-build-real (252:16)
    └─ (8:16)
   ...
heap-build (252:10)
 └─heap-build-real (252:16)
    └─ (8:16)

Any of course, if a device is part of MD raid set, that will also
cause an open with O_EXCL to fail.  So there are a number of different
reasons that an O_EXCL open can fail, and all you can do is to check
all of them, and some of them don't have a simply way of testing.

Sorry.... I'm actually not all that found with how the container folks
have implemented namespaces in the kernel, because they are not very
transparent since various namespaces are not first class named
objects, and so there is no way to actually list or name them all.

Regards,

				- Ted

BTW, have you considered actually creating a snapshot and then running
e2fsck on the snapshot?  Then you don't actually have to unmount it.

An example of how to do this can be found here:

https://git.kernel.org/cgit/fs/ext2/e2fsprogs.git/tree/contrib/e2croncheck

Cheers,

						- Ted

Hello Theodore,
please excuse the late reply, I was on business travel.

None that I know of. I use LVM, but neither namespace nor bind bounds,
LXC is not installed and schroot currently not configured on this
machine.

I don't have »/proc/fs/ext4«. (But then I don't use ext4, the
partition in question uses ext3).

Ok, this seems to point to something:
find /proc/ -name mounts -exec cat "{}" ";" | grep video
finds two entries.

Ok, the first one is:
/sbin/cgmanager --daemon -m name=systemd

(However, I don't use systemd on this machine, it does not really work
on this machine, and having it installed partially (though not as PID
1) caused only trouble; unfortunately sometimes hard depencendcies
require it, but I disgress).

The other entry points to the same programm, (/proc/2721/mounts and
/proc/2721/task/2721/mounts).

Now I removed cgmanager:

And e2fsck works again!

First of all, thanks for helping, I did not know about the trick with
/proc.

Secondly should this bug be assigned to cgmanager?

(And probably a wishlist bug for the kernel to offer a reliable means
to detect mounted partitions?).

Thanks again!

Greetings

           Helge

So cgmanager is different from systemd.  In fact, it's not clear in th
elong term whether systemd and cgmanager will play well together.

So e2fsck was always working correctly; it properly prevented file
system corruption by warning you that the file system was still
mounted --- which it was.  :-)

Well, what it is doing is documented:

"This daemon (cgmanager) runs on the host, mounts cgroupfs into a
separate mount namespace (so it's invisible from the host), binds
/sys/fs/cgroup/cgmanager/sock for incoming DBus queries and generally
handles all clients running directly on the host." -- https://cgmanager.linuxcontainers.org/

The net effect is that if you start cgmanager, since it create a
separate mount namespace, any file systems which are mounted at the
time when cgmanager is started will still be mounted in the cgmanager
namespace:

# mount -t ext4 /dev/heap/bitcoin /bitcoin
# service cgmanager start
# umount /dev/heap/bitcoin
# e2fsck -p /dev/heap/bitcoin
/dev/heap/bitcoin is in use.
e2fsck: Cannot continue, aborting.

# service cgmanager stop
# e2fsck -p /dev/heap/bitcoin
/dev/heap/bitcoin: clean, 15913/4587520 files, 14937110/18350080 blocks


Whether or not this is a _bug_ is a bit debateable.  It is a natural
consequences of its documented (and intentional) behavior.  Granted it
is has some very confusing and someone user-unfriendly consequences.
So, if you want try to reassign this to the cgmanager package, it's
not unreasonable.  But it's likely that any "fix" will require some
pretty serious design discussions with its upstream developers.

Again, it's not unreasonable, but it's also the case that it's beyond
what the Debian kernel developers are likely to have the bandwidth and
mandate to try to "fix", since it requires a fairly fundamental
rearchitecture of how namespaces work.  I've been grumbling about this
for years, and if there was an easy fix, it would have been fixed by
now.

Best regards,

					- Ted

clone 768406 -1 -2
reassign -1 cgmanager
retitle -1 Should honor umount to allow filesystem operations like e2fsck (at least in default install)
severity -1 important
reassign -2 lsof
retitle -2 Please show filesystems used by cgmanager for tracking
severity -2 wishlist
thanks

Hello Ted,

Thanks for the explanation. At least I did not install it on purpose.

I believe on default install (probably due to a suggests: from a
different package as in my case) this is a bug. Unless I wilingly
configure namespaces I expect basic tools like e2fsck to remain
operational (in the sense that I can easily use them, not in the sense
that they are broken, as it is definitly not).

I know the »easy« solution of a Debconf prompt upon installation is
frowned upon, thus I cannot derive an easy fix myself.

Well, I think at least lsof should be able to tell me about cgmanager,
opening a wishlist bug there as well.

I left this bug at e2fsprogs, feel free to deal with it as
appropriate, e.g. close it.

Thanks again very much for your help!

Greetings

          Helge