Dear Maintainer,

BTRFS_SCRUB_PRIORITY="idle" (or not setting it at all) in
/etc/default/btrfsmaintenance should cause
/usr/share/btrfsmaintenance/btrfs-scrub.sh to start scrub
with idle IO priority.

But it starts it with the default IO priority like any other
job.

The reason seems to be that the default value for ioprio
is not set:
--- /usr/share/btrfsmaintenance/btrfs-scrub.sh.old	2018-09-10 07:19:22.082165973 +0100
+++ /usr/share/btrfsmaintenance/btrfs-scrub.sh	2018-09-10 07:19:32.201975411 +0100
@@ -22,7 +22,7 @@
 	readonly=-r
 fi

Hi Wolfgang,

Thank you for filing this bug, I'm always happy to hear from users,
and a bug is an opportunity to improve something!
[ text from patch and System Information ]

From 'man btrfs-scrub', "The default IO priority of scrub is the idle
class", thus 'btrfs-scrub' should default to the idle class when
$ioprio is nil.

eg:
ioprio=
btrfs scrub start -Bd $ioprio /
# is equivalent to
# btrfs scrub start -Bd /
# which should default to -c 3 (idle)

The idle priority might not be applied for other reasons.  Would you
please share the output to the following commands:

First, please unapply your patch or reinstall btrfsmaintenance, since
it looks like it is applied to /usr/share/btrfsmaintenance/btrfs-scrub.sh

# while a btrfsmaintenance scheduled scrub is running
ionice -p `pidof btrfs-scrub`
# and
cat /sys/block/sd*/queue/scheduler
# and
cat /sys/module/*/parameters/use_blk_mq


Cheers,
Nicholas

Here is an upstream pull request related to the topic of this bug:

  "Add Balance ionice idle priority"
https://github.com/kdave/btrfsmaintenance/pull/66

I expect there will be a request for data that shows the existing
method is buggy and isn't working properly, but I could be wrong.

More generally I wonder if the idle class might not be working
properly on some btrfs systems using the new block-mq, but that's a
kernel thing.

Wolfgang, when you have a couple minutes of free time would you
please provide the information requested in the reply previous to this
one?

Kind regards,
Nicholas

Dear Maintainer,

This bug is actually preventing Debian to work as a multimedia server like
Plex, because during scrubbing, everything else becomes very slow and
unresponsive and with even a few TB to scrub, it takes already almost two days
(in my case 38h).
So this bug is actually for me more urgent than for the the other poster.

Thanks though for your work

Dear Philip,

Philip <phlomos@gmail.com> writes:

I've also noticed that btrfs has gotten slower sometime post ~linux-4.9
(with defaults), and I wish the fix proposed in this bug actually
resolved it.  Given that with btrfs a syncthing update of several
thousand files will also slow a desktop down to an unusable state, I
agree with upstream that it's a kernel issue.  For a server, I've had
best results with the old non-multiqueue scheduler and CFQ, and for
desktop I schedule scrubs or defrags for times I won't be using the
system and use the non-multiqueue deadline scheduler--this is documented
on our wiki.

In the worst-case scenario, I wonder if ancient (but still existing)
kernel issue that makes a desktop unusable when a USB drive is maxed out
with IO (any filesystem) is what causes this effect.  eg: that btrfs is
guaranteed to trigger the same condition on any type of drive that is
otherwise triggered by drive that has slow IO and high iowait (with any
filesystem).
issue than btrfsmaintenance)...back in 2015 I fully expected it to be
resolved by now.

The wiki documents a bunch of things that make this problem worse (eg:
using compression, keeping snapshots around, filling the disk more than
~80% full, etc).  Not a real solution, I know...  P.S. I'm also testing
a zfs system, and its performance also falls off a cliff when an aged fs
is above ~80% capacity--and once that happens, removing of files will
never restore the old performance.  That's why I prefer btrfs, despite
the known cases that cause terrible performance.


Regards,
Nicholas

Dear Maintainer,

I just saw, that independent of the scheduling class, which is set at start of
the scrubbing, the kernel worker processes seem to have always a scheduling
priority of TS and not IDL as you can see with the following ps:

# p|grep scrub
16288 root       - IDL  3.3  0.0    9  00:04 btrfs scrub start -Bd -c 3
/usr/local/share/backup
18331 root       0 TS   0.7  0.0    1  07:49 [kworker/u16:4-btrfs-scrub]
18126 root       0 TS   0.6  0.0    1  07:30 [kworker/u16:2-btrfs-scrub]
15827 root       0 TS   0.5  0.0    1  06:46 [kworker/u16:7-btrfs-scrub]
18022 root       0 TS   0.5  0.0    1  07:09 [kworker/u16:25-btrfs-scrubparity]
18025 root       0 TS   0.5  0.0    1  07:09 [kworker/u16:31-btrfs-scrubparity]
26890 root       0 TS   0.5  0.0    1  08:39 [kworker/u16:9-btrfs-scrub]
30744 root       0 TS   0.5  0.0    1  05:03 [kworker/u16:28-btrfs-scrub]
18021 root       0 TS   0.4  0.0    1  07:09 [kworker/u16:23-btrfs-scrub]
18065 root       0 TS   0.4  0.0    1  07:17 [kworker/u16:27-btrfs-scrub]
27865 root       0 TS   0.4  0.0    1  08:45 [kworker/u16:12-btrfs-scrub]
30836 root       0 TS   0.4  0.0    1  05:09 [kworker/u16:1-btrfs-scrub]
31648 root       0 TS   0.4  0.0    1  06:05 [kworker/u16:24-btrfs-scrub]
14798 root       - IDL  0.0  0.0    1  00:00 /bin/sh
/usr/share/btrfsmaintenance/btrfs-scrub.sh
14809 root       - IDL  0.0  0.0    1  00:00 /bin/sh
/usr/share/btrfsmaintenance/btrfs-scrub.sh

Do you think there's a reason for that or could we just re-ionice them?

Regards

Hi Philip,

Philip <phlomos@gmail.com> writes:

So raid5/6 profile.

I do not recommend attempting to renice them, because raid5/6 is still
fragile and bug-prone.  On that topic, if the objective is to survive
one disk failure, raid1 profile (aka: 2 copies on x disks) is better in
every way to the raid5 profile, and for the purposes of this bug the
load on the system will be less while scrubbing.  If you need 3 copies,
and have a new enough kernel, and want something in between the
stability of raid1 and raid6 profiles, then raid1c3 is worth a try.  I
believe you'll be more satisfied with its performance than raid5/6,
which are the worst performing profiles.

From what I remember of upstream threads on this topic it's not possible
to renice those threads...  IIRC there is more info at the forwarded
URL, but it might have been on LKML.


Regards,
Nicholas