#1057679 lvm2: LVM cache gets marked completely dirty on every boot when used for filesystem root

Package:
lvm2
Source:
lvm2
Description:
Linux Logical Volume Manager
Submitter:
progserega
Date:
2023-12-25 13:06:02 UTC
Severity:
normal
#1057679#5
Date:
2023-12-07 02:30:20 UTC
From:
To:
Dear Maintainer,

*** Reporter, please consider answering these questions, where appropriate ***

   * I create logical volume 'root_deb' on hdd and its writeback cache on ssd (nvme). And also I create logical volume 'home' and its cache on nvme.
   * All works. But after reboot PC - I get big value CacheDirtyBlocks for root_deb logical volume. After some tome (~30 minutes) cache synced - and it work again propertly. But next reboot - get big dirty value for root_deb again. For 'home' no such bug. Only for root logical volume.
   * I think that dirty value after reboot do not big (~12 Gb) - as in my situation.

LVM configuration:
lvs -v
LV       VG       #Seg Attr       LSize   Maj Min KMaj KMin Pool                    Origin           Data%  Meta%  Move Cpy%Sync Log Convert LV UUID                                LProfile
media    vg_media    4 -wi-ao----  <1,36t  -1  -1  253    0                                                                                  IOK4Zv-sm7h-Rcsx-QAXp-ZChT-dw50-Sketa1
boot     vg_root     1 -wi-ao---- 200,00m  -1  -1  253   12                                                                                  b0tdbO-f12D-TSHc-Szda-YQKJ-iudc-Z0h2cc
home     vg_root     1 Cwi-aoC--- 300,00g  -1  -1  253    4 [home_cache_data_cpool] [home_corig]     5,88   0,24        0,07                 D9NZx2-yX2M-KvBJ-1q0e-RP7J-cTyS-1ZOBns
root20   vg_root     1 -wi-ao----  60,00g  -1  -1  253    7                                                                                  YUkX03-ZVEe-Lplv-mU4a-uqlx-2a0E-naaQKv
root_deb vg_root     1 Cwi-aoC---  60,00g  -1  -1  253   11 [root_cache_data_cpool] [root_deb_corig] 30,72  1,13        0,00                 KZ1UYo-h9le-haHu-rgPR-6IEQ-jS9O-1bbAbq
swap     vg_root     1 -wi-ao----   4,00g  -1  -1  253    5                                                                                  MnyACP-ZRRX-LVMv-8kwX-Ff20-Kmrb-s1XIpo
swap2    vg_root     1 -wi-ao----  10,00g  -1  -1  253    6

#1057679#10
Date:
2023-12-07 03:07:11 UTC
From:
To:
After reboot I get such situatin (big dirty on root):

sudo /usr/sbin/lvdisplay -C -o lv_name,lv_size,metadata_percent,data_percent,sync_percent,cache_dirty_blocks,cache_read_hits,cache_read_misses,cache_write_hits,cache_write_misses --aligned --separator '|'|sed 's/</ /'
  LV      |LSize  |Meta% |Data% |Cpy%Sync|CacheDirtyBlocks|CacheReadHits   |CacheReadMisses |CacheWriteHits  |CacheWriteMisses
  media   |  1,36t|      |      |        |                |                |                |                |
  boot    |200,00m|      |      |        |                |                |                |                |
  home    |300,00g|0,24  |5,91  |0,00    |               0|          972822|          198634|         7650224|          136267
  root20  | 60,00g|      |      |        |                |                |                |                |
  root_deb| 60,00g|1,13  |30,74 |70,29   |          212423|           63082|           25693|           57618|           10209
  swap    |  4,00g|      |      |        |                |                |                |                |
  swap2   | 10,00g|      |      |        |                |                |                |                |

or (cmd: lvs -v):
  LV       VG       #Seg Attr       LSize   Maj Min KMaj KMin Pool                    Origin           Data%  Meta%  Move Cpy%Sync Log Convert LV UUID                                LProfile
  media    vg_media    4 -wi-ao----  <1,36t  -1  -1  253    0                                                                                  IOK4Zv-sm7h-Rcsx-QAXp-ZChT-dw50-Sketa1
  boot     vg_root     1 -wi-ao---- 200,00m  -1  -1  253   12                                                                                  b0tdbO-f12D-TSHc-Szda-YQKJ-iudc-Z0h2cc
  home     vg_root     1 Cwi-aoC--- 300,00g  -1  -1  253    4 [home_cache_data_cpool] [home_corig]     5,89   0,24        0,05                 D9NZx2-yX2M-KvBJ-1q0e-RP7J-cTyS-1ZOBns
  root20   vg_root     1 -wi-ao----  60,00g  -1  -1  253    7                                                                                  YUkX03-ZVEe-Lplv-mU4a-uqlx-2a0E-naaQKv
  root_deb vg_root     1 Cwi-aoC---  60,00g  -1  -1  253   11 [root_cache_data_cpool] [root_deb_corig] 30,73  1,13        99,47                KZ1UYo-h9le-haHu-rgPR-6IEQ-jS9O-1bbAbq
  swap     vg_root     1 -wi-ao----   4,00g  -1  -1  253    5                                                                                  MnyACP-ZRRX-LVMv-8kwX-Ff20-Kmrb-s1XIpo
  swap2    vg_root     1 -wi-ao----  10,00g  -1  -1  253    6                                                                                  JVa8Fc-0Wl5-kLyf-jWgA-oe34-a3Nu-raSnNG


It attachment previous boot log (from cmd: journalctl -b -1 > /tmp/prev_boot.log). It show how unmount volumes.

#1057679#15
Date:
2023-12-07 10:17:22 UTC
From:
To:
Change policy cache from writeback to writethrough - not change situation. After reboot - again 100% dirty cache and big I/O.
#1057679#20
Date:
2023-12-07 18:07:32 UTC
From:
To:
I assume this happens because the blockdevice is not cleanly unmounted
during shutdown.  Please test "dracut" as initramfs generator.  This
should support a shutdown mode that does not have that problem.

Bastian

#1057679#25
Date:
2023-12-25 13:03:29 UTC
From:
To:
Thank you very match, Bastian Blank!

Dracut help fix this issue:
0. add cache to root logical volume.
1. I install dracut (it remove initramfs-tools)
2. add modules lvm and mdadm to dracut
3. update all initrd-images by dracut
4. add GRUB_CMDLINE_LINUX="rd.md=1 rd.md.conf=1 rd.auto=1" to /etc/default/grub (becouse system not boot after switch to dracut in my case) for init mdadm by initramfs at boot.
5. dpgk-reconfigure grub-pc
6. reboot twice
7. after all reboot - dirty cache for root lv was normal.

dracut + some tuning = fix dirty cache "bug" on root.

But I think this problem was not in lvmcache, but in initramfs-tools, which not correct unmount root? So lvmcache simply show this problem... And without lvmcache but with initramfs-tools - at any reboot root was not correctly unmount?