#775169 lvm2: pvmove segfaults in find_pvmove_lv, no data loss

Package:
lvm2
Source:
lvm2
Description:
Linux Logical Volume Manager
Submitter:
Marc Lehmann
Date:
2017-11-05 12:45:06 UTC
Severity:
normal
#775169#5
Date:
2015-01-12 06:11:53 UTC
From:
To:
Dear Maintainer,

I tried to pvmove some lvs off of a device, everything from sdb2 (root
filesystem, swap, journals and dm-caches for other partitions) to sdf1:

  PV         VG         Fmt  Attr PSize   PFree
  /dev/sdb2  vg_cerebro lvm2 a--  400.00g 206.09g
  /dev/sdc1  vg_cerebro lvm2 a--   25.47t   4.00m
  /dev/sdf1  vg_cerebro lvm2 a--  220.00g 220.00g

these commands all segfault in find_pvmove_lv:

   pvmove /dev/sdb2
   pvmove /dev/sdb2 /dev/sdf1
   pvmove -n root /dev/sdb2 /dev/sdf1

Backtrace is not very englightening:

   Program received signal SIGSEGV, Segmentation fault.
   0x00005555555f401e in find_pvmove_lv ()
   (gdb) bt
   #0  0x00005555555f401e in find_pvmove_lv ()
   #1  0x0000555555591194 in pvmove ()
   #2  0x000055555558b248 in lvm_run_command ()
   #3  0x000055555558b8de in lvm2_main ()
   #4  0x00007fffef394b45 in __libc_start_main (main=0x5555555766f0 <main>, argc=2, argv=0x7fffffffe108, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe0f8) at libc-start.c:287
   #5  0x000055555557671e in _start ()

The segfault seemed to occur early enough before any action is taken, and
also happens in test mode. pvmove -t -vvv /dev/sdf1 outputs this before it
segfaults:

   #libdm-config.c:846       Setting data to localvol_cache_cdata
   #libdm-config.c:846       Setting metadata to localvol_cache_cmeta
   #libdm-config.c:846       Setting cache_mode to writeback
   #metadata/lv_manip.c:1183       Stack localvol_cache:0[0] on LV localvol_cache_cdata:0
   #metadata/lv_manip.c:770       Adding localvol_cache:0 as an user of localvol_cache_cdata
   #metadata/lv_manip.c:770       Adding localvol_cache:0 as an user of localvol_cache_cmeta
   Segmentation fault

I use pvmove rarely, but regularly. My suspicion is that it is dm-cache
related, as introducing caches was the only thing I changed to my lvm
config, but of course it could be completely unrelated.

lvs -a output follows:

  LV                     Attr       LSize   Devices
  bp                     Cwi-aoC---   6.80t bp_corig(0)
  [bp_cache]             Cwi---C---  15.00g bp_cache_cdata(0)
  [bp_cache_cdata]       Cwi-ao----  15.00g /dev/sdb2(37696)
  [bp_cache_cmeta]       ewi-ao---- 180.00m /dev/sdb2(16868)
  [bp_corig]             owi---C---   6.80t /dev/sdc1(4718592)
  db                     -wi-ao----  32.00g /dev/sdb2(21760)
  localvol               Cwi-aoC--- 681.75g localvol_corig(0)
  [localvol_cache]       Cwi---C---   8.00g localvol_cache_cdata(0)
  [localvol_cache_cdata] Cwi-ao----   8.00g /dev/sdb2(18723)
  [localvol_cache_cmeta] ewi-ao----  12.00m /dev/sdb2(20771)
  localvol_cachemeta     -wi-a-----  12.00m /dev/sdb2(18720)
  [localvol_corig]       owi---C--- 681.75g /dev/sdc1(6501172)
  log_bp                 -wi-ao---- 128.00m /dev/sdb2(29984)
  log_localvol           -wi-ao---- 128.00m /dev/sdb2(29952)
  log_wd                 -wi-ao---- 128.00m /dev/sdb2(16640)
  [lvol0_pmspare]        ewi------- 680.00m /dev/sdb2(20774)
  root                   -wi-ao----  65.00g /dev/sdb2(0)
  swap                   -wi-ao----  32.00g /dev/sdb2(41536)
  wd                     Cwi-aoC---  18.00t wd_corig(0)
  [wd_cache]             Cwi---C---  40.00g wd_cache_cdata(0)
  [wd_cache_cdata]       Cwi-ao----  40.00g /dev/sdb2(49728)
  [wd_cache_cmeta]       ewi-ao---- 680.00m /dev/sdb2(16672)
  [wd_corig]             owi---C---  18.00t /dev/sdc1(0)

#775169#10
Date:
2017-11-05 12:11:48 UTC
From:
To:
I was encountering same, or highly similar issue,
notably pvmove segfaults.  Currently on Debian oldstable.
I also found the relatively informative similar/related thread:
https://lists.debian.org/debian-user/2011/06/msg01947.html
https://lists.debian.org/debian-user/2011/06/msg02076.html

My situation "the same", or highly similar:
pvmove segfaults
lvs -a
and
lvdisplay -a
show a [pvmove0] volume
attempts at:
pvmove --abort
lvremove [-f] ...pvmove0
lvchange ... pvmove0
lvmremove [-f] ... pvmove0
all failed
The threat mentioned above had suggestion to remove the dm device,
however, using blkid on all dm devices, I found no DM device with
the UUID shown by lvdisplay -a
nor any devices/links with or containing name pvmove0, and pvmove0
(in lvdisplay -a, lvs -a)
also showed it as size 0, and everything LVM related reported it
as locked and inactive.  Not sure what created this situation, but ..

workaround for the above:
I ran vgcfgbackup twice (so I'd have both "current" config saved,
and one slightly older archive saved copy of same).
I then edited the most recent saved copy, removing the pvmove0 section.
I then used vgcfgrestore.
After that, all was fine and good.  :-)  No more complaints or signs
about pvmove0, and pvmove worked again fine and as expected.

Suggestion - see if something can be added to the lvm code to recognize this
situation and self-correct (or even prevent).  It may also be quite feasible
to reproduce the problem, by injecting such data into a file created by
vgcfgbackup, then using vgcfgrestore (I've not attempted that).  But,
in case that may be useful, here's the section I'd removed:
# pwd -P; sed -ne '/pvmove0 {/,/}/p' tigger_00754-383252791.vg
/etc/lvm/archive
                 pvmove0 {
                         id = "23mVuC-uEaK-gB3Z-9WU0-nYbM-91ls-kQ9KXf"
                         status = ["READ", "WRITE", "PVMOVE", "LOCKED"]
                         flags = []
                         allocation_policy = "contiguous"
                         segment_count = 0

                 }
#