In the transcript below:

  * I create a snapshot (of an idle volume, not mounted);

  * I inspect the devmapper mappings;

  * I try to remove the snapshot, but LVM fails partway and can't put the
    pieces back together.

  * Try again: this time LVM seems to follow through to completion.

  * I inspect the mappings again: the original volume has been put back OK,
    but the `real' mapping whcih `lvremove' complained about earlier is left
    over.

  * Finally, I delete the leftover mapping by hand.

If I stop udev before removing the snapshot then it all works OK, except that
dangling symlinks are left in /dev/vg-ibanez and /dev/mapper, so this may
well be related to #549691, #618016 and #641259.

Throughout all of this, the machine is essentially idle.  There are no other
users, no externally visible services other than SSH, no significant I/O, ...
Nothing touched either the original LV or the snapshot in between creation
and (attempted) removal.


[ibanez ~]r lvcreate --verbose --snapshot --size=4G --name=news.snap vg-ibanez/news
    Setting chunksize to 8 sectors.
    Setting logging type to disk
    Finding volume group "vg-ibanez"
    Archiving volume group "vg-ibanez" metadata (seqno 58).
    Creating logical volume news.snap
    Creating volume group backup "/etc/lvm/backup/vg-ibanez" (seqno 59).
    Found volume group "vg-ibanez"
    activation/volume_list configuration setting not defined, checking only host tags for vg-ibanez/news.snap
    Creating vg--ibanez-news.snap
    Loading vg--ibanez-news.snap table (253:19)
    Resuming vg--ibanez-news.snap (253:19)
    Clearing start of logical volume "news.snap"
    Creating logical volume snapshot0
    Found volume group "vg-ibanez"
    Found volume group "vg-ibanez"
    Creating vg--ibanez-news-real
    Loading vg--ibanez-news-real table (253:21)
    Resuming vg--ibanez-news-real (253:21)
    Loading vg--ibanez-news table (253:20)
    Creating vg--ibanez-news.snap-cow
    Loading vg--ibanez-news.snap-cow table (253:22)
    Resuming vg--ibanez-news.snap-cow (253:22)
    Loading vg--ibanez-news.snap table (253:19)
    Suspending vg--ibanez-news (253:20) with filesystem sync with device flush
    Suspending vg--ibanez-news-real (253:21) with filesystem sync with device flush
    Found volume group "vg-ibanez"
    Loading vg--ibanez-news-real table (253:21)
    Suppressed vg--ibanez-news-real identical table reload.
    Loading vg--ibanez-news.snap-cow table (253:22)
    Suppressed vg--ibanez-news.snap-cow identical table reload.
    Resuming vg--ibanez-news-real (253:21)
    Resuming vg--ibanez-news.snap (253:19)
    Resuming vg--ibanez-news (253:20)
    Creating volume group backup "/etc/lvm/backup/vg-ibanez" (seqno 60).
  Logical volume "news.snap" created
[ibanez ~]r dmsetup table vg--ibanez-news
0 67108864 snapshot-origin 253:21
[ibanez ~]r dmsetup table vg--ibanez-news-real
0 67108864 linear 253:0 5236591616
[ibanez ~]r lvremove --verbose --force vg-ibanez/news.snap
    Using logical volume(s) on command line
    Archiving volume group "vg-ibanez" metadata (seqno 60).
    Removing snapshot news.snap
    Found volume group "vg-ibanez"
    Found volume group "vg-ibanez"
    Loading vg--ibanez-news table (253:20)
    Loading vg--ibanez-news.snap table (253:19)
    Suspending vg--ibanez-news (253:20) with device flush
    Suspending vg--ibanez-news.snap (253:19) with device flush
    Suspending vg--ibanez-news-real (253:21) with device flush
    Suspending vg--ibanez-news.snap-cow (253:22) with device flush
    Found volume group "vg-ibanez"
    Resuming vg--ibanez-news.snap-cow (253:22)
    Resuming vg--ibanez-news-real (253:21)
    Resuming vg--ibanez-news.snap (253:19)
    Removing vg--ibanez-news.snap-cow (253:22)
    Found volume group "vg-ibanez"
    Resuming vg--ibanez-news (253:20)
  Unable to deactivate open vg--ibanez-news-real (253:21)
  Failed to resume news.
[ibanez ~]r dmsetup table vg--ibanez-news
0 67108864 linear 253:0 5236591616
[ibanez ~]r dmsetup table vg--ibanez-news-real
0 67108864 linear 253:0 5236591616
[ibanez ~]r lvremove --verbose --force vg-ibanez/news.snap
    Using logical volume(s) on command line
    Archiving volume group "vg-ibanez" metadata (seqno 61).
    Found volume group "vg-ibanez"
    Removing vg--ibanez-news.snap (253:19)
    Releasing logical volume "news.snap"
    Creating volume group backup "/etc/lvm/backup/vg-ibanez" (seqno 62).
  Logical volume "news.snap" successfully removed
[ibanez ~]r dmsetup table vg--ibanez-news
0 67108864 linear 253:0 5236591616
[ibanez ~]r dmsetup table vg--ibanez-news-real
0 67108864 linear 253:0 5236591616
[ibanez ~]r dmsetup remove vg--ibanez-news-real
[ibanez ~]

I'm not enjoying the lvm source code much.  There's obviously some kind
of race going on: if I stop for a short while at breakpoints at the two
`lv_resume' calls in `vg_remove_snapshot' (lines 218 and 222 of
lib/metadata/snapshot_manip.c) then everything works OK.

If I downgrade to 2.02.66-5 (stable), then `lvremove' succeeds, but
isn't happy:

[ibanez ~]r lvremove -vf vg-ibanez/news.snap
    Using logical volume(s) on command line
    Archiving volume group "vg-ibanez" metadata (seqno 154).
    Removing snapshot news.snap
    Found volume group "vg-ibanez"
    Removing vg--ibanez-news.snap (253:19)
    Removing vg--ibanez-news.snap-cow (253:22)
    Found volume group "vg-ibanez"
    Releasing logical volume "news.snap"
    Found volume group "vg-ibanez"
    Found volume group "vg-ibanez"
    Suspending vg--ibanez-news (253:20) with device flush
    Suspending vg--ibanez-news-real (253:21) with device flush
    Found volume group "vg-ibanez"
    Loading vg--ibanez-news table (253:20)
  Internal error: Performing unsafe table load while 2 device(s) are known to be suspended:  (253:20)
    Resuming vg--ibanez-news-real (253:21)
    Resuming vg--ibanez-news (253:20)
    Removing vg--ibanez-news-real (253:21)
    Creating volume group backup "/etc/lvm/backup/vg-ibanez" (seqno 155).
  Logical volume "news.snap" successfully removed

I also edited 60-persistent-storage.rules to disable setting the `watch'
option as suggested in

https://bugzilla.redhat.com/show_bug.cgi?id=577798#c5

though this didn't seem to make any difference.