#942579 qemu-utils: nbd disconnect/timout/system hang when writing to image mounted with qemu-nbd

Package:
qemu-utils
Source:
qemu
Description:
QEMU utilities
Submitter:
Marcel
Date:
2021-06-12 12:03:03 UTC
Severity:
normal
#942579#5
Date:
2019-10-18 12:36:18 UTC
From:
To:
Dear Maintainer,

I experience the following syptoms since Debian buster. In Debian stretch it
works just fine:

Load Kernel Module: sudo modprobe nbd max_part=16
Create an image: qemeu-image create vdi /path/to/img-file.vdi 120G
Bind the image: qemeu-nbd -c /dev/nbd0 /path/to/img-file.vdi

Create a partion, e.g.: cfdisk /dev/nbd0
Format the partition: sudo mkfs.ext4 /dev/nbd0p1
Mount the partition: sudo mount /dev/nbd0p1 /mnt/nbd0

Now, copy over a large file, e.g. 20G. After about 10GB the procedure will stop
and there will be things like the following in /var/log/kernel.log


Oct 18 12:04:07 free kernel: [  449.449298] block nbd1: Connection timed out
Oct 18 12:06:42 free kernel: [  605.016877] INFO: task kworker/u2:4:142 blocked
for more than 120 seconds.
Oct 18 12:06:42 free kernel: [  605.016882]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:06:42 free kernel: [  605.016883] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:06:42 free kernel: [  605.016885] kworker/u2:4    D    0   142      2
0x80004000
Oct 18 12:06:42 free kernel: [  605.016904] Workqueue: writeback wb_workfn
(flush-254:3)
Oct 18 12:06:42 free kernel: [  605.016906] Call Trace:
Oct 18 12:06:42 free kernel: [  605.016925]  ? __schedule+0x2bb/0x660
Oct 18 12:06:42 free kernel: [  605.016931]  ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:06:42 free kernel: [  605.016933]  ? wbt_exit+0x30/0x30
Oct 18 12:06:42 free kernel: [  605.016935]  ? __wbt_done+0x30/0x30
Oct 18 12:06:42 free kernel: [  605.016937]  schedule+0x2f/0xa0
Oct 18 12:06:42 free kernel: [  605.016939]  io_schedule+0x12/0x40
Oct 18 12:06:42 free kernel: [  605.016942]  rq_qos_wait+0xfe/0x160
Oct 18 12:06:42 free kernel: [  605.016955]  ? karma_partition+0x210/0x210
Oct 18 12:06:42 free kernel: [  605.016957]  ? wbt_exit+0x30/0x30
Oct 18 12:06:42 free kernel: [  605.016959]  wbt_wait+0x99/0xe0
Oct 18 12:06:42 free kernel: [  605.016962]  __rq_qos_throttle+0x23/0x30
Oct 18 12:06:42 free kernel: [  605.016964]  blk_mq_make_request+0x11d/0x590
Oct 18 12:06:42 free kernel: [  605.016973]  ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:06:42 free kernel: [  605.016975]  generic_make_request+0xcf/0x310
Oct 18 12:06:42 free kernel: [  605.016978]  submit_bio+0x45/0x140
Oct 18 12:06:42 free kernel: [  605.016998]  ext4_io_submit+0x49/0x60 [ext4]
Oct 18 12:06:42 free kernel: [  605.017014]  ext4_writepages+0x707/0xeb0 [ext4]
Oct 18 12:06:42 free kernel: [  605.017023]  ? check_preempt_wakeup+0x18d/0x230
Oct 18 12:06:42 free kernel: [  605.017028]  ? do_writepages+0x41/0xd0
Oct 18 12:06:42 free kernel: [  605.017042]  ?
ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Oct 18 12:06:42 free kernel: [  605.017045]  do_writepages+0x41/0xd0
Oct 18 12:06:42 free kernel: [  605.017055]  ? __wake_up_common_lock+0x89/0xc0
Oct 18 12:06:42 free kernel: [  605.017058]
__writeback_single_inode+0x3d/0x340
Oct 18 12:06:42 free kernel: [  605.017067]  writeback_sb_inodes+0x1e5/0x480
Oct 18 12:06:42 free kernel: [  605.017070]  __writeback_inodes_wb+0x5d/0xb0
Oct 18 12:06:42 free kernel: [  605.017072]  wb_writeback+0x25f/0x2f0
Oct 18 12:06:42 free kernel: [  605.017075]  wb_workfn+0x30c/0x400
Oct 18 12:06:42 free kernel: [  605.017078]  process_one_work+0x1a7/0x3b0
Oct 18 12:06:42 free kernel: [  605.017086]  worker_thread+0x1fa/0x390
Oct 18 12:06:42 free kernel: [  605.017087]  ? create_worker+0x1a0/0x1a0
Oct 18 12:06:42 free kernel: [  605.017090]  kthread+0x112/0x130
Oct 18 12:06:42 free kernel: [  605.017092]  ? __kthread_parkme+0x70/0x70
Oct 18 12:06:42 free kernel: [  605.017094]  ret_from_fork+0x35/0x40
Oct 18 12:06:42 free kernel: [  605.017137] INFO: task kworker/0:2H:2962
blocked for more than 120 seconds.
Oct 18 12:06:42 free kernel: [  605.017140]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:06:42 free kernel: [  605.017141] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:06:42 free kernel: [  605.017143] kworker/0:2H    D    0  2962      2
0x80004000
Oct 18 12:06:42 free kernel: [  605.017148] Workqueue: kblockd
blk_mq_timeout_work
Oct 18 12:06:42 free kernel: [  605.017149] Call Trace:
Oct 18 12:06:42 free kernel: [  605.017153]  ? __schedule+0x2bb/0x660
Oct 18 12:06:42 free kernel: [  605.017156]  schedule+0x2f/0xa0
Oct 18 12:06:42 free kernel: [  605.017159]  schedule_preempt_disabled+0xa/0x10
Oct 18 12:06:42 free kernel: [  605.017162]  __mutex_lock.isra.8+0x2b5/0x4a0
Oct 18 12:06:42 free kernel: [  605.017165]  ?
blk_mq_dispatch_rq_list+0x48d/0x5c0
Oct 18 12:06:42 free kernel: [  605.017171]  sock_shutdown+0x58/0x80 [nbd]
Oct 18 12:06:42 free kernel: [  605.017174]  nbd_xmit_timeout+0xb7/0x150 [nbd]
Oct 18 12:06:42 free kernel: [  605.017176]  blk_mq_check_expired+0xe7/0x100
Oct 18 12:06:42 free kernel: [  605.017179]
blk_mq_queue_tag_busy_iter+0x19a/0x2f0
Oct 18 12:06:42 free kernel: [  605.017181]  ? blk_poll+0x350/0x350
Oct 18 12:06:42 free kernel: [  605.017182]  ? __switch_to_asm+0x34/0x70
Oct 18 12:06:42 free kernel: [  605.017184]  ? blk_poll+0x350/0x350
Oct 18 12:06:42 free kernel: [  605.017186]  ? __switch_to+0x10/0x3e0
Oct 18 12:06:42 free kernel: [  605.017188]  blk_mq_timeout_work+0x54/0x120
Oct 18 12:06:42 free kernel: [  605.017190]  process_one_work+0x1a7/0x3b0
Oct 18 12:06:42 free kernel: [  605.017191]  worker_thread+0x30/0x390
Oct 18 12:06:42 free kernel: [  605.017193]  ? create_worker+0x1a0/0x1a0
Oct 18 12:06:42 free kernel: [  605.017195]  kthread+0x112/0x130
Oct 18 12:06:42 free kernel: [  605.017197]  ? __kthread_parkme+0x70/0x70
Oct 18 12:06:42 free kernel: [  605.017199]  ret_from_fork+0x35/0x40
Oct 18 12:06:42 free kernel: [  605.017201] INFO: task jbd2/dm-3-8:3222 blocked
for more than 120 seconds.
Oct 18 12:06:42 free kernel: [  605.017203]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:06:42 free kernel: [  605.017204] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:06:42 free kernel: [  605.017205] jbd2/dm-3-8     D    0  3222      2
0x80004000
Oct 18 12:06:42 free kernel: [  605.017207] Call Trace:
Oct 18 12:06:42 free kernel: [  605.017209]  ? __schedule+0x2bb/0x660
Oct 18 12:06:42 free kernel: [  605.017212]  ? bit_wait_timeout+0x90/0x90
Oct 18 12:06:42 free kernel: [  605.017213]  schedule+0x2f/0xa0
Oct 18 12:06:42 free kernel: [  605.017215]  io_schedule+0x12/0x40
Oct 18 12:06:42 free kernel: [  605.017217]  bit_wait_io+0xd/0x50
Oct 18 12:06:42 free kernel: [  605.017219]  __wait_on_bit+0x73/0x90
Oct 18 12:06:42 free kernel: [  605.017222]  out_of_line_wait_on_bit+0x91/0xb0
Oct 18 12:06:42 free kernel: [  605.017224]  ? init_wait_var_entry+0x40/0x40
Oct 18 12:06:42 free kernel: [  605.017230]
jbd2_journal_commit_transaction+0x14b5/0x1890 [jbd2]
Oct 18 12:06:42 free kernel: [  605.017232]  ? __switch_to_asm+0x40/0x70
Oct 18 12:06:42 free kernel: [  605.017239]  kjournald2+0xbd/0x270 [jbd2]
Oct 18 12:06:42 free kernel: [  605.017242]  ? finish_wait+0x80/0x80
Oct 18 12:06:42 free kernel: [  605.017247]  ? commit_timeout+0x10/0x10 [jbd2]
Oct 18 12:06:42 free kernel: [  605.017249]  kthread+0x112/0x130
Oct 18 12:06:42 free kernel: [  605.017251]  ? __kthread_parkme+0x70/0x70
Oct 18 12:06:42 free kernel: [  605.017252]  ret_from_fork+0x35/0x40
Oct 18 12:06:42 free kernel: [  605.017254] INFO: task ext4lazyinit:3224
blocked for more than 120 seconds.
Oct 18 12:06:42 free kernel: [  605.017256]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:06:42 free kernel: [  605.017256] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:06:42 free kernel: [  605.017258] ext4lazyinit    D    0  3224      2
0x80004000
Oct 18 12:06:42 free kernel: [  605.017259] Call Trace:
Oct 18 12:06:42 free kernel: [  605.017261]  ? __schedule+0x2bb/0x660
Oct 18 12:06:42 free kernel: [  605.017263]  ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:06:42 free kernel: [  605.017265]  ? wbt_exit+0x30/0x30
Oct 18 12:06:42 free kernel: [  605.017267]  ? __wbt_done+0x30/0x30
Oct 18 12:06:42 free kernel: [  605.017269]  schedule+0x2f/0xa0
Oct 18 12:06:42 free kernel: [  605.017271]  io_schedule+0x12/0x40
Oct 18 12:06:42 free kernel: [  605.017273]  rq_qos_wait+0xfe/0x160
Oct 18 12:06:42 free kernel: [  605.017275]  ? karma_partition+0x210/0x210
Oct 18 12:06:42 free kernel: [  605.017276]  ? wbt_exit+0x30/0x30
Oct 18 12:06:42 free kernel: [  605.017278]  wbt_wait+0x99/0xe0
Oct 18 12:06:42 free kernel: [  605.017281]  __rq_qos_throttle+0x23/0x30
Oct 18 12:06:42 free kernel: [  605.017282]  blk_mq_make_request+0x11d/0x590
Oct 18 12:06:42 free kernel: [  605.017288]  ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:06:42 free kernel: [  605.017290]  generic_make_request+0xcf/0x310
Oct 18 12:06:42 free kernel: [  605.017292]  submit_bio+0x45/0x140
Oct 18 12:06:42 free kernel: [  605.017294]  blk_next_bio+0x33/0x40
Oct 18 12:06:42 free kernel: [  605.017296]
__blkdev_issue_zero_pages+0x90/0x190
Oct 18 12:06:42 free kernel: [  605.017297]  blkdev_issue_zeroout+0xef/0x220
Oct 18 12:06:42 free kernel: [  605.017312]  ext4_init_inode_table+0x17b/0x390
[ext4]
Oct 18 12:06:42 free kernel: [  605.017329]  ext4_lazyinit_thread+0x2bd/0x3a0
[ext4]
Oct 18 12:06:42 free kernel: [  605.017343]  ?
ext4_unregister_li_request+0x60/0x60 [ext4]
Oct 18 12:06:42 free kernel: [  605.017346]  kthread+0x112/0x130
Oct 18 12:06:42 free kernel: [  605.017348]  ? __kthread_parkme+0x70/0x70
Oct 18 12:06:42 free kernel: [  605.017349]  ret_from_fork+0x35/0x40
Oct 18 12:08:43 free kernel: [  725.788441] INFO: task kworker/u2:4:142 blocked
for more than 241 seconds.
Oct 18 12:08:43 free kernel: [  725.788448]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:08:43 free kernel: [  725.788450] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:08:43 free kernel: [  725.788453] kworker/u2:4    D    0   142      2
0x80004000
Oct 18 12:08:43 free kernel: [  725.788465] Workqueue: writeback wb_workfn
(flush-254:3)
Oct 18 12:08:43 free kernel: [  725.788468] Call Trace:
Oct 18 12:08:43 free kernel: [  725.788479]  ? __schedule+0x2bb/0x660
Oct 18 12:08:43 free kernel: [  725.788483]  ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:08:43 free kernel: [  725.788487]  ? wbt_exit+0x30/0x30
Oct 18 12:08:43 free kernel: [  725.788490]  ? __wbt_done+0x30/0x30
Oct 18 12:08:43 free kernel: [  725.788493]  schedule+0x2f/0xa0
Oct 18 12:08:43 free kernel: [  725.788496]  io_schedule+0x12/0x40
Oct 18 12:08:43 free kernel: [  725.788500]  rq_qos_wait+0xfe/0x160
Oct 18 12:08:43 free kernel: [  725.788503]  ? karma_partition+0x210/0x210
Oct 18 12:08:43 free kernel: [  725.788506]  ? wbt_exit+0x30/0x30
Oct 18 12:08:43 free kernel: [  725.788509]  wbt_wait+0x99/0xe0
Oct 18 12:08:43 free kernel: [  725.788513]  __rq_qos_throttle+0x23/0x30
Oct 18 12:08:43 free kernel: [  725.788516]  blk_mq_make_request+0x11d/0x590
Oct 18 12:08:43 free kernel: [  725.788529]  ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:08:43 free kernel: [  725.788533]  generic_make_request+0xcf/0x310
Oct 18 12:08:43 free kernel: [  725.788536]  submit_bio+0x45/0x140
Oct 18 12:08:43 free kernel: [  725.788568]  ext4_io_submit+0x49/0x60 [ext4]
Oct 18 12:08:43 free kernel: [  725.788592]  ext4_writepages+0x707/0xeb0 [ext4]
Oct 18 12:08:43 free kernel: [  725.788600]  ? check_preempt_wakeup+0x18d/0x230
Oct 18 12:08:43 free kernel: [  725.788606]  ? do_writepages+0x41/0xd0
Oct 18 12:08:43 free kernel: [  725.788627]  ?
ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Oct 18 12:08:43 free kernel: [  725.788631]  do_writepages+0x41/0xd0
Oct 18 12:08:43 free kernel: [  725.788635]  ? __wake_up_common_lock+0x89/0xc0
Oct 18 12:08:43 free kernel: [  725.788640]
__writeback_single_inode+0x3d/0x340
Oct 18 12:08:43 free kernel: [  725.788644]  writeback_sb_inodes+0x1e5/0x480
Oct 18 12:08:43 free kernel: [  725.788648]  __writeback_inodes_wb+0x5d/0xb0
Oct 18 12:08:43 free kernel: [  725.788652]  wb_writeback+0x25f/0x2f0
Oct 18 12:08:43 free kernel: [  725.788656]  wb_workfn+0x30c/0x400
Oct 18 12:08:43 free kernel: [  725.788660]  process_one_work+0x1a7/0x3b0
Oct 18 12:08:43 free kernel: [  725.788663]  worker_thread+0x1fa/0x390
Oct 18 12:08:43 free kernel: [  725.788665]  ? create_worker+0x1a0/0x1a0
Oct 18 12:08:43 free kernel: [  725.788668]  kthread+0x112/0x130
Oct 18 12:08:43 free kernel: [  725.788672]  ? __kthread_parkme+0x70/0x70
Oct 18 12:08:43 free kernel: [  725.788675]  ret_from_fork+0x35/0x40
Oct 18 12:08:43 free kernel: [  725.788726] INFO: task kworker/0:2H:2962
blocked for more than 241 seconds.
Oct 18 12:08:43 free kernel: [  725.788729]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:08:43 free kernel: [  725.788730] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:08:43 free kernel: [  725.788732] kworker/0:2H    D    0  2962      2
0x80004000
Oct 18 12:08:43 free kernel: [  725.788738] Workqueue: kblockd
blk_mq_timeout_work
Oct 18 12:08:43 free kernel: [  725.788740] Call Trace:
Oct 18 12:08:43 free kernel: [  725.788744]  ? __schedule+0x2bb/0x660
Oct 18 12:08:43 free kernel: [  725.788748]  schedule+0x2f/0xa0
Oct 18 12:08:43 free kernel: [  725.788751]  schedule_preempt_disabled+0xa/0x10
Oct 18 12:08:43 free kernel: [  725.788754]  __mutex_lock.isra.8+0x2b5/0x4a0
Oct 18 12:08:43 free kernel: [  725.788758]  ?
blk_mq_dispatch_rq_list+0x48d/0x5c0
Oct 18 12:08:43 free kernel: [  725.788766]  sock_shutdown+0x58/0x80 [nbd]
Oct 18 12:08:43 free kernel: [  725.788771]  nbd_xmit_timeout+0xb7/0x150 [nbd]
Oct 18 12:08:43 free kernel: [  725.788774]  blk_mq_check_expired+0xe7/0x100
Oct 18 12:08:43 free kernel: [  725.788778]
blk_mq_queue_tag_busy_iter+0x19a/0x2f0
Oct 18 12:08:43 free kernel: [  725.788781]  ? blk_poll+0x350/0x350
Oct 18 12:08:43 free kernel: [  725.788783]  ? __switch_to_asm+0x34/0x70
Oct 18 12:08:43 free kernel: [  725.788786]  ? blk_poll+0x350/0x350
Oct 18 12:08:43 free kernel: [  725.788790]  ? __switch_to+0x10/0x3e0
Oct 18 12:08:43 free kernel: [  725.788793]  blk_mq_timeout_work+0x54/0x120
Oct 18 12:08:43 free kernel: [  725.788796]  process_one_work+0x1a7/0x3b0
Oct 18 12:08:43 free kernel: [  725.788798]  worker_thread+0x30/0x390
Oct 18 12:08:43 free kernel: [  725.788801]  ? create_worker+0x1a0/0x1a0
Oct 18 12:08:43 free kernel: [  725.788803]  kthread+0x112/0x130
Oct 18 12:08:43 free kernel: [  725.788806]  ? __kthread_parkme+0x70/0x70
Oct 18 12:08:43 free kernel: [  725.788809]  ret_from_fork+0x35/0x40
Oct 18 12:08:43 free kernel: [  725.788812] INFO: task jbd2/dm-3-8:3222 blocked
for more than 241 seconds.
Oct 18 12:08:43 free kernel: [  725.788815]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:08:43 free kernel: [  725.788816] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:08:43 free kernel: [  725.788818] jbd2/dm-3-8     D    0  3222      2
0x80004000
Oct 18 12:08:43 free kernel: [  725.788820] Call Trace:
Oct 18 12:08:43 free kernel: [  725.788824]  ? __schedule+0x2bb/0x660
Oct 18 12:08:43 free kernel: [  725.788828]  ? bit_wait_timeout+0x90/0x90
Oct 18 12:08:43 free kernel: [  725.788830]  schedule+0x2f/0xa0
Oct 18 12:08:43 free kernel: [  725.788833]  io_schedule+0x12/0x40
Oct 18 12:08:43 free kernel: [  725.788836]  bit_wait_io+0xd/0x50
Oct 18 12:08:43 free kernel: [  725.788840]  __wait_on_bit+0x73/0x90
Oct 18 12:08:43 free kernel: [  725.788843]  out_of_line_wait_on_bit+0x91/0xb0
Oct 18 12:08:43 free kernel: [  725.788847]  ? init_wait_var_entry+0x40/0x40
Oct 18 12:08:43 free kernel: [  725.788857]
jbd2_journal_commit_transaction+0x14b5/0x1890 [jbd2]
Oct 18 12:08:43 free kernel: [  725.788860]  ? __switch_to_asm+0x40/0x70
Oct 18 12:08:43 free kernel: [  725.788870]  kjournald2+0xbd/0x270 [jbd2]
Oct 18 12:08:43 free kernel: [  725.788875]  ? finish_wait+0x80/0x80
Oct 18 12:08:43 free kernel: [  725.788882]  ? commit_timeout+0x10/0x10 [jbd2]
Oct 18 12:08:43 free kernel: [  725.788885]  kthread+0x112/0x130
Oct 18 12:08:43 free kernel: [  725.788888]  ? __kthread_parkme+0x70/0x70
Oct 18 12:08:43 free kernel: [  725.788891]  ret_from_fork+0x35/0x40
Oct 18 12:08:43 free kernel: [  725.788894] INFO: task ext4lazyinit:3224
blocked for more than 241 seconds.
Oct 18 12:08:43 free kernel: [  725.788896]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:08:43 free kernel: [  725.788897] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:08:43 free kernel: [  725.788899] ext4lazyinit    D    0  3224      2
0x80004000
Oct 18 12:08:43 free kernel: [  725.788901] Call Trace:
Oct 18 12:08:43 free kernel: [  725.788905]  ? __schedule+0x2bb/0x660
Oct 18 12:08:43 free kernel: [  725.788908]  ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:08:43 free kernel: [  725.788911]  ? wbt_exit+0x30/0x30
Oct 18 12:08:43 free kernel: [  725.788914]  ? __wbt_done+0x30/0x30
Oct 18 12:08:43 free kernel: [  725.788917]  schedule+0x2f/0xa0
Oct 18 12:08:43 free kernel: [  725.788920]  io_schedule+0x12/0x40
Oct 18 12:08:43 free kernel: [  725.788923]  rq_qos_wait+0xfe/0x160
Oct 18 12:08:43 free kernel: [  725.788926]  ? karma_partition+0x210/0x210
Oct 18 12:08:43 free kernel: [  725.788929]  ? wbt_exit+0x30/0x30
Oct 18 12:08:43 free kernel: [  725.788931]  wbt_wait+0x99/0xe0
Oct 18 12:08:43 free kernel: [  725.788935]  __rq_qos_throttle+0x23/0x30
Oct 18 12:08:43 free kernel: [  725.788938]  blk_mq_make_request+0x11d/0x590
Oct 18 12:08:43 free kernel: [  725.788946]  ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:08:43 free kernel: [  725.788949]  generic_make_request+0xcf/0x310
Oct 18 12:08:43 free kernel: [  725.788953]  submit_bio+0x45/0x140
Oct 18 12:08:43 free kernel: [  725.788955]  blk_next_bio+0x33/0x40
Oct 18 12:08:43 free kernel: [  725.788958]
__blkdev_issue_zero_pages+0x90/0x190
Oct 18 12:08:43 free kernel: [  725.788960]  blkdev_issue_zeroout+0xef/0x220
Oct 18 12:08:43 free kernel: [  725.788984]  ext4_init_inode_table+0x17b/0x390
[ext4]
Oct 18 12:08:43 free kernel: [  725.789010]  ext4_lazyinit_thread+0x2bd/0x3a0
[ext4]
Oct 18 12:08:43 free kernel: [  725.789035]  ?
ext4_unregister_li_request+0x60/0x60 [ext4]
Oct 18 12:08:43 free kernel: [  725.789038]  kthread+0x112/0x130
Oct 18 12:08:43 free kernel: [  725.789041]  ? __kthread_parkme+0x70/0x70
Oct 18 12:08:43 free kernel: [  725.789044]  ret_from_fork+0x35/0x40
Oct 18 12:10:44 free kernel: [  846.559910] INFO: task kworker/u2:4:142 blocked
for more than 362 seconds.
Oct 18 12:10:44 free kernel: [  846.559917]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:10:44 free kernel: [  846.559918] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:10:44 free kernel: [  846.559921] kworker/u2:4    D    0   142      2
0x80004000
Oct 18 12:10:44 free kernel: [  846.559931] Workqueue: writeback wb_workfn
(flush-254:3)
Oct 18 12:10:44 free kernel: [  846.559933] Call Trace:
Oct 18 12:10:44 free kernel: [  846.559943]  ? __schedule+0x2bb/0x660
Oct 18 12:10:44 free kernel: [  846.559946]  ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:10:44 free kernel: [  846.559949]  ? wbt_exit+0x30/0x30
Oct 18 12:10:44 free kernel: [  846.559952]  ? __wbt_done+0x30/0x30
Oct 18 12:10:44 free kernel: [  846.559954]  schedule+0x2f/0xa0
Oct 18 12:10:44 free kernel: [  846.559957]  io_schedule+0x12/0x40
Oct 18 12:10:44 free kernel: [  846.559960]  rq_qos_wait+0xfe/0x160
Oct 18 12:10:44 free kernel: [  846.559963]  ? karma_partition+0x210/0x210
Oct 18 12:10:44 free kernel: [  846.559966]  ? wbt_exit+0x30/0x30
Oct 18 12:10:44 free kernel: [  846.559968]  wbt_wait+0x99/0xe0
Oct 18 12:10:44 free kernel: [  846.559971]  __rq_qos_throttle+0x23/0x30
Oct 18 12:10:44 free kernel: [  846.559974]  blk_mq_make_request+0x11d/0x590
Oct 18 12:10:44 free kernel: [  846.559984]  ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:10:44 free kernel: [  846.559987]  generic_make_request+0xcf/0x310
Oct 18 12:10:44 free kernel: [  846.559990]  submit_bio+0x45/0x140
Oct 18 12:10:44 free kernel: [  846.560016]  ext4_io_submit+0x49/0x60 [ext4]
Oct 18 12:10:44 free kernel: [  846.560035]  ext4_writepages+0x707/0xeb0 [ext4]
Oct 18 12:10:44 free kernel: [  846.560042]  ? check_preempt_wakeup+0x18d/0x230
Oct 18 12:10:44 free kernel: [  846.560047]  ? do_writepages+0x41/0xd0
Oct 18 12:10:44 free kernel: [  846.560064]  ?
ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Oct 18 12:10:44 free kernel: [  846.560067]  do_writepages+0x41/0xd0
Oct 18 12:10:44 free kernel: [  846.560071]  ? __wake_up_common_lock+0x89/0xc0
Oct 18 12:10:44 free kernel: [  846.560075]
__writeback_single_inode+0x3d/0x340
Oct 18 12:10:44 free kernel: [  846.560078]  writeback_sb_inodes+0x1e5/0x480
Oct 18 12:10:44 free kernel: [  846.560082]  __writeback_inodes_wb+0x5d/0xb0
Oct 18 12:10:44 free kernel: [  846.560085]  wb_writeback+0x25f/0x2f0
Oct 18 12:10:44 free kernel: [  846.560089]  wb_workfn+0x30c/0x400
Oct 18 12:10:44 free kernel: [  846.560092]  process_one_work+0x1a7/0x3b0
Oct 18 12:10:44 free kernel: [  846.560095]  worker_thread+0x1fa/0x390
Oct 18 12:10:44 free kernel: [  846.560097]  ? create_worker+0x1a0/0x1a0
Oct 18 12:10:44 free kernel: [  846.560100]  kthread+0x112/0x130
Oct 18 12:10:44 free kernel: [  846.560103]  ? __kthread_parkme+0x70/0x70
Oct 18 12:10:44 free kernel: [  846.560106]  ret_from_fork+0x35/0x40
Oct 18 12:10:44 free kernel: [  846.560147] INFO: task kworker/0:2H:2962
blocked for more than 362 seconds.
Oct 18 12:10:44 free kernel: [  846.560150]       Tainted: G           OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:10:44 free kernel: [  846.560151] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:10:44 free kernel: [  846.560153] kworker/0:2H    D    0  2962      2
0x80004000
Oct 18 12:10:44 free kernel: [  846.560158] Workqueue: kblockd
blk_mq_timeout_work
Oct 18 12:10:44 free kernel: [  846.560159] Call Trace:
Oct 18 12:10:44 free kernel: [  846.560163]  ? __schedule+0x2bb/0x660
Oct 18 12:10:44 free kernel: [  846.560166]  schedule+0x2f/0xa0
Oct 18 12:10:44 free kernel: [  846.560169]  schedule_preempt_disabled+0xa/0x10
Oct 18 12:10:44 free kernel: [  846.560172]  __mutex_lock.isra.8+0x2b5/0x4a0
Oct 18 12:10:44 free kernel: [  846.560175]  ?
blk_mq_dispatch_rq_list+0x48d/0x5c0
Oct 18 12:10:44 free kernel: [  846.560182]  sock_shutdown+0x58/0x80 [nbd]
Oct 18 12:10:44 free kernel: [  846.560187]  nbd_xmit_timeout+0xb7/0x150 [nbd]
Oct 18 12:10:44 free kernel: [  846.560189]  blk_mq_check_expired+0xe7/0x100
Oct 18 12:10:44 free kernel: [  846.560192]
blk_mq_queue_tag_busy_iter+0x19a/0x2f0
Oct 18 12:10:44 free kernel: [  846.560195]  ? blk_poll+0x350/0x350
Oct 18 12:10:44 free kernel: [  846.560197]  ? __switch_to_asm+0x34/0x70
Oct 18 12:10:44 free kernel: [  846.560199]  ? blk_poll+0x350/0x350
Oct 18 12:10:44 free kernel: [  846.560201]  ? __switch_to+0x10/0x3e0
Oct 18 12:10:44 free kernel: [  846.560204]  blk_mq_timeout_work+0x54/0x120
Oct 18 12:10:44 free kernel: [  846.560206]  process_one_work+0x1a7/0x3b0
Oct 18 12:10:44 free kernel: [  846.560208]  worker_thread+0x30/0x390
Oct 18 12:10:44 free kernel: [  846.560210]  ? create_worker+0x1a0/0x1a0
Oct 18 12:10:44 free kernel: [  846.560212]  kthread+0x112/0x130
Oct 18 12:10:44 free kernel: [  846.560226]  ? __kthread_parkme+0x70/0x70
Oct 18 12:10:44 free kernel: [  846.560228]  ret_from_fork+0x35/0x40
Oct 18 12:35:42 free kernel: [    0.000000] Linux version 5.2.0-3-amd64
(debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-23)) #1 SMP
Debian 5.2.17-1 (2019-10-06)
Oct 18 12:35:42 free kernel: [    0.000000] Command line:
BOOT_IMAGE=/vmlinuz-5.2.0-3-amd64 root=/dev/mapper/test--lvm--vg-root ro quiet


And/or this:

Oct 13 11:19:03 free kernel: [  213.420917] block nbd1: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [  213.421006] block nbd1: Disconnected due to
user request.
Oct 13 11:19:03 free kernel: [  213.421983] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:19:03 free kernel: [  213.421991] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:19:03 free kernel: [  213.422011] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:19:03 free kernel: [  213.422014] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:19:03 free kernel: [  213.422024] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:19:03 free kernel: [  213.422026] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:19:03 free kernel: [  213.422036] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:19:03 free kernel: [  213.422038] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:19:03 free kernel: [  213.422048] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:19:03 free kernel: [  213.422050] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:19:03 free kernel: [  213.422059] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:19:03 free kernel: [  213.422062] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:19:03 free kernel: [  213.422071] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:19:03 free kernel: [  213.422073] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:19:03 free kernel: [  213.422101] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:19:03 free kernel: [  213.422103] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:19:03 free kernel: [  213.422116] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:19:03 free kernel: [  213.422118] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:19:03 free kernel: [  213.422125] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:19:03 free kernel: [  213.422127] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:19:03 free kernel: [  213.422251] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:19:03 free kernel: [  213.422421] Dev nbd1: unable to read RDB block
0
Oct 13 11:19:03 free kernel: [  213.422599]  nbd1: unable to read partition
table
Oct 13 11:19:03 free kernel: [  213.422829] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:19:03 free kernel: [  213.423002] Dev nbd1: unable to read RDB block
0
Oct 13 11:19:03 free kernel: [  213.428372]  nbd1: unable to read partition
table
Oct 13 11:19:03 free kernel: [  213.436129] block nbd0: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [  213.457035] block nbd1: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [  213.461999] block nbd0: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [  213.826134] block nbd1: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [  213.832202] block nbd0: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [  213.852389] block nbd1: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [  213.857512] block nbd0: NBD_DISCONNECT
Oct 13 11:22:14 free kernel: [  404.790015] block nbd1: NBD_DISCONNECT
Oct 13 11:22:14 free kernel: [  404.797440] block nbd0: NBD_DISCONNECT
Oct 13 11:22:14 free kernel: [  404.821874] block nbd1: NBD_DISCONNECT
Oct 13 11:22:14 free kernel: [  404.828243] block nbd0: NBD_DISCONNECT
Oct 13 11:24:37 free kernel: [  547.343900] block nbd1: NBD_DISCONNECT
Oct 13 11:24:37 free kernel: [  547.343977] block nbd1: Disconnected due to
user request.
Oct 13 11:24:37 free kernel: [  547.344914] print_req_error: 182 callbacks
suppressed
Oct 13 11:24:37 free kernel: [  547.344917] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:24:37 free kernel: [  547.344923] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:24:37 free kernel: [  547.344925] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:24:37 free kernel: [  547.345114] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:24:37 free kernel: [  547.345119] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:24:37 free kernel: [  547.345129] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:24:37 free kernel: [  547.345131] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:24:37 free kernel: [  547.345139] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:24:37 free kernel: [  547.345141] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:24:37 free kernel: [  547.345151] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:24:37 free kernel: [  547.345153] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:24:37 free kernel: [  547.345161] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:24:37 free kernel: [  547.345163] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:24:37 free kernel: [  547.345171] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:24:37 free kernel: [  547.345173] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:24:37 free kernel: [  547.345185] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:24:37 free kernel: [  547.345187] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:24:37 free kernel: [  547.345203] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:24:37 free kernel: [  547.345205] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:24:37 free kernel: [  547.345213] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:24:37 free kernel: [  547.345215] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:24:37 free kernel: [  547.345357] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:24:37 free kernel: [  547.345555] Dev nbd1: unable to read RDB block
0
Oct 13 11:24:37 free kernel: [  547.345763]  nbd1: unable to read partition
table
Oct 13 11:24:37 free kernel: [  547.346034] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:24:37 free kernel: [  547.346231] Dev nbd1: unable to read RDB block
0
Oct 13 11:24:37 free kernel: [  547.346433]  nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [  597.462406] block nbd1: NBD_DISCONNECT
Oct 13 11:25:27 free kernel: [  597.462437] block nbd1: Disconnected due to
user request.
Oct 13 11:25:27 free kernel: [  597.462721] print_req_error: 182 callbacks
suppressed
Oct 13 11:25:27 free kernel: [  597.462722] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:25:27 free kernel: [  597.462723] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:25:27 free kernel: [  597.462724] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:25:27 free kernel: [  597.462730] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:25:27 free kernel: [  597.462731] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:25:27 free kernel: [  597.462734] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:25:27 free kernel: [  597.462734] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:25:27 free kernel: [  597.462737] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:25:27 free kernel: [  597.462737] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:25:27 free kernel: [  597.462740] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:25:27 free kernel: [  597.462741] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:25:27 free kernel: [  597.462743] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:25:27 free kernel: [  597.462744] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:25:27 free kernel: [  597.462746] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:25:27 free kernel: [  597.462747] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:25:27 free kernel: [  597.462751] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:25:27 free kernel: [  597.462751] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:25:27 free kernel: [  597.462757] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:25:27 free kernel: [  597.462757] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:25:27 free kernel: [  597.462760] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:25:27 free kernel: [  597.462760] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:25:27 free kernel: [  597.462805] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [  597.462867] Dev nbd1: unable to read RDB block
0
Oct 13 11:25:27 free kernel: [  597.462932]  nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [  597.464997] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [  597.465102] Dev nbd1: unable to read RDB block
0
Oct 13 11:25:27 free kernel: [  597.465168]  nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [  597.467891] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [  597.467954] Dev nbd1: unable to read RDB block
0
Oct 13 11:25:27 free kernel: [  597.468021]  nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [  597.470461] block nbd0: NBD_DISCONNECT
Oct 13 11:25:27 free kernel: [  597.471099] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [  597.471162] Dev nbd1: unable to read RDB block
0
Oct 13 11:25:27 free kernel: [  597.471227]  nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [  597.471312] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [  597.471374] Dev nbd1: unable to read RDB block
0
Oct 13 11:25:27 free kernel: [  597.471438]  nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [  597.474762] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [  597.474824] Dev nbd1: unable to read RDB block
0
Oct 13 11:25:27 free kernel: [  597.474889]  nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [  597.484450] block nbd1: NBD_DISCONNECT
Oct 13 11:25:27 free kernel: [  597.486235] block nbd0: NBD_DISCONNECT
Oct 13 11:26:20 free kernel: [  650.829740] block nbd1: NBD_DISCONNECT
Oct 13 11:26:20 free kernel: [  650.829838] block nbd1: Disconnected due to
user request.
Oct 13 11:26:20 free kernel: [  650.830814] print_req_error: 575 callbacks
suppressed
Oct 13 11:26:20 free kernel: [  650.830817] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:26:20 free kernel: [  650.830824] buffer_io_error: 574 callbacks
suppressed
Oct 13 11:26:20 free kernel: [  650.830826] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:26:20 free kernel: [  650.830846] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:26:20 free kernel: [  650.830849] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:26:20 free kernel: [  650.830860] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:26:20 free kernel: [  650.830862] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:26:20 free kernel: [  650.830872] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:26:20 free kernel: [  650.830874] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:26:20 free kernel: [  650.830884] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:26:20 free kernel: [  650.830886] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:26:20 free kernel: [  650.830895] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:26:20 free kernel: [  650.830898] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:26:20 free kernel: [  650.830907] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:26:20 free kernel: [  650.830909] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:26:20 free kernel: [  650.830924] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:26:20 free kernel: [  650.830926] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:26:20 free kernel: [  650.830944] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:26:20 free kernel: [  650.830947] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:26:20 free kernel: [  650.830956] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:26:20 free kernel: [  650.830959] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:26:20 free kernel: [  650.831169] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:26:20 free kernel: [  650.831402] Dev nbd1: unable to read RDB block
0
Oct 13 11:26:20 free kernel: [  650.831644]  nbd1: unable to read partition
table
Oct 13 11:26:20 free kernel: [  650.831960] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:26:20 free kernel: [  650.832193] Dev nbd1: unable to read RDB block
0
Oct 13 11:26:20 free kernel: [  650.832434]  nbd1: unable to read partition
table
Oct 13 11:26:20 free kernel: [  650.850299] block nbd0: NBD_DISCONNECT
Oct 13 11:26:20 free kernel: [  650.866829] block nbd1: NBD_DISCONNECT
Oct 13 11:26:20 free kernel: [  650.872238] block nbd0: NBD_DISCONNECT
Oct 13 11:27:07 free kernel: [  697.727569] block nbd1: NBD_DISCONNECT
Oct 13 11:27:07 free kernel: [  697.727602] block nbd1: Disconnected due to
user request.
Oct 13 11:27:07 free kernel: [  697.727916] print_req_error: 182 callbacks
suppressed
Oct 13 11:27:07 free kernel: [  697.727917] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:27:07 free kernel: [  697.727919] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:27:07 free kernel: [  697.727920] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:27:07 free kernel: [  697.727927] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:27:07 free kernel: [  697.727928] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:27:07 free kernel: [  697.727930] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:27:07 free kernel: [  697.727931] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:27:07 free kernel: [  697.727934] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:27:07 free kernel: [  697.727935] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:27:07 free kernel: [  697.727937] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:27:07 free kernel: [  697.727938] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:27:07 free kernel: [  697.727941] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:27:07 free kernel: [  697.727941] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:27:07 free kernel: [  697.727944] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:27:07 free kernel: [  697.727945] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:27:07 free kernel: [  697.727949] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:27:07 free kernel: [  697.727950] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:27:07 free kernel: [  697.727955] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:27:07 free kernel: [  697.727956] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:27:07 free kernel: [  697.727959] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:27:07 free kernel: [  697.727959] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:27:07 free kernel: [  697.728008] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:07 free kernel: [  697.728074] Dev nbd1: unable to read RDB block
0
Oct 13 11:27:07 free kernel: [  697.728143]  nbd1: unable to read partition
table
Oct 13 11:27:07 free kernel: [  697.728234] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:07 free kernel: [  697.728300] Dev nbd1: unable to read RDB block
0
Oct 13 11:27:07 free kernel: [  697.728367]  nbd1: unable to read partition
table
Oct 13 11:27:07 free kernel: [  697.729640] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:07 free kernel: [  697.729714] Dev nbd1: unable to read RDB block
0
Oct 13 11:27:07 free kernel: [  697.729783]  nbd1: unable to read partition
table
Oct 13 11:27:07 free kernel: [  697.730297] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:07 free kernel: [  697.730362] Dev nbd1: unable to read RDB block
0
Oct 13 11:27:07 free kernel: [  697.730430]  nbd1: unable to read partition
table
Oct 13 11:27:07 free kernel: [  697.737367] block nbd0: NBD_DISCONNECT
Oct 13 11:27:07 free kernel: [  697.751292] block nbd1: NBD_DISCONNECT
Oct 13 11:27:07 free kernel: [  697.756432] block nbd0: NBD_DISCONNECT
Oct 13 11:27:23 free kernel: [  713.452457] block nbd1: NBD_DISCONNECT
Oct 13 11:27:23 free kernel: [  713.452489] block nbd1: Disconnected due to
user request.
Oct 13 11:27:23 free kernel: [  713.452785] print_req_error: 383 callbacks
suppressed
Oct 13 11:27:23 free kernel: [  713.452786] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:27:23 free kernel: [  713.452788] buffer_io_error: 382 callbacks
suppressed
Oct 13 11:27:23 free kernel: [  713.452789] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:27:23 free kernel: [  713.452795] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:27:23 free kernel: [  713.452796] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:27:23 free kernel: [  713.452799] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:27:23 free kernel: [  713.452800] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:27:23 free kernel: [  713.452802] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:27:23 free kernel: [  713.452803] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:27:23 free kernel: [  713.452806] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:27:23 free kernel: [  713.452806] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:27:23 free kernel: [  713.452809] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:27:23 free kernel: [  713.452810] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:27:23 free kernel: [  713.452812] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:27:23 free kernel: [  713.452813] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:27:23 free kernel: [  713.452817] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:27:23 free kernel: [  713.452818] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:27:23 free kernel: [  713.452823] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:27:23 free kernel: [  713.452824] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:27:23 free kernel: [  713.452827] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:27:23 free kernel: [  713.452827] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:27:23 free kernel: [  713.452887] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:23 free kernel: [  713.452950] Dev nbd1: unable to read RDB block
0
Oct 13 11:27:23 free kernel: [  713.453015]  nbd1: unable to read partition
table
Oct 13 11:27:23 free kernel: [  713.453102] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:23 free kernel: [  713.453165] Dev nbd1: unable to read RDB block
0
Oct 13 11:27:23 free kernel: [  713.453230]  nbd1: unable to read partition
table
Oct 13 11:27:23 free kernel: [  713.454379] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:23 free kernel: [  713.454442] Dev nbd1: unable to read RDB block
0
Oct 13 11:27:23 free kernel: [  713.454507]  nbd1: unable to read partition
table
Oct 13 11:27:23 free kernel: [  713.454980] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:23 free kernel: [  713.455102] Dev nbd1: unable to read RDB block
0
Oct 13 11:27:23 free kernel: [  713.455166]  nbd1: unable to read partition
table
Oct 13 11:27:23 free kernel: [  713.465565] block nbd0: NBD_DISCONNECT
Oct 13 11:27:23 free kernel: [  713.477996] block nbd1: NBD_DISCONNECT
Oct 13 11:27:23 free kernel: [  713.482674] block nbd0: NBD_DISCONNECT
Oct 13 11:28:16 free kernel: [  766.759620] block nbd1: NBD_DISCONNECT
Oct 13 11:28:16 free kernel: [  766.765261] block nbd0: NBD_DISCONNECT
Oct 13 11:28:16 free kernel: [  766.772551] block nbd1: NBD_DISCONNECT
Oct 13 11:28:16 free kernel: [  766.777506] block nbd0: NBD_DISCONNECT
Oct 13 11:28:33 free kernel: [  783.019791] block nbd1: NBD_DISCONNECT
Oct 13 11:28:33 free kernel: [  783.019883] block nbd1: Disconnected due to
user request.
Oct 13 11:28:33 free kernel: [  783.020851] print_req_error: 383 callbacks
suppressed
Oct 13 11:28:33 free kernel: [  783.020855] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:28:33 free kernel: [  783.020861] buffer_io_error: 382 callbacks
suppressed
Oct 13 11:28:33 free kernel: [  783.020864] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:28:33 free kernel: [  783.020884] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:28:33 free kernel: [  783.020887] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:28:33 free kernel: [  783.020898] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:28:33 free kernel: [  783.020900] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:28:33 free kernel: [  783.020910] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:28:33 free kernel: [  783.020912] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:28:33 free kernel: [  783.020922] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:28:33 free kernel: [  783.020924] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:28:33 free kernel: [  783.020933] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:28:33 free kernel: [  783.020936] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:28:33 free kernel: [  783.020945] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:28:33 free kernel: [  783.020947] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:28:33 free kernel: [  783.020961] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:28:33 free kernel: [  783.020964] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:28:33 free kernel: [  783.020982] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:28:33 free kernel: [  783.020985] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:28:33 free kernel: [  783.020996] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:28:33 free kernel: [  783.020998] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:28:33 free kernel: [  783.021167] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:28:33 free kernel: [  783.021398] Dev nbd1: unable to read RDB block
0
Oct 13 11:28:33 free kernel: [  783.021640]  nbd1: unable to read partition
table
Oct 13 11:28:33 free kernel: [  783.021953] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:28:33 free kernel: [  783.022187] Dev nbd1: unable to read RDB block
0
Oct 13 11:28:33 free kernel: [  783.022423]  nbd1: unable to read partition
table
Oct 13 11:28:33 free kernel: [  783.036979] block nbd0: NBD_DISCONNECT
Oct 13 11:28:33 free kernel: [  783.051997] block nbd1: NBD_DISCONNECT
Oct 13 11:28:33 free kernel: [  783.057931] block nbd0: NBD_DISCONNECT
Oct 13 11:28:44 free kernel: [  794.232905] block nbd1: NBD_DISCONNECT
Oct 13 11:28:44 free kernel: [  794.238458] block nbd0: NBD_DISCONNECT
Oct 13 11:28:44 free kernel: [  794.247611] block nbd1: NBD_DISCONNECT
Oct 13 11:28:44 free kernel: [  794.252883] block nbd0: NBD_DISCONNECT
Oct 13 11:29:06 free kernel: [  816.240576] block nbd1: NBD_DISCONNECT
Oct 13 11:29:06 free kernel: [  816.246196] block nbd0: NBD_DISCONNECT
Oct 13 11:29:06 free kernel: [  816.254445] block nbd1: NBD_DISCONNECT
Oct 13 11:29:06 free kernel: [  816.258645] block nbd0: NBD_DISCONNECT
Oct 13 11:29:21 free kernel: [  831.380225] block nbd1: NBD_DISCONNECT
Oct 13 11:29:21 free kernel: [  831.380318] block nbd1: Disconnected due to
user request.
Oct 13 11:29:21 free kernel: [  831.381310] print_req_error: 182 callbacks
suppressed
Oct 13 11:29:21 free kernel: [  831.381313] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:29:21 free kernel: [  831.381320] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:29:21 free kernel: [  831.381337] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:29:21 free kernel: [  831.381358] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:29:21 free kernel: [  831.381361] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:29:21 free kernel: [  831.381370] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:29:21 free kernel: [  831.381373] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:29:21 free kernel: [  831.381381] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:29:21 free kernel: [  831.381384] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:29:21 free kernel: [  831.381392] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:29:21 free kernel: [  831.381394] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:29:21 free kernel: [  831.381403] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:29:21 free kernel: [  831.381405] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:29:21 free kernel: [  831.381414] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:29:21 free kernel: [  831.381416] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:29:21 free kernel: [  831.381428] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:29:21 free kernel: [  831.381431] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:29:21 free kernel: [  831.381447] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:29:21 free kernel: [  831.381450] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:29:21 free kernel: [  831.381458] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:29:21 free kernel: [  831.381461] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:29:21 free kernel: [  831.381615] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:29:21 free kernel: [  831.381830] Dev nbd1: unable to read RDB block
0
Oct 13 11:29:21 free kernel: [  831.382052]  nbd1: unable to read partition
table
Oct 13 11:29:21 free kernel: [  831.382338] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:29:21 free kernel: [  831.382551] Dev nbd1: unable to read RDB block
0
Oct 13 11:29:21 free kernel: [  831.382770]  nbd1: unable to read partition
table
Oct 13 11:29:21 free kernel: [  831.395494] block nbd0: NBD_DISCONNECT
Oct 13 11:29:21 free kernel: [  831.411843] block nbd1: NBD_DISCONNECT
Oct 13 11:29:21 free kernel: [  831.419893] block nbd0: NBD_DISCONNECT
Oct 13 11:29:39 free kernel: [  849.869377] block nbd1: NBD_DISCONNECT
Oct 13 11:29:39 free kernel: [  849.869382] block nbd1: Disconnected due to
user request.
Oct 13 11:29:39 free kernel: [  849.869383] block nbd1: shutting down sockets
Oct 13 11:29:39 free kernel: [  849.869534] print_req_error: 182 callbacks
suppressed
Oct 13 11:29:39 free kernel: [  849.869536] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:29:39 free kernel: [  849.869538] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:29:39 free kernel: [  849.869538] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:29:39 free kernel: [  849.869543] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:29:39 free kernel: [  849.869543] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:29:39 free kernel: [  849.869546] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:29:39 free kernel: [  849.869547] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:29:39 free kernel: [  849.869549] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:29:39 free kernel: [  849.869550] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:29:39 free kernel: [  849.869552] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:29:39 free kernel: [  849.869553] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:29:39 free kernel: [  849.869555] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:29:39 free kernel: [  849.869556] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:29:39 free kernel: [  849.869558] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:29:39 free kernel: [  849.869559] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:29:39 free kernel: [  849.869562] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:29:39 free kernel: [  849.869563] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:29:39 free kernel: [  849.869567] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:29:39 free kernel: [  849.869568] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:29:39 free kernel: [  849.869570] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:29:39 free kernel: [  849.869571] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:29:39 free kernel: [  849.869614] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:29:39 free kernel: [  849.869673] Dev nbd1: unable to read RDB block
0
Oct 13 11:29:39 free kernel: [  849.869734]  nbd1: unable to read partition
table
Oct 13 11:29:39 free kernel: [  849.871997] block nbd0: NBD_DISCONNECT
Oct 13 11:29:39 free kernel: [  849.882161] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:29:39 free kernel: [  849.882222] Dev nbd1: unable to read RDB block
0
Oct 13 11:29:39 free kernel: [  849.882303]  nbd1: unable to read partition
table
Oct 13 11:29:39 free kernel: [  849.882309] block nbd1: NBD_DISCONNECT
Oct 13 11:29:39 free kernel: [  849.882311] block nbd1: Send disconnect failed
-32
Oct 13 11:29:39 free kernel: [  849.885020] block nbd0: NBD_DISCONNECT
Oct 13 11:30:11 free kernel: [  881.354868] block nbd1: Connection timed out
Oct 13 11:30:11 free kernel: [  881.354879] block nbd1: Connection timed out
Oct 13 11:30:11 free kernel: [  881.354904] print_req_error: 182 callbacks
suppressed
Oct 13 11:30:11 free kernel: [  881.354907] print_req_error: I/O error, dev
nbd1, sector 251658120 flags 80700
Oct 13 11:30:11 free kernel: [  881.354931] print_req_error: I/O error, dev
nbd1, sector 251658184 flags 80700
Oct 13 11:30:11 free kernel: [  881.354974] print_req_error: I/O error, dev
nbd1, sector 251658120 flags 0
Oct 13 11:30:11 free kernel: [  881.354977] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:30:11 free kernel: [  881.354979] Buffer I/O error on dev nbd1,
logical block 251658120, async page read
Oct 13 11:30:11 free kernel: [  881.354990] print_req_error: I/O error, dev
nbd1, sector 251658121 flags 0
Oct 13 11:30:11 free kernel: [  881.354993] Buffer I/O error on dev nbd1,
logical block 251658121, async page read
Oct 13 11:30:11 free kernel: [  881.355001] print_req_error: I/O error, dev
nbd1, sector 251658122 flags 0
Oct 13 11:30:11 free kernel: [  881.355003] Buffer I/O error on dev nbd1,
logical block 251658122, async page read
Oct 13 11:30:11 free kernel: [  881.355011] print_req_error: I/O error, dev
nbd1, sector 251658123 flags 0
Oct 13 11:30:11 free kernel: [  881.355013] Buffer I/O error on dev nbd1,
logical block 251658123, async page read
Oct 13 11:30:11 free kernel: [  881.355021] print_req_error: I/O error, dev
nbd1, sector 251658124 flags 0
Oct 13 11:30:11 free kernel: [  881.355023] Buffer I/O error on dev nbd1,
logical block 251658124, async page read
Oct 13 11:30:11 free kernel: [  881.355031] print_req_error: I/O error, dev
nbd1, sector 251658125 flags 0
Oct 13 11:30:11 free kernel: [  881.355033] Buffer I/O error on dev nbd1,
logical block 251658125, async page read
Oct 13 11:30:11 free kernel: [  881.355041] print_req_error: I/O error, dev
nbd1, sector 251658126 flags 0
Oct 13 11:30:11 free kernel: [  881.355043] Buffer I/O error on dev nbd1,
logical block 251658126, async page read
Oct 13 11:30:11 free kernel: [  881.355052] print_req_error: I/O error, dev
nbd1, sector 251658127 flags 0
Oct 13 11:30:11 free kernel: [  881.355054] Buffer I/O error on dev nbd1,
logical block 251658127, async page read
Oct 13 11:30:11 free kernel: [  881.355247] Buffer I/O error on dev nbd1,
logical block 4096, async page read
Oct 13 11:30:11 free kernel: [  881.355257] Buffer I/O error on dev nbd1,
logical block 4097, async page read
Oct 13 11:32:39 free kernel: [    0.000000] Linux version 5.2.0-3-amd64
(debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-23)) #1 SMP
Debian 5.2.17-1 (2019-10-06)
Oct 13 11:32:39 free kernel: [    0.000000] Command line:
BOOT_IMAGE=/vmlinuz-5.2.0-3-amd64 root=/dev/mapper/test--lvm--vg-root ro quiet



I've also tried with different settings of the --cache option to qemu-nbd, but
this only resulted in faster speeds, but the problem was the same.

Like I said, if I try the same on a current Stretch system, it works like a
charm. I am not sure if this is a pure qemu-nbd (NBD) problem or related to the
kernel (or both).
#942579#10
Date:
2019-11-20 12:23:07 UTC
From:
To:
Here is some recent output from /var/log/syslog:


Nov 20 13:11:49 debian kernel: [  160.820705] block nbd1: Connection timed out
Nov 20 13:15:12 debian kernel: [  363.471204] INFO: task kworker/u2:3:128 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [  363.471210]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [  363.471212] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [  363.471214] kworker/u2:3    D    0   128      2 0x80000000
Nov 20 13:15:12 debian kernel: [  363.471237] Workqueue: writeback wb_workfn (flush-254:11)
Nov 20 13:15:12 debian kernel: [  363.471240] Call Trace:
Nov 20 13:15:12 debian kernel: [  363.471262]  ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [  363.471265]  schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [  363.471268]  io_schedule+0x12/0x40
Nov 20 13:15:12 debian kernel: [  363.471275]  wbt_wait+0x205/0x300
Nov 20 13:15:12 debian kernel: [  363.471291]  ? trace_event_raw_event_wbt_step+0x130/0x130
Nov 20 13:15:12 debian kernel: [  363.471295]  rq_qos_throttle+0x31/0x40
Nov 20 13:15:12 debian kernel: [  363.471298]  blk_mq_make_request+0x111/0x530
Nov 20 13:15:12 debian kernel: [  363.471302]  generic_make_request+0x1a4/0x400
Nov 20 13:15:12 debian kernel: [  363.471305]  submit_bio+0x45/0x140
Nov 20 13:15:12 debian kernel: [  363.471333]  ext4_io_submit+0x49/0x60 [ext4]
Nov 20 13:15:12 debian kernel: [  363.471353]  ext4_bio_write_page+0x24a/0x4d0 [ext4]
Nov 20 13:15:12 debian kernel: [  363.471371]  mpage_submit_page+0x53/0x70 [ext4]
Nov 20 13:15:12 debian kernel: [  363.471388]  mpage_map_and_submit_buffers+0x10c/0x1d0 [ext4]
Nov 20 13:15:12 debian kernel: [  363.471408]  ext4_writepages+0xa12/0xf00 [ext4]
Nov 20 13:15:12 debian kernel: [  363.471414]  ? do_writepages+0x41/0xd0
Nov 20 13:15:12 debian kernel: [  363.471430]  ? ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Nov 20 13:15:12 debian kernel: [  363.471432]  do_writepages+0x41/0xd0
Nov 20 13:15:12 debian kernel: [  363.471445]  ? __wb_calc_thresh+0x3a/0x130
Nov 20 13:15:12 debian kernel: [  363.471448]  __writeback_single_inode+0x3d/0x350
Nov 20 13:15:12 debian kernel: [  363.471459]  writeback_sb_inodes+0x1e3/0x450
Nov 20 13:15:12 debian kernel: [  363.471463]  __writeback_inodes_wb+0x5d/0xb0
Nov 20 13:15:12 debian kernel: [  363.471466]  wb_writeback+0x25f/0x2f0
Nov 20 13:15:12 debian kernel: [  363.471469]  wb_workfn+0x30d/0x400
Nov 20 13:15:12 debian kernel: [  363.471472]  ? __switch_to+0x8c/0x440
Nov 20 13:15:12 debian kernel: [  363.471476]  process_one_work+0x1a7/0x3a0
Nov 20 13:15:12 debian kernel: [  363.471487]  worker_thread+0x30/0x390
Nov 20 13:15:12 debian kernel: [  363.471491]  ? create_worker+0x1a0/0x1a0
Nov 20 13:15:12 debian kernel: [  363.471493]  kthread+0x112/0x130
Nov 20 13:15:12 debian kernel: [  363.471495]  ? kthread_bind+0x30/0x30
Nov 20 13:15:12 debian kernel: [  363.471499]  ret_from_fork+0x35/0x40
Nov 20 13:15:12 debian kernel: [  363.471504] INFO: task kworker/0:1H:165 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [  363.471506]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [  363.471508] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [  363.471510] kworker/0:1H    D    0   165      2 0x80000000
Nov 20 13:15:12 debian kernel: [  363.471515] Workqueue: kblockd blk_mq_timeout_work
Nov 20 13:15:12 debian kernel: [  363.471516] Call Trace:
Nov 20 13:15:12 debian kernel: [  363.471520]  ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [  363.471523]  schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [  363.471525]  schedule_preempt_disabled+0xa/0x10
Nov 20 13:15:12 debian kernel: [  363.471528]  __mutex_lock.isra.8+0x2b5/0x4a0
Nov 20 13:15:12 debian kernel: [  363.471535]  ? _dev_err+0x6c/0x90
Nov 20 13:15:12 debian kernel: [  363.471541]  sock_shutdown+0x4a/0x90 [nbd]
Nov 20 13:15:12 debian kernel: [  363.471545]  nbd_xmit_timeout+0xb0/0x180 [nbd]
Nov 20 13:15:12 debian kernel: [  363.471548]  blk_mq_check_expired+0xe9/0x100
Nov 20 13:15:12 debian kernel: [  363.471551]  bt_iter+0x4e/0x60
Nov 20 13:15:12 debian kernel: [  363.471554]  blk_mq_queue_tag_busy_iter+0x184/0x2e0
Nov 20 13:15:12 debian kernel: [  363.471556]  ? blk_mq_poll+0x360/0x360
Nov 20 13:15:12 debian kernel: [  363.471559]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471561]  ? blk_mq_poll+0x360/0x360
Nov 20 13:15:12 debian kernel: [  363.471564]  ? syscall_return_via_sysret+0x13/0x83
Nov 20 13:15:12 debian kernel: [  363.471567]  blk_mq_timeout_work+0x54/0x120
Nov 20 13:15:12 debian kernel: [  363.471570]  process_one_work+0x1a7/0x3a0
Nov 20 13:15:12 debian kernel: [  363.471573]  worker_thread+0x30/0x390
Nov 20 13:15:12 debian kernel: [  363.471576]  ? create_worker+0x1a0/0x1a0
Nov 20 13:15:12 debian kernel: [  363.471578]  kthread+0x112/0x130
Nov 20 13:15:12 debian kernel: [  363.471580]  ? kthread_bind+0x30/0x30
Nov 20 13:15:12 debian kernel: [  363.471583]  ret_from_fork+0x35/0x40
Nov 20 13:15:12 debian kernel: [  363.471620] INFO: task jbd2/dm-11-8:3239 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [  363.471622]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [  363.471623] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [  363.471625] jbd2/dm-11-8    D    0  3239      2 0x80000000
Nov 20 13:15:12 debian kernel: [  363.471627] Call Trace:
Nov 20 13:15:12 debian kernel: [  363.471631]  ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [  363.471639]  ? __wake_up_common_lock+0x89/0xc0
Nov 20 13:15:12 debian kernel: [  363.471641]  ? finish_wait+0x80/0x80
Nov 20 13:15:12 debian kernel: [  363.471644]  schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [  363.471651]  jbd2_journal_commit_transaction+0x257/0x1890 [jbd2]
Nov 20 13:15:12 debian kernel: [  363.471656]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471658]  ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [  363.471661]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471664]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471666]  ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [  363.471669]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471671]  ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [  363.471674]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471676]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471679]  ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [  363.471682]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471684]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471687]  ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [  363.471689]  ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [  363.471692]  ? finish_wait+0x80/0x80
Nov 20 13:15:12 debian kernel: [  363.471695]  ? lock_timer_base+0x67/0x80
Nov 20 13:15:12 debian kernel: [  363.471701]  kjournald2+0xbd/0x270 [jbd2]
Nov 20 13:15:12 debian kernel: [  363.471704]  ? finish_wait+0x80/0x80
Nov 20 13:15:12 debian kernel: [  363.471710]  ? commit_timeout+0x10/0x10 [jbd2]
Nov 20 13:15:12 debian kernel: [  363.471712]  kthread+0x112/0x130
Nov 20 13:15:12 debian kernel: [  363.471714]  ? kthread_bind+0x30/0x30
Nov 20 13:15:12 debian kernel: [  363.471718]  ret_from_fork+0x35/0x40
Nov 20 13:15:12 debian kernel: [  363.471720] INFO: task umount:3921 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [  363.471722]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [  363.471724] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [  363.471725] umount          D    0  3921   2085 0x00000000
Nov 20 13:15:12 debian kernel: [  363.471727] Call Trace:
Nov 20 13:15:12 debian kernel: [  363.471730]  ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [  363.471733]  ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [  363.471735]  schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [  363.471738]  rwsem_down_write_failed+0x17c/0x3a0
Nov 20 13:15:12 debian kernel: [  363.471741]  ? mnt_get_count+0x39/0x50
Nov 20 13:15:12 debian kernel: [  363.471746]  call_rwsem_down_write_failed+0x13/0x20
Nov 20 13:15:12 debian kernel: [  363.471749]  down_write+0x29/0x40
Nov 20 13:15:12 debian kernel: [  363.471752]  deactivate_super+0x43/0x50
Nov 20 13:15:12 debian kernel: [  363.471755]  cleanup_mnt+0x3f/0x70
Nov 20 13:15:12 debian kernel: [  363.471757]  task_work_run+0x8a/0xb0
Nov 20 13:15:12 debian kernel: [  363.471761]  exit_to_usermode_loop+0xeb/0xf0
Nov 20 13:15:12 debian kernel: [  363.471768]  do_syscall_64+0x10d/0x110
Nov 20 13:15:12 debian kernel: [  363.471772]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov 20 13:15:12 debian kernel: [  363.471775] RIP: 0033:0x7f08434ec507
Nov 20 13:15:12 debian kernel: [  363.471780] Code: Bad RIP value.
Nov 20 13:15:12 debian kernel: [  363.471782] RSP: 002b:00007ffed9c8ec38 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
Nov 20 13:15:12 debian kernel: [  363.471784] RAX: 0000000000000000 RBX: 000055b66751ca40 RCX: 00007f08434ec507
Nov 20 13:15:12 debian kernel: [  363.471786] RDX: 0000000000000003 RSI: 0000000000000002 RDI: 000055b66751d290
Nov 20 13:15:12 debian kernel: [  363.471787] RBP: 0000000000000002 R08: 000055b667525fd0 R09: 00007f084356de80
Nov 20 13:15:12 debian kernel: [  363.471788] R10: 0000000000000000 R11: 0000000000000202 R12: 000055b66751d290
Nov 20 13:15:12 debian kernel: [  363.471789] R13: 00007f08436121c4 R14: 000055b66751cb38 R15: 000055b667525890
Nov 20 13:15:12 debian kernel: [  363.471792] INFO: task dumpe2fs:3922 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [  363.471794]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [  363.471795] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [  363.471797] dumpe2fs        D    0  3922    599 0x00000000
Nov 20 13:15:12 debian kernel: [  363.471799] Call Trace:
Nov 20 13:15:12 debian kernel: [  363.471802]  ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [  363.471805]  schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [  363.471807]  io_schedule+0x12/0x40
Nov 20 13:15:12 debian kernel: [  363.471810]  wbt_wait+0x205/0x300
Nov 20 13:15:12 debian kernel: [  363.471812]  ? trace_event_raw_event_wbt_step+0x130/0x130
Nov 20 13:15:12 debian kernel: [  363.471816]  rq_qos_throttle+0x31/0x40
Nov 20 13:15:12 debian kernel: [  363.471818]  blk_mq_make_request+0x111/0x530
Nov 20 13:15:12 debian kernel: [  363.471822]  generic_make_request+0x1a4/0x400
Nov 20 13:15:12 debian kernel: [  363.471824]  submit_bio+0x45/0x140
Nov 20 13:15:12 debian kernel: [  363.471827]  ? guard_bio_eod+0x32/0x100
Nov 20 13:15:12 debian kernel: [  363.471830]  submit_bh_wbc+0x163/0x190
Nov 20 13:15:12 debian kernel: [  363.471833]  __block_write_full_page+0x234/0x420
Nov 20 13:15:12 debian kernel: [  363.471836]  ? check_disk_change+0x60/0x60
Nov 20 13:15:12 debian kernel: [  363.471838]  ? touch_buffer+0x60/0x60
Nov 20 13:15:12 debian kernel: [  363.471841]  __writepage+0x19/0x50
Nov 20 13:15:12 debian kernel: [  363.471843]  write_cache_pages+0x1e1/0x470
Nov 20 13:15:12 debian kernel: [  363.471845]  ? __wb_calc_thresh+0x130/0x130
Nov 20 13:15:12 debian kernel: [  363.471848]  generic_writepages+0x56/0x90
Nov 20 13:15:12 debian kernel: [  363.471851]  do_writepages+0x41/0xd0
Nov 20 13:15:12 debian kernel: [  363.471855]  ? __handle_mm_fault+0x1090/0x1270
Nov 20 13:15:12 debian kernel: [  363.471858]  __filemap_fdatawrite_range+0xbe/0xf0
Nov 20 13:15:12 debian kernel: [  363.471861]  file_write_and_wait_range+0x4c/0xa0
Nov 20 13:15:12 debian kernel: [  363.471864]  blkdev_fsync+0x16/0x40
Nov 20 13:15:12 debian kernel: [  363.471867]  do_fsync+0x38/0x70
Nov 20 13:15:12 debian kernel: [  363.471870]  __x64_sys_fsync+0x10/0x20
Nov 20 13:15:12 debian kernel: [  363.471873]  do_syscall_64+0x53/0x110
Nov 20 13:15:12 debian kernel: [  363.471877]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov 20 13:15:12 debian kernel: [  363.471878] RIP: 0033:0x7f0790255214
Nov 20 13:15:12 debian kernel: [  363.471881] Code: Bad RIP value.
Nov 20 13:15:12 debian kernel: [  363.471882] RSP: 002b:00007fff7f9474d8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Nov 20 13:15:12 debian kernel: [  363.471890] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0790255214
Nov 20 13:15:12 debian kernel: [  363.471891] RDX: 0000000000000008 RSI: 0000000000000000 RDI: 0000000000000003
Nov 20 13:15:12 debian kernel: [  363.471892] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007f07903ef680
Nov 20 13:15:12 debian kernel: [  363.471893] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f07903ef680
Nov 20 13:15:12 debian kernel: [  363.471894] R13: 000055cb7a993440 R14: 00007f07903ef680 R15: 0000000000000000
Nov 20 13:15:16 debian PackageKit: daemon quit
Nov 20 13:15:16 debian systemd[1]: packagekit.service: Main process exited, code=killed, status=15/TERM
Nov 20 13:15:16 debian systemd[1]: packagekit.service: Succeeded.
Nov 20 13:17:01 debian CRON[3941]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Nov 20 13:17:13 debian kernel: [  484.242755] INFO: task kworker/u2:3:128 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [  484.242759]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [  484.242759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [  484.242761] kworker/u2:3    D    0   128      2 0x80000000
Nov 20 13:17:13 debian kernel: [  484.242766] Workqueue: writeback wb_workfn (flush-254:11)
Nov 20 13:17:13 debian kernel: [  484.242767] Call Trace:
Nov 20 13:17:13 debian kernel: [  484.242772]  ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [  484.242773]  schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [  484.242774]  io_schedule+0x12/0x40
Nov 20 13:17:13 debian kernel: [  484.242776]  wbt_wait+0x205/0x300
Nov 20 13:17:13 debian kernel: [  484.242778]  ? trace_event_raw_event_wbt_step+0x130/0x130
Nov 20 13:17:13 debian kernel: [  484.242779]  rq_qos_throttle+0x31/0x40
Nov 20 13:17:13 debian kernel: [  484.242781]  blk_mq_make_request+0x111/0x530
Nov 20 13:17:13 debian kernel: [  484.242783]  generic_make_request+0x1a4/0x400
Nov 20 13:17:13 debian kernel: [  484.242784]  submit_bio+0x45/0x140
Nov 20 13:17:13 debian kernel: [  484.242797]  ext4_io_submit+0x49/0x60 [ext4]
Nov 20 13:17:13 debian kernel: [  484.242806]  ext4_bio_write_page+0x24a/0x4d0 [ext4]
Nov 20 13:17:13 debian kernel: [  484.242813]  mpage_submit_page+0x53/0x70 [ext4]
Nov 20 13:17:13 debian kernel: [  484.242820]  mpage_map_and_submit_buffers+0x10c/0x1d0 [ext4]
Nov 20 13:17:13 debian kernel: [  484.242829]  ext4_writepages+0xa12/0xf00 [ext4]
Nov 20 13:17:13 debian kernel: [  484.242832]  ? do_writepages+0x41/0xd0
Nov 20 13:17:13 debian kernel: [  484.242838]  ? ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Nov 20 13:17:13 debian kernel: [  484.242839]  do_writepages+0x41/0xd0
Nov 20 13:17:13 debian kernel: [  484.242841]  ? __wb_calc_thresh+0x3a/0x130
Nov 20 13:17:13 debian kernel: [  484.242842]  __writeback_single_inode+0x3d/0x350
Nov 20 13:17:13 debian kernel: [  484.242843]  writeback_sb_inodes+0x1e3/0x450
Nov 20 13:17:13 debian kernel: [  484.242852]  __writeback_inodes_wb+0x5d/0xb0
Nov 20 13:17:13 debian kernel: [  484.242854]  wb_writeback+0x25f/0x2f0
Nov 20 13:17:13 debian kernel: [  484.242856]  wb_workfn+0x30d/0x400
Nov 20 13:17:13 debian kernel: [  484.242857]  ? __switch_to+0x8c/0x440
Nov 20 13:17:13 debian kernel: [  484.242859]  process_one_work+0x1a7/0x3a0
Nov 20 13:17:13 debian kernel: [  484.242861]  worker_thread+0x30/0x390
Nov 20 13:17:13 debian kernel: [  484.242862]  ? create_worker+0x1a0/0x1a0
Nov 20 13:17:13 debian kernel: [  484.242863]  kthread+0x112/0x130
Nov 20 13:17:13 debian kernel: [  484.242864]  ? kthread_bind+0x30/0x30
Nov 20 13:17:13 debian kernel: [  484.242866]  ret_from_fork+0x35/0x40
Nov 20 13:17:13 debian kernel: [  484.242868] INFO: task kworker/0:1H:165 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [  484.242869]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [  484.242870] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [  484.242871] kworker/0:1H    D    0   165      2 0x80000000
Nov 20 13:17:13 debian kernel: [  484.242873] Workqueue: kblockd blk_mq_timeout_work
Nov 20 13:17:13 debian kernel: [  484.242874] Call Trace:
Nov 20 13:17:13 debian kernel: [  484.242875]  ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [  484.242876]  schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [  484.242878]  schedule_preempt_disabled+0xa/0x10
Nov 20 13:17:13 debian kernel: [  484.242879]  __mutex_lock.isra.8+0x2b5/0x4a0
Nov 20 13:17:13 debian kernel: [  484.242881]  ? _dev_err+0x6c/0x90
Nov 20 13:17:13 debian kernel: [  484.242884]  sock_shutdown+0x4a/0x90 [nbd]
Nov 20 13:17:13 debian kernel: [  484.242885]  nbd_xmit_timeout+0xb0/0x180 [nbd]
Nov 20 13:17:13 debian kernel: [  484.242887]  blk_mq_check_expired+0xe9/0x100
Nov 20 13:17:13 debian kernel: [  484.242888]  bt_iter+0x4e/0x60
Nov 20 13:17:13 debian kernel: [  484.242889]  blk_mq_queue_tag_busy_iter+0x184/0x2e0
Nov 20 13:17:13 debian kernel: [  484.242890]  ? blk_mq_poll+0x360/0x360
Nov 20 13:17:13 debian kernel: [  484.242891]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242892]  ? blk_mq_poll+0x360/0x360
Nov 20 13:17:13 debian kernel: [  484.242893]  ? syscall_return_via_sysret+0x13/0x83
Nov 20 13:17:13 debian kernel: [  484.242894]  blk_mq_timeout_work+0x54/0x120
Nov 20 13:17:13 debian kernel: [  484.242896]  process_one_work+0x1a7/0x3a0
Nov 20 13:17:13 debian kernel: [  484.242897]  worker_thread+0x30/0x390
Nov 20 13:17:13 debian kernel: [  484.242898]  ? create_worker+0x1a0/0x1a0
Nov 20 13:17:13 debian kernel: [  484.242899]  kthread+0x112/0x130
Nov 20 13:17:13 debian kernel: [  484.242900]  ? kthread_bind+0x30/0x30
Nov 20 13:17:13 debian kernel: [  484.242901]  ret_from_fork+0x35/0x40
Nov 20 13:17:13 debian kernel: [  484.242923] INFO: task jbd2/dm-11-8:3239 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [  484.242924]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [  484.242925] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [  484.242925] jbd2/dm-11-8    D    0  3239      2 0x80000000
Nov 20 13:17:13 debian kernel: [  484.242926] Call Trace:
Nov 20 13:17:13 debian kernel: [  484.242928]  ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [  484.242929]  ? __wake_up_common_lock+0x89/0xc0
Nov 20 13:17:13 debian kernel: [  484.242930]  ? finish_wait+0x80/0x80
Nov 20 13:17:13 debian kernel: [  484.242931]  schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [  484.242935]  jbd2_journal_commit_transaction+0x257/0x1890 [jbd2]
Nov 20 13:17:13 debian kernel: [  484.242937]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242938]  ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [  484.242939]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242940]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242941]  ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [  484.242942]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242943]  ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [  484.242944]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242945]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242946]  ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [  484.242947]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242948]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242949]  ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [  484.242950]  ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [  484.242951]  ? finish_wait+0x80/0x80
Nov 20 13:17:13 debian kernel: [  484.242952]  ? lock_timer_base+0x67/0x80
Nov 20 13:17:13 debian kernel: [  484.242955]  kjournald2+0xbd/0x270 [jbd2]
Nov 20 13:17:13 debian kernel: [  484.242956]  ? finish_wait+0x80/0x80
Nov 20 13:17:13 debian kernel: [  484.242959]  ? commit_timeout+0x10/0x10 [jbd2]
Nov 20 13:17:13 debian kernel: [  484.242960]  kthread+0x112/0x130
Nov 20 13:17:13 debian kernel: [  484.242960]  ? kthread_bind+0x30/0x30
Nov 20 13:17:13 debian kernel: [  484.242962]  ret_from_fork+0x35/0x40
Nov 20 13:17:13 debian kernel: [  484.242963] INFO: task umount:3921 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [  484.242964]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [  484.242964] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [  484.242965] umount          D    0  3921   2085 0x00000000
Nov 20 13:17:13 debian kernel: [  484.242966] Call Trace:
Nov 20 13:17:13 debian kernel: [  484.242967]  ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [  484.242968]  ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [  484.242969]  schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [  484.242970]  rwsem_down_write_failed+0x17c/0x3a0
Nov 20 13:17:13 debian kernel: [  484.242972]  ? mnt_get_count+0x39/0x50
Nov 20 13:17:13 debian kernel: [  484.242974]  call_rwsem_down_write_failed+0x13/0x20
Nov 20 13:17:13 debian kernel: [  484.242976]  down_write+0x29/0x40
Nov 20 13:17:13 debian kernel: [  484.242977]  deactivate_super+0x43/0x50
Nov 20 13:17:13 debian kernel: [  484.242979]  cleanup_mnt+0x3f/0x70
Nov 20 13:17:13 debian kernel: [  484.242980]  task_work_run+0x8a/0xb0
Nov 20 13:17:13 debian kernel: [  484.242981]  exit_to_usermode_loop+0xeb/0xf0
Nov 20 13:17:13 debian kernel: [  484.242983]  do_syscall_64+0x10d/0x110
Nov 20 13:17:13 debian kernel: [  484.242984]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov 20 13:17:13 debian kernel: [  484.242985] RIP: 0033:0x7f08434ec507
Nov 20 13:17:13 debian kernel: [  484.242988] Code: Bad RIP value.
Nov 20 13:17:13 debian kernel: [  484.242989] RSP: 002b:00007ffed9c8ec38 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
Nov 20 13:17:13 debian kernel: [  484.242990] RAX: 0000000000000000 RBX: 000055b66751ca40 RCX: 00007f08434ec507
Nov 20 13:17:13 debian kernel: [  484.242990] RDX: 0000000000000003 RSI: 0000000000000002 RDI: 000055b66751d290
Nov 20 13:17:13 debian kernel: [  484.242991] RBP: 0000000000000002 R08: 000055b667525fd0 R09: 00007f084356de80
Nov 20 13:17:13 debian kernel: [  484.242991] R10: 0000000000000000 R11: 0000000000000202 R12: 000055b66751d290
Nov 20 13:17:13 debian kernel: [  484.242992] R13: 00007f08436121c4 R14: 000055b66751cb38 R15: 000055b667525890
Nov 20 13:17:13 debian kernel: [  484.242993] INFO: task dumpe2fs:3922 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [  484.242994]       Tainted: G           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [  484.242994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [  484.242995] dumpe2fs        D    0  3922    599 0x00000000
Nov 20 13:17:13 debian kernel: [  484.242996] Call Trace:
Nov 20 13:17:13 debian kernel: [  484.242998]  ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [  484.242999]  schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [  484.242999]  io_schedule+0x12/0x40
Nov 20 13:17:13 debian kernel: [  484.243001]  wbt_wait+0x205/0x300
Nov 20 13:17:13 debian kernel: [  484.243002]  ? trace_event_raw_event_wbt_step+0x130/0x130
Nov 20 13:17:13 debian kernel: [  484.243003]  rq_qos_throttle+0x31/0x40
Nov 20 13:17:13 debian kernel: [  484.243004]  blk_mq_make_request+0x111/0x530
Nov 20 13:17:13 debian kernel: [  484.243006]  generic_make_request+0x1a4/0x400
Nov 20 13:17:13 debian kernel: [  484.243007]  submit_bio+0x45/0x140
Nov 20 13:17:13 debian kernel: [  484.243008]  ? guard_bio_eod+0x32/0x100
Nov 20 13:17:13 debian kernel: [  484.243009]  submit_bh_wbc+0x163/0x190
Nov 20 13:17:13 debian kernel: [  484.243010]  __block_write_full_page+0x234/0x420
Nov 20 13:17:13 debian kernel: [  484.243012]  ? check_disk_change+0x60/0x60
Nov 20 13:17:13 debian kernel: [  484.243013]  ? touch_buffer+0x60/0x60
Nov 20 13:17:13 debian kernel: [  484.243014]  __writepage+0x19/0x50
Nov 20 13:17:13 debian kernel: [  484.243015]  write_cache_pages+0x1e1/0x470
Nov 20 13:17:13 debian kernel: [  484.243016]  ? __wb_calc_thresh+0x130/0x130
Nov 20 13:17:13 debian kernel: [  484.243017]  generic_writepages+0x56/0x90
Nov 20 13:17:13 debian kernel: [  484.243018]  do_writepages+0x41/0xd0
Nov 20 13:17:13 debian kernel: [  484.243020]  ? __handle_mm_fault+0x1090/0x1270
Nov 20 13:17:13 debian kernel: [  484.243021]  __filemap_fdatawrite_range+0xbe/0xf0
Nov 20 13:17:13 debian kernel: [  484.243023]  file_write_and_wait_range+0x4c/0xa0
Nov 20 13:17:13 debian kernel: [  484.243024]  blkdev_fsync+0x16/0x40
Nov 20 13:17:13 debian kernel: [  484.243025]  do_fsync+0x38/0x70
Nov 20 13:17:13 debian kernel: [  484.243027]  __x64_sys_fsync+0x10/0x20
Nov 20 13:17:13 debian kernel: [  484.243028]  do_syscall_64+0x53/0x110
Nov 20 13:17:13 debian kernel: [  484.243029]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov 20 13:17:13 debian kernel: [  484.243030] RIP: 0033:0x7f0790255214
Nov 20 13:17:13 debian kernel: [  484.243031] Code: Bad RIP value.
Nov 20 13:17:13 debian kernel: [  484.243032] RSP: 002b:00007fff7f9474d8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Nov 20 13:17:13 debian kernel: [  484.243032] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0790255214
Nov 20 13:17:13 debian kernel: [  484.243033] RDX: 0000000000000008 RSI: 0000000000000000 RDI: 0000000000000003
Nov 20 13:17:13 debian kernel: [  484.243033] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007f07903ef680
Nov 20 13:17:13 debian kernel: [  484.243034] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f07903ef680
Nov 20 13:17:13 debian kernel: [  484.243034] R13: 000055cb7a993440 R14: 00007f07903ef680 R15: 0000000000000000
#942579#15
Date:
2019-11-20 13:40:00 UTC
From:
To:
So, I started to dig a bit deeper by simply downgrading the qmeu-utils package.

Running the recent version of buster but version qemu-utils_2.8+dfsg-6+deb9u8 (http://snapshot.debian.org/package/qemu/1%3A2.8%2Bdfsg-6%2Bdeb9u8/#qemu-utils_1:3a:2.8:2b:dfsg-6:2b:deb9u8) works.

All other versions greater than this one did not succeed. But with qemu-utils_2.8+dfsg-6+deb9u8 all works again, as expected.

I hope this helps.

Cheers,
Marcel
#942579#20
Date:
2021-06-08 04:01:56 UTC
From:
To:
Lest anyone think this is resolved, I just experienced it on buster,
running kernel 4.19.0-16-amd64 and 5.2+dfsg-9~bpo10+1 from
buster-backports.

In my case, though, I've used qemu-nbd without difficulty for raw
files on the order of 40G before without difficulty - but this time,
with a VDI of actual size ~19G and...let's call it "virtual" size
100GB, I wrote around a GB of data to it and then suddenly

[2005064.948700] block nbd0: Connection timed out
[2005064.951474] block nbd0: shutting down sockets
[2005064.951479] print_req_error: I/O error, dev nbd0, sector 39028592
[2005064.954230] block nbd0: Connection timed out
[2005064.956982] print_req_error: I/O error, dev nbd0, sector 39029104
[2005064.958271] block nbd0: Connection timed out
[2005064.959345] print_req_error: I/O error, dev nbd0, sector 39029360
[2005064.960527] block nbd0: Connection timed out
[2005064.961608] print_req_error: I/O error, dev nbd0, sector 39029616
[2005064.962677] block nbd0: Connection timed out
[2005064.963726] print_req_error: I/O error, dev nbd0, sector 39029872

(I presume if I hadn't noticed and had waited long enough I too would
see "task blocked for xyz seconds")

The fact that the original report on this particular bug was using a
VDI as well makes me suspect it might be a problem with handling VDIs
- I'm going to try converting it and report back...

- Rich
#942579#25
Date:
2021-06-12 11:57:53 UTC
From:
To:
I come bearing data!

After a kind of long bisect, the bad commit was:
commit 1e886639791762e89b51aa0507f523c6a1448831
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Thu Jun 29 15:27:41 2017 +0200

    vdi: make it thread-safe

And then, when I went to verify it on git master before reporting it
against qemu upstream, I couldn't reproduce it against
894fc4fd670aaf04a67dc7507739f914ff4bacf2.

Another round of bisecting points to
commit 050de36b13f7a841b7805391bca44f36370e86e4
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Thu Mar 25 12:29:39 2021 +0100

    coroutine-lock: Reimplement CoRwlock to fix downgrade bug

as the first commit where it works reliably. Which is unfortunate, as
cherrypicking that looks a bit more invasive than is probably
reasonable.

- Rich