Dear Maintainer,

I run autopkgtest -U -B -u debci mutipath-tools -- qemu with a testbed made by
debci setup -f -s sid -a amd64 -b qemu.

The testsuite gives
kpartx-file-loopback FAIL stderr: Warning: Partition table header claims that the size of partition table
tgtbasedmpaths       FAIL non-zero exit status 1

Looking at the log, failure of tgtbasedmpaths seems a real error.

The log is attached.

Best regards, Ryutaroh Matsumoto

Hi Ryutaroh Matsumoto,


I skimmed into the logs but I'm not sure what failure is being referred
to here.

Hi Ritesh Raj Sarraf,
Thank you for paying your attention to this.
if you consider this as a false positive.
In the following, the actual test results seem different from what are
expected by the test script in multipath-tools.

NVMe module may not be loaded
===== paths list =====
uuid hcil    dev dev_t pri dm_st chk_st vend/prod/rev    dev_st
     2:0:0:1 sda 8:0   -1  undef undef  IET,VIRTUAL-DISK unknown
     3:0:0:1 sdb 8:16  -1  undef undef  IET,VIRTUAL-DISK unknown
     4:0:0:1 sdc 8:32  -1  undef undef  IET,VIRTUAL-DISK unknown
     5:0:0:1 sdd 8:48  -1  undef undef  IET,VIRTUAL-DISK unknown
No devices found
Test WWN should now point to DM
+ lsscsi -liv
list_ndevices: scandir: /sys/class/nvme/: No such file or directory
+ multipath -v3 -ll
Feb 23 00:09:07 | set open fds limit to 1048576/1048576
Feb 23 00:09:07 | loading //lib/multipath/libchecktur.so checker
Feb 23 00:09:07 | checker tur: message table size = 3
Feb 23 00:09:07 | loading //lib/multipath/libprioconst.so prioritizer
Feb 23 00:09:07 | _init_foreign: foreign library "nvme" is not enabled
Feb 23 00:09:07 | sr0: device node name blacklisted
Feb 23 00:09:07 | vda: device node name blacklisted
Feb 23 00:09:07 | fd0: device node name blacklisted
Feb 23 00:09:07 | sda: size = 204800
Feb 23 00:09:07 | sda: vendor = IET
Feb 23 00:09:07 | sda: product = VIRTUAL-DISK
Feb 23 00:09:07 | sda: rev = 0001
Feb 23 00:09:07 | sda: h:b:t:l = 2:0:0:1
Feb 23 00:09:07 | sda: tgt_node_name = iqn.2016-11.foo.com:target.iscsi
Feb 23 00:09:07 | sda: 1024 cyl, 4 heads, 50 sectors/track, start at 0
Feb 23 00:09:07 | sda: vpd_vendor_id = 0 "undef" (setting: multipath internal)
Feb 23 00:09:07 | sda: serial = beaf11
Feb 23 00:09:07 | sda: detect_checker = yes (setting: multipath internal)
Feb 23 00:09:07 | sda: path_checker = tur (setting: multipath internal)
Feb 23 00:09:07 | sda: checker timeout = 30 s (setting: kernel sysfs)
Feb 23 00:09:07 | sda: tur state = up
Feb 23 00:09:07 | sdb: size = 204800
Feb 23 00:09:07 | sdb: vendor = IET
Feb 23 00:09:07 | sdb: product = VIRTUAL-DISK
Feb 23 00:09:07 | sdb: rev = 0001
Feb 23 00:09:07 | sdb: h:b:t:l = 3:0:0:1
Feb 23 00:09:07 | sdb: tgt_node_name = iqn.2016-11.foo.com:target.iscsi
Feb 23 00:09:07 | sdb: 1024 cyl, 4 heads, 50 sectors/track, start at 0
Feb 23 00:09:07 | sdb: vpd_vendor_id = 0 "undef" (setting: multipath internal)
Feb 23 00:09:07 | sdb: serial = beaf11
Feb 23 00:09:07 | sdb: detect_checker = yes (setting: multipath internal)
Feb 23 00:09:07 | sdb: path_checker = tur (setting: multipath internal)
Feb 23 00:09:07 | sdb: checker timeout = 30 s (setting: kernel sysfs)
Feb 23 00:09:07 | sdb: tur state = up
Feb 23 00:09:07 | sdc: size = 204800
Feb 23 00:09:07 | sdc: vendor = IET
Feb 23 00:09:07 | sdc: product = VIRTUAL-DISK
Feb 23 00:09:07 | sdc: rev = 0001
Feb 23 00:09:07 | sdc: h:b:t:l = 4:0:0:1
Feb 23 00:09:07 | sdc: tgt_node_name = iqn.2016-11.foo.com:target.iscsi
Feb 23 00:09:07 | sdc: 1024 cyl, 4 heads, 50 sectors/track, start at 0
Feb 23 00:09:07 | sdc: vpd_vendor_id = 0 "undef" (setting: multipath internal)
Feb 23 00:09:07 | sdc: serial = beaf11
Feb 23 00:09:07 | sdc: detect_checker = yes (setting: multipath internal)
Feb 23 00:09:07 | sdc: path_checker = tur (setting: multipath internal)
Feb 23 00:09:07 | sdc: checker timeout = 30 s (setting: kernel sysfs)
Feb 23 00:09:07 | sdc: tur state = up
Feb 23 00:09:07 | sdd: size = 204800
Feb 23 00:09:07 | sdd: vendor = IET
Feb 23 00:09:07 | sdd: product = VIRTUAL-DISK
Feb 23 00:09:07 | sdd: rev = 0001
Feb 23 00:09:07 | sdd: h:b:t:l = 5:0:0:1
Feb 23 00:09:07 | sdd: tgt_node_name = iqn.2016-11.foo.com:target.iscsi
Feb 23 00:09:07 | sdd: 1024 cyl, 4 heads, 50 sectors/track, start at 0
Feb 23 00:09:07 | sdd: vpd_vendor_id = 0 "undef" (setting: multipath internal)
Feb 23 00:09:07 | sdd: serial = beaf11
Feb 23 00:09:07 | sdd: detect_checker = yes (setting: multipath internal)
Feb 23 00:09:07 | sdd: path_checker = tur (setting: multipath internal)
Feb 23 00:09:07 | sdd: checker timeout = 30 s (setting: kernel sysfs)
Feb 23 00:09:07 | sdd: tur state = up
Feb 23 00:09:07 | libdevmapper version 1.02.175 (2021-01-08)
Feb 23 00:09:07 | DM multipath kernel driver v1.14.0
Feb 23 00:09:07 | unloading const prioritizer
Feb 23 00:09:07 | unloading tur checker
+ dmsetup table
+ echo Test WWN should now point to DM
+ grep dm
+ readlink /dev/disk/by-id/scsi-360000000000000000e00000000010001

Best regards, Ryutaroh

I really don't have much insight into these tests. So I can't judge if
the report is correct or not.

I usually do manual tests as a pre-requisite to my uploads.

Looking at the git logs, these tests were contributed by the Ubuntu
folks (CCed). Hopefully they may have more insight into it.

Other than the tests, does multipath-tools work proper for you ?
From the logs you've shared, multipath does report to see the iSCSI
LUNs. Now whether a proper map was created or not, is not clear from
the logs.

I have seen no problems in bin. packages from src:multipath-tools.
I run autopkgtest qemu on all packages with standard priority and
in task-gnome-desktop, and reported all the errors I observed (as
tests finished) except the packages already erred on ci.debian.net.
Some of them turned out to be real errors...

Best regards, Ryutaroh

It's similar to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=981231
and a warning from the new gdisks version which is printed on stderr, or
autopkgtest fails on stderr content by default

Using Restrictions: allow-stderr should workaround the issue

I tried to reproduce it with the setup you mentioned.

```
rrs@lenovo:~$ autopkgtest -B -U -u debci -o /tmp/multipath-tools-qemu
multipath-tools -- qemu -d --show-boot -c 1 --ram-size=3072
/var/lib/debci/qemu/sid-amd64.img
autopkgtest [20:11:41]: starting date: 2021-03-03
autopkgtest [20:11:41]: version 5.16
autopkgtest [20:11:41]: host lenovo; command line: /usr/bin/autopkgtest
-B -U -u debci -o /tmp/multipath-tools-qemu multipath-tools -- qemu -d
--show-boot -c 1 --ram-size=3072 /var/lib/debci/qemu/sid-amd64.img
autopkgtest-virt-qemu: DBG: executing open
autopkgtest-virt-qemu: DBG: find_free_port: trying 10022
autopkgtest-virt-qemu: DBG: find_free_port: 10022 is free
autopkgtest-virt-qemu: DBG: execute-timeout: qemu-img info --
output=json /var/lib/debci/qemu/sid-amd64.img
autopkgtest-virt-qemu: DBG: Creating temporary overlay image in
/tmp/autopkgtest-qemu.3tdfx4at/overlay.img
autopkgtest-virt-qemu: DBG: execute-timeout: qemu-img create -f qcow2 -
F qcow2 -b /var/lib/debci/qemu/sid-amd64.img /tmp/autopkgtest-
qemu.3tdfx4at/overlay.img
autopkgtest-virt-qemu: DBG: Forwarding local port 10022 to VM ssh port
22
autopkgtest-virt-qemu: DBG: Detected KVM capable Intel host CPU,
enabling nested KVM
autopkgtest-virt-qemu: DBG: expect: " login: "
```

and


```
SeaBIOS (version 1.14.0-2)


iPXE (http://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+BFF8F2D0+BFECF2D0
CA00



Booting from Hard Disk...
autopkgtest-virt-qemu: DBG: cleanup...
<VirtSubproc>: failure: timed out waiting for "login prompt on ttyS0"
autopkgtest [20:12:41]: ERROR: testbed failure: cannot send to testbed:
[Errno 32] Broken pipe
20:12 ♒♒♒    ☹ 😟=> 16
```


Is it something you can help me with, to reproduce the actual
multipath-tools issue ?

I figured out the setup for debci/autopkgtest all thanks to your bug
reports that motivated me. :-)

Hello Sebastien
test.

```
Tests: kpartx-file-loopback
Depends: kpartx, qemu-utils, gdisk
Restrictions: needs-root, isolation-machine, allow-stderr

Tests: tgtbasedmpaths
Depends: multipath-tools, tgt, open-iscsi, fio, lsscsi
Restrictions: needs-root, isolation-machine, allow-stderr
```

But it still fails.

```
rrs@lenovo:/tmp/multipath-tools-qemu$ cat summary
kpartx-file-loopback PASS
tgtbasedmpaths       FAIL non-zero exit status 1
21:07 ♒♒♒   ☺ 😄
rrs@lenovo:/tmp/multipath-tools-qemu$ tail -n 50 log
Mar 03 15:33:13 | sdd: checker timeout = 30 s (setting: kernel sysfs)
Mar 03 15:33:13 | sdd: tur state = up
===== paths list =====
uuid hcil    dev dev_t pri dm_st chk_st vend/prod/rev    dev_st
     2:0:0:1 sda 8:0   -1  undef undef  IET,VIRTUAL-DISK unknown
     3:0:0:1 sdb 8:16  -1  undef undef  IET,VIRTUAL-DISK unknown
     4:0:0:1 sdc 8:32  -1  undef undef  IET,VIRTUAL-DISK unknown
     5:0:0:1 sdd 8:48  -1  undef undef  IET,VIRTUAL-DISK unknown
No devices found
Test WWN should now point to DM
Mar 03 15:33:13 | libdevmapper version 1.02.175 (2021-01-08)
Mar 03 15:33:13 | DM multipath kernel driver v1.14.0
Mar 03 15:33:13 | unloading const prioritizer
Mar 03 15:33:13 | unloading tur checker
+ dmsetup table
+ echo Test WWN should now point to DM
+ grep dm
+ readlink /dev/disk/by-id/scsi-360000000000000000e00000000010001
autopkgtest [21:03:14]: test tgtbasedmpaths: -----------------------]
autopkgtest-virt-qemu: DBG: executing copyup
/tmp/autopkgtest.6dFWDl/tgtbasedmpaths-stdout /tmp/multipath-tools-
qemu/tgtbasedmpaths-stdout
autopkgtest-virt-qemu: DBG: ['cmdls', "(['/tmp/autopkgtest-
qemu.0ejkv6rf/runcmd', 'sh', '-ec', 'cat
</tmp/autopkgtest.6dFWDl/tgtbasedmpaths-stdout'], ['cat'])"]
autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader
name='/dev/null'>", 'deststdout', "<_io.BufferedWriter
name='/tmp/multipath-tools-qemu/tgtbasedmpaths-stdout'>",
'devnull_read', <_io.BufferedReader name='/]
autopkgtest-virt-qemu: DBG:  +< /tmp/autopkgtest-qemu.0ejkv6rf/runcmd
sh -ec cat </tmp/autopkgtest.6dFWDl/tgtbasedmpaths-stdout
autopkgtest-virt-qemu: DBG:  +> cat
autopkgtest-virt-qemu: DBG:  +>?
autopkgtest-virt-qemu: DBG:  +<?
autopkgtest-virt-qemu: DBG: executing copyup
/tmp/autopkgtest.6dFWDl/tgtbasedmpaths-stderr /tmp/multipath-tools-
qemu/tgtbasedmpaths-stderr
autopkgtest-virt-qemu: DBG: ['cmdls', "(['/tmp/autopkgtest-
qemu.0ejkv6rf/runcmd', 'sh', '-ec', 'cat
</tmp/autopkgtest.6dFWDl/tgtbasedmpaths-stderr'], ['cat'])"]
autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader
name='/dev/null'>", 'deststdout', "<_io.BufferedWriter
name='/tmp/multipath-tools-qemu/tgtbasedmpaths-stderr'>",
'devnull_read', <_io.BufferedReader name='/]
autopkgtest-virt-qemu: DBG:  +< /tmp/autopkgtest-qemu.0ejkv6rf/runcmd
sh -ec cat </tmp/autopkgtest.6dFWDl/tgtbasedmpaths-stderr
autopkgtest-virt-qemu: DBG:  +> cat
autopkgtest-virt-qemu: DBG:  +>?
autopkgtest-virt-qemu: DBG:  +<?
autopkgtest [21:03:15]: test tgtbasedmpaths:  - - - - - - - - - -
results - - - - - - - - - -
tgtbasedmpaths       FAIL non-zero exit status 1
autopkgtest-virt-qemu: DBG: executing copyup
/tmp/autopkgtest.6dFWDl/tgtbasedmpaths-artifacts/ /tmp/multipath-tools-
qemu/artifacts/
autopkgtest-virt-qemu: DBG: ['cmdls', "(['/tmp/autopkgtest-
qemu.0ejkv6rf/runcmd', 'sh', '-ec', 'cd
/tmp/autopkgtest.6dFWDl/tgtbasedmpaths-artifacts/; tar --warning=none -
c . -f -'], ['tar', '--directory', '/tmp/multipa]
autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader
name='/dev/null'>", 'deststdout', "<_io.BufferedReader
name='/dev/null'>", 'devnull_read', <_io.BufferedReader
name='/dev/null'>]
autopkgtest-virt-qemu: DBG:  +< /tmp/autopkgtest-qemu.0ejkv6rf/runcmd
sh -ec cd /tmp/autopkgtest.6dFWDl/tgtbasedmpaths-artifacts/; tar --
warning=none -c . -f -
autopkgtest-virt-qemu: DBG:  +> tar --directory /tmp/multipath-tools-
qemu/artifacts/ --warning=none --preserve-permissions --extract --no-
same-owner -f -
autopkgtest-virt-qemu: DBG:  +>?
autopkgtest-virt-qemu: DBG:  +<?
autopkgtest [21:03:15]: @@@@@@@@@@@@@@@@@@@@ summary
kpartx-file-loopback PASS
tgtbasedmpaths       FAIL non-zero exit status 1
autopkgtest-virt-qemu: DBG: executing close
autopkgtest-virt-qemu: DBG: cleanup...
qemu-system-x86_64: terminating on signal 15 from pid 101442
(/usr/bin/python3)
autopkgtest-virt-qemu: DBG: executing quit
autopkgtest-virt-qemu: DBG: cleanup...
21:07 ♒♒♒   ☺ 😄
```

This should work...
You could try the same command by root, but
it should work under an ordinary user.

Have you seen "Grub" or "Linux"?
If not, the installation of grub and/or linux-image-amd64 is broken
in the VM image /var/lib/debci/qemu/sid-amd64.img.
I wonder if /var/lib/debci/qemu/sid-amd64.img can be booted under virt-manager.

I also see a problem in grub-pc/sid.
Could you try

# debci setup -f -s bullseye -a amd64 -b qemu
Then
(sid replaced by bullseye).

Best regards, Ryutaroh

Control: tag -1 help

I was finally able to boot and test a suggestion from Sebastien. But
the testcase still fails.

And, by now, you must have realized that I'm complete noob in
debci/autopkgtest so I'll have to rely on some other expert in
resolving this.