#797513 Regression in support for custom CPU configuration

#797513#5
Date:
2015-08-31 09:06:04 UTC
From:
To:
Since the upgrade to 1.2.18-2, most of my VMs fail to start:

virsh # start sid
error: Failed to start domain sid
error: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel

This is the CPU I have:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 42
model name	: Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz
stepping	: 7
microcode	: 0x29
cpu MHz		: 825.703
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid xsaveopt
bugs		:
bogomips	: 5183.58
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

[other 3 threads omitted]

and here's the CPU configuration for one of those domains:

  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='avx'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='xsave'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='lm'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='sse4.1'/>
    <feature policy='require' name='sse4.2'/>
    <feature policy='require' name='pclmuldq'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='aes'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='invtsc'/>
    <feature policy='require' name='rdtscp'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='cx16'/>
    <feature policy='require' name='pse36'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='popcnt'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='disable' name='syscall'/>
  </cpu>

(Why a custom CPU configuration?  Because 'copy host configuration'
does something crazy.  The VM gets an 'Atom N270' (a 32-bit processor
in reality) with a random assortment of CPU flags, including 'lm' but
not 'syscall'.  llvmpipe also seems to be unhappy with this, resulting
in a sadface when gdm3 starts.)

If I delete the <cpu> element entirely from the XML file, the VM works
again.

Ben.

#797513#10
Date:
2015-08-31 09:37:55 UTC
From:
To:
Hi,

Can you check which version you updated from so I can go digging what
changed in this area?

Looks pretty much like a Intel CPU :(. Any chance you can check
1.2.19-rc1 from experimental to see if this regression got fixed.

Cheers,
 -- Guido

#797513#15
Date:
2015-08-31 16:27:40 UTC
From:
To:
1.2.16-2

I'll try that later.

Ben.

#797513#18
Date:
2015-08-31 22:24:14 UTC
From:
To:
Hi, same problem here:

$ sudo virsh start win7
error: Failed to start domain win7
error: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
$ sudo virsh dumpxml win7 | sed -n '\|<cpu|,\|</cpu|p'
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>coreduo</model>
    <vendor>Intel</vendor>
  </cpu>
$ lscpu | grep -e Vendor -e Model
Vendor ID:             GenuineIntel
Model:                 14
Model name:            Genuine Intel(R) CPU           T2400  @ 1.83GHz

libvirt 1.2.19~rc1-1 from experimental has this bug as well.

HAND,
Nikolaus

#797513#21
Date:
2015-08-31 22:24:14 UTC
From:
To:
Hi, same problem here:

$ sudo virsh start win7
error: Failed to start domain win7
error: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
$ sudo virsh dumpxml win7 | sed -n '\|<cpu|,\|</cpu|p'
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>coreduo</model>
    <vendor>Intel</vendor>
  </cpu>
$ lscpu | grep -e Vendor -e Model
Vendor ID:             GenuineIntel
Model:                 14
Model name:            Genuine Intel(R) CPU           T2400  @ 1.83GHz

libvirt 1.2.19~rc1-1 from experimental has this bug as well.

HAND,
Nikolaus

#797513#26
Date:
2015-09-01 05:38:10 UTC
From:
To:
Hi,

Can you please attach the output of

    virsh capabilities | xpath -e //host

I'd be keen to know what the <vendor> element says. I retried on a
Westmere and the CPU type is detected correctly.

Cheers,
 -- Guido

#797513#31
Date:
2015-09-01 10:40:52 UTC
From:
To:
Attached.

Ben.

#797513#36
Date:
2015-09-01 11:27:30 UTC
From:
To:
Hi,
On Tue, Sep 01, 2015 at 11:40:52AM +0100, Ben Hutchings wrote:
[..snip..]

It's indeed lacking the vendor attribute. I have:

  $ virsh capabilities | xpath -q -e //host/cpu/vendor
  <vendor>Intel</vendor>

while yours is empty which seems to be the regression. Can you check
syslog if there are any warnings or errors printed during libvirt
startup? If not I'm a bit puzzled since nothing changed recently in
this area.
Cheers,
 -- Guido

#797513#41
Date:
2015-09-01 11:41:22 UTC
From:
To:
Aug 30 19:10:50 deadeye libvirtd[11493]: libvirt version: 1.2.18, package: 2 (buildd 2015-08-23-18:38:12 x86-grnet-01)
Aug 30 19:10:50 deadeye libvirtd[11493]: End of file while reading data: Input/output error
Aug 30 19:15:17 deadeye libvirtd[11493]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 30 19:20:03 deadeye libvirtd[11493]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 30 19:21:52 deadeye libvirtd[18146]: libvirt version: 1.2.18, package: 2 (buildd 2015-08-23-18:38:12 x86-grnet-01)
Aug 30 19:21:52 deadeye libvirtd[18146]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 30 19:21:59 deadeye libvirtd[18146]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 30 19:22:02 deadeye libvirtd[18146]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 31 09:39:40 deadeye libvirtd[1233]: libvirt version: 1.2.18, package: 2 (buildd 2015-08-23-18:38:12 x86-grnet-01)
Aug 31 09:39:40 deadeye libvirtd[1233]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 31 09:40:35 deadeye libvirtd[1233]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 31 09:45:20 deadeye libvirtd[1233]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 31 09:52:18 deadeye libvirtd[1233]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 31 09:52:43 deadeye libvirtd[1233]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Aug 31 10:00:16 deadeye libvirtd[1233]: internal error: End of file from monitor
Aug 31 10:00:49 deadeye libvirtd[1233]: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: syscall
Aug 31 10:01:40 deadeye libvirtd[1233]: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: syscall
Aug 31 10:03:56 deadeye libvirtd[1233]: internal error: End of file from monitor
Aug 31 10:12:03 deadeye libvirtd[1233]: internal error: End of file from monitor
Aug 31 10:12:33 deadeye libvirtd[1233]: internal error: End of file from monitor
Aug 31 10:12:35 deadeye libvirtd[1233]: End of file while reading data: Input/output error
Sep  1 01:30:44 deadeye libvirtd[1233]: unsupported configuration: guest and host CPU are not compatible: host CPU vendor does not match required CPU vendor Intel
Sep  1 02:31:00 deadeye libvirtd[1233]: internal error: End of file from monitor
Sep  1 02:31:03 deadeye libvirtd[1233]: End of file while reading data: Input/output error
Sep  1 11:39:17 deadeye libvirtd[26426]: libvirt version: 1.2.18, package: 2 (buildd 2015-08-23-18:38:12 x86-grnet-01)
Sep  1 11:39:17 deadeye libvirtd[26426]: Failed to acquire pid file '/run/user/1000/libvirt/libvirtd.pid': Resource temporarily unavailable
Sep  1 11:39:17 deadeye libvirtd[26428]: libvirt version: 1.2.18, package: 2 (buildd 2015-08-23-18:38:12 x86-grnet-01)
Sep  1 11:39:17 deadeye libvirtd[26428]: Failed to acquire pid file '/run/user/1000/libvirt/libvirtd.pid': Resource temporarily unavailable
Sep  1 11:39:17 deadeye libvirtd[26430]: libvirt version: 1.2.18, package: 2 (buildd 2015-08-23-18:38:12 x86-grnet-01)
Sep  1 11:39:17 deadeye libvirtd[26430]: Failed to acquire pid file '/run/user/1000/libvirt/libvirtd.pid': Resource temporarily unavailable
Sep  1 11:39:58 deadeye libvirtd[30636]: libvirt version: 1.2.18, package: 2 (buildd 2015-08-23-18:38:12 x86-grnet-01)
Sep  1 11:39:58 deadeye libvirtd[30636]: Failed to acquire pid file '/run/user/1000/libvirt/libvirtd.pid': Resource temporarily unavailable
Sep  1 11:39:58 deadeye libvirtd[30638]: libvirt version: 1.2.18, package: 2 (buildd 2015-08-23-18:38:12 x86-grnet-01)
Sep  1 11:39:58 deadeye libvirtd[30638]: Failed to acquire pid file '/run/user/1000/libvirt/libvirtd.pid': Resource temporarily unavailable
Sep  1 11:39:58 deadeye libvirtd[30645]: libvirt version: 1.2.18, package: 2 (buildd 2015-08-23-18:38:12 x86-grnet-01)
Sep  1 11:39:58 deadeye libvirtd[30645]: Failed to acquire pid file '/run/user/1000/libvirt/libvirtd.pid': Resource temporarily unavailable

By the way, please don't capture the build hostname or time as this
makes the package unreproducible.

Ben.

#797513#44
Date:
2015-09-01 22:38:38 UTC
From:
To:
Again, same here.  I've downgraded libvirt to 1.2.16 again, and here's
the delta for the capabilities host node:
--- virsh-cap-1.2.16.xml 2015-09-02 00:20:50.989297520 +0200 +++ virsh-cap-1.2.19.xml 2015-09-02 00:19:58.709038277 +0200 @@ -1,10 +1,20 @@ <host> - <uuid>638c5e1e-b80b-4ae0-a519-73a90ca4c1ad</uuid> + <uuid>8c1ecff4-070d-4374-94f0-ce62022e53db</uuid> <cpu> <arch>i686</arch> <model>coreduo</model> - <vendor>Intel</vendor> <topology sockets="1" cores="2" threads="1" /> + <feature name="avx512cd" /> + <feature name="avx512pf" /> + <feature name="adx" /> + <feature name="rdseed" /> + <feature name="avx512f" /> + <feature name="invpcid" /> + <feature name="bmi2" /> + <feature name="smep" /> + <feature name="avx2" /> + <feature name="hle" /> + <feature name="bmi1" /> <feature name="pdcm" /> <feature name="xtpr" /> <feature name="tm2" /> The features avx512cd..bmi1 are actually *not* supported by my old Core Duo T2400 CPU, and libvirt 1.2.16 rightly complains about that if I feed it such a cpu definition on my box. I have no libvirt related errors in my log. Regards, Nikolaus.
#797513#47
Date:
2015-09-01 22:54:19 UTC
From:
To:
Ugh. Looking into /var/log/libvirt instead of syslog, I find plenty, see
attachment.

Nikolaus

#797513#54
Date:
2015-09-02 08:52:36 UTC
From:
To:
Hi,
On Mon, Aug 31, 2015 at 10:06:04AM +0100, Ben Hutchings wrote:
[..snip..]

I found s system with this cpu family / model / cpuid level and
microcode version and am not setting this bug. The <vendor> element is
perfectly there:

    $ virsh capabilities | xpath -q -e  //host/cpu/vendor
    <vendor>Intel</vendor>

Puzzled,
 -- Guido

#797513#59
Date:
2015-09-08 14:28:10 UTC
From:
To:
On Wed, Sep 02, 2015 at 00:54:19 +0200, Nikolaus Schulz wrote:
...

Interesting, so it seems syscall feature is missing in both cases, which
is why libvirt detects the CPU as n270 or coreduo (all new CPU models
require syscall feature). Would you mind attaching the output of the
"cpuid -1r" command? The cpuid binary is the one from a Debian package
called cpuid according to
https://packages.debian.org/search?keywords=cpuid

Could you also set log_filters="1:cpu 1:qemu 1:daemon" in
/etc/libvirt/libvirtd.conf, restart libvirtd and attach libvirtd.log?

Jirka

#797513#66
Date:
2015-09-15 10:05:51 UTC
From:
To:
Hi,

Could somebody seeing this provide Jiri with the requested information?
He's one of libvirt's upstream developers and very familiar with the cpu
detection code.
Cheers,
 -- Guido

#797513#69
Date:
2015-09-17 22:38:31 UTC
From:
To:
Attached.

Ditto.  Note that I temporarily upgraded to libvirt 1.2.19 for this, the
larger part of the log before that is still with 1.2.16.

Hope this helps!
Nikolaus

#797513#76
Date:
2015-09-18 13:53:02 UTC
From:
To:
Thanks, the attached file confirms the CPU does not support syscall.
Which means libvirt is correct in this part, I will play more with the
data to look into the vendor issue.

Hmm, the log doesn't contain any debug messages. Did you change
log_filters settings as suggested? And if so, does log_outputs look
similar to

    log_outputs="1:file:/var/log/libvirt/libvirtd.log"

(note the "1" at the beginning)?

Thanks

Jirka

#797513#81
Date:
2015-09-18 14:04:38 UTC
From:
To:
Hi Jiri,

Debian does not build with --enable-debug at the moment so the
VIR_DEBUG_INT is stubbed out. Looking at the spec file it seems
Fedora doesn't enable it either.

I already wondered ad debconf if it wouldn't be better if distros
would build with debugging enabled?

Cheers
 -- Guido

#797513#86
Date:
2015-09-18 14:24:13 UTC
From:
To:
So are you explicitly running configure with --disable-debug or
--enable-debug=no? Because debugging is enabled by default.

Jirka

#797513#91
Date:
2015-09-18 14:36:01 UTC
From:
To:
Hi,

I should have checked better. We're having:

   configure:             Debug: yes

so it should be on. Sorry for the noise.
Cheers,
 -- Guido

#797513#96
Date:
2015-09-18 15:13:59 UTC
From:
To:
[...]

Could this be due to running 32-bit userland on a 64-bit kernel?  Intel
CPUs do not support syscall from 32-bit code.

Ben.