Dear Maintainer,

I have a server setup with Xen and 12 domU's strictly using Debian,
since squeeze, wheezy, and now jessie.  I had been minimally using
libvirt with munin-node to poll statistics.  After upgrading to jessie,
and converting Xen configuration to xl.cfg format, libvirt no longer
enumerates any of the domU's, and does not report any errors other than
information being empty.

For example:

  # virsh -c 'xen:///' list --all
   Id    Name                           State
  ----------------------------------------------------

Attached is "LIBVIRT_DEBUG=1 libvirtd" output.  At line 10640 is a
conneection from the aforementioned 'virsh' command.  Shortly thereafter
there are logs like:

  2015-05-01 04:15:52.835+0000: 23215: debug : do_open:1145 : trying driver 5 (xenlight) ...
  2015-05-01 04:15:52.835+0000: 23211: debug : virEventPollRunOnce:642 : EVENT_POLL_RUN: nhandles=10 timeout=4999
  2015-05-01 04:15:52.835+0000: 23215: debug : virObjectRef:296 : OBJECT_REF: obj=0x7f4ec3c232c0
  2015-05-01 04:15:52.835+0000: 23215: debug : virAccessManagerCheckConnect:218 : manager=0x7f4ec3c232c0(name=stack) driver=xenlight perm=0
  2015-05-01 04:15:52.835+0000: 23215: debug : virAccessManagerCheckConnect:218 : manager=0x7f4ec3c26940(name=none) driver=xenlight perm=0
  2015-05-01 04:15:52.835+0000: 23215: debug : virObjectUnref:259 : OBJECT_UNREF: obj=0x7f4ec3c232c0
  2015-05-01 04:15:52.835+0000: 23215: debug : do_open:1152 : driver 5 xenlight returned SUCCESS

AFAICT this looks like the new libxl driver is getting used.

Hi,

See:

https://libvirt.org/drvxen.html

you might still have xend running, but

...this looks like xenlight indeed but you need to create xenlight
configs as described at the above URL.
Cheers,
 -- Guido

I read that before reporting.  xend is no longer included in jessie (was
in wheezy xen-utils-4.1 package, no longer in jessie xen-utils-4.4
package).  I rebooted after dom0 dist-upgrade.  Definitely no xend
running.

Interesting.  I had been running libvirt thru the lifetime of squeeze
and wheezy without customizing anything under /etc/libvirt, given my
use-case of minimally using libvirt to report stats.  AIUI libvirt was
able to communicate with xend (4.0 and 4.1) and enumerate the domU's
managed by xend, however under Xen 4.4 and the switch to libxl, libvirt
requires configuration.  If that is true then this bug should probably
be closed or at the very least changed to priority 'wishlist'.

The operations side of things seems a bit under-documented, so pardon me
for being verbose, as this may be helpful to someone.

Following the “Converting from XM config files to domain XML”, it's not
obvious where to store the converted config.

Running libvirtd with LIBVIRT_DEBUG=1, I see the following message:

  info : virDomainObjListLoadAllConfigs:19450 : Scanning for configs in /etc/libvirt/libxl

Looks like libvirtd expects configuration in this directory.

  # mkdir /etc/libvirt/libxl
  # virsh -c xen:/// domxml-from-native xen-xm /etc/xen/host1.cfg \
     > /etc/libvirt/libxl/host1.xml

Then restart libvirtd.  Nothing has changed.  More debug output:

  info : virDomainObjListLoadAllConfigs:19450 : Scanning for configs in /etc/libvirt/libxl
  info : virDomainObjListLoadAllConfigs:19474 : Loading config file 'host1.xml'
  error : virDomainDiskDefParseXML:5949 : internal error: Invalid harddisk device name: raw

Inspecting the converted domain XML, I see disk declarations like:

  <disk type='block' device='disk'>
    <driver name='phy'/>
    <source dev='/dev/vg-ssd/lv-host1-root'/>
    <target dev='raw' bus='xen'/>
  </disk>

… dev=raw should probably be dev=xvda, etc.  Otherwise everything else
looks okay.  The libvirt Xen driver documentation speaks of 'XM', while
Xen 4.4 has changed to 'XL' configiration files.  Looking at the libvirt
ChangeLog, I see upstream versions that probably fix this problem:

  * 1.2.12: Jan 27 2015
      libxl: Add support for parsing/formating Xen XL config (Kiarie Kahurani)
      Introduce support for parsing/formatting Xen xl config format (Jim Fehlig)

After correcting the domain XML, and restarting libvirtd once more,
things start progressing:

  # virsh -c xen:/// list --all
   Id    Name                           State
  ----------------------------------------------------
   -     host1                          shut off

… however “shut off” is incorrect.

Not sure how to proceed, other than abandon libvirt, for Xen anyway.
Maybe look at it again during 'stretch'.

On Fri, May 01, 2015 at 12:51:37PM -0700, Gerald Turner wrote:
[..snip..]

We could cherry pick those to our stable release.

Stupid question but did you start the domain via virsh beforehand? If
so, is there anything interesting in the daemon log? AFAIK
configurations are now managed entierly via /etc/libvirt/libxl/ .

Cheers,
 -- Guido

Control: severity -1 wishlist

Nice!  However unless somebody else comes along and has the same
expectations that I had (i.e. libvirt recognizing Xen guests with little
or no configuration), I wouldn't put any effort into this.

Not stupid because that's part of the point I've subtly been trying to
make - I'm sticking with domU's started by /etc/init.d/xendomains and
/usr/sbin/xl, the "Xen way".  For four years during squeeze and wheezy,
I had dropped in munin-libvirt-plugins and libvirt-bin (0.8.3 and
0.9.12) for the sole purpose of monitoring the domU's memory/cpu/io.
This worked without any configuration other than specifying "uri
xen:///" in the munin-node plugin configuration.  During the
"heartbleed" panic, after hastily reacting to the output of
checkrestart, I accidently discovered that "service libvirt-guests
restart" actually handles restarts of these "unmanaged" domU's.
Fantastic!  Consequently I've been telling sysadmin colleagues that
libvirt is great because it integrates non-intrusively and can handle
heterogeneous VM environments.

Sorry - I was too quick to react after upgrading to jessie and opening
this bug.  FWICT libvirt+Xen/xl no longer supports this "unmanaged"
use-case like the way libvirt+Xen/xend was able to in previous releases.
In the meantime I've dropped in a non-Debian-packaged Xen munin plugin¹
and have stopped relying on libvirt.

Perhaps one day I'll experiment with a managed libvirt+Xen setup on a
non-Production system and report some real bugs ;-)

¹ http://munin-monitoring.org/browser/munin-contrib/plugins/virtualization/xen-multi

retitle 783901 libvirtd does not recognize Xen guests managed via /usr/sbin/xl

Hi,

Could you attach a configuration to this bug that shows he conversion
errors. I'll try to get this fixed so we ease the migration. That said
there are lots of libxl changes in newer libvirt so running a backported
1.2.15 might make more sense.
managed xen domains aren't picked up by /usr/sbin/xl and vice versa.

It's bad that this isn't true for Jessie anymore but looking at the
uptream list it seems most distros are switching over to libvirt managed
xen domains.

I think the bug is valid and we should leave it open since others might
stomp on it as well.

Cheers,
 -- Guido

Attached example.cfg in xl.cfg(5) format, and the produced example.xml
when running ‘virsh domxml-from-native xen-xm example.cfg >|
example.xml’.

The only problem seems to be parsing the disks, which is specified in
the xl-disk-configuration.txt¹ document, wherein it looks like the
<vdev> and <format> positional parameters were swapped from the older xm
format.

Not quite.

xl managed xen domains aren't picked up by libvirt.

As for vice-versa, sorry but I haven't tried whether libvirt managed xen
domains are picked up by xl.

Interesting.  I wonder if systemd-machined has been integrated with
libvirt - yet another compelling reason to change management stacks.

¹ http://xenbits.xen.org/docs/4.4-testing/misc/xl-disk-configuration.txt

libvirt/libxl with libvirt managed domU a bit further:

  * shutdown a particular domU (xl shutdown host)
  * removed it's /etc/xen/host.cfg file (no way it's xl managed anymore)
  * created /etc/libvirt/libxl/host.xml file
  * restarted libvirtd
  * started the domU with libvirt (virsh start host)

This worked fine.  The ‘virsh list’ output properly reports ‘running’.
This also answers your earlier vice-versus question: the xl tool *does*
recognize the libvirt managed domU just fine.  In other words, the
best-of-both-worlds setup.

However after doing this I discovered that there are many features in
the libvirt/libxl driver that are unimplemented¹, in particular, missing
functions that the originally sought after munin-libvirt-plugins
requires, for example:

  virsh # domblkstat host
  error: Failed to get block stats host
  error: this function is not supported by the connection driver: virDomainBlockStats

Scanning through git logs², I see no indication that these missing
features are being worked on.

Given that nothing has been gained, and that I'm not using any libvirt
software that doesn't depend on unimplemented features of the
libvirt/libxl driver, I'll be reverting back to xl managed.

BTW, to answer my off-topic question about whether the libvirt stack
integrates with systemd-machined, apparently not - nothing changed with
machinectl output: systemd is unaware of the libvirt managed domU.

¹ https://libvirt.org/hvsupport.html
² http://libvirt.org/git/?p=libvirt.git;a=history;f=src/libxl/libxl_driver.c