(I realise that posting this on debian-devel [1] was probably not the
most appropriate place, as it's actually a bug report.)

I am not sure whether this is a bug in autopkgtest-build-lxc, a bug in
lxc itself or a user error.  Please feel free to redirect as
appropriate!

This is what I did:

Step 1: I installed the lxc and autopkgtest packages
That went smoothly.  (lxc version 1:4.0.11-1, autopkgtest version
5.21; autopkgtest was already installed, and I installed lxc from

Step 2: I ran the command "autopkgtest-build-lxc debian sid"
as root.  I got various warning messages to begin with:
lxc-create: autopkgtest-sid: storage/btrfs.c: btrfs_create: 938 Inappropriate ioctl for device - Failed to create btrfs subvolume "/var/lib/lxc/autopkgtest-sid/rootfs"
lxc-create: autopkgtest-sid: storage/zfs.c: zfs_create: 735 Failed to create zfs dataset "zfs:lxc/autopkgtest-sid": lxc-create: autopkgtest-sid: utils.c: run_command_internal: 1588
lxc-create: autopkgtest-sid: storage/lvm.c: do_lvm_create: 165 Failed to create logical volume "autopkgtest-sid":   Volume group "lxc" not found
  Cannot process volume group lxc
lxc-create: autopkgtest-sid: storage/lvm.c: lvm_create: 623 Error creating new logical volume "lvm:/dev/lxc/autopkgtest-sid" of size "1073741824 bytes"
<<<<<

after which things ran smoothly for a bit:
debootstrap is /usr/sbin/debootstrap
Checking cache download in /var/cache/lxc/debian/rootfs-sid-amd64 ...
Downloading debian minimal ...
I: Target architecture can be executed
I: Retrieving InRelease
[... downloading and installing base system ...]
I: Base system installed successfully.
Download complete.
<<<<<

but then there were lots of warning messages about libeatmydata.so
interspersed with information messages; I assume that these are mostly
harmless:

But then I received several fatal error messages:
lxc-start: autopkgtest-sid: lxccontainer.c: wait_on_daemonized_start: 867 Received container state "ABORTING" instead of "RUNNING"
lxc-start: autopkgtest-sid: tools/lxc_start.c: main: 306 The container failed to start
lxc-start: autopkgtest-sid: tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode
lxc-start: autopkgtest-sid: tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options
<<<<<

Since autopkgtest-build-lxc doesn't allow a --logfile option, I
attempted to start the container manually, using the command
  lxc-start -n autopkgtest-sid --logfile /tmp/lxc.log --logpriority INFO
and got the following warnings and errors in the log file (I've
excluded the INFO entries):
lxc-start autopkgtest-sid 20220501145802.680 NOTICE   conf - conf.c:lxc_setup:4450 - The container "autopkgtest-sid" is set up
lxc-start autopkgtest-sid 20220501145802.681 WARN     cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable devices controller
lxc-start autopkgtest-sid 20220501145802.681 ERROR    cgfsng - cgroups/cgfsng.c:cg_legacy_set_data:2675 - No such file or directory - Failed to setup limits for the "devices" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy
lxc-start autopkgtest-sid 20220501145802.681 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2742 - No such file or directory - Failed to set "devices.deny" to "a"
lxc-start autopkgtest-sid 20220501145802.681 ERROR    start - start.c:lxc_spawn:1890 - Failed to setup legacy device cgroup controller limits
lxc-start autopkgtest-sid 20220501145802.681 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:867 - Received container state "ABORTING" instead of "RUNNING"
lxc-start autopkgtest-sid 20220501145802.681 ERROR    lxc_start - tools/lxc_start.c:main:306 - The container failed to start
lxc-start autopkgtest-sid 20220501145802.681 ERROR    lxc_start - tools/lxc_start.c:main:309 - To get more details, run the container in foreground mode
lxc-start autopkgtest-sid 20220501145802.681 ERROR    lxc_start - tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start autopkgtest-sid 20220501145802.682 ERROR    start - start.c:__lxc_start:2068 - Failed to spawn container "autopkgtest-sid"
lxc-start autopkgtest-sid 20220501145802.682 WARN     start - start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 20 for process 389747
<<<<<

I found something like this reported at this GitHub issue against lxc:
https://github.com/lxc/lxc/issues/2268
so I followed the advice there and ran the commands:

mount -o remount,rw /sys/fs/cgroup
mkdir /sys/fs/cgroup/devices
mount -t cgroup devices -o devices /sys/fs/cgroup/devices
mount -o remount,ro /sys/fs/cgroup

But that seems to be really bad, as now systemd-logind.service seems
to have broken and cannot be restarted, so I don't recommend doing
that!

I've restarted my system and started again.  The above solution is
very bad at least partly because /sys/fs/cgroup is type cgroup2.  But
I still can't start the LXC container, which makes running autopkgtest
impossible.

I did read the Debian Wiki pages on LXC and the
/usr/share/doc/lxc/README.Debian.gz file, but to no avail.  (I do have
apparmor installed.)

Any help much appreciated!

Best wishes,

   Julian


[1] https://lists.debian.org/debian-devel/2022/04/msg00372.html

clone 1010437 -1
reassign -1 lxc 1:4.0.11-1
retitle -1 lxc: as root, lxc-start fails to start with cgroups/cgfsng error setting up limits for devices
retitle 1010437 autopkgtest-build-lxc: eatmydata gives lots of LD_PRELOAD warnings
thanks

I have now run the autopkgtest-build-lxc script "by hand" to see where
the issues are arising in the issue below, and I think there are two
separate things going on, hence splitting this bug report into two.
A few comments interspersed below.

This looks to be similar to
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963508 so I wonder
whether the apparmor settings for lxc mean that LD_PRELOAD cannot be
used with lxc, and LD_PRELOAD is needed by eatmydata.  This is a minor
issue with autopkgtest-build-lxc; maybe it should just not try using
eatmydata, or maybe there is some way to change the lxc apparmor
settings (if indeed that is the thing preventing the use of
LD_PRELOAD) to allow eatmydata?  I don't know anything about apparmor,
so I am just speculating here.

This also happens when I create the lxc container manually (as root),
using the recipe in autopkgtest-build-lxc:
  lxc-create -B best --name=autopkgtest-unstable -t debian -- -r unstable
the resulting container fails to start in the same way as just
described (when lxc-start is run as root).

This is the bug report against lxc.

Best wishes,

   Julian

An update...

I've now done more searching, and the conclusion I've come to is that
this is that this is the same issue discussed in
https://wiki.debian.org/LXC/CGroupV2#LXC_containers_started_by_root
(and in various other bug reports); by adding the two lines

lxc.cgroup.devices.allow =
lxc.cgroup.devices.deny =

to the file /var/lib/lxc/autopkgtest-unstable/config, I was able to
start the container.  But I'm running lxc version 1:4.0.11-1 and that
wiki page says this change is unnecessary from version 4.0.2-1~1
onwards, which does not seem to be the case.

lxc: I don't know whether the wiki is wrong or some change made in
4.0.2-1 has been reverted more recently.  Either way, it would be
great to resolve this discrepancy.

autopkgtest-build-lxc: perhaps it would be good to add these lines at
the end of the config file when the container is built, especially if
the lxc folks can't fix this.

Best wishes,

   Julian

Hi Julian,

Sorry for the silence, you're doing great work.

I wonder if you refer to
https://bugs.debian.org/902394
https://bugs.debian.org/904732

Our infrastructure (where we don't experiences issues and are using the
autopkgtest/debci/autodep8 packages from unstable on an otherwise stable
system) runs lxc version 1:4.0.6-2. So that maybe limiting the changes
further.

Paul

Hi Paul and lxc maintainers,

lxc maintainers: sorry, I intended to copy in #1010469 in my previous
message but didn't do so; a possible cause of this bug is discussed
below

Yes, I was determined to get it to work!

I was thinking more of
https://bugs.debian.org/944389

And I'm on 1:4.0.11-1.  So perhaps there was a regression in this
regard?

Best wishes,

   Julian

No. I use this version and lxc just works for me. In fact everyone else
who is on testing/unstable is also using it. lxc also just works on a
clean VM. There is something wrong with your system that is causing
this, but it's in no way a general problem.

Hello,

Bug #1010437 in autopkgtest reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/ci-team/autopkgtest/-/commit/f84726464e07a9c9c34afbe43499db57f3e76437
------------------------------------------------------------------------
a-build-lxc: remove broken eatmydata LD_PRELOAD

Doing `eatmydata lxc-create ...` exports LD_PRELOAD, but that is
a preload that refers to a library on the host system. The chroot does
not have libeatmydata.so yet (and in principle it could be named
differently, or be installed somewhere else), so the preloading does not
work and produces a lot of warnings.

Note that this (broken) optimization was only about chroot creation,
with no effect on test execution.

Closes: #1010437
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1010437

Hello,

Bug #1010437 in autopkgtest reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:

https://salsa.debian.org/ci-team/autopkgtest/-/commit/f84726464e07a9c9c34afbe43499db57f3e76437
------------------------------------------------------------------------
a-build-lxc: remove broken eatmydata LD_PRELOAD

Doing `eatmydata lxc-create ...` exports LD_PRELOAD, but that is
a preload that refers to a library on the host system. The chroot does
not have libeatmydata.so yet (and in principle it could be named
differently, or be installed somewhere else), so the preloading does not
work and produces a lot of warnings.

Note that this (broken) optimization was only about chroot creation,
with no effect on test execution.

Closes: #1010437
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1010437