(I realise that posting this on debian-devel [1] was probably not the most appropriate place, as it's actually a bug report.) I am not sure whether this is a bug in autopkgtest-build-lxc, a bug in lxc itself or a user error. Please feel free to redirect as appropriate! This is what I did: Step 1: I installed the lxc and autopkgtest packages That went smoothly. (lxc version 1:4.0.11-1, autopkgtest version 5.21; autopkgtest was already installed, and I installed lxc from Step 2: I ran the command "autopkgtest-build-lxc debian sid" as root. I got various warning messages to begin with: lxc-create: autopkgtest-sid: storage/btrfs.c: btrfs_create: 938 Inappropriate ioctl for device - Failed to create btrfs subvolume "/var/lib/lxc/autopkgtest-sid/rootfs" lxc-create: autopkgtest-sid: storage/zfs.c: zfs_create: 735 Failed to create zfs dataset "zfs:lxc/autopkgtest-sid": lxc-create: autopkgtest-sid: utils.c: run_command_internal: 1588 lxc-create: autopkgtest-sid: storage/lvm.c: do_lvm_create: 165 Failed to create logical volume "autopkgtest-sid": Volume group "lxc" not found Cannot process volume group lxc lxc-create: autopkgtest-sid: storage/lvm.c: lvm_create: 623 Error creating new logical volume "lvm:/dev/lxc/autopkgtest-sid" of size "1073741824 bytes" <<<<< after which things ran smoothly for a bit: debootstrap is /usr/sbin/debootstrap Checking cache download in /var/cache/lxc/debian/rootfs-sid-amd64 ... Downloading debian minimal ... I: Target architecture can be executed I: Retrieving InRelease [... downloading and installing base system ...] I: Base system installed successfully. Download complete. <<<<< but then there were lots of warning messages about libeatmydata.so interspersed with information messages; I assume that these are mostly harmless: But then I received several fatal error messages: lxc-start: autopkgtest-sid: lxccontainer.c: wait_on_daemonized_start: 867 Received container state "ABORTING" instead of "RUNNING" lxc-start: autopkgtest-sid: tools/lxc_start.c: main: 306 The container failed to start lxc-start: autopkgtest-sid: tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode lxc-start: autopkgtest-sid: tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options <<<<< Since autopkgtest-build-lxc doesn't allow a --logfile option, I attempted to start the container manually, using the command lxc-start -n autopkgtest-sid --logfile /tmp/lxc.log --logpriority INFO and got the following warnings and errors in the log file (I've excluded the INFO entries): lxc-start autopkgtest-sid 20220501145802.680 NOTICE conf - conf.c:lxc_setup:4450 - The container "autopkgtest-sid" is set up lxc-start autopkgtest-sid 20220501145802.681 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable devices controller lxc-start autopkgtest-sid 20220501145802.681 ERROR cgfsng - cgroups/cgfsng.c:cg_legacy_set_data:2675 - No such file or directory - Failed to setup limits for the "devices" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy lxc-start autopkgtest-sid 20220501145802.681 ERROR cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2742 - No such file or directory - Failed to set "devices.deny" to "a" lxc-start autopkgtest-sid 20220501145802.681 ERROR start - start.c:lxc_spawn:1890 - Failed to setup legacy device cgroup controller limits lxc-start autopkgtest-sid 20220501145802.681 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:867 - Received container state "ABORTING" instead of "RUNNING" lxc-start autopkgtest-sid 20220501145802.681 ERROR lxc_start - tools/lxc_start.c:main:306 - The container failed to start lxc-start autopkgtest-sid 20220501145802.681 ERROR lxc_start - tools/lxc_start.c:main:309 - To get more details, run the container in foreground mode lxc-start autopkgtest-sid 20220501145802.681 ERROR lxc_start - tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options lxc-start autopkgtest-sid 20220501145802.682 ERROR start - start.c:__lxc_start:2068 - Failed to spawn container "autopkgtest-sid" lxc-start autopkgtest-sid 20220501145802.682 WARN start - start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 20 for process 389747 <<<<< I found something like this reported at this GitHub issue against lxc: https://github.com/lxc/lxc/issues/2268 so I followed the advice there and ran the commands: mount -o remount,rw /sys/fs/cgroup mkdir /sys/fs/cgroup/devices mount -t cgroup devices -o devices /sys/fs/cgroup/devices mount -o remount,ro /sys/fs/cgroup But that seems to be really bad, as now systemd-logind.service seems to have broken and cannot be restarted, so I don't recommend doing that! I've restarted my system and started again. The above solution is very bad at least partly because /sys/fs/cgroup is type cgroup2. But I still can't start the LXC container, which makes running autopkgtest impossible. I did read the Debian Wiki pages on LXC and the /usr/share/doc/lxc/README.Debian.gz file, but to no avail. (I do have apparmor installed.) Any help much appreciated! Best wishes, Julian [1] https://lists.debian.org/debian-devel/2022/04/msg00372.html
clone 1010437 -1 reassign -1 lxc 1:4.0.11-1 retitle -1 lxc: as root, lxc-start fails to start with cgroups/cgfsng error setting up limits for devices retitle 1010437 autopkgtest-build-lxc: eatmydata gives lots of LD_PRELOAD warnings thanks I have now run the autopkgtest-build-lxc script "by hand" to see where the issues are arising in the issue below, and I think there are two separate things going on, hence splitting this bug report into two. A few comments interspersed below. This looks to be similar to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963508 so I wonder whether the apparmor settings for lxc mean that LD_PRELOAD cannot be used with lxc, and LD_PRELOAD is needed by eatmydata. This is a minor issue with autopkgtest-build-lxc; maybe it should just not try using eatmydata, or maybe there is some way to change the lxc apparmor settings (if indeed that is the thing preventing the use of LD_PRELOAD) to allow eatmydata? I don't know anything about apparmor, so I am just speculating here. This also happens when I create the lxc container manually (as root), using the recipe in autopkgtest-build-lxc: lxc-create -B best --name=autopkgtest-unstable -t debian -- -r unstable the resulting container fails to start in the same way as just described (when lxc-start is run as root). This is the bug report against lxc. Best wishes, Julian
Hi Paul and lxc maintainers, lxc maintainers: sorry, I intended to copy in #1010469 in my previous message but didn't do so; a possible cause of this bug is discussed below Yes, I was determined to get it to work! I was thinking more of https://bugs.debian.org/944389 And I'm on 1:4.0.11-1. So perhaps there was a regression in this regard? Best wishes, Julian
No. I use this version and lxc just works for me. In fact everyone else who is on testing/unstable is also using it. lxc also just works on a clean VM. There is something wrong with your system that is causing this, but it's in no way a general problem.
[excluding 1010437@bugs.d.o from reply list, as that's to do with eatmydata] It seems that you are right: I tried booting into Debian Live, upgrading to testing and running lxc; it ran without a problem. So now I'm in a quandry: despite installing lxc from scratch, and just redoing so (purging the packages and removing all of the cached files, /etc/lxc, /var/lib/lxc* and so on before reinstalling), I am still experiencing the same problem. I am running what I believe to be a standard system - I first installed it in September 2020 or thereabouts and have kept it up-to-date with testing ever since. I have no idea what might be causing this strange behaviour, and therefore I have got no clue how to fix it. I also don't know whether what is wrong with my setup might affect other people as well. If you have any suggestions of things I could look at on my system (configuration files, other packages, ...) I'm all ears! Many thanks, Julian
Are all packages recommended by lxc installed?
Yes, they are. It's a standard Debian kernel (currently linux-image-5.17.0-1-amd64 5.17.3-1). I'm not aware of doing any customisations that might have caused problems :( /etc/lxc/default.conf is unmodified: lxc.net.0.type = veth lxc.net.0.link = lxcbr0 lxc.net.0.flags = up lxc.apparmor.profile = generated lxc.apparmor.allow_nesting = 1 and when I created a trial container, I get /var/lib/lxc/debian-unstable-trial/config: # Template used to create this container: /usr/share/lxc/templates/lxc-debian # Parameters passed to the template: -r unstable # For additional config options, please look at lxc.container.conf(5) # Uncomment the following line to support nesting containers: #lxc.include = /usr/share/lxc/config/nesting.conf # (Be aware this has security implications) lxc.net.0.type = veth lxc.net.0.hwaddr = 00:16:3e:78:11:12 lxc.net.0.link = lxcbr0 lxc.net.0.flags = up lxc.apparmor.profile = generated lxc.apparmor.allow_nesting = 1 lxc.rootfs.path = dir:/var/lib/lxc/debian-unstable-trial/rootfs # Common configuration lxc.include = /usr/share/lxc/config/debian.common.conf # Container specific configuration lxc.tty.max = 4 lxc.uts.name = debian-unstable-trial lxc.arch = amd64 lxc.pty.max = 1024 I've no idea if that is of any help. Thanks! Julian
I could not find anything wrong in those. I'm sorry but I don't know what's wrong with your system. can you debug to check what is the exact point where it fails to start a container?
Thanks Antonio! Here's a log file with logpriority at DEBUG; I've no idea if this will help. I'm so stumped. I wondered if it was perhaps some extra kernel modules (using dkms) causing the problem, so I've purged those and rebooted, but it didn't help. Something seems to be confused with the cgroups. Here's the result of mount | grep cgroup: cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime) none on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls) And ls /sys/fs/cgroup gives: cgroup.controllers dev-hugepages.mount misc.capacity cgroup.max.depth dev-mqueue.mount net_cls cgroup.max.descendants init.scope proc-fs-nfsd.mount cgroup.procs io.cost.model proc-sys-fs-binfmt_misc.mount cgroup.stat io.cost.qos sys-fs-fuse-connections.mount cgroup.subtree_control io.pressure sys-kernel-config.mount cgroup.threads io.stat sys-kernel-debug.mount cpu.pressure lxc.pivot sys-kernel-tracing.mount cpuset.cpus.effective memory.numa_stat system.slice cpuset.mems.effective memory.pressure user.slice cpu.stat memory.stat Ho hum :-/ Julian lxc-start debian-unstable-trial 20220511204109.279 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:987 - Set process title to [lxc monitor] /var/lib/lxc debian-unstable-trial lxc-start debian-unstable-trial 20220511204109.280 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:848 - First child 502040 exited lxc-start debian-unstable-trial 20220511204109.280 INFO lsm - lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor lxc-start debian-unstable-trial 20220511204109.281 DEBUG seccomp - seccomp.c:parse_config_v2:656 - Host native arch is [3221225534] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "reject_force_umount # comment this to allow umount -f; not recommended" lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "[all]" lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "kexec_load errno 1" lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "open_by_handle_at errno 1" lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "init_module errno 1" lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "finit_module errno 1" lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "delete_module errno 1" lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886] lxc-start debian-unstable-trial 20220511204109.281 INFO seccomp - seccomp.c:parse_config_v2:1017 - Merging compat seccomp contexts into main context lxc-start debian-unstable-trial 20220511204109.511 INFO start - start.c:lxc_init:883 - Container "debian-unstable-trial" is initialized lxc-start debian-unstable-trial 20220511204109.531 INFO cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1028 - The monitor process uses "lxc.monitor.debian-unstable-trial" as cgroup lxc-start debian-unstable-trial 20220511204109.531 DEBUG storage - storage/storage.c:get_storage_by_name:209 - Detected rootfs type "dir" lxc-start debian-unstable-trial 20220511204109.532 INFO cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1136 - The container process uses "lxc.payload.debian-unstable-trial" as inner and "lxc.payload.debian-unstable-trial" as limit cgroup lxc-start debian-unstable-trial 20220511204109.534 INFO start - start.c:lxc_spawn:1759 - Cloned CLONE_NEWNS lxc-start debian-unstable-trial 20220511204109.534 INFO start - start.c:lxc_spawn:1759 - Cloned CLONE_NEWPID lxc-start debian-unstable-trial 20220511204109.534 INFO start - start.c:lxc_spawn:1759 - Cloned CLONE_NEWUTS lxc-start debian-unstable-trial 20220511204109.534 INFO start - start.c:lxc_spawn:1759 - Cloned CLONE_NEWIPC lxc-start debian-unstable-trial 20220511204109.534 INFO start - start.c:lxc_spawn:1759 - Cloned CLONE_NEWNET lxc-start debian-unstable-trial 20220511204109.534 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 21 and stashed path as mnt:/proc/502041/fd/21 lxc-start debian-unstable-trial 20220511204109.534 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 22 and stashed path as pid:/proc/502041/fd/22 lxc-start debian-unstable-trial 20220511204109.534 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 23 and stashed path as uts:/proc/502041/fd/23 lxc-start debian-unstable-trial 20220511204109.534 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 24 and stashed path as ipc:/proc/502041/fd/24 lxc-start debian-unstable-trial 20220511204109.534 DEBUG start - start.c:lxc_try_preserve_namespace:139 - Preserved net namespace via fd 25 and stashed path as net:/proc/502041/fd/25 lxc-start debian-unstable-trial 20220511204109.534 INFO cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2749 - Limits for the legacy cgroup hierarchies have been setup lxc-start debian-unstable-trial 20220511204109.534 WARN cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2809 - Invalid argument - Ignoring cgroup2 limits on legacy cgroup system lxc-start debian-unstable-trial 20220511204109.534 INFO start - start.c:do_start:1206 - Unshared CLONE_NEWCGROUP lxc-start debian-unstable-trial 20220511204109.542 INFO network - network.c:netdev_configure_server_veth:655 - Retrieved mtu 1500 from lxcbr0 lxc-start debian-unstable-trial 20220511204109.557 INFO network - network.c:netdev_configure_server_veth:720 - Attached "vethS9TtPh" to bridge "lxcbr0" lxc-start debian-unstable-trial 20220511204109.557 DEBUG network - network.c:netdev_configure_server_veth:851 - Instantiated veth tunnel "vethS9TtPh <--> vethVbX9aI" lxc-start debian-unstable-trial 20220511204109.558 DEBUG conf - conf.c:lxc_mount_rootfs:1423 - Mounted rootfs "/var/lib/lxc/debian-unstable-trial/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)" lxc-start debian-unstable-trial 20220511204109.558 INFO conf - conf.c:setup_utsname:866 - Set hostname to "debian-unstable-trial" lxc-start debian-unstable-trial 20220511204109.558 INFO network - network.c:lxc_setup_network_in_child_namespaces:4005 - Finished setting up network devices with caller assigned names lxc-start debian-unstable-trial 20220511204109.583 DEBUG network - network.c:setup_hw_addr:3807 - Mac address "00:16:3e:98:ce:14" on "eth0" has been setup lxc-start debian-unstable-trial 20220511204109.583 DEBUG network - network.c:lxc_network_setup_in_child_namespaces_common:3948 - Network device "eth0" has been setup lxc-start debian-unstable-trial 20220511204109.583 INFO network - network.c:lxc_setup_network_in_child_namespaces:4021 - Finished setting up network devices with kernel assigned names lxc-start debian-unstable-trial 20220511204109.583 INFO conf - conf.c:mount_autodev:1206 - Preparing "/dev" lxc-start debian-unstable-trial 20220511204109.583 INFO conf - conf.c:mount_autodev:1267 - Prepared "/dev" lxc-start debian-unstable-trial 20220511204109.583 DEBUG conf - conf.c:lxc_mount_auto_mounts:733 - Invalid argument - Tried to ensure procfs is unmounted lxc-start debian-unstable-trial 20220511204109.584 DEBUG conf - conf.c:lxc_mount_auto_mounts:756 - Invalid argument - Tried to ensure sysfs is unmounted lxc-start debian-unstable-trial 20220511204109.584 DEBUG conf - conf.c:mount_entry:2403 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options lxc-start debian-unstable-trial 20220511204109.584 DEBUG conf - conf.c:mount_entry:2422 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14 lxc-start debian-unstable-trial 20220511204109.584 DEBUG conf - conf.c:mount_entry:2466 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none" lxc-start debian-unstable-trial 20220511204109.584 DEBUG conf - conf.c:mount_entry:2466 - Mounted "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/proc" with filesystem type "proc" lxc-start debian-unstable-trial 20220511204109.584 DEBUG conf - conf.c:mount_entry:2466 - Mounted "sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/sys" with filesystem type "sysfs" lxc-start debian-unstable-trial 20220511204109.584 INFO conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "debian-unstable-trial", config section "lxc" lxc-start debian-unstable-trial 20220511204109.652 INFO conf - conf.c:lxc_fill_autodev:1304 - Populating "/dev" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_fill_autodev:1313 - Created device node "full" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_fill_autodev:1313 - Created device node "null" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_fill_autodev:1313 - Created device node "random" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_fill_autodev:1313 - Created device node "tty" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_fill_autodev:1313 - Created device node "urandom" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_fill_autodev:1313 - Created device node "zero" lxc-start debian-unstable-trial 20220511204109.652 INFO conf - conf.c:lxc_fill_autodev:1392 - Populated "/dev" lxc-start debian-unstable-trial 20220511204109.652 INFO conf - conf.c:lxc_transient_proc:3757 - Caller's PID is 1; /proc/self points to 1 lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_setup_devpts_child:1738 - Attached detached devpts mount 24 to 22/pts lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_setup_devpts_child:1824 - Created "/dev/ptmx" file as bind mount target lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_setup_devpts_child:1831 - Bind mounted "/dev/pts/ptmx" to "/dev/ptmx" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_allocate_ttys:1092 - Created tty with ptx fd 26 and pty fd 27 and index 1 lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_allocate_ttys:1092 - Created tty with ptx fd 28 and pty fd 29 and index 2 lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_allocate_ttys:1092 - Created tty with ptx fd 30 and pty fd 31 and index 3 lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_allocate_ttys:1092 - Created tty with ptx fd 32 and pty fd 33 and index 4 lxc-start debian-unstable-trial 20220511204109.652 INFO conf - conf.c:lxc_allocate_ttys:1097 - Finished creating 4 tty devices lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_setup_ttys:1056 - Bind mounted "" onto "tty1" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_setup_ttys:1056 - Bind mounted "" onto "tty2" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_setup_ttys:1056 - Bind mounted "" onto "tty3" lxc-start debian-unstable-trial 20220511204109.652 DEBUG conf - conf.c:lxc_setup_ttys:1056 - Bind mounted "" onto "tty4" lxc-start debian-unstable-trial 20220511204109.653 INFO conf - conf.c:lxc_setup_ttys:1063 - Finished setting up 4 /dev/tty<N> device(s) lxc-start debian-unstable-trial 20220511204109.654 INFO conf - conf.c:setup_personality:1904 - Set personality to "0lx0" lxc-start debian-unstable-trial 20220511204109.654 DEBUG conf - conf.c:capabilities_deny:3187 - Dropped mac_admin (33) capability lxc-start debian-unstable-trial 20220511204109.654 DEBUG conf - conf.c:capabilities_deny:3187 - Dropped mac_override (32) capability lxc-start debian-unstable-trial 20220511204109.654 DEBUG conf - conf.c:capabilities_deny:3187 - Dropped sys_time (25) capability lxc-start debian-unstable-trial 20220511204109.654 DEBUG conf - conf.c:capabilities_deny:3187 - Dropped sys_module (16) capability lxc-start debian-unstable-trial 20220511204109.654 DEBUG conf - conf.c:capabilities_deny:3187 - Dropped sys_rawio (17) capability lxc-start debian-unstable-trial 20220511204109.654 DEBUG conf - conf.c:capabilities_deny:3190 - Capabilities have been setup lxc-start debian-unstable-trial 20220511204109.654 NOTICE conf - conf.c:lxc_setup:4450 - The container "debian-unstable-trial" is set up lxc-start debian-unstable-trial 20220511204109.654 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set_at:1186 - Set AppArmor label to "lxc-debian-unstable-trial_</var/lib/lxc>//&:lxc-debian-unstable-trial_<-var-lib-lxc>:" lxc-start debian-unstable-trial 20220511204109.654 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set:1231 - Changed AppArmor profile to lxc-debian-unstable-trial_</var/lib/lxc>//&:lxc-debian-unstable-trial_<-var-lib-lxc>: lxc-start debian-unstable-trial 20220511204109.655 WARN cgfsng - cgroups/cgfsng.c:get_hierarchy:142 - There is no useable devices controller lxc-start debian-unstable-trial 20220511204109.655 ERROR cgfsng - cgroups/cgfsng.c:cg_legacy_set_data:2675 - No such file or directory - Failed to setup limits for the "devices" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy lxc-start debian-unstable-trial 20220511204109.655 ERROR cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2742 - No such file or directory - Failed to set "devices.deny" to "a" lxc-start debian-unstable-trial 20220511204109.655 ERROR start - start.c:lxc_spawn:1890 - Failed to setup legacy device cgroup controller limits lxc-start debian-unstable-trial 20220511204109.655 DEBUG network - network.c:lxc_delete_network:4159 - Deleted network devices lxc-start debian-unstable-trial 20220511204109.655 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:867 - Received container state "ABORTING" instead of "RUNNING" lxc-start debian-unstable-trial 20220511204109.655 ERROR lxc_start - tools/lxc_start.c:main:306 - The container failed to start lxc-start debian-unstable-trial 20220511204109.655 ERROR lxc_start - tools/lxc_start.c:main:309 - To get more details, run the container in foreground mode lxc-start debian-unstable-trial 20220511204109.655 ERROR lxc_start - tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options lxc-start debian-unstable-trial 20220511204109.655 ERROR start - start.c:__lxc_start:2068 - Failed to spawn container "debian-unstable-trial" lxc-start debian-unstable-trial 20220511204109.655 WARN start - start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 20 for process 502047 lxc-start debian-unstable-trial 20220511204109.900 INFO conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "debian-unstable-trial", config section "lxc"
tags 1010469 unreproducible fixed 1010469 1:4.0.11-1 thanks I decided to reinstall my system from scratch, and now this bug has gone away. So as no-one else could reproduce it and I have no idea what has changed on my system as a result of reinstalling, I'm closing it with an "unreproducible" tag. Thanks for your help along the way! Best wishes, Julian
notfixed 1010469 1:4.0.11-1 thanks Oh dear, oh dear, oh dear. It's just happened again. I am so completely stumped by this one. Julian
Julian Gilbey <jdg@debian.org> wrote on 08/06/2022 at 10:50:18+0200: What apparmor profile are you trying to run your container with?
Dear Pierre-Elliott, I'm not sure which profile I'm using; I just installed lxc and am using whatever the default is. Looking at /var/lib/lxc/containername/config, I see the lines: lxc.apparmor.profile = generated lxc.apparmor.allow_nesting = 1 which hopefully means something to you! Best wishes, Julian
Julian Gilbey <jdg@debian.org> wrote on 08/08/2022 at 15:47:08+0100: Hrmpf, this one slipped out of my todolist, I'm sorry for this, this is bad. When you indeed reinstalled your system, which version of Debian did you install? Did you do anything specific before things turned bad again? Cheers,
Hi Pierre-Elliott, I was using debian testing (whatever state it was in at the time). I've just tried reinstalling lxc from scratch with the current debian testing. I haven't been able to get as far as reproducing this error, as I've hit a different snag: # lxc-create -n debian-sid -t download -- -d debian -r sid -a amd64 # lxc-start -n debian-sid --logfile /tmp/lxc.log --logpriority DEBUG lxc-start: debian-sid: ../src/lxc/lxccontainer.c: wait_on_daemonized_start: 878 Received container state "ABORTING" instead of "RUNNING" lxc-start: debian-sid: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start lxc-start: debian-sid: ../src/lxc/tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode lxc-start: debian-sid: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options The resulting log file contains the cryptic error messages: lxc-start debian-sid 20230511122856.360 ERROR network - ../src/lxc/network.c:netdev_configure_server_veth:711 - No such file or directory - Failed to attach "vethQ4rt4x" to bridge "lxcbr0", bridge interface doesn't exist That's super-weird; I have no idea what "vethQ4rt4x" is meant to mean. I think this should probably be a separate bug report, though. Despite some web searching, I have no idea how to fix this problem, but I now can't use lxc at all :( I think it's something about lxc-net not connecting the bridging device to the correct network device (which in my case is enp5s0). Best wishes, Julian
Julian Gilbey <jdg@debian.org> wrote on 11/05/2023 at 16:41:46+0200: the LXC container to get network. Inside the container it'll be eth0, outside it's a veth intervace, named veth$RANDOM stuff. The issue is in the message: you configured the container to bind this interface on a bridge named lxcbr0 that doesn't seem to exist on the host. enp5s0 is a physical interface, bridging a container directly on it might not achieve what you expect. The usual way is to either use the lxc-net service, or to create a manual bridge (with network/interfaces or systemd-networkd config), allow forwarding on it and the physical interface, and bind the containers on it. You will find some doc on LXC network configuration on LXC's website. :)
Thanks so much for the quick response, that's really helpful!
Unfortunately, this doesn't seem to be the issue, though:
# systemctl status lxc-net.service
● lxc-net.service - LXC network bridge setup
Loaded: loaded (/lib/systemd/system/lxc-net.service; enabled; preset: enab>
Active: active (exited) since Thu 2023-05-11 20:35:48 BST; 13h ago
Docs: man:lxc
Process: 81843 ExecStart=/usr/libexec/lxc/lxc-net start (code=exited, statu>
Main PID: 81843 (code=exited, status=0/SUCCESS)
Tasks: 1 (limit: 76868)
Memory: 1.3M
CPU: 70ms
CGroup: /system.slice/lxc-net.service
└─81884 dnsmasq --conf-file=/dev/null -u dnsmasq --strict-order -->
May 11 20:35:48 euler systemd[1]: Starting lxc-net.service - LXC network bridge>
May 11 20:35:48 euler dnsmasq[81884]: started, version 2.89 cachesize 150
May 11 20:35:48 euler dnsmasq[81884]: compile time options: IPv6 GNU-getopt DBu>
May 11 20:35:48 euler dnsmasq-dhcp[81884]: DHCP, IP range 10.0.3.2 -- 10.0.3.25>
May 11 20:35:48 euler dnsmasq-dhcp[81884]: DHCP, sockets bound exclusively to i>
May 11 20:35:48 euler dnsmasq[81884]: reading /etc/resolv.conf
May 11 20:35:48 euler dnsmasq[81884]: using nameserver 10.0.0.243#53
May 11 20:35:48 euler dnsmasq[81884]: read /etc/hosts - 7 names
May 11 20:35:48 euler systemd[1]: Finished lxc-net.service - LXC network bridge>
And with some details snipped:
# ifconfig
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet [...] netmask 255.255.255.0 broadcast 192.168.0.255
inet6 [...] prefixlen 64 scopeid 0x20<link>
ether [...] txqueuelen 1000 (Ethernet)
[...]
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
[...]
lxcbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 10.0.3.1 netmask 255.255.255.0 broadcast 10.0.3.255
ether 00:16:3e:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
tun0: [...]
wlp3s0: [...]
# bridge vlan show
port vlan-id
lxcbr0 1 PVID Egress Untagged
So lxc-net was established, and it still didn't work :( (And yes,
I've just checked that lxc-start still fails.) But maybe the bridge
is meant to be in the lxc container itself?
So I'm still totally stumped.
Any further ideas/suggestions/things to check would be welcomely
received!
Best wishes,
Julian
Julian Gilbey <jdg@debian.org> wrote on 12/05/2023 at 11:39:33+0200: What do you have in /etc/lxc/lxc-usernet ? Also, what is your container config, please?
Hi Pierre-Elliott, Thanks! :-) jdg veth lxcbr0 10 (But I'm trying to start the container as root, having su'd from jdg, in case that's relevant. And I have the subuid/subgid set up correctly.) I assume you mean /var/lib/lxc/debian-sid/config? Here's the content of this file: # Template used to create this container: /usr/share/lxc/templates/lxc-download # Parameters passed to the template: -d debian -r sid -a amd64 # For additional config options, please look at lxc.container.conf(5) # Uncomment the following line to support nesting containers: #lxc.include = /usr/share/lxc/config/nesting.conf # (Be aware this has security implications) # Distribution configuration lxc.include = /usr/share/lxc/config/common.conf lxc.arch = linux64 # Container specific configuration lxc.apparmor.profile = generated lxc.apparmor.allow_nesting = 1 lxc.rootfs.path = dir:/var/lib/lxc/debian-sid/rootfs lxc.uts.name = debian-sid # Network configuration lxc.net.0.type = veth lxc.net.0.link = lxcbr0 lxc.net.0.flags = up As a non-expert, this looks fine, but there may be something awry here. I've had numerous lxc problems with my machine, and have no idea why :( Best wishes, Julian
OK, I rebooted my machine, and just tried again, and the bridge worked fine. So it was some random transient thing. Who knows what?! But now we're back to the original problem cgfsng problem (running with --logpriority TRACE): lxc-start debian-sid 20230515092650.376 WARN cgfsng - ../src/lxc/cgroups/cgfsng.c:get_hierarchy:149 - There is no useable devices controller lxc-start debian-sid 20230515092650.376 ERROR cgfsng - ../src/lxc/cgroups/cgfsng.c:cg_legacy_set_data:3098 - No such file or directory - Failed to setup limits for the "devices" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy lxc-start debian-sid 20230515092650.376 ERROR cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits_legacy:3165 - No such file or directory - Failed to set "devices.deny" to "a" lxc-start debian-sid 20230515092650.376 ERROR start - ../src/lxc/start.c:lxc_spawn:1893 - Failed to setup legacy device cgroup controller limits The full log (with TRACE) is attached. Running findmnt gives: TARGET SOURCE FSTYPE OPTIONS / /dev/nvme0n1p2 ext4 rw,noatime ├─/sys sysfs sysfs rw,nosuid,nodev,noexec,rel │ ├─/sys/kernel/security securityfs securit rw,nosuid,nodev,noexec,rel │ ├─/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,rel │ │ └─/sys/fs/cgroup/net_cls none cgroup rw,relatime,net_cls │ ├─/sys/fs/pstore pstore pstore rw,nosuid,nodev,noexec,rel [...] Ho hum :( Best wishes, Julian
Ah, success! I followed the recipe on https://wiki.debian.org/LXC/CGroupV2 referenced in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=944389 (adding the lines lxc.cgroup.devices.allow = lxc.cgroup.devices.deny = to the end of /var/lib/lxc/debian-sid/config) and it now works. But there's no mention of this in /usr/share/doc/lxc/README.Debian.gz, and I don't need to do this on my other machine, so there's still something weird going on on this machine. Perhaps it's a hardware thing? Oh joys! Best wishes, Julian
Julian Gilbey <jdg@debian.org> wrote on 15/05/2023 at 22:05:37+0200: Ah, I don't remember seeing these logs before, maybe I forgot to ask for a full trace, sorry. Do you see anything in /var/log/audit or /var/log/syslog or /var/log/kern.log about apparmor denies? Cheers,
Hi Pierre-Elliott, Thanks for all your time on this! No, I don't think so; here's the relevant part of /var/log/syslog: 2023-05-15T21:47:21.645645+01:00 euler kernel: [94706.953337] audit: type=1400 audit(1684183641.640:92): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-debian-sid_</var/lib/lxc>" pid=547941 comm="apparmor_parser" 2023-05-15T21:47:21.741618+01:00 euler kernel: [94707.049379] lxcbr0: port 1(vethPUhTGL) entered blocking state 2023-05-15T21:47:21.741632+01:00 euler kernel: [94707.049385] lxcbr0: port 1(vethPUhTGL) entered disabled state 2023-05-15T21:47:21.741633+01:00 euler kernel: [94707.049489] device vethPUhTGL entered promiscuous mode 2023-05-15T21:47:21.741635+01:00 euler kernel: [94707.049738] lxcbr0: port 1(vethPUhTGL) entered blocking state 2023-05-15T21:47:21.741636+01:00 euler kernel: [94707.049741] lxcbr0: port 1(vethPUhTGL) entered forwarding state 2023-05-15T21:47:21.741637+01:00 euler kernel: [94707.049956] lxcbr0: port 1(vethPUhTGL) entered disabled state 2023-05-15T21:47:21.741637+01:00 euler kernel: [94707.050570] eth0: renamed from vethX4vupZ 2023-05-15T21:47:21.742023+01:00 euler NetworkManager[1647]: <info> [1684183641.7416] manager: (vethPUhTGL): new Veth device (/org/freedesktop/NetworkManager/Devices/18) 2023-05-15T21:47:21.764024+01:00 euler NetworkManager[1647]: <info> [1684183641.7639] device (vethPUhTGL): carrier: link connected 2023-05-15T21:47:21.764243+01:00 euler NetworkManager[1647]: <info> [1684183641.7641] device (lxcbr0): carrier: link connected 2023-05-15T21:47:21.765623+01:00 euler kernel: [94707.072272] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2023-05-15T21:47:21.765639+01:00 euler kernel: [94707.072351] IPv6: ADDRCONF(NETDEV_CHANGE): vethPUhTGL: link becomes ready 2023-05-15T21:47:21.765640+01:00 euler kernel: [94707.072481] lxcbr0: port 1(vethPUhTGL) entered blocking state 2023-05-15T21:47:21.765642+01:00 euler kernel: [94707.072487] lxcbr0: port 1(vethPUhTGL) entered forwarding state 2023-05-15T21:47:22.017893+01:00 euler kernel: [94707.323563] audit: type=1400 audit(1684183642.012:93): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-debian-sid_</var/lib/lxc>" pid=547969 comm="apparmor_parser" 2023-05-15T21:47:22.085623+01:00 euler kernel: [94707.392045] lxcbr0: port 1(vethPUhTGL) entered disabled state 2023-05-15T21:47:22.085634+01:00 euler kernel: [94707.392996] device vethPUhTGL left promiscuous mode 2023-05-15T21:47:22.085635+01:00 euler kernel: [94707.393002] lxcbr0: port 1(vethPUhTGL) entered disabled state and /var/log/kern.log, which seems very similar: 2023-05-15T21:47:21.645645+01:00 euler kernel: [94706.953337] audit: type=1400 a udit(1684183641.640:92): apparmor="STATUS" operation="profile_load" profile="/us r/bin/lxc-start" name="lxc-debian-sid_</var/lib/lxc>" pid=547941 comm="apparmor_ parser" 2023-05-15T21:47:21.741618+01:00 euler kernel: [94707.049379] lxcbr0: port 1(vet hPUhTGL) entered blocking state 2023-05-15T21:47:21.741632+01:00 euler kernel: [94707.049385] lxcbr0: port 1(vet hPUhTGL) entered disabled state 2023-05-15T21:47:21.741633+01:00 euler kernel: [94707.049489] device vethPUhTGL entered promiscuous mode 2023-05-15T21:47:21.741635+01:00 euler kernel: [94707.049738] lxcbr0: port 1(vet hPUhTGL) entered blocking state 2023-05-15T21:47:21.741636+01:00 euler kernel: [94707.049741] lxcbr0: port 1(vet hPUhTGL) entered forwarding state 2023-05-15T21:47:21.741637+01:00 euler kernel: [94707.049956] lxcbr0: port 1(vethPUhTGL) entered disabled state 2023-05-15T21:47:21.741637+01:00 euler kernel: [94707.050570] eth0: renamed from vethX4vupZ 2023-05-15T21:47:21.765623+01:00 euler kernel: [94707.072272] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2023-05-15T21:47:21.765639+01:00 euler kernel: [94707.072351] IPv6: ADDRCONF(NETDEV_CHANGE): vethPUhTGL: link becomes ready 2023-05-15T21:47:21.765640+01:00 euler kernel: [94707.072481] lxcbr0: port 1(vethPUhTGL) entered blocking state 2023-05-15T21:47:21.765642+01:00 euler kernel: [94707.072487] lxcbr0: port 1(vethPUhTGL) entered forwarding state 2023-05-15T21:47:22.017893+01:00 euler kernel: [94707.323563] audit: type=1400 audit(1684183642.012:93): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-debian-sid_</var/lib/lxc>" pid=547969 comm="apparmor_parser" 2023-05-15T21:47:22.085623+01:00 euler kernel: [94707.392045] lxcbr0: port 1(vethPUhTGL) entered disabled state 2023-05-15T21:47:22.085634+01:00 euler kernel: [94707.392996] device vethPUhTGL left promiscuous mode 2023-05-15T21:47:22.085635+01:00 euler kernel: [94707.393002] lxcbr0: port 1(vethPUhTGL) entered disabled state (and I don't have a /var/log/audit). Hmmm.... Julian