Dear Maintainer,

I spin up containers using LXD. I'm primarily doing this for learning
purposes (for example installing and configuring web services).

I launched my first container (test) to get my feet wet by:

```
lxc launch images:debian/12 test
```

When I tried to set hostname in the container via (`hostnamectl set-hostname
test.test`), I got `Could not set pretty hostname: Connection timed out`
error. journalctl on the container showed errors related to network
namespacing:

```
Jun 17 03:03:12 test dbus-daemon[81]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.7' (uid=0 pid=104 comm="hostnamectl set-hostname test.test")
Jun 17 03:03:12 test (ostnamed)[105]: systemd-hostnamed.service: Failed to set up network namespacing: Permission denied
Jun 17 03:03:12 test systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Jun 17 03:03:12 test (ostnamed)[105]: systemd-hostnamed.service: Failed at step NETWORK spawning /lib/systemd/systemd-hostnamed: Permission denied
Jun 17 03:03:12 test systemd[1]: systemd-hostnamed.service: Main process exited, code=exited, status=225/NETWORK
Jun 17 03:03:12 test systemd[1]: systemd-hostnamed.service: Failed with result 'exit-code'.
Jun 17 03:03:12 test systemd[1]: Failed to start systemd-hostnamed.service - Hostname Service.
Jun 17 03:03:37 test dbus-daemon[81]: [system] Failed to activate service 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms)
```

dmesg on the host revealed that these errors above are due to AppArmor policy
violations:

```
[10673.299973] audit: type=1400 audit(1686970915.519:84): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_</var/lib/lxd>" pid=12984 comm="(crub_all)" family="unix" sock_type="dgram" protocol=0 requested_mask="send"
[10673.299988] audit: type=1400 audit(1686970915.519:85): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_</var/lib/lxd>" pid=12984 comm="(crub_all)" family="unix" sock_type="dgram" protocol=0 requested_mask="send"
[10675.793944] audit: type=1400 audit(1686970918.015:86): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_</var/lib/lxd>" pid=12991 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 requested_mask="send"
[10675.793966] audit: type=1400 audit(1686970918.015:87): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_</var/lib/lxd>" pid=12991 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 requested_mask="send"
[10750.671804] audit: type=1400 audit(1686970992.896:88): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_</var/lib/lxd>" pid=13038 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 requested_mask="send"
[10750.671817] audit: type=1400 audit(1686970992.896:89): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_</var/lib/lxd>" pid=13038 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 requested_mask="send"
```

From the upstream discussion [1], the workaround is to put AppArmor profile
for the container unconfined. To do this, I have to run:

```
lxc config set test raw.lxc "lxc.apparmor.profile=unconfined"
```

See lxc.container.conf(5) manpage [2] for explanation of lxc config key.

The upstream discussion stated that this bug should have been already fixed
in recent systemd, AppArmor, and Linux kernel versions (at least as shipped in
Ubuntu Bionic release). However, I can (still) reproduce it on Debian testing.

Thanks.

[1]: https://discuss.linuxcontainers.org/t/bionic-containers-on-xenial-host-systemd-hostnamed-unable-to-start/1732
[2]: https://manpages.ubuntu.com/manpages/lunar/en/man5/lxc.container.conf.5.html

I'm unsure if current breakage is due to apparmor itself (the bug reported
there was apparently fixed a while ago), lxd's apparmor profile, or
somewhere else (as satisfying as blaming systemd would be...). See linked
bugs listed below.

Is it possible that the AppArmor socket mediation patches have not made it
into upstream and/or Debian kernels? :thinking_face: If so, then this bug
needs to be reassigned to the kernel.

I'm writing here because this also breaks the plocate-updatedb service
inside LXD containers.

I don't think it's a bug with network namespacing inside containers, as

   unshare -u ifconfig

works fine, for example.

In the container:

    root@pat:~# systemctl status plocate-updatedb.service
    × plocate-updatedb.service - Update the plocate database
         Loaded: loaded (/lib/systemd/system/plocate-updatedb.service; static)
        Drop-In: /run/systemd/system/service.d
                 └─zzz-lxc-service.conf
         Active: failed (Result: exit-code) since Fri 2023-06-23 09:53:56 AWST; 5h 31min ago
    TriggeredBy: ● plocate-updatedb.timer
        Process: 33437 ExecStart=/usr/sbin/updatedb.plocate (code=exited, status=225/NETWORK)
       Main PID: 33437 (code=exited, status=225/NETWORK)
            CPU: 584us

    Jun 23 09:53:56 pat systemd[1]: Starting plocate-updatedb.service - Update the plocate database...
    Jun 23 09:53:56 pat systemd[1]: plocate-updatedb.service: Main process exited, code=exited, status=225/NETWORK
    Jun 23 09:53:56 pat systemd[1]: plocate-updatedb.service: Failed with result 'exit-code'.
    Jun 23 09:53:56 pat systemd[1]: Failed to start plocate-updatedb.service - Update the plocate database.

On the host after attempting to start the service inside the guest:

    2023-06-23T09:53:56.040427+08:00 grook kernel: [772843.931461] audit: type=1400 audit(1687485236.036:118): apparmor="DENIED" operation="file_lock" profile="lxd-pat_</var/lib/lxd>" pid=3334600 comm="(.plocate)" family="unix" sock_type=ram" protocol=0 requested_mask="send"
    2023-06-23T09:53:56.040437+08:00 grook kernel: [772843.931469] audit: type=1400 audit(1687485236.036:119): apparmor="DENIED" operation="file_lock" profile="lxd-pat_</var/lib/lxd>" pid=3334600 comm="(.plocate)" family="unix" sock_type=ram" protocol=0 requested_mask="send"

Host information (both host and guest are running bookworm BTW):

    Kernel: Linux 6.1.0-9-amd64 (SMP w/8 CPU threads; PREEMPT)
    Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
    Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), LANGUAGE=en_AU:en
    Shell: /bin/sh linked to /usr/bin/dash
    Init: systemd (via /run/systemd/system)
    LSM: AppArmor: enabled

    Versions of packages lxd depends on:
    ii  adduser              3.134
    ii  attr                 1:2.5.1-4
    ii  ca-certificates      20230311
    ii  init-system-helpers  1.65.2
    ii  libacl1              2.3.1-3
    ii  libc6                2.36-9
    ii  libcap2              1:2.66-4
    ii  libdqlite0           1.11.1-1
    ii  libgcc-s1            12.2.0-14
    ii  liblxc-common        1:5.0.2-1
    ii  liblxc1              1:5.0.2-1
    ii  libsqlite3-0         3.40.1-2
    ii  libudev1             252.6-1
    ii  lxcfs                5.0.3-1
    ii  lxd-client           5.0.2-5
    ii  rsync                3.2.7-1
    ii  squashfs-tools       1:4.5.1-1
    ii  uidmap               1:4.13+dfsg1-1+b1
    ii  xz-utils             5.4.1-0.2

    Versions of packages lxd recommends:
    ii  apparmor                     3.0.8-3
    ii  dnsmasq-base [dnsmasq-base]  2.89-1
    ii  lxd-agent                    5.0.2-5

    Versions of packages lxd suggests:
    pn  btrfs-progs     <none>
    pn  ceph-common     <none>
    ii  gdisk           1.0.9-2.1
    ii  lvm2            2.03.16-2
    ii  lxd-tools       5.0.2-5
    ii  zfsutils-linux  2.1.11-1

The container itself does not have apparmour installed.

systemd-hostnamed.service is probably also affected, but in my case I paved
over the issue by setting PrivateNetwork=no in an override.

Related:
 - https://bugs.launchpad.net/bugs/1575779 and https://bugs.launchpad.net/bugs/1780227
 - https://bugs.launchpad.net/bugs/1635382
 - https://github.com/lxc/lxc/issues/820 and https://github.com/lxc/lxd/issues/1603

Hi Bagas, Michael,

  At first glance, this looks like an apparmor problem. Unfortunately
all the issues that have been mentioned previously are about five years
old. Ubuntu has carried custom patches for the kernel and apparmor in
the past, but at least the relevant kernel changes seem to have been
merged quite a while back[1].

  I will see what I can figure out on my end and update this when I
know more.

  For the moment setting PrivateNetwork=no for affected services is
probably a better approach than totally removing the container's
apparmor profile.

Mathias

[1] -- https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=80a17a5f501ea048d86f81d629c94062b76610d4

Hi Mathias, sorry for late reply.

Indeed, overriding affected services with PrivateNetwork=no works.
Thanks for the better workaround!

Dear lxd and systemd maintainers,

Michael Biebl (2023-09-11):

FTR I did not dare merging these myself: perhaps you want to keep
separate bug reports to track workarounds on top of #1050256 that's
tracking the root cause, or something.

Cheers,

Dear submitter,

as the package lxd has just been removed from the Debian archive
unstable we hereby close the associated bug reports.  We are sorry
that we couldn't deal with your issue properly.

For details on the removal, please see https://bugs.debian.org/1110735

The version of this package that was in Debian prior to this removal
can still be found using https://snapshot.debian.org/.

Please note that the changes have been done on the master archive and
will not propagate to any mirrors until the next dinstall run at the
earliest.

This message was generated automatically; if you believe that there is
a problem with it please contact the archive administrators by mailing
ftpmaster@ftp-master.debian.org.

Debian distribution maintenance software
pp.
Thorsten Alteholz (the ftpmaster behind the curtain)