LXD: no network on Archlinux, working on Ubuntu container
This issue was reported on LXD forums first, but it seems to be an issue with Manjaro. Here are the essentials of the thread:
I’m trying to start a unprivileged Archlinux container on a Manjaro host. All of the settings are default or set according to the arch wiki.
My problem is that no matter what, the Archlinux container does not get an ipv4 address assigned. An Ubuntu container, on the other hand, works out of the box.
`lxc launch images:ubuntu/20.04 hlos-ubuntu` # gets ipv4, internet works out of the box
`lxc launch images:archlinux hlos-arch` # no ipv4 -> no internet
+-------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| hlos-arch | RUNNING | | fd42:9baf:9d56:538d:216:3eff:fe6a:ef95 (eth0) | CONTAINER | 0 |
+-------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| hlos-ubuntu | RUNNING | 10.25.199.228 (eth0) | fd42:9baf:9d56:538d:216:3eff:fe2f:3a57 (eth0) | CONTAINER | 0 |
+-------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
There are A LOT of wrong leads on the internet because there was a bug in systemd for a while, which prevented networking in lxc.
My best guess (after working trough dozens of google results, github issues and countless mailing lists) that there is something broken in the archlinux container. journalctl inside the container has some leads but looking them up lead me back to old/fixed issues.
container - journalctl output
Full output. Some lines of interst:
Jun 16 19:44:04 hlos-arch systemd-udevd[50]: Failed to chown '/dev/net/tun' 0 0: Operation not permitted
Jun 16 19:44:04 hlos-arch systemd-udevd[50]: Failed to apply permissions on static device nodes: Operation not permitted
Jun 16 19:44:04 hlos-arch systemd[1]: Started udev Kernel Device Manager.
Jun 16 19:44:04 hlos-arch systemd-udevd[58]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
...
Jun 16 19:44:04 hlos-arch systemd[59]: systemd-networkd.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Permission denied
Jun 16 19:44:04 hlos-arch systemd[59]: systemd-networkd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-networkd: Permission denied
...
Jun 16 19:44:04 hlos-arch systemd[1]: Starting Network Service...
Jun 16 19:44:04 hlos-arch systemd[61]: systemd-logind.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Permission denied
Jun 16 19:44:04 hlos-arch systemd[61]: systemd-logind.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-logind: Permission denied
host - dmesg output
Looks a lot like an apparmor failure to me. dmesg would probably show some denials
[ 7345.930676] lxdbr0: port 1(veth6d57013c) entered blocking state
[ 7345.930882] lxdbr0: port 1(veth6d57013c) entered disabled state
[ 7345.933167] device veth6d57013c entered promiscuous mode
[ 7345.933182] audit: type=1700 audit(1592405655.520:406): dev=veth6d57013c prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[ 7345.948519] IPv6: ADDRCONF(NETDEV_CHANGE): vethbdb942a5: link becomes ready
[ 7345.948583] lxdbr0: port 1(veth6d57013c) entered blocking state
[ 7345.948584] lxdbr0: port 1(veth6d57013c) entered forwarding state
[ 7346.025935] audit: type=1400 audit(1592405655.614:407): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-hlos-arch_</var/lib/lxd>" pid=84363 comm="apparmor_parser"
[ 7346.087598] eth0: renamed from vethbdb942a5
[ 7346.103718] lxdbr0: port 1(veth6d57013c) entered disabled state
[ 7346.105624] lxdbr0: port 1(veth6d57013c) entered blocking state
[ 7346.105625] lxdbr0: port 1(veth6d57013c) entered forwarding state
[ 7346.484299] audit: type=1400 audit(1592405656.074:408): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84441 comm="(d-logind)" flags="ro, remount, noatime, bind"
[ 7346.485522] audit: type=1400 audit(1592405656.074:409): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84442 comm="(networkd)" flags="ro, remount, noatime, bind"
[ 7346.492335] audit: type=1400 audit(1592405656.080:410): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84445 comm="(d-logind)" flags="ro, remount, noatime, bind"
[ 7346.494926] audit: type=1400 audit(1592405656.084:411): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84446 comm="(networkd)" flags="ro, remount, noatime, bind"
[ 7346.500453] audit: type=1400 audit(1592405656.087:412): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84449 comm="(d-logind)" flags="ro, remount, noatime, bind"
[ 7346.503002] audit: type=1400 audit(1592405656.090:413): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84450 comm="(networkd)" flags="ro, remount, noatime, bind"
[ 7346.508783] audit: type=1400 audit(1592405656.097:414): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84453 comm="(d-logind)" flags="ro, remount, noatime, bind"
[ 7346.511098] audit: type=1400 audit(1592405656.104:415): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84454 comm="(networkd)" flags="ro, remount, noatime, bind"
lxd profiles
Our apparmor policy for the container specifically allows that set of flags and path in the base profile so it’s quite weird…
Can you look for the relevant profile in /var/snap/lxd/common/lxd/security and confirm that you see the ro,remount,noatime,bind mount entries in there?
(not a snap install...) contents of /var/lib/lxd/security/apparmor/profiles/lxd-hlos-arch here
conclusion
Yeah, the profile looks correct with the exact thing you tried to do being allowed according to it… This feels like an apparmor-parser or kernel bug in this case.
The security.nesting=true workaround should be fine in this case but you probably should file a bug against apparmor in your distro
So here we are. I lack deeper knowledge of apparmor to properly debug this on my own. But if you need any more information, I'm happy to help.