New network stack without systemd?

[davdr]

It looks like aardvark-dns requires a systemd user session to exist. Without it, container name lookups fail. This makes that the new network stack can't be fully used under WSL2, due to WSL2 being incompatible with systemd out of the box (without resorting to the process namespace workaround as used by Podman's own WSL2 backend).

I'm assuming this is intentional so I'm not filing this as an Issue. My question rather is: is there any chance that this dependency can be removed in future versions?

Some more info on my tests below.

For my use case I'm running rootless Podman in a Debian-based WSL2 distro. I tried the new network stack with Debian Sid, and Podman 4.2 (Podman and friends from current Debian Experimental) and 4.3RC1 (Podman and friends from curren Kubic repos).

The following fails without a systemd user session (dig can't resolve the container name):

$ podman network create test
$ podman run -dt --name aone --network test quay.io/libpod/alpine_nginx:latest
$ podman exec aone dig +short aone

When redoing this after running loginctl enable-linger <username> for the rootless user, the name can be resolved as expected. But this only works on a VM, not under WSL2, because systemd can't run as PID 1 under WSL2.

[rhatdan]

@Luap99 @baude @flouthoc @mheon I think that the network stack should be able to run without systemd? Some Linux Platforms and especially WSL2 require it.

[flouthoc]

@davdr I think following PR ( containers/netavark#436 ) should close this, could you try the patch in your environment ?

[davdr]

@davdr I think following PR ( containers/netavark#436 ) should close this, could you try the patch in your environment ?

Thank you for the patch! I've built a patched 1.2.0-dev version and I confirm it still works an a VM with systemd (still on the condition lingering is enabled for the user).

Under WSL2 however no success for now:

• Debian Sid/Experimental have netavark 1.0.3-3 in the repos. When I replace that executable, I can't start containers in the user-defined network: "podman run" hangs with "failed to connect to syslog: Initialization". I can still start containers in the default network where I believe aardvark-dns isn't started (as per default config, the default network has no dns enabled).
• I figured maybe replacing a Debian-built 1.0.3 by a patched upstream-built 1.2.0 is too much of a difference. So I wanted to try the same with packages from Kubic that had netavark 1.2.0 last week (as I have in my VM), but it turns out the Kubic repos no longer have netavark (nor aardvark-dns) in them...

I hope at least the first point is of some help.

[Luap99]

you have to match aardvark-dns version with the netavark version

[davdr]

you have to match aardvark-dns version with the netavark version

Okay, I'll give that a try too!

[davdr]

Indeed matching the versions solved the error of the container not starting. For the rest of my conclusions, see under #16082 (reply in thread)

[Luap99]

Does the /run/systemd/system/ directory exists in your WSL image? According to sd_booted(3) this means that the system was booted with systemd.

[flouthoc]

@flouthoc WSL2: success! Having netavark and aardvark-dns at 1.2.0 (with your patch for netavark) works for me under WSL2 (other Podman packages from Debian Experimental).

Later tonight I'll validate one more time without the patch too, given @Luap99's point about the systemd path.

Thanks for verifying and trying the patch.

[davdr]

Interesting results: it turns out that the aardvark-dns 1.0.3(-1) from Debian Sid was causing the problem of container DNS not working.

• When I replace the aardvark-dns executable with a build from the upstream 1.0.3 version, container DNS works (while keeping netavark at Debian's 1.0.3(-1) version). Tested and doublechecked both under WSL2, and in a VM with systemd and a user session.
• Also when I replace aardvark-dns and netavark with builds form their current "main" branches (basically 1.2.0-dev) it still works. Still so if I include aardvark-dns: verify if init is actually systemd before using systemd-run netavark#436 in the netavark build. This part I've only tested under WSL2.
• Debian Experimental currently doesn't currently have a more recent version, but Kubic has (netavark and aardvark-dns at 1.2.0). With Podman and friends from Kubic, it also works. Tested and doublechecked both under WSL2, and in a VM with systemd and a user session

So in conclusion, I was looking at two problems:

1. in a VM booted with systemd, you need loginctl enable-linger <username> for the rootless user, for container DNS to work using the new network stack
2. however, both in a VM and in WSL2, my specific distribution's aardvark-dns build caused container DNS not to work

Many thanks for your support and insights!

[davdr]

In case anyone's interested, here is a minimal reproducer for the issue with Debian Sid's aardvark-dns 1.0.3-1.

(@siretart sorry to take the liberty to tag you, but this might interest you?)

Starting from a fresh Debian 11 Bullseye VM:

As root:

# update to Sid + Experimental
cat > /etc/apt/sources.list <<EOF
deb http://deb.debian.org/debian sid main
deb http://deb.debian.org/debian experimental main
EOF
apt update
apt upgrade -y
reboot
apt install -y fuse-overlayfs slirp4netns catatonit uidmap iptables
# install Podman 4.2 and friends from Experimental, note that netavark and aardvark-dns actually are the versions from sid at 1.0.3-1
apt -t experimental install -y podman netavark aardvark-dns containers-storage conmon crun
update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
loginctl enable-linger <rootless-user>

As rootless-user:

podman network create test
podman run -dt --name aone --network test quay.io/libpod/alpine_nginx:latest
podman inspect --format "{{.NetworkSettings.Networks.test.IPAddress}}" aone
podman exec aone dig +short aone
#  --> should return the IP address returned by podman inspect, instead returns nothing
podman kill aone
podman rm aone
podman network rm test

Now fix by replacing with aardvark-dns built from upstream version 1.0.3, as root:

mv /usr/lib/podman/aardvark-dns /usr/lib/podman/aardvark-dns-1.0.3-1
mv /path/to/bin/aardvark-dns /usr/lib/podman/aardvark-dns
chown root:root /usr/lib/podman/aardvark-dns
chmod 755 /usr/lib/podman/aardvark-dns
reboot  # overkill to make sure no aardvark-dns processes are running anymore

Then as rootless user, repeat the test starting with "network create". The "dig" command can now correctly look up the container name.

[davdr]

Interesting results: it turns out that the aardvark-dns 1.0.3(-1) from Debian Sid was causing the problem of container DNS not working.

For future reference: it's this bug: #1024980
Cannot resolve container names with DNS

[davdr]

Solved in Debian Sid since Jan 9, 2023 with aardvark-dns 1.4.0-2!

[fsdrw08]

I dont found this problem in Alpine linux with rootless podman config, alpine linux did not use systemd

InfraSvc-Alpine:~$ podman network create test
test
InfraSvc-Alpine:~$ podman run -dt --name aone --network test quay.io/libpod/alpine_nginx:latest
Trying to pull quay.io/libpod/alpine_nginx:latest...
Getting image source signatures
Copying blob d2c7362ca710 done
Copying blob df9b9388f04a done
Copying blob 71895e83ea49 done
Copying config ecea49d99d done
Writing manifest to image destination
Storing signatures
3b279b8c6b007f32ab02ce21b759ee8b40a3b60321c267e87ec95c7471bd05f6
InfraSvc-Alpine:~$ podman exec aone dig +short aone
10.89.1.2

[davdr]

Debian Sid also no longer has the problem since Jan 9, 2023 with aardvark-dns 1.4.0-2.