New podman install - can't create networks - Error: could not find free subnet from subnet pools

[interpretazioni-mataniani]

Issue Description

Hi! I'm running podman on a brand new Fedora 39 Silverblue install.
Whenever trying to create a network with podman network create, or using podman-compose, I get an error:

Error: could not find free subnet from subnet pools

This is a brand-new install - I've even tried reinstalling, with the same results.
Would appreciate your help troubleshooting.
Thanks!

Steps to reproduce the issue

Steps to reproduce the issue

1. Run podman network create [network-name] or podman-compose up -d with a valid docker-compose.yml file

Describe the results you received

Error: could not find free subnet from subnet pools

Describe the results you expected

Creation of a network shown under podman network ls

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-3.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 87.79
    systemPercent: 2.3
    userPercent: 9.92
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: silverblue
    version: "39"
  eventLogger: journald
  freeLocks: 2047
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.5.6-300.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1407729664
  memTotal: 16699150336
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-2.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-2.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc39.x86_64
    version: |
      pasta 0^20230908.g05627dc-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 6826225664
  swapTotal: 8589930496
  uptime: 29h 44m 0.00s (Approximately 1.21 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/shmick/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/shmick/.local/share/containers/storage
  graphRootAllocated: 248355225600
  graphRootUsed: 27910668288
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 8
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/shmick/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.0
  Built: 1695838680
  BuiltTime: Wed Sep 27 21:18:00 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

I've attempted this rootless first, but the issue remains when running rootfull as well

Additional information

Could only replicate this on my main machine - when installing Fedora 39 Silverblue on a different host, the issue does not occur.

[Luap99]

What is the output of ip addr?
You either already use up 10.89.0.0-10.255.255.0/24 on your host or you created enough networks they all uses up this range (check podman network ls).
Or you configured containers.conf incorrectly and you do not have any subnet pool configured, see default_subnet_pools in https://github.com/containers/common/blob/main/docs/containers.conf.5.md

Either way this is not a bug so I convert it to a discussion.

[interpretazioni-mataniani]

Hi, thanks for your help :)

This is ip addr:

[shmick@fedora ~]$ ip addr 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp24s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 30:9c:23:03:f1:6d brd ff:ff:ff:ff:ff:ff
    inet  [my-static-ip] /0 brd 255.255.255.255 scope global noprefixroute enp24s0
       valid_lft forever preferred_lft forever
    inet6 [ipv6-ip] scope link noprefixroute 
       valid_lft forever preferred_lft forever

And podman network ls:

[shmick@fedora ~]$ podman network ls
NETWORK ID    NAME             DRIVER
2f259bab93aa  podman           bridge
537ede6a024d  traefik_default  bridge

As far as I can tell I have had no containers.conf file initially - as mentioned, this is a brand new install. I've also tried the following contents with no success:

default_subnet_pools = [
        {"base" = "10.89.0.0/16", "size" = 24},
        {"base" = "10.90.0.0/15", "size" = 24},
        {"base" = "10.92.0.0/14", "size" = 24},
        {"base" = "10.96.0.0/11", "size" = 24},
        {"base" = "10.128.0.0/9", "size" = 24},
]

Please let me know if there's any more information I can provide.

Thanks!

[Luap99]

[my-static-ip] /0 That does not look like a valid setup, you made all ipv4 addresses part of your local subnet which that netmask. Thus there is no way to assign new ones without causing a conflict.

[interpretazioni-mataniani]

Thanks, that was it! I set the netmask to /32 and the issue went away.

[rdaton]

[my-static-ip] /0 That does not look like a valid setup, you made all ipv4 addresses part of your local subnet which that netmask. Thus there is no way to assign new ones without causing a conflict.

Thanks.... It works. I restored my laptop's network interface to defaults (DHCP) and it worked.

I had manually added an additional static IP to my physical network interface.