Can't start rootless container with --userns=keep-id when graphroot is on a different drive: OCI permission denied

[Supreeeme]

Issue Description

Starting a rootless container with --userns=keep-id when graphroot is on a different drive fails. This happens when configured via storage.conf and when it is a symlink to a folder on a different drive.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a storage.conf in ~/.config/containers with graphroot set to a folder on a different drive:

[storage]
driver = "overlay"
runroot = "$XDG_RUNTIME_DIR/containers/storage"
graphroot = "/run/media/mmcblk0p1/containers/storage"

2. Create a (rootless?) container: podman create --name=arch --userns=keep-id -it archlinux:latest /bin/bash
3. Try to start it: podman start arch

Describe the results you received

Got the following error:

Error: unable to start container "408221eb3b9b69a1b9633a4f7b6363b890ffb22940e4403276f6fe3fbdbb91e2": crun: open `/run/media/deck/4c4f3a65-a785-4f13-a085-17bea7c88ec3/containers/storage/overlay/0a9a697edc2b4c8ba94bc9d128c2766a2e796afa9cb632595db81214ea34c761/merged`: Permission denied: OCI permission denied

Describe the results you expected

Expected the container to start successfully.

podman info output

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /nix/store/d5ypxvhg6macxn9y6ayxfqq58k75ma0c-podman-helper-binary-wrapper/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 85.7
    systemPercent: 2.74
    userPercent: 11.56
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: holo
    distribution: steamos
    variant: steamdeck
    version: 3.6.20
  eventLogger: journald
  freeLocks: 2043
  hostname: steamdeck
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.5.0-valve22-1-neptune-65-g9a338ed8a75e
  linkmode: dynamic
  logDriver: journald
  memFree: 2368831488
  memTotal: 15531683840
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /nix/store/7qjh34d4ijk6cnjw755ihkd9lc3m91cg-podman-5.3.0/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.0
    package: Unknown
    path: /nix/store/7qjh34d4ijk6cnjw755ihkd9lc3m91cg-podman-5.3.0/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: Unknown
    path: /nix/store/d5ypxvhg6macxn9y6ayxfqq58k75ma0c-podman-helper-binary-wrapper/bin/crun
    version: |-
      crun version 1.18.2
      commit: 1.18.2
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /nix/store/7qjh34d4ijk6cnjw755ihkd9lc3m91cg-podman-5.3.0/libexec/podman/pasta
    package: Unknown
    version: |
      pasta 2024_09_06.6b38f07
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 8838963200
  swapTotal: 8839487488
  uptime: 2h 26m 23.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/deck/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /run/media/deck/4c4f3a65-a785-4f13-a085-17bea7c88ec3/containers/storage
  graphRootAllocated: 503513473024
  graphRootUsed: 387375247360
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers/storage
  transientStore: false
  volumePath: /run/media/deck/4c4f3a65-a785-4f13-a085-17bea7c88ec3/containers/storage/volumes
version:
  APIVersion: 5.3.0
  Built: 315532800
  BuiltTime: Mon Dec 31 19:00:00 1979
  GitCommit: ""
  GoVersion: go1.23.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

This is on a Steam Deck, but podman has been installed from Nix.

Additional information

Without --userns=keep-id I can start the container fine, but obviously I'm root in the container, which I don't necessarily want.
If the graphroot is in my home directory, --userns=keep-id works without issue.
I've seen a couple issues with the OCI permission denied issue but didn't see any resolution, so thought I'd open a new one.

[giuseppe]

do the parent directories for your graphroot have the executable bit set?

Please check that /run, /run/media, /run/media/mmcblk0p1, /run/media/mmcblk0p1/containers and /run/media/mmcblk0p1/containers/storage have +x set for other users.

[Supreeeme]

Ah, that was it - /run/media/mmcblk0p1 is a symlink and one of the directories in the hierarchy it was symlinked to was missing +x for other users. Thanks! I wonder if the error message could be improved?