Hardware transcoding does not work for some vertical videos

[Knallli]

The bug

Hardware transcoding fails on vertical videos since the vertical resolution goes too high on 4k content.

VAAPI reports following resolution constraints on HEVC:

[hevc_vaapi @ 0x5565e27c180] Hardware does not support encoding at size 1080x2344 (constraints: width 256-4096 height 128-2304).

Possible workaround could be to rotate the video for transcoding and then rotate it back on finish no?

The OS that Immich Server is running on

Unraid 6.12.6 w/ Linux 6.8rc3

Version of Immich Server

v1.94.1

Version of Immich Mobile App

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/user/appdata/immich/thumbs:/usr/src/app/upload/thumbs
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml 
      service: vaapi
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/user/appdata/immich/thumbs:/usr/src/app/upload/thumbs
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: openvino
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/user/appdata/immich/model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:latest
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - /mnt/user/appdata/immich/pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=xxx
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/mnt/user/main/immich/uploads
TYPESENSE_API_KEY=xxx
PUBLIC_LOGIN_PAGE_MESSAGE=Hello
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

Reproduction steps

1. Have a vertical video exceeding the vertical constraint of vaapi
2. Enable HW Transcoding with VAAPI
3. Set transcoding resolution to original 
4. crash hw transcoding by transcoding a vertical video with resolution above the constraint

Additional information

Console output of the FFMPEG error.

[Nest] 7  - 02/05/2024, 4:05:35 PM     LOG [MediaService] Start encoding video a535251b-bdb7-43a8-977b-6212f8954c2c {"inputOptions":["-init_hw_device vaapi=accel:/dev/dri/renderD128","-filter_hw_device accel"],"outputOptions":["-c:v hevc_vaapi","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-g 256","-tag:v hvc1","-v verbose","-vf format=nv12,hwupload","-compression_level 7","-qp 28","-global_quality 28","-rc_mode 1"],"twoPass":false}
[Nest] 7  - 02/05/2024, 4:05:35 PM   ERROR [MediaRepository] ffmpeg version 6.0-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[AVHWDeviceContext @ 0x5565e170200] libva: VA-API version 1.18.0
[AVHWDeviceContext @ 0x5565e170200] libva: Trying to open /usr/lib/jellyfin-ffmpeg/lib/dri/radeonsi_drv_video.so
[AVHWDeviceContext @ 0x5565e170200] libva: Found init function __vaDriverInit_1_18
amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description.
If they do, bad things may happen!
[AVHWDeviceContext @ 0x5565e170200] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x5565e170200] Initialised VAAPI connection: version 1.18
[AVHWDeviceContext @ 0x5565e170200] VAAPI driver: Mesa Gallium driver 23.2.0-devel for AMD Radeon Graphics (gfx1103_r1, LLVM 15.0.6, DRM 3.57, 6.8.0-rc3-thor-Unraid+).
[AVHWDeviceContext @ 0x5565e170200] Driver not found in known nonstandard list, using standard behaviour.
[h264 @ 0x5565e27ad80] Reinit context to 1088x2352, pix_fmt: yuv420p
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'upload/library/eea6a086-0ea0-4706-b3d2-4528dde59942/2019/2019-05-26/19-05-26-04-38-18+1.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2019-05-26T02:38:31.000000Z
    com.android.version: 9
  Duration: 00:00:12.10, start: 0.000000, bitrate: 15090 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuv420p(tv, smpte170m/bt470bg/smpte170m, progressive, left), 1080x2340 (1088x2352), 14990 kb/s, SAR 1:1 DAR 6:13, 27.43 fps, 45k tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2019-05-26T02:38:31.000000Z
      handler_name    : VideoHandle
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 96 kb/s (default)
    Metadata:
      creation_time   : 2019-05-26T02:38:31.000000Z
      handler_name    : SoundHandle
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_vaapi))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[graph_1_in_0_1 @ 0x5565e1a9840] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:mono
[h264 @ 0x5565e27d180] Reinit context to 1088x2352, pix_fmt: yuv420p
[graph 0 input from stream 0:0 @ 0x5565e1a6540] w:1080 h:2340 pixfmt:yuv420p tb:1/90000 fr:3735000/136141 sar:1/1
[auto_scale_0 @ 0x5565e0c4280] w:iw h:ih flags:'' interl:0
[Parsed_format_0 @ 0x5565e1a66c0] auto-inserting filter 'auto_scale_0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_format_0'
[auto_scale_0 @ 0x5565e0c4280] w:1080 h:2340 fmt:yuv420p sar:1/1 -> w:1080 h:2340 fmt:nv12 sar:1/1 flags:0x00000004
    Last message repeated 3 times
[hevc_vaapi @ 0x5565e27c180] Using input frames context (format vaapi) with hevc_vaapi encoder.
[hevc_vaapi @ 0x5565e27c180] Input surface format is nv12.
[hevc_vaapi @ 0x5565e27c180] Using VAAPI profile VAProfileHEVCMain (17).
[hevc_vaapi @ 0x5565e27c180] Using VAAPI entrypoint VAEntrypointEncSlice (6).
[hevc_vaapi @ 0x5565e27c180] Using VAAPI render target format YUV420 (0x1).
[hevc_vaapi @ 0x5565e27c180] Using CTU size 64x64, min CB size 8x8.
[hevc_vaapi @ 0x5565e27c180] RC mode: CQP.
[hevc_vaapi @ 0x5565e27c180] RC quality: 28.
[hevc_vaapi @ 0x5565e27c180] RC framerate: 65432/2385 (27.43 fps).
[hevc_vaapi @ 0x5565e27c180] Driver does not report any additional prediction constraints.
[hevc_vaapi @ 0x5565e27c180] Using intra and P-frames (supported references: 1 / 0).
[hevc_vaapi @ 0x5565e27c180] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[hevc_vaapi @ 0x5565e27c180] Hardware does not support encoding at size 1080x2344 (constraints: width 256-4096 height 128-2304).
[vost#0:0/hevc_vaapi @ 0x5565e2e7c80] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x5565e27cd80] Qavg: 53678.656
[aac @ 0x5565e27cd80] 2 frames left in the queue on closing
[AVIOContext @ 0x5565e1b8700] Statistics: 0 bytes written, 0 seeks, 0 writeouts
Terminating demuxer thread 0
[AVIOContext @ 0x5565e21a2c0] Statistics: 344470 bytes read, 3 seeks
Conversion failed!

[Nest] 7  - 02/05/2024, 4:05:35 PM   ERROR [MediaService] Error: ffmpeg exited with code 1: Conversion failed!

[aviv926]

Can you attach a sample file so we can reproduce the problem?

[Knallli]

https://www.pexels.com/video/road-trip-4434242/

@aviv926 the above sample can reproduce the problem on my end.

Edit: I have also attached the ffmpeg log for the sample and my vainfo
vainfo.txt
ffmpeg_log.txt

[mertalev]

It seems this is an issue with AMD devices in particular. Intel supports 4096x4096 and I can't see a limit on this for Nvidia. I also don't know if it affects discrete AMD GPUs or just APUs, and if there are different limits for different APUs.

All in all, it might be worth rotating for VAAPI only, but needs more investigation as to the surface area of the issue and how much of a performance hit it would be for devices that don't need this.

[Knallli]

Could this issue be on VAAPI itself? According to the datasheet for my Phoenix iGPU there should be 8k encode, decode support for h265 8/10 bit

[ext4xfs]

https://patchwork.freedesktop.org/patch/577299/ - I believe this is related.

[mertalev]

That does look related. I think we can wait for that to make its way into an FFmpeg release.

[jrasm91]

Is this still an issue?

[mertalev]

Yes. It should be fixed once we upgrade to FFmpeg 7.0. jellyfin-ffmpeg's relevant PR was merged recently, so that might be possible soon.

[sand-r]

I still have the same issue on latest release with ffmpeg 7.0

[Nest] 7  - 10/19/2024, 4:56:00 PM     LOG [Microservices:MediaService] Encoding video 97ea2c5f-cb3a-49d8-a18b-d5c7c8183623 with VAAPI acceleration
[Nest] 7  - 10/19/2024, 4:56:00 PM   ERROR [Microservices:MediaRepository]         vendor_id       : [0][0][0][0]
        encoder         : HEVC
      Side data:
        DOVI configuration record: version: 1.0, profile: 8, level: 10, rpu flag: 1, el flag: 0, bl flag: 1, compatibility id: 4
        displaymatrix: rotation of -90.00 degrees
        Ambient Viewing Environment, ambient_illuminance=314.000000, ambient_light_x=0.312700, ambient_light_y=0.329000
  Stream #0:1[0x2](und): Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 102 kb/s (default)
      Metadata:
        creation_time   : 2024-10-19T16:51:55.000000Z
        handler_name    : Core Media Audio
        vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](und): Data: none (mebx / 0x7862656D), 0 kb/s (default)
      Metadata:
        creation_time   : 2024-10-19T16:51:55.000000Z
        handler_name    : Core Media Metadata
  Stream #0:3[0x4](und): Data: none (mebx / 0x7862656D), 0 kb/s (default)
      Metadata:
        creation_time   : 2024-10-19T16:51:55.000000Z
        handler_name    : Core Media Metadata
  Stream #0:4[0x5](und): Data: none (mebx / 0x7862656D), 69 kb/s (default)
      Metadata:
        creation_time   : 2024-10-19T16:51:55.000000Z
        handler_name    : Core Media Metadata
  Stream #0:5[0x6](und): Data: none (mebx / 0x7862656D), 5 kb/s (default)
      Metadata:
        creation_time   : 2024-10-19T16:51:55.000000Z
        handler_name    : Core Media Metadata
  Stream #0:6[0x7](und): Data: none (mebx / 0x7862656D), 0 kb/s (default)
      Metadata:
        creation_time   : 2024-10-19T16:51:55.000000Z
        handler_name    : Core Media Metadata
[out#0/mp4 @ 0x5d31a1f5f40] Adding streams from explicit maps...
[vost#0:0/h264_vaapi @ 0x5d31a2e5280] Created video stream from input stream 0:0
[aost#0:1/libopus @ 0x5d31a2e5580] Created audio stream from input stream 0:1
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (aac (native) -> opus (libopus))
[vost#0:0/h264_vaapi @ 0x5d31a2e5280] Starting thread...
[aost#0:1/libopus @ 0x5d31a2e5580] Starting thread...
[vf#0:0 @ 0x5d31a14aa40] Starting thread...
[af#0:1 @ 0x5d31a147fc0] Starting thread...
[vist#0:0/hevc @ 0x5d31a17ba00] [dec:hevc @ 0x5d31a17c780] Starting thread...
[aist#0:1/aac @ 0x5d31a17bb80] [dec:aac @ 0x5d31a066580] Starting thread...
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0x5d31a123f80] Starting thread...
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0x5d332050480] w:3840 h:2160 pixfmt:yuv420p10le tb:1/600 fr:60000/1001 sar:0/1 csp:bt2020nc range:tv
[auto_scale_0 @ 0x5d332050900] w:iw h:ih flags:'' interl:0
[transpose @ 0x5d332050540] auto-inserting filter 'auto_scale_0' between the filter 'graph 0 input from stream 0:0' and the filter 'transpose'
[auto_scale_0 @ 0x5d332050900] w:3840 h:2160 fmt:yuv420p10le csp:bt2020nc range:tv sar:0/1 -> w:3840 h:2160 fmt:nv12 csp:bt2020nc range:tv sar:0/1 flags:0x00000004
[transpose @ 0x5d332050540] w:3840 h:2160 dir:1 -> w:2160 h:3840 rotation:clockwise vflip:0
[graph 0 input from stream 0:0 @ 0x5d332050480] video frame properties congruent with link at pts_time: 0
[h264_vaapi @ 0x5d31a26d980] Using input frames context (format vaapi) with h264_vaapi encoder.
[h264_vaapi @ 0x5d31a26d980] Input surface format is nv12.
[h264_vaapi @ 0x5d31a26d980] Using VAAPI profile VAProfileH264High (7).
[h264_vaapi @ 0x5d31a26d980] Using VAAPI entrypoint VAEntrypointEncSlice (6).
[h264_vaapi @ 0x5d31a26d980] Using VAAPI render target format YUV420 (0x1).
[h264_vaapi @ 0x5d31a26d980] RC mode: CQP.
[h264_vaapi @ 0x5d31a26d980] Block Level bitrate control: OFF.
[h264_vaapi @ 0x5d31a26d980] RC quality: 31.
[h264_vaapi @ 0x5d31a26d980] RC framerate: 60000/1001 (59.94 fps).
[h264_vaapi @ 0x5d31a26d980] Driver does not report any additional prediction constraints.
[h264_vaapi @ 0x5d31a26d980] Using intra and P-frames (supported references: 1 / 0).
[h264_vaapi @ 0x5d31a26d980] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[h264_vaapi @ 0x5d31a26d980] Hardware does not support encoding at size 1440x2560 (constraints: width 256-4096 height 128-2304).
[vost#0:0/h264_vaapi @ 0x5d31a2e5280] Error while opening encoder - maybe incorrect parameters such as bit_rate, rate, width or height.
[vf#0:0 @ 0x5d31a14aa40] Error sending frames to consumers: Invalid argument
[vf#0:0 @ 0x5d31a14aa40] Task finished with error code: -22 (Invalid argument)
[vf#0:0 @ 0x5d31a14aa40] Terminating thread with return code -22 (Invalid argument)
[vist#0:0/hevc @ 0x5d31a17ba00] [dec:hevc @ 0x5d31a17c780] Decoder returned EOF, finishing
[vist#0:0/hevc @ 0x5d31a17ba00] [dec:hevc @ 0x5d31a17c780] Terminating thread with return code 0 (success)

[mertalev]

What processor do you have? 7.0 should only help if the true dimension range is enough and was only misreported in earlier versions.

[sand-r]

It's a ryzen5 5600h

Edit:

Looks like it should support 8k encoding? I think it should have the Cezanna gpu
https://en.wikipedia.org/wiki/Unified_Video_Decoder#Format_support

[mertalev]

https://patchwork.freedesktop.org/patch/577299/ - I believe this is related.

I looked at this again and it's actually a driver patch rather than for FFmpeg. All in all, this doesn't seem like an immich issue to me.