WebGPU branch doesn't build

[thearperson]

Description of Issue

Check out: https://github.com/autodesk-forks/USD/tree/adsk/feature/webgpu

Steps to Reproduce

1. Run docker command to build

docker build -t usd-webgpu .

2. Got errors:

44.67 emcmake: error: 'cmake -DCMAKE_INSTALL_PREFIX=/src/USD_emscripten -DCMAKE_PREFIX_PATH=/src/USD_emscripten -DCMAKE_BUILD_TYPE=Release -DTINT_BUILD_SPV_READER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_HLSL_WRITER=OFF -DTINT_BUILD_MSL_WRITER=OFF -DTINT_BUILD_SPV_WRITER=OFF -DTINT_BUILD_WGSL_WRITER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_DOCS=OFF -DTINT_BUILD_SAMPLES=OFF -DTINT_BUILD_TESTS=OFF "-DCMAKE_CXX_FLAGS=-Wno-unsafe-buffer-usage -Wno-disabled-macro-expansion -Wno-#warnings -Wno-error -pthread" "-DCMAKE_EXE_LINKER_FLAGS=-sSTACK_SIZE=5MB -sSTACK_SIZE=5MB -sDEFAULT_PTHREAD_STACK_SIZE=2MB" -DBUILD_SHARED_LIBS=OFF /src/USD_emscripten/src/tint -DCMAKE_TOOLCHAIN_FILE=/tmp/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=/tmp/emsdk/node/16.20.0_64bit/bin/node' failed (returned 1)
44.67
44.67 ERROR: Failed to run 'emcmake cmake -DCMAKE_INSTALL_PREFIX="/src/USD_emscripten" -DCMAKE_PREFIX_PATH="/src/USD_emscripten" -DCMAKE_BUILD_TYPE=Release    -DTINT_BUILD_SPV_READER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_HLSL_WRITER=OFF -DTINT_BUILD_MSL_WRITER=OFF -DTINT_BUILD_SPV_WRITER=OFF -DTINT_BUILD_WGSL_WRITER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_DOCS=OFF -DTINT_BUILD_SAMPLES=OFF -DTINT_BUILD_TESTS=OFF -DCMAKE_CXX_FLAGS="-Wno-unsafe-buffer-usage -Wno-disabled-macro-expansion -Wno-#warnings -Wno-error -pthread" -DCMAKE_EXE_LINKER_FLAGS="-sSTACK_SIZE=5MB -sSTACK_SIZE=5MB -sDEFAULT_PTHREAD_STACK_SIZE=2MB" -DBUILD_SHARED_LIBS=OFF "/src/USD_emscripten/src/tint"'

System Information (OS, Hardware)

Ubuntu 22.04

Package Versions

Build Flags

[+] Building 159.8s (16/16) FINISHED                                                                                                                                                                                   docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                             0.1s
 => => transferring dockerfile: 1.04kB                                                                                                                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                0.1s
 => => transferring context: 2B                                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                                  0.9s
 => [ 1/12] FROM docker.io/library/ubuntu:20.04@sha256:33a5cc25d22c45900796a1aca487ad7a7cb09f09ea00b779e3b2026b4fc2faba                                                                                                          0.0s
 => [internal] load build context                                                                                                                                                                                              113.3s
 => => transferring context: 16.52GB                                                                                                                                                                                           113.0s
 => CACHED [ 2/12] RUN apt-get -y update && apt-get install -y        python-setuptools         libglew-dev         libxrandr-dev         libxcursor-dev         libxinerama-dev         libxi-dev         python3-pip           0.0s
 => CACHED [ 3/12] RUN curl -fsSL https://deb.nodesource.com/setup_current.x | bash - && DEBIAN_FRONTEND=noninteractive apt-get install -y nodejs libglib2.0-0                                                                   0.0s
 => CACHED [ 4/12] RUN pip install --upgrade pip                                                                                                                                                                                 0.0s
 => CACHED [ 5/12] RUN pip install PySide2 PyOpenGL Jinja2                                                                                                                                                                       0.0s
 => CACHED [ 6/12] RUN apt remove cmake                                                                                                                                                                                          0.0s
 => CACHED [ 7/12] RUN pip install cmake==3.20.3                                                                                                                                                                                 0.0s
 => CACHED [ 8/12] RUN mkdir -p tmp && cd tmp && git clone --recursive https://github.com/emscripten-core/emsdk                                                                                                                  0.0s
 => CACHED [ 9/12] RUN cd /tmp/emsdk && ./emsdk install 2.0.24                                                                                                                                                                   0.0s
 => CACHED [10/12] RUN cd /tmp/emsdk && ./emsdk activate 2.0.24 --permanent                                                                                                                                                      0.0s
 => CACHED [11/12] COPY . /src/                                                                                                                                                                                                  0.0s
 => ERROR [12/12] RUN source /tmp/emsdk/emsdk_env.sh && cd /src && python3 ./build_scripts/build_usd.py --emscripten USD_emscripten                                                                                             45.0s
------
 > [12/12] RUN source /tmp/emsdk/emsdk_env.sh && cd /src && python3 ./build_scripts/build_usd.py --emscripten USD_emscripten:
1.787 Setting up EMSDK environment (suppress these messages with EMSDK_QUIET=1)
1.787 Adding directories to PATH:
1.787 PATH += /tmp/emsdk
1.787 PATH += /tmp/emsdk/upstream/emscripten
1.787
1.787 Setting environment variables:
1.787 PATH = /tmp/emsdk:/tmp/emsdk/upstream/emscripten:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
1.787 EMSDK = /tmp/emsdk
1.787 EMSDK_NODE = /tmp/emsdk/node/16.20.0_64bit/bin/node
44.67 The following components were disabled because they are not compatible with emscripten: Python, examples, tutorials, tools, usdview, materialX
44.67
44.67 Building with settings:
44.67   USD source directory          /src
44.67   USD install directory         /src/USD_emscripten
44.67   3rd-party source directory    /src/USD_emscripten/src
44.67   3rd-party install directory   /src/USD_emscripten
44.67   Build directory               /src/USD_emscripten/build
44.67   CMake generator               Default
44.67   CMake toolset                 Default
44.67   Emscripten                    Enabled
44.67   Node                          Disabled
44.67   DAWN                          Disabled
44.67   Downloader                    curl
44.67
44.67   Building                      Shared libraries
44.67     Variant                     Release
44.67     Target
44.67     Imaging                     On
44.67       Ptex support:             Off
44.67       OpenVDB support:          Off
44.67       OpenImageIO support:      Off
44.67       OpenColorIO support:      Off
44.67       PRMan support:            Off
44.67     UsdImaging                  On
44.67       usdview:                  Off
44.67     Python support              Off
44.67       Python Debug:             Off
44.67       Python docs:              Off
44.67     Documentation               Off
44.67     Tests                       Off
44.67     Examples                    Off
44.67     Tutorials                   Off
44.67     Tools                       Off
44.67     Alembic Plugin              Off
44.67       HDF5 support:             Off
44.67     Draco Plugin                Off
44.67     MaterialX Plugin            Off
44.67
44.67   Dependencies                  boost, TBB, Tint, Shaderc, OpenSubdiv
44.67 STATUS: Installing boost...
44.67 STATUS: Installing TBB...
44.67 STATUS: Installing Tint...
44.67 -- -- Listing dependencies from /src/USD_emscripten/src/tint
44.67 -- -- Fetching dependency 'third_party/vulkan-deps'
44.67 -- -- Shallow cloning 'https://chromium.googlesource.com/vulkan-deps' at '5b82cc4a7eaa9befcf893bd0528b9cf93aae740a' into '/src/USD_emscripten/src/tint/third_party/vulkan-deps'
44.67 -- -- Checking out tag '5b82cc4a7eaa9befcf893bd0528b9cf93aae740a'
44.67 -- -- Listing dependencies from /src/USD_emscripten/src/tint/third_party/vulkan-deps
44.67 -- -- Fetching dependency 'spirv-headers/src'
44.67 -- -- Shallow cloning 'https://chromium.googlesource.com/external/github.com/KhronosGroup/SPIRV-Headers' at 'aa331ab0ffcb3a67021caa1a0c1c9017712f2f31' into '/src/USD_emscripten/src/tint/third_party/vulkan-deps/spirv-headers/src'
44.67 -- -- Checking out tag 'aa331ab0ffcb3a67021caa1a0c1c9017712f2f31'
44.67 -- -- Fetching dependency 'spirv-tools/src'
44.67 -- -- Shallow cloning 'https://chromium.googlesource.com/external/github.com/KhronosGroup/SPIRV-Tools' at '95f93810bbae12e1a601a3a5a5d975e5558a2994' into '/src/USD_emscripten/src/tint/third_party/vulkan-deps/spirv-tools/src'
44.67 -- -- Checking out tag '95f93810bbae12e1a601a3a5a5d975e5558a2994'
44.67 -- -- Listing dependencies from /src/USD_emscripten/src/tint/third_party/vulkan-deps/spirv-tools/src
44.67 -- -- Fetching dependency 'glslang/src'
44.67 -- -- Shallow cloning 'https://chromium.googlesource.com/external/github.com/KhronosGroup/glslang' at '6d41bb9c557c5a0eec61ffba1f775dc5f717a8f7' into '/src/USD_emscripten/src/tint/third_party/vulkan-deps/glslang/src'
44.67 -- -- Checking out tag '6d41bb9c557c5a0eec61ffba1f775dc5f717a8f7'
44.67 -- -- Listing dependencies from /src/USD_emscripten/src/tint/third_party/vulkan-deps/glslang/src
44.67 -- -- Fetching dependency 'third_party/abseil-cpp'
44.67 -- -- Shallow cloning 'https://chromium.googlesource.com/chromium/src/third_party/abseil-cpp' at 'bc3ab29356a081d0b5dd4ac55e30f4b45d8794cc' into '/src/USD_emscripten/src/tint/third_party/abseil-cpp'
44.67 -- -- Checking out tag 'bc3ab29356a081d0b5dd4ac55e30f4b45d8794cc'
44.67 2023-08-03 23:33
44.67 emcmake cmake -DCMAKE_INSTALL_PREFIX="/src/USD_emscripten" -DCMAKE_PREFIX_PATH="/src/USD_emscripten" -DCMAKE_BUILD_TYPE=Release    -DTINT_BUILD_SPV_READER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_HLSL_WRITER=OFF -DTINT_BUILD_MSL_WRITER=OFF -DTINT_BUILD_SPV_WRITER=OFF -DTINT_BUILD_WGSL_WRITER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_DOCS=OFF -DTINT_BUILD_SAMPLES=OFF -DTINT_BUILD_TESTS=OFF -DCMAKE_CXX_FLAGS="-Wno-unsafe-buffer-usage -Wno-disabled-macro-expansion -Wno-#warnings -Wno-error -pthread" -DCMAKE_EXE_LINKER_FLAGS="-sSTACK_SIZE=5MB -sSTACK_SIZE=5MB -sDEFAULT_PTHREAD_STACK_SIZE=2MB" -DBUILD_SHARED_LIBS=OFF "/src/USD_emscripten/src/tint"
44.67 configure: cmake -DCMAKE_INSTALL_PREFIX=/src/USD_emscripten -DCMAKE_PREFIX_PATH=/src/USD_emscripten -DCMAKE_BUILD_TYPE=Release -DTINT_BUILD_SPV_READER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_HLSL_WRITER=OFF -DTINT_BUILD_MSL_WRITER=OFF -DTINT_BUILD_SPV_WRITER=OFF -DTINT_BUILD_WGSL_WRITER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_DOCS=OFF -DTINT_BUILD_SAMPLES=OFF -DTINT_BUILD_TESTS=OFF "-DCMAKE_CXX_FLAGS=-Wno-unsafe-buffer-usage -Wno-disabled-macro-expansion -Wno-#warnings -Wno-error -pthread" "-DCMAKE_EXE_LINKER_FLAGS=-sSTACK_SIZE=5MB -sSTACK_SIZE=5MB -sDEFAULT_PTHREAD_STACK_SIZE=2MB" -DBUILD_SHARED_LIBS=OFF /src/USD_emscripten/src/tint -DCMAKE_TOOLCHAIN_FILE=/tmp/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=/tmp/emsdk/node/16.20.0_64bit/bin/node
44.67 -- Tint build samples: OFF
44.67 -- Tint build docs: OFF
44.67 -- Tint build docs with warn as error: OFF
44.67 -- Tint build SPIR-V reader: ON
44.67 -- Tint build WGSL reader: OFF
44.67 -- Tint build GLSL writer: ON
44.67 -- Tint build HLSL writer: OFF
44.67 -- Tint build MSL writer: OFF
44.67 -- Tint build SPIR-V writer: OFF
44.67 -- Tint build WGSL writer: ON
44.67 -- Tint build Syntax Tree writer: OFF
44.67 -- Tint build IR: ON
44.67 -- Tint build fuzzers: OFF
44.67 -- Tint build SPIRV-Tools fuzzer: OFF
44.67 -- Tint build AST fuzzer: OFF
44.67 -- Tint build regex fuzzer: OFF
44.67 -- Tint build benchmarks: OFF
44.67 -- Tint build tests: OFF
44.67 -- Tint build with ASAN: OFF
44.67 -- Tint build with MSAN: OFF
44.67 -- Tint build with UBSAN: OFF
44.67 -- Tint build checking [chromium-style]: OFF
44.67 -- Tint build remote-compile tool: OFF
44.67 -- Using python3
44.67 -- Found PythonInterp: /usr/bin/python3 (found suitable version "3.8.10", minimum required is "3")
44.67 CMake Warning (dev) at third_party/vulkan-deps/spirv-tools/src/source/CMakeLists.txt:380 (add_library):
44.67   ADD_LIBRARY called with SHARED option but the target platform does not
44.67   support dynamic linking.  Building a STATIC library instead.  This may lead
44.67   to problems.
44.67 This warning is for project developers.  Use -Wno-dev to suppress it.
44.67
44.67 CMake Warning at third_party/abseil-cpp/absl/copts/AbseilConfigureCopts.cmake:70 (message):
44.67   Value of CMAKE_SYSTEM_PROCESSOR (x86) is unknown and cannot be used to set
44.67   ABSL_RANDOM_RANDEN_COPTS
44.67 Call Stack (most recent call first):
44.67   third_party/abseil-cpp/CMake/AbseilHelpers.cmake:18 (include)
44.67   third_party/abseil-cpp/CMakeLists.txt:81 (include)
44.67
44.67
44.67 -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
44.67 -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
44.67 -- Looking for pthread_create in pthreads
44.67 -- Looking for pthread_create in pthreads - not found
44.67 -- Looking for pthread_create in pthread
44.67 -- Looking for pthread_create in pthread - not found
44.67 -- Check if compiler accepts -pthread
44.67 -- Check if compiler accepts -pthread - no
44.67 CMake Error at /usr/local/lib/python3.8/dist-packages/cmake/data/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
44.67   Could NOT find Threads (missing: Threads_FOUND)
44.67 Call Stack (most recent call first):
44.67   /usr/local/lib/python3.8/dist-packages/cmake/data/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
44.67   /usr/local/lib/python3.8/dist-packages/cmake/data/share/cmake-3.20/Modules/FindThreads.cmake:238 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
44.67   third_party/abseil-cpp/CMakeLists.txt:106 (find_package)
44.67
44.67
44.67 -- Configuring incomplete, errors occurred!
44.67 See also "/src/USD_emscripten/build/tint/CMakeFiles/CMakeOutput.log".
44.67 See also "/src/USD_emscripten/build/tint/CMakeFiles/CMakeError.log".
44.67 emcmake: error: 'cmake -DCMAKE_INSTALL_PREFIX=/src/USD_emscripten -DCMAKE_PREFIX_PATH=/src/USD_emscripten -DCMAKE_BUILD_TYPE=Release -DTINT_BUILD_SPV_READER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_HLSL_WRITER=OFF -DTINT_BUILD_MSL_WRITER=OFF -DTINT_BUILD_SPV_WRITER=OFF -DTINT_BUILD_WGSL_WRITER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_DOCS=OFF -DTINT_BUILD_SAMPLES=OFF -DTINT_BUILD_TESTS=OFF "-DCMAKE_CXX_FLAGS=-Wno-unsafe-buffer-usage -Wno-disabled-macro-expansion -Wno-#warnings -Wno-error -pthread" "-DCMAKE_EXE_LINKER_FLAGS=-sSTACK_SIZE=5MB -sSTACK_SIZE=5MB -sDEFAULT_PTHREAD_STACK_SIZE=2MB" -DBUILD_SHARED_LIBS=OFF /src/USD_emscripten/src/tint -DCMAKE_TOOLCHAIN_FILE=/tmp/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=/tmp/emsdk/node/16.20.0_64bit/bin/node' failed (returned 1)
44.67
44.67 ERROR: Failed to run 'emcmake cmake -DCMAKE_INSTALL_PREFIX="/src/USD_emscripten" -DCMAKE_PREFIX_PATH="/src/USD_emscripten" -DCMAKE_BUILD_TYPE=Release    -DTINT_BUILD_SPV_READER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_HLSL_WRITER=OFF -DTINT_BUILD_MSL_WRITER=OFF -DTINT_BUILD_SPV_WRITER=OFF -DTINT_BUILD_WGSL_WRITER=ON -DTINT_BUILD_WGSL_READER=OFF -DTINT_BUILD_DOCS=OFF -DTINT_BUILD_SAMPLES=OFF -DTINT_BUILD_TESTS=OFF -DCMAKE_CXX_FLAGS="-Wno-unsafe-buffer-usage -Wno-disabled-macro-expansion -Wno-#warnings -Wno-error -pthread" -DCMAKE_EXE_LINKER_FLAGS="-sSTACK_SIZE=5MB -sSTACK_SIZE=5MB -sDEFAULT_PTHREAD_STACK_SIZE=2MB" -DBUILD_SHARED_LIBS=OFF "/src/USD_emscripten/src/tint"'
44.67 See /src/USD_emscripten/build/tint/log.txt for more details.
------
Dockerfile:32
--------------------
  30 |
  31 |     COPY . /src/
  32 | >>> RUN source /tmp/emsdk/emsdk_env.sh && cd /src && python3 ./build_scripts/build_usd.py --emscripten USD_emscripten
  33 |
--------------------
ERROR: failed to solve: process "/bin/bash -c source /tmp/emsdk/emsdk_env.sh && cd /src && python3 ./build_scripts/build_usd.py --emscripten USD_emscripten" did not complete successfully: exit code: 1

[francis-wangfr]

Thank you Steve for the reporting the issue. This branch didn't tried by us on linux before. From error above, it seems pthreads lib doesn't support well on your OS. Maybe you would like to check pthreads lib or install it if there is not.

[thearperson]

Thanks for the prompt response!

yeah, there is something going on the pthread, but since this is building towards emscripten I thought the pthread comes with emsdk? see: https://emscripten.org/docs/porting/pthreads.html

Also note that pthread is already widely used in other places in the code base.

The problem, instead, seems to be related to tint and abseil lib:

This warning right before the pthread error seems to be suspicious. I'm wondering whether absl is configured for emsdk in this case.

44.67 CMake Warning at third_party/abseil-cpp/absl/copts/AbseilConfigureCopts.cmake:70 (message):
44.67   Value of CMAKE_SYSTEM_PROCESSOR (x86) is unknown and cannot be used to set
44.67   ABSL_RANDOM_RANDEN_COPTS
44.67 Call Stack (most recent call first):
44.67   third_party/abseil-cpp/CMake/AbseilHelpers.cmake:18 (include)
44.67   third_party/abseil-cpp/CMakeLists.txt:81 (include)

[francis-wangfr]

Yes, you are right Steve, it is emscripten pthreads. Glad to know you got document for pthreads. From above error you mentioned, it seems CMAKE_SYSTEM_PROCESSOR is x86, are you using x86 arch device? Not sure what is wrong.

[thearperson]

Looks like using newer version of emsdk solves the problem:

diff --git a/Dockerfile b/Dockerfile
index b0d7b230e..d10f618d8 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -2,7 +2,7 @@ FROM ubuntu:20.04
 SHELL ["/bin/bash", "-c"]

 ARG CMAKE_VERSION=3.20.3
-ARG EMSCRIPTEN_VERSION=2.0.24
+ARG EMSCRIPTEN_VERSION=3.1.31

 RUN apt-get -y update && apt-get install -y\
         python-setuptools \

[francis-wangfr]

It is glad to know you get problem resolved, Steve, yes, upper version is better. The 3.1.33 is recommended to use from our readme here.