Headless Rendering

Headless rendering using ssc scripts

We use https://github.com/stackgl/headless-gl/ for offscreen/headless rendering. There are some known issues/difficulties with this.

Remote headless rendering

If you are trying to use a remote machine for rendering, you will need to make sure you can get a WebGL context. There are several options for achieving this.

Using forked headless-gl with EGL support

We provide an adapted version of the headless-gl library which uses EGL to create an OpenGL context (allowing for easier offscreen rendering on machines with Nvidia GPUs). Many thanks to Erik Wijmans for adapting the headless-gl code. Download the headless-gl-nvidia.zip archive from the latest sstk release, and follow the instructions in the contained source code to compile a new build/Release/webgl.node binary file (you may also attempt to use the precompiled one, in case it works with your OS and driver combination). Either way, copy the build/Release/webgl.node file into node_modules/gl/build/Release/webgl.node, overriding the file that is there.

Using xvfb-run

It is possible to run the code on machines without a full X server session using the xvfb-run tool (see https://github.com/stackgl/headless-gl#how-can-headless-gl-be-used-on-a-headless-linux-machine for some details).

1. Install Xvfb (which should include the xvfb-run script) on your system. On Ubuntu distributions this is typically done with sudo apt-get install xvfb.
2. Check that you can obtain a glx-enabled display context using xvfb-run -s "-ac -screen 0 1280x1024x24" glxinfo. You can also run xvfb-run -s "-ac -screen 0 1280x1024x24" xvinfo to print available X extension information.
3. On some systems, issues with the above are due to the LIBGL_ALWAYS_INDIRECT environment variable being not defined. If it is defined, you can use unset LIBGL_ALWAYS_INDIRECT to undefine it and then try again.
4. Prefix command calls that require offscreen rendering with xvfb-run in this way: e.g., xvfb-run -s "-ac -screen 0 1280x1024x24" render-file.js --input <filename>.

Using VirtualGL and VNC

If you have X11 and a desktop manager setup (or are willing to install them on your system), you can setup VirtualGL and VNC for rendering on a remote desktop session. See basic steps: https://gist.github.com/cyberang3l/422a77a47bdc15a0824d5cca47e64ba2 See detailed instructions: https://virtualgl.org/vgldoc/2_1_1/

Issue with creating WebGL context

If you get the following error:

THREE.WebGLRenderer 73
THREE.WebGLRenderer: Error creating WebGL context.
...
TypeError: Cannot read property 'getExtension' of null

you were not able to get a WebGL context. To fix this, please setup your machine to support remote headless rendering using xvfb-run or VirtualGL + VNC.

Issue with headless rendering on machines with nvidia drivers

If you have issues doing headless rendering using the xvfb-run tool, and running xvfb-run -s "-ac -screen 0 1280x1024x24" glxinfo gives messages like these:

name of display: :99
Error: couldn't find RGB GLX visual or fbconfig
Error: couldn't find RGB GLX visual or fbconfig

Then most likely the issue is due to the OpenGL drivers installed along with the nvidia drivers. The solution is to use the --no-opengl-libs parameter when installing both the nvidiaa driver and CUDA. See a guide describing this (with more details specific to AWS deployment) at https://davidsanwald.github.io/2016/11/13/building-tensorflow-with-gpu-support.html

Issue with using nVidia GPUs on Ubuntu

There is a known issue with support of headless-gl on nVidia GPU on Ubuntu: https://github.com/stackgl/headless-gl/issues/65

You have likely encountered this if you see the following error:

node: symbol lookup error: ssc/node_modules/gl/build/Release/webgl.node: undefined symbol: _Z15XextFindDisplayP15_XExtensionInfoP9_XDisplay

To resolve this, try following the steps described in this comment: https://github.com/stackgl/headless-gl/issues/65#issuecomment-252742795 . It requires building the headless-gl library manually, and then copying the output webgl.node into the ssc/node_modules/gl/build/Release/webgl.node (overriding the version installed by default).

Be sure to save your built webgl.node somewhere safe. If you ever do npm install in the ssc directory you will need to find your manually built webgl.node and copy it again.