Kubric Docker Image GPU Rendering Issue with NVIDIA GPUs

I’m new to GPU rendering programs and currently working on a project using Kubric, a tool for synthetic data generation, and I’m facing an issue with GPU rendering. I’ve updated the Blender image, which Kubric’s image based on, from 2.9 to 3.6. Everything works fine under CPU rendering.

Despite successfully updating the Kubric Docker image based on Blender to incorporate GPU support for rendering (by referring to this github issue), I am unable to utilize the GPU effectively. The Blender docker image in question is hosted on Docker Hub under kangweiliao/blender:latest.

The modification was aimed at enabling GPU rendering by setting up the @use_gpu.setter in blender.py within Kubric as follows:

@use_gpu.setter
def use_gpu(self, value: bool):
  self.blender_scene.cycles.device = "GPU" if value else "CPU"
  if value:
    cycles_preferences = bpy.context.preferences.addons['cycles'].preferences
    cycles_preferences.compute_device_type = "CUDA"
    cycles_preferences.get_devices()
    devices = [d for d in cycles_preferences.devices]
    for d in devices:
      if d.type == "CUDA":
        d.use = True
      else:
        d.use = False  # disable all other devices
    logger.info("Using the following GPU Device: %s", [d.name for d in devices if d.use])

When running the Kubric image with the following command, the console correctly identifies my NVIDIA GeForce RTX 4070 Ti GPU, but it seems like the rendering process does not effectively utilize the GPU as expected.

docker run --rm --interactive --gpus all --env KUBRIC_USE_GPU=1 --volume "$(pwd):/kubric" local/kubruntu:gpu /usr/bin/python3 examples/helloworld.py

The console output indicates the GPU is detected:
INFO:kubric.renderer.blender:Using the following GPU Device: ['NVIDIA GeForce RTX 4070 Ti']

In addition, the program taking much longer time than the CPU rendering and printed hundred lines of ptxas info, like this:

...
ptxas info    : Used 71 registers, 380 bytes cmem[0], 16 bytes cmem[2]
ptxas info    : Compiling entry function 'kernel_gpu_integrator_reset' for 'sm_89'
ptxas info    : Function properties for kernel_gpu_integrator_reset
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 8 registers, 356 bytes cmem[0]
...

Despite the example output is corrected generated for Kubric after a long wait, it’s not working for my project and throwing error:

ptxas fatal   : Output file '/.cache/cycles/kernels/cycles_kernel_sm_89_D995D600D82B1FAE12AF1F8318F836BD.cubin' could not be opened
Failed to execute compilation command, see console for details.

Refer to the Cycles GPU rendering documentation for possible solutions:
https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html

Traceback (most recent call last):
Error: Failed to execute compilation command, see console for details.
Error: Failed to execute compilation command, see console for details.
Error: Failed to execute compilation command, see console for details.
  File "/workspace/scenes/movi.py", line 267, in <module>
    main(sys.argv)
  File "/workspace/scenes/movi.py", line 121, in main
    renderSetup(FLAGS)
  File "/workspace/scenes/movi.py", line 248, in renderSetup
    render(FLAGS, scene, renderer, simulator, output_dir, rng)
  File "/workspace/scenes/movi_render.py", line 568, in render
    data_stack = renderer.render()
  File "/usr/local/lib/python3.10/dist-packages/kubric/renderer/blender.py", line 286, in render  
    bpy.ops.render.render(animation=False, write_still=True)
  File "/blender-git/build_linux_bpy/bin/bpy/3.6/scripts/modules/bpy/ops.py", line 113, in __call__
    ret = _op_call(self.idname_py(), None, kw)
RuntimeError: Error: Failed to execute compilation command, see console for details.
Error: Failed to execute compilation command, see console for details.

The same setup with the official unmodified Kubric Docker image does not detect the GPU at all, suggesting there might be an underlying issue with how Kubric or Blender with Docker or the GPU.

The docker images I used to build the container could be found here

Has anyone encountered a similar issue or has suggestions on how to ensure the GPU is being utilized correctly for rendering in Kubric within a Docker environment? Any insights or advice would be greatly appreciated.

  • Please trim your code to make it easier to find your problem. Follow these guidelines to create a minimal reproducible example.

    – 
    Bot

  • This is a write permission or disk space problem

    – 

  • @talonmies Thanks. I’m new to GPU rendering, I noticed that along with this fatal error occurring, my WSL consumes almost all of my available RAM but not VRAM. Is this related to the problem you said?

    – 

  • 1

    I have no idea. The error means that the nvidia tool chain tried to create and then read back temporary files and failed. Usually that means a read/write permission issue or no free disk space. How that translates to your system is a question unrelated to rendering or cuda

    – 

  • @talonmies Thanks for the information! I will see what I can do to identify this problem.

    – 

Leave a Comment