Skip to content

Render Tests

Render, compositing and video sequencer tests compare render results against reference images.

Inspection

Tests result can be inspected visually through a HTML file that shows both the new image, reference image and a diff. This can be found at:

~/blender-git/build/tests/report.html

By default ctest output is sparse. More details including the Blender commands can be logged as follows:

cd ~/blender-git/build
BLENDER_VERBOSE=1 ctest -VV -R cycles

Adding and Updating Tests

Updating Reference Images

When adding a new test blend file, or when there are benign test failures due to intentional changes, the reference images can be created or updated as follows:

cd ~/blender-git/build
# Update reference images for cycles tests
BLENDER_TEST_UPDATE=1 ctest -R cycles

After this all the tests should pass, and .blend files and references images can be committed.

EEVEE

For EEVEE, always use the Nvidia OpenGL result as reference image. These references can be easily fetched from a failing test build result and copied back in the source folder eevee_renders inside each test suite. The threshold then need to be adjusted to pass on the different platforms in eevee_render_tests.py.

In case this is for adding a Cycles test, add the test to the blocklist in eevee_render_tests.py and to this task.

Categories

Tests are organized in folders per category.

When creating a new category folder, it must be added in tests/python/CMakeLists.txt.

Files

Each blend file generates a test in its category. It always renders frame number 1 to a PNG image.

Feature Location
Cycles, EEVEE, Workbench, Hydra tests/files/render/{category}/*.blend
Compositor tests/files/compositing/{category}/*.blend
Video Sequencer tests/files/sequence_editing/{category}/*.blend
Mesh Sculpting tests/files/sculpting/{category}/*.blend

GPU

By default only CPU tests are run. To enable GPU testing, change the CMake configuration.

Feature CMake Variable
Cycles CYCLES_TEST_DEVICES=CPU;{DEVICE}
with CUDA OPTIX ONEAPI HIP HIP-RT METAL METAL-RT
EEVEE, Workbench, Hydra WITH_GPU_RENDER_TESTS=ON
Compositor WITH_GPU_COMPOSITOR_TESTS=ON
Mesh Sculpting WITH_GPU_MESH_PAINT_TESTS=ON

Result may sometimes be different between CPU and GPU, or between different GPUs. For this reason some tests are blacklisted or use higher diff thresholds. See more about this in the next section.

Threshold udpate

For EEVEE, the update policy is:

  • Keep a very strict threshold for the reference platform (Nvidia OpenGL) for all tests.
  • Only bump the failure threshold when tests are not passing on other platform.
  • Bump the fail percentage in priority.
  • Bump the fail threshold if there are a lot of pixels failing (> 0.5%).
  • Bump these numbers per platform if the bump is large.
  • Tests that are non-deterministic should be added to the blocklist.

Batching

By default multiple tests are run with a single Blender launch, which speeds up test execution.

For investigating threading or memory corruption issues, disabling this can help avoid interference between tests. This is done by turning off WITH_TESTS_BATCHED.