Paper

This repo is the code distribution for the Objects with Lighting dataset. It contains the evaluation script (scripts/evaluation.py) and the tools used for building the dataset.

If you find the data or code useful please cite

@inproceedings{Ummenhofer2024OWL,
  author       = {Benjamin Ummenhofer and
                  Sanskar Agrawal and
                  Rene Sep{\'{u}}lveda and
                  Yixing Lao and
                  Kai Zhang and
                  Tianhang Cheng and
                  Stephan R. Richter and
                  Shenlong Wang and
                  Germ{\'{a}}n Ros},
  title        = {Objects With Lighting: {A} Real-World Dataset for Evaluating Reconstruction
                  and Rendering for Object Relighting},
  booktitle    = {3DV},
  publisher    = {{IEEE}},
  year         = {2024}
}

Downloads

Please download the dataset from the current release or use the links below for the latest version. Extracting the files in the repository root will create the dataset directory.

wget https://github.com/isl-org/objects-with-lighting/releases/download/v1/objects-with-lighting-dataset-v1_{1,2}.tgz

Directory structure

├─ calibration      # Data files for calibration and generated calibration parameters
├─ dataset          # Contains the data meant for consumption
├─ docs             # Images and markdown for documentation
├─ methods          # Documentation and scripts for the baseline method and other state-of-the-art methods
├─ scripts          # This dir contains scripts for creating data and evaluation
├─ utils            # Utility python modules used by all scripts

Evaluation script

The script scripts/evaluate.py can be used to compute the common metrics PSNR, SSIM, LPIPS for predicted images. The results will be stored in a json file.

Supported image file formats are .png, .exr, and .npy.
We assume .exr and .npy files store unclipped linear images, while .png stores values after applying the tonemapping as used for the dataset.
For linear images the evaluation script computes the optimal exposure value minimizing the least squares error before computing the error metrics.
The predicted images have to be stored with the same folder structure as the dataset and should be named pr_image_xxxx.{npy,exr,png}.

The script can be invoked as

python scripts/evaluate.py -p path/to/predictions results.json

Dataset format

├─ dataset
    ├─ object_name     # The dataset is grouped into objects
        ├─ test        # Files in the test dir are meant for evaluation
            ├─ inputs  # The inputs dir contains all files that are allowed to be used by the methods

Each of the test directories contains the following data.

File	Description
`inputs/`	This directory contains all data for reconstructing the object. For a fair evaluation only data inside this folder may be used.
`inputs/image_xxxx.png`	Image files with 8-bit RGB images after tonemapping.
`inputs/camera_xxxx.txt`	Camera parameters for the corresponding image file.
`inputs/mask_xxxx.png`	An approximate mask for methods that require it.
`inputs/exposure.txt`	The exposure value that has been used in the tonemapping.
`inputs/object_bounding_box.txt`	The axis aligned bounding box of the object. This box is not a tight bounding box.
`env.hdr`	An equirectangular image of the environment where the input images have been taken. This image is provided for debugging purposes and should not be used for reconstruction or evaluation.
`env_512_rotated.hdr`	This environment map is downscaled to 1024x512 and has been rotated with the 'world_to_env' transform for easier usage. This image is provided for debugging purposes and should not be used for reconstruction or evaluation.
`world_to_env.txt`	The 4x4 world to camera transform that transforms a point into the coordinate system of the equirectangular image 'env.hdr'.
`gt_image_xxxx.png`	A ground truth image used in evaluation.
`gt_camera_xxxx.txt`	The corresponding camera parameters for a ground truth image.
`gt_mask_xxxx.png`	The mask used for evaluation. Valid pixels are marked with the value 255.
`gt_exposure_xxxx.txt`	The exposure used in the tonemapping of the corresponding ground truth image.
`gt_env_xxxx.hdr`	An equirectangular image of the environment where the corresponding ground truth image was taken.
`gt_world_to_env_xxxx.txt`	The 4x4 world to camera transform that transforms a point into the coordinate system of the equirectangular image 'gt_env_xxxx.hdr'.
`gt_env_512_rotated_xxxx.hdr`	This environment map is downscaled to 1024x512 and has been rotated with the 'world_to_env' transform for easier usage.

Tone mapping

We generate the tonemapped 8-bit images with the following function.