GitHub

Dark3R: Low-Light 3D Reconstruction

Dark3R extends MASt3R (Matching Anything by Segmenting with 3D Representations) to work effectively with low-light and noisy images. The method uses LoRA (Low-Rank Adaptation) fine-tuning to adapt the pre-trained MASt3R model for low-light conditions while maintaining performance on clean images.

1. Installation Instructions

Dark3R uses a conda environment for dependency management. The project includes an environment.yml file that specifies all required dependencies.

Prerequisites

  • Conda (Miniconda or Anaconda)
  • CUDA-capable GPU (recommended for training and inference)
  • CUDA 12.1+ (for GPU support)

Installation Steps

  1. Clone the repository (if not already done):

    git clone <repository-url>
cd Dark3R

  2. Create the conda environment from the environment.yml file:

    conda env create -f environment.yml

  3. Activate the environment:

    conda activate dark3r

  4. Verify installation:

    python -c "import torch; print(f'PyTorch version: {torch.__version__}')"
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"

The environment includes all necessary dependencies including PyTorch, CUDA support, computer vision libraries, 3D reconstruction tools, and evaluation utilities.

Downloading MASt3R Checkpoints

Follow the instructions in the original MASt3R repository to download the pre-trained MASt3R weights into checkpoints/.

Make sure to download both the main model and the retrieval model. Your checkpoints folder should have the following:

checkpoints/
├── MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth
├── MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric_retrieval_codebook.pkl
└── MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric_retrieval_trainingfree.pth

If those are not available, we have a backup of these weights in this folder.

2. Downloading the Dataset

Dataset Structure

Dark3R datasets are organized with the following structure:

<DATASET_ROOT>/
├── bc040_chapel/                  # scene
│   ├── iso102400_s1_00080/        # exposure
│   │   ├── downsampled_004/       # resolution
│   │   ├── downsampled_016/
│   │   ├── images_arw/
│   ├── iso102400_s1_00125/
│   │   ├── downsampled_016/
│   │   └── images_arw/
│   └── ...
├── bc041_traincar/
│   └── ...
└── ...

The dataset is organized on 3 levels: scene, exposure, and resolution.

Folder Nomenclature

Each scene folder (starting with bc) contains multiple exposures (starting with iso), each containing different resolutions. The naming convention is iso<iso value>_s<1 or 0>_<exposure time>. For example, iso102400_s1_00125 means ISO 102400 and exposure time of 1/125s. iso102400_s0_00003 would mean ISO 102400 and exposure time of 3s.

Downloading Datasets

The dataset can be downloaded using the script download/download_dataset.py. There are pre-defined configuration files in download/config that control which scenes, exposures, and resolutions are downloaded.

Real Captures (Tripod)

If you do not need the original raw files, which are extremely large in size, we recommend using the config download/config/full_scenes_downsampled.yaml.

python download/download_dataset.py download/config/full_scenes_downsampled.yaml

Synthetic Captures (Handheld)

To download synthetic captures, we also provide a pre-defined config.

python download/download_dataset.py download/config/synthetic_downsampled.yaml

COLMAP Poses: We also provide COLMAP poses on the long exposure images to be used as reference for computing pose error. They can be found in the colmap_long_exposure folder inside where the dataset was unzipped.

It is recommended to use the script above to download the dataset. However, the full dataset can also be found using this Dropbox Link.

Additional Info

For more details on the dataset, please see download/dataset_info.md.

3. Training Instructions

Note, this is only required for training Dark3R. If you are using the pretrained weights, continue to step 4.

Generating Pairs

To train Dark3R, the model requires pairs of images with visual overlap to learn dense matching and 3D consistency. Instead of computing these on the fly, we pre-calculate valid pairs using image retrieval, following the MASt3R-SfM protocol.

Run the following script to generate a JSON file containing these pairs:

python utils/generate_training_pairs.py --train_dataset_root data/dark3r_dataset

Output: This will create a pairs file (e.g., pairs_rf0050_ws0050.json) that you must pass to the --precomputed_pairs argument when running the training script.

Note: This script will only compute pairs for scenes in the training dataset. These are defined in dataset/dataset_config/finetuning_real_datasets_1023.yml.

Running the Training Script

To train the model, run train_dark3r.py. The script uses torch.distributed for multi-GPU support.

Below is the configuration we found to have the best results and correspond to the quantitative results in the paper.

python -m torch.distributed.run --nproc_per_node=1 train_dark3r.py \
    --batch_size 2 \
    --num_epochs 20 \
    --save_dir results/train_dark3r \
    --real_dataset_config dataset/dataset_config/finetuning_real_datasets_1023.yml \
    --real_dataset_root data/dark3r_dataset \
    --sim_dataset_root data/synthetic_dataset \
    --precomputed_pairs pairs_rf0050_ws0050.json

Key Arguments

  • --nproc_per_node: Number of GPUs to use. The paper uses 8 A6000 GPUs.

  • --losses: Specifies the loss components. encoder and desc refer to the consistency losses on noisy inputs.

  • --real_dataset_root: Path to the real-world captured dataset (handheld/tripod) used for "captured data" supervision.

  • --sim_dataset_root: Path to the dataset where noise is synthesized using a Poisson-Gaussian model.

  • --precomputed_pairs: JSON file containing the pairs of images to be used for training (generated via retrieval).

Training Outputs

Checkpoints and logs will be saved to the directory specified by -o / --output_dir. The final model weights should be used for the inference steps in Section 4.

4. Downloading Pretrained Weights

Skip this step and continue to step 5 if you are not using the pretrained weights.

To download the pretrained weights, run the command:

python download/download_dataset.py download/config/pretrained_weights.yaml

5. Inference Instructions

Dark3R inference performs 3D reconstruction from a set of low-light images. The canonical format for running inference is:

python forward_dark3r.py \
    --filelist "data/dark3r_dataset/bc040_chapel/iso102400_s1_02000/downsampled_016/images_npy_undistorted/*.npy" \
    --outdir results/test_dark3r \
    --finetuned_model <FINETUNED_LORA_PATH> \
    --winsize 15 --refid 10

Key Arguments:

  • --filelist: Glob pattern or list of image file paths (typically .npy files)
  • --outdir: Output directory for reconstruction results
  • --finetuned_model: Path to trained LoRA weights file (typically ends with _lora.pth)
  • --winsize: number of key images (

Read the original on github.com ↗