Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces

Setup

Install the required libraries

conda create -n openfungraph python=3.10
conda activate openfungraph
##### Install Pytorch according to your own setup #####
# For example, if you have a GPU with CUDA 11.8
# Note that this version is compatible with the LLaVA repo
# Here we install cudatoolkit via Conda for installation of Grounded-SAM
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=11.8 cudatoolkit=11.8 -c pytorch -c nvidia
# Install the Faiss library (CPU version should be fine)
conda install -c pytorch faiss-cpu=1.7.4 mkl=2021 blas=1.0=mkl
# Install Pytorch3D by 
https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md
# We recommend installing from a local clone to avoid confliction
# Install the required libraries
pip install tyro open_clip_torch wandb h5py openai hydra-core distinctipy pyviz3d line_profiler
# Install the gradslam package and its dependencies
git clone https://github.com/krrish94/chamferdist.git
cd chamferdist
pip install .
cd ..
git clone https://github.com/gradslam/gradslam.git
cd gradslam
git checkout conceptfusion
pip install .

Install Grounded-SAM package

Follow the instructions on the original repo.

First checkout the package by

git clone git@github.com:IDEA-Research/Grounded-Segment-Anything.git

Then, install the package Following the commands listed in the original GitHub repo. You can skip the Install osx step and the "optional dependencies".

During this process, you will need to set the CUDA_HOME to be where the CUDA toolkit is installed. The CUDA tookit can be set up system-wide or within a conda environment. We tested it within a conda environment, i.e. installing cudatoolkit-dev using conda by former commands.

# and you need to replace `export CUDA_HOME=/path/to/cuda-11.3/` by 
export CUDA_HOME=/path/to/anaconda3/envs/openfungraph/

You also need to download ram_swin_large_14m.pth, groundingdino_swint_ogc.pth, sam_vit_h_4b8939.pth following the instruction here.

After installation, set the path to Grounded-SAM as an environment variable.

export GSA_PATH=/path/to/Grounded-Segment-Anything

Set up LLaVA

Follow the instructions on the LLaVA repo to set it up. We have tested with model checkpoint LLaVA-7B-v1.6.

Install this repo

cd OpenFunGraph
pip install -e .

Prepare dataset

Download the customized SceneFun3D dataset and the newly recorded FunGraph3D dataset.

In their top repo, file structure is introduced. Note that related path should be set as env_vars.bash.template.

export FUNGRAPH3D_ROOT=
export FUNGRAPH3D_CONFIG_PATH=${FG_FOLDER}/openfungraph/dataset/dataconfigs/fungraph3d/fungraph3d.yaml
export SCENEFUN3D_ROOT=  # for SceneFun3D, it should be with dev / test
export SCENEFUN3D_CONFIG_PATH=${FG_FOLDER}/openfungraph/dataset/dataconfigs/scenefun3d/scenefun3d.yaml

OpenFunGraph can also be easily run on other dataset. See dataset/datasets_common.py for how to write your own dataloader.

The env variables needed can be found in env_vars.bash.template. When following the setup guide below, you should change the variables accordingly for easy setup.

The following commands should be run in the openfungraph folder.

cd openfungraph

Functional Scene Graph Node Detection

export SCENE_NAME=
bash scenegraph/detection_scenefun3d.sh (or *_fungraph3d.sh)

The above commands will save the 2D node detection and segmentation results.

You can ignore the There's a wrong phrase happen, this is because of our post-process merged wrong tokens, which will be modified in the future. We will assign it with a random label at this time. message.

3D Functional Scene Graph Construction

Ensure that the openai package is installed and that your APIKEY is set. We recommend using GPT-4.

export OPENAI_API_KEY=<your GPT-4 API KEY here>

CUDA_VISIBLE_DEVICES=0 python scenegraph/build_fungraph_whole_openai.py --dataset_root ${SCENEFUN3D_ROOT} ``or`` ${FUNGRAPH3D_ROOT}  --scene_name ${SCENE_NAME} --mapfile ``<SCENE_PATH>/pcd_saves/full_pcd_ram_withbg_allclasses_overlap_maskconf0.3_bbox0.9_simsum1.2_dbscan.1_post.pkl.gz`` --part_file ``<SCENE_PATH>/part/pcd_saves/full_pcd_ram_withbg_allclasses_overlap_maskconf0.15_bbox0.1_simsum1.2_dbscan.1_parts_post.pkl.gz``

Visualize and Evaluation

After running the algorithm, you can get three modified key assets of object-level nodes <SCENE_PATH>/pcd_saves/full_pcd_ram_withbg_allclasses_overlap_maskconf0.3_bbox0.9_simsum1.2_dbscan.1_post.pkl.gz (the name could be varied depending on what parameters you choose), sub-object-level elements <SCENE_PATH>/part/pcd_saves/full_pcd_ram_withbg_allclasses_overlap_maskconf0.15_bbox0.1_simsum1.2_dbscan.1_parts_post.pkl.gz, and the finel graph edges <SCENE_PATH>/cfslam_funcgraph_edges.pkl (or with confidence).

Visualize them by

python scripts/pyviz3d_interactable_results.py --inter_result_path <PATH TO OBJECT-LEVEL NODES> --part_result_path <PATH TO SUB-OBJECT-LEVEL ELEMENTS> --edge_file <PATH TO GRAPH EDGES> --pc_path <PATH TO SCENE POINT CLOUD> (--pose_path (only for SCENEFUN3D) <SCENE_PATH>/*_transform.npy)

Evaluation scripts: For node evaluation

"python eval/eval_node.py --dataset --root_path --scene --video