¹Shanghai AI Lab ²Peking University ³S-Lab, NTU ⁴Shanghai Jiao Tong University
^*Equal Contribution ^†Corresponding Author

GeneMAN is a generalizable framework for single-view-to-3D human reconstruction, built on a collection of multi-source human data. Given a single in-the-wild image of a person, GeneMAN can reconstruct a high-quality 3D human model, regardless of its clothing, pose, or body proportions (e.g., a full-body, a half-body, or a close-up shot) in the given image.

Installation

Environment Setup

This part is the same as original threestudio. Skip it if you already have installed the environment.

See installation.md for additional information, including installation via Docker.

You must have an NVIDIA graphics card with at least 20GB VRAM and have CUDA installed.
Install Python >= 3.8.
(Optional, Recommended) Create a virtual environment:

conda create -n geneman python==3.10
conda activate geneman

Install PyTorch >= 1.12. We have tested on torch1.12.1+cu113 and torch2.0.0+cu118, but other versions should also work fine.

# torch1.12.1+cu113
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
# or torch2.0.0+cu118
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

(Optional, Recommended) Install ninja to speed up the compilation of CUDA extensions:

pip install ninja

Install dependencies:

git clone https://github.com/3DTopia/GeneMAN
cd GeneMAN
pip install -r requirements.txt

Models

Download our pre-trained GeneMAN models from HuggingFace.
Copy the pretrained_models folder into GeneMAN/pretrained_models. Copy the tets folder into GeneMAN/extern.
Download HumanNorm pretrained models on HuggingFace: Normal-adapted-model, Depth-adapted-model, Normal-aligned-model and ControlNet. Place HumanNorm pretrained models into GeneMAN/pretrained_models.
Download required model checkpoints for pre-processing:

For background removal, download ViT-H SAM model to GeneMAN/pretrained_models/seg.
YOLO11, BLIP2 and Sapiens models will be downloaded automatically on first use.

After downloading, the GeneMAN folder should be structured as follows:

GeneMAN/
├── extern/
│   └── tets/
├── pretrained_models/
│   ├── normal-adapted-sd1.5/
│   ├── depth-adapted-sd1.5/
│   ├── normal-aligned-sd1.5/
│   ├── controlnet-normal-sd1.5/
│   ├── geneman-prior2d/
│   ├── geneman-prior3d/
│   ├── sapiens/
│   └── seg/
│       ├── sam_vit_h_4b8939.pth
│       └── yolo11x.pt
└── …

QuickStart

Preprocessing

Pre-process the human images to remove background and obtain normals, depths, keypoints, and text prompts:

sh script/preprocess.sh

Usage

Our model is trained in multiple stages. Run 3D human reconstruction from a single image:

sh script/run.sh

[Note]: We have now switched to the Stage-3 strategy proposed in HumanNorm, based on our prior model. In our original pipeline, the texture-refinement stage was carried out by refining the UV map. We observed that a small fraction of the results exhibited instability and visible artifacts introduced by our original UV-map texture refinement. As an alternative, we have now switched to the Stage-3 strategy proposed in HumanNorm. If you would like to use UV map texture refine, you can refer to the Stage-3 scheme described in DreamGaussian. We will further enhance this stage in future updates.

Export Meshes

sh script/export_mesh.sh

TODO

Release the code.
Upload pretrained models.
Enhance texture refine stage.

Acknowledgments

Our project benefits from the amazing open-source projects:

We are grateful for their contribution.

Citation

If you find this work useful for your research, please consider citing our paper:

@article{wang2024geneman,
  title={GeneMAN: Generalizable Single-Image 3D Human Reconstruction from Multi-Source Human Data},
  author={Wang, Wentao and Ye, Hang and Hong, Fangzhou and Yang, Xue and Zhang, Jianfu and Wang, Yizhou and Liu, Ziwei and Pan, Liang},
  journal={arXiv preprint arXiv:2411.18624},
  year={2024}
}