This repository contains the official PyTorch implementation of the 3DV 2022 paper:

Neural Point-based Shape Modeling of Humans in Challenging Clothing
Qianli Ma, Jinlong Yang, Michael J. Black and Siyu Tang
Paper | Supp | Project website

Installation

This repository is based on our prior work, POP and SCALE. If you have successfully run either one of them before, you can skip steps 1 and 2 below. The code has been tested with python 3.8 on Ubuntu 20.04 + CUDA 11.0.

1. First, in the folder of this repository, run the following commands to create a new virtual environment and install dependencies:

python3 -m venv $HOME/.virtualenvs/SkiRT
source $HOME/.virtualenvs/SkiRT/bin/activate
pip install -U pip setuptools
pip install -r requirements.txt

2. Install the Chamfer Distance package (MIT license, taken from this implementation). The compilation is verified to be successful under CUDA 11.0.

cd chamferdist
python setup.py install
cd ..

3. Install the psbody.mesh mesh processing library. Simply download the appropriate python wheels from their Releases site and simply install pip install <downloaded_wheel_file>.whl.
4. Download our pre-processed body model related assets for training SkiRT, such as the barycentric coordinates and face indices of the UV query points on the SMPL/SMPL-X body meshes.

wget https://keeper.mpdl.mpg.de/f/9fc3ed8add544b4b9d8a/?dl=1 -O assets.zip && unzip assets.zip -d assets && rm assets.zip

You are now good to go with the next steps! All the commands below are assumed to be run from the SkiRT repository folder, within the virtual environment created above.

Download data

• Create folders for data and SMPL-X body model: mkdir body_models data.

• Download SMPL-X body model and place the model .pkl files under body_models/smplx/.

• Download the packed data ReSynth dataset.

  • After a simple registration, go to the "Download" tab, section "Option 2" (Download Data for Each Subject") there. Choose the subject(s) of interest, download the "packed packed npz files", unzip respectively to the data/resynth/ folder.
  • On the same download page, section "Pre-processed Attributes for SkiRT (3DV 2022)", download our pre-processed extra attributes of your selected subject(s) and unzip to the data/resynth_extra_params/ folder.

• The data file organization will look like this:

SkiRT
├── body_models
│   ├── smplx
│   │   │── SMPLX_FEMALE.pkl
│   │   │── SMPLX_MALE.pkl
│   │   ├── SMPLX_NEUTRAL.pkl
├── data
│   ├── resynth
│   │   ├── rp_anna_posed_001
│   │   │   ├── test
│   │   │   │   ├── <per-frame npz files>
│   │   │   ├── train
│   │   │   │   ├── <per-frame npz files>
│   │   ├── rp_beatrice_posed_025
│   │   ├── ...
│   ├── resynth_extra_params
│   │   ├── rp_anna_posed_001
│   │   │   ├── test
│   │   │   │   ├── <per-frame npz files>
│   │   │   ├── train
│   │   │   │   ├── <per-frame npz files>
│   │   ├── rp_beatrice_posed_025
│   │   ├── ...

Training and Inference

Overview

Training SkiRT involves training a coarse stage for pose-independent, coarse "template" of clothed body, followed by a fine stage that predicts the LBS weights and pose-dependent geometry.

Below we take the ReSynth dataset, subject rp_beatrice_posed_025, as an example, assuming the data are in their corresponding folders under data/.

Train the coarse stage

python main.py --name experiment_rp_beatrice_posed_025 --config configs/config_coarse_stage.yaml --outfit_name rp_beatrice_posed_025 --mode train

• This command will start training for a pose-independent, coarse geometry of the clothed body represented by a point set (paper Sec. 4.1, Eq. 4).
• On an RTX6000 GPU this takes 0.15 min/epoch for a ReSynth subject, and ca. 20 minutes till convergence (150 epochs).
• After training, the script will automatically evaluate it on the unsee poses of the same subject, and produce the learned coarse clothed body geometry under results/saved_samples/experiment_rp_beatrice_posed_025/test_seen/coarse_stage_256/rp_beatrice_posed_025, to be used in training the fine stage below, including:
  • <frame_name>_pred.ply: posed, coarse clothed body shape (not used, just for visualization).
  • <frame_name>_pred_cano.ply: predicted clothing on a T-posed, subject's personalized shaped body. This will be the base where the fine shape is added on the fine stage, i.e.