AiOS

AiOS

[CVPR 2024] Official Code for "AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation

Stars: 121

Visit
 screenshot

AiOS is a tool for human pose and shape estimation, performing human localization and SMPL-X estimation in a progressive manner. It consists of body localization, body refinement, and whole-body refinement stages. Users can download datasets for evaluation, SMPL-X body models, and AiOS checkpoint. Installation involves creating a conda virtual environment, installing PyTorch, torchvision, Pytorch3D, MMCV, and other dependencies. Inference requires placing the video for inference and pretrained models in specific directories. Test results are provided for NMVE, NMJE, MVE, and MPJPE on datasets like BEDLAM and AGORA. Users can run scripts for AGORA validation, AGORA test leaderboard, and BEDLAM leaderboard. The tool acknowledges codes from MMHuman3D, ED-Pose, and SMPLer-X.

README:

AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation

[Homepage]      [arXiv]      [Code]     

method

AiOS performs human localization and SMPL-X estimation in a progressive manner. It is composed of (1) the body localization stage that predicts coarse human location; (2) the Body refinement stage that refines body features and produces face and hand locations; (3) the Whole-body Refinement stage that refines whole-body features and regress SMPL-X parameters.

Preparation

Organize them according to this datastructure:

AiOS/
├── config/
└── data
    ├── body_models
        └── smplx
    |       ├──MANO_SMPLX_vertex_ids.pkl
    |       ├──SMPL-X__FLAME_vertex_ids.npy
    |       ├──SMPLX_NEUTRAL.pkl
    |       ├──SMPLX_to_J14.pkl
    |       ├──SMPLX_NEUTRAL.npz
    |       ├──SMPLX_MALE.npz
    |       └──SMPLX_FEMALE.npz
    ├── cache
    ├── checkpoint
    │   └── aios_checkpoint.pth
    ├── datasets
    │   ├── agora
    │   └── bedlam
    └── multihuman_data
        └── agora_validation_multi_3840_1010.npz

Installtion

# Create a conda virtual environment and activate it.
conda create -n aios python=3.8 -y
conda activate aios

# Install PyTorch and torchvision.
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge

# Install Pytorch3D
git clone -b v0.6.1 https://github.com/facebookresearch/pytorch3d.git
cd pythorch3d
pip install -v -e .
cd ..

# Install MMCV, build from source
git clone -b v1.6.1 https://github.com/open-mmlab/mmcv.git
cd mmcv
export MMCV_WITH_OPS=1
export FORCE_MLU=1
pip install -v -e .
cd ..

# Install other dependencies
conda install -c conda-forge ffmpeg
pip install -r requirements.txt 

# Build deformable detr
cd models/aios/ops
python setup.py build install
cd ../../..

Inference

  • Place the mp4 video for inference under AiOS/demo/
  • Prepare the pretrained models to be used for inference under AiOS/data/checkpoint
  • Inference output will be saved in AiOS/demo/{INPUT_VIDEO}_out
cd main
sh scripts/inference.sh {INPUT_VIDEO} {OUTPUT_DIR} 

# For inferencing short_video.mp4 with output directory of demo/short_video_out
sh scripts/inference.sh short_video demo

Test

NMVE NMJE MVE MPJPE
DATASETS FB B FB B FB B F LH/RH FB B F LH/RH
BEDLAM 87.6 57.7 85.8 57.7 83.2 54.8 26.2 28.1/30.8 81.5 54.8 26.2 25.9/28.0
AGORA-Test 102.9 63.4 100.7 62.5 98.8 60.9 27.7 42.5/43.4 96.7 60.0 29.2 40.1/41.0
AGORA-Val 105.1 60.9 102.2 61.4 100.9 60.9 30.6 43.9/45.6 98.1 58.9 32.7 41.5/43.4

a. Make test_result dir

mkdir test_result

b. AGORA Validatoin

Run the following command and it will generate a 'predictions/' result folder which can evaluate with the agora evaluation tool

sh scripts/test_agora_val.sh data/checkpoint/aios_checkpoint.pth agora_val

b. AGORA Test Leaderboard

Run the following command and it will generate a 'predictions.zip' which can be submitted to AGORA Leaderborad

sh scripts/test_agora.sh data/checkpoint/aios_checkpoint.pth agora_test

c. BEDLAM

Run the following command and it will generate a 'predictions.zip' which can be submitted to BEDLAM Leaderborad

sh scripts/test_bedlam.sh data/checkpoint/aios_checkpoint.pth bedlam_test

Acknowledge

Some of the codes are based on MMHuman3D, ED-Pose and SMPLer-X.

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for AiOS

Similar Open Source Tools

For similar tasks

For similar jobs