This is the official code repository for our SIG'25 paper:

Text-based Animatable 3D Avatars with Morphable Model Alignment

ACM SIGGRAPH 2025 (Conference Track)

Yiqian Wu, Malte Prinzler, Xiaogang Jin*, Siyu Tang

1. Python 3.10
2. CUDA>=12.1
3. At least 24 GB of memory
4. Linux
5. Tested on NVIDIA TITAN RTX GPU (3.5h per portrait)

Clone AnimPortrait3D to /path/to/AnimPortrait3D

git clone [email protected]:oneThousand1000/AnimPortrait3D.git

Create environment

cd AnimPortrait3D
conda env create -f environment.yaml 
source activate AnimPortrait3D

Run Install Script

bash ./install.sh

Run fetch_data.sh to download all required files. Before proceeding, ensure you have registered accounts for SMPLX, SMPL, and SMPLify, FLAME to access the necessary models.

Note: The script downloads pretrained diffusion models, ControlNet, and the Sapiens model to /path/to/pretrained_model. Please ensure that the specified path has sufficient storage space (> 50 GB).

bash ./fetch_data.sh \
    /path/to/pretrained_model \
    /path/to/AnimPortrait3D

# example:

bash ./fetch_data.sh \
    /path/to/AnimPortrait3D/pretrained_model \
    /path/to/AnimPortrait3D

AnimPortrait3D builds upon the outputs of Portrait3D as its starting point. To generate your own starting avatar, please refer to Portrait3D. Alternatively, you can download pre-generated results from our Hugging Face gallery. For optimal performance, we recommend using a Portrait3D avatar with a neutral expression and without long hair.

Our data preparation script (fetch_data.sh) also downloads an exemplar Portrait3D result to outputs/Portrait3D_results. Ensure that your Portrait3D results are organized in the following structure:

/path/to/Portrait3D_output
│
└─── {face_id}.pth [the pyramid trigrid file generated by Portrait3D]  
│
└─── {face_id}.txt [the prompt]
│
└─── {face_id}.png [preview image (optional)]
│   
└─── ...

If you encounter any issues while running this project, please first check our Q&A for possible solutions.

The following script trains a high-quality 3D avatar mesh from the Portrait3D avatar.

Please ensure you use absolute paths.

The face_id corresponds to the filename in /path/to/Portrait3D_output. Use 000 to input the exemplar data.

bash 0_mesh_optimize.sh \
    face_id \
    /path/to/Portrait3D_output \
    /path/to/pretrained_model \
    /path/to/AnimPortrait3D_output

# example:

bash 0_mesh_optimize.sh \
    000 \
    /path/to/AnimPortrait3D/Portrait3D_output \
    /path/to/AnimPortrait3D/pretrained_model \
    /path/to/AnimPortrait3D/AnimPortrait3D_output

The following script generates meshes for hair and clothing and fits an SMPL-X model to the Portrait3D avatar.

face segmentation

We use Sapiens for face segmentation. Due to the complexity of its installation, we do not include it directly in our project. Instead, please follow the instructions in Sapiens to set up a separate environment.

Next, place face_seg.sh in ./seg/scripts/demo/local/face_seg.sh under the Sapiens project root. Then, run face_seg.sh within the Sapiens environment as follows:

face_seg.sh \
    face_id \
    /path/to/AnimPortrait3D_output \
    /path/to/pretrained_sapiens

# example

face_seg.sh \
    000 \
    /path/to/AnimPortrait3D/AnimPortrait3D_output \
    /path/to/AnimPortrait3D/pretrained_model/sapiens

Note we have already downloaded sapiens models in fetch_data.sh, please specify /path/to/pretrained_sapiens as /path/to/pretrained_model/sapiens

Asset mesh segment

Please ensure you use absolute paths.

bash 1_asset_mesh_gen.sh \
    face_id \
    /path/to/Portrait3D_output \
    /path/to/pretrained_model \
    /path/to/AnimPortrait3D_output

# example

bash 1_asset_mesh_gen.sh \
    000 \
    /path/to/AnimPortrait3D/Portrait3D_output \
    /path/to/AnimPortrait3D/pretrained_model \
    /path/to/AnimPortrait3D/AnimPortrait3D_output

The following script trains a 3DGS avatar with carefully initialized geometry and appearance.

Please ensure you use absolute paths.

bash 2_avatar_initialization.sh \
    face_id \
    /path/to/Portrait3D_output \
    /path/to/pretrained_model \
    /path/to/AnimPortrait3D_output

# example

bash 2_avatar_initialization.sh \
    000 \
    /path/to/AnimPortrait3D/Portrait3D_output \
    /path/to/AnimPortrait3D/pretrained_model \
    /path/to/AnimPortrait3D/AnimPortrait3D_output

The following script trains a 3DGS avatar with dynamic expressions and poses. The training process consists of two stages: first, we pre-train the eye and mouth regions separately, and then we train the full avatar.

Please ensure you use absolute paths.

Here, the "abstract_prompt" refers to a simple description that broadly categorizes the avatar, such as "a boy", "an old man", or "a woman".

bash 3_dynamic_avatar_optimization.sh \
    face_id \
    /path/to/Portrait3D_output \
    /path/to/pretrained_model \
    /path/to/AnimPortrait3D_output \
    "abstract_prompt"

# example

bash 3_dynamic_avatar_optimization.sh \
    000 \
    /path/to/AnimPortrait3D/Portrait3D_output \
    /path/to/AnimPortrait3D/pretrained_model \
    /path/to/AnimPortrait3D/AnimPortrait3D_output \
    "a boy"

This script also outputs the path of the generated final avatar and the fitted parameters, which are used for rendering and visualization.

We provide generated results at HuggingFace, download and enjoy!

Please specify the path to the final PLY file using --points_cloud and the path to the fitted parameters using --fitted_parameters.

For input expressions and poses, we provide two exemplars at ./GSAvatar/test_motion. You can also use VHAP to reconstruct motion from in-the-wild videos, or extract them from the NerSemble dataset.

We provide some motion sequences reconstructed from VFHQ dataset using VHAP, please refer to google drive for download.

cd ./GSAvatar
python render_animation.py \
        --points_cloud=/path/to/points_cloud  \
        --fitted_parameters=/path/to/fitted_parameters  \
        --video_out_path=/path/to/output_video  \
        --exp_path=/path/to/expression_params \
        --pose_path=/path/to/poses
cd ..

cd ./GSAvatar
python local_viewer.py \
    --points_cloud=/path/to/points_cloud  \
    --fitted_parameters=/path/to/fitted_parameters
cd ..

[email protected]

If you find this project helpful to your research, please consider citing:

@inproceedings{10.1145/3721238.3730680,
author = {Wu, Yiqian and Prinzler, Malte and Jin, Xiaogang and Tang, Siyu},
title = {Text-based Animatable 3D Avatars with Morphable Model Alignment},
year = {2025},
isbn = {9798400715402},
publisher = {Association for Computing Machinery},
url = {https://doi.org/10.1145/3721238.3730680},
doi = {10.1145/3721238.3730680}, 
booktitle = {Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers},
articleno = {26},
numpages = {11},
keywords = {Animatable 3D avatar generation, Gaussian splatting, diffusion models},
series = {SIGGRAPH Conference Papers '25}
}

We want to express our thanks to those in the open-source community for their valuable contributions.

This project incorporates modified versions of the following projects:

• gaussian-splatting
• GaussianAvatars
• pytorch-hair-segmentation
• Unique3D
• LucidDreamer

Please note the following limitations:

1. The project is provided strictly for non-commercial research and evaluation purposes.
2. All original copyright, patent, trademark, and attribution notices must remain intact.

By using or distributing this project, you agree to comply with the licensing conditions specified by each of the above projects.