https://github.com/jerenchen/simple-diffusion-pose-gen

A simple Python/Godot example of an AI prompt-based 3D human pose generator
https://github.com/jerenchen/simple-diffusion-pose-gen

3d-graphics ai-prompts character-animation godot-engine human-pose-estimation inverse-kinematics python stable-diffusion

Last synced: 4 months ago
JSON representation

A simple Python/Godot example of an AI prompt-based 3D human pose generator

• Host: GitHub
• URL: https://github.com/jerenchen/simple-diffusion-pose-gen
• Owner: jerenchen
• License: bsd-3-clause
• Created: 2024-10-18T06:02:12.000Z (12 months ago)
• Default Branch: main
• Last Pushed: 2025-06-26T04:10:10.000Z (4 months ago)
• Last Synced: 2025-06-26T05:19:45.832Z (4 months ago)
• Topics: 3d-graphics, ai-prompts, character-animation, godot-engine, human-pose-estimation, inverse-kinematics, python, stable-diffusion
• Language: Python
• Homepage:
• Size: 1.46 MB
• Stars: 5
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Simple Diffusion Pose Gen

A quick attempt to utilize [Stable Diffusion](https://huggingface.co/blog/stable_diffusion) (SD) image generative models (v1.5 & XL) and MediaPipe's [Pose Landmarker](https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker) vision models to generate 3D human poses from an AI prompt.

## News

* 2025-06-22: Minor updates on docs and dependencies.

* 2024-11-12: Added a Godot full-body IK pose solver.

* 2024-10-24: Initial implementation.

## Installation

1. Clone repo: `git clone https://github.com/jerenchen/simple-diffusion-pose-gen.git`.

2. Change dir into `python` and (optionally) use a virtual enivronment (e.g. conda).

3. Install Python depedencies: `pip install -r requirements.txt`.

4. Download MediaPipe [Pose Landmarker (Full)](https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_full/float16/latest/pose_landmarker_full.task) and save the file under `python/tasks`.

## Python Pose Gen + Godot IK Solver

1. Inside dir `python`, Initialize the Python pose-gen service: `python posegen.py --base sd15 --steps 8`.

2. Open and run `projec.godot` inside dir `godot` with [Godot Engine](https://godotengine.org/).

3. Enter a prompot to generate a pose.

| ![SD Godot Pose Gen](img/sd_godot_pose_gen.png) |

| :---: |

| Generating a pose using prompt *"A basketball player making a 3-pointer jump shot"* |

## Standalone Python Pose Gen

> NOTE: The standalone demo requires PySide6 >= v6.7.

Inside dir `python`, run: `python simple-diffusion-pose-gen.py`, and then enter a prompt to generate a pose.

| ![SD Settings](img/sd_settings.png) |

| :---: |

| ![SD Pose Gen](img/sd_pose_gen.png) |

## Stable Diffusion Model Settings

* *SD15* has fewer model parameters and therefore requires less memory whereas *SDXL* could generate images with higher fidelity.

* [Hyper-SD](https://hyper-sd.github.io/) (a SD inference acceleration technique) Steps: *2*, *4*, or *8*, trade-off between speed (fewer bigger steps) and quality (more smaller steps).

* PyTorch device for running SD inference (if available): *CPU*, *CUDA*, or *MPS* (Apple Silicon).

## Acknowledgement

* [Godot + Python Communication with UDP](https://github.com/trflorian/godot-python-comm) by [Florian Trautweiler](https://github.com/trflorian).

* [Open 3D Mannequine](https://github.com/gdquest-demos/godot-3d-mannequin) by [GDQuest](https://github.com/gdquest-demos).