EasyAnimate
EasyAnimateV1
? EasyAnimate is an end-to-end solution for generating high-resolution and long videos. We can train transformer based diffusion generators, train VAEs for processing long videos, and preprocess metadata.
? We use DIT and transformer as a diffuser for video and image generation.
? Welcome!
English | 简体中文 | 日本語
EasyAnimate is a pipeline based on the transformer architecture, designed for generating AI images and videos, and for training baseline models and Lora models for Diffusion Transformer. We support direct prediction from pre-trained EasyAnimate models, allowing for the generation of videos with various resolutions, approximately 6 seconds in length, at 8fps (EasyAnimateV5, 1 to 49 frames). Additionally, users can train their own baseline and Lora models for specific style transformations.
We will support quick pull-ups from different platforms, refer to Quick Start.
New Features:
Function：
Our UI interface is as follows:
DSW has free GPU time, which can be applied once by a user and is valid for 3 months after applying.
Aliyun provide free GPU time in Freetier, get it and use in Aliyun PAI-DSW to start EasyAnimate within 5min!
Our ComfyUI is as follows, please refer to ComfyUI README for details.
If you are using docker, please make sure that the graphics card driver and CUDA environment have been installed correctly in your machine.
Then execute the following commands in this way:
# pull image
docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:easyanimate
# enter image
docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:easyanimate
# clone code
git clone https://github.com/aigc-apps/EasyAnimate.git
# enter EasyAnimate's dir
cd EasyAnimate
# download weights
mkdir models/Diffusion_Transformer
mkdir models/Motion_Module
mkdir models/Personalized_Model
# Please use the hugginface link or modelscope link to download the EasyAnimateV5 model.
# I2V models
# https://huggingface.co/alibaba-pai/EasyAnimateV5-12b-zh-InP
# https://modelscope.cn/models/PAI/EasyAnimateV5-12b-zh-InP
# T2V models
# https://huggingface.co/alibaba-pai/EasyAnimateV5-12b-zh
# https://modelscope.cn/models/PAI/EasyAnimateV5-12b-zh
We have verified EasyAnimate execution on the following environment:
The detailed of Windows:
The detailed of Linux:
We need about 60GB available on disk (for saving weights), please check!
We'd better place the weights along the specified path:
EasyAnimateV5:
? models/
├── Diffusion_Transformer/
│ ├── EasyAnimateV5-12b-zh-InP/
│ └── EasyAnimateV5-12b-zh/
├── Personalized_Model/
│ └── your trained trainformer model / your trained lora model (for UI load)
The results displayed are all based on image.
00000008.mp4 |
00000004.mp4 |
00000003.mp4 |
00000002.mp4 |
00000012.mp4 |
00000011.mp4 |
00000006.mp4 |
00000005.mp4 |
00000016.mp4 |
00000056.mp4 |
00000060.mp4 |
00000065.mp4 |
1.mp4 |
2.mp4 |
3.mp4 |
4.mp4 |
5.mp4 |
6.mp4 |
7.mp4 |
8.mp4 |
demo_pose.mp4 |
demo_scribble.mp4 |
demo_depth.mp4 |
demo_pose_out.mp4 |
demo_scribble_out.mp4 |
demo_depth_out.mp4 |
Please refer to ComfyUI README for details.
Due to the large parameters of EasyAnimateV5, we need to consider GPU memory saving schemes to conserve memory. We provide a GPU_memory_mode option for each prediction file, which can be selected from model_cpu_offload, model_cpu_offload_and_qfloat8, and sequential_cpu_offload.
model_cpu_offload indicates that the entire model will be offloaded to the CPU after use, saving some GPU memory.model_cpu_offload_and_qfloat8 indicates that the entire model will be offloaded to the CPU after use, and the transformer model is quantized to float8, saving even more GPU memory.sequential_cpu_offload means that each layer of the model will be offloaded to the CPU after use, which is slower but saves a substantial amount of GPU memory.A complete EasyAnimate training pipeline should include data preprocessing, Video VAE training, and Video DiT training. Among these, Video VAE training is optional because we have already provided a pre-trained Video VAE.
We have provided a simple demo of training the Lora model through image data, which can be found in the wiki for details.
A complete data preprocessing link for long video segmentation, cleaning, and description can refer to README in the video captions section.
If you want to train a text to image and video generation model. You need to arrange the dataset in this format.
? project/
├── datasets/
│ ├── internal_datasets/
│ ├── train/
│ │ ├── ? 00000001.mp4
│ │ ├── ? 00000002.jpg
│ │ └── ? .....
│ └── ? json_of_internal_datasets.json
The json_of_internal_datasets.json is a standard JSON file. The file_path in the json can to be set as relative path, as shown in below:
[
{
"file_path": "train/00000001.mp4",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"type": "video"
},
{
"file_path": "train/00000002.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"type": "image"
},
.....
]You can also set the path as absolute path as follow:
[
{
"file_path": "/mnt/data/videos/00000001.mp4",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"type": "video"
},
{
"file_path": "/mnt/data/train/00000001.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"type": "image"
},
.....
]Video VAE training is an optional option as we have already provided pre trained Video VAEs. If you want to train video vae, you can refer to README in the video vae section.
If the data format is relative path during data preprocessing, please set scripts/train.sh as follow.
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/json_of_internal_datasets.json"
If the data format is absolute path during data preprocessing, please set scripts/train.sh as follow.
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/json_of_internal_datasets.json"
Then, we run scripts/train.sh.
sh scripts/train.shFor details on setting some parameters, please refer to Readme Train and Readme Lora.
EasyAnimateV5:
| Name | Type | Storage Space | Hugging Face | Model Scope | Description |
|---|---|---|---|---|---|
| EasyAnimateV5-12b-zh-InP | EasyAnimateV5 | 34 GB | ?Link | ?Link | Official image-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports bilingual prediction in Chinese and English. |
| EasyAnimateV5-12b-zh-Control | EasyAnimateV5 | 34 GB | ?Link | ?Link | Official video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc. Supports video prediction at multiple resolutions (512, 768, 1024) and is trained with 49 frames at 8 frames per second. Bilingual prediction in Chinese and English is supported. |
| EasyAnimateV5-12b-zh | EasyAnimateV5 | 34 GB | ?Link | ?Link | Official text-to-video weights. Supports video prediction at multiple resolutions (512, 768, 1024), trained with 49 frames at 8 frames per second, and supports bilingual prediction in Chinese and English. |
| Name | Type | Storage Space | Url | Hugging Face | Description |
|---|---|---|---|---|---|
| EasyAnimateV4-XL-2-InP.tar.gz | EasyAnimateV4 | Before extraction: 8.9 GB / After extraction: 14.0 GB | Download | ?Link | Our official graph-generated video model is capable of predicting videos at multiple resolutions (512, 768, 1024, 1280) and has been trained on 144 frames at a rate of 24 frames per second. |
| Name | Type | Storage Space | Url | Hugging Face | Description |
|---|---|---|---|---|---|
| EasyAnimateV3-XL-2-InP-512x512.tar | EasyAnimateV3 | 18.2GB | Download | ?Link | EasyAnimateV3 official weights for 512x512 text and image to video resolution. Training with 144 frames and fps 24 |
| EasyAnimateV3-XL-2-InP-768x768.tar | EasyAnimateV3 | 18.2GB | Download | ?Link | EasyAnimateV3 official weights for 768x768 text and image to video resolution. Training with 144 frames and fps 24 |
| EasyAnimateV3-XL-2-InP-960x960.tar | EasyAnimateV3 | 18.2GB | Download | ?Link | EasyAnimateV3 official weights for 960x960 text and image to video resolution. Training with 144 frames and fps 24 |
| Name | Type | Storage Space | Url | Description |
|---|---|---|---|---|
| easyanimate_v1_mm.safetensors | Motion Module | 4.1GB | download | Training with 80 frames and fps 12 |
| Name | Type | Storage Space | Url | Description |
|---|---|---|---|---|
| PixArt-XL-2-512x512.tar | Pixart | 11.4GB | download | Pixart-Alpha official weights |
| easyanimate_portrait.safetensors | Checkpoint of Pixart | 2.3GB | download | Training with internal portrait datasets |
| easyanimate_portrait_lora.safetensors | Lora of Pixart | 654.0MB | download | Training with internal portrait datasets |
This project is licensed under the Apache License (Version 2.0).
