ComfyUI Colors MZ

Recent changes

• [2024-08-02] Support faceid, add some related nodes, see examples/workflow_ipa_faceid.png for workflow

• [2024-07-27] Added MZ_KolorsControlNetLoader node for loading the official ControlNet model

• [2024-07-26] Added MZ_ApplySDXLSamplingSettings node for V2 version to return to SDXL scheduler configuration.

• [2024-07-25] Corrected sampling_settings, parameters come from scheduler_config.json, only V2 takes effect

• [2024-07-21] Thanks to the branch from yiwangsimple for Mac repair and testing https://github.com/yiwangsimple/ComfyUI-Kolors-MZ

• [2024-07-21] Added MZ_ChatGLM3TextEncodeAdvanceV2 node

• [2024-07-18] IPA related nodes have been supported in ComfyUI_IPAdapter_plus

• [2024-07-17] Added new loaders and advanced application nodes that support IPAdapter_plus MZ_KolorsCLIPVisionLoader, MZ_IPAdapterModelLoaderKolors, MZ_IPAdapterAdvancedKolors

• [2024-07-14] Delete automatic compatibility with ControlNet and add MZ_KolorsControlNetPatch node

Implementation of Kolors on ComfyUI

Reference from https://github.com/kijai/ComfyUI-KwaiKolorsWrapper

Use ComfyUI native sampling

The workflow is obtained in examples/workflow.png

UNET model download

The unet model is placed under the models/unet/ folder

Model homepage: https://huggingface.co/Kwai-Kolors/Kolors

Download address: https://huggingface.co/Kwai-Kolors/Kolors/resolve/main/unet/diffusion_pytorch_model.fp16.safetensors

ChatGLM3 model download

chatglm3 is placed under the models/LLM/ folder

Model homepage: https://huggingface.co/Kijai/ChatGLM3-safetensors

Download address: https://huggingface.co/Kijai/ChatGLM3-safetensors/resolve/main/chatglm3-fp16.safetensors

It is recommended to use cubiq/ComfyUI_IPAdapter_plus for IPAdapter implementation

The faceid related nodes are already supported, and the IPAdapter implementation needs to update ComfyUI_IPAdapter_plus to the latest version.

IPAdapter workflow

https://github.com/cubiq/ComfyUI_IPAdapter_plus/blob/main/examples/ipadapter_kolors.json

IPAdapter_FaceIDv2 workflow

https://github.com/cubiq/ComfyUI_IPAdapter_plus/blob/main/examples/IPAdapter_FaceIDv2_Kolors.json

Official IP-Adapter-Plus model download address

Model homepage: https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-Plus

https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-Plus/resolve/main/ip_adapter_plus_general.bin Download to models/ipadapter/

https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-Plus/resolve/main/image_encoder/pytorch_model.bin Download to models/clip_vision/

Official Kolors-IP-Adapter-FaceID-Plus model download address

Model homepage: https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus

https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus/resolve/main/ipa-faceid-plus.bin Download to models/ipadapter/

https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus/resolve/main/clip-vit-large-patch14-336/pytorch_model.bin Download to models/clip_vision/

https://huggingface.co/MonsterMMORPG/tools/resolve/main/antelopev2.zip Download and extract to models/insightface/models/antelopev2/*.onnx

Official ControlNet model download address

Model homepage (Depth): https://huggingface.co/Kwai-Kolors/Kolors-ControlNet-Depth

Model homepage (Canny): https://huggingface.co/Kwai-Kolors/Kolors-ControlNet-Canny

Model homepage (Pose): https://huggingface.co/Kwai-Kolors/Kolors-ControlNet-Pose

Kolors-Inpainting model download address

Model homepage: https://huggingface.co/Kwai-Kolors/Kolors-Inpainting

https://huggingface.co/Kwai-Kolors/Kolors-Inpainting/resolve/main/unet/diffusion_pytorch_model.safetensors Download to models/unet/

Implementation of Kolors on ComfyUI

Reference from https://github.com/kijai/ComfyUI-KwaiKolorsWrapper

Using ComfyUI Native Sampling

The workflow is obtained in examples/workflow.png

UNET model download

The unet model is placed in the models/unet/ folder

Model homepage: https://huggingface.co/Kwai-Kolors/Kolors

Download link: https://huggingface.co/Kwai-Kolors/Kolors/resolve/main/unet/diffusion_pytorch_model.fp16.safetensors

ChatGLM3 model download

The chatglm3 is placed in the models/LLM/ folder

Model homepage: https://huggingface.co/Kijai/ChatGLM3-safetensors

Download link: https://huggingface.co/Kijai/ChatGLM3-safetensors/resolve/main/chatglm3-fp16.safetensors

IPAdapter implementation is recommended to use cubiq/ComfyUI_IPAdapter_plus

The faceid related nodes have been supported in it, and the IPAdapter implementation needs to update ComfyUI_IPAdapter_plus to the latest version

IPAdapter workflow

https://github.com/cubiq/ComfyUI_IPAdapter_plus/blob/main/examples/ipadapter_kolors.json

IPAdapter_FaceIDv2 workflow

https://github.com/cubiq/ComfyUI_IPAdapter_plus/blob/main/examples/IPAdapter_FaceIDv2_Kolors.json

Official IP-Adapter-Plus model download link

Model homepage: https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-Plus

https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-Plus/resolve/main/ip_adapter_plus_general.bin Download to models/ipadapter/

https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-Plus/resolve/main/image_encoder/pytorch_model.bin Download to models/clip_vision/

Official Kolors-IP-Adapter-FaceID-Plus model download link

Model homepage: https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus

https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus/resolve/main/ipa-faceid-plus.bin Download to models/ipadapter/

https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus/resolve/main/clip-vit-large-patch14-336/pytorch_model.bin Download to models/clip_vision/

https://huggingface.co/MonsterMMORPG/tools/resolve/main/antelopev2.zip Download and unzip to models/insightface/models/antelopev2/*.onnx

Official ControlNet model download link

Model homepage(Depth): https://huggingface.co/Kwai-Kolors/Kolors-ControlNet-Depth

Model homepage(Canny): https://huggingface.co/Kwai-Kolors/Kolors-ControlNet-Canny

Model homepage(Pose): https://huggingface.co/Kwai-Kolors/Kolors-ControlNet-Pose

Kolors-Inpainting model download link

Model homepage: https://huggingface.co/Kwai-Kolors/Kolors-Inpainting

https://huggingface.co/Kwai-Kolors/Kolors-Inpainting/resolve/main/unet/diffusion_pytorch_model.safetensors Download to models/unet/

Testing results with the same seed using ComfyUI-KwaiKolorsWrapper

Test workflow comes from examples/workflow_same_seed_test.png

The test workflow comes from examples/workflow_same_seed_test.png

FAQ

Error while loading model

• There are currently two versions of kolors, one is the unet type and uses the unet loader, and the other is the checkpoints folder and uses the KolorsCheckpointLoaderSimple loader. If your model comes from the unet folder of huggingface, try to use the unet loader first (Currently, there are two versions of kolors, one is unet type using unet loader, and the other is placed in the checkpoints folder using KolorsCheckpointLoaderSimple loader. If your model comes from the huggingface unet folder, try to use the unet loader first)
• If you are not sure about the model type, try both

Mac users cannot use (Mac users cannot use)

• Mac users can go to ComfyUI-Kolors-MZ (Mac users can go to ComfyUI-Kolors-MZ )

Errors related to IPAdapter

• Make sure ComfyUI ontology and ComfyUI_IPAdapter_plus are updated to the latest version

name 'round_up' is not defined

• Reference: THUDM/ChatGLM2-6B#272 (comment), use pip install cpm_kernels or pip install -U cpm_kernels to update cpm_kernels

module 'comfy.model_detection' has no attribute 'unet_prefix_from_state_dict'

• Update ComfyUI ontology to the latest version (Update ComfyUI ontology to the latest version)

RuntimeError: Only Tensors of floating point dtype can require gradients

• Try using the fp16 version of the model: https://huggingface.co/Kijai/ChatGLM3-safetensors/blob/main/chatglm3-fp16.safetensors

Error occurred when executing MZ_ChatGLM3Loader: 'ChatGLMModel' object has no attribute 'transformer'

• Check whether the model selected by the ChatGLM3Loader node has been downloaded correctly

Credits

• Kolors
• ComfyUI
• ComfyUI_IPAdapter_plus

Star History

Contact

• WeChat: minrszone
• Bilibili: minus_zone
• Little Red Book: MinusZoneAI

Stargazers

Sponsorship