The open-source ControlNeXt image/video generation guidance tool by the Hong Kong Chinese Jiajiaya team can be called a "weight loss artifact" in the AI industry. It is small in size, perfectly compatible with the Stable Diffusion series models, and is plug-and-play, greatly simplifying the operation process. ControlNeXt supports a variety of control modes, such as edge guidance, posture control, masking and depth of field control. Its powerful control capabilities can even accurately control Iron Man's dance movements. What's even more amazing is that its training speed and generation speed are far faster than ControlNet, and its efficiency is greatly improved.
ControlNeXt supports a variety of control modes, including edge guidance, posture control, masking and depth of field control. It can even let Iron Man dance a beautiful dance with precise movements down to his fingers, demonstrating his powerful control capabilities.
The secret of ControlNeXt's weight loss is that it cleverly removes the big eater control branch in ControlNet and replaces it with a light meal package composed of a small number of ResNet blocks. Although this small and exquisite module is only one-tenth of its original size, it can perfectly extract features of various control conditions.
Not only that, ControlNeXt is also a learning genius. It only takes 400 steps to learn a new skill, while ControlNet takes thousands of steps. In terms of generation speed, ControlNeXt is even better, bringing only 10.4% delay, while ControlNet requires a full 41.9%.
Another unique skill of ControlNeXt is cross normalization. This technology is like holding a party for the features, so that their data distributions are as aligned as possible, thus avoiding the sensitivity of parameter initialization and allowing control conditions to take effect at the beginning of training.
ControlNeXt is like a Transformer in the AI world, small and flexible yet powerful. It not only allows the two-dimensional girl to perfectly fit the control lines, but also creates cross-dimensional characters with different styles. With this artifact, I believe we will be able to see more amazing AI artworks soon!
Project homepage: https://pbihao.github.io/projects/controlnext/index.html
ControlNeXt brings new possibilities to the field of AI image/video generation with its efficient, lightweight and powerful control capabilities. Its open source features also facilitate developers to conduct more in-depth research and applications. I believe there will be more innovative applications based on ControlNeXt in the future. We look forward to ControlNeXt shining in the field of AI art creation!