The research team of Alibaba Damo Academy published a paper titled "SHMT: Self-supervised Hierarchical Makeup Transfer" at the NeurIPS 2024 conference. The research proposed a novel makeup effect transfer technology. This technology uses Latent Diffusion Models to accurately generate makeup images, and has great application prospects in the fields of makeup applications and image processing. The SHMT model only needs a makeup reference picture and a photo of the target person to transfer the makeup effects to the target face, greatly simplifying the editing and application process of makeup effects. The team has open sourced the training code, test code and pre-trained model to facilitate further research and development by researchers.
Recently, the research team of Alibaba Damo Academy released an important research result called "SHMT: Self-supervised Hierarchical Makeup Transfer". The paper has been accepted by the top international academic conference NeurIPS2024. This research demonstrates a new makeup effect transfer technology that uses latent diffusion models (Latent Diffusion Models) to achieve accurate generation of makeup images, injecting new vitality into the fields of makeup application and image processing.
Simply put, SHMT is a makeup transfer technology. As long as a makeup reference picture and a photo of the target character are used, the makeup effect can be transferred to the target face.
The team adopted an open source approach in the project and released training code, test code and pre-training models, making it easier for researchers to conduct related research and development.
During the model building process, the team recommends users to create a conda environment named "ldm" and quickly complete the setup through the provided environment file. In addition, VQ-f4 was selected as a pre-trained autoencoding model in the study. Users need to download it and put it into the specified checkpoint folder in order to start inference smoothly.
Data preparation is key to the successful operation of the SHMT model. The research team recommends downloading the makeup transfer dataset provided by “BeautyGAN” and integrating different makeup and non-makeup images. At the same time, the preparation of facial parsing and 3D facial data is also crucial, and relevant tools and data paths are detailed in the study to ensure that users can effectively prepare data.
In terms of model training and inference, the research team provides detailed command line scripts so that users can adjust parameters according to their own needs. The team also particularly emphasized the importance of data structure, providing clear directory structure examples to guide users on how to prepare data.
The launch of the SHMT model marks the successful application of self-supervised learning in the field of makeup effect transfer, and may be widely used in beauty, cosmetics, image processing and other industries in the future. This research not only demonstrates the potential of the technology, but also lays a solid foundation for in-depth research in related fields.
Project entrance: https://github.com/Snowfallingplum/SHMT
Highlights:
1. The SHMT model uses the latent diffusion model to achieve makeup effect transfer, and has been accepted by NeurIPS2024.
2. The team provides complete open source code and pre-trained models to facilitate researchers’ application and improvement.
3. Data preparation and parameter adjustment are crucial, and the study provides detailed guidance on the operation process and directory structure.
All in all, the open source release of the SHMT model provides powerful tools and resources for makeup effect migration research, and its application prospects in the fields of beauty, cosmetics, and image processing are worth looking forward to. The innovativeness and practicality of this research make it an important breakthrough in the field and lay a solid foundation for future related research.