Today, with the rapid development of virtual reality and augmented reality technology, how to create a realistic, natural and easy-to-operate 3D digital avatar has become a hot research topic. The ExAvatar technology jointly launched by DGIST and Meta's Codec Avatars Lab provides us with a brand new solution. It can quickly generate a lifelike 3D avatar that can imitate the user's whole body movements, facial expressions and hand postures through a simple mobile phone scan, and achieves high-quality dynamic performance and rendering effects through advanced algorithms and technologies, providing a powerful platform for the Metaverse. fields have brought new possibilities.
In the digital age, virtual reality (VR) and augmented reality (AR) technologies are gradually changing the way we live and work. Imagine being able to create a 3D avatar that mimics your every move and expression. What would it be like?
ExAvatar, jointly developed by DGIST and Meta's Codec Avatars Lab, is turning this imagination into reality. This technology captures your full body movements, facial expressions and even hand gestures through a video and converts them into a lifelike 3D digital image.
Two innovations of ExAvatar: First, it uses the SMPL-X full-body parametric mesh model to accurately capture and reproduce various postures of the human body; second, combined with 3D Gaussian sputtering technology (3DGS), it makes ExAvatar more realistic and efficient rendering capabilities.
Core functions:
Full-body 3D driver: supports comprehensive animation of the body, hands and face, generating a variety of poses and expressions.
Hybrid representation: combines 3D Gaussian and surface meshes to ensure geometric and appearance consistency and reduce artifacts.
Convenient capture: You can create a 3D avatar by scanning with your mobile phone in a short time, and the operation is simple.
High-quality rendering: Use advanced algorithms and technologies to achieve high-quality dynamic performance and rendering effects.
Better than existing technology: It surpasses previous 3D avatar generation technology in terms of natural movement and appearance, and is adaptable to a wider range of application scenarios.
ExAvatar solves some of the challenges of previous technologies, such as a lack of diversity in facial expressions and poses, and a lack of observational data such as 3D scans and depth images. By blending representations, ExAvatar improves the naturalness of animations and reduces artifacts that can appear in new poses.
Before training ExAvatar, the research team jointly registered the body, hands and face with the SMPL-X model, introduced joint offsets and facial offsets, optimized the hand bone length and facial area shape, and improved The expressiveness and naturalness of the avatar.
ExAvatar's technical architecture extracts the features of each Gaussian and processes them through a multi-layer perceptron (MLP), combined with a standard grid to form a 3D avatar that can be animated in standard space. The linear blending interpolation (LBS) algorithm is used for animation processing, and the avatar is rendered to the screen with the help of 3DGS technology, ensuring high-quality visual effects.
ExAvatar's convenience is reflected in the fact that users can create 3D avatars with a simple mobile phone scan, support animation with novel body poses, gestures and facial expressions, and render from any viewing angle. This technique uses a hybrid representation method that treats each 3D Gaussian as a vertex of the surface, with predefined connectivity between these vertices, matching the mesh topology of SMPL-X.
Project address: https://top.aibase.com/tool/exavatar
Paper address: https://arxiv.org/pdf/2407.21686
With its high precision, efficiency and ease of use, ExAvatar is expected to be widely used in fields such as virtual reality, augmented reality, game development and digital content creation, bringing users a more immersive and interactive digital experience. In the future, this technology may also play an important role in remote collaboration, virtual training, etc.