Nanyang Technological University in Singapore has developed an amazing AI technology - SOLAMI, which can create lifelike 3D virtual characters and achieve real-time interaction in a VR environment. This technology breaks through the rigid limitations of traditional AI characters. Virtual characters can not only understand your voice and movement instructions, but can also interact with each other in natural and smooth conversations, dance, and even boxing. This marks a major leap forward for AI technology in fields such as virtual reality, social entertainment, and education and training, bringing revolutionary changes to future interactive entertainment and learning methods.
A research team from Nanyang Technological University in Singapore recently unveiled an AI technology called SOLAMI, which can create vivid 3D virtual characters that can not only interact with you in real time in the VR world, but also understand your voice and movements to accompany you. You chat, dance and even box! This is simply a blessing for games, virtual socializing and singles!
SOLAMI is an end-to-end social vision-language-action (VLA) modeling framework that uses deep learning technology to convert the user's voice and actions into a "language" that the virtual character can understand, and generates corresponding voice and action responses . Speaking in human language means that it can translate your voice and movements into a language that AI can understand, and then let the virtual character react accordingly, just like a real person, naturally and smoothly, saying goodbye to the rigid and mechanical feeling of previous AI characters.
In order to train this AI "social master", the research team also took great pains.
They created a synthetic dataset called SynMSI, which contains massive amounts of dialogue, motion and speech data. These data are not collected casually, but are generated through careful design and processing using existing action databases and powerful language models.
What's even more amazing is that SOLAMI is also equipped with a VR interface, allowing you to interact with virtual characters immersively.
When you put on a VR device, you can see a virtual character standing in front of you, chatting and doing actions with you, as if you are in a real social scene.
The research team stated that the application prospects of SOLAMI technology are very broad and may subvert many fields such as gaming, virtual social networking, and education and training in the future.
For example, NPC characters in games can become more intelligent and interact with you more like real people; virtual avatars on virtual social platforms can be more personalized, allowing you to find like-minded friends in the virtual world; you can even create A virtual teacher appears to make learning more lively and interesting.
Of course, SOLAMI is still in the research stage, but the huge potential it shows has already excited the technology circle.
Through a series of experiments, the research team proved that SOLAMI is superior to existing methods in terms of movement quality, voice quality and response speed. More importantly, user test results show that everyone is very satisfied with the virtual characters created by SOLAMI. It seems that the era of "AI wife/husband" is really coming!
Core highlights of SOLAMI technology:
End-to-end VLA model: Directly convert the user's voice and actions into the virtual character's voice and action responses to achieve a natural and smooth interactive experience.
SynMSI synthetic data set: Utilizes existing action data sets and large language models to automatically generate a large amount of multi-round multi-modal dialogue data to solve the problem of lack of training data.
Immersive VR interface: Users can communicate face-to-face with virtual characters through VR equipment and experience a more realistic interactive experience.
Smarter and more humane: SOLAMI can create smarter and more realistic virtual characters, making the virtual interactive experience more "human".
The research team said that the application prospects of SOLAMI technology are very broad, and it can be used in many fields such as games, virtual social networking, education and training, etc. For example, in games, SOLAMI can create smarter and more realistic NPC characters to enhance players’ gaming experience; in virtual social interactions, SOLAMI can help users create more personalized avatars and enhance the immersion of virtual social interactions; In education and training, SOLAMI can create more vivid virtual teachers and improve teaching effects.
The research team also conducted a series of experiments, and the results showed that SOLAMI technology is superior to other existing methods in terms of action quality, voice quality and inference delay. User research also shows that users are very satisfied with the 3D virtual characters built with SOLAMI technology.
At present, SOLAMI technology is still in the research stage, but its future development potential is huge and it is expected to bring us a smarter and more humane virtual interactive experience.
Project homepage: https://solami-ai.github.io/
Technical report: https://arxiv.org/abs/2412.00174
Complete introduction video: https://www.bilibili.com/video/BV1D6zpYHEyc/
All in all, SOLAMI technology shows the huge potential of AI in the field of virtual interaction. Its future application prospects are worth looking forward to, and may completely change the way we entertain, socialize and learn. Let’s wait and see how this technology will shape the digital world of the future.