NVIDIA has released the amazing AI audio model Fugatto, which is like a magic wand, giving users the ability to control sounds through text. Fugatto is more than just a simple audio generator. It can mix music, speech and various sounds, and understand complex text instructions to achieve unprecedented audio creation and manipulation. This model has a wide range of application prospects. From music production to advertising dubbing, from language learning to game development, Fugatto can provide powerful support, greatly improving efficiency and creativity. Its powerful multi-language and multi-accent processing capabilities, as well as precise control of sound details, demonstrate its breakthrough technical strength.
Do you still remember the scene in the science fiction movie where the protagonist waves a magic wand and can control the sound at will? Now, this magical ability is no longer a fantasy! NVIDIA’s latest AI model Fugatto is like a “sound magic wand” , allowing users to control music, sounds and voices using text alone, and create a variety of wonderful auditory effects.
Fugatto, whose full name is "Foundational Generative Audio Transformer Opus1", is an audio processing model based on generative AI technology. Unlike other AI models that can only create music or modify speech, Fugatto has the more powerful ability to generate or convert any mixture of music, speech, and sounds, and is able to understand and execute instructions entered by users through text and audio files.
Fugatto's powerful features have amazed users from all walks of life, including music producers, advertising agencies, language learning tool developers, and game developers. Music producers can use it to quickly experiment with different musical styles, vocals, and instruments, and even add effects or improve the sound quality to existing songs. Advertising companies can use it to add different accents and emotions to the dubbing of advertisements, and easily promote advertisements to different regions and target groups. Language learning tool developers can use Fugatto to convert course content into any voice the user wants, such as that of a family member or friend, to make learning more personalized. Game developers can use Fugatto to modify in-game sound materials in real time based on game progress, or create new game sound effects based on text commands and audio input.
The magic of Fugatto is its ability to understand and generate sounds just like a human. Not only can it carry out specific instructions given by the user, it can also create new sounds that have never been heard before. For example, it can make the trumpet make a dog sound, and the saxophone make a cat sound. As long as the user can describe it, Fugatto can create it.
Another groundbreaking ability of Fugatto is its ability to combine instructions learned separately during training to produce more complex effects. For example, users can ask it to generate a French-accented voice with a sad emotion. What's even more amazing is that Fugatto also allows users to make subtle adjustments to the instructions, such as controlling the thickness of the accent or the intensity of the sadness, allowing users to create like an artist.
Fugatto can also generate sounds that change over time, such as a storm approaching from a distance and thunder building in intensity before slowly fading into the distance. Users can precisely control the sound changing process and create a variety of vivid sound effects.
Fugatto is a collaborative effort between researchers from around the world, with team members from countries such as India, Brazil, China, Jordan and South Korea. Their diverse backgrounds give Fugatto greater multi-accent and multi-language capabilities.
The birth of Fugatto is the culmination of NVIDIA's years of research in the fields of speech modeling, audio coding, and audio understanding. It uses 2.5 billion parameters and is trained on a cluster of NVIDIA DGX systems equipped with 32 NVIDIA H100Tensor Core GPUs.
The emergence of Fugatto marks a new era in audio processing technology. It will bring unlimited possibilities to various fields such as music, movies, games, education, etc. Let us look forward to it creating more amazing auditory feasts!
Official blog: https://blogs.nvidia.com/blog/fugatto-gen-ai-sound-model/
The emergence of Fugatto is not only an advancement in technology, but also an infinite extension of creativity. It provides artists and developers with unprecedented creative tools and heralds the endless possibilities of future audio technology. Let's wait and see how Fugatto will change our auditory world.