BodyTalk, the AI video translation tool launched by Panjaya, is revolutionizing the field of video translation. This tool can accurately convert the voices of the characters in the video into other languages, and at the same time adjust the characters' facial expressions and body movements to naturally match the speech patterns of the new language, truly achieving the effect of "fake it as real". The editor of Downcodes will give you an in-depth understanding of this amazing technology and the story behind it.
A startup called Panjaya is revolutionizing the video translation game. After three years of secret research and development, the company launched its breakthrough product BodyTalk, an AI tool that can accurately convert the voices of people in videos into other languages. Unlike traditional dubbing, BodyTalk not only perfectly replicates the speaker's original voice characteristics, but also automatically adjusts the facial expressions and body movements of the characters in the video to naturally match the speech patterns of the new language.
Panjaya was founded by Hilik Shani and Ariel Shalom, two Israeli government deep learning experts. In 2021, the two left the government sector to start their own business, and then welcomed Guy Piekarz, who has rich industry experience, as CEO. Matcha, a streaming media discovery and recommendation platform previously founded by Pickez, was acquired by Apple in 2013.
Currently, BodyTalk supports translation in 29 languages. Its workflow starts with audio translation, then generates a new voice that imitates the original speaker, and finally automatically adjusts the mouth shape and movements of the speaker in the video to match the new language expression. While processing times are approaching real-time, it currently still takes several minutes to process a video.
At the technical level, Panjaya adopts a hybrid strategy, using third-party large language models and independently developing core technologies. According to Pickez, the company's lip sync engine was completely developed by the internal AI research team because there is no solution on the market that can meet the needs of its complex business scenarios such as multiple angles and multiple speakers.
The company is currently focusing on the B2B market and has reached cooperation with institutions such as JFrog and TED. TED said that video views of lectures dubbed using Panjaya tools increased by 115% and completion rates doubled. The company plans to further expand applications in sports, education, marketing and medical fields.
To avoid misuse of the technology, Panjaya strictly controls access to tools and plans to develop features such as watermarks to identify synthesized video content. Although subtitles have become a standard feature of video content - according to a CBS survey, more than half of American viewers turn on subtitles when watching - the demand for dubbed content in the international market is still huge. Data from the research organization CSA shows that, especially in the B2B field, native language content can bring higher user engagement.
This round of US$9.5 million in financing was participated by a number of investment institutions and individuals, including Viola Ventures, R-Squared Ventures, and JFrog co-founder and CEO Shlomi Ben Haim. In the future, Panjaya plans to launch API interfaces and further improve the processing speed, moving towards the goal of real-time processing.
BodyTalk's technological breakthrough has brought new possibilities to the video translation industry, and its future development is worth looking forward to. Panjaya’s success also demonstrates the huge potential of AI technology in solving practical problems. I believe that we will see more similar innovative applications emerge in the near future. The editor of Downcodes will continue to pay attention to its subsequent development.