The editor of Downcodes will take you to understand the latest ReCapture technology released by Google Research! This amazing technology reinterprets your videos from a whole new perspective, giving you an unprecedented viewing experience. It is not a simple video editing, but uses AI technology to generate a new version with customized camera trajectories based on the video you provide, making you seem to have the ability to shoot from multiple cameras, even for daily videos shot with your mobile phone. It can also have a movie-like effect. Are you ready? Let's explore the magic of ReCapture together!
Google Research recently launched a new technology called ReCapture, which allows you to re-experience your own videos from a new perspective. ReCapture technology can generate a new version with customized camera tracks based on the video provided by the user, which means you can watch the video content from a perspective not found in the original video, and still maintain the original movement of the characters and scenes in the video.
ReCapture is like a magical editor that can generate a new version with a fresh perspective based on the video you provide. For example, if you take a video of a dog playing with your mobile phone, ReCapture can help you generate a video taken from the dog's perspective. Isn't it amazing?
So, how does ReCapture achieve this "magic"? In fact, the principle behind it is not complicated. It first uses a multi-view diffusion model or point cloud rendering technology to generate a rough video based on the new perspective you want. This rough video is like an unpolished jade, the picture may be incomplete, the time may be inconsistent, and it may be swaying like a drunk.
Next, ReCapture will use its secret weapon - "mask video fine-tuning" technology to "finely polish" this rough video. This technology is like a skilled craftsman who uses two special tools - spatial LoRA and temporal LoRA to repair and optimize the video. Spatial LoRA is like a "beautician", responsible for learning the characters and scene information in the original video to make the picture clearer and more beautiful. Time LoRA is a "rhythm master" responsible for learning scene movements from new perspectives to make video playback smoother and more natural.
Through the joint efforts of these two "masters", the rough video was transformed into a clear, coherent and dynamic new video. Not only that, in order to make the video effect more perfect, ReCapture will also use SDEdit technology to add final touches to the video, just like makeup, making the video more refined and delicate.
Google researchers say ReCapture can handle various types of videos and perspective transitions without requiring large amounts of training data. This means that even if you are just an ordinary video enthusiast, you can easily create professional-grade "multi-camera" videos with ReCapture.
Project address: https://generative-video-camera-controls.github.io/
The emergence of ReCapture technology has undoubtedly brought new possibilities to the field of video creation. It simplifies the multi-view video production process, allowing more people to easily experience the joy of creation. I believe that in the future, ReCapture will be applied in more fields and bring us more surprises!