Tencent AI Lab and Tencent PCG's ARC Lab jointly released a new framework called StereoCrafter. This technology can convert ordinary 2D videos into high-fidelity stereoscopic 3D videos, bringing revolutionary changes to the immersive experience. StereoCrafter uses deep learning technology to overcome the limitations of traditional 3D video conversion methods, significantly improve the generation effect, and can adapt to the high-fidelity requirements of various display devices to meet the growing demand for 3D content.
Recently, Tencent AI Lab and Tencent PCG's ARC Lab jointly launched a new framework called StereoCrafter, which can convert ordinary 2D videos into high-fidelity stereoscopic 3D videos.
This innovation responds to the growing demand for 3D content, especially in the field of immersive experiences. StereoCrafter makes full use of the advantages of the basic model, overcomes the limitations of traditional conversion methods, significantly improves the generation effect, and ensures that the generated content can meet the high-fidelity requirements of various display devices.
The core of the system is divided into two main steps. The first step is to remap the video based on depth information, extract occlusion information and perform video transformation at the same time; the second step is to repair the stereoscopic video. The system uses a pre-trained stable video diffusion model as the basis and introduces a fine-tuning protocol for the stereoscopic video inpainting task. In order to handle video inputs of different lengths and resolutions, the team also explored autoregressive strategies and slicing processing techniques to ensure that the system can flexibly adapt to various input conditions.
To support training, the team built a sophisticated data processing pipeline that generated large-scale, high-quality datasets. During the data set construction process, the research team selected from a large number of stereoscopic videos and generated corresponding video depth, transformed video and occlusion information to ensure that the video on the right serves as a real benchmark. These innovative methods provide practical solutions for converting 2D videos into 3D videos, allowing Apple Vision Pro and other 3D display devices to present a more exciting immersive experience.
StereoCrafter is not only a technological breakthrough, it also brings potential changes to the way digital media is experienced, potentially changing the way we watch and experience digital content.
Project entrance: https://stereocrafter.github.io/
Highlights:
StereoCrafter uses new technology to efficiently convert 2D videos into immersive stereoscopic 3D videos.
The system is divided into two main steps: depth video reconstruction and stereoscopic video repair to improve the generation effect.
The research team constructed high-quality data sets to support algorithm training and ensure output quality.
The emergence of StereoCrafter marks a major leap in 2D to 3D video conversion technology. Its efficient conversion efficiency and high-fidelity output quality will greatly enrich the creation and consumption of 3D content, bringing users more immersive audio-visual experience. experience. In the future, this technology is expected to be widely used in movies, games, virtual reality and other fields.