See3D, the latest 3D generation model released by Beijing Zhiyuan Artificial Intelligence Research Institute (BAAI), has achieved a technological breakthrough in using massive unlabeled Internet videos to generate 3D scenes. This model does not need to rely on traditional camera parameters and 3D annotations. It can generate multi-view images with controllable camera directions and consistent geometry using only visual clues in the video, greatly reducing the cost and difficulty of 3D data collection. See3D supports a variety of 3D generation methods, including text-based, single-view and sparse-view generation, and is capable of 3D editing and Gaussian rendering. Its application range covers many fields such as 3D interactive world, 3D reconstruction and open world 3D generation. Demonstrates strong application potential. The model code and demo have been open sourced to facilitate further exploration and application by researchers.
The training of the See3D model is based on a WebVi3D dataset containing 16 million video clips and 320 million frames of images. By adding time-dependent noise to the masked video data, camera-free 3D generation is achieved. Its advantages lie in data scalability, camera controllability and geometric consistency. It can generate scenes under any complex camera trajectories and maintain the geometric consistency of the previous and next frame views. See3D provides new ideas for the development of 3D generation technology, which is expected to promote the 3D research community's attention to large-scale camera-free annotation data and narrow the gap with existing closed-source 3D solutions. Project address: https://vision.baai.ac.cn/see3d
Through clever design, the See3D model solves the problem of high cost of traditional 3D data collection and provides a more convenient and efficient solution for 3D content creation. Its open source nature also encourages more researchers to participate and jointly promote the advancement of 3D generation technology. I believe that the emergence of See3D will have a profound impact on the field of 3D vision.