Unduhan awesome 3D gaussian splatting - unduhan kode sumber awesome 3D gaussian splatting

Sumber Daya Percikan Gaussian 3D yang Luar Biasa

Daftar makalah dan sumber daya sumber terbuka yang dikurasi dan berfokus pada 3D Gaussian Splatting, dimaksudkan untuk mengimbangi lonjakan penelitian yang diantisipasi dalam beberapa bulan mendatang. Jika Anda memiliki tambahan atau saran, silakan berkontribusi. Sumber daya tambahan seperti postingan blog, video, dll. juga diterima.

Daftar isi

Makalah Seminal memperkenalkan 3D Gaussian Splatting

Deteksi Objek 3D
Mengemudi Otonom
Avatar
Pekerjaan klasik
Kompresi
Difusi
Dinamika dan Deformasi
Mengedit
Penyematan Bahasa
Ekstraksi Mesh dan Fisika
Lain-lain
Regularisasi dan Optimasi
Rendering
Ulasan
MEMBANTU
Jarang
Navigasi dan Mengemudi Otonom
Pose
Skala Besar

Data
Kursus

Implementasi Sumber Terbuka
- Referensi
- Implementasi Tidak Resmi
- Percikan Gaussian 2D
- Mesin Permainan
- Pemirsa
- Utilitas
- tutorial
- Kerangka
- Lainnya

Postingan Blog
Video Tutorial
Kredit

Log Pembaruan:

24 Oktober 2024

Menambahkan 2 makalah: IGS, V^3

16 Oktober 2024

Menambahkan satu makalah:DGD

07 September 2024

Menambahkan satu makalah:MoDGS

10 Mei 2024

Menambahkan 18 makalah: Z-Splat, Dual-Camera, StylizedGS, Hash3D, Revisiting Densification, Gaussian Pancakes, 3D-aware Deformable Gaussians, SpikeNVS, penyelesaian PC Zero-shot, SplatPose, DreamScene360, RealmDreamer, Gaussian-ILC, Pembelajaran Penguatan dengan GGS , GoMAvatar, OccGaussian, LoopGaussian, Ulasan

11 April 2024

Pelepasan kode latentSplat

9 April 2024

Menambahkan 1 makalah: EgoLifter

8 April 2024

Menambahkan 3 makalah: Robust Gaussian Splatting, SC4D, dan MM-Gaussian

5 April 2024

Menambahkan 5 makalah: Rekonstruksi Permukaan, TCLC-GS, GaSpCT, OmniGS, dan Per-Gaussian Embedding,
Perbaikan

2 April 2024

Menambahkan 11 makalah: HO, SGD, HGS, Snap-it, InstantSplat, 3DGSR, MM3DGS, HAHA, CityGaussain, Mirror-3DGS, dan Feature Splatting

30 Maret 2024

Menambahkan 8 makalah: Pemodelan ketidakpastian, GRM, Gamba, CoherentGS, TOGS, SA-GS, dan GaussianCube

27 Maret 2024

Menambahkan Implementasi Lainnya: 360-gaussian-splatting
Label CVPR '24 ditambahkan
Menambahkan 5 makalah: Comp4D, DreamPolisher, DN-Splatter, 2D GS, dan Octree-GS

26 Maret 2024

Menambahkan 13 makalah: latentSplat, GS on the Move, RadSplat, Mini-Splatting, SyncTweedies, HAC, STAG4D, EndoGSLAM, Pixel-GS, Semantic Gaussians, Gaussian in the Wild, CG-SLAM, dan GSDF

24 Maret 2024 :

Kertas tambahan: Gaussian Frosting

20 Maret 2024 :

Menambahkan 4 makalah: GVGEN, HUGS, RGBD GS-ICP SLAM, dan High-Fidelity SLAM

19 Maret 2024 :

Menambahkan Pointrix
Menambahkan tutorial 3DGS oleh penulis asli
Menambahkan GauStudio
Menambahkan 23 makalah: Touch-GS, GGRt, FDGaussian, SWAG, Den-SOFT, Gaussian-Flow, Pengeditan 3D Konsisten Tampilan, BAGS, GeoGaussian, GS-Pose, Analytic-Splatting, Peta 3D Mulus, Tekstur-GS, Kemajuan Terkini dalam 3DGS, 3DGS Ringkas untuk SLAM Visual Padat, BrightDreamer, 3DGS-Reloc, Melampaui Ketidakpastian, 3DGS Sadar Gerakan, Fed3DGS, GaussNav, 3DGS-Calib, dan NEDS-SLAM

17 Maret 2024 :

Perbarui nama repo dan tautan untuk 3DGS.cpp (awalnya VulkanSplatting)

16 Maret 2024 :

PercikanTV
Menambahkan 6 makalah: GaussianGrasper, algoritma pemisahan baru, Pembuatan Teks-ke-3D Terkendali, 3DGS Spring-Mass, Hyper-3DGS, dan DreamScene

14 Maret 2024 :

Menambahkan 6 makalah: SemGauss, StyleGaussian, Gaussian Splatting in Style, GaussCtrl, GaussianImage, dan RAIN-GS

8 Maret 2024 :

Tutorial: Cara mengambil gambar untuk 3DGS
Menambahkan 6 makalah: SplattingAvatar, DNGaussian, Radiative Gaussians, BAGS, GSEdit, dan ManiGaussian

8 Maret 2024 :

Menambahkan Penampil 3DGStream

6 Maret 2024 :

1 makalah ditambahkan: Splat-Nav

5 Maret 2024 :

1 makalah ditambahkan: 3DGStream
Rilis kode
Penampil baru ditambahkan

2 Maret 2024 :

1 makalah ditambahkan: Model Gaussian 3D untuk Animasi dan Tekstur
Bagian baru: Kursus yang juga mengajarkan 3DGS.

28 Februari 2024 :

Gaussian Luas

27 Februari 2024 :

2 makalah ditambahkan: Spec-Gaussian dan GEA
Kode SC-GS dirilis

24 Februari 2024 :

2 makalah ditambahkan: Mengidentifikasi Gaussians dan Gaussian Pro yang tidak perlu

23 Februari 2024 :

Penulis yang Dikoreksi dan abstrak yang diperbarui untuk EndoGS: Rekonstruksi Jaringan Endoskopi yang Dapat Diubah Bentuk dengan Gaussian Splatting

21 Februari 2024 :

Menambahkan satu makalah: Membentuk Kembali SLAM: Survei

20 Februari 2024 :

Kode GaussianObject dirilis
Menambahkan satu makalah: GaussianHair

19 Februari 2024 :

Entri blog ditambahkan: NeRFs vs. 3DGS.

16 Februari 2024 :

2 makalah ditambahkan: IM-3D dan GES
Kode GameS dirilis

14 Februari 2024 :

Penampil yang ditambahkan: VulkanSplatting - perender 3DGS lintas platform dan berkinerja tinggi dalam C++ dan Vulkan Compute

13 Februari 2024 :

Rilis kode: (16 Jan 2024) Representasi dan Rendering Pemandangan Dinamis Fotorealistik Real-time dengan 4D Gaussian Splatting
3 makalah ditambahkan: 3DGala, ImplicitDeepFake, dan 3D Gaussians sebagai Era Visi Baru.

9 Februari 2024 :

1 makalah ditambahkan: HeadStudio

8 Februari 2024 :

3 makalah ditambahkan: Rig3DGS, GS berbasis Mesh, dan LGM 6 Februari 2024 :
Menambahkan 2 makalah: SGS-SLAM dan 4D Gaussian Splatting

5 Februari 2024 :

Memindahkan SWAGS ke bagian Dinamika dan Deformasi
Menambahkan 2 makalah: GaussianObject dan GaMeSh
GS++ berganti nama menjadi Proyeksi Optimal

2 Februari 2024 :

Menambahkan 6 makalah: VR-GS, Segment Anything, Gaussian Splashing, GS++, 360-GS, dan StopThePop
Rilis kode TRIPS

30 Januari 2024 :

Perubahan kode: Kode GaussianAvatars diubah menjadi pribadi

29 Januari 2024 :

Menambahkan 2 makalah: LIV-GaussMap dan TIP-Editor

26 Januari 2024 :

Kertas yang ditarik kembali dihapus: Gaussians 3D yang Dapat Dianimasikan untuk Sintesis Gerakan Manusia dengan Ketelitian Tinggi
3 makalah ditambahkan: EndoGaussians, PSAvatar, dan GauU-Scene

25 Januari 2024 :

Penampil yang ditambahkan: Splatapult - penyaji percikan gaussian 3d di C++ dan OpenGL, bekerja dengan OpenXR untuk VR yang ditambatkan

24 Januari 2024 :

Utilitas tambahan: GSOP (Gaussian Splat Operator) untuk SideFX Houdini
Rilis kode: GaussianAvatars

23 Januari 2024 :

3 makalah ditambahkan: Gen3D yang Diamortisasi, Jaringan Endoskopi yang Dapat Diubah Bentuk, Pembuatan Objek 3D dinamis yang cepat
Rilis kode: Avatar Animasi, Gaussians 3D Terkompresi, GaussianAvatar

13 Januari 2024 :

4 makalah ditambahkan: CoSSegGaussians, TRIPS, Gaussian Shadow Casting untuk Karakter Neural dan DISTWAR

9 Januari 2024 :

1 makalah ditambahkan: Survei tentang 3D Gaussian Splatting (Survei pertama)

8 Januari 2024 :

4 makalah ditambahkan: SWAGS (makalah tambahan tahun 2023 yang lupa saya tambahkan sebelumnya, ), makalah review pertama, 3DGS terkompresi, dan makalah aplikasi Karakterisasi Geometri Satelit.

7 Januari 2024 :

1 Implementasi sumber terbuka: taichi-splatting - karya awalnya berasal dari Taichi 3D Gaussian Splatting, dengan pengaturan ulang dan perubahan yang signifikan.

5 Januari 2024 :

3 makalah ditambahkan: FMGS, PEGASUS, dan Repaint123.

2 Januari 2024 :

1 makalah ditambahkan: Street Gaussians.

2 Januari 2024 :

Tautan kertas Gaussian yang menghilangkan blur diperbarui.
Kode SAGA dirilis.
2 makalah dari tahun 2023 ditambahkan: Text2Immersion dan Segmentasi 3DG Terpandu 2D.
Suplemen matematika dari gsplat lib.
Tambahkan tahun dalam kategori.
Kode GSM dirilis.

29 Desember 2023 :

1 makalah ditambahkan (tampaknya melewatkan yang sebelumnya): Gaussian-Head-Avatar.
Avatar kepala postingan blog ditambahkan.

29 Desember 2023 :

3 makalah ditambahkan: DreamGaussian4D, 4DGen, dan Spacetime Gaussian.

27 Desember 2023 :

3 makalah ditambahkan: LangSplat, Deformable 3DGS, dan Human101.
Entri blog ditambahkan: Tinjauan Komprehensif 3DGS.

25 Desember 2023 :

Representasi Gaussian 3D yang Efisien untuk kode Pemandangan Dinamis Bermata/Multi-tampilan dirilis.
Kode GPS-Gaussian dirilis.

24 Desember 2023 :

2 makalah ditambahkan: Self-Organization Gaussian Grids dan Gaussian Splitting.
Menambahkan repo untuk meningkatkan rendering Gaussian untuk memodelkan adegan yang lebih kompleks.

21 Desember 2023 :

3 makalah ditambahkan: Splatter Image, pixelSplat, dan sejajarkan gaussians Anda.
Kode Pengelompokan Gaussian dirilis.

19 Desember 2023 :

2 makalah ditambahkan: GAvatar dan GauFRe.

18 Desember 2023 :

Utilitas tambahan: SpectacularAI - Skrip konversi untuk konvensi 3DGS yang berbeda.
Kode SuGaR dirilis.

16 Desember 2023 :

Menambahkan penampil WebGL 3: Gauzilla.

15 Desember 2023 :

4 makalah ditambahkan: DrivingGaussian, iComMa, Triplane, dan 3DGS-Avatar.
Kode Gaussian yang dapat dihidupkan kembali dirilis.

13 Desember 2023 :

5 makalah ditambahkan: Gaussian-SLAM, CoGS, ASH, CF-GS, dan Photo-SLAM.

11 Desember 2023 :

2 makalah ditambahkan: Gaussian Splatting SLAM dan Denoising Scores untuk Generasi 3D.
Kode ScaffoldGS dirilis.

8 Desember 2023 :

2 makalah ditambahkan: EAGLES dan MonoGaussianAvatar.

7 Desember 2023 :

Kode LucidDreamer dirilis.
9 makalah ditambahkan: GauHuman, HeadGaS, HiFi4G, Gaussian-Flow, Feature-3DGS, Gaussian-Avatar, FlashAvatar, Relightable, dan Deblurring Gaussians.

5 Desember 2023 :

9 makalah ditambahkan: NeuSG, GaussianHead, GaussianAvatars, GPS-Gaussian, Neural Parametric Gaussians untuk Rekonstruksi Objek Non-Kaku Bermata, SplTAM, MANUS, Segment Any, dan Language embedded 3D Gaussians.

4 Desember 2023 :

8 makalah ditambahkan: Gaussian Grouping, MD Splatting, DynMF, Scaffold-GS, SparseGS, FSGS, Control4D, dan SC-GS.

1 Desember 2023 :

4 makalah ditambahkan: Compact3D, GaussianShader, Gaussian Getaran Berkala, dan Gaussian Shell Maps untuk Generasi Manusia 3D yang Efisien.
Membuat Daftar Isi untuk setiap kategori dan menambahkan jeda baris.

30 November 2023 :

Menambahkan implementasi mesin game Unreal.
5 makalah ditambahkan: LightGaussian, FisherRF, HUGS, HumanGaussian, CG3D, dan Multi Scale 3DGS.

29 November 2023 :

Menambahkan dua makalah: Point and Move dan IR-GS.

28 November 2023 :

Menambahkan lima makalah: GaussinEditor, Relightable Gaussians, GART, Mip-Splatting, HumanGaussian.

27 November 2023 :

Menambahkan dua makalah: Gaussian Editing dan Compact 3D Gaussians.

25 November 2023 :

Proyek Gaussians yang dapat dianimasikan ditambahkan (makalah belum dirilis).

22 November 2023 :

3 makalah GS baru ditambahkan: Animatable, Depth-Regularized, dan Monocular/Multi-view 3DGS.
Menambahkan beberapa makalah klasik.
Menambahkan kertas GS lain yang juga disebut LucidDreamer.

21 November 2023 :

3 makalah GS baru ditambahkan: GaussianDiffusion, LucidDreamer, PhysGaussian.
2 makalah GS lainnya ditambahkan: SuGaR, PhysGaussian.

21 November 2023 :

Menambahkan kertas GS-SLAM

17 November 2023 :

Menambahkan implementasi PlayCanvas ke bagian Game Engine.

16 November 2023 :

Kode Gaussians 3D yang dapat dideformasi dirilis.
Kertas Avatar Gaussian 3D yang dapat dikendarai ditambahkan.

8 November 2023 :

Beberapa catatan mengenai implementasi 3DGS dan pembahasan format unsive/rsal.

4 November 2023 :

Menambahkan percikan gaussian 2D.
Menambahkan postingan blog (teknis) yang sangat detail yang menjelaskan percikan gaussian 3D.

28 Oktober 2023 :

Bagian Utilitas Ditambahkan.
Menambahkan Konverter 3DGS untuk mengedit file .ply 3DGS di Cloud Bandingkan dengan Utilitas.
Menambahkan Kapture (untuk konversi model bundler ke colmap) dan skrip pemangkas gambar Kapture dengan instruksi konversi ke Utilitas.

23 Oktober 2023 :

Menambahkan penampil python WebGL 2.
Menambahkan Intro ke blog video gaussian splatting (dan Unity viewer).

21 Oktober 2023 :

Menambahkan penampil python OpenGL.
Menambahkan penampil WebGPU skrip ketikan.

20 Oktober 2023 :

Membuat abstrak dapat dibaca (menghilangkan tanda hubung).
Menambahkan tutorial Windows.
Perbaikan teks kecil lainnya.
Menambahkan penampil buku catatan Jupyter.

19 Oktober 2023 :

Menambahkan tautan halaman Github untuk Representasi Pemandangan Dinamis Fotorealistik Waktu Nyata.
Judul yang disusun ulang.
Menambahkan implementasi tidak resmi lainnya.
Memindahkan Nerfstudio gsplat dan cepat: C++/CUDA ke Implementasi Tidak Resmi.
Menambahkan penampil Nerfstudio, Blender, WebRTC, iOS & Metal.

17 Oktober 2023 :

Kode GaussianDreamer dirilis.
Menambahkan Representasi Pemandangan Dinamis Fotorealistik Waktu Nyata.

16 Oktober 2023 :

Menambahkan kertas Gaussians 3D yang Dapat Dideformasi.
Kode Gaussians 3D dinamis dirilis. 15 Oktober 2023 : Daftar awal dengan 6 makalah pertama.

Makalah Seminal yang memperkenalkan 3D Gaussian Splatting:

Percikan Gaussian 3D untuk Rendering Bidang Cahaya Waktu Nyata

Penulis : Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis

Abstrak

Metode Radiance Field baru-baru ini merevolusi sintesis pemandangan baru yang diambil dengan banyak foto atau video. Namun, untuk mencapai kualitas visual yang tinggi masih memerlukan jaringan saraf yang mahal untuk dilatih dan dirender, sementara metode yang lebih cepat saat ini pasti mengorbankan kecepatan demi kualitas. Untuk adegan tak terbatas dan lengkap (bukan objek terisolasi) dan rendering resolusi 1080p, tidak ada metode saat ini yang dapat mencapai kecepatan tampilan real-time. Kami memperkenalkan tiga elemen kunci yang memungkinkan kami mencapai kualitas visual tercanggih sambil mempertahankan waktu pelatihan yang kompetitif dan yang terpenting memungkinkan sintesis tampilan novel real-time (≥ 30 fps) berkualitas tinggi pada resolusi 1080p. Pertama, mulai dari titik jarang yang dihasilkan selama kalibrasi kamera, kami merepresentasikan pemandangan dengan Gaussians 3D yang mempertahankan properti bidang pancaran volumetrik kontinu yang diinginkan untuk optimalisasi pemandangan sekaligus menghindari komputasi yang tidak perlu di ruang kosong; Kedua, kami melakukan optimasi interleaved/kontrol kepadatan Gaussians 3D, terutama mengoptimalkan kovarians anisotropik untuk mencapai representasi pemandangan yang akurat; Ketiga, kami mengembangkan algoritme rendering yang sadar visibilitas dan cepat yang mendukung percikan anisotropik dan mempercepat pelatihan serta memungkinkan rendering waktu nyata. Kami mendemonstrasikan kualitas visual tercanggih dan rendering real-time pada beberapa kumpulan data yang sudah ada.

Deteksi Objek 3D

2024

1. 3DGS-DET: Memberdayakan 3D Gaussian Splatting dengan Panduan Batas dan Pengambilan Sampel Berfokus Kotak untuk Deteksi Objek 3D

Penulis : Yang Cao, Yuanliang Jv, Dan Xu

Abstrak

Neural Radiance Fields (NeRF) banyak digunakan untuk sintesis tampilan baru dan telah diadaptasi untuk Deteksi Objek 3D (3DOD), menawarkan pendekatan yang menjanjikan untuk deteksi objek 3D melalui representasi sintesis tampilan. Namun, NeRF menghadapi keterbatasan yang melekat: (i) NeRF memiliki kapasitas representasional yang terbatas untuk 3DOD karena sifatnya yang implisit, dan (ii) kecepatan renderingnya lambat. Baru-baru ini, 3D Gaussian Splatting (3DGS) telah muncul sebagai representasi 3D eksplisit yang mengatasi keterbatasan ini dengan kemampuan rendering yang lebih cepat. Terinspirasi oleh keunggulan ini, makalah ini memperkenalkan 3DGS ke dalam 3DOD untuk pertama kalinya, dan mengidentifikasi dua tantangan utama: (i) Distribusi spasial yang ambigu dari blob Gaussian – 3DGS terutama mengandalkan pengawasan tingkat piksel 2D, sehingga menghasilkan distribusi spasial 3D yang tidak jelas dari blob Gaussian dan diferensiasi yang buruk antara objek dan latar belakang, sehingga menghambat 3DOD; (ii) Gumpalan latar belakang yang berlebihan – Gambar 2D sering kali menyertakan banyak piksel latar belakang, sehingga menghasilkan 3DGS yang direkonstruksi secara padat dengan banyak gumpalan Gaussian berisik yang mewakili latar belakang, sehingga berdampak negatif pada deteksi. Untuk mengatasi tantangan (i), kami memanfaatkan fakta bahwa rekonstruksi 3DGS berasal dari gambar 2D, dan mengusulkan solusi yang elegan dan efisien dengan menggabungkan Panduan Batas 2D untuk secara signifikan meningkatkan distribusi spasial gumpalan Gaussian, sehingga menghasilkan diferensiasi yang lebih jelas antara objek dan latar belakang mereka (lihat Gambar 1). Untuk mengatasi tantangan (ii), kami mengusulkan strategi Pengambilan Sampel Berfokus Kotak menggunakan kotak 2D untuk menghasilkan distribusi probabilitas objek dalam ruang 3D, memungkinkan pengambilan sampel probabilistik yang efektif dalam 3D untuk mempertahankan lebih banyak blob objek dan mengurangi blob latar belakang yang berisik. Memanfaatkan usulan Panduan Batas dan Pengambilan Sampel Berfokus Kotak, metode terakhir kami, 3DGS-DET, mencapai peningkatan yang signifikan (+5,6 pada [email protected], +3.7 pada [email protected]) dibandingkan versi pipeline dasar kami, tanpa memperkenalkan parameter tambahan apa pun yang dapat dipelajari . Selain itu, 3DGS-DET secara signifikan mengungguli metode berbasis NeRF yang canggih, NeRF-Det, mencapai peningkatan sebesar +6,6 pada [email protected] dan +8.1 pada [email protected] untuk kumpulan data ScanNet, dan +31.5 yang mengesankan pada [email protected] untuk kumpulan data ARKITScenes. Kode dan model tersedia untuk umum di: https://github.com/yangcaoai/3DGS-DET.

? Kertas | Kode (belum)

Mengemudi Otonom:

Despite recent advancements in high-fidelity human reconstruction techniques, the requirements for densely captured images or time-consuming per-instance optimization significantly hinder their applications in broader scenarios. To tackle these issues, we present HumanSplat that predicts the 3D Gaussian Splatting properties of any human from a single input image in a generalizable manner. In particular, HumanSplat comprises a 2D multi-view diffusion model and a latent reconstruction transformer with human structure priors that adeptly integrate geometric priors and semantic features within a unified framework. A hierarchical loss that incorporates human semantic information is further designed to achieve high-fidelity texture modeling and better constrain the estimated multiple views. Comprehensive experiments on standard benchmarks and in-the-wild images demonstrate that HumanSplat surpasses existing state-of-the-art methods in achieving photorealistic novel-view synthesis. Project page: https://humansplat.github.io/.

? Kertas | Halaman Proyek

Classic work:

1. A Generalization of Algebraic Surface Drawing

Authors : James F. Blinn

Comment: : First paper rendering 3D gaussians.

Abstrak

The mathematical description of three-dimensional surfaces usually falls into one of two classifications: parametric and implicit. An implicit surface is defined to be all points which satisfy some equation F (x, y, z) = 0. This form is ideally suited for image space shaded picture drawing; the pixel coordinates are substituted for x and y, and the equation is solved for z. Algorithms for drawing such objects have been developed primarily for first- and second-order polynomial functions, a subcategory known as algebraic surfaces. This paper presents a new algorithm applicable to other functional forms, in particular to the summation of several Gaussian density distributions. The algorithm was created to model electron density maps of molecular structures, but it can be used for other artistically interesting shapes.

? Kertas

2. Approximate Differentiable Rendering with Algebraic Surfaces

Authors : Leonid Keselman and Martial Hebert

Comment: : First paper to do differentiable rendering optimization of 3D gaussians.

Abstrak

Differentiable renderers provide a direct mathematical link between an object's 3D representation and images of that object. In this work, we develop an approximate differentiable renderer for a compact, interpretable representation, which we call Fuzzy Metaballs. Our approximate renderer focuses on rendering shapes via depth maps and silhouettes. It sacrifices fidelity for utility, producing fast runtimes and high-quality gradient information that can be used to solve vision tasks. Compared to mesh-based differentiable renderers, our method has forward passes that are 5x faster and backwards passes that are 30x faster. The depth maps and silhouette images generated by our method are smooth and defined everywhere. In our evaluation of differentiable renderers for pose estimation, we show that our method is the only one comparable to classic techniques. In shape from silhouette, our method performs well using only gradient descent and a per-pixel loss, without any surrogate losses or regularization. These reconstructions work well even on natural video sequences with segmentation artifacts.

? Kertas | Project Page | Code | ? Short Presentation

3. Unbiased Gradient Estimation for Differentiable Surface Splatting via Poisson Sampling

Authors : Jan U. Müller, Michael Weinmann, Reinhard Klein

Comment: Builds 2D screen-space gaussians from underlying 3D representations.

Abstrak

We propose an efficient and GPU-accelerated sampling framework which enables unbiased gradient approximation for differentiable point cloud rendering based on surface splatting. Our framework models the contribution of a point to the rendered image as a probability distribution. We derive an unbiased approximative gradient for the rendering function within this model. To efficiently evaluate the proposed sample estimate, we introduce a tree-based data-structure which employs multi-pole methods to draw samples in near linear time. Our gradient estimator allows us to avoid regularization required by previous methods, leading to a more faithful shape recovery from images. Furthermore, we validate that these improvements are applicable to real-world applications by refining the camera poses and point cloud obtained from a real-time SLAM system. Finally, employing our framework in a neural rendering setting optimizes both the point cloud and network parameters, highlighting the framework's ability to enhance data driven approaches.

? Kode Kertas

4. Generating and Real-Time Rendering of Clouds

Authors : Petr Man

Comment: Splatting of anisotropic gaussians. Basically a non-differentiable implementation of 3DGS.

Abstrak

This paper presents a method for generation and real-time rendering of static clouds. Perlin noise function generates three dimensional map of a cloud. We also present a twopass rendering algorithm that performs physically based approximation. In the first preprocessed phase it computes multiple forward scattering. In the second phase first order anisotropic scattering at runtime is evaluated. The generated map is stored as voxels and is unsuitable for the real-time rendering. We introduce a more suitable inner representation of cloud that approximates the original map and contains much less information. The cloud is then represented by a set of metaballs (spheres) with parameters such as center positions, radii and density values. The main contribution of this paper is to propose a method, that transforms the original cloud map to the inner representation. This method uses the Radial Basis Function (RBF) neural network.

? Kertas

Kompresi:

3D Gaussian Splatting has recently emerged as a highly promising technique for modeling of static 3D scenes. In contrast to Neural Radiance Fields, it utilizes efficient rasterization allowing for very fast rendering at high-quality. However, the storage size is significantly higher, which hinders practical deployment, eg on resource constrained devices. In this paper, we introduce a compact scene representation organizing the parameters of 3D Gaussian Splatting (3DGS) into a 2D grid with local homogeneity, ensuring a drastic reduction in storage requirements without compromising visual quality during rendering. Central to our idea is the explicit exploitation of perceptual redundancies present in natural scenes. In essence, the inherent nature of a scene allows for numerous permutations of Gaussian parameters to equivalently represent it. To this end, we propose a novel highly parallel algorithm that regularly arranges the high-dimensional Gaussian parameters into a 2D grid while preserving their neighborhood structure. During training, we further enforce local smoothness between the sorted parameters in the grid. The uncompressed Gaussians use the same structure as 3DGS, ensuring a seamless integration with established renderers. Our method achieves a reduction factor of 17x to 42x in size for complex scenes with no increase in training time, marking a substantial leap forward in the domain of 3D scene distribution and consumption.

? Kertas | Project Page | Kode

Difusi:

2024:

1. AGG: Amortized Generative 3D Gaussians for Single Image to 3D

Authors : Dejia Xu, Ye Yuan, Morteza Mardani, Sifei Liu, Jiaming Song, Zhangyang Wang, Arash Vahdat

Abstrak

Given the growing need for automatic 3D content creation pipelines, various 3D representations have been studied to generate 3D objects from a single image. Due to its superior rendering efficiency, 3D Gaussian splatting-based models have recently excelled in both 3D reconstruction and generation. 3D Gaussian splatting approaches for image to 3D generation are often optimization-based, requiring many computationally expensive score-distillation steps. To overcome these challenges, we introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image, eliminating the need for per-instance optimization. Utilizing an intermediate hybrid representation, AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. Moreover, we propose a cascaded pipeline that first generates a coarse representation of the 3D data and later upsamples it with a 3D Gaussian super-resolution module. Our method is evaluated against existing optimization-based 3D Gaussian frameworks and sampling-based pipelines utilizing other 3D representations, where AGG showcases competitive generation abilities both qualitatively and quantitatively while being several orders of magnitude faster.

? Kertas | Project Page| ? Short Presentation

2. Fast Dynamic 3D Object Generation from a Single-view Video

Authors : Zijie Pan, Zeyu Yang, Xiatian Zhu, Li Zhang

Abstrak

Generating dynamic three-dimensional (3D) object from a single-view video is challenging due to the lack of 4D labeled data. Existing methods extend text-to-3D pipelines by transferring off-the-shelf image generation models such as score distillation sampling, but they are slow and expensive to scale (eg, 150 minutes per object) due to the need for back-propagating the information-limited supervision signals through a large pretrained model. To address this limitation, we propose an efficient video-to-4D object generation framework called Efficient4D. It generates high-quality spacetime-consistent images under different camera views, and then uses them as labeled data to directly train a novel 4D Gaussian splatting model with explicit point cloud geometry, enabling real-time rendering under continuous camera trajectories. Extensive experiments on synthetic and real videos show that Efficient4D offers a remarkable 10-fold increase in speed when compared to prior art alternatives while preserving the same level of innovative view synthesis quality. For example, Efficient4D takes only 14 minutes to model a dynamic object.

? Kertas | Project Page | Code | ? Short Presentation

3. GaussianObject: Just Taking Four Images to Get A High-Quality 3D Object with Gaussian Splatting

Authors : Chen Yang, Sikuang Li, Jiemin Fang, Ruofan Liang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian

Abstrak

Reconstructing and rendering 3D objects from highly sparse views is of critical importance for promoting applications of 3D vision techniques and improving user experience. However, images from sparse views only contain very limited 3D information, leading to two significant challenges: 1) Difficulty in building multi-view consistency as images for matching are too few; 2) Partially omitted or highly compressed object information as view coverage is insufficient. To tackle these challenges, we propose GaussianObject, a framework to represent and render the 3D object with Gaussian splatting, that achieves high rendering quality with only 4 input images. We first introduce techniques of visual hull and floater elimination which explicitly inject structure priors into the initial optimization process for helping build multi-view consistency, yielding a coarse 3D Gaussian representation. Then we construct a Gaussian repair model based on diffusion models to supplement the omitted object information, where Gaussians are further refined. We design a self-generating strategy to obtain image pairs for training the repair model. Our GaussianObject is evaluated on several challenging datasets, including MipNeRF360, OmniObject3D, and OpenIllumination, achieving strong reconstruction results from only 4 views and significantly outperforming previous state-of-the-art methods.

? Kertas | Project Page | Code | ? Short Presentation

Authors : Heng Yu, Chaoyang Wang, Peiye Zhuang, Willi Menapace, Aliaksandr Siarohin, Junli Cao, Laszlo A Jeni, Sergey Tulyakov, Hsin-Ying Lee

Authors : Junwu Zhang, Zhenyu Tang, Yatian Pang, Xinhua Cheng, Peng Jin, Yida Wei, Munan Ning, Li Yuan

Abstrak

Recent one image to 3D generation methods commonly adopt Score Distillation Sampling (SDS). Despite the impressive results, there are multiple deficiencies including multi-view inconsistency, over-saturated and over-smoothed textures, as well as the slow generation speed. To address these deficiencies, we present Repaint123 to alleviate multi-view bias as well as texture degradation and speed up the generation process. The core idea is to combine the powerful image generation capability of the 2D diffusion model and the texture alignment ability of the repainting strategy for generating high-quality multi-view images with consistency. We further propose visibility-aware adaptive repainting strength for overlap regions to enhance the generated image quality in the repainting process. The generated high-quality and multi-view consistent images enable the use of simple Mean Square Error (MSE) loss for fast 3D content generation. We conduct extensive experiments and show that our method has a superior ability to generate high-quality 3D content with multi-view consistency and fine textures in 2 minutes from scratch.

? Kertas | Project Page | Code (not yet)

Dynamics and Deformation:

Recently, 3D Gaussian, as an explicit 3D representation method, has demonstrated strong competitiveness over NeRF (Neural Radiance Fields) in terms of expressing complex scenes and training duration. These advantages signal a wide range of applications for 3D Gaussians in 3D understanding and editing. Meanwhile, the segmentation of 3D Gaussians is still in its infancy. The existing segmentation methods are not only cumbersome but also incapable of segmenting multiple objects simultaneously in a short amount of time. In response, this paper introduces a 3D Gaussian segmentation method implemented with 2D segmentation as supervision. This approach uses input 2D segmentation maps to guide the learning of the added 3D Gaussian semantic information, while nearest neighbor clustering and statistical filtering refine the segmentation results. Experiments show that our concise method can achieve comparable performances on mIOU and mAcc for multi-object segmentation as previous single-object segmentation methods.

? Kertas

Language Embedding:

3D Gaussian Splatting (3DGS) has recently revolutionized radiance field reconstruction, achieving high quality novel view synthesis and fast rendering speed without baking. However, 3DGS fails to accurately represent surfaces due to the multi-view inconsistent nature of 3D Gaussians. We present 2D Gaussian Splatting (2DGS), a novel approach to model and reconstruct geometrically accurate radiance fields from multi-view images. Our key idea is to collapse the 3D volume into a set of 2D oriented planar Gaussian disks. Unlike 3D Gaussians, 2D Gaussians provide view-consistent geometry while modeling surfaces intrinsically. To accurately recover thin surfaces and achieve stable optimization, we introduce a perspective-accurate 2D splatting process utilizing ray-splat intersection and rasterization. Additionally, we incorporate depth distortion and normal consistency terms to further enhance the quality of the reconstructions. We demonstrate that our differentiable renderer allows for noise-free and detailed geometry reconstruction while maintaining competitive appearance quality, fast training speed, and real-time rendering.

1. [CVPR '24] PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics

Authors : Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, Chenfanfu Jiang

Abstrak

We introduce PhysGaussian, a new method that seamlessly integrates physically grounded Newtonian dynamics within 3D Gaussians to achieve high-quality novel motion synthesis. Employing a custom Material Point Method (MPM), our approach enriches 3D Gaussian kernels with physically meaningful kinematic deformation and mechanical stress attributes, all evolved in line with continuum mechanics principles. A defining characteristic of our method is the seamless integration between physical simulation and visual rendering: both components utilize the same 3D Gaussian kernels as their discrete representations. This negates the necessity for triangle/tetrahedron meshing, marching cubes, "cage meshes," or any other geometry embedding, highlighting the principle of "what you see is what you simulate (WS2)." Our method demonstrates exceptional versatility across a wide variety of materials--including elastic entities, metals, non-Newtonian fluids, and granular materials--showcasing its strong capabilities in creating diverse visual content with novel viewpoints and movements.

? Paper | Project Page | Code | ? Short Presentation

2. [CVPR '24] SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering

Authors : Antoine Guédon, Vincent Lepetit

Abstrak

We propose a method to allow precise and extremely fast mesh extraction from 3D Gaussian Splatting. Gaussian Splatting has recently become very popular as it yields realistic rendering while being significantly faster to train than NeRFs. It is however challenging to extract a mesh from the millions of tiny 3D gaussians as these gaussians tend to be unorganized after optimization and no method has been proposed so far. Our first key contribution is a regularization term that encourages the gaussians to align well with the surface of the scene. We then introduce a method that exploits this alignment to sample points on the real surface of the scene and extract a mesh from the Gaussians using Poisson reconstruction, which is fast, scalable, and preserves details, in contrast to the Marching Cubes algorithm usually applied to extract meshes from Neural SDFs. Finally, we introduce an optional refinement strategy that binds gaussians to the surface of the mesh, and jointly optimizes these Gaussians and the mesh through Gaussian splatting rendering. This enables easy editing, sculpting, rigging, animating, compositing and relighting of the Gaussians using traditional softwares by manipulating the mesh instead of the gaussians themselves. Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art methods on neural SDFs, while providing a better rendering quality.

? Paper | Project Page | Code | ? Short Presentation

3. NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance

Authors : Hanlin Chen, Chen Li, Gim Hee Lee

Abstrak

Existing neural implicit surface reconstruction methods have achieved impressive performance in multi-view 3D reconstruction by leveraging explicit geometry priors such as depth maps or point clouds as regularization. However, the reconstruction results still lack fine details because of the over-smoothed depth map or sparse point cloud. In this work, we propose a neural implicit surface reconstruction pipeline with guidance from 3D Gaussian Splatting to recover highly detailed surfaces. The advantage of 3D Gaussian Splatting is that it can generate dense point clouds with detailed structure. Nonetheless, a naive adoption of 3D Gaussian Splatting can fail since the generated points are the centers of 3D Gaussians that do not necessarily lie on the surface. We thus introduce a scale regularizer to pull the centers close to the surface by enforcing the 3D Gaussians to be extremely thin. Moreover, we propose to refine the point cloud from 3D Gaussians Splatting with the normal priors from the surface predicted by neural implicit models instead of using a fixed set of points as guidance. Consequently, the quality of surface reconstruction improves from the guidance of the more accurate 3D Gaussian splatting. By jointly optimizing the 3D Gaussian Splatting and the neural implicit model, our approach benefits from both representations and generates complete surfaces with intricate details. Experiments on Tanks and Temples verify the effectiveness of our proposed method.

? Kertas

Misc:

In this paper, we address the limitations of Adaptive Density Control (ADC) in 3D Gaussian Splatting (3DGS), a scene representation method achieving high-quality, photorealistic results for novel view synthesis. ADC has been introduced for automatic 3D point primitive management, controlling densification and pruning, however, with certain limitations in the densification logic. Our main contribution is a more principled, pixel-error driven formulation for density control in 3DGS, leveraging an auxiliary, per-pixel error function as the criterion for densification. We further introduce a mechanism to control the total number of primitives generated per scene and correct a bias in the current opacity handling strategy of ADC during cloning operations. Our approach leads to consistent quality improvements across a variety of benchmark scenes, without sacrificing the method's efficiency.

? Kertas

2023:

1. [CVPRW '24] Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images

Authors : Jaeyoung Chung, Jeongtaek Oh, Kyoung Mu Lee

Abstrak

In this paper, we present a method to optimize Gaussian splatting with a limited number of images while avoiding overfitting. Representing a 3D scene by combining numerous Gaussian splats has yielded outstanding visual quality. However, it tends to overfit the training views when only a small number of images are available. To address this issue, we introduce a dense depth map as a geometry guide to mitigate overfitting. We obtained the depth map using a pre-trained monocular depth estimation model and aligning the scale and offset using sparse COLMAP feature points. The adjusted depth aids in the color-based optimization of 3D Gaussian splatting, mitigating floating artifacts, and ensuring adherence to geometric constraints. We verify the proposed method on the NeRF-LLFF dataset with varying numbers of few images. Our approach demonstrates robust geometry compared to the original method that relies solely on images.

? Paper | Project Page | Kode

2. EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS

Authors : Sharath Girish, Kamal Gupta, Abhinav Shrivastava

Abstrak

Recently, 3D Gaussian splatting (3D-GS) has gained popularity in novel-view scene synthesis. It addresses the challenges of lengthy training times and slow rendering speeds associated with Neural Radiance Fields (NeRFs). Through rapid, differentiable rasterization of 3D Gaussians, 3D-GS achieves real-time rendering and accelerated training. They, however, demand substantial memory resources for both training and storage, as they require millions of Gaussians in their point cloud representation for each scene. We present a technique utilizing quantized embeddings to significantly reduce memory storage requirements and a coarse-to-fine training strategy for a faster and more stable optimization of the Gaussian point clouds. Our approach results in scene representations with fewer Gaussians and quantized representations, leading to faster training times and rendering speeds for real-time rendering of high resolution scenes. We reduce memory by more than an order of magnitude all while maintaining the reconstruction quality. We validate the effectiveness of our approach on a variety of datasets and scenes preserving the visual quality while consuming 10-20x less memory and faster training/inference speed.

? Paper | Project Page | Kode

3. [CVPR '24] COLMAP-Free 3D Gaussian Splatting

Authors : Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, Xiaolong Wang

Abstrak

While neural rendering has led to impressive advances in scene reconstruction and novel view synthesis, it relies heavily on accurately pre-computed camera poses. To relax this constraint, multiple efforts have been made to train Neural Radiance Fields (NeRFs) without pre-processed camera poses. However, the implicit representations of NeRFs provide extra challenges to optimize the 3D structure and camera poses at the same time. On the other hand, the recently proposed 3D Gaussian Splatting provides new opportunities given its explicit point cloud representations. This paper leverages both the explicit geometric representation and the continuity of the input video stream to perform novel view synthesis without any SfM preprocessing. We process the input frames in a sequential manner and progressively grow the 3D Gaussians set by taking one input frame at a time, without the need to pre-compute the camera poses. Our method significantly improves over previous approaches in view synthesis and camera pose estimation under large motion changes.

? Paper | Project Page | Code (not yet) | ? Short Presentation

4. iComMa: Inverting 3D Gaussians Splatting for Camera Pose Estimation via Comparing and Matching

Authors : Yuan Sun, Xuan Wang, Yunfan Zhang, Jie Zhang, Caigui Jiang, Yu Guo, Fei Wang

Abstrak

We present a method named iComMa to address the 6D pose estimation problem in computer vision. The conventional pose estimation methods typically rely on the target's CAD model or necessitate specific network training tailored to particular object classes. Some existing methods address mesh-free 6D pose estimation by employing the inversion of a Neural Radiance Field (NeRF), aiming to overcome the aforementioned constraints. However, it still suffers from adverse initializations. By contrast, we model the pose estimation as the problem of inverting the 3D Gaussian Splatting (3DGS) with both the comparing and matching loss. In detail, a render-and-compare strategy is adopted for the precise estimation of poses. Additionally, a matching module is designed to enhance the model's robustness against adverse initializations by minimizing the distances between 2D keypoints. This framework systematically incorporates the distinctive characteristics and inherent rationale of render-and-compare and matching-based approaches. This comprehensive consideration equips the framework to effectively address a broader range of intricate and challenging scenarios, including instances with substantial angular deviations, all while maintaining a high level of prediction accuracy. Experimental results demonstrate the superior precision and robustness of our proposed jointly optimized framework when evaluated on synthetic and complex real-world data in challenging scenarios.

? Paper | Kode

Rendering:

Neural Radiance Fields (NeRFs) have demonstrated the remarkable potential of neural networks to capture the intricacies of 3D objects. By encoding the shape and color information within neural network weights, NeRFs excel at producing strikingly sharp novel views of 3D objects. Recently, numerous generalizations of NeRFs utilizing generative models have emerged, expanding its versatility. In contrast, Gaussian Splatting (GS) offers a similar renders quality with faster training and inference as it does not need neural networks to work. We encode information about the 3D objects in the set of Gaussian distributions that can be rendered in 3D similarly to classical meshes. Unfortunately, GS are difficult to condition since they usually require circa hundred thousand Gaussian components. To mitigate the caveats of both models, we propose a hybrid model that uses GS representation of the 3D object's shape and NeRF-based encoding of color and opacity. Our model uses Gaussian distributions with trainable positions (ie means of Gaussian), shape (ie covariance of Gaussian), color and opacity, and neural network, which takes parameters of Gaussian and viewing direction to produce changes in color and opacity. Consequently, our model better describes shadows, light reflections, and transparency of 3D objects.

? Paper | Kode

Ulasan:

? Kertas

SLAM:

The integration of neural rendering and the SLAM system recently showed promising results in joint localization and photorealistic view reconstruction. However, existing methods, fully relying on implicit representations, are so resource-hungry that they cannot run on portable devices, which deviates from the original intention of SLAM. In this paper, we present Photo-SLAM, a novel SLAM framework with a hyper primitives map. Specifically, we simultaneously exploit explicit geometric features for localization and learn implicit photometric features to represent the texture information of the observed environment. In addition to actively densifying hyper primitives based on geometric features, we further introduce a Gaussian-Pyramid-based training method to progressively learn multi-level features, enhancing photorealistic mapping performance. The extensive experiments with monocular, stereo, and RGB-D datasets prove that our proposed system Photo-SLAM significantly outperforms current state-of-the-art SLAM systems for online photorealistic mapping, eg, PSNR is 30% higher and rendering speed is hundreds of times faster in the Replica dataset. Moreover, the Photo-SLAM can run at real-time speed using an embedded platform such as Jetson AGX Orin, showing the potential of robotics applications.

? Paper | Project Page | Kode

Jarang:

We introduce the Splatter Image, an ultra-fast approach for monocular 3D object reconstruction which operates at 38 FPS. Splatter Image is based on Gaussian Splatting, which has recently brought real-time rendering, fast training, and excellent scaling to multi-view reconstruction. For the first time, we apply Gaussian Splatting in a monocular reconstruction setting. Our approach is learning-based, and, at test time, reconstruction only requires the feed-forward evaluation of a neural network. The main innovation of Splatter Image is the surprisingly straightforward design: it uses a 2D image-to-image network to map the input image to one 3D Gaussian per pixel. The resulting Gaussians thus have the form of an image, the Splatter Image. We further extend the method to incorporate more than one image as input, which we do by adding cross-view attention. Owning to the speed of the renderer (588 FPS), we can use a single GPU for training while generating entire images at each iteration in order to optimize perceptual metrics like LPIPS. On standard benchmarks, we demonstrate not only fast reconstruction but also better results than recent and much more expensive baselines in terms of PSNR, LPIPS, and other metrics.

? Paper | Project Page | Code | ? Short Presentation

Navigasi:

Memperluas