A method is developed to dramatically improve the accuracy of '3D Gaussian splitting,' which converts images into 3D
A research team from the University of British Columbia and Google has announced that they have developed a method called '3D Gaussian Splatting as a Markov Chain Monte Carlo Method' that dramatically improves the rendering quality of '3D Gaussian Splatting,' a method of 'neural rendering' that combines images taken from multiple viewpoints to generate images with depth.
3D Gaussian Splatting as Markov Chain Monte Carlo
3D Gaussian splitting is a method of expressing 3D images by overlapping millions of Gaussian distributions, and the accuracy is improved by first setting the initial state of the Gaussian distribution appropriately and then adjusting each Gaussian distribution to match the actual image. Conventional 3D Gaussian splitting has the problem that the accuracy depends on the initial state, so in order to improve the accuracy of the output image, it is necessary to use a different method to properly arrange the initial state to a certain extent.
A research team from the University of British Columbia and Google discovered that in the adjustment phase of 3D Gaussian splitting, they could adjust the Gaussian distributions using Stochastic Gradient Langevin Dynamics by treating millions of Gaussian distributions as samples for the Markov Chain Monte Carlo method .
Using this new method, it is possible to output 3D images with high accuracy even if the initial state is random. The figures below show examples of 3D images output using the conventional method and the newly developed method from a random initial state. The image on the left is from the conventional method, and there are many blurry parts overall, while the output from the new method on the right shows clear images down to the fine details. The difference is particularly noticeable in the buildings in the background of the screen.
Here's another example. The top row is the output using the conventional method, and the bottom row is the output using the new method. The left row has a random initial state, and the right row has an appropriate initial state. The new method shows that distant scenery can be depicted in more detail.
The image below is an example of converting a photo of a plant into 3D. When the image was output using the conventional method from the random initial state shown in the upper left, many white dots appeared.
With conventional methods, it seems that even when the initial state is set, it is difficult to express fine details such as the lower leaves.
The new technique allows for precise expression of even the smallest details.
The implementation of the new method, '3D Gaussian splatting as a Markov chain Monte Carlo method,' is available on GitHub .
Related Posts:
