Dec 29, 2021 08:00:00

Google researchers have finally developed 'HyperNeRF' that responds to 'changes in the shape of objects' with technology for building 3D models from images.

Is a technique from the image and video to produce a three-dimensional image 'Neural Radiance Fields (neural Radiance field:

NeRF )' So far, the target is changing the shape scene, such as the change and Yuku broken object of expression can not respond did. Newly developed by researchers at Google and the University of Washington, 'HyperNeRF' is a further development of conventional technology that can respond to changes in the shape of objects.

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
https://hypernerf.github.io/

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields-YouTube


In 2020, research teams such as Kun Hong Park of the University of Washington and Utokarsh Shinha of Google released ' Nerfies ', which is accurate even from a video of a dog that can not stay still due to the improvement of NeRF (left). It is a technology that makes it possible to create a model (center).

However, it cannot respond well to changes in the shape of the object, so if you enter a person's face that changes the facial expression significantly ...

My mouth is distorted into a strange shape.

On the other hand, 'HyperNeRF (left)', which is a further improvement of Nerfies, solves this problem.

Therefore, even if the facial expression changes so much that the shape of the face changes, an image that is faithful to the original facial expression can be obtained.

Of course, it corresponds to other than the face. It's a bit confusing in the captured image, but HyperNeRF (left) can also model a shimmering fire, compared to the traditional Nerfies (right) where the object being struck by the burner is scooped out. ..

Even in the video of working by moving your hands ...

I was able to grasp the structure of the part hidden by my hand.

The HyperNeRF improvements were inspired by the

level set method . The level set method is to build a model that takes into account the detailed structure of an object by adding the information of the 'slice image of the object' seen from a virtual high dimension to the three-dimensional model.

In the conventional level set method, a plane was used for the slice image, but ...

With the idea of bending a plane, HyperNeRF has made it possible to model the subject more efficiently and accurately.

As a result, in the past, it was necessary to shoot images taken with multiple cameras fixed at a fixed point, but with a single camera like shooting with a smartphone, it is possible to shoot from various angles by shaking. Even in such a case, it is possible to accurately capture the changes in the object. For example, a scene of breaking a cookie ...

Coffee poured into a cup.

Even with a 3D printer that moves at high speed, it is possible to realize a model that is clear and has few failures.