AR technology 'Niantic Occlusion' that can realize real and digital interaction far beyond conventional AR

Matrix Mill recently bought by Niantic is a team that has worked for many years in the construction of a deep neural network that can gather information on surrounding objects from multiple cameras and estimate shape. The deep neural network that Matrix Mill has been engaged in over the years in development has redefined "How computers understand and understand the real world", and how deep real objects and digital objects interact with each other There is a possibility of greatly changing whether it works or not. Based on such technology, Niantic has developed an AR technology " Niantic Occlusion " that is much more realistic than the existing augmented reality (AR), which can elaborately interact between reality and digital, surprised A demonstration of kaku is released.

A Peek Inside the Niantic Real World Platform - Niantic
https://nianticlabs.com/blog/nianticrealworldplatform/

You can see how Niantic Occlusion differs from conventional AR in a single shot when you see the following movie.

Codename: Niantic Occlusion - Real World AR Occlusion featuring Pikachu and Eevee - YouTube


In the case of conventional AR

Pikachu of the AR object is displayed on the image of the real world.

However, since we are just placing AR objects, we will completely ignore the objects that exist in the real world.

Although "this AR object can not interact with real world objects" was one of the major restrictions in the conventional AR, Niantic Occlusion solved spectacularly.

That's why it will be a demonstration of Niantic Occlusio from here.

It was the same until Pikachu was displayed, but ......

Pikachu runs and goes to the back of the planter.

Then it appeared from the back of the pedestrian.

In this way, the neural network recognizes the shape of the object existing in the real world, and it becomes possible to display "as if Pikachu is in the real world".

Case of Niantic Occlusion Part 2.

Evey also appeared.

Even if the number of AR objects increases.

Like Niantic Occlusion, Niantic aims to understand real-world objects in detail and aim to make digital objects interact with real-world objects "as if they are in the real world" "Niantic Real World Platform" to advance.

Below is the state that computer vision under development finely recognizes real world objects and each object explains "something" with quantifiable confidence score. By letting the computer vision algorithm learn objects such as "table" and "chair", the vocabulary increases and it seems that various objects can be recognized.


If computer vision can correctly understand real world objects, besides "interaction with real world objects" like Niantic Occlusion, for example, if you find "real world flower" When displaying "digital bees" or "real world lakes" you can display "digital ducks", Niantic notes that the possibilities as AR will expand to infinity It is.

In addition to "Niantic Occlusion", Niantic is developing new technology for "Niantic Real World Platform". For example, we have developed our own low latency AR networking technology for multiplayer games, which makes it possible to seamlessly realize AR multiplayer games that can be enjoyed by multiple people like the following movie I will.

Codename: Neon - Real World Multiplayer AR Demo - YouTube


Codename: Tonehenge - Real World Multiplayer AR Demo - YouTube