New technology 'Soccer On Your Tabletop' reproducing football game three-dimensionally on the table at AR

Constantine - Remastas of the University of Washington announced a technology that enables interactive rendering of football game images on the table using extended silver stone (AR). This system relocates the 3D player data extracted from the soccer game into the 3D space using the convolution neural network (CNN), thereby converting the two-dimensional game image stereoscopically and freely from various directions You will be able to watch soccer games.

Soccer On Your Tabletop
http://grail.cs.washington.edu/projects/soccer/

Football matches land on your table thanks to augmented reality | TechCrunch
https://techcrunch.com/2018/06/19/football-matches-land-on-your-table-thanks-to-augmented-reality/

I can understand exactly what the movie "Soccer On Your Tabletop" will be able to see football stereoscopically devised by Mr. Rematus, if you can see the following movie.

Soccer On Your Tabletop - YouTube


The man wearing in the image is wearing Microsoft's mixed reality (MR) goggle " HoloLens "

The pitch of soccer spreads on the desk in front of the man's eyes ... ...

The game is spreading in front of us. Of course the pitch is fixed on the desk so the user can watch the game freely from any angle of 360 degrees.

The original match video is unlisted on YouTube.

It was reconstructed with 3DCG using CNN and game data.

Then, how exactly is this realized?

First of all, the original match video.

A white line is drawn on the football pitch, so use this for camera calibration.

Subsequently, we found players lined up on the pitch.

Then estimate the movement of each player's body ... ...

Tracking the movements of the front and rear players.

This completes the division of players and pitches.

From the player's pose and movement tracking, it is possible to estimate including the player's pose up to the depth, and you will be able to know which one of the left and right legs is in front.

Finally we generate a mesh ... ...

We rebuild the game itself using football game player data.

After that, an example of reconstructing various game images will flow.

This picture ...

This is converted.

As "Soccer On Your Tabletop" is under development, the movements of the players are not reproduced with high precision, and the resolution is still too long and far from perfection. Even so, TechCrunch of the overseas technology news site is evaluating that it is "a wonderful idea" to be able to reconstruct images from one direction into three-dimensional images seen from all angles.

The source code of "Soccer On Your Tabletop" is published on GitHub.

GitHub - krematas / soccerontable: Upconverting YouTube soccer videos in 3D for viewing in AR / VR devices.Soccer On Your Tabletop