Technology that replaces people and cars in movies with CG in real time will appear, and anime characters and VTuber may be able to walk in the city
The AI system 'TransforMR ' that detects and deletes objects such as cars and people from the images taken by the camera and replaces them with CG models in real time, the AI team of Dusburg Essen University, ETH Zurich, and Porsche, an automobile manufacturer Announced jointly. TransforMR allows you to edit what you see on the fly, so you can expect applications in augmented reality (AR) technology.
TransforMR: Pose-Aware Object Substitution for Composing Alternate Mixed Realities
Why look at reality when you can edit what you see in real time? --The Verge
https://www.theverge.com/2021/10/13/22723969/mixed-augmented-reality-edit-ar-transformr-research
You can understand what TransforMR is like by watching the following movie.
TransforMR: Pose-Aware Object Substitution for Composing Alternate Mixed Realities (ISMAR '21) --YouTube
TransforMR is a technology that deletes the target in real time from the image taken by the camera and replaces it with a CG model prepared in advance.
TransforMR performs two processes in parallel in real time: 'erasing the target' and 'following the movement of the target'. It is a mechanism that recognizes the target and compensates for the cut out place according to the background, and overlays the CG model linked to the movement of the target.
The processed video is displayed on the user client such as a PC or smartphone, but the main processing of Transfor MR is performed on the back-end server. Since the data is sent to the cloud, it seems that a 4G line or more is required to execute Transfor MR.
The CG model to be replaced is loaded by Unity.
For example, take a picture of a car running on an iPad.
There is a car running in front of me, but on the iPad the car has been replaced by a science fiction-like vehicle.
Below, the woman walking in the foreground and the car running in the back are replaced by robots and science fiction-like vehicles, respectively. There is a height difference between the female and the CG model, and there are traces of the background being cut off above the robot's head.
Make a car a witch ...
It is also possible to replace women with bears.
Since the processing is performed in real time, the CG model can be changed on the spot.
Porsche AI researcher Mohammed Cali cites the ability to detect poses as one of the key points of Transfor MR. Being able to detect poses means that, for example, when a person is detected, the joints of the whole body can be identified, and the movement of the CG model can be linked to the target person. By using this system, it is also possible to 'replace the movement of the performer with a CG model of a fictitious character and perform live distribution outdoors or in a place without special equipment'.
However, Kali said, 'At the moment, running Transfor MR at a large resolution requires a lot of calculations, so there is a technical limit. The demonstration is done with a very small image of 512 x 512 pixels. Therefore, the amount of communication is very small. ' In addition, the frame rate of the video is about 15 fps, lag occurs for 50 to 100 milliseconds, and the CG model to be replaced is not the highest quality, so Transfor MR seems to be still in the development stage.
Regarding why Porsche is participating in the research of this technology, Porsche AI architect Tobias Grosse-Puppendal said, 'To improve the experience of passengers and drivers. In the future, traffic congestion It is also possible to apply TransforMR to entertain those involved in the process. Our main question was how to make the reality fun and changeable. ' Porsche is also developing a system called '(PDF file) SoundRide ' that detects changes in the surrounding landscape and plays appropriate music while driving.
Related Posts:
