Technology will be developed to enable ultra-slow-motion video shooting at 5400 fps with event cameras and AI
A research team from the
High Speed and High Dynamic Range Video with an Event Camera
http://rpg.ifi.uzh.ch/E2VID.html
Scientists Use Camera with Human-Like Vision to Capture 5,400 fps Video
https://petapixel.com/2019/07/09/scientists-use-camera-with-human-like-vision-to-capture-5400-fps-video/
When shooting 24 fps images with a regular camera, 24 still images are captured per second. In the case of slow motion video, shooting with 120 fps or 240 fps, etc., with an extremely large number of frames per second, and setting the playback frame to 24 fps or 30 fps, the video will be played more slowly than it actually is. It is not.
The event camera used by the research team this time is a camera that captures only the 'brightness change' of the light striking the sensor, unlike ordinary cameras that digitize and output the brightness of the light striking the sensor through a lens. The research team believes that 'even though the event camera can not obtain a clear image like a normal camera, it has an extremely wide dynamic range , no subject blur, and the delay is only a few microseconds.' We built a system to reconstruct frames from event camera data.
The research team first developed an algorithm to manually reconstruct the frames from the event camera data. Then, using the simulated data, we trained a neural network that could reconstruct event data with the same algorithm, and input the actual data.
The following movie is a slow motion movie actually released by the research team.
High Speed and High Dynamic Range Video with an Event Camera-YouTube
Although the movie taken with the event camera and reconstructed with AI is black and white, it can be taken with the same image quality as the 240 fps movie taken with Huawei P20 Pro , and the shape and movement of the fine object can be captured I understand well.
Below is a comparison of scenes taken by men breaking water balloons, taken with an
The left is event camera data, and the one reconstructed using AI is the center. In the movie (right) taken with the smartphone, motion blur (subject blur) is produced on the male face, while the central movie shows the male face clearly.
The right-most image is read and reconstructed by reading the luminance separately for red, green, and blue. After all it is difficult to reproduce full color with AI completely, but although the screen is squeezed without mixing colors, it is a pattern that can be photographed to some extent.
In addition to rebuilding from the event camera, the following scene shows what is being identified by image recognition.
Humans have two eyes, but since cameras are basically monocular, it is difficult to recognize the depth of objects. However, the AI trained by the research team can also estimate the depth of objects from the image of the event camera.
According to the research team, the slow motion video reconstructed by AI had a 20% improvement in image quality over regular slow motion video. When shooting a high frame rate movie over 5000 fps, the amount of light entering the sensor is extremely reduced, which makes the normal camera lighting conditions extremely severe, but the event camera data for reading changes in brightness In order to use it, you can shoot high-quality slow motion video even at high frame rates.
Most cameras that can shoot at frame rates as high as 5000 fps or higher are for business use, and there are cameras that are extremely expensive in terms of price and availability in the general market. PetaPixel, a technology media that handles cameras, is expecting to use this technology to be able to shoot super slow motion HDR video at an affordable price.
Related Posts:
