MIT · Google researchers announced technology that can take photo retouching processing like professional even with smartphones using machine learning

When submitting photos taken with smartphones to SNS etc., it is very common to do light retouching. However, due to limitations of smartphone performance, the work that can be done is limited. MIT and Google researchers,SIGGRAPH 2017In Machine Learning, we announced a system that can perform retouching like image editing professionals in real time with smartphones.

Deep Bilateral Learning
https://groups.csail.mit.edu/graphics/hdrnet/

Automatic image retouching on your phone | MIT News
http://news.mit.edu/2017/automatic-image-retouching-phone-0802

The outline of the technology for easily processing image retouching processing on a non-smart phone using a machine learning system developed by MIT and Google researchers is explained in the following movie.

Deep Bilateral Learning for Real-Time Image Enhancement - YouTube


This time, retouching technology jointly developed by MIT and Google researchers is a major feature of executing most of image processing with low resolution images. Machine learning systems using high resolution photographs are required for high machine performance, but low resolution images can be processed with smartphones. However, from using low resolution images, the color values ​​of individual pixels of high resolution pictures faced the difficulty of having to anticipate from the rough output from the machine learning system.


So researchers changed the output of the machine learning system to a simple "set of expressions" instead of "images", to change the color of the pixel. During training, the performance of the system was evaluated in terms of how similar it is to the expression outputted when applied to the original high resolution photo and the expression from the retouched image. Furthermore, we applied a three-dimensional grid of 16 × 16 × 8 to fit the formula to individual pixels of the high resolution image. The 16 × 16 face was set to the pixel position of the original image, and the 8 layers were made to correspond to the brightness of different pixels.


Researchers trained the system with datasets created by Durand's group and Adobe. There were 5000 images in this data set, each of which was retouched by 5 different photographers. In addition, we trained the system using thousands of pairs of images generated by applying image processing algorithms and others to create HDR images.


To examine the performance of the final system, were compared when treated with the same resolution as the original image rather than a development system and the low-resolution image to be processed at a lower resolution image, if using the original image is required memory 12GB whereas it was, 100MB memory is required for the processing of low-resolution image, the necessary memory seems to have run out in less than 1 of 100 minutes.

Professor John Baron of MIT said, "The technology of the machine learning system developed this time has the potential to make real-time image retouching processing on smartphones very convenient.In machine learning of images, on smartphones · Although there are severe limitations on power resources, you may avoid this limitation and bring the experience of shooting beautiful photos in real time. "