Twitter introduces optimization of automatic trimming of images by AI technology

Images posted on Twitter are automatically trimmed according to the UI design, but it was often done to cut out "I was different from what I thought ...". Twitter announces on the official blog that by utilizing AI technology, it has become possible to perform trimming faster and optimum than before.

Speedy Neural Networks for Smart Auto-Cropping of Images
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-Auto-Cropping-of-Images.html

Twitter, which enabled image posting in 2011, has cropped photos according to the screen layout of smartphones and displayed them as thumbnails (preview). However, because the accuracy of the automatic trimming function is low, it was often that you could not cut out with the image you want. Although we aimed to create a thumbnail close to the image of the tweeters by trimming the image by focusing on the face of the person shown in the photograph by using the face detection function etc. so far, It seems that there was no reason for a person to appear, and it did not go well. If the face can not be detected, it is basically a disappointing specification that trimming is based on the center of the picture.

Traditional automatic trimming on the left. Ideal automatic trimming on the right.


To improve the automatic trimming function on Twitter, we decided to introduce the idea of ​​focusing on the "salient (dominant)" area in the image and trimming it. A remarkable area is that there is a high possibility that people who viewed the image will have a viewpoint.


In general, people tend to pay attention to high-contrast areas in addition to objects such as "face", "text" and "animals" in images, so Twitter can train neural networks and specific algorithms , He said he developed a technology to accurately predict what people want to see.


Accuracy predicting "remarkable area" which people can pay attention by advancement of machine learning technology is very high accuracy, but posting multiple images reads a large amount of images from the feature of Twitter service It was inappropriate because it took too long to trim the image using the neural network trained with. For Twitter it is unnecessary to predict at a fine pixel level and it is necessary to know roughly where the outstanding area is located, so a network called "knowledge distillation" (distillation of knowledge) predicts a prominent area I heard that he decided to drastically reduce its size.


After reducing the size by distilling the knowledge, I did the work called "pruning" to eliminate features that waste computing power, not contributing to image detection accuracy. Combining the two kinds of AI techniques such as neural net distillation and pruning, the speed of trimming the image based on the "salient region" predicted using machine learning can be increased by 10 times compared with the conventional technique Twitter says.

With a new technology that allows natural trimming with little waiting time when posting, the disappointing thumbnails that traditionally focused on "under the feet" can be trimmed focusing on "expression of children" To


Even in situations where there are many things, it has become possible to accurately see through important elements such as text and cut it out.


Even if you submit multiple images, you can see that it is optimally optimized.


It is said that the automatic trimming function utilizing AI technology has already been introduced in browser version, iOS / Android version application.