How is 'artwork' that influences Netflix's viewing generated?

According to Netflix, "artwork" displayed on the background of the work page plays a very big role in deciding whether to watch or not when the user is looking for a new work. Artwork is created by cutting out one scene in the work, but to attract users, Netflix's proprietary technology is stuffed up in generation. Netflix explains how you are generating artwork.

AVA: The Art and Science of Image Discovery at Netflix
https://medium.com/netflix-techblog/ava-the-art-and-science-of-image-discovery-at-netflix-a442f163af6

While new works are appearing one after another, Netflix will show artworks that highlight the unique elements of the work as frames of the movie, about 30 still images per second constituting the movie We sort out and display it. For example, in the one hour episode of " Stranger Things ", approximately 86,000 frames are included.

Many of the artwork so far have been made by human beings sorted out from the frame. We found that using the A / B test we can find artworks that can further promote viewers' viewing of the target audience, and screening experts have tried more artworks, Netflix says that it is too challenging to continue sorting by humans as the number of works in the world continues to increase.

Netflix developed as a substitute for humans is a system called AVA. AVA is a tool to capture the most suitable moment to express a work, creative team creates artwork based on "moment" caught by AVA. An example of a scene selected by AVA is the following image. The work that became a source is " The Crown " on the upper left, " Stranger Things " on the upper right, " Black Mirror " on the lower left, and " Dark " on the lower right.

In the AVA, each frame is evaluated from various viewpoints, scoring is performed, and frames are extracted according to the viewpoints of "beauty", "creativity" and "diversity" using a ranking algorithm. For example, here is the frame that AVA chose in terms of "Bright".

Elements that form the basis of frame evaluation can be roughly divided into three categories, the first being " visual metadata " such as "brightness" "hue" "contrast" "blur".

The second is " context metadata " such as "whether the face is included", "how it moves" "how the camera moves," "what objects other than humans are included".

And the third is " composition metadata " of elements based on design principles such as "three division method", "depth of field", "symmetry" such as photographs, movies, and art works.

When AVA chooses the best candidate, it is also taken into consideration whether the reflected person is the main character. AVA examines other candidate frames and automatically determines whether it is a key figure by capturing similarity.

Also, it has become diversified so that many similar images are not selected. For example, in the figure below, three types of shots are selected from the left: "medium shot", "close-up" and "extreme close".

In this way, because the work of extracting "the best scene" is automated by AVA, the creative team is designed to be able to design attractive artwork everyday.