An amazing algorithm that can automatically convert existing images to all kinds of painting taste with deep learning

The development of an algorithm that recognizes the style of a prominent painter and converts another image to that taste is proceeding by using "deep learning" which is remarkable evolution of technology. A state of actual conversion performed in the study by Manuel Ruder of University of Freiburg, University of Freiburg, Germany, etc. has been released in the movie, but that state has reached a level that surprises can not be forbidden.

Supplementary video for "Artistic style transfer for videos" - YouTube


This algorithm is based on computer scientist Leon A. Gatys et alThe paper published in 2015It is a further development of the contents of, and it can reflect the style of a prominent painter in the material image. For example, in the CG animation film "Ice Age", Van Gogh's work "Star moonlidayIf you multiply the taste of ... "


Change to such a feeling. However, there are things like spots and blur on the screen.


This is because the image was processed on a frame-by-frame basis. If you look at the actual video, you can see that unnatural stains and changes in light and darkness are seen and it makes us feel unnatural.


Therefore, we succeeded in suppressing the occurrence of unnatural noise by adopting temporal constraint which takes into account the deviation existing between two frames.


It can be said that the taste of Van Gogh work is added to the image of ice age.


The next video is CG animation multiplied by Munch 's scream. The original image ......


Change to such a feeling. I was surprised that the image of animation which was realistic was completely replaced by an impressive thing.


CG animation work "Sintel"Of the French painter who represents FauvismHenri MatisseBy multiplying the taste of "La femme au chapeau" drawn by ... ...


Like this. Compared with "a woman in a hat" it is somewhat soft focus and somewhatStipplingIt is finished in a picture which feels somewhat like a nuance like.


When CG animation is multiplied by Picasso's cubism work "Weated female nude" ... ....


The world view changes completely. However, it may be felt that the sense of cubism is somewhat weak.


Of course, this algorithm supports not only CG but also live action video. TV drama"Miss · MarpleBy multiplying Van Gogh's stars, moons and nights with one scene ... ...


Like this. I am surprised that this atmosphere is reproduced not by still images but by moving images.


It is possible to redraw the grand scenery by CG with the taste of the water painting.


The texture of the image changes to a glitter, you can see that the nuance that makes the detail of the rock and others feel the mountain water painting is added.


Even in the following movies, you can see the changes and processing differences.

Artistic style transfer for videos - YouTube


The stain of "Ice · Age" that appeared at the beginning of the article was due to processing of the image for each frame alone ... ...


By adopting temporal constraint, we succeeded in greatly reducing unnatural noise from the image.


The image of the square in the vicinity of the Munster cathedral in Freiburg, Germany, was captured by Van Gogh's painting "Cafe Terrace at night"Converting ......


Like this. It is a level where you can not distinguish which one is no longer a real Van Gogh picture if you do not know it.


When I superimposed Sintel "a woman with a hat" like this ... ...


Russian abstract painter,Wassily KandinskyCompositon VII "is multiplied, it is like this.


This is Picasso's self-portrait. It is a technology which is surprised anyway by the change from the original material.


When converting a scene of Star Wars into a German impressionist, Heinrich Schlief ... ...


Like this. It is noteworthy that a lot of change is added also to the state of the wall surface reflected in the back.


To Ice Age,Mural painting of Lascaux caveTogether with the taste ... ...


Now this. It is also a texture with mural painting, which makes it a nice feeling with this.


The paper on this newly announced algorithm in June 2016 can be viewed from the following link.

[1604.08610] Artistic style transfer for videos
http://arxiv.org/abs/1604.08610

(PDF)Artistic style transfer for videos


Also, since the source code is published in GitHub, anyone can actually try it out.

GitHub - manuelruder / artistic - videos: Torch implementation for the paper "Artistic style transfer for videos"
https://github.com/manuelruder/artistic-videos