GAN-based 'STIT' that edits the face of a person in a movie in real time is now available
GAN-based technology ' STIT ' has been announced, which allows you to edit the face of a person in a movie in real time without any discomfort.
[2201.08361] Stitch it in Time: GAN-Based Facial Editing of Real Videos
Stitch it in Time: GAN-Based Facial Editing of Real Videos
https://stitch-time.github.io/
Generative Adversarial Network (GAN) is a widely used technique for facial image editing. However, it is said that it is very difficult to apply this 'image editing using GAN' to a movie, because there is 'temporal consistency' in the movie. Movies are always temporally consistent, so poorly editing them can make the movie look strange and make it clear that it was edited.
However, a research team such as Rotem Tzaban has announced a technology 'Stitch it in Time (STIT)' that allows you to edit the face of a person in a movie without discomfort by using StyleGAN , which is a type of GAN.
STIT allows you to edit your movie at the frame level without trying to enforce temporal consistency. This allows you to edit your movie for smoothness and low jitter . STIT can analyze various components existing in GAN's editing pipeline, identify consistent components, and use them to generate 'movies that you do not know have been edited'.
What kind of movie can actually be generated is as follows. By clicking the case images listed below, it is possible to play the movie edited by STIT and the original movie before editing at the same time.
From the left, an 'Original' movie, a movie with a person as 'Smile', and a movie with a person as 'Young'.
From the left, former President Barack Obama's 'Original' movie, 'Smile' movie, 'Gender' changed movie, 'Young' movie edited to look, 'Old' movie
From left to right, actress Emma Watson's 'Original' movie, 'Angry' movie, and 'Young' movie edited to look like
From the left, an 'Original' movie, a movie with 'Eyeshadow' added, a movie with 'Glasses', and a movie edited with 'Bushy Eyebrows'.
From the left, 'Original' movie, 'French Beard' movie, 'Glasses Removal' movie
It is possible to change facial expressions not only in live-action movies but also in 3DCG animation movies.
'STIT' has been announced on arXiv, a preprint server for publishing papers, and the source code is also available on GitHub.
GitHub --rotemtzaban / STIT
https://github.com/rotemtzaban/STIT
Related Posts:
in Software, Posted by logu_ii