An AI researcher analyzes the greatness of the subject camera filter that converts gender in real time



When a man takes a picture of his face with a camera, the face instantly changes to a woman, and when the face direction is changed or the expression is changed, the change is expressed in real time as a 'female expression'. Photo sharing app

Snapchat has released a great gender exchange filter. Of course, conversion from female to male is also possible, and because of the high accuracy, it has become a function enough for new installers of Snapchat to be installed for filtering purposes. Chan is analyzing the depth of this filter.

Eric Jang: Fun with Snapchat's Gender Swapping Filter
https://blog.evjang.com/2019/05/fun-with-snapchats-gender-swapping.html

You can check what kind of 'sex exchange filter' released by Snapchat from the following. When the camera recognizes a male face, it is instantaneously converted to a female face. What's more, because the movements and expressions of the face are converted into women in real time, it seems that they are only shooting women.




There were also people who realized a one-person duet using a filter.




Zhang says in his blog, 'If anyone who has used machine learning algorithms sees this feature, it's magically powerful.'

What kind of system is it? So, Mr. Chan actually downloaded the app and actually checked what he could do and what he could not do with the gender exchange filter. First, in the following 3 photos, the middle is unprocessed Mr. Zhang, the left is 'male filter' and the right is 'female filter'.



One of the features of the filter is that it does gender exchange in real time without Internet connection. It is natural to express hair when you take off your hat. You can see the moment you take off your hat with GIF animation when you click. I put my hand on a hat .........



As you move the hat backward, the hair appears with a fluffy movement.



If you change the face orientation, you will find that there is an acceptable range of face orientations, as there is a moment when gender returns. It seems that the filter is applied when the condition of the boolean variable is satisfied.



The filter is applied regardless of the lightness or darkness of the place, but the hair cover does not fall on the person's body.



However, the hair is affected by the light source. The following picture shows that the hair on the left is reflecting light well. Since the filter reflects the light source, the situation after gender exchange looks very natural.



Well, how does the filter fail? First of all, it is out when an obstacle gets in front of the camera and face. When moving an obstacle from right to left ...



A woman's face changes to a male at the moment an obstacle covers almost half of the face.



Below is an example of moving an obstacle from top to bottom. Gender exchange filters are applied according to 'how much of the face is blocked' rather than 'if the important features of the face are blocked'. Before this is blocked by an obstacle.



It will change to such a feeling when it cuts off. The interesting thing is that the obstacles become translucent and the hair changes like a blonde just before the app judges 'false' to the question 'Do you want to replace the face?' 'This is an

artifact of the statistical data the app has trained.' 'Does blondes tend to make makeup tutorials,' said Chang.



Covering the cheeks with a black mask will still work, but the parts of the mask will not be converted to skin color. In addition, it can be seen that the range of the filter is limited to the rectangular area tracking the face because the hair is cut suddenly and linearly.



However, if you cover most of the face with the mask, the mask will not work properly. The round part exposed to the skin color of the eyes and mouth seems to be recognized as the 'face', and we are trying to create hair around it.



It became stable when the mask was removed and half or more of the face was exposed.



One of the things most impressed by Mr. Chang is the realism of the hair. According to Zhang's analysis, the hair is rendered as a Photoshop-like layer, but unlike layers that add cat ears etc. to photos, the

alpha channel is partially transparent. When you put your hand over the hair, you can see that the layer has clear segmentation because some appear under the hand and some appear over the hand. The gender exchange filter performs head tracking using a 2D alpha mask to process hair, and it appears that the user's head is located and the hair is expressed.



This filter is based on the billions of data Snapchat has collected so far, and it is Mr. Zhang's impression that 'the machine learning algorithm ' CycleGAN ' is not used?' CycleGAN is a style conversion method using GAN, which enables learning and conversion even from images with different shapes and positions. Zhang sees that CycleGAN is regularized by using a limited data set of ground and unpaired objects paired with ground truth.

In addition, face and hair are combined in separate layers, 'this is the first time I saw GAN being used in the alpha channel,' said Zhang. It seems that whether or not all hair is generated by GAN is a subtle place, but where changes in hair color due to changes in highlights or obstacles indicate that we are learning from data. On the other hand, there are doubts as to whether the hair expression is so stable that it is completely generated by GAN. There are few apps that replace Southeast Asian men's faces with East Asian women's hairstyles, and it can be inferred that a large library of hairstyles can be found here.

Related Posts:

in Software, Posted by darkhorse_log