May 27, 2023 23:06:00

I tried using 'Kalidoface Lab', an app that allows anyone to become a VTuber by tracking facial expressions and movements using only a webcam

To become a VTuber, you need a 3D model as well as a tracking device and software to read your facial movements and expressions. However, the cost of preparing a 3D model and tracking device and software is a drawback. Google AI researcher

Rich has developed a free web app called Kalidoface Lab , which allows you to easily become a VTuber by tracking your facial expressions and body movements using only a webcam, so I tried it out.

Kalidoface Lab
https://lab.kalidoface.com/

When you access Kalidoface Lab from the link above, a 3D avatar will be displayed on the screen. To perform tracking using a webcam, click the camera button on the right side of the screen.

If you are asked for permission to use the camera, click 'Allow.'

Now, you can make facial expressions and move your face while facing the webcam, and your movements will be reflected on your 3D avatar.

Below is a video of the 3D avatar actually moving around. In addition to moving the face left and right and turning the neck, the mouth opening and closing, eyes opening and closing, and gaze can also be reproduced.

I tried using 'Kalidoface Lab' which allows you to become a VTuber with just a webcam - YouTube


Kalidoface Lab also allows you to change the pose of your 3D avatar. To switch poses, click 'Preference' in the upper left corner of the screen, select 'Avatar', and click 'Resting Pose'.

A pose selection box will then appear, so click on it.

There are various poses available for selection, such as 'sitting' and 'operating a smartphone.' Select 'Sitting' and click 'Done.'

The 3D avatar's pose changed as instructed.

Kalidoface Lab also allows you to load VRM files and control your favorite 3D avatars. To load a VRM file, click 'Upload' in the upper left corner of the screen.

When the file selection screen appears, select the VRM file you want to animate. In this example, we selected the VRM file for GIGAZINE's anthropomorphized character '

GIGAZINE-chan (tentative) .'

After waiting a little while, GIGAZINE-chan (provisional) was displayed on the screen. For some reason, the front and back were inverted, so I dragged the screen to adjust the camera direction.

He turned towards me.

Below is a glimpse of GIGAZINE-chan (provisional) in action. With Kalidoface Lab, you can move a 3D avatar without downloading any software, so it's useful when you have a 3D avatar but don't know how to move it.

I tried moving GIGAZINE-chan (provisional) using the free web application 'Kalidoface Lab' - YouTube