May 28, 2023 18:53:00

Motion capture technology 'SysMocap' that can reflect the movement of the face, hands and whole body in 3D avatar free & in real time

For VTuber (virtual YouTuber), AR, VR, motion capture technology ' SysMocap ' that reflects the movement of the face, hands and whole body taken with a web camera in real time to the 3D avatar has been released. It runs on Windows, macOS, and Linux, and can be easily installed in the local environment for free, so I actually tried using it.

GitHub - xianfei/SysMocap: A real-time motion capture system for 3D virtual character animating.

https://github.com/xianfei/SysMocap

Node.js must be installed to install SysMocap. Node.js installer can be downloaded from the following page.

Download | Node.js
https://nodejs.org/en/download

This time, click '64bit' of Windows Installer to download. The file size is 30.0MB.

Launch the downloaded installer.

Click 'Next'.

You will be prompted to agree to the license, so check 'I accept the terms in the License Agreement' and click 'Next'.

Since the place to install is displayed, click 'Next'.

Since the installation contents are displayed, click 'Next'.

Click Next.

Click 'Install' to start the installation.

When the installation is finished, click 'Finish' to close the installer.

Then type ' git clone https://github.com/xianfei/SysMocap.git ' in the terminal to clone the repository.

Type cd SysMocap to navigate to SysMocap's local repository.

Run it by typing ' npm i '.

Type npm start to run it.

Then the SysMocap window will start. The 3D avatar file formats supported by SysMocap are

VRM format , GLB format , and FBX format , and in addition to being able to load your own files, 9 types of avatars are included by default.

Select 'SETTING' from the top and select the camera in the 'Input' field.

Next, select 'MOCAP' from the top. Select the 3D avatar in '3D Model', select 'Camera' in 'Source', and click 'START'.

Then, loading is done and the image of the avatar and web camera is displayed as follows. My face and body in the webcam have bones that indicate recognition. Below the camera image are 'whole body', 'half body' and 'special shooting', and special shooting is a mode that recognizes bust up.

The avatar also moves in response to the direction of the face, the movement of the mouth and hands. Although the hand recognizes each finger, it was not perfectly reflected in the movement of the avatar.

'Half body' recognizes the movement of the upper body. Bones are also visible below the shoulder.

'Whole body' reads the movement of the whole body. The accuracy with which webcam recognition is reflected in the avatar is still not that high, but the rough movement of ``bending the left leg and extending the right leg'' was firmly reflected in the avatar.