The source code of 'Kalido Kit' that can move 3D avatars by tracking the movement of face and body with a webcam is now available



The source code of 'KalidoKit ', a tool that tracks facial expressions, body and hand movements and reflects them in 3D models, has been released on GitHub. By using this Kalido Kit, it is also possible to reflect the movement of yourself taken with the webcam on the 2D model or 3D model and operate the VTuber avatar.

GitHub --yeemachine / kalidokit: Blendshape and kinematics solver for Mediapipe / Tensorflow.js face, eyes, pose, and hand tracking models.

https://github.com/yeemachine/kalidokit



KalidKit is the core technology of 'Kalidoface 3D ' that allows you to become a VTuber by linking the movement captured by the camera with a 3D avatar on the browser. You can understand what Kalidoface 3D looks like by reading the following article.

'Kalidoface 3D' review that anyone can easily incarnate beautiful girl VTuber on the browser for free --GIGAZINE



The image looks like the following. Taking the reality of the body in the camera, from the movement of the extracted face and body MediaPipe or TensorFlow.js using the do the pose estimation. Then, KalidoKit gives feedback to the 3D avatar.



KalidoKit consists of three classes that calculate face, pose, and hand, and receive key points from Facemesh , Blazepose , Handpose, and Holistic.

[code] // Accepts an array (468 or 478 with iris tracking) of vectors
Kalidokit.Face.solve (facelandmarkArray, {
runtime: 'tfjs', // `mediapipe` or` tfjs`
video: HTMLVideoElement,
imageSize: {height: 0, width: 0},
smoothBlink: false, // smooth left and right eye blink delays
blinkSettings: [0.25, 0.75], // adjust upper and lower bound blink sensitivity
});

// Accepts arrays (33) of Pose keypoints and 3D Pose keypoints
Kalidokit.Pose.solve (poseWorld3DArray, poseLandmarkArray, {
runtime: 'tfjs', // `mediapipe` or` tfjs`
video: HTMLVideoElement,
imageSize: {height: 0, width: 0},
enableLegs: true,
});

// Accepts array (21) of hand landmark vectors; specify'Right' or'Left' side
Kalidokit.Hand.solve (handLandmarkArray, 'Right');

// Using exported classes directly
Face.solve (facelandmarkArray);
Pose.solve (poseWorld3DArray, poseLandmarkArray);
Hand.solve (handLandmarkArray, 'Right'); [/ code]

You can see how the Kalido Kit is actually used to feed back the movements of the face and body to the 3D avatar in real time in the following movie released by the developer Rich.

Related Posts:

in Software,   Video, Posted by log1i_yk