May 08, 2023 17:00:00

'ChatVRM' that allows you to talk with 3D characters on the browser is open source

As a front-end engineer at

pixiv , Mr. @keshigomu, who is involved in the development of a viewer for displaying 3D characters for VRoid Hub , a 3D character posting and sharing platform, has created a technology demonstration called ChatVRM that allows you to talk to 3D characters on your browser. ” and released the code as open source.

``ChatVRM'' that allows you to talk with 3D characters on the browser has been released as open source-pixiv inside
https://inside.pixiv.blog/2023/04/28/160000

'ChatVRM' is a technology demonstration in which a character responds to text or spoken words in full voice. It is possible to operate on a web browser, import 3D character data, and adjust the voice according to the character.

ChatVRM demo can be run from the following page. However, it is necessary to obtain the OpenAI API key independently, so be careful.

Application error: a client-side exception has occurred
https://pixiv.github.io/ChatVRM/

When you access the page, the following screen will be displayed, so click 'OpenAI site'.

Then, it will lead to

OpenAI's API issuance page , so click '+ Create new secret key'.

Give an arbitrary name and click 'Create secret key'.

Then an API key will be issued, so copy this.

Return to the ChatVRM demo page, paste the copied API key in the text box labeled 'OpenAI API key', and click 'Enter API key to start'.

Then the 3D character is displayed like this.

You can chat with the 3D character by pressing the microphone button at the bottom of the screen or clicking the text box and entering text.

You can see what it looks like when you actually talk to a 3D character with ChatVRM by watching the video below.

'ChatVRM' that allows you to talk with 3D characters on the browser - YouTube


From 'Settings' in the upper left of the screen, it is possible to import 3D characters in VRM format, adjust Koeiro API used as the voice of 3D characters, and specify character settings.

You can download 3D character data in VRM format by searching for a model on VRoid Hub that allows 'use by non-model registrants'.

In addition, ChatVRM publishes the code as open source on GitHub.

GitHub - pixiv/ChatVRM
https://github.com/pixiv/ChatVRM

The technical details of ChatVRM are summarized on the following page, `` @pixiv/three-vrm '' for displaying 3D characters, `` Web Speech API '' for recognizing user speech, and `` ChatGPT ”, and the use of “ Koeiro API ” for generating read-aloud voice are explained.

``ChatVRM'' that allows you to talk with 3D characters on the browser has been released as open source-pixiv inside
https://inside.pixiv.blog/2023/04/28/160000