Meta released AI 'SeamlessM4T' that can execute 'transcription', 'translation' and 'dubbing' simply by inputting Japanese voice, and a demo that can be translated into English and Chinese and can be used for free was also released, so I tried using it
AI `` SeamlessM4T '' that can execute `` transcription '', `` translation into another language '', and `` dubbing into another language '' just by inputting voice by Meta was released on August 22, 2023 (Tuesday). SeamlessM4T also supports Japanese voice input, and supports output to 35 languages including Japanese, English, Chinese, French, etc. A demo that anyone can use was also released, so I tried using the demo to see how accurately it could be transcribed and dubbed.
Introducing SeamlessM4T, a Multimodal AI Model for Speech and Text Translations | Meta
Introducing a foundational multimodal model for speech translation
https://ai.meta.com/blog/seamless-m4t/
There are multiple transcription AIs and translation AIs, but most of the existing AIs have only a single function such as ``transcription only'' and ``translation only''. SeamlessM4T, newly developed by Meta, is a multimodal AI that can perform multiple operations such as ``transcription'', ``translation into another language'', and ``dubbing into another language'' with just a single AI.
A demo that allows you to try the SeamlessM4T function is available at the link below, so I actually used it to try the Japanese voice processing performance.
Seamless Communication Translation Demo
https://seamless.metademolab.com/
When you access the demo, the following screen will be displayed, so click the blue button marked 'START DEMO'.
If you see a note about using the demo, carefully read
Then you can run the demo, so click 'START RECORDING'.
Click 'Allow' when asked for permission to use the microphone.
When 'Recording' is displayed on the screen, read the sentence you want to process into the microphone, and click 'STOP RECORDING' when you finish reading.
When the screen switches, scroll down to select the translated language.
When the translation language list is displayed, click the desired language and then click 'TRANSLATE'.
After waiting for a while, 'transcribed text', 'translated text into the selected language', and 'audio dubbed into the selected language' will be output. The dubbed audio can be played by clicking the play button.
By playing the following movie, you can check the sequence of steps from reading the sentence to translation and the accuracy of the dubbed voice. Both the transcription and translation results are good, and the dubbing results are also output clearly.
I tried translating Japanese voice into English and Chinese using Meta's transcription & translation & dubbing AI 'SeamlessM4T' - YouTube
Note that SeamlessM4T can also be installed and operated locally. Detailed installation instructions can be found at the link below.
GitHub - facebookresearch/seamless_communication: Foundational Models for State-of-the-Art Speech and Text Translation
https://github.com/facebookresearch/seamless_communication
Related Posts:
