Meta's voice generation AI 'Voicebox' can 'read text in someone else's voice' without permission, but Meta avoids public release because it is too dangerous

Meta, the company behind Facebook and Instagram that also focuses on AI research, announced the voice generation AI ' Voicebox ' on June 16, 2023. Voicebox not only reads input text aloud, but also allows for potentially abusive operations such as 'editing parts of the audio' and 'reading text in someone else's voice.' Meta acknowledges the potential for abuse of Voicebox and is refraining from publicly releasing Voicebox's model data and code.

Introducing Voicebox: The Most Versatile AI for Speech Generation | Meta
https://about.fb.com/news/2023/06/introducing-voicebox-ai-for-speech-generation/

Introducing Voicebox: The first generative AI model for speech to generalize across tasks with state-of-the-art performance
https://ai.facebook.com/blog/voicebox-generative-ai-model-speech/

Voicebox is a speech generation AI that can perform operations such as 'reading input text in a natural voice,' 'recording someone else's voice and reading the input text,' 'recording someone else's voice and reading the input text with a specified intonation,' and 'recording someone else's voice and editing part of it.' You can see the high performance of Voicebox in one shot by playing the movie included in the tweet below.

Introducing Voicebox, a new breakthrough generative speech system based on Flow Matching, a new method proposed by Meta AI. It can synthesize speech across six languages, perform noise removal, edit content, transfer audio style & more.

More details on this work & examples ⬇️

— Meta AI (@MetaAI) June 16, 2023

Many demos of the feature 'record someone else's voice and edit parts' are available on the following page. When you play each demo, you can see that it is possible to edit with such precision that it is difficult to tell which parts have been edited.

Editing
https://voicebox.metademolab.com/edit.html

Furthermore, you can have the app read out long sentences by simply recording someone else's voice for about three seconds. On the demo page below, you can see that you can freely record sentences by simply recording a short audio recording, such as a tweet.

Zero-Shot TTS
https://voicebox.metademolab.com/zs_tts.html

Various demos of Voicebox reveal that it can read text in someone else's voice with extremely high accuracy, making it vulnerable to abuse. Meta also acknowledges the risk of Voicebox being misused and has refrained from publicly releasing Voicebox's training model and code. However, Meta states that it has developed an 'effective system for distinguishing between real voices and those generated by Voicebox,' and that in the future it may be possible to build a mechanism for distinguishing between real voices and AI-generated voices.