Why can voice AI respond so quickly? OpenAI explains the mechanism of 'GPT-Live'.
OpenAI has published how it built the real-time system supporting 'GPT-Live,' a voice AI that responds quickly to human conversations, in about six months. The article explains how the system allows users and the AI to talk simultaneously, how it allows conversations to continue while delegating complex processing to other AIs, and how it reduces the waiting time before communication begins.
How we built a realtime system for responsive voice AI in six months | OpenAI
If a voice AI starts responding mid-sentence, or if there's a long silence after the user finishes speaking, the conversation becomes unnatural. Traditional voice AIs generated responses only after detecting the end of the user's speech, which presented challenges: if the detection was too early, it would interrupt the conversation; if it was too late, the response would also be delayed.
GPT-Live removes a dedicated model for determining the end of speech from the speech processing path and employs 'full-duplex communication,' where the AI itself listens to the speech while speaking. Like a telephone, both parties can send and receive speech simultaneously, and GPT-Live makes decisions such as speaking, continuing to listen, waiting, and interrupting multiple times per second.
When searching or performing complex inference is required, GPT-Live continues the conversation while handing over the processing to a separate, high-performance model. Even while waiting for the results of the other model's processing, voice communication continues on a dedicated high-speed path, so the conversation is less likely to be interrupted even if the search or tool execution is delayed.
For long conversations, a backup model instance is prepared in the background, and the context of the conversation is loaded before switching to it. This design ensures that users do not experience any waiting time due to switching execution environments or context compression.
The low-latency communication technology 'WebRTC' is used for voice transmission. While standard WebRTC requires multiple communications to initiate a connection, OpenAI has developed WebRTC Abridged Roundtrip Protocol (WARP), which reduces the number of round trips required on the network to initiate voice and data communication from six to one.
The GPT-Live mechanism is used in ChatGPT Voice, and OpenAI states that it will provide a GPT-Live API in the future to extend voice interaction to more devices and applications.
Related Posts:
in AI, Posted by log1d_ts