OpenAI revealed last week the “system card” of the GPT-4o model from ChatGPT, detailing both its limitations and the security testing procedures carried out. One of the most notable points is the model’s ability to mimic voices, which, in a strange case, managed to reproduce the voice of one of the red teamers hired to test the AI’s security, without their consent.
The advanced voice mode of ChatGPT allows users to have voice conversations with the AI assistant, using predefined voices based on various dubbing actors and actresses. However, in an episode documented by OpenAI in the GPT-4o system card, the model replicated the voice of a tester after receiving an audio input with some noise, bypassing the safeguards implemented by the company.
OpenAI explains that GPT-4o can synthesize any sound, including voices, based on small audio clips. This capability is controlled by an output classifier that ensures the model only uses preselected voices. Despite these controls, the noise in the user’s audio input in this example confused the model, causing it to mimic the user’s voice instead of the authorized voice.
The system card reveals that OpenAI takes these risks seriously, ensuring that their system catches 100% of significant deviations in voice. Simon Willison, an independent researcher, commented that OpenAI has implemented robust protection to prevent the use of unauthorized voices, but warns that in the near future there could be very similar technologies. “ElevenLabs can already clone voices, and soon we will see models that enable this capability on our own machines,” said Willison, highlighting the rapid evolution of voice synthesis technology thanks to new AI models.
Source: Ars Technica
