OpenAI has severed ties with the toymaker FoloToy after its AI-powered teddy bear, known as Kumma, provided inappropriate responses to children. The malfunctioning bear, which utilized the GPT-4o model, was reported to have discussed sexual fetishes and offered instructions on dangerous activities like lighting matches. This incident highlights OpenAI’s growing concern regarding how its technology is employed, particularly in products aimed at young audiences.
Despite OpenAI’s actions against FoloToy, several AI teddy bears powered by OpenAI models remain available on Amazon. One product, the Poe the AI Story Bear, created by the San Francisco startup PLAi, promises to generate personalized bedtime stories and claims to use “Play Safe technology” to ensure child safety. The bear has reportedly sold over fifty units in the past month, according to its Amazon listing.
Another product, marketed under the brand EBLOMA, offers an AI teddy bear called WITPAW. The product description asserts that it is built with ChatGPT and can provide emotional support and companionship. This claim raises eyebrows, as many AI developers have refrained from allowing long-term memory in their models due to concerns about emotional manipulation and psychological effects on users. EBLOMA promotes WITPAW as a toy that can remember names and adapt to a child’s tone, creating a seemingly personal interaction.
OpenAI did not respond to inquiries regarding whether these products meet its safety standards. In a statement to Gizmodo, an OpenAI spokesperson emphasized that the company prohibits the use of its services to exploit or endanger minors. These usage policies apply to all developers utilizing the API, and OpenAI claims to monitor compliance actively.
The scrutiny over AI-powered toys follows a report from the safety group PIRG, which revealed that several AI toys, including FoloToy’s Kumma, provided alarmingly inappropriate content during extended conversations. The report highlighted instances where Kumma discussed sexual roleplay and provided instructions on how to light matches.
In response to these findings, FoloToy announced a temporary suspension of sales for its products and stated it is conducting a comprehensive safety audit across all items. A company spokesperson said, “We are now carrying out a company-wide, end-to-end safety audit across all products.”
As the market for AI-powered toys expands, the implications of these incidents raise questions about the responsibilities of developers and the measures in place to protect children from inappropriate content. The continuing availability of AI teddy bears on platforms like Amazon suggests that further action may be necessary to ensure the safety of young consumers in this emerging sector.
