Home→Startups & Technology→OpenAI adds CSAM detection tools and ChatGPT warnings
Startups & Technology

OpenAI adds CSAM detection tools and ChatGPT warnings

OpenAI is introducing a tool to flag child sexual abuse material (CSAM), new ChatGPT warnings and temporary account suspensions, developer guidelines, and AI features for law enforcement as the company faces scrutiny over AI-related safety risks.

OpenAI adds CSAM detection tools and ChatGPT warnings

The image classifier is designed to identify visual signs of abuse, including in AI-generated images it has not encountered before. It assigns a score to help reviewers assess flagged material. OpenAI is offering select companies access through its API at no cost and said it developed the tool with the National Center for Missing and Exploited Children (NCMEC), while working with Tech Coalition and ROOST to share access.

In early partner testing, the classifier flagged 48% fewer images while identifying the same number of abusive images, OpenAI said. Images it flagged were twice as likely to be abusive as images flagged through other means, according to the company.

ChatGPT will display an alert when users try prompts seeking material that sexualizes children, direct them to specialized support, and suspend their accounts for 24 hours. The service already blocks CSAM and related requests; repeated violations could result in permanent removal, OpenAI said. The company developed its messages and guidelines with Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of the deterrent messages.

OpenAI is also publishing guidance to help developers prevent and detect CSAM and report it. The company said early testing helped some developers adopt child-safety tools and register to report material to NCMEC. For law enforcement, it is providing dedicated plugins, investigative tools, complex data analysis, investigator training, and a field guide. Access to the features will be managed by program administrators.

The announcements come amid lawsuits alleging that OpenAI’s technology has harmed users, particularly children. Common Sense Media has called the company’s technology an “unacceptable risk” for teens. OpenAI says it has made a series of safety changes over the past 12 months, including a safety blueprint, updated content guidelines, model adjustments to prevent harmful outputs, and a ChatGPT version for teens. Other chatbots, including Grok, have also been used to create nonconsensual sexualized imagery, including images of children.

Comments (0)

Leave a comment

No comments yet. Be the first!