NewsDesk World

Press releases and business news from around the world

Published through our partners

OpenAI Adds CSAM Image Classifier and ChatGPT Account Suspensions

Artificial Intelligence · Updated ·

OpenAI is offering a free child abuse image detector to select companies. ChatGPT will also warn and temporarily suspend users who seek sexualized content involving children.

OpenAI Adds CSAM Image Classifier and ChatGPT Account Suspensions

OpenAI has unveiled a package of child safety tools, and the main one is a classifier that spots child sexual abuse material, including images made by AI. The release comes as worries grow about how AI products are used to harm children.

The package has four parts: an image classifier, warnings and temporary blocks inside ChatGPT, guidelines for developers, and tools for law enforcement.

The pressure behind the move

OpenAI faces several lawsuits that blame its technology for harming users, children in particular. The nonprofit Common Sense Media has labeled its tech an "unacceptable risk" for teens.

Other chatbots, Grok among them, have been used to make nonconsensual sexualized imagery, including of children. That is the backdrop for the new announcement.

Over the past 12 months OpenAI has released a safety blueprint, updated its content guidelines, retuned its models to avoid harmful output and launched a special version of ChatGPT for teens.

Warnings and a 24-hour suspension in ChatGPT

ChatGPT is getting new warning screens for users who try to reach material that sexualizes children. Anyone who sends such a prompt sees an alert and is pointed to specialized support to stop harmful behaviour. The account is also suspended for 24 hours as a warning.

The app already refuses CSAM content and related requests, so what changes is a penalty. OpenAI says that users who keep offending risk losing the account permanently.

The wording and approach were shaped with outside organizations and researchers. They include Protect Children, the Lucy Faithfull Foundation and Joel Scanlan of the University of Tasmania. Scanlan independently evaluated a trial of deterrent messages.

A classifier built with child-safety groups

A classifier is software that sorts content into categories. This one scans images it has never seen for visual signs of abuse and gives each a score. The score points reviewers toward how to deal with the material.

OpenAI is giving select companies access through its API at no cost. It developed the tool with the National Center for Missing and Exploited Children (NCMEC), and it is working with Tech Coalition and ROOST to share access.

In early tests with partners, the classifier flagged 48% fewer images yet caught as many abusive ones. Images it flagged were twice as likely to be abusive, which reduces the work for human reviewers.

Guidelines for developers and tools for police

OpenAI is also publishing guidelines for developers who build on its tools. They cover how to prevent, detect and help users report CSAM. In early testing, some developers used the guidelines to adopt child safety tooling and to register with NCMEC for reporting.

Investigators get AI-powered features too: plugins built for them, investigative tools and complex data analysis. Program administrators manage access. OpenAI will also train investigators and supply a field guide on using AI.

Source: TechCrunch