OpenAI is employing hundreds of contractors to read actual ChatGPT conversations to evaluate the chatbot’s responses, but the company has not clearly informed users about this human review process, potentially violating data privacy obligations set by the European Court of Justice.
- Hundreds of contractors read real ChatGPT conversations under Project Lily
- OpenAI has not clearly informed users about human review of chats
- Europe's top court holds companies must inform users before data collection
What happened
OpenAI has been employing hundreds of contractors to read real user conversations with ChatGPT and rate the chatbot’s replies as part of an internal effort known as Project Lily. These reviewers do not have access to usernames but can see a summary of users’ memory data, which sometimes reveals prior chatbot usage and approximate location. The company uses an automated Privacy Filter to remove personal data before human review, but the filter is not flawless and can miss uncommon identifiers.
When pressed by journalists over where OpenAI informs users about this human review, the company did not initially provide a clear answer. It later pointed to a support page. Similar practices exist at other firms like Anthropic and Google, which disclose limited human review under certain conditions. Meanwhile, Italy’s data protection authority has already fined OpenAI €15 million and mandated six months of public education for failing to adequately warn users about data processing.
Why it matters
The European Court of Justice has clarified that data controllers like OpenAI must inform users about personal data processing at the moment of collection. It is irrelevant whether contractors reading the data can identify individuals; the obligation rests entirely with the data controller before the transfer or processing. This ruling underscores how AI companies must ensure transparency and lawful legal bases for reviewing user inputs with human contractors.
This issue impacts millions of AI chatbot users who may not realize their conversations are subject to human scrutiny, raising privacy and trust concerns. Despite filters, residual personal data can still be exposed through reviewers seeing contextual memory summaries. Regulatory actions like the fine imposed in Italy signal tighter scrutiny and the need for better user communication and privacy safeguards in large-scale AI deployments.
What to watch next
OpenAI is advancing privacy options by allowing enterprise customers to opt out of data retention and human review, previewing zero data retention capabilities. However, the default setting for most regular and paid ChatGPT users includes data being used to improve models, with opt-out only affecting new conversations. The evolution of these settings and their adoption rates will be important to monitor as regulators continue enforcement and users demand stronger privacy guarantees.
Regulatory bodies across Europe and beyond are likely to increase scrutiny of AI companies’ transparency and legal compliance around human data review processes. OpenAI and its competitors may need to revise user disclosures, improve data anonymization technology, and adopt more granular consent mechanisms to align with privacy laws and rebuild consumer trust.