San Francisco's Attorney General David Chiu has issued cease-and-desist orders compelling Apple and Google to remove 13 AI-powered nudify apps from their stores due to violations of California laws against deepfake pornography. These apps enable the creation of explicit, non-consensual images, often targeting women and children.
- San Francisco AG demands removal of 13 AI nudify apps for illegal deepfake content.
- Apps facilitate non-consensual sexualized imagery, primarily harming women and children.
- Apple and Google reportedly earned millions from app fees despite the controversy.
What happened
San Francisco Attorney General David Chiu sent cease-and-desist letters this week to Apple and Google, instructing them to remove a total of 13 nudify apps from their app stores. These apps utilize AI technology to manipulate photos by removing clothing or placing subjects in sexualized contexts, often without their consent. Chiu cited violations of California’s laws that prohibit services enabling deepfake pornography and emphasized the apps’ widespread availability and harmful impact.
Wired reported that five such apps were flagged on Google Play and eight on Apple’s App Store, though specific app names were withheld to avoid promoting them. One of the targeted apps alone had over one million downloads and openly marketed features designed to sexualize images of women. The AG's office estimates Apple and Google have collectively earned millions from fees generated by these harmful apps, raising concerns about their role in facilitating and profiting from abusive technology.
Why it matters
The actions underscore the increasing scrutiny app stores face for insufficiently policing AI-driven tools that produce deepfake pornography and non-consensual intimate images. Such tools perpetuate serious harms including humiliation, bullying, threats, and mental health crises, especially for women and children. Chiu highlighted the severe consequences victims endure, including damage to reputation and autonomy, and incidences of suicidal ideation.
Both Apple and Google maintain policies banning explicit content, but enforcement gaps remain evident. Google has suspended some apps identified by the AG for violating content policies and implemented search restrictions for related terms. Apple, meanwhile, has removed some flagged apps but has otherwise been less vocal in addressing the issue publicly. The case also exposes challenges in detecting apps that mask nudifying features as standard face-swapping tools, complicating efforts to protect users.
What to watch next
App stores will likely face continued legal and public pressure to bolster detection methods and enforcement against apps enabling non-consensual intimate imagery. The controversy follows broader concerns over AI misuse, including recent incidents like the Grok chatbot generating illegal content, which raises questions about liability and proactive platform governance. Apple and Google’s responses and transparency around removing harmful AI applications will be closely observed by regulators and the public.
Additionally, stakeholders will watch for potential legislative or regulatory developments targeting AI-generated deepfake pornography. Enhanced policies or mandated detection technologies could become necessary to prevent app stores from profiting off abusive content. Advocacy groups and victims’ rights organizations are also expected to push for stronger accountability measures within the mobile app ecosystem.