San Francisco orders Apple, Google to remove nudify apps from app stores

The city of San Francisco has sued Apple and Google over mobile apps that use AI to turn ordinary uploaded photos into non-consensual explicit images. According to the city attorney's office, both companies aren't merely tolerating the apps by continuing to host them in their stores — they're profiting directly from purchase commissions and subscription fees generated by them.

So-called "nudify" apps use generative AI models to transform an ordinary photo of a person into a nude or sexually explicit image without that person's consent. The technology has increasingly been weaponized for bullying and harassment, particularly among teenagers, with victims spanning both adults and minors.

The lawsuit alleges that while Apple's and Google's app store policies explicitly ban this kind of content, enforcement in practice has been inconsistent. City officials say similar apps have repeatedly been removed only to reappear under different names — a cycle they argue points to a gap in the platforms' review systems.

At the heart of the legal argument is the claim that the platforms aren't merely passive intermediaries — they actively profit from this content through app-store commissions. That framing shifts the case from a pure content-moderation dispute into a question of the companies' financial accountability.

Apple and Google have previously said, in response to similar criticism, that they've tightened policies and removed apps found to be in violation. Critics counter that in an ecosystem reviewing millions of apps, malicious developers can easily dodge enforcement simply by changing an app's name and description.

The lawsuit is part of a broader legal push at the local and state level to regulate the misuse of AI tools. Many US states have passed laws in recent years criminalizing the creation of non-consensual synthetic explicit content, but holding app stores directly liable is a relatively new legal strategy.

Digital rights groups are watching closely to see how the case's outcome might reshape platforms' approach to content moderation. If a court finds that app stores can be held responsible not just for listing an app but for the revenue generated from it, that could set a precedent reaching far beyond this specific category of apps.

Critics note that the rapidly falling cost and rising accessibility of AI image-generation technology is making this kind of abuse increasingly difficult to detect. An app that disguises its "nudify" function, or markets itself as serving some other purpose, can more easily slip past automated review systems.

Victim advocacy groups are calling the lawsuit a turning point, arguing that accountability has, until now, fallen mostly on the individual developer who built the app — while this case puts major platforms' own accountability on the table for the first time.

Whatever the outcome, the development signals mounting legal pressure on major tech companies over the misuse of AI tools — pressure that is increasingly targeting not just the creators of harmful content, but the distribution channels that carry it.