Silicon Valley’s $117 Million Blind Spot: The Purge of AI Nudify Apps

Key takeaways

• San Francisco City Attorney David Chiu has issued a formal order for Apple and Google to remove 13 specific AI deepfake apps.
• A Tech Transparency Project report revealed that nudify apps have generated $117 million in revenue and 705 million downloads, highlighting a massive moderation failure.
• App store algorithms, including autocomplete and search ads, have been found to actively steer users toward these harmful applications.
• New 2025 California laws are providing the legal framework to hold tech platforms liable as facilitators of nonconsensual content.

The Digital Wild West Meets the Law

Your smartphone is currently hosting a multibillion dollar industry that thrives on the nonconsensual digital stripping of others, and for the first time, city officials are putting the tech giants on notice. San Francisco City Attorney David Chiu issued a series of cease and desist letters to Apple and Google this week, demanding the immediate removal of 13 specific nudify apps. These applications use artificial intelligence to generate nonconsensual deepfake pornography, a practice that Chiu asserts violates California law criminalizing the facilitation of such content.

While Apple and Google have long maintained that such apps are strictly forbidden, the scale of the problem is far larger than previously admitted. According to a report by the Tech Transparency Project (TTP) released earlier this year, the systemic failure of app moderation has allowed these tools to flourish. TTP identified 55 nudify apps on Google Play and 47 on the Apple App Store. Collectively, these applications have generated a staggering $117 million in revenue and have been downloaded over 705 million times globally. This is no longer a niche corner of the internet; it is a mainstream, high profit sector of the app economy.

What Changed: From Self-Regulation to Civil Liability

The shift in tone from San Francisco officials marks a turning point in how tech platforms are held accountable for AI generated content. Previously, companies like Apple and Google were largely shielded by federal protections that limited their liability for what users did with apps on their stores. However, a new California law enacted in 2025 has changed the landscape, allowing victims to pursue civil actions against third party facilitators. The City Attorney is now arguing that by hosting these apps and taking a cut of the revenue, these companies are effectively aiding and abetting illegal activity.

The pressure is not just coming from local officials. As reported by CNBC and Business Today, Democratic senators have also begun questioning the presence of Elon Musk’s Grok AI on the X platform, which has been used to generate millions of nonconsensual sexual images. While Apple and Google have removed dozens of apps following the TTP investigation, critics argue that the platforms are playing a game of digital whack-a-mole rather than addressing the core of the problem.

The Algorithm Problem: Steering Users Toward Harm

One of the most damning findings in the recent investigation involves the app stores' own search and advertising algorithms. Researchers at the Tech Transparency Project found that autocomplete features and search ads were actually steering users toward these harmful tools. When a user searches for innocent terms or related keywords like "undress," the platforms often suggest nudify apps or display advertisements for them. This creates a feedback loop where the platforms are not just hosting the content, but actively marketing it to new audiences.

Context: How Nudify AI Works

Nudify apps use deep learning models to analyze a photograph of a clothed person and predict what they would look like underneath. By utilizing massive datasets of existing adult content, these algorithms can generate highly realistic, though entirely fake, nude images in seconds. Unlike traditional photo editing, this process requires no technical skill, making it a powerful and dangerous tool for harassment and digital abuse.

Why It Matters: The Human and Cultural Cost

This crackdown coincides with a broader cultural anxiety about the impact of AI on human expression and safety. During a recent meeting with OpenAI staff, author Dave Eggers warned that generative AI is effectively silencing an entire generation. Eggers argued that by providing instant, algorithmic output, tools like ChatGPT and deepfake generators discourage critical thinking and authentic creative expression. This cognitive offloading, as some experts call it, replaces the hard work of human thought with a homogenized, instant product.

Furthermore, the geopolitical dimension of the AI race adds another layer of complexity. While Western regulators focus on safety, models like Moonshot AI's Kimi are emerging from China with advanced long context capabilities. According to analysis from TechCrunch, Kimi represents a significant competitive threat to Western models like ChatGPT because it can process massive amounts of text, potentially bypassing some of the filters and constraints found in US based systems. As regulators in San Francisco and beyond attempt to rein in the dangers of AI, they are doing so in an environment where global competition and domestic innovation are often at odds.

What to Watch Next

Expect the legal battle in California to serve as a blueprint for other states and nations. The European Union and the UK are already conducting separate investigations into how major platforms manage AI generated content. The next big test will be whether Apple and Google are forced to apply these same standards to major social media platforms like X, which currently host AI tools like Grok that have similar capabilities to the banned apps. As autonomous systems like Waymo continue to navigate the physical streets of San Francisco, the city is simultaneously becoming the primary laboratory for how we govern the virtual streets of the digital age.

The Final Word

The era of tech platforms claiming they are mere neutral pipelines is over. When algorithms profit from the degradation of individuals to the tune of $117 million, the definition of innovation must be called into question. The move by San Francisco is not just a legal maneuver; it is a demand for a new ethical standard in the age of generative AI. Whether the tech giants can truly sanitize their ecosystems without breaking the open nature of the app store remains the billion dollar question.