A new report by Rati has uncovered unsafe patterns in the rapid adoption of artificial intelligence (AI) tools. Specifically, it draws attention to the fact that “nudification” or nudify apps have created the simplest way ever to alter people’s likenesses. These tools are capable of removing clothing from photos. Consequently, they have ignited a wave of online vitriol that had once seemed too severe to exist. These same technological advancements have left young, urban, professional women in India feeling vulnerable and silenced. This has led them to reconsider their approach to their website.
The report shines a light on the uncomfortable reality for law enforcement in India as they try to tackle this online victimization. Authorities often describe the process of having abusive content removed from platforms as “opaque, resource-intensive, inconsistent, and often ineffective.” Almost half of all women continue to experience harassment with no recourse to address it. This concern is addressed in a companion report by Equality Now. This report reveals that women in India are increasingly being abused, shamed, and silenced online, often at the hands of AI-generated content.
Additionally, in India, deepfakes exist in a legal grey area. No specific laws currently recognize them as distinct forms of harm, although several existing laws could apply to cases of online harassment and intimidation. While these laws allow women to report AI-generated deepfakes, the system is still riddled with issues.
Rati’s insights show that nearly 1 in 10 cases reported to its helpline are of images edited with generative AI tools. Unlike AI-generated nonconsensual intimate images, this abuse tends to expose victims to a continuous cycle of re-victimization that traumatizes them over and over again. Rati states, “One of the abiding characteristics of AI-generated abuse is its tendency to multiply. It is created easily, shared widely and tends to resurface repeatedly.”
The report includes case studies, highlighting how these technologies are affecting people on the ground. Rana Ayyub, an internationally-known journalist, wrote about this environment and became the target of a vicious doxing campaign. Consequently, a wave of deepfake sexualized images of her began spreading on social media. Asha Bhosle, the legendary Bollywood singer, had her image and sound recreated through AI programming. This footage was later posted to YouTube, again without her consent.
Gaatha Sarvaiya, a recent Indian law graduate has resolved to do something about it. She has privatized her own social media to protect herself from threats of AI-generated violence. She expressed her concerns with the risks associated with posting personal photos on the web. She added, I’d think to myself, ‘What if it’s dangerous? What if people just take our pictures and use them for their own purposes?
As the report emphasizes, AI makes it easier than ever to produce hyper-realistic content. This low barrier to entry helps create a place where online harassment can be magnified in a myriad of ways. One other especially concerning example was a woman targeted simply for her extortion. After she uploaded a personal photo with her loan application, someone edited it into a pornographic image to profit off her likeness. The experience of victim blaming Osiecki’s victim reported feeling ‘shamed and socially tagged, as if she had been ‘participating in something unclean.
The chilling effect of this fear has larger consequences for the online presence of women more generally. Lakshané also pointed to a strong public sentiment against the misappropriation of photographs. This fear especially paralyzes women who work in the public eye, or dare to express political dissent. The result of this fear is that too often women end up silencing themselves or staying less active online.
Tarunima Prabhakar elaborated on this phenomenon: “The consequence of facing online harassment is actually silencing yourself or becoming less active online.” She added that this fatigue can lead women to withdraw completely from online spaces: “And the consequence of that fatigue is that you just completely recede from these online spaces.”
Platforms can ban accounts that are generating this sort of abusive content, but that content just reappears on a different account. Rati has referred to this unfortunate phenomenon as “content recidivism.” This pervasive atmosphere creates an even more hostile environment for women who want to express their opinions and post pictures online.