The Oversight Board, also known as Meta’s semi-independent policy council, has shifted its focus to examining how the company’s social platforms handle explicit, AI-generated images. In a statement released on Tuesday, the board announced that it will be launching investigations into two separate cases regarding Instagram’s practices in India and Facebook’s practices in the United States. These investigations were sparked by Meta’s failure to quickly detect and respond to AI-generated images featuring public figures that were deemed inappropriate.
In both of the aforementioned cases, the offensive content has since been removed. However, to protect the privacy and avoid further harassment of the individuals targeted by the AI-generated images, the board has chosen not to disclose their identities. According to an email sent by Meta to TechCrunch, this decision was made “to avoid gender-based harassment.”
The Oversight Board handles cases related to Meta’s moderation decisions, and users must first appeal to Meta before approaching the board. The board plans to publish their full findings and conclusions in the near future.
The Cases
Describing the first case, the board stated that a user had reported the AI-generated nude of a public figure in India on Instagram as pornography. The image was originally posted by an account that only featured AI-generated images of Indian women, and the majority of users who interact with these images are based in India.
Meta failed to remove the image after the initial report, and the ticket was automatically closed after 48 hours without further review. When the complainant appealed the decision, the report was once again automatically closed without any oversight from Meta. In other words, despite two reports, the explicit image created by AI remained on Instagram.
The user then appealed to the board, prompting Meta to take action and remove the offensive content for violating their community standards on bullying and harassment.
The second case involves Facebook, where an explicit AI-generated image resembling a U.S. public figure was posted in a Group focused on AI creations. In this instance, the social network swiftly removed the image as it had been previously posted by another user and flagged as “derogatory sexualized photoshop or drawings” in Meta’s Media Matching Service Bank.
When TechCrunch inquired about the board’s decision to select a case where the company ultimately removed the AI-generated image, the board explained that they choose cases “that represent larger issues across Meta’s platforms.” They elaborated that these cases provide insight into the global effectiveness of Meta’s policies and processes regarding various topics.
“We are aware that Meta’s ability to moderate content varies depending on the market and language. By selecting one case from the U.S. and one from India, we hope to determine whether Meta is equally protecting all women worldwide,” said Oversight Board Co-Chair Helle Thorning-Schmidt in a statement. “The Board believes it is crucial to investigate whether Meta’s policies and enforcement practices are effectively addressing this issue.”
The Problem of Deepfake Porn and Online Gender-based Violence
Some AI tools have recently been expanded to allow users to generate porn, although this is not the case with all AI tools. As reported by TechCrunch, groups such as Unstable Diffusion have attempted to monetize AI porn with questionable ethical lines and potential bias in data.
In countries like India, the increasing prevalence of deepfakes has become a major concern. According to a BBC report from last year, the number of deepfake videos featuring Indian actresses has significantly risen in recent times. Data suggests that women are often the subjects of deepfaked videos.
Earlier this year, Deputy IT Minister Rajeev Chandrasekhar expressed dissatisfaction with tech companies’ approaches to combating deepfakes.
“If a platform believes they can get away with not removing deepfake videos, or they do not take the issue seriously, we have the power to protect our citizens by blocking these platforms,” Chandrasekhar said during a press conference.
While India considers implementing specific laws related to deepfakes, nothing has been formally established yet.
Although reporting online gender-based violence is possible under Indian law, the process can be time-consuming and lacks support. In a study published last year by Indian advocacy group IT for Change, it was noted that courts in India need to have robust methods in place to address online gender-based violence and take these cases seriously.
Aparajita Bharti, co-founder of The Quantum Hub and a public policy consultant in India, believes that limits should be placed on AI models to prevent them from creating harmful explicit content.
“The primary issue with generative AI is that it significantly increases the amount of explicit content available, as it is relatively easy to generate and can be done with a high level of sophistication. In order to prevent the creation of such content, AI models must be trained to restrict output when the intent to harm is clear. Additionally, default labels should be implemented for easy detection,” Bharti explained in an email to TechCrunch.
Currently, there are only a small number of laws worldwide that address the production and distribution of AI-generated porn. A handful of U.S. states have legislation targeting deepfakes, and the UK recently passed a law to criminalize the creation of sexually explicit AI-powered imagery.
Meta’s Response and Next Steps
Meta has confirmed that both pieces of explicit content have been removed in response to the Oversight Board’s cases. However, the company did not address its failure to remove the content on Instagram after initial user reports, nor did they specify how long the content remained on the platform before being taken down.
The social media giant stated that they use a combination of AI and human review to identify sexually suggestive content and do not recommend this type of content in places like Instagram Explore or Reels recommendations.
The Oversight Board has requested public comments, with a deadline of April 30, to gather input on the issue of deepfake porn and the proliferation of such content in countries like the U.S. and India. They are also seeking insight into potential flaws in Meta’s methods for detecting AI-generated explicit imagery.
After reviewing public comments and investigating the cases, the board will publish their decision on the matter in the coming weeks.
These cases serve as a reminder that large platforms are still struggling to effectively moderate content while AI-powered tools have made it easier for users to create and share various types of content. Companies like Meta are continually experimenting with AI tools for content generation and detection, but offenders continue to find ways to bypass these systems and post problematic content on social media platforms.
