Ruma’s nightmare began in 2021 when she joined Telegram, a cloud-based encrypted instant messaging service, to receive movie ticket alerts. The very next day, she received explicit images of her face superimposed onto pornographic material, accompanied by humiliating and degrading captions.

“I felt my hands and entire body start to shake,” she said. “I can’t forget the images, no matter how hard I try.”

“Misogynistic phrases such as ‘slut,’ ‘cock holder,’ ‘toilet,’ synthesized onto my photos along with more photos of men’s genitals overlaid on my face were indescribably disgusting,” Ruma (a pseudonym) told Korea Pro.

She discovered that a separate room of 40 people on Telegram shared degrading images and messages of her, underscoring growing concerns about a new type of digital sexual crime using deepfakes.

The rapid advancement of artificial intelligence (AI) has fueled the spread of deepfake technology, which allows the digital manipulation of videos and images to represent individuals in ways they would not otherwise appear or act.

Perpetrators misuse this technology to synthesize images of acquaintances or others, particularly women, into pornographic materials and distribute them without the victims’ consent.

Ruma’s case is part of a nationwide story known as the “Seoul National University (SNU) Deepfake Case” or “SNU Nth Room,” referencing a previous case involving the spread of sexual exploitation videos on Telegram.

The case, only made public this year, involves Seoul National University graduates who created and distributed illegal synthetic materials using photos of at least 61 confirmed female victims on Telegram since 2019.

So far, only two SNU alums have been arrested.

Members of the Korea Women’s Hot Line put up signs in front of South Korea’s Supreme Court after the court confirmed a 42-year prison sentence for Cho Ju-bin, a South Korean criminal who was convicted as part of the “Nth Room” case, Oct. 14, 2021 | Korea Women’s Hot Line via Facebook

RISE OF DEEPFAKES

Deepfake technology, which has been around for decades, initially focused on rudimentary approaches such as tweaking video footage to show a person mouthing along to different audio. As technology advances, new AI tools have led to increasingly sophisticated deepfake programs.

While deepfakes are used constructively in education, film and TV production to boost accessibility and communication about important social development measures, the ability to generate deepfakes has become increasingly accessible. Some have exploited this rapidly evolving technology to fuel disinformation campaigns and crimes targeting specific individuals.

In South Korea, the government and law enforcement authorities emphasized the danger of deepfakes ahead of April’s parliamentary elections, highlighting the risks of AI in a misinformation environment already supercharged by political divides. 

However, the country has yet to reckon with the abuse of deepfake tools for digital sex crimes.

Data obtained by Korea Pro from Democratic Party lawmaker Ko Min-jung’s office and the Korea Communications Standards Commission (KCSC) reveals a nearly fourfold increase in requests for explicit videos to be blocked or removed, rising from 1,913 in 2021 to 7,187 in 2023.

The actual scale of the damage is presumed to be much larger, as many victims do not report the incidents or remain unaware they have been victimized.

A report by South Korean Supreme Court Justice Lee Suk-yeon, citing data from Home Security Heroes, a U.S. security-related startup, revealed that the number of deepfake videos distributed online worldwide in 2023 reached 95,820, a 550% increase compared to 2019. 

Among these, 98% were identified as deepfake pornography, with 99% of the targets being women and 53% being South Korean singers and actresses.

The effectiveness of efforts to combat deepfake sex crimes is limited. From June 2020 to April 2024, the KCSC’s removal rate for explicit fake videos stood at a mere 2.3%. In 2021, only 157 of the country’s 258 police stations had dedicated cyber investigation teams, with each cybercrime investigator handling an average of 200 cases.

An illustration of a person using the Telegram app on their smartphone | Image: Korea Pro

LACK OF LEGAL PROTECTIONS

Ruma told Korea Pro that simply deleting content or limiting access to deepfake images and videos already posted online is ineffective. She emphasized the importance of swiftly arresting perpetrators to prevent further incidents rather than merely removing photos from the public internet.

Criminal defense lawyer Choi Seok-ho of law firm Truth pointed out that while a special act addresses penalties for those who create and disseminate deepfakes, there is a lack of legal framework explicitly targeting the purchase or download of such content.

Regarding deepfakes involving adolescents, Choi noted that “cases involving minors could lead to more severe punishment under existing laws protecting children and adolescents from sexual exploitation.”

Korea Pro identified a Telegram room with over 460 members where the identities of at least four girls were explicitly exposed, along with sensitive information such as their schools, home addresses, birthdates and government ID numbers.

The group chat was active, with users offering to edit pictures of acquaintances and requesting sexually explicit edits on others’ images.

Telegram did not respond to Korea Pro’s inquiries about its efforts to report users, delete the room, or prevent further digital sex crimes, nor did it address repeated reports of suspected deepfake images of women within one of the chatrooms.

Choi clarified that even if a messenger service is based in another country, jurisdiction typically lies in Korea “if crimes involving Korean individuals occur on the platform or if the crime’s impact is within Korea.”

But he acknowledged that modifying sentencing could be challenging, as “increasing penalties for one crime may necessitate reevaluating penalties for others.”

Noelle Martin, an award-winning activist, lawyer and researcher in the University of Western Australia Tech and Policy Lab who was targeted by perpetrators fabricating and distributing fake pornographic images and deepfakes of her at the age of 17, told Korea Pro that digitally altering women’s images into non-consensual sexualized content is not a new issue but has recently surged due to several factors.

She highlighted the lack of effective regulation on digital platforms that host and profit from deepfake abuse and failure to remove related content and advertisements.

“The core of the problem is the dehumanization and degradation of women, the failure to see women as people with sexual agency and bodily autonomy and the treatment of women as objects, commodities and means to others’ ends,” Martin said.

“We are also dealing with a lack of accountability for all actors in the deepfake pipeline, including payment processors, such as Visa and MasterCard, who facilitate the trading of this material. And search engines, such as Google, who direct traffic to these services.”

Ruma told Korea Pro that since the incident, she has become extremely sensitive to meeting new people, has isolated herself from many friends and is continuously receiving psychological counseling for complex post-traumatic stress disorder.

She changed her number, deleted her Kakaotalk messenger application, stopped using her real name and avoids gatherings where she might have to exchange numbers with new people, especially men.

The Korean Women Workers Association and the Telegram Sexual Exploitation Joint Countermeasure Committee hold a joint press conference and protest digital sexual exploitation, Nov. 29, 2020 | Image: Korean Women Workers Association

TACKLING DEEPFAKE CRIME

In response to the potential use of generative AI to spread disinformation ahead of April’s parliamentary election, South Korean authorities cracked down on reported deepfakes earlier this year, particularly those targeting the government and politicians.

Last December, South Korea amended the Public Official Election Act with Article 82-8, which prohibited politicians from using deepfake videos 90 days before election day.

When a suspected deepfake video of South Korean President Yoon Suk-yeol speaking disparagingly about his government spread online before the 22nd National Assembly election, the Korean National Police Agency and KCSC took action to delete and block the video. 

Post-election reports indicated that 388 confirmed deepfake videos appeared during the campaign period.

However, deepfakes are not just a matter for politicians and public figures in South Korea. A petition urging the passage of a law mandating clear labeling of AI-generated content to prevent unethical use was filed and signed by over 50,000 people within a month in May.

One potential way to counter emerging deepfake technologies is to leverage other technologies to detect manipulated videos and photos. The Korean Policy Agency announced in early May that it developed deepfake detection software with an accuracy rate of approximately 80% when verifying authenticity.

Although recent media reports have questioned the software’s effectiveness, the police maintain that their response will continue to adapt to the constantly evolving threats.

“Since AI technology itself is newly developed, detection technology also needs to be continually developed,” Im Yoon-sang, the head of Cyber Sexual Violence Investigation at the Korean National Police Agency, told Korea Pro. 

Activist and lawyer Martin told Korea Pro that countries could do more to tackle this global and borderless problem.

“Even if deepfake laws are introduced, that’s just one step. The real test will be how well they operate in practice, and that will depend on how well these abuses are investigated by law enforcement and how seriously they are treated by courts,” she stated.

Shreyas Reddy contributed to this analysis. Edited by John Lee