Levine, Kumar Receive Oak Foundation Grant to Increase the Safety of Children Online

UMass Amherst College of Information and Computer Sciences Professor Brian Levine and Research Fellow Jagath Jai Kumar, along with Professor Hany Farid of the University of California Berkeley were awarded a $250,000 grant by the Oak Foundation to increase the safety of children online. The project will address a simple question asked by parents worldwide: "Is this app safe for my child?" Unfortunately, not enough information is available to parents from the app stores and third-party sources to guide their decision. 

A recent Pew study found that children younger than 12 are active on social media, including TikTok, Snapchat, Facebook's Instagram, and Facebook. For example, although the app is self-rated as 12+, the same study found that 5% of children 11 and younger in the U.S. use Facebook's Instagram. In 2019, TikTok removed 8.2 million videos per month for violating its own policies. Of these, about 26% involved adult nudity and sexual activities; another 25% of videos depicted harmful, dangerous, or illegal behavior by or of minors, including child sexual abuse material. TikTok reported that, despite moderation based on both machine learning and human review, it was unable to remove 10% (i.e., 800,000 per month) of these videos before they were viewed by users. 

At the same time, apps intended for children are used by predatory adults who portray themselves as children to meet potential victims. And some popular cartoon avatar games designed for young children are infiltrated with adults who dress their characters in lingerie and engage in adult situations.

The goal of the project is to quantitatively measure the presence of exploitation, grooming, and content inappropriate and potentially dangerous for young children on mobile apps. "With these measurements and analysis, the project will educate children, parents, industry, and policymakers about the privacy risks and personal danger children face from social networking apps and online interactions," says Farid. These results will also serve as an evaluation of the quality of the machine-learning-based moderation employed by industry platforms. 

This project aligns well with the Oak Foundation's existing programs on preventing child sexual abuse. Increasingly, child sexual exploitation begins for victims with an online interaction. Levine says, "An important strategy for addressing our child sexual abuse public health crisis is publicizing the dangers present in specific apps and internet environments.  In doing so, this project will seek to educate all and raise the bar for social responsibility in industry."