Meet the 15-year-old deepfake victim pushing Congress into action
This article is from The Technocrat, MIT Technology Review’s weekly tech policy newsletter about power, politics, and Silicon Valley. To receive it in your inbox every Friday, sign up here.
I want to share a story about an inspirational young woman and her mother, who have stepped into the fray on AI policy issues after a horrific incident. Too often, new technologies disproportionately harm women and girls without drawing enough attention or bringing about change. In reporting out a recent story, I was so impressed with this family’s drive to fix that.
In October, Francesca Mani was one of reportedly more than 30 girls at Westfield High School in New Jersey who were victims of deepfake pornography. Boys at the school had taken photos of Francesca and her classmates and manipulated them with artificial intelligence to create sexually explicit images of them without their consent. (Westfield High School said in an email that “matters involving students are confidential” but claims that “far fewer” than 30 students were affected.)
The practice is actually stunningly commonplace, but we rarely hear such stories—at least in part because many victims of sexual harassment very understandably don’t want to talk publicly about incidents that are so private. But within just a day of learning about the violation, which she calls “shocking,” 15-year-old Francesca started speaking out and calling on lawmakers to do something about the broader problem. Her efforts are already starting to pay off with new momentum behind proposals for state and federal legislation, which I wrote about in a story published this morning. That includes a bill cosponsored by New Jersey state senators Jon Bramnick and Kristin Corrado that would establish civil and criminal penalties for the nonconsensual creation and sharing of deepfake porn.
Francesca and her mother, Dorota, say that their activism aims particularly to support women and girls who might not have their own ability to make change. I spoke with the Manis earlier this week to more deeply understand what this experience has been like, and I want to share parts of my conversation with them, which has been edited for length and clarity.
Could you walk me through what happened to you and how you found out?
Francesca: Every single girl was worried on October 20. Rumors were going around, people were worried, and all the girls thought they were going to be one of the AI victims. And at the end of the day it was confirmed by the administration that I was one of many AI deepfake victims. Of course, I was shocked, because me and the other girls were betrayed by our classmates. We didn’t know our own classmates would do that to us.
It’s really brave what you’re doing by speaking out and talking to all these people about your experience. What made you want to take action?
Francesca: Until I was one of the victims, I didn’t really know how complex and scary AI technology is. So it’s made me understand the importance of self-education in regards to the technology, because AI is here to stay, and we need to learn how to live with it without hurting ourselves and others. So this is actually why I have created a website called AI Help, which will help educate and protect us from AI. And this tool will provide resources that will help AI victims self-advocate. I also want to make sure that we have state and federal laws to protect us—children and women—from deepfakes, and that’s already being put into action.
What are some of the key things that you think other girls and women should know about the risks when it comes to AI?
Francesca: It’s important to know that it can happen to anyone, by anyone; like, it could be your classmates. That’s what happened to me. People should realize when they start posting stuff on Instagram or any type of social media that it can happen to you. Protect your image, make your account private, and have only certain followers on it, like people you know, instead of having a public account.
Can you tell me about the conversations you’ve been having with lawmakers about this issue?
Francesca: I spoke to Senator Bramnick—he’s actually from Westfield—to help me advocate for our state and to [make] new AI laws. We actually spoke in person and he promised that he will do all he can to protect our state from deepfakes. He also immediately cosponsored Senator Corrado’s bill. And if everything goes well with the joint effort, we’ll be able to protect New Jersey with an AI bill by January 2024. And this makes me incredibly happy, to know that my own town senator cares enough to fight for the important cause.
Congressman [Joe] Morelle [of New York] also invited us to Washington, DC, to meet with other congressmen [from both parties].
Dorota: We are hoping after our visit in Washington we’re gonna be able to have more and more support and make sure that we start with something. Then we can always better it.
What has this taught you about politics or the way the American government works?
Francesca: I was so happy to know that—well, when I reached out I was 14, I just turned 15—but I’m, like, a 14-year-old, and [the lawmakers] listened to me and they helped me. They were willing to protect me and other girls. Something I learned from this is to speak up and not to be afraid.
I know you filed a police report about this, but what is the legal recourse you are hoping for?
Francesca: I would really like for whoever’s doing this to be suspended or expelled, because I think it’s important for everyone to feel comfortable if it’s someone from your school. And I would also like an apology. I would forgive that person, but, you know, I would never forget.
Dorota, you mentioned you haven’t been pleased with the school’s response. Is there anything you would like schools to do differently to either respond to situations like this or prevent them from happening in the first place?
Dorota: I think education is so important on this matter—educating our children, educating ourselves—and then taking responsibility. I think we should use this [opportunity] to educate our girls that they’re worth it; even though they have been victimized, it doesn’t mean they should be ashamed and they should just accept things as they are and hope for them to pass. We have a wonderful school district. Our teachers are fantastic. Francesca wouldn’t be able to go through this whole situation without the support of her teachers. I cannot say good enough things.
But the administration simply is hoping for things to die down. I have no report. There’s no consequences, there’s no accountability, there’s no apology. I’m an educator myself. I own a private school in Jersey City. And I think as a mother and as a woman, I’m advocating for something different. I’m supporting my daughter, but as an educator, I’m advocating to create a safe place for our children, because it could happen to anybody. It does not necessarily have to be a woman, and I think we should send a clear message that this is not acceptable.
[In a statement to MIT Technology Review, Westfield superintendent Raymond González said, “The Westfield Public School District has safeguards in place to prevent this from happening on our network and school-issued devices. We continue to strengthen our efforts by educating our students and establishing clear guidelines to ensure that these new technologies are used responsibly in our schools and beyond.” The school also said it conducted an immediate investigation and is working with police.]
Do you think there’s a role for education that also teaches kids what’s appropriate and inappropriate, as well as how to protect themselves from the harms?
Dorota: Oh, absolutely. I think on many occasions [digital education] will be given to a, you know, physical education teacher, and they will be teaching the dangers of AI. Let’s be honest with each other: this is such a complex and sophisticated technology, and it’s ever changing. We should make sure that a specialist is teaching that class, and it shouldn’t be just at the beginning of the school year. It should be at least twice or three times, and it should be delivered in a meaningful way.
It really affects people’s lives. And Francesca has such a strong personality. She always has been a fighter. And I applaud her for speaking for herself. At the beginning, when she told me, Mom, I wanna fight, I said, Francesca, I want you to know that it can go either way. You’re going to hear people that are happy and you’re going to hear people that are really against you, and you have to be prepared for it. And she said, I am not a child. I can take people’s opinions and I want to speak up. But not everybody’s gonna have the same character as Francesca. Not everybody’s going to have the same support at home as Francesca. And there are going to be girls or boys who will not see the light at the end of the tunnel and will go for suicide or self-harm. And I don’t think we should wait for that. Education in a meaningful way is the most important thing.
Francesca: I also want to urge all school districts to update their policies on cyber-harassment to add a definition of AI and to add defined consequences if deepfakes are being created by a student. Laws can take time to be passed, but school policies can and should be updated immediately.
It feels like you really have created a lot of momentum. Have you had any negative reactions?
Francesca: No, not really. It’s so cool to know that I have such a great community and support from my friends and teachers, and I just want to thank them. I’m so proud to be an American. I live in a country where the voice of a now-15-year-old girl can make positive change.
What I am reading this week
- Instagram influencers in India are being paid by political campaigns to sway local elections. It’s part of a growing trend in which smaller and nonpolitical social media personalities are tapped for campaign messaging.
- The EU AI Act is entering its final stage of negotiations, and some people are worried that the discussions are not going great. The EU Parliament, Commission, and Council have a preliminary deadline of December 6 to finalize the deal, and apparently tech lobbyists are making discussions sticky.
- The US government will no longer notify Meta when foreign disinformation campaigns are operating on its platforms. This reversal of a years-long policy is a result of a conservative legal campaign. And in the run-up to the US elections, it’s not good news for the health of the internet.
What I learned this week
Using AI to generate an image uses a ton of energy, according to a new research study from Hugging Face and Carnegie Mellon University. As my colleague Melissa Heikkilä reported, “Their work, which is yet to be peer reviewed, shows that while training massive AI models is incredibly energy intensive, it’s only one part of the puzzle. Most of their carbon footprint comes from their actual use.” The research marks the first time the carbon emissions associated with using AI for different purposes, like image or text generation, have been calculated.
