The Evolution of Disinformation Campaigns: AI’s Role in Creating Deepfakes
Disinformation is undergoing a significant transformation. Historically, simple memes — images, videos, or text snippets shared on social media — have been powerful tools in disinformation campaigns. Russia, for example, used memes to influence the 2016 U.S. election, while China employed similar tactics to target protesters in Hong Kong. These traditional disinformation methods, including fake news websites, fabricated stories, and provocative Facebook posts, have been effective in undermining confidence in U.S. elections and increasing divisions within the American electorate.
However, recent advancements in computer science and artificial intelligence (AI) have introduced a new and highly compelling method for spreading disinformation: deepfakes. Deepfake videos involve synthetically altered footage where faces or bodies are digitally modified to appear as someone else. These videos are becoming increasingly lifelike, raising concerns that this technology will significantly enhance foreign and domestic disinformation threats. The threat has already been realized in some areas, such as AI-enabled pornography targeting women.
While the full potential for deepfake-induced havoc has yet to be realized, the risk remains significant. For instance, some commentators feared that deepfake videos would disrupt the 2020 U.S. election, and although this did not happen, the risk persists for future elections.
The rise of deepfakes and related AI-generated fake content comes at a particularly vulnerable time for the United States and the global community. In their seminal report “Truth Decay: An Initial Exploration of the Diminishing Role of Facts and Analysis in American Public Life,” RAND researchers Jennifer Kavanagh and Michael D. Rich identify four key trends contributing to the declining importance of truth in society: increasing disagreement over facts and their interpretations, blurring the line between opinion and fact, the dominance of opinion and personal experience over factual information, and declining trust in traditionally respected sources of information. If these trends continue, they suggest that the audience for deepfakes will become increasingly susceptible.
This perspective aims to provide policymakers with a comprehensive overview of the deepfake threat. It begins by examining the technology underpinning deepfakes and associated AI-driven technologies that enable the creation of deepfake videos, voice cloning, deepfake images, and generative text. The perspective highlights the specific threats posed by deepfakes and explores factors that could mitigate these threats. It then reviews current efforts to detect and counter deepfakes and concludes with recommendations for policymakers. This analysis is based on an extensive review of the published literature on deepfake and AI-disinformation technologies.
Artificial Intelligence Systems in Disinformation Campaigns
Various AI technologies are increasingly being exploited for disinformation campaigns. While deepfake videos are a prominent concern, voice cloning, deepfake images, and generative text pose significant threats. This section reviews the technologies and capabilities underpinning these AI-based disinformation tools.
Deepfake Videos
Deepfake videos involve synthetically modified footage that alters subjects’ faces or bodies. These synthetic videos are created using generative adversarial networks (GANs). GANs consist of a generator that produces images from random noise and a discriminator that evaluates whether an input image is actual or generated. The adversarial nature of these components — comparable to a forger and a detective — enables the creation of high-fidelity fake images after sufficient training.
In spring 2021, a TikTok account (@deeptomcruise) released a series of highly realistic deepfake videos of Tom Cruise. These videos amassed over 15.9 million views and heightened public concern about the emerging era of deepfake disinformation. Crafting well-made deepfakes requires substantial computing resources, time, money, and skill. For instance, the @deeptomcruise deepfakes necessitated hours of authentic footage to train the AI models, a training period of two months, and a pair of high-end NVIDIA RTX 8000 graphics processing units (GPUs), each costing upwards of $5,795. Additionally, developers had to meticulously review the final footage frame by frame for any unnatural movements. The process also relied on a talented actor to convincingly mimic Tom Cruise’s mannerisms.
NOTE: As of April 12, 2022, this TikTok video had more than 16.1 million
views.
As technology advances, the cost and complexity of creating deepfake videos will decrease, making it easier to produce high-quality fakes. The deepfake videos of Tom Cruise followed earlier notable examples, such as a 2018 deepfake of Barack Obama using profanity and a 2020 deepfake of Richard Nixon delivering a speech he never gave. With each iteration, the realism of these videos improves, making them increasingly challenging to detect with the naked eye.
Deepfake Services
Numerous online platforms now offer access to deepfake technology. Popular sites include:
Reface: Allows users to swap faces in existing videos and GIFs.
MyHeritage: Animates photos of deceased relatives.
Zao: A Chinese app that enables users to impose their faces onto a selection of movie characters.
DeepNude: Notoriously allows users to upload photos (primarily of women) and receive an output where the subject appears nude.
These services democratize access to deepfake technology, raising significant concerns about their potential misuse in disinformation campaigns and beyond.
Voice Cloning
Voice cloning is another application of deepfake technology with significant potential for misuse. Various online and phone applications now enable users to mimic the voices of celebrities or other individuals. There are already notable instances of the malicious use of such services. In one case, the CEO of a UK-based energy firm received a phone call from someone who sounded like his boss at a parent company. Following the instructions given by the cloned voice, which was allegedly produced using voice-cloning software, the CEO transferred €220,000 (approximately US$243,000) to a Hungarian supplier’s bank account. In another instance, a man from Philadelphia reported falling victim to a voice-cloning scam. He received a call that he believed was from his son, who claimed to be in jail and needed money for a lawyer. Trusting the familiar voice, the man wired US$9,000 to a stranger. These examples highlight the dangers of voice cloning technology and its potential to facilitate fraud and deception.
Deepfake Images
Deepfake images are another significant cause for concern in AI-generated disinformation. These images often appear as highly realistic headshots nearly indistinguishable from genuine photographs. Websites like Generated Photos allow users to easily create fake headshots, making them readily accessible for malicious purposes.
A notable example of deepfake images involved a LinkedIn profile of “Katie Jones,” purportedly a Russia and Eurasia fellow at the Center for Strategic and International Studies. This profile, connected to a network of influential accounts, was part of a state-run espionage operation. Experts identified the profile photo as a deepfake image.
Deepfake images in fake social media accounts are becoming more prevalent. In one significant discovery, Facebook identified dozens of state-sponsored accounts using fake images as profile photos. Propaganda planners prefer fake images over stolen real ones because researchers can use tools like Google’s reverse image search to trace the origins of stolen photos. By using deepfake images, propagandists circumvent this defensive measure, making the photos untraceable.
Generative Text
AI can also generate realistic yet artificial text using natural language processing models. On September 8, 2020, The Guardian published an article titled “A Robot Wrote This Entire Article. Are You Scared Yet, Human?” This article was generated by OpenAI’s Generative Pre-Trained Transformer-3 (GPT-3), trained on vast datasets including CommonCrawl, WebText, Wikipedia, and a corpus of books. The Guardian provided GPT-3 with an introductory paragraph and specific instructions, resulting in eight essays edited into a cohesive article. The text was realistic enough to pass as human-written to an unsuspecting reader. Despite its capabilities, GPT-3 was not foolproof. For instance, a GPT-3-powered bot was released on a Reddit community, generating one post per minute for over a week. Some posts, such as advice to formerly suicidal users, were identified as autogenerated, highlighting the potential for large-scale text-based propaganda. Foreign adversaries could exploit text-generation programs like GPT-4 to produce divisive social media content or mass-produce fake news stories. FireEye researchers demonstrated this by training the old GPT-2 , a precursor to GPT-3, to mimic the social media posts used by Russia’s troll farms during the 2016 election. Additionally, adversaries might use text generators to create fake news, overwhelming genuine coverage of sensitive topics. An example is China’s use of this tactic to flood the #Xinjiang hashtag with posts about cotton production, diverting attention from human rights abuses in the region. These developments underscore the growing sophistication and accessibility of AI technologies in disinformation campaigns, presenting new challenges for detection and mitigation.
Risks and Implications
The risks associated with deepfakes and other AI-generated content are vast and varied, constrained only by one’s imagination. Society’s trust in video footage and the multitude of applications for such footage make it easy to envision numerous ways deepfakes could impact society and national security.
Here are four key ways adversaries or malicious actors might weaponize deepfakes:
1. Election Manipulation: Deepfake content could be used to manipulate elections. For example, on the eve of a closely contested election, a video could surface showing a candidate engaging in a scandalous or controversial act. Such a video could influence voters and sway the outcome of the election.
2. Exacerbating Social Divisions: Deepfake content could deepen social divisions. Russia has already used propaganda to divide the U.S. public. Within the U.S., the increasing partisan divide sees various propaganda-like tactics used to attack and defame opponents. Research has shown that online echo chambers reinforce partisanship, and deepfake content could further exacerbate this effect by presenting highly believable yet false narratives.
3. Eroding Trust in Institutions: Deepfake content could erode trust in institutions and authorities. For example, a fake but viral video of a police officer acting violently, a judge discussing ways to circumvent the judiciary, or border guards using racist language could all have devastating effects on public trust in these institutions.
4. Undermining Journalism and Information Sources: The advent of highly believable deepfakes means that even accurate video content or recordings can be dismissed as deepfakes by those who find the content unfavorable. This risk is particularly acute in developing regions with lower levels of education and literacy, fragile democracies, and existing interethnic strife. Deepfakes could exacerbate misinformation-fueled violence, such as the atrocities against Rohingya Muslims in Myanmar, violence against Muslims in India, and interethnic violence in Ethiopia. Additionally, deepfakes and AI-generated media may disproportionately harm women due to the prevalence of gendered content in pornography. Deepfake pornography can convincingly overlay a selected face onto that of a pornography actor, often without the subject’s consent, providing unlimited potential for abuse and exploitation. Such videos can pose broader national security threats by being used to embarrass, undermine, or exploit intelligence operatives, political candidates, journalists, or leaders. Research on the societal implications of deepfakes is still in its infancy. A systematic review of the scientific literature identified only 21 studies that used experiments to understand the true impact of deepfakes on users. The research yields conflicting results regarding users’ ability to detect deepfake videos and how much such videos influence them. For example, Nils C. Köbis, Barbora Doležalová, and Ivan Soraperra found that despite their confidence, users were routinely fooled by “hyper-realistic” deepfake content. Conversely, another study suggested that humans often detect deepfake content better than machines.
The Impact of Deepfake Videos
Deepfake videos can profoundly impact, often more significant than news articles or traditional disinformation videos. Research by Yoori Hwang, Ji Youn Ryu, and Se-Hoon Jeong (2021) found deepfake videos are perceived as more vivid, persuasive, and credible than fake news articles. Participants in their study were also more likely to share disinformation on social media when it included a deepfake video. This finding is supported by a comprehensive study conducted by Chloe Wittenberg, Ben M. Tappin, Adam J. Berinsky, and David G. Rand (2021), which involved over 7,000 participants. The researchers discovered that participants were likelier to believe an event occurred when presented with a fake video rather than fake textual evidence. However, the persuasive power of these videos was less significant than expected, producing only “small effects on attitudes and behavioral intentions.” The authors caution that deepfakes might be more persuasive outside a controlled laboratory setting. Still, they suggest that “current concerns about the unparalleled persuasiveness of video-based misinformation, including deepfakes, may be somewhat premature.” Another study by Barari, Lucas, and Munger (2021) found that deepfakes are no more likely than textual headlines or audio recordings to persuade large groups of people to believe in fabricated scandals. One anticipated impact of deepfakes is a general decline in trust in media, a concern validated by some research. Cristian Vaccari and Andrew Chadwick (2020) used survey experiments to show that participants who viewed deepfakes felt more uncertain rather than outright misled. This uncertainty contributed to a reduced trust in social media-based news content.
Factors Mitigating the Use of Deepfakes
Several factors currently mitigate the harmful use of deepfakes. While numerous papers predict doomsday scenarios involving deepfakes, Tim Hwang of the Center for Security and Emerging Technology offers a more measured assessment of the associated risks.
1. Prevalence of “Shallow” Fakes: Although experts debate the future dangers of deepfakes, “shallow” fakes represent a more immediate threat. These are videos manually altered or selectively edited to mislead audiences. A notable example is a video that shows Speaker of the U.S. House of Representatives Nancy Pelosi slurring her words, edited to slow down her speech and make her seem intoxicated. Such videos can be highly realistic and practical because they confirm existing biases. As Hwang notes, deepfakes are less attractive for spreading false narratives when considering the technology’s costs and risks.
2. High Cost and Complexity: Creating high-quality deepfake videos requires substantial resources, including expensive equipment, extensive training data, specialized technical skills, and talented actors. This limits the use of deepfake technology to a small number of well-funded actors. For example, the creator of the Tom Cruise deepfake video noted that the one-click, high-quality deepfakes era has yet to arrive. While technology will eventually democratize access, only a limited range of actors can effectively utilize deepfake technology.
3. Time-Consuming Production: Deepfake videos take months to create, requiring extensive planning and reducing the number of situations where they can be practically used. This time requirement limits rapid-fire operations and opportunistic use. The extended production time also allows intelligence communities to detect and mitigate planned deepfake releases in advance.
4. Extensive Training Data Requirements: High-quality deepfakes require thousands of training images, often featuring celebrities or politicians with abundant publicly available footage. This requirement limits the ability to create convincing deepfakes of lesser-known individuals, such as intelligence agents. Adversaries might develop custom generative models that evade detection, holding them in reserve for critical moments like elections or symbolically essential events. However, deepfake videos are likely to be detected, especially those intended to have a significant impact.
Factors such as cost, time, technology, and aptitude suggest that perpetrators of deepfake attacks would likely be caught and face consequences, including international pressure or economic sanctions. Adversaries must weigh their decision-making’s political, economic, and security costs.
These mitigating factors are, however, time-bound. As technology advances, deepfake videos will become easier and faster to produce, requiring less training data. The day will come when individuals can create highly realistic deepfakes using only a smartphone app. Increasing realism will make detection more difficult, potentially increasing the number of actors creating and disseminating deepfakes and reducing the risk of detection and geopolitical consequences.
Ongoing Initiatives
Given the inevitable rise of deepfakes, various approaches are being explored to mitigate the threat to information integrity. These include:
Detection Technologies: Developing automated systems to detect deepfake content.
Provenance Technologies: Ensuring the authenticity of digital content through secure methods that trace its origins.
Regulatory Initiatives: Implementing laws and regulations to address the use and distribution of deepfakes.
Open-Source Intelligence Techniques (OSINTs): Utilizing tools and methods to identify and counter deepfake content.
Journalistic Approaches: Equipping journalists with the tools and skills to verify the authenticity of content.
Media Literacy: Educating the public to recognize and critically assess deepfake content.
These efforts collectively aim to preserve information integrity and mitigate the potentially devastating impacts of deepfakes.
Detection
One primary approach to mitigating the rise of deepfakes is developing and implementing automated systems to detect them. Deepfake videos are created using generative adversarial networks (GANs), where a generator produces images, and a discriminator determines whether the images are real or fake. Programs to enhance detection capabilities focus on building increasingly effective discriminators to identify deepfake content.
Significant investments have been made in detection technologies. The Defense Advanced Research Projects Agency (DARPA) has funded two essential programs: the Media Forensics (MediFor) program, which concluded in 2021, and the ongoing Semantic Forensics (SemaFor) program, which received $19.7 million in funding for fiscal year 2021 and requested $23.4 million for fiscal year 2022. Facebook’s “Deepfake Challenge Competition” involved over 2,000 participants developing and testing models for detecting deepfakes.
While detection capabilities have improved, so has the sophistication of deepfake videos, leading to an arms race favoring deepfake creators. For instance, researchers discovered in 2018 that deepfake subjects did not blink at average human rates. This insight was quickly incorporated by deepfake artists, who adjusted their models to create more realistic eye-blinking rates. As GANs continue improving, distinguishing between natural and synthetic images becomes increasingly complex, even for high-quality detectors.
Results from the Facebook deepfake-detection challenge illustrated this challenge. Detectors achieved only 65 percent accuracy on a “black box dataset” of real-world examples but performed better, with 82 percent accuracy, on a public dataset of deepfakes.
Recommended initiatives to enhance detection capabilities:
1. Supporting Detection with Data Repositories: Social media platforms could provide access to their vast collections of images, including synthetic media, as training data to keep detection programs updated with the latest deepfake technologies. For example, Google released an extensive database of deepfakes in 2019 to aid detection efforts.
2. Creating Radioactive Training Data: Radioactive training data, imbued with imperceptible changes, can leave identifiable marks on trained models, making the resulting deepfake content easier to detect. Experiments by Alexandre Sablayrolles and colleagues demonstrated high confidence in detecting the use of radioactive training data, even when only a tiny percentage was used.
3. Limiting Access to Detection Tools: The Partnership on AI suggested that publicly available high-tech detection tools enable adversaries to create undetectable deepfakes. Instead, a multistakeholder process should determine who gains access to these tools and training datasets.
4. Labeling Fake Content: Social media platforms need effective ways to label detected deepfake content. Methods could include watermarks, platform warnings, metadata warnings, or side-by-side fake versus authentic content comparisons. Labeling schemes are effective in mitigating the effects of misinformation. Research by Nathan Walter and colleagues found that social media interventions like real-time corrections, crowdsourced fact-checking, and algorithmic tagging can reduce misinformation.
Provenance
Another approach to mitigating deepfakes is through content provenance. The Content Authenticity Initiative (CAI), a collaboration between Adobe, Qualcomm, Trupic, the New York Times, and other partners, has developed a method to digitally capture and present photo images’ provenance digitally. This initiative enables photographers to use a secure mode on their smartphones that embeds critical information into the metadata of digital images. This secure mode employs cryptographic asset hashing to provide verifiable, tamper-evident signatures, ensuring that the image and metadata have not been unknowingly altered.
When photos taken with this technology are shared on news sites or social media platforms, they include a visible icon — a small, encircled “i.” Clicking on this icon reveals the original photo, identifies any edits made, and provides information such as the time and location where the photo was taken and the type of device used. Initially developed for still images and videos, this technology will eventually extend to other forms of digital content. While not a complete solution for deepfakes, it offers viewers confidence that an image has not been synthetically altered and helps reputable news organizations build public trust regarding the authenticity of their content. The effectiveness of this technology depends on its adoption when the photo is taken, making widespread promotion and usage crucial. In January 2022, the Coalition for Content Provenance and Authority (C2PA) established technical standards to guide the implementation of content provenance for creators, editors, publishers, media platforms, and consumers. C2PA, which unites the efforts of CAI and Project Origin, aims to promote the global adoption of digital provenance techniques.
Open-Source Intelligence Techniques and Journalistic Approaches
Open-source intelligence Techniques (OSINTs), journalistic tools, and tradecraft provide critical methods for addressing the deepfake problem. These approaches aim to develop and disseminate open-source tools that can identify deepfakes and other disinformation-related content. Such tools are particularly vital for journalists from small to midsize news organizations, who rely on them to verify the authenticity of reported content. Additionally, OSINTs and related tools are essential for civil society actors involved in fact-checking and educational efforts.
One widely used tool is reverse image search. By capturing a screenshot of a suspicious image or video and running it through Google’s or a third party’s reverse image search platform, users can help validate its authenticity. Identical search results suggest the content is genuine, while discrepancies may indicate manipulation. However, enhancing the accuracy and quality of search results is necessary for more efficient use of this tool.
In his blog Witness, Sam Gregory highlights several open-source tools for forensic analysis and “provenance-based image verification.” For example, FotoForensics can identify elements added to a photo and Forensically offers tools for clone detection, noise analysis, and metadata analysis to support image forensics. InVID provides a web extension allowing users to freeze-frame videos, conduct reverse image searches on video frames, magnify frozen video images, and more. The Image Verification Assistant aims to build a “comprehensive tool for media verification” with features such as image-tampering detection algorithms, reverse image search, and metadata analysis.
Another notable tool is Ghiro, a “fully automated tool designed to run forensic analysis over massive sets of images, using a user-friendly web application.” These tools collectively empower journalists and civil society to combat the spread of deepfakes and disinformation effectively.
Implications and Recommendations
Based on a brief review of the technology and related issues, here are five recommendations for addressing the threat of deepfakes:
1. Adversarial Use and Deterrence: The use of deepfakes by adversaries involves a complex decision-making process that balances opportunities, benefits, and risks. The United States should conduct wargames and develop deterrence strategies to influence the decision-making of foreign adversaries. The intelligence community should also invest in strategies to gather intelligence on adversary efforts to develop and deploy deepfake technology, providing early warnings and mitigating risks before the content is created.
2. Investment in Detection Technology: The U.S. government, the research community, social media platforms, and private stakeholders should continue enhancing detection technology. It includes creating a “deepfake zoo” of known deepfake content to inform detection technology development. The government should work with the private sector to increase radioactive datasets of video content, making trained deepfake videos more easily detectable. Researchers should also explore best practices for labeling deepfake content. Additionally, limiting access to high-performance deepfake detectors may be necessary through a strategic reserve or a multistakeholder deliberation process.
3. Media Literacy Efforts: Media literacy efforts should continue on two fronts. First, broad media literacy skills should be promoted to build resilience against disinformation, with evidence-based training implemented at multiple levels, including school curricula and online interventions. Second, efforts should continue to directly warn audiences about the realities of deepfake technology and its potential use in disinformation. Media literacy interventions must sow mistrust in non-provenance-based video content while promoting trust in provenance-based content. Various actors should support these efforts, including news organizations, social media platforms, civil society groups, state and local governments, and the U.S. government.
4. Development of New OSINTs: Efforts to develop new Open-Source Intelligence Techniques (OSINTs) must continue to assist journalists, media organizations, civic actors, and other nontechnical experts in detecting and researching deepfake content. Essential tools include high-quality GAN-based detectors, enhanced reverse video search capabilities, cross-platform content trackers, and network-mapping tools. These tools should be accessible and user-friendly for non-technically trained individuals in the U.S. and abroad. The U.S. government should invest in these technologies through programs like the Networking and Information Technology Research and Development program. Major technology industry players, mainly social media platforms, should fund tool development and promote their utility and availability.
5. Adoption of Provenance-Based Approaches: Expanding the adoption of provenance-based approaches is crucial. The Coalition for Content Provenance and Authority (C2PA) has developed the necessary technical specifications, and stakeholders should promote the global adoption of this technology. The Deepfake Task Force Act, introduced by Senators Portman and Peters, is one potential approach to further this goal. The continued focus of both the White House and Congress on advancing content provenance initiatives can play a critical role in countering the harmful impact of deepfakes by enhancing transparency and building public trust.
Conclusion
The disinformation landscape is rapidly evolving, with deepfakes and other AI-generated content representing the latest frontier in digital deception. While traditional methods like memes and fake news have been effective in sowing division and undermining trust, the advent of deepfakes introduces a far more sophisticated and potentially devastating tool for disinformation campaigns. The implications of deepfakes are profound, posing risks to national security, social cohesion, and the integrity of democratic processes. These AI-generated fabrications can manipulate elections, exacerbate social divisions, erode trust in institutions, and undermine journalism. Moreover, the technology’s misuse can uniquely impact vulnerable groups, particularly women, through deepfake pornography and other forms of exploitation. Despite these threats, several mitigating factors currently limit the widespread harmful use of deepfakes, including the high cost and complexity of creating high-quality deepfakes and the extensive training data required. However, these barriers will likely diminish as technology advances, increasing the potential for deepfake misuse. A multifaceted approach is necessary to address this growing threat. Investment in detection technologies, the development of provenance-based tools, enhanced media literacy, and the creation of new OSINT techniques are crucial. Additionally, regulatory initiatives and collaborative efforts between the government, private sector, and civil society will play a vital role in combating the spread of deepfakes.
By understanding the evolving nature of disinformation and implementing comprehensive strategies to counter deepfakes, stakeholders can better protect the integrity of information and build resilience against the challenges posed by this new era of digital deception.