September 18, 2023 | Lois Yoksoulian, University of Illinois News Bureau
New research led by data science experts at the University of Illinois Urbana-Champaign and found that there is no strong evidence that YouTube promoted anti-vaccine sentiment during the COVID-19 pandemic.
The study, published in the Journal of Medical Internet Research, performed an algorithmic audit to examine if YouTube鈥檚 recommendation system acted as a 鈥渞abbit hole,鈥� leading users searching for vaccine-related videos to anti-vaccine content.
For the study, the researchers asked World Health Organization-trained participants and workers from Amazon Mechanical Turk to intentionally find an anti-vaccine video with as few clicks as possible, starting from an initial informational COVID-19 video posted by the WHO. They compared the recommendations seen by these users to related videos that are obtained from the YouTube application programming interface and to YouTube鈥檚 Up-Next recommended videos that were seen by clean browsers without any user-identifying cookies.
The team then used machine learning methods to classify anti-vaccine content, analyzing more than 27,000 video recommendations made by YouTube.
鈥淲e found no evidence that YouTube promotes anti-vaccine content to its users,鈥� said Margaret Yee Man Ng, an Illinois assistant professor of journalism with an appointment in the Institute of Communications Research and lead author of the study. 鈥淭he average share of anti-vaccine or vaccine hesitancy videos remained below 6% at all steps in users鈥� recommendation trajectories.鈥�
The initial goal of the research was to better understand YouTube鈥檚 famously opaque techniques for content recommendations鈥攇oing beyond querying the platform鈥檚 application programming interfaces to collect real-world data鈥攁nd whether these techniques funnel users toward anti-vaccine sentiment and vaccine hesitancy.
鈥淲e wanted to learn about how different entities were using the platform to disseminate their content so that we could develop recommendations for how YouTube could do a better job of not pushing misinformation,鈥� said UN Global Pulse researcher Katherine Hoffmann Pham, a co-author of the study. 鈥淐ontrary to public belief, YouTube wasn鈥檛 promoting anti-vaccine content. The study reveals that YouTube鈥檚 algorithms instead recommended other health-related content that was not explicitly related to vaccination.鈥�
鈥淭he videos that users were directed to were longer and contained more popular content, and attempted to push a blockbuster strategy to engage users by promoting other reliably successful content across the platform,鈥� Ng said.
The study also allowed the researchers to examine how users鈥� real-world experiences differ from the personalized recommendations obtained by querying YouTube鈥檚 鈥淩elatedToVideo鈥� application programming interface. This API is designed to help programmers search for related content on the platform or using clean browsers, replicating the experience of a new user visiting YouTube with no search or view history, which is often used to study the platform鈥檚 recommendation system.
The study reports that the watch histories of users significantly affect video recommendations, suggesting that data from the API or a clean browser does not offer an accurate picture of the suggestions that real users are seeing. Real users saw slightly more pro-vaccine content as they advanced through their recommendation trajectories. In contrast, searches performed by the API or clean browsers during the study were drawn toward irrelevant recommendations as they advanced.
鈥淚 think one benefit of this study relative to others is that it proposes a relatively lightweight methodology to gather real data on how people navigate through YouTube鈥檚 video recommendations,鈥� Pham said. 鈥淪o unlike the APIs, which will just sort of randomly suggest new links, the users can critically review the links and pick one, which sort of mimics the behavior that many people would use on YouTube in reality.鈥�
Understanding recommendation systems is important because it promotes transparency and holds them accountable,鈥� added co-author Miguel Luengo-Oroz, a professor at the Telecommunications School of the Universidad Politecnica de Madrid. 鈥漈his helps people understand the choices being made for them by platform designers.鈥�
Ng is also affiliated with the , the , and the at Illinois.
Editor鈥檚 notes:
To contact Margaret Yee Man Ng, call 217-300-8186; email ymn@illinois.edu.
The paper 鈥淓xploring YouTube鈥檚 recommendation system in the context of COVID-19 vaccines: Computational and comparative analysis of video trajectories鈥� is available. DOI: