citizen polling is a crucial function of contemporary political campaigns and movements, but it has transformed significantly. Recent US election cycles have generated numerous analyses explaining both the victories and the deficiencies of public surveys. There are two primary causes for the shortcomings in polling.
Firstly, nonresponse has surged. It has become vastly more challenging to reach individuals than before. Few people complete surveys sent via postal mail these days. Few people answer their phone when an unfamiliar caller rings. According to Pew Research, the percentage of people willing to converse with them dropped from 36% in 1997 to only 6% by 2018. Pollsters across the globe have encountered similar obstacles.
Secondly, individuals do not always candidly share their thoughts with pollsters. Some conceal their genuine opinions out of embarrassment. Others act in a biased manner, expressing what they believe their political affiliation wants to hear—or what they know the opposing party does not want to hear.
Despite these weaknesses, an intense fascination with polling continues to dominate our politics. Headlines are more inclined to highlight the latest shifts in survey numbers rather than the substantial policy matters at stake in the campaign. This represents a tragedy for a democratic system. Elections should be viewed as decisions with repercussions for our livelihood and well-being, not as competitions determining who secures which desirable position.
Can Polling Benefit from AI Technology?
artificial intelligence (AI) holds the potential to revolutionize polling. AI can provide the capacity to swiftly survey and consolidate the expressed perspectives of individuals and communities online, grasp trends based on demographics, and provide projections for novel circumstances and policy topics to a level comparable with human experts. Future politicians will not fret about the outcomes of a survey conducted the previous week; they will simply inquire a chatbot about public sentiments. This would massively enhance our accessibility to real-time, detailed insights on public opinions, although it could also amplify concerns regarding the accuracy of this data.
It may seem far-fetched, but stay with us.
Substantial language models, the AI frameworks supporting tools like ChatGPT, are constructed on vast collections of data gathered from the Internet. These models are trained to replicate the responses of millions of individuals across a multitude of subjects, contexts, and scenarios. For over a decade, campaigns have scoured social media for clues on how individuals are reacting to current political events. Interacting with an AI chatbot is akin to conducting analytics on social media, with the distinction that they can provide novel responses to inquiries that have not been previously addressed. This enables the generation of additional data from too-small demographics, as well as an immediate opportunity to seek clarifications from the simulated constituents for a better grasp of their rationale.
Researchers and enterprises are already leveraging large language models (LLMs) to replicate polling outcomes. Current methodologies are rooted in AI agent concepts. An AI agent is an instance of an AI model programmed to act in a specific manner. For instance, it may be primed to respond as a person exhibiting particular demographic traits with access to news sources from designated outlets. Studies have organized groups comprising thousands of AI agents that respond as individual members of a survey group, akin to human panelists periodically called upon to answer inquiries.
The significant contrast between humans and AI agents is that the AI agents are always available, metaphorically speaking, regardless of the frequency of interaction. A political candidate or strategist can inquire an AI agent about voter inclinations towards supporting them based on adopting position A versus B, or variations thereof, such as policy A-1 versus A-2. They can pose such queries to distinct segments like male versus female voters, or further refine the probe to married male voters of retirement age residing in rural districts of Illinois with no college degrees, who faced unemployment during the previous economic downturn; the AI will factor in as much contextual information as requested.
The value of this system lies in its ability to extrapolate to fresh scenarios and polling themes and provide plausible responses, even though absolute accuracy is not assured. In numerous instances, it will predict these responses as competently as a human political expert. If the outcomes seem perplexing, the human analyst can promptly prompt the AI with a series of subsequent inquiries.
Enhancing the Accuracy of AI Agents as Survey Subjects
In experiments conducted with early versions of the ChatGPT model (GPT-3.5), we observed that it performed relatively well in mimicking human survey answers. The ChatGPT agents generally aligned with the responses of actual individuals across a variety of survey questions, including stances on abortion and approval ratings for the US Supreme Court. The AI-based polling results demonstrated average responses and distributions across demographic attributes like age and gender, mirroring those of genuine human survey participants.
Our most prominent shortcoming arose in a query related to US involvement in the Ukraine conflict. In our tests, the liberal-biased AI agents mainly opposed US intervention in Ukraine and compared it to the Iraq war, while conservative AI agents favored aggressive stances supporting US intervention. This outcome complies with historical political equilibrium in US foreign policy at the decade’s outset, although it is contrary to current political sentiments.
This error primarily boils down to timing. Humans provided input after Russia’s full-fledged invasion in 2022, while the AI model was only trained on data up to September 2021. The AI erred due to a lack of awareness about evolving political dynamics. The model lacked adequate context on crucial recent events.
We believe that AI agents can surmount these limitations. While AI models rely on the training data available and its inherent constraints, the unique aspect of AI agents lies in their capacity to harness and integrate new data automatically upon receiving queries. AI models can update the context for opinion generation by assimilating data from the same sources as humans. Each AI agent within a simulated group can be exposed to identical news sources and social platforms as individuals from the corresponding demographic before responding to survey queries. The system operates effectively as AI agents can carry out multiple steps, such as reviewing a query, fetching information from a defined database (like Google, the New York Times, Fox News, or Reddit), and formulating responses.
In this manner, AI polling tools can replicate the process of enlightening their synthetic survey group with the most pertinent news concerning a subject, tailored to each AI agent’s prevalent information sources. Furthermore, they can conduct queries onadditional contextual details, including demographic shifts and historical records. Similar to human surveyors, they can refine their predictions by considering factors such as the cost of residences in the respondent’s vicinity or the voter turnout in that region during the previous election cycle.
Potential applications for AI polling
The utilization of AI polling is expected to be appealing to political campaigns and the media. However, ongoing research is uncovering situations where this technology might not be effective. While AI polling will always have limitations in its precision, these limitations align it with traditional polling methods rather than setting it apart. Present-day pollsters face the challenge of obtaining sample sizes that are sufficiently large to detect statistically significant variances between similar populations, and issues like nonresponse and insincere responses can introduce systematic errors. Despite these constraints, both traditional and AI-driven polls will retain their value. Despite concerns regarding the accuracy of political polling in the US, national surveys on key issues typically yield results that are reliable within a few percentage points. Whether you are vying for a local council position, participating in a tightly contested national election, or making policy decisions at a local government level, these slight yet localized distinctions can be highly significant. While imperfect, these signals can still assist campaigns and decision-makers in tracking trends over time, discerning variations between demographic segments, and revealing which messages resonate most with different groups.
The most effective application of AI lies in complementing traditional human-centric polling processes. Over time, AI algorithms will improve their ability to predict human responses and identify instances where their predictions are likely inaccurate or uncertain. They will be able to recognize the issues and communities undergoing significant changes, where the training data might lead them astray. In such scenarios, AI models can signal the need for human involvement to ensure accurate calibration with real-world perspectives. These AI agents can even be programmed to automate this process, leveraging existing survey tools to solicit genuine human feedback when necessary.
This combination of human and AI polling somewhat mirrors the current landscape of survey research. Decades of social science research have driven significant advancements in statistical methodologies for analyzing survey data. Contemporary polling techniques already involve substantial modeling and projection to predict characteristics of a broader population based on limited survey samples. Presently, humans participate in the surveys while computers assist in extrapolating missing data. In the future, this dynamic may reverse: AI could complete the surveys, with humans stepping in to provide input in situations where the AI is uncertain. If the prospect of political leaders relying on machines for insights into public sentiment concerns you, then you should be equally wary of the current techniques, not just the future ones.
Although AI-generated results may improve rapidly, they are unlikely to gain immediate credibility. Surveying people directly for their opinions feels more dependable than relying on a computer for such insights. Initially, AI-supported polls are expected to be predominantly utilized internally by campaigns, while news outlets stick to more conventional methodologies. A significant electoral event where AI predictions are accurate and human forecasts falter may be required to shift this trend.
This article was collaboratively written with Aaron Berger, Eric Gong, and Nathan Sanders, and was originally published on the Harvard Kennedy School Ash Center’s website.
About Author
