Public opinion polls can be tricky. They can be inflammatory. They can be wrong.
They may even be right.
But more often than not, the surface results of a poll — particularly on questions other than a political up-or-down vote on candidates — do not reflect nuances in opinion or outside factors that can affect how people answer any particular polling question.
A recent poll by one of the nation’s most highly respected news organizations and one of the top polling groups illustrates the difficulty of taking a poll at face value. The results of the poll by the Wall Street Journal and the National Opinion Research Center at the University of Chicago spread virally online in late March because they seemed to portray a seismic, downward shift in cultural values.
One of the most breathless, ominous, overwrought takes on the poll that I saw came from Mike Allen of the news site Axios, who wrote, “Rarely does one poll stare so deeply into the soul of a nation and tell its story.”
Let’s be clear: Polls do not stare anywhere, let alone into a nation’s soul.
A poll may be a snapshot (taken with a cloudy lens, held by someone whose hand-steadiness you may not be able to gauge), or a thermometer. But it has no eyes. It has no brain. It has no empathic ability to divine the feelings or intent of those who are surveyed.
This particular poll’s results that generated so much alarm were a cratering, compared to a poll done in 2019, of the public perception of the importance of patriotism, religion, having children and community involvement. (Allen compared the new poll to a Journal/NBC poll done in 1998.)
And indeed, if true, those trends would be alarming.
But are they true?
Not long after this poll went viral, I saw an analysis by Patrick Ruffini, a Republican pollster who co-founded the political research firm Eschelon Insights, that sought to lower the temperature of the discussion.
“These findings fit into a declinist narrative we are already predisposed to believe,” Ruffini wrote. But he cautioned, “If these numbers had been produced by my firm, I would immediately assume we had made a mistake and send them back to an analyst to double check.”
“The point here is not that the Wall Street Journal and NORC released bad data. The Journal is one of the more thoughtful media sponsors of polling and NORC is the premier survey data-collection organization in the country. Rather, the dramatically different results we see from 2019 and 2023 are because the data was collected differently,” he wrote.
The 2019 poll, like ones before it (including the 1998 poll cited by Allen), was done by telephone interviews with survey participants. The March 2023 survey was collected via NORC’s Amerispeak, which Ruffini called “an extremely high-quality online panel,” but that difference is the key.
“Surveying the exact same types of respondents online and over the phone will yield different results. And it matters most for exactly the kinds of values questions that the Journal asked in its survey,” he wrote.
“The basic idea is this: if I’m speaking to another human being over the phone, I am much more likely to answer in ways that make me look like an upstanding citizen, one who is patriotic and values community involvement. My answers to the same questions online will probably be more honest, since the format is impersonal and anonymous.”
Ruffini’s take is that the online survey results are probably a more accurate reflection of how people really feel. You might find THAT alarming.
But if he’s correct, it will take future online survey panels to show whether there is an actual trend in any direction.
