Welcome Tim Johnson to Novus Sanity Check.
Thank you. It's a pleasure to be here.
I'm so glad you had the time to do this and maybe you could tell
me a little bit or the listeners a little bit about yourself to
start with. Sure, sure.
Well, where to begin? I I grew up in the western part
of New York State in the US. The closest landmark would be
Niagara Falls. We were very close to Canada,
which makes me a big ice hockey fan and I have a number of scars
from my teenage hockey career that remind me of it on a, on a
daily basis almost. I, I heard my doctorate at the
University of Kentucky in, in Lexington, KY in sociology.
And while I was there, I, I, I had a, a nice job coordinating
research at a survey Research Center that the university had
opened. And this was during the heyday
of random digit dialing. It had it really just been
established as as the go to methodology for probability
telephone survey sampling. I, I remember arriving at the,
at, at the survey center and they had this amazing library of
telephone directories, paper telephone directories that were
all organized and with some overlap would cover the entire,
the entire state of Kentucky, which is where we, we were
focusing most of our, our data collection.
And I remember bringing in a, a, a very talented undergraduate
programmer to develop our first RDD program where we would feed
in the, the area codes and exchanges to generate random
samples and, and then to improve those using the Matoski Woksburg
sample designs. That was really the heyday of
RDD surveys. We routinely exceeded 70%
response rates and we're taught to look down at any, any surveys
that had response rates of less than than 70%.
So that, that was, that was a long time ago.
That was the early 1980s. But I, I ultimately graduated
and moved up to Chicago. It was hired by Dick Warneke, a
cancer control epidemiologist who directed the survey research
laboratory at the University of Illinois at Chicago.
And I had done a lot of health survey research as, as as part
of my own studies and my dissertation while while in
Kentucky. So we, we hit it off right away.
And so I, I was blessed to collaborate for more than 20
years with, with Dick Wernicke and also with Seymour Sudman,
who was our deputy director down on the Urbana Champaign campus
and also a a quite a renowned methodologist himself.
So I, I, I thought I had hit the jackpot, an opportunity to study
and and work with both Wernicky and and Sundman.
I ended up Dick stepped down after a few years and I I became
director and directed the survey lab for maybe 2/3 of of my
academic career at the University of Illinois and
ultimately retired a few years ago.
But I'm still active engaged in, in my own personal research and
collaborations and also in in leadership for professional
associations concerned with public opinion and and survey
research. So that's a long winded
introduction. Yeah.
And and you also been president of both a per and wafer among
other things. And I looked up your your bio.
So, so you you basically let them two of the most prestigious
research organizations in the world, according to my book at
least, Yeah. And which is which is really,
really something, I think. Well, well, it's, it's a
willingness to help out and sometimes they have a hard time
finding candidates. So a little bit of each I think.
Yeah, OK, you're like me. You'll forget to back down when
someone asks, asks you, and then suddenly you're standing there
alone or. Well, you know, yeah, some of us
were like just a kind of a knee jerk reaction to help when asked
to help with a project or a problem.
And then only later do you think more carefully about what you've
gotten yourself into. Yeah.
But, but as you said also that there's been a lot of changes in
the methodology. As you said, telephone was, was
the the gold standard for a very long time.
And also the response rate, you said that it was about 70% or or
over. And now in the US and, and, and
several other places, it's, it's close to 7% and 70 I think,
which really, really makes it hard to, to, to look at the
response rate as a good indicator of, of a properly
conducted survey. And, and maybe a bit an
interesting detail. But, but one thing that we as
researchers always know is probably the first 100
interviews, you know more or less where the results will end
up. So you get an indication,
although there's only 10% of the total respondents who have
answers, if you have 1000 respondents into you.
And still we talk about the, the, the dropout rate as being
the biggest thing because the theoretical dropout rate there
is 90% higher than it is later. And so, so, which is something
I've been thinking about for quite a long time actually.
And I'm, I'm, I'm, we've contacted some experiments here
and realised, OK, but if it is OK in the 1st 100 interviews,
more or less giving a good indication, then the entire data
collection is actually just going more granular in, in the,
in the subgroups. And also, of course, the margin
of error will get smaller with a hard higher number of
interviews. But, but it's, it also gives an
indication that the, the dropout rate is not the, the indication
that it's often being talked about like, oh, if it's under
this level, it doesn't work. So, but, but, but I also tend
to, to, to see the challenge here because now a lot of people
say, OK, the response rate is so low, so you can't rely on the
pulse anymore. And I think both of you and I
know that that's not really that simple, is it?
Not even close. Yes, no, it is really
challenging. I what I've always tried to tell
students is that it it, it's really variable specific.
There, there, there are some variables that you can estimate
pretty reliably with those very low response rates and there are
others for which other variables for which the selection effects
of a low response rate render it not, not terribly useful.
But those tend to be behavioral questions, attitudinal questions
is my in my experience and my readings aren't as vulnerable to
those those types of selection effects as as our behavioral
measures. So when we're doing public
opinion polling, it's it's seems to be a little bit more robust
to these the these low response rates that that we're going to
be living with for the foreseeable future.
Yeah, yeah, we are, aren't we? And and, but, but that is also
part of the magic with the probability sample is that you
the response rate tend to be random as well.
The the the non response. I mean, sorry, because as you
said it, it's the the the attitudinal, the attitudes are
seem are quite reliable still, but but the behavioural and may
maybe just to try to explain it to to the not so nerdy person.
I if if someone is not deep into this swamp as we are.
I, I, I tried to explain it that OK, if, if you want to find
someone helping you with their plumbing, it's really easy to
find the guys who don't have a job, but they might not be the
best ones. And, and from that perspective,
also in the research perspective, if you have very
niche, if you want to find a very niche competence there, you
might have a really big problem with the non response because
you will not reach those. But in attitudes, what you're
going to vote for in the next election, sort of, for instance,
that that is such a broad opinion, which it doesn't matter
if you don't find the perfect or the greatest plumber there who'd
really are going to fix your, your emergency in your, in your
bathroom. So, so that that's me trying to
dumb it down. Maybe I'm completely wrong, but
what's your take on this? Well, it's, it's you're right,
it's hard to dig even deeper into into it off the top of my
head, but but there is a strong correlation between the types of
questions and and the the effects of the of the low
response rates with the behavioral questions just just
be more difficult. Yeah, yeah, yeah.
And and difficult actually means cost more to to to for often
because you can still do it, but it costs a lot more money
because you have to work harder to get the the correct
respondents or reach the correct respondents.
Right, right. I'm, I'm sorry and, and, and
maybe it's, it's easier for us to to, to use some external
methods to validate those behaviours.
That, that it is, it's how do you, how do you validate
somebody's attitude can be done but but indirectly, but with
with behaviors. If we're talking about your,
your visits to the doctor and, and and and so forth.
There are external external records that can be used to in,
you know, the, you know, it takes a lot of effort to do it.
But to, you know, to, to validate what, what, what people
are telling you. And for a lot of other those
types of variables. You know, some of these
wonderful registries that you have in in in Europe are very,
very useful for those those purposes that that you you can
validate health conditions as as as well as health services
receipt that that you can't do with opinions.
The the only the only external way to validate self-reports on
public opinion. You know, the the common one
that we're familiar with and that we get in a lot of trouble
arguing about at the professional conferences is a
horse race questions in the polls who's you know, because we
can there is an external way to validate that and we get in a
lot of trouble when the snapshot polls are are not successful in
in incorrectly validating or correctly, you know, predicting
I guess is a better term to use the.
Outcome, yeah, that's the challenge as well because it's
actually not predictor Tory methods.
So it's as you said, it is a snapshot.
But then we say yeah and enough. And if nothing changes, this
will be the outcome because. That's a big caveat you've got
there, exactly. Exactly.
And that's what actually, sure, you can add, you can, you can
work with predictatorial models, but pollsters, the, the public
opinion pollsters actually don't do that.
They, they, they have a snapshot of today.
This is the last day that we measured before the lecture that
this would be the outcome if it were the election today.
But then it's being talked as being that the a prognosis which
is at per to sign isn't so. So there you also have this
challenge and, and, and just just as an example, in Sweden in
the last two national elections, 4% of the voters changed their
mind after we as an industry did the last questions for for the
election and the election. Interesting.
So we know now that the voter mobility is so much higher than
we used to because of things like Internet, because people
that they're that and and that, that the drama in in media often
tend to create some small, small catastrophe for some candidate
the last couple of days, which really move the voters because
you, you don't, you're not really if in the US you have two
parties and it's more polarized by by the sign maybe.
But but in Sweden, we have eight parties.
But for instance, the days where you were born, once a Social
Democrats, always a Social Democrats is over long ago.
Now we can move the entire spectrum depending on where you
think of the present and you think of the future.
And then the parties are really triangulating this a lot.
So the the polls is actually not a good indication of how good.
No, the sorry, the election is not a good indication of how the
polls are going. But thankfully, although 4%
changed their mind, there's still a pretty good indication.
But if 8% changed their mind, it would be worse.
Yes, well, and, and especially in, in, in in most Western style
democracies anymore the the outcomes of the election,
there's no more landslides. It's it's always the United
States being a a terrible example where it's very close
even when it shouldn't be. It it turns out to be very
close. And so people are going to
continue to use the election outcomes because that's all
we've got. That's the only Yeah, yeah, but
but the old sick available. Exactly.
But at the same time, we have an entire industry in in, in the US
spending billions of dollars getting the voters to change
their minds, which we don't have in much other issues.
So, so, so the pollsters are trying to measure something that
the, that there's $1 million at stake or maybe billions of
dollars at stake, getting the voters to change their mind in
the until the last second, which is a fascinating weird.
It's it's like someone is actually doping the horse in the
horse race. It's interesting analogy.
There are a lot of moving parts. You're right in the election and
there, there are a lot of actors and, and I, you know, they're
not all good actors that, that are, are attempting to influence
the outcome. You know, the political parties,
of course, that's their job is to try to sway voters.
But there are a lot of external groups that, that are also
sending messages and more and more we see these disinformation
campaigns and so forth. And it's also challenging to to
measure their impact. Yeah, yeah, it is.
And and I'm maybe maybe back then to, to, to the the only
joystick we have for for opinions being the horse racing.
But at the same time, proper polls is still a very good
indicator both if there's something bad going on about
real voter temping tampering, exit polls are often the only
proof you have or an indicator of of something not going
properly in in an international election.
But but same time you can also see when the voters change speak
of of of Canada, there was a radical shift in opinions going
going like AUS effect, but going going the opposite way.
We don't. Have to talk more.
That was very interesting. Yeah, some of my Canadian
friends have actually thanked us, you know, because by, by
electing your current president, our current president, we saved
them from from a similar fate. Yeah.
Yeah, which is, which is really tragic and fascinating at the
same time. But you have all this that's
probably an unintended consequence for, for those who
thought they were going to actually drive it home in
Canada. But but at the same time, the
voters actually react to what's happening in the world.
But the polls showed the shift of opinions during this change.
And, and so if we move from, but that's, that maybe that also the
stories a little bit of the argument.
Because if we, if we say that the, the, the election is not a
good indicator of how good the polls are because people change
their mind. And until the last second, then
you have no way of proving that the polls work on measuring
opinions. But the same time we know that
polls is a good indication of explaining when and how the
opinions changed. But you can't prove it if they
changed their mind after you stop measuring.
Because at some sense me, me being in the private sector, I
also, I, I, I really feel the responsibility because if I
present the poll at the borning of the Election Day, I might
change the voter turn out. Because we, we have science
showing that for, for Brexit, for instance, when, when, when,
and also the US and, and Denmark.
There's a bunch of examples that when someone calls the election
based on polls saying this is over, it actually affects the
turn out rate of the election because the people who support
the winning candidates say oh good, I don't have to vote.
Exactly that. I'm certain that's what happened
in 2016 in the in the US. Yeah, yeah, yeah, exactly.
And, and, and, and the Brexit, there's several indicators as
well, because the voter turn out was lower in the remain areas
than in the leave areas. And there were also a lot of
maybe anecdotal reports afterwards saying that hang on,
I voted for Brexit because I didn't think it would matter.
I just wanted to protest because everyone said it would be
remain. So I just want to put in my my
own. So yeah, they have to get get
their act together, but I I still didn't really want to
leave. So it's, it's, it's really
challenging. You're right that the public has
this assumption that the polls are correct within half of a
percentage point. And and it's just not realistic
given our, our, our models to, to expect that that kind of
precision, especially as you said earlier, when we stopped
polling a few days before the actual election takes place.
You cannot capture all of that churning and and last minute
movement in the electorate that's out there.
In the old days we thought we could understand voter turn out
by by the weather conditions on the day of the election.
That certainly doesn't work, but but there is something that
drives turn out that all of our likely voter models cannot
cannot adequately capture, especially with the the fairly
rudimentary questions that that are used, you know, they're
they're all realistic, but but I just I don't think that we have
have a good way of of adequately modeling, you know, the the turn
out. No.
And I, I also think because it also has to do with, with,
because you said something, I'm just going to air that idea as
well because it's always a 5050 race.
And, and I'm starting to think that we're, we're getting so
close to game theory in the political consultant realm that
you've, you're creating this 5050 choices for everyone
because you're trying to try also using really reliable
polls, really expensive polls to dig out.
Where is the key issue that I have to get the people to tip?
But, but I think that they're too professional here in getting
this. So it's like a, a, a coin toss
for the voter to to much that the people actually care because
there's something weird here. How come it's always a 5050
race? It was the same in Poland the
other day. Two, one or two percentage point
over for, for, for one of them. There's something here that's so
weird. It hasn't to do with polls.
It's it's actual, actual election outcome.
Sorry. Yeah.
Right. No, that's OK.
That's, it's polarization is, is what I, you know, yeah, see
discussed in the literature and in the media at, at a regular
basis. And they're right.
There is just a heavy dose of, of polarization in, in most of
our Western societies these days.
And it's produced all kinds of negative outcomes, including
what we see in, in these elections.
These outcomes in these elections are probably just a a
by product of of of those broader social divisions that
that are being discussed. Yeah.
But but I also think that it's it's because you focus on these
polarizing issues because so, so because the communication is so
heavily, you research it so deeply to making sure that you
get this polarization, these false choices, basically black
or white choices. Yeah.
They refer to them as wedge issues.
Oh yeah, yeah, yeah. And, and, and, and then when you
have a normal distribution curve on anything, because that's what
you have, you're being actually forced to, you're at the tip tip
of the edge all the time and just tipping over for one or
others issue. There's something there that
hasn't to do much with poles except that poles are actually
being used to to find these issues.
But it's also how you use this information that is the
challenge here. And when the grayscale
disappears and everything is black and white, yeah, you're,
you're either black or white. So, so there's something there.
But I realise now that we're completely off the topic but but
it's OK. But still interesting.
Yeah, Yeah, it is. But I also, but because I said
something that I, I think that listeners might react to because
I said if, if the, if the elections is not a good
indicator of reliable polls, because if the voter mobility
increase, then it will be a worse and worse indicator.
But that actually doesn't mean that the, because you're, you're
talked about this a little bit as well.
That does actually doesn't mean that you cannot validate other
polls because you can do it on some other stuff.
There are some, some behavioural issues, as you talked about or,
or topics that that you could, you could look at.
But I think also one thing that is getting forgotten is that
actually the theories that's being used has been working for
almost over 70 years now. And the theory includes also the
margin of error and and that's it.
We're 95% confident that it's within the margin of error.
OK, that's a that it's, it seems a little bit dodgy to say that,
but it's also you're 95% confident it's within the margin
of error, which is quite good in most issues and the theories are
still there, yes. Now that's largely sampling
error, right? Yeah, that we're talking about.
Yeah. You know, sometimes I, I, I use
the rule of thumb that you, you double that margin of error to
account for non sampling errors. If if, if it's really, really
important to you to understand you know what the odds of one
candidate or the other winning the election is, it's it's
probably safer to double that that margin of error.
Yeah, but but but. It's a rule of.
Thumb yeah, yeah, that, that is a good point because if you're
stuck at the the 1-2 percentage point or whatever, it's like,
OK, if it's really important, why not double it?
That that's a good, that's a good point, especially when it's
a 5050 because the margin is actually bigger than on the
middle than they saw on there and and but it also needs to be
a random sample or probably. Absolutely.
And that's the thing that's getting forgotten because
there's a lot of arguments saying, OK, the response rate is
so low, you cannot do probably probabilistic research anymore.
It's no point. And, and I see no proof for
that, no actual proof. It's, it's just like an
accusation being thrown around all the time.
But but I see it all the time. Although there are no real proof
that the the response rate low responses, it is affecting the
opinions as as we talked about, but it's still being OK.
It's no point doing random samples.
We have to go to alternative methods.
But then you'd completely lose the margin of error.
Totally. And you know some of these
studies, these these poll aggregators, our, our colleague
Claire Durand in Montreal does a lot of work looking at the
accuracy of of final polls based upon sampling methods.
And she finds consistently that the probability based sample
frames are, are more accurate, not not tremendously more
accurate, but more accurate consistently.
Yeah. And and there you have a, a, a
complicating factor as well because no one knows really what
happened in this last election. But but the polls were so close,
it was basically mathematically impossible that they were
independent of each other. So there was something where
some say it's hurting, I don't know because I'm not there, but
there's something there because if you don't know, if it is a
really independent poll from the others, you also you have that
variable here as well so that no one knows what's happening
because the, this is being done in commercial actors.
But there's also I, I often joke I could make a perfect poll in
Sweden without asking any questions because there are so
many good polls to look at. And, and if you're, if you're at
in close, the close to the other ones, no one would rise and
raise an eyebrow. And you will save a lot of money
doing that, but you will of course be lying.
But if it's your reputation at stake, which is what what we're
talking about as as a poster and you have to show you that you're
actually are good enough to be trusted for the commercial work
because you don't make money on on on the horse race as as a
poster for the media, you actually lose money.
You make that. You make money working for the
parties, but and they're they're they, I talked to a lot of of
those and they say, yeah, we never know, use anything else
but proper probability samples, random samples, because they
can't afford to be wrong and they also get paid to be right.
So that's true. Yeah, that's true.
You know, I, I don't think that there's a lot of hurting, you
know, I, I understand. And the concern about that I,
yeah, I, I've seen so many polls that that seem to be outliers
and the, the pollster was kind of embarrassed about it.
Yeah, but this is what the data showed.
I, those would be the ideal candidates for a little hurting.
My I maybe they tweak and this is not hurting, tweak the
variables in their models, their weights and sample weights and
so forth. But that's well.
Yeah, Yeah, yeah. I get to what you mean.
And as I think there's different layer layers of this as well,
because OK, in Sweden, I've seen so many obvious indicators of
this because some non probability polls were so way
off from from us, Ipsos and Kantar.
We were in the same space all the entire time during because
in general, we we publish a poll every month during the election
year or even with between elections of all the time
actually, but but close to the two or three latest polls before
the election, the non probability base subtly close up
to us, which is really impossible to explain because
and that has happened with so many providers for non
probability based panels in Sweden.
And because and the argument for them has always been we're
better because we don't have the problem.
We don't response. We are using the water methods,
whatever, whatever. And if and that's how they
explain to be different between the election from us, but it
doesn't explain how we continue to to that the probability based
are to continue to be close to each other the entire time.
Of course, there's some noise there because and it should be,
but it's giving the same view of the opinion and the non
probability being completely different until exactly in
before the election. So that that is that I've seen
this for at least three elections because I'm so now
I've followed this for such a long time.
So this cannot be a coincidence, but I can never prove it.
But it's still annoying me because either you're completely
different and you get it right, but then you should be
completely different even in at the Election Day, because then
the traditional methods doesn't work and they have a new perfect
method. Yeah, boy, you know, you, I, I
almost, I, I don't have an answer to that either.
No, no one has. Yeah.
But, but I think it, it, it's going to require some forensic
analysis of those non probability polls because when
we talk about non probability, what's the definition?
It, it, it's, it's what it's not, it's not a probability
poll, but it could be anything. Yes, there's such a variety.
They're they're not all the same animal, these non probability
polls. So I I don't know that.
That is true and and exactly and it's it's like yeah, yeah.
And that is part of the problem as well as as as you know as
because you know, you've left the scientific field to be just
to be very broad. OK, we have a definition of what
probability based sample is and that's the the theory we have
for 70 + 8. OK, there's details.
There's so there's so many things there that also makes
that a, a, a sliding scale to some to, to a lot of sense.
But it's actually at least there's some sort of theory bit
that that I usually say that you try to replicate it to the best
way possible because we all know back to the response rate.
You cannot do a perfect probability samples probably
anywhere anymore. But you can try to replicate it
to the best way possible. And and maybe in the simplest
form, you as a poster have a big bucket when you randomly sample
people. And then we ask you, but not the
opposite around. You cannot stand in line saying
I want to give my opinion to you because that that's the simplest
difference. But but, but as you said that
there is no definition of non probability based samples.
There, there are so many different things.
And maybe that's also why why you can't say a method works
that's not based on probability based because it's like, OK,
it's just not what we know. What's the underlying theory?
Yeah, exactly. Because I'm, I'm still waiting
for that. I would love to see another
method that actually works, but it has to be some theory that
can be confirmed. Now at at least quota sampling,
you understood the rationale there there was, there was a
reason for what they were doing. Yeah, that's true, but.
A lot of these other approaches it's it's it's not so clear and
and the problem well, the general problem would be
inaccuracy in in a lot of these election polls leads to a lot of
public suspicion regarding the quality of the the polls.
They don't, they don't believe the polls, especially you think
about the motivated reasoning theories that they don't like
the polls when they don't show the results that they would like
to see. And if you if, if in a heavily
polarized society, there's always going to be a significant
fraction of the population that does not like what, what the
polls are, are, are telling us. And and that they first thing
they do is well, it's fake polls and some of our candidates are
capitalized on that. Yeah.
And that and yeah, that's a good point.
And it's also, it's hitting the, the integratives of, of, of
science in general as well. Yes, it is.
Because polls is polls and, and you have spent the entire career
not political polls, but doing, using polls as part of your
research. And and, but, but no one knows
the difference between these horse race polls that no one
pays for because media just gets them and the proper research
that's being based on on, on on the same theories, which which
is a quite big dilemma because there also the census bureaus
use the same theories to a large extent on the same methods.
You know, you're absolutely right about that and, and it's
an interesting point. A lot of my work is back to my
collaborations with Richard Wernke is, is related to cancer
control and there are dozens of validation studies that did you
get a mammogram And then can we have your permission to look at
your medical records to confirm the dates?
And a lot of it is is not whether or not they received
the, the the exam, but the, the, the, is it in the proper time
interval? Do they recall was it in the
last five years or whatever the the Cancer Society
recommendations are for, for speeding those studies?
Yeah, we've got a pretty good idea what the, the concordance
rates between the reports and what the medical records tell us
are across all of these cancer screening medical procedures
today. Every single one of those
studies is based on probability sampling it if If they came in
and they were non probability sampling I I, I, I'm not sure
they'd even get published. Yeah, yeah.
But but when the polls in quotation marks are wrong in in
in in an election, I think it also hurts the willingness to
finance these kinds of standards because the general idea is that
OK, polls don't work, the response rates this to low,
everyone knows it doesn't work. Look at the last election.
Why should we spend a bunch of money on these non working
methods? So I, I see indicators of that
for, for several places because it's so hard, because the
politicians are the one who, who gives grants and also they have
the voters who the most vocal voters.
And as we say now, if it's a poll you don't like, yeah, it's
the polls wrong. They they are biased.
Can't be right. Yeah, exactly.
Because everyone I know thinks like me so and, and the poll is
saying something different, which I have heard a lot of
times because I work for politics, political issues.
But that's how it works. Because you are in your own
bubble. And of course you don't meet the
opposition, no, no matter where where you are to that extent
because you don't talk about those issues with people who are
probably on the complete opposite.
But but. Partially, that's true.
Yeah, which is which is fine as well because you don't want to
fight all the time. To be honest.
It could be it's it's OK to live your life.
But if you start extrapolating that and also say, OK, the post
don't work. So I don't trust the science, I
don't trust the statistics from the census bureaus.
Suddenly it's like quickly unraveling worldview where
everything is contested and that that's basically the core.
Why why I'm I'm constantly nagging about this is about this
part because the knowledge society and a fact based society
is so much based on what the core of what we're actually
doing. Although I'm commercial and and
you, you come from the university and, and and the real
science part, it's the same. Overlap.
We're all part of the same statistical knowledge
infrastructure. Yeah, yeah, yeah, you're right
in in the US right now that that that infrastructure is being
taken down. The the the NISTA of a national
survey of drug use and health. They fired the entire staff.
Yeah, the federal government did.
Yeah, it's tragic. And you, you start politicizing
knowledge and, and we've seen this before in other countries
in, in Russia or the Soviet Union, the, the science was
heavily politicized. You, you only were allowed to
come up with the, with the findings that was in line with
the party. So, so we, we know what happens
and we don't know where we're going.
But but just to stay out the politics because it's it's
really it's it's really too sad to talk about even people who
don't think like you need to trust facts like gravity works.
It's a good thing. We need a common, a Common Core
of knowledge. Yeah, we did.
It's uncontested. Exactly.
And and, and sure. And, and we also have to trust
the spokesperson who stands behind science because
politicizing that I, I, I know John Krosnick did a lot of
studies on this. When, when, when in climate, in
climate studies, when he did experiments, a scientist saying
something about climate change, but then he did the same, but
adding a political hat on the same person.
The trust was degrading rapidly, of course, because suddenly you
weren't this independent scientist, you were a political
actor using science to get your point through it.
That's how it was the the person was saying the same thing, but
it was being perceived as trying to getting political about the
issue. So, so there's something there
that that really needs it's, it's, it's a really, really
important issue just to move away from trying to push your
view into explaining this is the way it is.
So yeah, I don't know what the fuck to make of that, sorry.
No, that's OK. We're we're we're down the
rabbit hole now. Yeah.
Really, I so don't mention the war.
Oh, OK, it's all right. Like I, I always do that.
I'm, I'm sorry, but oh. No, that's OK.
But but it but it's really interesting to but we also have
a challenge because we shouldn't contest facts because it in the
long time it benefits no one how you use the facts.
That's a different thing That could be a political thing, but
facts should still be there. How many people living in a
country whatever all these data or or or also the views, the
opinions of people is often really important.
The population estimation of how how people think is often a a
vital part. Even between the election of of
so many things, how we behaved. We had a pandemic, but the
opinion of of the people and and the behavioural that that that
the opinions resulted in was vital to handle the to do not
panic, for instance, the simplest part and and we need to
know that and we need to trust that those opinions are
representative for the population.
Well, it, it, it, it, it really is a challenge.
I you, you mentioned the COVID and, and just the, the ongoing
controversy. It's, it's almost becoming like
the Kennedy assassination. You know, what is the origin of
the COVID virus? Is it, is it from that, that
laboratory and that game of function research, or is it a,
an, a terrible, terrible accident that, that took place
in the, in the, in that open wet market?
I don't know that we'll ever know.
No, that that exactly. It also has to do with with the
trust between people. Yeah.
Agreed that the the inter human trust is is the key in in a
democracy. We can trust our neighbors and
then realise that we have laws and rules and regulation that we
follow them because they are the consensus of the culture of, of,
of, of a democracy. Yeah.
And, and there's an incredible amount of trust associated with
the scientific enterprise that that what you are reporting is
actually what you did. And, you know, I, I, I think in
the US and the transparency initiative and other efforts
around the world, you know, to reveal as much information as
possible, not not only so that the results can be replicated,
but also so that that other scientists will accept as
credible. Yeah, the the research that
that, that you're reporting and. Yeah, yeah, that, that that's
true. And and also the the the
complicating part in actually explaining complicated details
in a trustworthy way without having the audience to read all
the footnotes. But they should be there.
But it's yeah, so so that. That that information should be
there for those who are interested in primarily other
other scientists and in in understanding those details.
Yeah. And maybe that's the last topic
because I realized we have completely way off this, but
actually you, you presented a, a paper with, with one, one of
your colleagues at the latest. Peter Miller.
Peter Miller Yes, at the latest April conference about what
happens when you, when you present more information about
what's the transparency. Can you just talk a little bit
about that? Oh, sure, yeah, sure.
So the, the E POR transfer initiative was actually Peter's
brainchild when he was president of the association back about 15
years ago. And it, it was a heavy lift on
the part of the whole association to to develop a, a
workable model for a, a, a program that would encourage
survey organizations, commercial government, academic survey
organizations to make a pledge, A voluntary pledge to report on
a routine basis for all studies that they publicly report all of
the mythological details that are in the official code of
conduct, professional code of conduct for the association.
And there are over 100 members of the association of, of the
Transparency Initiative today that that have taken that pledge
and, and that report that information, it's never been
evaluated. And, you know, and it became
obvious sometime back Peter and I had discussions that, well,
the, the methodological disclosure of those details that
we were talking about, you know, it's scientists talking to other
scientists. Nobody else understands or cares
or wants to read all that, that nerd speak or that Godly go.
So we were wondering, what are we going about this wrong?
We're not really reaching the audience that, you know, has
declining trust in, in, in public opinion research and
science in general. So what can we do about that?
We took advantage of an opportunity and, and, and the
people at the University of Southern California's
Understanding America study, what we're kind enough to give
us some time on, on one of their nationwide panel survey
questionnaires to, to conduct an experiment where we, we took a
fairly neutral study, news media report, I think it was about
weight loss drugs. And then the latest findings
from a public opinion poll about the use of, of, of weight loss
drugs. And we developed three different
types of mythological disclosures to, to, you know,
for that news story and then randomly assigned each
respondent to one of these three disclosures. 1 is what we call
the minimal disclosure. And it was about two sentences.
The survey was done in this month and it was a sample of X
number of Americans, very basic minimal disclosure.
The second condition was what we call the informational
disclosure, which is basically what the Transparency Initiative
requires. There's a much, much longer list
of, of data elements that need to be disclosed to conform with
the the Transparency Initiative's requirements.
And then there's something that Peter developed called the
explanatory disclosure. And rather than just throwing
all of these numbers at you, the response rate was this, well,
what does that mean? If if you're you're that talking
to a scientist, what exactly does that mean?
What, what Peter did, he very carefully crafted an explanatory
description of all of this data so that people would understand
why decision, why methodological decisions were being made and
and what how this information should be interpreted.
It was a much longer piece, as you, you might imagine.
So the experiment ranged from 2 sentences to kind of, it's kind
of like Goldilocks with three different levels of, of, of
obial. But we, we randomly assigned a
large sample. We had 3000 respondents and they
each received one of these, these disclosure texts, the
minimal to the maximal, the explanatory.
And then we, we measured their, we had this very nice scale 4
out of scale that had great reliability and, and construct
validity that measured their perceptions of the credibility
of the pole that they were, you know, that was the focus of the
of, of the experiment. And we did not find tremendous
differences in terms of the effects of this methodological
disclosure on respondents. We found a little bit that that
the explanatory text, despite the fact that it was a much
longer block of information written probably at a slightly
higher reading level, nonetheless elicited higher
perceptions of of credibility relative to the minimal.
We interviewed these many people On this date and we, we had a
series of, we looked at a series of, of potential moderators of
the, of that relationship and, and, and found virtually no
effects there. And we also learned that not a
lot of people know about the transparency initiative or it
just just mentioned the transparency initiative as a
Good Housekeeping seal has no effect on, on people.
So that, that that's the word and, and the final piece of of
interesting information was we randomized the presentation of
this material such that half we just embedded it the
methodological details in the study.
For the other half, we said For more information about how this
study was done, click here on this hyperlink.
And so they, they had to actually go out of their way to
to access the information. And what we found was just under
20%, about one in five respondents had any interest in
even going to look at at the methodological information that
we got. That tells me we've got a long
way to go. Yeah, yeah.
Sorry. That was a long winded no, no,
no but. I think it's, it's, it's good
because I, I, yeah, I found it really interesting as well.
And, and also the challenge here.
But of course, there's so many more challenges because most of
this polls or, or research result is presented in, in news
media and, and, and the trust in media is, will influence the
trust in the poll if you're unlucky as well.
So there's so many layers. But I, I thought it was really
interesting to just look at this because at the same time, it is
good to have some sort of disclosure of what's what, how,
how the, the poll is constructed because.
Yeah, just going back to the probability based, non
probability based, that's one step.
But also OK, there is some science behind this and we have
a method that we we actually know it is possible to reproduce
the method and and in that case also a result.
So I thought it was really interesting and, and that there
are, as you said, there's so much more to do.
But but I, but I think we all have to figure out how to work
more on increasing the trust in the science behind this and and
also, of course, protecting the neutrality of the science,
because that's not facts or facts.
And we should treat it like that.
Then you might not like it, but just it's still probably the
good to know it. That's basically my my take on
facts. I'm with you.
We have to protect the neutrality of the facts.
They're not. Yeah.
And then? Partisan.
Yeah, yeah, exactly. Because then every, every fact
will be a partisan. That's that's an entire
different challenge. But but yeah, we, we, we still
have to acknowledge that there are science that actually works
and that we know how to do it. And and that in turn creates a
lot of vital facts from the, from in the Western world that
we need in our daily life. It affects the economy.
It affects everything actually what we're doing.
And we tend to forget that that's, but I'm really glad you
took the time because I realized we have run out of time right
now. Oh.
Have we? Yeah.
You know, it's been a pleasure. It's been a pleasure talking
with you. Yeah.
These are important problems that aren't going away.
We'll have to continue our dialogue on these topics.
I totally agree. I'm so glad it took the time and
now my day is ending. I think your day is starting,
but it's amazing that we can talk across the world like this.
Thank you. So much a great invention, yeah.
It is. I hope you have a great day.
Thank you. Thanks so much.