1 00:00:09,600 --> 00:00:12,600 Welcome Tim Johnson to Novus Sanity Check. 2 00:00:13,200 --> 00:00:15,160 Thank you. It's a pleasure to be here. 3 00:00:15,520 --> 00:00:20,000 I'm so glad you had the time to do this and maybe you could tell 4 00:00:20,000 --> 00:00:24,200 me a little bit or the listeners a little bit about yourself to 5 00:00:24,200 --> 00:00:26,320 start with. Sure, sure. 6 00:00:27,720 --> 00:00:34,000 Well, where to begin? I I grew up in the western part 7 00:00:34,000 --> 00:00:37,560 of New York State in the US. The closest landmark would be 8 00:00:37,560 --> 00:00:40,240 Niagara Falls. We were very close to Canada, 9 00:00:40,240 --> 00:00:46,160 which makes me a big ice hockey fan and I have a number of scars 10 00:00:46,160 --> 00:00:51,560 from my teenage hockey career that remind me of it on a, on a 11 00:00:51,560 --> 00:00:55,480 daily basis almost. I, I heard my doctorate at the 12 00:00:55,520 --> 00:00:59,680 University of Kentucky in, in Lexington, KY in sociology. 13 00:00:59,680 --> 00:01:05,840 And while I was there, I, I, I had a, a nice job coordinating 14 00:01:05,840 --> 00:01:10,200 research at a survey Research Center that the university had 15 00:01:10,200 --> 00:01:12,920 opened. And this was during the heyday 16 00:01:12,920 --> 00:01:18,160 of random digit dialing. It had it really just been 17 00:01:18,160 --> 00:01:22,080 established as as the go to methodology for probability 18 00:01:22,560 --> 00:01:27,240 telephone survey sampling. I, I remember arriving at the, 19 00:01:27,280 --> 00:01:30,800 at, at the survey center and they had this amazing library of 20 00:01:30,800 --> 00:01:34,040 telephone directories, paper telephone directories that were 21 00:01:34,040 --> 00:01:38,680 all organized and with some overlap would cover the entire, 22 00:01:39,880 --> 00:01:42,280 the entire state of Kentucky, which is where we, we were 23 00:01:42,280 --> 00:01:45,120 focusing most of our, our data collection. 24 00:01:45,200 --> 00:01:51,360 And I remember bringing in a, a, a very talented undergraduate 25 00:01:51,360 --> 00:01:56,200 programmer to develop our first RDD program where we would feed 26 00:01:56,200 --> 00:02:00,320 in the, the area codes and exchanges to generate random 27 00:02:00,320 --> 00:02:05,520 samples and, and then to improve those using the Matoski Woksburg 28 00:02:06,480 --> 00:02:09,120 sample designs. That was really the heyday of 29 00:02:09,120 --> 00:02:14,120 RDD surveys. We routinely exceeded 70% 30 00:02:14,120 --> 00:02:18,800 response rates and we're taught to look down at any, any surveys 31 00:02:18,800 --> 00:02:21,640 that had response rates of less than than 70%. 32 00:02:21,640 --> 00:02:25,240 So that, that was, that was a long time ago. 33 00:02:25,280 --> 00:02:30,520 That was the early 1980s. But I, I ultimately graduated 34 00:02:30,640 --> 00:02:37,360 and moved up to Chicago. It was hired by Dick Warneke, a 35 00:02:37,360 --> 00:02:41,920 cancer control epidemiologist who directed the survey research 36 00:02:41,920 --> 00:02:44,920 laboratory at the University of Illinois at Chicago. 37 00:02:45,400 --> 00:02:50,560 And I had done a lot of health survey research as, as as part 38 00:02:50,560 --> 00:02:53,320 of my own studies and my dissertation while while in 39 00:02:53,320 --> 00:02:55,760 Kentucky. So we, we hit it off right away. 40 00:02:55,760 --> 00:02:59,160 And so I, I was blessed to collaborate for more than 20 41 00:02:59,160 --> 00:03:03,160 years with, with Dick Wernicke and also with Seymour Sudman, 42 00:03:03,240 --> 00:03:09,120 who was our deputy director down on the Urbana Champaign campus 43 00:03:09,120 --> 00:03:12,720 and also a a quite a renowned methodologist himself. 44 00:03:12,720 --> 00:03:17,240 So I, I, I thought I had hit the jackpot, an opportunity to study 45 00:03:17,240 --> 00:03:20,720 and and work with both Wernicky and and Sundman. 46 00:03:21,480 --> 00:03:26,400 I ended up Dick stepped down after a few years and I I became 47 00:03:26,400 --> 00:03:31,320 director and directed the survey lab for maybe 2/3 of of my 48 00:03:31,320 --> 00:03:35,360 academic career at the University of Illinois and 49 00:03:35,800 --> 00:03:38,400 ultimately retired a few years ago. 50 00:03:38,720 --> 00:03:43,000 But I'm still active engaged in, in my own personal research and 51 00:03:43,000 --> 00:03:47,560 collaborations and also in in leadership for professional 52 00:03:47,560 --> 00:03:50,280 associations concerned with public opinion and and survey 53 00:03:50,280 --> 00:03:52,520 research. So that's a long winded 54 00:03:52,520 --> 00:03:54,440 introduction. Yeah. 55 00:03:54,440 --> 00:04:01,400 And and you also been president of both a per and wafer among 56 00:04:01,800 --> 00:04:04,960 other things. And I looked up your your bio. 57 00:04:04,960 --> 00:04:10,440 So, so you you basically let them two of the most prestigious 58 00:04:10,880 --> 00:04:14,120 research organizations in the world, according to my book at 59 00:04:14,120 --> 00:04:18,640 least, Yeah. And which is which is really, 60 00:04:19,640 --> 00:04:24,160 really something, I think. Well, well, it's, it's a 61 00:04:24,760 --> 00:04:28,080 willingness to help out and sometimes they have a hard time 62 00:04:28,080 --> 00:04:32,040 finding candidates. So a little bit of each I think. 63 00:04:32,840 --> 00:04:35,440 Yeah, OK, you're like me. You'll forget to back down when 64 00:04:35,440 --> 00:04:37,920 someone asks, asks you, and then suddenly you're standing there 65 00:04:37,920 --> 00:04:40,840 alone or. Well, you know, yeah, some of us 66 00:04:41,560 --> 00:04:44,840 were like just a kind of a knee jerk reaction to help when asked 67 00:04:44,840 --> 00:04:46,480 to help with a project or a problem. 68 00:04:47,080 --> 00:04:49,400 And then only later do you think more carefully about what you've 69 00:04:49,400 --> 00:04:52,280 gotten yourself into. Yeah. 70 00:04:52,280 --> 00:04:56,760 But, but as you said also that there's been a lot of changes in 71 00:04:56,760 --> 00:04:59,880 the methodology. As you said, telephone was, was 72 00:04:59,880 --> 00:05:04,640 the the gold standard for a very long time. 73 00:05:06,000 --> 00:05:10,400 And also the response rate, you said that it was about 70% or or 74 00:05:10,400 --> 00:05:13,240 over. And now in the US and, and, and 75 00:05:13,760 --> 00:05:18,840 several other places, it's, it's close to 7% and 70 I think, 76 00:05:20,880 --> 00:05:25,360 which really, really makes it hard to, to, to look at the 77 00:05:25,360 --> 00:05:29,160 response rate as a good indicator of, of a properly 78 00:05:29,160 --> 00:05:36,200 conducted survey. And, and maybe a bit an 79 00:05:36,200 --> 00:05:39,560 interesting detail. But, but one thing that we as 80 00:05:39,560 --> 00:05:43,640 researchers always know is probably the first 100 81 00:05:43,640 --> 00:05:46,520 interviews, you know more or less where the results will end 82 00:05:46,520 --> 00:05:50,720 up. So you get an indication, 83 00:05:50,720 --> 00:05:56,640 although there's only 10% of the total respondents who have 84 00:05:56,640 --> 00:05:59,680 answers, if you have 1000 respondents into you. 85 00:06:00,320 --> 00:06:04,520 And still we talk about the, the, the dropout rate as being 86 00:06:04,560 --> 00:06:07,400 the biggest thing because the theoretical dropout rate there 87 00:06:07,400 --> 00:06:15,240 is 90% higher than it is later. And so, so, which is something 88 00:06:15,240 --> 00:06:17,240 I've been thinking about for quite a long time actually. 89 00:06:17,240 --> 00:06:19,920 And I'm, I'm, I'm, we've contacted some experiments here 90 00:06:19,920 --> 00:06:24,360 and realised, OK, but if it is OK in the 1st 100 interviews, 91 00:06:25,160 --> 00:06:29,640 more or less giving a good indication, then the entire data 92 00:06:29,640 --> 00:06:35,840 collection is actually just going more granular in, in the, 93 00:06:35,880 --> 00:06:40,000 in the subgroups. And also, of course, the margin 94 00:06:40,000 --> 00:06:44,360 of error will get smaller with a hard higher number of 95 00:06:44,360 --> 00:06:47,000 interviews. But, but it's, it also gives an 96 00:06:47,000 --> 00:06:51,880 indication that the, the dropout rate is not the, the indication 97 00:06:51,880 --> 00:06:54,920 that it's often being talked about like, oh, if it's under 98 00:06:54,920 --> 00:07:01,200 this level, it doesn't work. So, but, but, but I also tend 99 00:07:01,200 --> 00:07:06,720 to, to, to see the challenge here because now a lot of people 100 00:07:06,720 --> 00:07:10,600 say, OK, the response rate is so low, so you can't rely on the 101 00:07:10,600 --> 00:07:15,160 pulse anymore. And I think both of you and I 102 00:07:15,160 --> 00:07:18,720 know that that's not really that simple, is it? 103 00:07:19,800 --> 00:07:24,560 Not even close. Yes, no, it is really 104 00:07:24,560 --> 00:07:28,880 challenging. I what I've always tried to tell 105 00:07:28,880 --> 00:07:32,080 students is that it it, it's really variable specific. 106 00:07:32,400 --> 00:07:34,680 There, there, there are some variables that you can estimate 107 00:07:35,120 --> 00:07:38,360 pretty reliably with those very low response rates and there are 108 00:07:38,360 --> 00:07:41,400 others for which other variables for which the selection effects 109 00:07:41,400 --> 00:07:45,360 of a low response rate render it not, not terribly useful. 110 00:07:45,560 --> 00:07:50,640 But those tend to be behavioral questions, attitudinal questions 111 00:07:51,200 --> 00:07:56,360 is my in my experience and my readings aren't as vulnerable to 112 00:07:56,400 --> 00:08:00,080 those those types of selection effects as as our behavioral 113 00:08:00,080 --> 00:08:02,400 measures. So when we're doing public 114 00:08:02,400 --> 00:08:07,600 opinion polling, it's it's seems to be a little bit more robust 115 00:08:07,840 --> 00:08:11,600 to these the these low response rates that that we're going to 116 00:08:11,600 --> 00:08:13,360 be living with for the foreseeable future. 117 00:08:13,800 --> 00:08:17,200 Yeah, yeah, we are, aren't we? And and, but, but that is also 118 00:08:17,520 --> 00:08:22,160 part of the magic with the probability sample is that you 119 00:08:23,160 --> 00:08:27,000 the response rate tend to be random as well. 120 00:08:27,280 --> 00:08:31,080 The the the non response. I mean, sorry, because as you 121 00:08:31,080 --> 00:08:35,840 said it, it's the the the attitudinal, the attitudes are 122 00:08:36,720 --> 00:08:42,080 seem are quite reliable still, but but the behavioural and may 123 00:08:42,080 --> 00:08:47,560 maybe just to try to explain it to to the not so nerdy person. 124 00:08:49,520 --> 00:08:54,640 I if if someone is not deep into this swamp as we are. 125 00:08:56,440 --> 00:09:01,080 I, I, I tried to explain it that OK, if, if you want to find 126 00:09:02,720 --> 00:09:06,080 someone helping you with their plumbing, it's really easy to 127 00:09:06,080 --> 00:09:10,080 find the guys who don't have a job, but they might not be the 128 00:09:10,080 --> 00:09:13,480 best ones. And, and from that perspective, 129 00:09:13,480 --> 00:09:16,640 also in the research perspective, if you have very 130 00:09:16,640 --> 00:09:20,800 niche, if you want to find a very niche competence there, you 131 00:09:20,800 --> 00:09:24,480 might have a really big problem with the non response because 132 00:09:24,480 --> 00:09:28,320 you will not reach those. But in attitudes, what you're 133 00:09:28,320 --> 00:09:31,320 going to vote for in the next election, sort of, for instance, 134 00:09:31,680 --> 00:09:34,800 that that is such a broad opinion, which it doesn't matter 135 00:09:34,800 --> 00:09:38,320 if you don't find the perfect or the greatest plumber there who'd 136 00:09:38,320 --> 00:09:43,280 really are going to fix your, your emergency in your, in your 137 00:09:43,280 --> 00:09:46,920 bathroom. So, so that that's me trying to 138 00:09:46,920 --> 00:09:48,840 dumb it down. Maybe I'm completely wrong, but 139 00:09:49,000 --> 00:09:53,560 what's your take on this? Well, it's, it's you're right, 140 00:09:53,600 --> 00:09:56,440 it's hard to dig even deeper into into it off the top of my 141 00:09:56,440 --> 00:10:01,480 head, but but there is a strong correlation between the types of 142 00:10:01,480 --> 00:10:04,960 questions and and the the effects of the of the low 143 00:10:04,960 --> 00:10:09,560 response rates with the behavioral questions just just 144 00:10:09,560 --> 00:10:11,920 be more difficult. Yeah, yeah, yeah. 145 00:10:12,400 --> 00:10:16,600 And and difficult actually means cost more to to to for often 146 00:10:16,600 --> 00:10:19,280 because you can still do it, but it costs a lot more money 147 00:10:19,280 --> 00:10:23,160 because you have to work harder to get the the correct 148 00:10:23,520 --> 00:10:26,040 respondents or reach the correct respondents. 149 00:10:26,040 --> 00:10:29,760 Right, right. I'm, I'm sorry and, and, and 150 00:10:29,800 --> 00:10:34,280 maybe it's, it's easier for us to to, to use some external 151 00:10:34,280 --> 00:10:36,600 methods to validate those behaviours. 152 00:10:36,600 --> 00:10:39,000 That, that it is, it's how do you, how do you validate 153 00:10:39,000 --> 00:10:43,800 somebody's attitude can be done but but indirectly, but with 154 00:10:43,800 --> 00:10:47,360 with behaviors. If we're talking about your, 155 00:10:47,360 --> 00:10:52,560 your visits to the doctor and, and and and so forth. 156 00:10:52,560 --> 00:10:59,360 There are external external records that can be used to in, 157 00:10:59,680 --> 00:11:01,640 you know, the, you know, it takes a lot of effort to do it. 158 00:11:01,640 --> 00:11:05,600 But to, you know, to, to validate what, what, what people 159 00:11:05,600 --> 00:11:09,600 are telling you. And for a lot of other those 160 00:11:09,600 --> 00:11:11,400 types of variables. You know, some of these 161 00:11:11,400 --> 00:11:15,000 wonderful registries that you have in in in Europe are very, 162 00:11:15,000 --> 00:11:19,400 very useful for those those purposes that that you you can 163 00:11:19,400 --> 00:11:24,240 validate health conditions as as as well as health services 164 00:11:25,600 --> 00:11:29,440 receipt that that you can't do with opinions. 165 00:11:29,600 --> 00:11:34,480 The the only the only external way to validate self-reports on 166 00:11:34,480 --> 00:11:36,880 public opinion. You know, the the common one 167 00:11:36,880 --> 00:11:38,920 that we're familiar with and that we get in a lot of trouble 168 00:11:38,920 --> 00:11:42,760 arguing about at the professional conferences is a 169 00:11:44,080 --> 00:11:47,080 horse race questions in the polls who's you know, because we 170 00:11:47,080 --> 00:11:50,800 can there is an external way to validate that and we get in a 171 00:11:50,800 --> 00:11:56,200 lot of trouble when the snapshot polls are are not successful in 172 00:11:56,240 --> 00:12:01,920 in incorrectly validating or correctly, you know, predicting 173 00:12:01,920 --> 00:12:03,680 I guess is a better term to use the. 174 00:12:03,720 --> 00:12:06,640 Outcome, yeah, that's the challenge as well because it's 175 00:12:06,640 --> 00:12:09,000 actually not predictor Tory methods. 176 00:12:09,400 --> 00:12:11,480 So it's as you said, it is a snapshot. 177 00:12:11,480 --> 00:12:16,640 But then we say yeah and enough. And if nothing changes, this 178 00:12:16,640 --> 00:12:20,240 will be the outcome because. That's a big caveat you've got 179 00:12:20,240 --> 00:12:22,240 there, exactly. Exactly. 180 00:12:22,240 --> 00:12:26,360 And that's what actually, sure, you can add, you can, you can 181 00:12:26,360 --> 00:12:33,000 work with predictatorial models, but pollsters, the, the public 182 00:12:33,000 --> 00:12:35,000 opinion pollsters actually don't do that. 183 00:12:36,760 --> 00:12:38,720 They, they, they have a snapshot of today. 184 00:12:38,720 --> 00:12:41,960 This is the last day that we measured before the lecture that 185 00:12:41,960 --> 00:12:44,840 this would be the outcome if it were the election today. 186 00:12:45,400 --> 00:12:49,800 But then it's being talked as being that the a prognosis which 187 00:12:49,800 --> 00:12:53,200 is at per to sign isn't so. So there you also have this 188 00:12:53,200 --> 00:12:57,760 challenge and, and, and just just as an example, in Sweden in 189 00:12:57,760 --> 00:13:01,760 the last two national elections, 4% of the voters changed their 190 00:13:01,760 --> 00:13:07,880 mind after we as an industry did the last questions for for the 191 00:13:07,880 --> 00:13:09,760 election and the election. Interesting. 192 00:13:10,720 --> 00:13:15,840 So we know now that the voter mobility is so much higher than 193 00:13:15,840 --> 00:13:20,760 we used to because of things like Internet, because people 194 00:13:20,760 --> 00:13:25,360 that they're that and and that, that the drama in in media often 195 00:13:25,360 --> 00:13:30,400 tend to create some small, small catastrophe for some candidate 196 00:13:30,400 --> 00:13:34,720 the last couple of days, which really move the voters because 197 00:13:34,720 --> 00:13:39,720 you, you don't, you're not really if in the US you have two 198 00:13:39,720 --> 00:13:43,200 parties and it's more polarized by by the sign maybe. 199 00:13:43,200 --> 00:13:45,360 But but in Sweden, we have eight parties. 200 00:13:45,560 --> 00:13:50,120 But for instance, the days where you were born, once a Social 201 00:13:50,120 --> 00:13:53,280 Democrats, always a Social Democrats is over long ago. 202 00:13:53,640 --> 00:13:58,800 Now we can move the entire spectrum depending on where you 203 00:13:58,800 --> 00:14:01,880 think of the present and you think of the future. 204 00:14:01,880 --> 00:14:04,440 And then the parties are really triangulating this a lot. 205 00:14:05,000 --> 00:14:10,000 So the the polls is actually not a good indication of how good. 206 00:14:10,440 --> 00:14:13,160 No, the sorry, the election is not a good indication of how the 207 00:14:13,160 --> 00:14:15,840 polls are going. But thankfully, although 4% 208 00:14:15,840 --> 00:14:19,040 changed their mind, there's still a pretty good indication. 209 00:14:19,640 --> 00:14:23,120 But if 8% changed their mind, it would be worse. 210 00:14:24,040 --> 00:14:28,400 Yes, well, and, and especially in, in, in in most Western style 211 00:14:28,400 --> 00:14:31,040 democracies anymore the the outcomes of the election, 212 00:14:31,040 --> 00:14:35,120 there's no more landslides. It's it's always the United 213 00:14:35,120 --> 00:14:38,120 States being a a terrible example where it's very close 214 00:14:38,520 --> 00:14:40,960 even when it shouldn't be. It it turns out to be very 215 00:14:40,960 --> 00:14:43,920 close. And so people are going to 216 00:14:43,920 --> 00:14:46,760 continue to use the election outcomes because that's all 217 00:14:46,760 --> 00:14:50,640 we've got. That's the only Yeah, yeah, but 218 00:14:50,640 --> 00:14:52,200 but the old sick available. Exactly. 219 00:14:52,200 --> 00:14:56,720 But at the same time, we have an entire industry in in, in the US 220 00:14:57,760 --> 00:15:01,840 spending billions of dollars getting the voters to change 221 00:15:01,840 --> 00:15:06,040 their minds, which we don't have in much other issues. 222 00:15:06,040 --> 00:15:09,360 So, so, so the pollsters are trying to measure something that 223 00:15:09,360 --> 00:15:13,160 the, that there's $1 million at stake or maybe billions of 224 00:15:13,160 --> 00:15:16,680 dollars at stake, getting the voters to change their mind in 225 00:15:16,680 --> 00:15:21,800 the until the last second, which is a fascinating weird. 226 00:15:22,480 --> 00:15:25,120 It's it's like someone is actually doping the horse in the 227 00:15:25,120 --> 00:15:28,320 horse race. It's interesting analogy. 228 00:15:28,760 --> 00:15:32,200 There are a lot of moving parts. You're right in the election and 229 00:15:33,720 --> 00:15:36,600 there, there are a lot of actors and, and I, you know, they're 230 00:15:36,600 --> 00:15:40,960 not all good actors that, that are, are attempting to influence 231 00:15:40,960 --> 00:15:43,920 the outcome. You know, the political parties, 232 00:15:43,920 --> 00:15:48,320 of course, that's their job is to try to sway voters. 233 00:15:48,320 --> 00:15:52,120 But there are a lot of external groups that, that are also 234 00:15:52,120 --> 00:15:55,640 sending messages and more and more we see these disinformation 235 00:15:55,640 --> 00:16:00,640 campaigns and so forth. And it's also challenging to to 236 00:16:00,640 --> 00:16:02,440 measure their impact. Yeah, yeah, it is. 237 00:16:02,520 --> 00:16:06,440 And and I'm maybe maybe back then to, to, to the the only 238 00:16:06,440 --> 00:16:10,200 joystick we have for for opinions being the horse racing. 239 00:16:10,640 --> 00:16:16,200 But at the same time, proper polls is still a very good 240 00:16:16,200 --> 00:16:20,480 indicator both if there's something bad going on about 241 00:16:20,680 --> 00:16:25,200 real voter temping tampering, exit polls are often the only 242 00:16:25,200 --> 00:16:29,080 proof you have or an indicator of of something not going 243 00:16:29,160 --> 00:16:31,360 properly in in an international election. 244 00:16:31,720 --> 00:16:35,600 But but same time you can also see when the voters change speak 245 00:16:35,600 --> 00:16:41,440 of of of Canada, there was a radical shift in opinions going 246 00:16:41,520 --> 00:16:48,960 going like AUS effect, but going going the opposite way. 247 00:16:50,440 --> 00:16:51,400 We don't. Have to talk more. 248 00:16:51,400 --> 00:16:55,080 That was very interesting. Yeah, some of my Canadian 249 00:16:55,080 --> 00:16:58,960 friends have actually thanked us, you know, because by, by 250 00:16:59,080 --> 00:17:04,119 electing your current president, our current president, we saved 251 00:17:04,119 --> 00:17:06,560 them from from a similar fate. Yeah. 252 00:17:07,280 --> 00:17:10,720 Yeah, which is, which is really tragic and fascinating at the 253 00:17:10,720 --> 00:17:13,640 same time. But you have all this that's 254 00:17:13,640 --> 00:17:16,000 probably an unintended consequence for, for those who 255 00:17:16,000 --> 00:17:19,319 thought they were going to actually drive it home in 256 00:17:19,319 --> 00:17:21,800 Canada. But but at the same time, the 257 00:17:21,800 --> 00:17:25,040 voters actually react to what's happening in the world. 258 00:17:25,480 --> 00:17:30,640 But the polls showed the shift of opinions during this change. 259 00:17:30,800 --> 00:17:35,040 And, and so if we move from, but that's, that maybe that also the 260 00:17:35,040 --> 00:17:36,560 stories a little bit of the argument. 261 00:17:36,560 --> 00:17:40,960 Because if we, if we say that the, the, the election is not a 262 00:17:40,960 --> 00:17:45,400 good indicator of how good the polls are because people change 263 00:17:45,400 --> 00:17:48,240 their mind. And until the last second, then 264 00:17:48,240 --> 00:17:52,720 you have no way of proving that the polls work on measuring 265 00:17:52,720 --> 00:17:55,200 opinions. But the same time we know that 266 00:17:55,200 --> 00:17:59,880 polls is a good indication of explaining when and how the 267 00:17:59,880 --> 00:18:03,240 opinions changed. But you can't prove it if they 268 00:18:03,240 --> 00:18:05,240 changed their mind after you stop measuring. 269 00:18:05,520 --> 00:18:10,160 Because at some sense me, me being in the private sector, I 270 00:18:10,160 --> 00:18:13,720 also, I, I, I really feel the responsibility because if I 271 00:18:14,120 --> 00:18:18,440 present the poll at the borning of the Election Day, I might 272 00:18:18,440 --> 00:18:21,840 change the voter turn out. Because we, we have science 273 00:18:21,840 --> 00:18:25,840 showing that for, for Brexit, for instance, when, when, when, 274 00:18:26,280 --> 00:18:28,160 and also the US and, and Denmark. 275 00:18:28,160 --> 00:18:31,960 There's a bunch of examples that when someone calls the election 276 00:18:31,960 --> 00:18:35,640 based on polls saying this is over, it actually affects the 277 00:18:35,640 --> 00:18:39,840 turn out rate of the election because the people who support 278 00:18:39,840 --> 00:18:42,560 the winning candidates say oh good, I don't have to vote. 279 00:18:43,240 --> 00:18:45,240 Exactly that. I'm certain that's what happened 280 00:18:45,240 --> 00:18:49,560 in 2016 in the in the US. Yeah, yeah, yeah, exactly. 281 00:18:49,560 --> 00:18:52,680 And, and, and, and the Brexit, there's several indicators as 282 00:18:52,680 --> 00:18:56,200 well, because the voter turn out was lower in the remain areas 283 00:18:56,400 --> 00:18:59,440 than in the leave areas. And there were also a lot of 284 00:18:59,680 --> 00:19:02,680 maybe anecdotal reports afterwards saying that hang on, 285 00:19:02,760 --> 00:19:06,040 I voted for Brexit because I didn't think it would matter. 286 00:19:06,200 --> 00:19:09,440 I just wanted to protest because everyone said it would be 287 00:19:09,440 --> 00:19:11,760 remain. So I just want to put in my my 288 00:19:11,760 --> 00:19:14,520 own. So yeah, they have to get get 289 00:19:14,520 --> 00:19:17,840 their act together, but I I still didn't really want to 290 00:19:17,840 --> 00:19:20,560 leave. So it's, it's, it's really 291 00:19:20,560 --> 00:19:23,520 challenging. You're right that the public has 292 00:19:23,520 --> 00:19:28,040 this assumption that the polls are correct within half of a 293 00:19:28,040 --> 00:19:31,760 percentage point. And and it's just not realistic 294 00:19:33,600 --> 00:19:38,120 given our, our, our models to, to expect that that kind of 295 00:19:38,120 --> 00:19:42,000 precision, especially as you said earlier, when we stopped 296 00:19:42,000 --> 00:19:45,800 polling a few days before the actual election takes place. 297 00:19:45,800 --> 00:19:51,040 You cannot capture all of that churning and and last minute 298 00:19:51,040 --> 00:19:52,840 movement in the electorate that's out there. 299 00:19:53,880 --> 00:19:58,720 In the old days we thought we could understand voter turn out 300 00:19:58,800 --> 00:20:02,000 by by the weather conditions on the day of the election. 301 00:20:02,840 --> 00:20:07,160 That certainly doesn't work, but but there is something that 302 00:20:07,160 --> 00:20:10,840 drives turn out that all of our likely voter models cannot 303 00:20:12,480 --> 00:20:17,000 cannot adequately capture, especially with the the fairly 304 00:20:17,000 --> 00:20:21,240 rudimentary questions that that are used, you know, they're 305 00:20:21,240 --> 00:20:25,280 they're all realistic, but but I just I don't think that we have 306 00:20:25,400 --> 00:20:28,960 have a good way of of adequately modeling, you know, the the turn 307 00:20:28,960 --> 00:20:30,080 out. No. 308 00:20:30,080 --> 00:20:34,400 And I, I also think because it also has to do with, with, 309 00:20:34,880 --> 00:20:39,040 because you said something, I'm just going to air that idea as 310 00:20:39,040 --> 00:20:41,160 well because it's always a 5050 race. 311 00:20:41,760 --> 00:20:45,120 And, and I'm starting to think that we're, we're getting so 312 00:20:45,120 --> 00:20:49,880 close to game theory in the political consultant realm that 313 00:20:49,880 --> 00:20:53,600 you've, you're creating this 5050 choices for everyone 314 00:20:53,840 --> 00:20:58,120 because you're trying to try also using really reliable 315 00:20:58,120 --> 00:21:00,680 polls, really expensive polls to dig out. 316 00:21:00,680 --> 00:21:03,720 Where is the key issue that I have to get the people to tip? 317 00:21:04,520 --> 00:21:08,560 But, but I think that they're too professional here in getting 318 00:21:08,560 --> 00:21:11,760 this. So it's like a, a, a coin toss 319 00:21:12,200 --> 00:21:16,760 for the voter to to much that the people actually care because 320 00:21:16,760 --> 00:21:20,280 there's something weird here. How come it's always a 5050 321 00:21:20,280 --> 00:21:22,080 race? It was the same in Poland the 322 00:21:22,080 --> 00:21:26,040 other day. Two, one or two percentage point 323 00:21:26,480 --> 00:21:30,880 over for, for, for one of them. There's something here that's so 324 00:21:30,880 --> 00:21:34,240 weird. It hasn't to do with polls. 325 00:21:34,240 --> 00:21:36,960 It's it's actual, actual election outcome. 326 00:21:36,960 --> 00:21:37,560 Sorry. Yeah. 327 00:21:38,040 --> 00:21:38,840 Right. No, that's OK. 328 00:21:39,080 --> 00:21:43,400 That's, it's polarization is, is what I, you know, yeah, see 329 00:21:43,560 --> 00:21:46,400 discussed in the literature and in the media at, at a regular 330 00:21:46,400 --> 00:21:47,280 basis. And they're right. 331 00:21:47,520 --> 00:21:53,320 There is just a heavy dose of, of polarization in, in most of 332 00:21:53,320 --> 00:21:55,040 our Western societies these days. 333 00:21:55,360 --> 00:22:02,200 And it's produced all kinds of negative outcomes, including 334 00:22:02,200 --> 00:22:04,760 what we see in, in these elections. 335 00:22:05,040 --> 00:22:09,240 These outcomes in these elections are probably just a a 336 00:22:09,240 --> 00:22:13,080 by product of of of those broader social divisions that 337 00:22:13,080 --> 00:22:15,360 that are being discussed. Yeah. 338 00:22:15,480 --> 00:22:18,720 But but I also think that it's it's because you focus on these 339 00:22:18,720 --> 00:22:23,760 polarizing issues because so, so because the communication is so 340 00:22:23,760 --> 00:22:28,320 heavily, you research it so deeply to making sure that you 341 00:22:28,320 --> 00:22:33,240 get this polarization, these false choices, basically black 342 00:22:33,240 --> 00:22:35,320 or white choices. Yeah. 343 00:22:36,000 --> 00:22:37,960 They refer to them as wedge issues. 344 00:22:38,240 --> 00:22:41,560 Oh yeah, yeah, yeah. And, and, and, and then when you 345 00:22:41,560 --> 00:22:44,920 have a normal distribution curve on anything, because that's what 346 00:22:44,920 --> 00:22:48,880 you have, you're being actually forced to, you're at the tip tip 347 00:22:48,880 --> 00:22:51,560 of the edge all the time and just tipping over for one or 348 00:22:51,560 --> 00:22:54,200 others issue. There's something there that 349 00:22:54,200 --> 00:22:56,840 hasn't to do much with poles except that poles are actually 350 00:22:56,840 --> 00:22:59,360 being used to to find these issues. 351 00:22:59,360 --> 00:23:02,560 But it's also how you use this information that is the 352 00:23:02,560 --> 00:23:04,960 challenge here. And when the grayscale 353 00:23:05,880 --> 00:23:08,840 disappears and everything is black and white, yeah, you're, 354 00:23:09,080 --> 00:23:12,760 you're either black or white. So, so there's something there. 355 00:23:12,760 --> 00:23:15,600 But I realise now that we're completely off the topic but but 356 00:23:16,560 --> 00:23:18,720 it's OK. But still interesting. 357 00:23:18,720 --> 00:23:21,520 Yeah, Yeah, it is. But I also, but because I said 358 00:23:21,520 --> 00:23:25,120 something that I, I think that listeners might react to because 359 00:23:25,120 --> 00:23:27,880 I said if, if the, if the elections is not a good 360 00:23:27,880 --> 00:23:34,080 indicator of reliable polls, because if the voter mobility 361 00:23:34,200 --> 00:23:37,920 increase, then it will be a worse and worse indicator. 362 00:23:39,240 --> 00:23:42,040 But that actually doesn't mean that the, because you're, you're 363 00:23:42,040 --> 00:23:43,560 talked about this a little bit as well. 364 00:23:43,760 --> 00:23:47,320 That does actually doesn't mean that you cannot validate other 365 00:23:47,320 --> 00:23:49,840 polls because you can do it on some other stuff. 366 00:23:50,440 --> 00:23:54,320 There are some, some behavioural issues, as you talked about or, 367 00:23:54,320 --> 00:23:57,160 or topics that that you could, you could look at. 368 00:23:57,800 --> 00:24:02,200 But I think also one thing that is getting forgotten is that 369 00:24:02,200 --> 00:24:06,680 actually the theories that's being used has been working for 370 00:24:07,120 --> 00:24:11,960 almost over 70 years now. And the theory includes also the 371 00:24:11,960 --> 00:24:14,280 margin of error and and that's it. 372 00:24:14,320 --> 00:24:17,760 We're 95% confident that it's within the margin of error. 373 00:24:18,280 --> 00:24:24,040 OK, that's a that it's, it seems a little bit dodgy to say that, 374 00:24:24,040 --> 00:24:28,360 but it's also you're 95% confident it's within the margin 375 00:24:28,360 --> 00:24:32,520 of error, which is quite good in most issues and the theories are 376 00:24:32,520 --> 00:24:36,200 still there, yes. Now that's largely sampling 377 00:24:36,200 --> 00:24:38,600 error, right? Yeah, that we're talking about. 378 00:24:38,960 --> 00:24:43,600 Yeah. You know, sometimes I, I, I use 379 00:24:43,600 --> 00:24:47,320 the rule of thumb that you, you double that margin of error to 380 00:24:47,320 --> 00:24:51,240 account for non sampling errors. If if, if it's really, really 381 00:24:51,240 --> 00:24:57,600 important to you to understand you know what the odds of one 382 00:24:57,600 --> 00:25:00,720 candidate or the other winning the election is, it's it's 383 00:25:00,760 --> 00:25:03,720 probably safer to double that that margin of error. 384 00:25:04,080 --> 00:25:05,720 Yeah, but but but. It's a rule of. 385 00:25:05,840 --> 00:25:08,240 Thumb yeah, yeah, that, that is a good point because if you're 386 00:25:08,240 --> 00:25:11,680 stuck at the the 1-2 percentage point or whatever, it's like, 387 00:25:11,680 --> 00:25:14,600 OK, if it's really important, why not double it? 388 00:25:14,600 --> 00:25:17,720 That that's a good, that's a good point, especially when it's 389 00:25:17,720 --> 00:25:21,320 a 5050 because the margin is actually bigger than on the 390 00:25:21,320 --> 00:25:27,160 middle than they saw on there and and but it also needs to be 391 00:25:27,920 --> 00:25:30,760 a random sample or probably. Absolutely. 392 00:25:31,080 --> 00:25:33,600 And that's the thing that's getting forgotten because 393 00:25:34,600 --> 00:25:37,800 there's a lot of arguments saying, OK, the response rate is 394 00:25:37,800 --> 00:25:42,720 so low, you cannot do probably probabilistic research anymore. 395 00:25:42,880 --> 00:25:47,560 It's no point. And, and I see no proof for 396 00:25:47,560 --> 00:25:50,480 that, no actual proof. It's, it's just like an 397 00:25:50,480 --> 00:25:53,280 accusation being thrown around all the time. 398 00:25:56,520 --> 00:26:00,120 But but I see it all the time. Although there are no real proof 399 00:26:00,280 --> 00:26:05,000 that the the response rate low responses, it is affecting the 400 00:26:05,000 --> 00:26:08,760 opinions as as we talked about, but it's still being OK. 401 00:26:08,760 --> 00:26:11,720 It's no point doing random samples. 402 00:26:11,720 --> 00:26:14,200 We have to go to alternative methods. 403 00:26:14,840 --> 00:26:17,680 But then you'd completely lose the margin of error. 404 00:26:19,400 --> 00:26:21,520 Totally. And you know some of these 405 00:26:22,080 --> 00:26:26,760 studies, these these poll aggregators, our, our colleague 406 00:26:26,880 --> 00:26:31,480 Claire Durand in Montreal does a lot of work looking at the 407 00:26:31,480 --> 00:26:35,560 accuracy of of final polls based upon sampling methods. 408 00:26:35,560 --> 00:26:40,280 And she finds consistently that the probability based sample 409 00:26:40,280 --> 00:26:43,800 frames are, are more accurate, not not tremendously more 410 00:26:43,800 --> 00:26:45,800 accurate, but more accurate consistently. 411 00:26:46,760 --> 00:26:48,480 Yeah. And and there you have a, a, a 412 00:26:49,120 --> 00:26:52,840 complicating factor as well because no one knows really what 413 00:26:52,840 --> 00:26:56,440 happened in this last election. But but the polls were so close, 414 00:26:56,440 --> 00:26:59,400 it was basically mathematically impossible that they were 415 00:26:59,560 --> 00:27:02,440 independent of each other. So there was something where 416 00:27:02,440 --> 00:27:06,040 some say it's hurting, I don't know because I'm not there, but 417 00:27:06,040 --> 00:27:09,200 there's something there because if you don't know, if it is a 418 00:27:09,200 --> 00:27:14,080 really independent poll from the others, you also you have that 419 00:27:14,080 --> 00:27:17,320 variable here as well so that no one knows what's happening 420 00:27:17,320 --> 00:27:20,480 because the, this is being done in commercial actors. 421 00:27:20,720 --> 00:27:24,560 But there's also I, I often joke I could make a perfect poll in 422 00:27:24,560 --> 00:27:27,200 Sweden without asking any questions because there are so 423 00:27:27,200 --> 00:27:31,640 many good polls to look at. And, and if you're, if you're at 424 00:27:31,640 --> 00:27:34,840 in close, the close to the other ones, no one would rise and 425 00:27:34,920 --> 00:27:38,800 raise an eyebrow. And you will save a lot of money 426 00:27:38,800 --> 00:27:41,360 doing that, but you will of course be lying. 427 00:27:42,360 --> 00:27:46,240 But if it's your reputation at stake, which is what what we're 428 00:27:46,240 --> 00:27:51,120 talking about as as a poster and you have to show you that you're 429 00:27:51,120 --> 00:27:55,040 actually are good enough to be trusted for the commercial work 430 00:27:55,280 --> 00:27:58,320 because you don't make money on on on the horse race as as a 431 00:27:59,320 --> 00:28:02,160 poster for the media, you actually lose money. 432 00:28:03,280 --> 00:28:06,880 You make that. You make money working for the 433 00:28:06,880 --> 00:28:14,680 parties, but and they're they're they, I talked to a lot of of 434 00:28:14,680 --> 00:28:18,320 those and they say, yeah, we never know, use anything else 435 00:28:18,320 --> 00:28:23,160 but proper probability samples, random samples, because they 436 00:28:23,160 --> 00:28:28,600 can't afford to be wrong and they also get paid to be right. 437 00:28:29,200 --> 00:28:31,160 So that's true. Yeah, that's true. 438 00:28:31,240 --> 00:28:34,360 You know, I, I don't think that there's a lot of hurting, you 439 00:28:34,360 --> 00:28:37,840 know, I, I understand. And the concern about that I, 440 00:28:39,120 --> 00:28:43,280 yeah, I, I've seen so many polls that that seem to be outliers 441 00:28:43,280 --> 00:28:45,760 and the, the pollster was kind of embarrassed about it. 442 00:28:45,760 --> 00:28:47,680 Yeah, but this is what the data showed. 443 00:28:48,160 --> 00:28:51,800 I, those would be the ideal candidates for a little hurting. 444 00:28:52,080 --> 00:28:57,120 My I maybe they tweak and this is not hurting, tweak the 445 00:28:57,120 --> 00:29:00,680 variables in their models, their weights and sample weights and 446 00:29:00,680 --> 00:29:03,040 so forth. But that's well. 447 00:29:03,560 --> 00:29:05,160 Yeah, Yeah, yeah. I get to what you mean. 448 00:29:05,160 --> 00:29:08,800 And as I think there's different layer layers of this as well, 449 00:29:08,800 --> 00:29:13,840 because OK, in Sweden, I've seen so many obvious indicators of 450 00:29:13,840 --> 00:29:18,120 this because some non probability polls were so way 451 00:29:18,120 --> 00:29:21,040 off from from us, Ipsos and Kantar. 452 00:29:21,080 --> 00:29:25,040 We were in the same space all the entire time during because 453 00:29:25,080 --> 00:29:28,160 in general, we we publish a poll every month during the election 454 00:29:28,160 --> 00:29:32,360 year or even with between elections of all the time 455 00:29:32,360 --> 00:29:37,440 actually, but but close to the two or three latest polls before 456 00:29:37,440 --> 00:29:40,560 the election, the non probability base subtly close up 457 00:29:40,560 --> 00:29:46,400 to us, which is really impossible to explain because 458 00:29:46,640 --> 00:29:49,280 and that has happened with so many providers for non 459 00:29:49,280 --> 00:29:51,160 probability based panels in Sweden. 460 00:29:52,920 --> 00:29:55,880 And because and the argument for them has always been we're 461 00:29:55,880 --> 00:29:58,000 better because we don't have the problem. 462 00:29:58,000 --> 00:30:00,200 We don't response. We are using the water methods, 463 00:30:00,200 --> 00:30:03,720 whatever, whatever. And if and that's how they 464 00:30:03,720 --> 00:30:08,880 explain to be different between the election from us, but it 465 00:30:08,880 --> 00:30:14,360 doesn't explain how we continue to to that the probability based 466 00:30:14,440 --> 00:30:17,440 are to continue to be close to each other the entire time. 467 00:30:18,800 --> 00:30:21,200 Of course, there's some noise there because and it should be, 468 00:30:21,200 --> 00:30:25,360 but it's giving the same view of the opinion and the non 469 00:30:25,360 --> 00:30:30,240 probability being completely different until exactly in 470 00:30:30,360 --> 00:30:34,080 before the election. So that that is that I've seen 471 00:30:34,080 --> 00:30:37,280 this for at least three elections because I'm so now 472 00:30:37,280 --> 00:30:39,480 I've followed this for such a long time. 473 00:30:39,480 --> 00:30:44,000 So this cannot be a coincidence, but I can never prove it. 474 00:30:44,000 --> 00:30:47,320 But it's still annoying me because either you're completely 475 00:30:47,320 --> 00:30:50,320 different and you get it right, but then you should be 476 00:30:50,320 --> 00:30:54,680 completely different even in at the Election Day, because then 477 00:30:55,000 --> 00:30:58,680 the traditional methods doesn't work and they have a new perfect 478 00:30:58,680 --> 00:31:02,000 method. Yeah, boy, you know, you, I, I 479 00:31:02,000 --> 00:31:04,040 almost, I, I don't have an answer to that either. 480 00:31:04,560 --> 00:31:05,600 No, no one has. Yeah. 481 00:31:05,600 --> 00:31:08,560 But, but I think it, it, it's going to require some forensic 482 00:31:08,560 --> 00:31:11,480 analysis of those non probability polls because when 483 00:31:11,480 --> 00:31:15,480 we talk about non probability, what's the definition? 484 00:31:15,520 --> 00:31:19,240 It, it, it's, it's what it's not, it's not a probability 485 00:31:19,240 --> 00:31:22,640 poll, but it could be anything. Yes, there's such a variety. 486 00:31:22,640 --> 00:31:25,360 They're they're not all the same animal, these non probability 487 00:31:25,360 --> 00:31:28,520 polls. So I I don't know that. 488 00:31:28,880 --> 00:31:32,560 That is true and and exactly and it's it's like yeah, yeah. 489 00:31:33,600 --> 00:31:37,600 And that is part of the problem as well as as as you know as 490 00:31:37,840 --> 00:31:42,000 because you know, you've left the scientific field to be just 491 00:31:42,000 --> 00:31:44,480 to be very broad. OK, we have a definition of what 492 00:31:44,480 --> 00:31:48,040 probability based sample is and that's the the theory we have 493 00:31:48,040 --> 00:31:50,560 for 70 + 8. OK, there's details. 494 00:31:50,560 --> 00:31:54,720 There's so there's so many things there that also makes 495 00:31:54,720 --> 00:31:58,080 that a, a, a sliding scale to some to, to a lot of sense. 496 00:31:58,280 --> 00:32:01,200 But it's actually at least there's some sort of theory bit 497 00:32:01,200 --> 00:32:05,200 that that I usually say that you try to replicate it to the best 498 00:32:05,200 --> 00:32:09,240 way possible because we all know back to the response rate. 499 00:32:09,240 --> 00:32:12,600 You cannot do a perfect probability samples probably 500 00:32:12,600 --> 00:32:16,120 anywhere anymore. But you can try to replicate it 501 00:32:16,120 --> 00:32:19,400 to the best way possible. And and maybe in the simplest 502 00:32:19,400 --> 00:32:24,440 form, you as a poster have a big bucket when you randomly sample 503 00:32:24,440 --> 00:32:28,000 people. And then we ask you, but not the 504 00:32:28,080 --> 00:32:30,480 opposite around. You cannot stand in line saying 505 00:32:30,480 --> 00:32:34,440 I want to give my opinion to you because that that's the simplest 506 00:32:34,440 --> 00:32:37,400 difference. But but, but as you said that 507 00:32:37,400 --> 00:32:40,360 there is no definition of non probability based samples. 508 00:32:40,360 --> 00:32:42,280 There, there are so many different things. 509 00:32:42,280 --> 00:32:46,240 And maybe that's also why why you can't say a method works 510 00:32:46,320 --> 00:32:50,000 that's not based on probability based because it's like, OK, 511 00:32:51,000 --> 00:32:55,920 it's just not what we know. What's the underlying theory? 512 00:32:55,920 --> 00:32:59,120 Yeah, exactly. Because I'm, I'm still waiting 513 00:32:59,120 --> 00:33:00,680 for that. I would love to see another 514 00:33:00,680 --> 00:33:03,800 method that actually works, but it has to be some theory that 515 00:33:03,800 --> 00:33:08,440 can be confirmed. Now at at least quota sampling, 516 00:33:08,440 --> 00:33:11,120 you understood the rationale there there was, there was a 517 00:33:11,120 --> 00:33:14,920 reason for what they were doing. Yeah, that's true, but. 518 00:33:15,400 --> 00:33:18,400 A lot of these other approaches it's it's it's not so clear and 519 00:33:18,440 --> 00:33:21,680 and the problem well, the general problem would be 520 00:33:22,720 --> 00:33:27,400 inaccuracy in in a lot of these election polls leads to a lot of 521 00:33:27,720 --> 00:33:32,960 public suspicion regarding the quality of the the polls. 522 00:33:32,960 --> 00:33:36,560 They don't, they don't believe the polls, especially you think 523 00:33:36,560 --> 00:33:39,920 about the motivated reasoning theories that they don't like 524 00:33:39,920 --> 00:33:43,200 the polls when they don't show the results that they would like 525 00:33:43,200 --> 00:33:46,520 to see. And if you if, if in a heavily 526 00:33:46,520 --> 00:33:49,640 polarized society, there's always going to be a significant 527 00:33:50,720 --> 00:33:53,800 fraction of the population that does not like what, what the 528 00:33:53,800 --> 00:33:56,680 polls are, are, are telling us. And and that they first thing 529 00:33:56,680 --> 00:34:00,680 they do is well, it's fake polls and some of our candidates are 530 00:34:00,680 --> 00:34:02,800 capitalized on that. Yeah. 531 00:34:02,800 --> 00:34:04,720 And that and yeah, that's a good point. 532 00:34:04,720 --> 00:34:09,320 And it's also, it's hitting the, the integratives of, of, of 533 00:34:09,320 --> 00:34:11,639 science in general as well. Yes, it is. 534 00:34:12,040 --> 00:34:17,000 Because polls is polls and, and you have spent the entire career 535 00:34:17,199 --> 00:34:20,280 not political polls, but doing, using polls as part of your 536 00:34:20,280 --> 00:34:23,760 research. And and, but, but no one knows 537 00:34:23,760 --> 00:34:27,120 the difference between these horse race polls that no one 538 00:34:27,120 --> 00:34:32,000 pays for because media just gets them and the proper research 539 00:34:32,000 --> 00:34:36,120 that's being based on on, on on the same theories, which which 540 00:34:36,120 --> 00:34:40,120 is a quite big dilemma because there also the census bureaus 541 00:34:40,120 --> 00:34:44,520 use the same theories to a large extent on the same methods. 542 00:34:45,440 --> 00:34:47,480 You know, you're absolutely right about that and, and it's 543 00:34:47,480 --> 00:34:51,719 an interesting point. A lot of my work is back to my 544 00:34:51,719 --> 00:34:55,520 collaborations with Richard Wernke is, is related to cancer 545 00:34:55,520 --> 00:35:02,320 control and there are dozens of validation studies that did you 546 00:35:02,320 --> 00:35:05,400 get a mammogram And then can we have your permission to look at 547 00:35:05,400 --> 00:35:07,320 your medical records to confirm the dates? 548 00:35:07,320 --> 00:35:09,600 And a lot of it is is not whether or not they received 549 00:35:09,600 --> 00:35:13,640 the, the the exam, but the, the, the, is it in the proper time 550 00:35:13,640 --> 00:35:15,280 interval? Do they recall was it in the 551 00:35:15,280 --> 00:35:18,640 last five years or whatever the the Cancer Society 552 00:35:18,640 --> 00:35:22,800 recommendations are for, for speeding those studies? 553 00:35:23,120 --> 00:35:26,400 Yeah, we've got a pretty good idea what the, the concordance 554 00:35:26,400 --> 00:35:30,360 rates between the reports and what the medical records tell us 555 00:35:30,360 --> 00:35:35,320 are across all of these cancer screening medical procedures 556 00:35:35,320 --> 00:35:37,760 today. Every single one of those 557 00:35:37,760 --> 00:35:43,200 studies is based on probability sampling it if If they came in 558 00:35:43,200 --> 00:35:48,200 and they were non probability sampling I I, I, I'm not sure 559 00:35:48,200 --> 00:35:50,560 they'd even get published. Yeah, yeah. 560 00:35:50,640 --> 00:35:56,280 But but when the polls in quotation marks are wrong in in 561 00:35:56,280 --> 00:36:03,120 in in an election, I think it also hurts the willingness to 562 00:36:03,120 --> 00:36:07,920 finance these kinds of standards because the general idea is that 563 00:36:07,920 --> 00:36:10,440 OK, polls don't work, the response rates this to low, 564 00:36:10,440 --> 00:36:12,840 everyone knows it doesn't work. Look at the last election. 565 00:36:12,840 --> 00:36:18,600 Why should we spend a bunch of money on these non working 566 00:36:18,600 --> 00:36:21,280 methods? So I, I see indicators of that 567 00:36:21,280 --> 00:36:24,360 for, for several places because it's so hard, because the 568 00:36:24,360 --> 00:36:28,960 politicians are the one who, who gives grants and also they have 569 00:36:28,960 --> 00:36:31,000 the voters who the most vocal voters. 570 00:36:31,360 --> 00:36:34,280 And as we say now, if it's a poll you don't like, yeah, it's 571 00:36:34,280 --> 00:36:36,160 the polls wrong. They they are biased. 572 00:36:36,160 --> 00:36:37,960 Can't be right. Yeah, exactly. 573 00:36:38,480 --> 00:36:41,360 Because everyone I know thinks like me so and, and the poll is 574 00:36:41,360 --> 00:36:45,200 saying something different, which I have heard a lot of 575 00:36:45,200 --> 00:36:48,360 times because I work for politics, political issues. 576 00:36:49,720 --> 00:36:52,320 But that's how it works. Because you are in your own 577 00:36:52,320 --> 00:36:55,280 bubble. And of course you don't meet the 578 00:36:55,280 --> 00:36:59,560 opposition, no, no matter where where you are to that extent 579 00:36:59,560 --> 00:37:02,040 because you don't talk about those issues with people who are 580 00:37:02,400 --> 00:37:03,800 probably on the complete opposite. 581 00:37:04,960 --> 00:37:06,320 But but. Partially, that's true. 582 00:37:06,600 --> 00:37:09,680 Yeah, which is which is fine as well because you don't want to 583 00:37:09,680 --> 00:37:10,920 fight all the time. To be honest. 584 00:37:10,920 --> 00:37:13,240 It could be it's it's OK to live your life. 585 00:37:13,240 --> 00:37:17,000 But if you start extrapolating that and also say, OK, the post 586 00:37:17,000 --> 00:37:20,240 don't work. So I don't trust the science, I 587 00:37:20,240 --> 00:37:23,800 don't trust the statistics from the census bureaus. 588 00:37:24,000 --> 00:37:30,240 Suddenly it's like quickly unraveling worldview where 589 00:37:30,240 --> 00:37:34,040 everything is contested and that that's basically the core. 590 00:37:34,040 --> 00:37:38,600 Why why I'm I'm constantly nagging about this is about this 591 00:37:38,600 --> 00:37:45,000 part because the knowledge society and a fact based society 592 00:37:45,000 --> 00:37:49,400 is so much based on what the core of what we're actually 593 00:37:49,400 --> 00:37:52,440 doing. Although I'm commercial and and 594 00:37:52,440 --> 00:37:55,520 you, you come from the university and, and and the real 595 00:37:55,520 --> 00:37:59,120 science part, it's the same. Overlap. 596 00:37:59,120 --> 00:38:02,800 We're all part of the same statistical knowledge 597 00:38:02,800 --> 00:38:05,960 infrastructure. Yeah, yeah, yeah, you're right 598 00:38:05,960 --> 00:38:09,480 in in the US right now that that that infrastructure is being 599 00:38:09,920 --> 00:38:13,320 taken down. The the the NISTA of a national 600 00:38:13,320 --> 00:38:18,680 survey of drug use and health. They fired the entire staff. 601 00:38:18,760 --> 00:38:20,400 Yeah, the federal government did. 602 00:38:20,640 --> 00:38:25,000 Yeah, it's tragic. And you, you start politicizing 603 00:38:25,000 --> 00:38:29,680 knowledge and, and we've seen this before in other countries 604 00:38:30,920 --> 00:38:34,200 in, in Russia or the Soviet Union, the, the science was 605 00:38:34,200 --> 00:38:37,480 heavily politicized. You, you only were allowed to 606 00:38:38,360 --> 00:38:41,560 come up with the, with the findings that was in line with 607 00:38:41,560 --> 00:38:45,520 the party. So, so we, we know what happens 608 00:38:45,520 --> 00:38:47,480 and we don't know where we're going. 609 00:38:47,480 --> 00:38:52,280 But but just to stay out the politics because it's it's 610 00:38:52,280 --> 00:38:58,200 really it's it's really too sad to talk about even people who 611 00:38:58,200 --> 00:39:03,720 don't think like you need to trust facts like gravity works. 612 00:39:03,840 --> 00:39:07,040 It's a good thing. We need a common, a Common Core 613 00:39:07,040 --> 00:39:09,120 of knowledge. Yeah, we did. 614 00:39:09,120 --> 00:39:11,320 It's uncontested. Exactly. 615 00:39:11,640 --> 00:39:15,200 And and, and sure. And, and we also have to trust 616 00:39:15,440 --> 00:39:19,760 the spokesperson who stands behind science because 617 00:39:19,760 --> 00:39:23,880 politicizing that I, I, I know John Krosnick did a lot of 618 00:39:23,880 --> 00:39:26,800 studies on this. When, when, when in climate, in 619 00:39:26,800 --> 00:39:32,000 climate studies, when he did experiments, a scientist saying 620 00:39:32,000 --> 00:39:35,040 something about climate change, but then he did the same, but 621 00:39:35,040 --> 00:39:37,720 adding a political hat on the same person. 622 00:39:38,320 --> 00:39:42,040 The trust was degrading rapidly, of course, because suddenly you 623 00:39:42,040 --> 00:39:45,800 weren't this independent scientist, you were a political 624 00:39:45,800 --> 00:39:48,960 actor using science to get your point through it. 625 00:39:48,960 --> 00:39:52,320 That's how it was the the person was saying the same thing, but 626 00:39:52,320 --> 00:39:57,240 it was being perceived as trying to getting political about the 627 00:39:57,240 --> 00:39:59,440 issue. So, so there's something there 628 00:39:59,440 --> 00:40:03,400 that that really needs it's, it's, it's a really, really 629 00:40:03,400 --> 00:40:07,960 important issue just to move away from trying to push your 630 00:40:07,960 --> 00:40:11,760 view into explaining this is the way it is. 631 00:40:12,720 --> 00:40:16,040 So yeah, I don't know what the fuck to make of that, sorry. 632 00:40:16,880 --> 00:40:18,600 No, that's OK. We're we're we're down the 633 00:40:18,600 --> 00:40:19,600 rabbit hole now. Yeah. 634 00:40:20,080 --> 00:40:24,640 Really, I so don't mention the war. 635 00:40:24,640 --> 00:40:29,560 Oh, OK, it's all right. Like I, I always do that. 636 00:40:29,560 --> 00:40:31,800 I'm, I'm sorry, but oh. No, that's OK. 637 00:40:31,840 --> 00:40:35,000 But but it but it's really interesting to but we also have 638 00:40:35,000 --> 00:40:41,480 a challenge because we shouldn't contest facts because it in the 639 00:40:41,480 --> 00:40:46,160 long time it benefits no one how you use the facts. 640 00:40:46,160 --> 00:40:51,000 That's a different thing That could be a political thing, but 641 00:40:51,000 --> 00:40:56,000 facts should still be there. How many people living in a 642 00:40:56,000 --> 00:41:02,160 country whatever all these data or or or also the views, the 643 00:41:02,160 --> 00:41:05,360 opinions of people is often really important. 644 00:41:06,160 --> 00:41:10,320 The population estimation of how how people think is often a a 645 00:41:10,320 --> 00:41:12,760 vital part. Even between the election of of 646 00:41:12,760 --> 00:41:16,600 so many things, how we behaved. We had a pandemic, but the 647 00:41:16,600 --> 00:41:20,800 opinion of of the people and and the behavioural that that that 648 00:41:20,800 --> 00:41:26,320 the opinions resulted in was vital to handle the to do not 649 00:41:26,320 --> 00:41:29,920 panic, for instance, the simplest part and and we need to 650 00:41:29,920 --> 00:41:33,720 know that and we need to trust that those opinions are 651 00:41:33,840 --> 00:41:35,600 representative for the population. 652 00:41:36,080 --> 00:41:38,640 Well, it, it, it, it, it really is a challenge. 653 00:41:38,640 --> 00:41:42,560 I you, you mentioned the COVID and, and just the, the ongoing 654 00:41:42,560 --> 00:41:45,000 controversy. It's, it's almost becoming like 655 00:41:45,000 --> 00:41:48,480 the Kennedy assassination. You know, what is the origin of 656 00:41:48,480 --> 00:41:50,760 the COVID virus? Is it, is it from that, that 657 00:41:50,760 --> 00:41:55,400 laboratory and that game of function research, or is it a, 658 00:41:56,200 --> 00:41:59,200 an, a terrible, terrible accident that, that took place 659 00:41:59,200 --> 00:42:01,600 in the, in the, in that open wet market? 660 00:42:02,520 --> 00:42:03,640 I don't know that we'll ever know. 661 00:42:04,280 --> 00:42:07,440 No, that that exactly. It also has to do with with the 662 00:42:08,640 --> 00:42:10,560 trust between people. Yeah. 663 00:42:10,920 --> 00:42:14,680 Agreed that the the inter human trust is is the key in in a 664 00:42:14,680 --> 00:42:17,440 democracy. We can trust our neighbors and 665 00:42:17,440 --> 00:42:21,400 then realise that we have laws and rules and regulation that we 666 00:42:21,400 --> 00:42:26,360 follow them because they are the consensus of the culture of, of, 667 00:42:26,360 --> 00:42:28,200 of, of a democracy. Yeah. 668 00:42:28,200 --> 00:42:30,880 And, and there's an incredible amount of trust associated with 669 00:42:30,880 --> 00:42:35,960 the scientific enterprise that that what you are reporting is 670 00:42:35,960 --> 00:42:39,400 actually what you did. And, you know, I, I, I think in 671 00:42:39,400 --> 00:42:43,400 the US and the transparency initiative and other efforts 672 00:42:43,400 --> 00:42:48,000 around the world, you know, to reveal as much information as 673 00:42:48,000 --> 00:42:51,760 possible, not not only so that the results can be replicated, 674 00:42:51,760 --> 00:42:54,680 but also so that that other scientists will accept as 675 00:42:54,680 --> 00:42:57,840 credible. Yeah, the the research that 676 00:42:58,240 --> 00:43:02,480 that, that you're reporting and. Yeah, yeah, that, that that's 677 00:43:02,480 --> 00:43:04,320 true. And and also the the the 678 00:43:04,880 --> 00:43:09,360 complicating part in actually explaining complicated details 679 00:43:09,360 --> 00:43:14,240 in a trustworthy way without having the audience to read all 680 00:43:14,240 --> 00:43:17,960 the footnotes. But they should be there. 681 00:43:18,480 --> 00:43:23,560 But it's yeah, so so that. That that information should be 682 00:43:23,560 --> 00:43:26,240 there for those who are interested in primarily other 683 00:43:26,360 --> 00:43:29,320 other scientists and in in understanding those details. 684 00:43:30,840 --> 00:43:33,080 Yeah. And maybe that's the last topic 685 00:43:33,080 --> 00:43:36,240 because I realized we have completely way off this, but 686 00:43:36,520 --> 00:43:40,320 actually you, you presented a, a paper with, with one, one of 687 00:43:40,320 --> 00:43:42,360 your colleagues at the latest. Peter Miller. 688 00:43:42,600 --> 00:43:48,120 Peter Miller Yes, at the latest April conference about what 689 00:43:48,120 --> 00:43:51,120 happens when you, when you present more information about 690 00:43:51,120 --> 00:43:55,040 what's the transparency. Can you just talk a little bit 691 00:43:55,040 --> 00:43:56,960 about that? Oh, sure, yeah, sure. 692 00:43:57,760 --> 00:44:02,400 So the, the E POR transfer initiative was actually Peter's 693 00:44:02,400 --> 00:44:06,800 brainchild when he was president of the association back about 15 694 00:44:06,800 --> 00:44:10,000 years ago. And it, it was a heavy lift on 695 00:44:10,000 --> 00:44:15,160 the part of the whole association to to develop a, a 696 00:44:15,160 --> 00:44:20,400 workable model for a, a, a program that would encourage 697 00:44:21,040 --> 00:44:26,680 survey organizations, commercial government, academic survey 698 00:44:26,680 --> 00:44:33,640 organizations to make a pledge, A voluntary pledge to report on 699 00:44:33,640 --> 00:44:39,160 a routine basis for all studies that they publicly report all of 700 00:44:39,160 --> 00:44:42,320 the mythological details that are in the official code of 701 00:44:42,320 --> 00:44:45,960 conduct, professional code of conduct for the association. 702 00:44:45,960 --> 00:44:50,760 And there are over 100 members of the association of, of the 703 00:44:50,760 --> 00:44:54,120 Transparency Initiative today that that have taken that pledge 704 00:44:54,120 --> 00:44:57,080 and, and that report that information, it's never been 705 00:44:57,080 --> 00:45:01,560 evaluated. And, you know, and it became 706 00:45:01,560 --> 00:45:05,480 obvious sometime back Peter and I had discussions that, well, 707 00:45:05,720 --> 00:45:08,000 the, the methodological disclosure of those details that 708 00:45:08,000 --> 00:45:11,720 we were talking about, you know, it's scientists talking to other 709 00:45:11,720 --> 00:45:14,800 scientists. Nobody else understands or cares 710 00:45:14,800 --> 00:45:18,440 or wants to read all that, that nerd speak or that Godly go. 711 00:45:20,360 --> 00:45:22,680 So we were wondering, what are we going about this wrong? 712 00:45:22,680 --> 00:45:26,680 We're not really reaching the audience that, you know, has 713 00:45:26,680 --> 00:45:30,360 declining trust in, in, in public opinion research and 714 00:45:30,360 --> 00:45:33,640 science in general. So what can we do about that? 715 00:45:33,960 --> 00:45:37,640 We took advantage of an opportunity and, and, and the 716 00:45:37,640 --> 00:45:40,200 people at the University of Southern California's 717 00:45:40,200 --> 00:45:43,640 Understanding America study, what we're kind enough to give 718 00:45:43,640 --> 00:45:48,480 us some time on, on one of their nationwide panel survey 719 00:45:48,480 --> 00:45:54,400 questionnaires to, to conduct an experiment where we, we took a 720 00:45:54,400 --> 00:45:59,720 fairly neutral study, news media report, I think it was about 721 00:45:59,720 --> 00:46:02,080 weight loss drugs. And then the latest findings 722 00:46:02,080 --> 00:46:06,560 from a public opinion poll about the use of, of, of weight loss 723 00:46:06,560 --> 00:46:09,800 drugs. And we developed three different 724 00:46:10,000 --> 00:46:14,360 types of mythological disclosures to, to, you know, 725 00:46:14,520 --> 00:46:18,360 for that news story and then randomly assigned each 726 00:46:18,560 --> 00:46:21,680 respondent to one of these three disclosures. 1 is what we call 727 00:46:21,680 --> 00:46:24,280 the minimal disclosure. And it was about two sentences. 728 00:46:24,800 --> 00:46:29,800 The survey was done in this month and it was a sample of X 729 00:46:29,800 --> 00:46:34,640 number of Americans, very basic minimal disclosure. 730 00:46:35,680 --> 00:46:38,960 The second condition was what we call the informational 731 00:46:38,960 --> 00:46:42,040 disclosure, which is basically what the Transparency Initiative 732 00:46:42,200 --> 00:46:44,640 requires. There's a much, much longer list 733 00:46:44,960 --> 00:46:48,760 of, of data elements that need to be disclosed to conform with 734 00:46:48,760 --> 00:46:52,240 the the Transparency Initiative's requirements. 735 00:46:53,000 --> 00:46:55,440 And then there's something that Peter developed called the 736 00:46:55,440 --> 00:46:59,240 explanatory disclosure. And rather than just throwing 737 00:46:59,320 --> 00:47:03,720 all of these numbers at you, the response rate was this, well, 738 00:47:03,720 --> 00:47:07,280 what does that mean? If if you're you're that talking 739 00:47:07,280 --> 00:47:09,240 to a scientist, what exactly does that mean? 740 00:47:09,800 --> 00:47:14,920 What, what Peter did, he very carefully crafted an explanatory 741 00:47:14,920 --> 00:47:18,360 description of all of this data so that people would understand 742 00:47:18,680 --> 00:47:22,080 why decision, why methodological decisions were being made and 743 00:47:22,080 --> 00:47:24,920 and what how this information should be interpreted. 744 00:47:25,840 --> 00:47:28,720 It was a much longer piece, as you, you might imagine. 745 00:47:29,160 --> 00:47:34,120 So the experiment ranged from 2 sentences to kind of, it's kind 746 00:47:34,120 --> 00:47:37,720 of like Goldilocks with three different levels of, of, of 747 00:47:37,880 --> 00:47:42,040 obial. But we, we randomly assigned a 748 00:47:42,040 --> 00:47:45,840 large sample. We had 3000 respondents and they 749 00:47:45,840 --> 00:47:49,480 each received one of these, these disclosure texts, the 750 00:47:49,480 --> 00:47:52,080 minimal to the maximal, the explanatory. 751 00:47:53,160 --> 00:47:57,840 And then we, we measured their, we had this very nice scale 4 752 00:47:57,840 --> 00:48:00,760 out of scale that had great reliability and, and construct 753 00:48:00,760 --> 00:48:04,080 validity that measured their perceptions of the credibility 754 00:48:04,080 --> 00:48:07,600 of the pole that they were, you know, that was the focus of the 755 00:48:07,720 --> 00:48:12,760 of, of the experiment. And we did not find tremendous 756 00:48:12,760 --> 00:48:16,360 differences in terms of the effects of this methodological 757 00:48:16,360 --> 00:48:21,480 disclosure on respondents. We found a little bit that that 758 00:48:21,680 --> 00:48:24,840 the explanatory text, despite the fact that it was a much 759 00:48:24,840 --> 00:48:29,680 longer block of information written probably at a slightly 760 00:48:29,680 --> 00:48:36,200 higher reading level, nonetheless elicited higher 761 00:48:36,400 --> 00:48:40,080 perceptions of of credibility relative to the minimal. 762 00:48:40,960 --> 00:48:45,320 We interviewed these many people On this date and we, we had a 763 00:48:45,320 --> 00:48:48,600 series of, we looked at a series of, of potential moderators of 764 00:48:48,600 --> 00:48:52,240 the, of that relationship and, and, and found virtually no 765 00:48:52,240 --> 00:48:55,960 effects there. And we also learned that not a 766 00:48:55,960 --> 00:49:01,760 lot of people know about the transparency initiative or it 767 00:49:02,720 --> 00:49:05,000 just just mentioned the transparency initiative as a 768 00:49:05,000 --> 00:49:07,760 Good Housekeeping seal has no effect on, on people. 769 00:49:07,760 --> 00:49:10,840 So that, that that's the word and, and the final piece of of 770 00:49:10,840 --> 00:49:16,480 interesting information was we randomized the presentation of 771 00:49:16,480 --> 00:49:19,880 this material such that half we just embedded it the 772 00:49:19,880 --> 00:49:21,640 methodological details in the study. 773 00:49:22,000 --> 00:49:25,320 For the other half, we said For more information about how this 774 00:49:25,320 --> 00:49:28,040 study was done, click here on this hyperlink. 775 00:49:28,400 --> 00:49:33,400 And so they, they had to actually go out of their way to 776 00:49:33,760 --> 00:49:38,760 to access the information. And what we found was just under 777 00:49:38,760 --> 00:49:42,000 20%, about one in five respondents had any interest in 778 00:49:42,000 --> 00:49:45,920 even going to look at at the methodological information that 779 00:49:45,920 --> 00:49:47,880 we got. That tells me we've got a long 780 00:49:47,880 --> 00:49:50,760 way to go. Yeah, yeah. 781 00:49:50,760 --> 00:49:52,160 Sorry. That was a long winded no, no, 782 00:49:52,160 --> 00:49:54,440 no but. I think it's, it's, it's good 783 00:49:54,440 --> 00:49:57,560 because I, I, yeah, I found it really interesting as well. 784 00:49:57,560 --> 00:50:00,600 And, and also the challenge here. 785 00:50:00,600 --> 00:50:03,560 But of course, there's so many more challenges because most of 786 00:50:03,560 --> 00:50:07,800 this polls or, or research result is presented in, in news 787 00:50:07,800 --> 00:50:12,320 media and, and, and the trust in media is, will influence the 788 00:50:12,320 --> 00:50:14,560 trust in the poll if you're unlucky as well. 789 00:50:14,960 --> 00:50:17,680 So there's so many layers. But I, I thought it was really 790 00:50:17,680 --> 00:50:22,320 interesting to just look at this because at the same time, it is 791 00:50:22,320 --> 00:50:26,520 good to have some sort of disclosure of what's what, how, 792 00:50:26,520 --> 00:50:28,840 how the, the poll is constructed because. 793 00:50:29,720 --> 00:50:32,520 Yeah, just going back to the probability based, non 794 00:50:32,520 --> 00:50:34,200 probability based, that's one step. 795 00:50:34,200 --> 00:50:37,000 But also OK, there is some science behind this and we have 796 00:50:37,000 --> 00:50:42,600 a method that we we actually know it is possible to reproduce 797 00:50:42,640 --> 00:50:45,280 the method and and in that case also a result. 798 00:50:45,280 --> 00:50:49,200 So I thought it was really interesting and, and that there 799 00:50:49,200 --> 00:50:52,160 are, as you said, there's so much more to do. 800 00:50:52,160 --> 00:50:57,040 But but I, but I think we all have to figure out how to work 801 00:50:57,040 --> 00:51:00,640 more on increasing the trust in the science behind this and and 802 00:51:00,640 --> 00:51:04,360 also, of course, protecting the neutrality of the science, 803 00:51:04,360 --> 00:51:07,800 because that's not facts or facts. 804 00:51:08,120 --> 00:51:10,000 And we should treat it like that. 805 00:51:10,320 --> 00:51:13,240 Then you might not like it, but just it's still probably the 806 00:51:13,240 --> 00:51:17,080 good to know it. That's basically my my take on 807 00:51:17,080 --> 00:51:18,640 facts. I'm with you. 808 00:51:18,640 --> 00:51:20,680 We have to protect the neutrality of the facts. 809 00:51:22,120 --> 00:51:23,640 They're not. Yeah. 810 00:51:24,400 --> 00:51:25,280 And then? Partisan. 811 00:51:25,680 --> 00:51:28,240 Yeah, yeah, exactly. Because then every, every fact 812 00:51:28,240 --> 00:51:31,120 will be a partisan. That's that's an entire 813 00:51:31,120 --> 00:51:34,640 different challenge. But but yeah, we, we, we still 814 00:51:34,640 --> 00:51:37,960 have to acknowledge that there are science that actually works 815 00:51:37,960 --> 00:51:43,160 and that we know how to do it. And and that in turn creates a 816 00:51:43,160 --> 00:51:46,800 lot of vital facts from the, from in the Western world that 817 00:51:46,800 --> 00:51:50,840 we need in our daily life. It affects the economy. 818 00:51:50,840 --> 00:51:53,280 It affects everything actually what we're doing. 819 00:51:53,280 --> 00:51:57,600 And we tend to forget that that's, but I'm really glad you 820 00:51:57,600 --> 00:52:00,480 took the time because I realized we have run out of time right 821 00:52:00,480 --> 00:52:00,880 now. Oh. 822 00:52:00,880 --> 00:52:01,440 Have we? Yeah. 823 00:52:01,600 --> 00:52:04,480 You know, it's been a pleasure. It's been a pleasure talking 824 00:52:04,480 --> 00:52:05,320 with you. Yeah. 825 00:52:05,320 --> 00:52:07,440 These are important problems that aren't going away. 826 00:52:07,440 --> 00:52:10,240 We'll have to continue our dialogue on these topics. 827 00:52:10,560 --> 00:52:13,560 I totally agree. I'm so glad it took the time and 828 00:52:14,200 --> 00:52:17,360 now my day is ending. I think your day is starting, 829 00:52:17,360 --> 00:52:20,880 but it's amazing that we can talk across the world like this. 830 00:52:21,320 --> 00:52:23,160 Thank you. So much a great invention, yeah. 831 00:52:23,280 --> 00:52:26,080 It is. I hope you have a great day. 832 00:52:26,080 --> 00:52:27,120 Thank you. Thanks so much.