1 00:00:00,200 --> 00:00:06,900 Today, on episode number 787 of cxo, talk, we're discussing data 2 00:00:06,900 --> 00:00:10,600 and intuition. The role in marketing, we call 3 00:00:10,600 --> 00:00:15,100 that quantitative marketing. Our guests are two folks from 4 00:00:15,100 --> 00:00:18,900 the Columbia business school at Columbia University in New York 5 00:00:18,900 --> 00:00:22,300 City ODed. Netzer is the vice Dean for 6 00:00:22,300 --> 00:00:26,500 research and a professor and author of the new book decisions 7 00:00:26,500 --> 00:00:31,500 over decimals and Amy Jake is the chief marketing officer of 8 00:00:31,500 --> 00:00:34,000 the institution for me marketing. 9 00:00:34,000 --> 00:00:38,200 Is that part of any organization that is on charge of 10 00:00:38,700 --> 00:00:41,600 facilitating and enabling successful transaction with the 11 00:00:41,600 --> 00:00:43,800 customer successful? Whether it could mean profit if 12 00:00:43,800 --> 00:00:46,800 it's for profit could be engagement for example if it's 13 00:00:46,800 --> 00:00:50,800 non profit but the ones who are in charge of making sure that 14 00:00:50,800 --> 00:00:56,000 such transactions relationships do exist and and now you want to 15 00:00:56,000 --> 00:00:58,700 layer on it. The quantitative part of it in 16 00:00:58,700 --> 00:01:01,700 order to create this. Successful interactions. 17 00:01:02,100 --> 00:01:05,800 I really need to understand if under firm who's on the other 18 00:01:05,800 --> 00:01:07,800 side and it's not exactly the other side will have my 19 00:01:07,800 --> 00:01:11,600 partner's, my customers. And for that, I need a lot of 20 00:01:11,600 --> 00:01:15,000 data, a lot of information about my customers and who they are 21 00:01:15,200 --> 00:01:19,000 and good understanding of who I am, what am I capabilities and 22 00:01:19,000 --> 00:01:23,000 that again requires data and analytics. 23 00:01:23,700 --> 00:01:26,700 And specifically the analytics is often done in order to try 24 00:01:26,700 --> 00:01:29,100 and understand this match the met with the match between the 25 00:01:29,100 --> 00:01:32,200 customer. And and they the firm and what 26 00:01:32,200 --> 00:01:34,600 the firm has to offer and even changing what the firm has to 27 00:01:34,600 --> 00:01:37,900 offer to make sure it matches. What what customers want? 28 00:01:39,100 --> 00:01:42,900 Particularly in, in, in, in recent years where a lot of the 29 00:01:42,900 --> 00:01:47,900 effort went over Z, he's getting better and better understanding 30 00:01:47,900 --> 00:01:50,800 of the customers because the better and the more we 31 00:01:50,800 --> 00:01:53,500 understand the customers, the better, we can actually serve 32 00:01:53,500 --> 00:01:57,800 them, what it is that they want, and that requires a lot of data, 33 00:01:57,800 --> 00:02:00,800 a lot of analytics and a lot of quantitative marketing. 34 00:02:01,200 --> 00:02:08,199 Amy, what is the impact of this data driven approach on your 35 00:02:08,199 --> 00:02:11,900 work right now? There was a period of time where 36 00:02:12,200 --> 00:02:17,300 it was somewhat difficult to explain to senior leadership the 37 00:02:17,300 --> 00:02:21,100 value of marketing. The increase in the data that we 38 00:02:21,100 --> 00:02:24,800 have, the ability to show quantitatively, and to connect 39 00:02:24,800 --> 00:02:28,500 an action that marketing is taken to a bottom line impact. 40 00:02:28,500 --> 00:02:30,800 That's what a lot of the data has done. 41 00:02:31,100 --> 00:02:35,500 It has allowed us to be better marketers, we understand a lot 42 00:02:35,500 --> 00:02:37,500 more. So dead was saying about the 43 00:02:37,500 --> 00:02:41,800 audience, we can engage in different tactics and strategies 44 00:02:41,800 --> 00:02:46,900 personalization is so important. You know, we are inundated with 45 00:02:46,900 --> 00:02:52,500 material and it is very difficult for us as humans to 46 00:02:52,500 --> 00:02:55,400 have that, much flying at us and to be able to figure out what to 47 00:02:55,400 --> 00:02:58,500 pay attention to. You know, cognitively, if 48 00:02:58,500 --> 00:03:02,300 there's this influx of information, we have to pick, we 49 00:03:02,300 --> 00:03:06,300 have to sort, we have to think about what makes sense for us to 50 00:03:06,300 --> 00:03:08,500 pay attention to what limited time do we have. 51 00:03:08,700 --> 00:03:14,200 And how We react to that and in Mass marketing sometimes you can 52 00:03:14,200 --> 00:03:17,300 get that benefit with your audience but more often what 53 00:03:17,300 --> 00:03:20,400 we're seeing is that the more personalized, that the marketing 54 00:03:20,400 --> 00:03:24,400 is, the more likely that person who is otherwise inundated will 55 00:03:24,400 --> 00:03:27,200 take a moment will stop and will engage with you. 56 00:03:27,600 --> 00:03:31,700 And the only way to do that is to be able to leverage the data 57 00:03:31,700 --> 00:03:35,100 that you have to say, this is the right course, for this 58 00:03:35,100 --> 00:03:38,100 person, this is the right path. This is the right Next Step for 59 00:03:38,100 --> 00:03:40,900 person a But for person B, it's going to be a little bit 60 00:03:40,900 --> 00:03:44,200 different and every time that our audience or consumer or 61 00:03:44,200 --> 00:03:47,800 customer has that experience, the more likely, they are to 62 00:03:47,800 --> 00:03:50,400 come back because there's a benefit for them. 63 00:03:50,400 --> 00:03:53,400 They're getting what they need. They've seen that it works. 64 00:03:53,400 --> 00:03:56,900 They know it was worth their time and so I think data both 65 00:03:56,900 --> 00:04:00,800 helps us on the strategy side. Develop better strategies. 66 00:04:00,800 --> 00:04:06,400 It helps us on showing results to senior leaders to, to 67 00:04:06,400 --> 00:04:10,400 shareholders to stakeholders and it helps Us to deliver a better 68 00:04:10,400 --> 00:04:14,300 experience to the customer. So they're all wrapped up in 69 00:04:14,300 --> 00:04:17,200 sort of one larger marketing effort. 70 00:04:17,500 --> 00:04:19,800 But I think those three components and thinking about 71 00:04:19,800 --> 00:04:21,899 it, in those ways is really helpful. 72 00:04:22,100 --> 00:04:25,200 I've been teaching the marketing core for quite a few years here 73 00:04:25,500 --> 00:04:27,200 at Columbia business. School is probably the course 74 00:04:27,200 --> 00:04:31,400 that has changed the most actually em out of any course we 75 00:04:31,400 --> 00:04:33,300 teach. And the reason is that the world 76 00:04:33,300 --> 00:04:35,800 of marketing is changing tremendously. 77 00:04:35,800 --> 00:04:39,100 And if you think for example about advertising, The 78 00:04:39,300 --> 00:04:42,400 advertising. Again, many of us have watched 79 00:04:42,400 --> 00:04:45,000 madman, right? And that's that's really the old 80 00:04:45,000 --> 00:04:48,600 days of marketing. But even up to the early 2000's, 81 00:04:48,600 --> 00:04:52,400 the center of advertising was here in New York was in Madison 82 00:04:52,400 --> 00:04:54,600 Avenue. If you think about what are the 83 00:04:54,600 --> 00:04:58,500 three biggest advertising companies in the world then not 84 00:04:58,500 --> 00:05:01,700 anymore in New York and there are not any more advertising 85 00:05:01,700 --> 00:05:06,000 company companies, there are Google Facebook and I'm an 86 00:05:06,000 --> 00:05:08,600 Amazon. These are these companies in 87 00:05:08,600 --> 00:05:12,600 Silicon Valley and they sit in, in, in Seattle, which means 88 00:05:12,600 --> 00:05:15,300 there are technology companies, and with that comes the 89 00:05:15,300 --> 00:05:17,700 quantitative aspect of marketing. 90 00:05:18,100 --> 00:05:23,300 Amy made a very interesting point, she spoke about the 91 00:05:23,600 --> 00:05:29,200 enhanced ability to connect the marketing activities, back to 92 00:05:29,200 --> 00:05:33,000 impact on the organization. Can you talk about that? 93 00:05:33,000 --> 00:05:35,700 Because that's what Business Leaders ultimately care about, 94 00:05:35,700 --> 00:05:37,000 right? We're spending. 95 00:05:37,200 --> 00:05:39,800 A certain amount of money on our marketing and what is the 96 00:05:39,800 --> 00:05:41,800 ultimate impact? And how do we measure that? 97 00:05:42,100 --> 00:05:46,000 The one of the biggest impact of quantitative marketing on their 98 00:05:46,000 --> 00:05:48,800 world of marketing? Was this notion of our eye and 99 00:05:48,800 --> 00:05:50,400 there is return on investment, right? 100 00:05:50,400 --> 00:05:54,300 I mean, will the famous quote and, you know, half of my 101 00:05:54,300 --> 00:05:56,300 marketing expenses are? Well, spent, I just don't know 102 00:05:56,300 --> 00:05:59,600 which half and then came the world of online marketing, 103 00:05:59,600 --> 00:06:01,000 right? Then came the world of digital 104 00:06:01,000 --> 00:06:04,000 marketing and with this promise of we're going to see what 105 00:06:04,000 --> 00:06:05,900 people clicked on. We actually going to see whether 106 00:06:06,100 --> 00:06:09,500 someone clicked on the ad where the summit eventually even 107 00:06:09,500 --> 00:06:14,100 bought some finally, we can get our iron marketing, we know what 108 00:06:14,100 --> 00:06:17,100 is the impact on of a marketing on consumers? 109 00:06:18,100 --> 00:06:21,300 I think it's true. It we have moved a long long way 110 00:06:21,300 --> 00:06:24,700 in the ability to show return on investment on marketing. 111 00:06:25,200 --> 00:06:28,200 And we, it is much easier now to have that conversation. 112 00:06:28,200 --> 00:06:32,200 And I think that marketeers that haven't taken that step have 113 00:06:32,200 --> 00:06:34,800 truly left behind. And if still struggle with are 114 00:06:34,900 --> 00:06:37,000 within their organizations, I will. 115 00:06:37,100 --> 00:06:41,500 So the mention that that promise may have been even too strong. 116 00:06:41,500 --> 00:06:44,700 It, we now started seeing the pendulum shifting on that 117 00:06:44,700 --> 00:06:47,800 statement in the sense that the fact that I see whether customer 118 00:06:47,800 --> 00:06:51,400 clicked on an ad doesn't yet fully tell me whether it's a 119 00:06:51,400 --> 00:06:53,000 full return on investment, right? 120 00:06:53,000 --> 00:06:55,600 It could have been that this customer would have bought 121 00:06:55,600 --> 00:06:59,600 anyway, so we need some sometimes more sophisticated 122 00:06:59,600 --> 00:07:02,900 quantitative techniques in order to truly ask questions, like 123 00:07:02,900 --> 00:07:05,100 attributional in current through a mentality. 124 00:07:05,500 --> 00:07:08,000 And whether we truly have Roi on these. 125 00:07:08,000 --> 00:07:11,900 It seemed as if we solve the problem and now we realize the 126 00:07:11,900 --> 00:07:15,000 problem is, we have way way ahead, but there are still 127 00:07:15,200 --> 00:07:17,200 things to be done, which is where the world of quantitative 128 00:07:17,200 --> 00:07:21,000 marketing works on today. A few years ago, a marketer had 129 00:07:21,000 --> 00:07:24,100 to say I see a correlation but I may not be able to show 130 00:07:24,100 --> 00:07:26,700 causation and now I think you're right there. 131 00:07:26,700 --> 00:07:31,200 Are there still are numerous questions that we have there's 132 00:07:31,200 --> 00:07:34,600 still challenges with attribution models but I think 133 00:07:34,600 --> 00:07:38,400 we are much closer to being able to say It's not just happening 134 00:07:38,400 --> 00:07:40,500 at the same time in, isn't that nice? 135 00:07:40,600 --> 00:07:44,200 We're able to at least connect often some of those dots and 136 00:07:44,200 --> 00:07:49,300 say, well, I know that when my marketing spend goes up 13% 27 137 00:07:49,300 --> 00:07:55,200 percent of customers, you know, purchase more or 13, you know my 138 00:07:55,200 --> 00:07:58,500 marketing spend goes up 13 percent and 50 percent more 139 00:07:58,500 --> 00:08:01,000 individuals enter this online store. 140 00:08:01,300 --> 00:08:05,400 So we're able to do it in a way that you know, if our 141 00:08:05,400 --> 00:08:08,500 attribution models are right and we Have the right technology at 142 00:08:08,500 --> 00:08:11,200 some point in the path we can say there was a direct 143 00:08:11,200 --> 00:08:14,600 connection and we know that it's actually going to make a 144 00:08:14,608 --> 00:08:16,900 difference. And I think directionality is 145 00:08:16,900 --> 00:08:21,300 important, but directionality with, you know, some very 146 00:08:21,300 --> 00:08:25,300 specific models that show that relationship is what ultimately 147 00:08:25,300 --> 00:08:30,500 gets you farther in terms of your budget, in terms of being 148 00:08:30,500 --> 00:08:33,799 able to enter into new markets, and really buying into sort of 149 00:08:33,799 --> 00:08:37,000 the confidence of the c-suite, I think, which is super important. 150 00:08:37,500 --> 00:08:39,799 I think one of the things that truly help with that, and 151 00:08:39,799 --> 00:08:42,000 particularly, in the world of digital marketing is the ability 152 00:08:42,000 --> 00:08:44,200 to run these A/B tests, right? Yes. 153 00:08:44,200 --> 00:08:46,800 I mean it's very difficult to say, you know, how about my 154 00:08:46,800 --> 00:08:49,900 customer going to see one store and a half of my customer going 155 00:08:49,900 --> 00:08:51,600 to see a different version of my store. 156 00:08:51,900 --> 00:08:54,100 You know, my story. My story is very difficult to 157 00:08:54,100 --> 00:08:57,400 play with it in the online World it because it became very easy. 158 00:08:57,400 --> 00:09:01,100 And you know I split my time between Colombia and Amazon, I 159 00:09:01,100 --> 00:09:05,500 spend some of my time at Amazon advertising, the practice at 160 00:09:05,500 --> 00:09:09,100 Amazon advertising is running. These A/B test these, let's try 161 00:09:09,100 --> 00:09:12,700 version a version version B, and when we do that, we truly can 162 00:09:12,700 --> 00:09:15,000 get to RI. We truly can get to. 163 00:09:15,300 --> 00:09:18,000 What is the difference between offering the customer version a 164 00:09:18,008 --> 00:09:20,400 and version B? It's still easier to doing 165 00:09:20,700 --> 00:09:24,600 digital environment where we can take, you know, we have 100,000 166 00:09:24,600 --> 00:09:27,500 visitors, we can split them half or any any way. 167 00:09:27,500 --> 00:09:31,400 We split them randomly and see, throw to to different customers 168 00:09:31,400 --> 00:09:34,500 different different offerings and see how it works. 169 00:09:34,900 --> 00:09:36,900 It's still a little bit more difficult in the offline mode. 170 00:09:37,100 --> 00:09:40,100 But even there, we are making progress toward thinking in the, 171 00:09:40,100 --> 00:09:45,600 in a language of A/B tests in the language of testing, in 172 00:09:45,600 --> 00:09:48,500 order to measure Roi. And in order to measure, truly 173 00:09:48,500 --> 00:09:51,800 what customers want? I'll give you just an example of 174 00:09:51,800 --> 00:09:54,700 that. Several years ago, I was working 175 00:09:54,700 --> 00:09:59,400 at a large company and my role and my team's role was focused 176 00:09:59,400 --> 00:10:02,300 around telling investors the story of the success of the 177 00:10:02,300 --> 00:10:05,200 business. And when we had our hypothesis 178 00:10:05,200 --> 00:10:08,500 about which creative would work, The best to get them to engage 179 00:10:08,500 --> 00:10:13,100 with our financial results. I would have thought one thing 180 00:10:13,100 --> 00:10:15,800 and we a be tested over a period of about a year. 181 00:10:15,800 --> 00:10:19,000 So that's four, four quarters worth of data and we actually 182 00:10:19,000 --> 00:10:23,200 found that in telling the story with our IP, so the show's 183 00:10:23,200 --> 00:10:27,300 themselves when we use that as a creative investors, now, this 184 00:10:27,300 --> 00:10:30,800 isn't an audience that you would naturally think would pay more 185 00:10:30,800 --> 00:10:34,500 attention to, you know, a character on a show than a 186 00:10:34,508 --> 00:10:38,700 financial times headline. But when we Talked about our 187 00:10:38,700 --> 00:10:41,100 efforts. And when we use the creative, 188 00:10:41,100 --> 00:10:44,900 when we use our IP, we found that the interest in the 189 00:10:44,900 --> 00:10:49,600 engagement, in the information, the financial information about 190 00:10:49,600 --> 00:10:53,200 the company increased, you know, exponentially. 191 00:10:53,500 --> 00:10:56,800 And so I think to your point that's an a/b test that 10 years 192 00:10:56,800 --> 00:10:59,300 ago or 20 years ago, one, you wouldn't have been able to run 193 00:10:59,500 --> 00:11:02,400 but to you wouldn't have hypothesized that. 194 00:11:02,500 --> 00:11:05,700 But when we just have the ability to continue to test and 195 00:11:05,700 --> 00:11:09,000 swap in and swap out, Out and we see, very clear results in the 196 00:11:09,000 --> 00:11:12,300 data, you know, it gives us confidence to both try new 197 00:11:12,300 --> 00:11:15,900 things, but then it gives us Security in the decisions that 198 00:11:15,908 --> 00:11:18,500 we make. Because we know now that we can 199 00:11:18,500 --> 00:11:23,400 see that that happening, please subscribe to our YouTube channel 200 00:11:23,400 --> 00:11:26,900 and hit the Subscribe button at the bottom of our website. 201 00:11:26,900 --> 00:11:30,000 So you can get our newsletter and we can keep you up-to-date 202 00:11:30,000 --> 00:11:34,700 on upcoming live shows. What about the data? 203 00:11:34,700 --> 00:11:40,000 What kinds of. Of data, should marketers be 204 00:11:40,300 --> 00:11:43,600 collecting? What's the best way to get that 205 00:11:43,600 --> 00:11:46,700 data and ODed? Can you take us a little bit 206 00:11:46,700 --> 00:11:51,700 behind the scenes of the kinds of models that you're looking 207 00:11:51,700 --> 00:11:55,200 at? And again, my real interest here 208 00:11:55,300 --> 00:12:02,200 is the impact of that data. And those models on the real 209 00:12:02,200 --> 00:12:05,700 lives of marketers, hey, it sounds like a reality TV show, 210 00:12:05,700 --> 00:12:07,300 right? The real lives of ders. 211 00:12:08,100 --> 00:12:09,800 We have a fire hose of data, right? 212 00:12:09,800 --> 00:12:11,900 We have more data than we ever had before. 213 00:12:12,500 --> 00:12:15,300 It's still the situation that we don't always have the data. 214 00:12:15,300 --> 00:12:17,800 We exactly need. We always sometimes, looking for 215 00:12:17,800 --> 00:12:20,900 the right data, we need there is this something we talked about 216 00:12:20,900 --> 00:12:23,400 in the book that you mentioned earlier, then we call the the 217 00:12:23,400 --> 00:12:25,900 certain teammate. There was this belief that 218 00:12:25,900 --> 00:12:28,000 finally when we have all of these data will get to the 219 00:12:28,000 --> 00:12:31,400 certain decisions. Certainties I me that we still 220 00:12:31,400 --> 00:12:34,300 make decisions under uncertainty, but we do have much 221 00:12:34,300 --> 00:12:38,700 more data than we ever had. And Specifically, the data that 222 00:12:38,700 --> 00:12:44,500 marketeers value and cherish tend to be data that tells us 223 00:12:44,500 --> 00:12:47,600 about customer preferences, being able to measure and 224 00:12:47,600 --> 00:12:51,700 understand customer preferences and the reality of Eden. 225 00:12:51,700 --> 00:12:54,400 And the reason why it was difficult, it may be still is 226 00:12:54,400 --> 00:12:58,800 difficult in some ways is because of heterogeneity, 227 00:12:59,100 --> 00:13:02,100 because customers are so different than 0 to customers 228 00:13:02,100 --> 00:13:04,800 that are the same. And, you know, I always tell my 229 00:13:04,800 --> 00:13:07,900 students to know they and always Find within my class. 230 00:13:07,900 --> 00:13:12,300 This type of students, those who treat the Apple Store, like a 231 00:13:12,400 --> 00:13:14,600 like a called. And those who would never be 232 00:13:14,600 --> 00:13:16,300 caught dead in an Apple store, right? 233 00:13:16,300 --> 00:13:19,000 And the marketeers needs to understand this and he's trying 234 00:13:19,000 --> 00:13:22,600 to stand this often with limited data, depending who you are, if 235 00:13:22,600 --> 00:13:25,300 your Google or Amazon, you have a lot of data. 236 00:13:25,300 --> 00:13:29,100 If you just, if you are media Outlet, you may have less data 237 00:13:29,100 --> 00:13:33,100 about about the customer and who they are and we tend to see 238 00:13:33,100 --> 00:13:35,000 unlike. If you go to data for example, 239 00:13:35,000 --> 00:13:39,800 in finance, I can always Go one more year and have longer data 240 00:13:39,800 --> 00:13:43,000 about this about stocks, right? I can also just do an analysis 241 00:13:43,000 --> 00:13:48,900 of the entire Market when it comes to stocks, when it comes 242 00:13:48,900 --> 00:13:51,300 to to Consumers. A I cannot do data just on the 243 00:13:51,300 --> 00:13:54,500 entire Market because of the heterogeneity that I'm talking 244 00:13:54,500 --> 00:13:56,400 about. I do need to understand each 245 00:13:56,400 --> 00:13:59,100 customer in their own preferences and I'm limited with 246 00:13:59,100 --> 00:14:02,900 respect to the history because if I'm talking about a travel, 247 00:14:02,900 --> 00:14:07,600 for example, if I'm Expedia one of these dry only see EU, you 248 00:14:07,600 --> 00:14:09,600 know, five ten times, that's all I have. 249 00:14:09,700 --> 00:14:12,200 So I need to work with that length of history of what I 250 00:14:12,200 --> 00:14:17,200 observed about the customer, but and that's where economics helps 251 00:14:17,200 --> 00:14:19,800 so. Or statistics, there is a 252 00:14:19,808 --> 00:14:25,200 trade-off when it comes to data between how good your data is 253 00:14:25,200 --> 00:14:29,700 and how complicated your model needs to be the better data, you 254 00:14:29,700 --> 00:14:32,200 have the simpler, the model you can use. 255 00:14:32,200 --> 00:14:35,000 If we run the A/B test of a just talked about before. 256 00:14:35,500 --> 00:14:36,900 All I need to do is compare average. 257 00:14:37,100 --> 00:14:40,300 Is a customer in the control but that much customer in treatment 258 00:14:40,300 --> 00:14:44,400 but that much, that's it. I mean, level of math, that that 259 00:14:44,400 --> 00:14:48,800 is, you know, a sixth grader. Could do if I need now to build 260 00:14:48,800 --> 00:14:52,700 a model where I am again a Expedia and I'm trying to 261 00:14:52,708 --> 00:14:56,300 understand customer preferences from this visit and the previous 262 00:14:56,300 --> 00:15:00,200 visits and so on, that's where I need a much more sophisticated 263 00:15:00,200 --> 00:15:03,500 type of modeling and type of analysis. 264 00:15:04,000 --> 00:15:06,800 The other, maybe distinction I want to make about data. 265 00:15:07,200 --> 00:15:09,500 Is it extinct distinction between structure data and 266 00:15:09,500 --> 00:15:16,300 unstructured data until fairly recently, Circa 2010, most of 267 00:15:16,300 --> 00:15:20,000 our analysis was done on what is called structure data structure. 268 00:15:20,000 --> 00:15:23,600 Data are numbers data that comes in the form that we've by the 269 00:15:23,600 --> 00:15:27,700 way, generally think about data in Excel, or in a table with 270 00:15:27,700 --> 00:15:30,300 numbers in it. But if you think about it, 271 00:15:30,300 --> 00:15:34,600 majority of the data we have as business people, as marketeers 272 00:15:35,200 --> 00:15:40,200 is actually unstructured data, it can Ones in text image, 273 00:15:40,200 --> 00:15:43,200 video, Audio customer calls us to the call center. 274 00:15:43,500 --> 00:15:47,600 And we have data in terms of tens of thousands, if not 275 00:15:47,600 --> 00:15:50,700 hundreds of thousands of ads that we can, we can analyze ads 276 00:15:50,700 --> 00:15:56,700 are images and text, we have company reports, different 277 00:15:56,700 --> 00:15:59,700 estimates, depends who you ask, but 80 to 95 percent of data 278 00:15:59,700 --> 00:16:02,200 available. For business is likely in the 279 00:16:02,200 --> 00:16:06,800 form of unstructured data in text audio video and image. 280 00:16:07,000 --> 00:16:12,000 And it is only since 2010 that we actually know how to analyze 281 00:16:12,000 --> 00:16:14,900 this data at scale. I mean I'm structure that 282 00:16:14,900 --> 00:16:17,400 existed probably since the Ten Commandments right. 283 00:16:17,400 --> 00:16:21,200 We're waiting on a stone but but really they build it or to 284 00:16:21,200 --> 00:16:24,600 analyze this data at scale, came with machine learning type 285 00:16:24,700 --> 00:16:29,700 machine, learning methods, type tools and and we seeing more and 286 00:16:29,700 --> 00:16:32,600 more companies are using it. And, of course, recently, the 287 00:16:32,600 --> 00:16:37,200 whole chechi PT and generative AI is an example of Jing, 288 00:16:37,200 --> 00:16:40,300 unstructured data, Amy your marketer. 289 00:16:40,300 --> 00:16:48,600 I'm assuming that you are not and expert statistician and data 290 00:16:48,600 --> 00:16:53,900 scientist. And so given what ODed was just 291 00:16:53,900 --> 00:16:59,400 describing in terms of the quantity of data, the quality of 292 00:16:59,400 --> 00:17:04,400 data, what can marketers do to take advantage of these 293 00:17:04,400 --> 00:17:10,000 important points? But without Without having to be 294 00:17:10,000 --> 00:17:13,800 a deep level statistician. For example, don't be scared of 295 00:17:13,800 --> 00:17:16,400 data. I'm not a statistician as you 296 00:17:16,400 --> 00:17:19,300 very correctly, pointed out, I never have been. 297 00:17:19,900 --> 00:17:23,599 But I think for marketers, when we talk about quantitative 298 00:17:23,599 --> 00:17:27,200 marketing or we talked about data-driven marketing, for some 299 00:17:27,200 --> 00:17:31,400 especially individuals early on in their career, it sounds like 300 00:17:31,400 --> 00:17:34,000 they're going to have to be gathering the data sets. 301 00:17:34,000 --> 00:17:36,800 It sounds like they're going to have to be building the model. 302 00:17:37,000 --> 00:17:39,100 It sounds like they're going to have to be interpreting. 303 00:17:39,600 --> 00:17:43,600 And I think we're very lucky in that, we have a whole generation 304 00:17:43,600 --> 00:17:48,000 of individuals who understand data, who are data scientists, 305 00:17:48,000 --> 00:17:51,500 who are insights and research-driven colleagues. 306 00:17:51,500 --> 00:17:55,100 And the more that we can develop very close relationships with 307 00:17:55,100 --> 00:17:57,900 those individuals as marketers the better. 308 00:17:57,900 --> 00:18:01,300 Because most marketers, I know aren't as you said 309 00:18:01,300 --> 00:18:04,400 statisticians. But what they do know is how to 310 00:18:04,400 --> 00:18:06,400 look at data and I know we're going to talk a little bit about 311 00:18:06,400 --> 00:18:09,800 this and and say, well, based on my understanding of the 312 00:18:09,800 --> 00:18:14,100 audience, this seems odd, let's explore that or they have a 313 00:18:14,100 --> 00:18:16,500 question that they know they want to answer. 314 00:18:16,700 --> 00:18:20,100 And they know that the data is an important part of that and so 315 00:18:20,100 --> 00:18:22,400 they can ask for help in building the model. 316 00:18:22,400 --> 00:18:25,800 So the first is, you know, you don't have to be the expert. 317 00:18:25,900 --> 00:18:28,500 Don't be scared, don't not use it. 318 00:18:28,600 --> 00:18:31,500 Use it, find a partner, find a colleague, you know, find 319 00:18:31,500 --> 00:18:35,900 somebody who is the expert and the second is think about the 320 00:18:35,900 --> 00:18:41,100 data in Meaningful way. So you as a dad said, there's 321 00:18:41,600 --> 00:18:46,400 infinite data available so it really is about what are you 322 00:18:46,400 --> 00:18:48,400 looking for? And then when something and I 323 00:18:48,408 --> 00:18:50,700 know Dad's going to talk a little bit about this, when 324 00:18:50,700 --> 00:18:54,300 something doesn't feel right, ask more questions and a good 325 00:18:54,300 --> 00:19:00,900 example of using kind of numbers and qualitative elements 326 00:19:01,800 --> 00:19:05,100 happened a couple of years ago. I worked in a publishing company 327 00:19:05,400 --> 00:19:09,300 and it's very common for people to unsubscribe from 328 00:19:09,400 --> 00:19:12,400 subscription-based businesses, including magazines. 329 00:19:12,800 --> 00:19:16,500 And so, you build a model and, you know, that churn will be X 330 00:19:16,500 --> 00:19:20,000 based on historical data. And you know, that there might 331 00:19:20,000 --> 00:19:23,500 be some macro or micro conditions that will impact that 332 00:19:23,500 --> 00:19:29,500 Year's churn and you go out and you acquire customers, knowing 333 00:19:29,500 --> 00:19:33,000 this percent is going to leave and you need to have, you know, 334 00:19:33,000 --> 00:19:36,100 X percent of new people coming in and subscribe. 335 00:19:37,000 --> 00:19:42,600 But as a marketer it's not just about making sure that you can, 336 00:19:42,700 --> 00:19:45,100 you know, fix the revenue equation on both sides. 337 00:19:45,300 --> 00:19:47,400 It's also about understanding your customers. 338 00:19:47,400 --> 00:19:51,000 So you have this model we have this model, we know that X 339 00:19:51,000 --> 00:19:53,100 percent, you know, are unsubscribing. 340 00:19:53,500 --> 00:19:56,200 And so then you have to dig deeper to think about why what 341 00:19:56,200 --> 00:19:58,800 is it? That what's the reason that you 342 00:19:58,800 --> 00:20:01,800 know, they've chosen to cancel this subscription and we could 343 00:20:01,800 --> 00:20:06,700 get a little bit of insight in asking structured questions and 344 00:20:06,700 --> 00:20:10,100 our Hypothesis was it was a value situation. 345 00:20:10,300 --> 00:20:13,600 Somebody decided that that product that the magazine was no 346 00:20:13,600 --> 00:20:16,400 longer worth what they were. Paying feels pretty simple, 347 00:20:16,400 --> 00:20:18,700 right? But we actually had done some 348 00:20:18,700 --> 00:20:22,200 brand research and had some qualitative components and what 349 00:20:22,200 --> 00:20:26,800 the qualitative components told us was that actually had nothing 350 00:20:26,800 --> 00:20:30,500 to do with value, it had nothing to do with the product, it was 351 00:20:30,500 --> 00:20:35,200 about unfinished ability. So people felt that this 352 00:20:35,200 --> 00:20:40,400 particular magazine Worth the price worth, you know, worth had 353 00:20:40,400 --> 00:20:43,800 great value, but they couldn't finish it, and they would stack 354 00:20:43,800 --> 00:20:46,000 it up in a pile. And they would say, I'll come 355 00:20:46,000 --> 00:20:48,800 back to it, and they didn't. And every time they looked at 356 00:20:48,800 --> 00:20:50,900 the pile, it crew and their guilt. 357 00:20:51,000 --> 00:20:52,500 So this is an emotional reaction. 358 00:20:52,600 --> 00:20:56,700 Their guilt over not finishing, the magazine was actually, what 359 00:20:56,700 --> 00:21:02,200 was contributing to canceling subscriptions, so that Insight 360 00:21:02,200 --> 00:21:04,500 is not in sight. You were ever going to get from 361 00:21:04,500 --> 00:21:06,700 structured data. It's not even in sight, you were 362 00:21:06,900 --> 00:21:09,500 to get from some of your survey questions. 363 00:21:09,500 --> 00:21:13,000 It came exclusive these through some qualitative conversations 364 00:21:13,300 --> 00:21:17,100 and there were some solutions to that digital. 365 00:21:18,100 --> 00:21:20,900 Combated the problem a little bit because people can't finish 366 00:21:20,900 --> 00:21:23,400 the internet. So if you can't finish a digital 367 00:21:23,400 --> 00:21:25,500 issue of a magazine, you didn't feel as guilty. 368 00:21:26,100 --> 00:21:29,500 You know, you can introduce newsletters and quick quicker, B 369 00:21:29,500 --> 00:21:31,800 of content and summaries and things like that. 370 00:21:32,600 --> 00:21:36,700 But I think the other important thing is I and I want to really 371 00:21:36,900 --> 00:21:38,400 Reiterate. This is that there's the 372 00:21:38,400 --> 00:21:40,400 structured data. There's everything, you know, 373 00:21:40,600 --> 00:21:44,900 there's quantitative and then there's this other qualitative 374 00:21:44,900 --> 00:21:47,000 element. That is equally as important for 375 00:21:47,000 --> 00:21:50,100 marketers to pair together and I think it is important that will 376 00:21:50,100 --> 00:21:53,900 think about data in the most General sense of the word, a 377 00:21:53,900 --> 00:21:57,700 Tamiya conversation with a person is data or just maybe 378 00:21:57,700 --> 00:22:01,100 want to touch on one thing with respect to the, you know, when I 379 00:22:01,108 --> 00:22:05,600 build the end model statistical model and I truly mean it. 380 00:22:05,600 --> 00:22:09,200 I would I would actually Really go to a me, first to tell me 381 00:22:09,200 --> 00:22:12,400 this model is right, before I would go to a statistician to 382 00:22:12,400 --> 00:22:14,800 tell me whether it, whether they think my model is, right, 383 00:22:14,800 --> 00:22:18,100 because the True Value comes from, let's say it's a model 384 00:22:18,100 --> 00:22:21,400 that predicts click click through on an ad and the model 385 00:22:21,400 --> 00:22:24,400 showed, you know, twenty percent click-through rate. 386 00:22:24,900 --> 00:22:27,800 It's the decision wouldn't know to tell me that 20% clicks. 387 00:22:27,800 --> 00:22:30,700 All right, on an ad is unheard of, unless you're Ed truly 388 00:22:30,700 --> 00:22:33,000 offered money to customers. No one. 389 00:22:33,000 --> 00:22:34,300 There's not, there's no one in five. 390 00:22:34,300 --> 00:22:35,900 Customers click and click on a man. 391 00:22:36,400 --> 00:22:37,300 I needed. Cat ears. 392 00:22:37,300 --> 00:22:39,200 I need a me to tell me. I don't know what you've done 393 00:22:39,200 --> 00:22:42,600 wrong, but I can tell you this is wrong because it's it's 394 00:22:42,600 --> 00:22:45,600 impossible. That customers. 20% 1811 in five 395 00:22:45,600 --> 00:22:47,000 customers would click on your ad. 396 00:22:47,300 --> 00:22:51,000 We have a couple of questions that have come up on Twitter, 397 00:22:51,200 --> 00:22:54,700 actually, to folks, Chris Peterson, and our Salon Khan, 398 00:22:54,700 --> 00:22:57,100 who are both regular listeners of cxo Takin. 399 00:22:57,100 --> 00:23:00,700 So thank you guys are asking independently. 400 00:23:00,700 --> 00:23:05,500 Essentially the same question and that is, where do we draw 401 00:23:05,500 --> 00:23:09,500 the line for? Privacy in terms of data-driven 402 00:23:09,700 --> 00:23:11,600 marketing, that's from Chris Peterson. 403 00:23:11,600 --> 00:23:15,900 And arsalan, Khan says how, and who should strike a balance 404 00:23:15,900 --> 00:23:21,600 between collecting data that can affect privacy and who decides, 405 00:23:21,600 --> 00:23:22,900 what's private? What's not? 406 00:23:22,900 --> 00:23:26,100 And also, then you have the issue of biases. 407 00:23:26,300 --> 00:23:29,400 So can we talk about that? And then we'll switch over to 408 00:23:29,500 --> 00:23:34,100 quantitative intuition privacy is an extremely important topic 409 00:23:34,100 --> 00:23:40,000 to think about to discuss. To realize that that companies 410 00:23:40,000 --> 00:23:45,800 have huge responsibility when we are collecting individual data. 411 00:23:47,300 --> 00:23:50,500 I think there are, there are there is the issue of data being 412 00:23:50,500 --> 00:23:54,300 shared and then there is the issue of data being used by, by 413 00:23:54,300 --> 00:23:55,900 the company. I mean despite what actually 414 00:23:55,900 --> 00:24:00,700 most people think most, at least the large companies do not sell 415 00:24:00,700 --> 00:24:02,900 data. Now, I'm not sure it is, by the 416 00:24:02,900 --> 00:24:07,500 way, being done because and because of pure Ethical issues 417 00:24:07,500 --> 00:24:11,500 of because data is just way too expensive to sail, the use of 418 00:24:11,500 --> 00:24:13,700 data by by company. By again, the thing about the 419 00:24:13,700 --> 00:24:18,300 top companies like Facebook, Google Amazon is so important 420 00:24:18,300 --> 00:24:20,600 and useful for their own purposes that actually selling. 421 00:24:20,600 --> 00:24:23,100 It would be, would not be very wise. 422 00:24:23,700 --> 00:24:26,600 So generally, at least from these companies data generally, 423 00:24:26,600 --> 00:24:30,800 do not do not flow around. It generally does tend to stay 424 00:24:30,800 --> 00:24:34,500 Within These these companies but even there they should treat it 425 00:24:34,500 --> 00:24:38,600 with the with the Ability that these data should be treated 426 00:24:38,600 --> 00:24:43,700 with and that's where recent regulation came about first 427 00:24:43,700 --> 00:24:47,500 party data where you're allowed to use your own data. 428 00:24:47,500 --> 00:24:50,200 But you're you're not allowed to actually use someone else's 429 00:24:50,200 --> 00:24:52,800 data, which by the way in, and of itself, there is an 430 00:24:52,808 --> 00:24:56,300 interesting tweak there. It's on the one hand, it sounds 431 00:24:56,300 --> 00:24:59,500 right. It's indeed protects customers. 432 00:24:59,500 --> 00:25:04,300 In terms of their data, being being in floating around but it 433 00:25:04,300 --> 00:25:08,400 also increases, maybe the Gap between the top companies that 434 00:25:08,400 --> 00:25:12,100 have really good first-party data their own data and the 435 00:25:12,108 --> 00:25:14,500 smaller companies that don't have their own data. 436 00:25:14,500 --> 00:25:16,800 And that's what that where they could have closed the Gap by 437 00:25:17,000 --> 00:25:20,000 getting data, Maybe from, from someone else. 438 00:25:20,700 --> 00:25:25,200 I do think that we shouldn't let the cat Garden milk there. 439 00:25:25,200 --> 00:25:27,100 I think we do need regulation there, right? 440 00:25:27,400 --> 00:25:30,400 And Europe is much better than the u.s. in that science. 441 00:25:30,900 --> 00:25:34,500 And we've made that mistake previously, with social media, 442 00:25:34,700 --> 00:25:37,900 where we let the the Guard themselves. 443 00:25:38,100 --> 00:25:40,100 We've made it with the privacy and I think we need to move 444 00:25:40,100 --> 00:25:43,000 there and I'm happy to see if there is generative an RA. 445 00:25:43,000 --> 00:25:45,000 I this conversation is starting already now. 446 00:25:45,000 --> 00:25:49,600 And not once this is fully fully out there and I mean, I don't 447 00:25:49,600 --> 00:25:52,500 know if you have other selves. Yeah, I mean it going to an end 448 00:25:52,500 --> 00:25:57,200 even kind of larger question is, who owns the data? 449 00:25:57,300 --> 00:26:00,300 We're now in a world where our Behavior has been tracked for a 450 00:26:00,500 --> 00:26:04,000 very long period of time, and there are reams of data, people 451 00:26:04,000 --> 00:26:06,600 probably know that I prefer baked macaroni. 452 00:26:06,800 --> 00:26:09,600 And cheese, you know, over brussels sprouts, but who 453 00:26:09,600 --> 00:26:12,900 doesn't, you know, that, that data is gone. 454 00:26:12,900 --> 00:26:16,200 It's with somebody, I hope I get an ad for macaroni and cheese. 455 00:26:16,200 --> 00:26:21,300 I will let everybody know shortly, but I think there's a 456 00:26:21,300 --> 00:26:24,900 larger question as we think about a next-generation, right. 457 00:26:24,900 --> 00:26:28,100 Let's talk about kids for instance, for whom maybe there's 458 00:26:28,100 --> 00:26:31,200 some data, there's not a lot who owns that data, who should own 459 00:26:31,200 --> 00:26:36,000 that data is my child, the holder of that data or is it the 460 00:26:36,008 --> 00:26:38,300 platform? Who's on which they're engaging, 461 00:26:38,300 --> 00:26:42,400 and there have been these really large high-level conversations 462 00:26:42,400 --> 00:26:45,900 around, you know, should I be able to own my data and I choose 463 00:26:45,900 --> 00:26:48,100 where when, and how to monetize that? 464 00:26:48,200 --> 00:26:51,900 Or is there a trade-off? I get to go and watch cat videos 465 00:26:51,900 --> 00:26:55,300 or, you know, whatever it is videos that I like and in 466 00:26:55,300 --> 00:26:59,900 exchange, I'm giving somebody an insight and I know that because 467 00:27:00,000 --> 00:27:04,200 I understand that's, you know, the way that a lot of practices 468 00:27:04,200 --> 00:27:07,100 work because I'm getting a pop-up that says There are 469 00:27:07,100 --> 00:27:09,800 cookies and I'm willingly engaging in a behavior. 470 00:27:09,800 --> 00:27:13,900 So I think there's both the kind of what is happening now. 471 00:27:13,900 --> 00:27:17,000 And then how do we think about data the value of data and 472 00:27:17,000 --> 00:27:21,100 ownership of data for a whole new generation, little bit off 473 00:27:21,300 --> 00:27:22,800 topic. But I think an important 474 00:27:22,800 --> 00:27:26,800 conversation for any of us who are, who are using data and 475 00:27:26,800 --> 00:27:30,000 whose data is being used in traded, in some way and we need 476 00:27:30,000 --> 00:27:33,100 to move too much more of an opt-in environment, right? 477 00:27:33,100 --> 00:27:37,300 We're at least, we make the decision of whether to Charity. 478 00:27:37,500 --> 00:27:40,700 Obviously, this is a crucially important topic. 479 00:27:40,700 --> 00:27:44,000 This issue of data privacy and data ownership. 480 00:27:44,200 --> 00:27:49,700 And we're not going to solve that one now, but we can get a 481 00:27:49,700 --> 00:27:52,100 better understanding of is, oh, Dave. 482 00:27:52,400 --> 00:27:56,800 Aude of your concept of quantitative intuition. 483 00:27:56,800 --> 00:28:00,900 What is that, and why does it matter what even Division? 484 00:28:00,900 --> 00:28:03,000 I think this particular is important in the world of 485 00:28:03,000 --> 00:28:05,600 marketing. Where again, there is a, the 486 00:28:05,600 --> 00:28:10,100 more If you will, they the the unstructured data that Amy 487 00:28:10,100 --> 00:28:13,100 talked about or the the conversations that we have, we 488 00:28:13,100 --> 00:28:16,600 tend to use our intuition regularly in our private lives. 489 00:28:17,100 --> 00:28:20,400 You know, we talked about surprising thing, an interesting 490 00:28:20,400 --> 00:28:22,100 thing. Sometimes we even call them 491 00:28:22,100 --> 00:28:27,200 gossip, but we are very worried of using our intuition, when it 492 00:28:27,200 --> 00:28:31,600 comes to the world of business, to our to our work life. 493 00:28:32,000 --> 00:28:35,200 And the idea here is not to just, you know, trust your gut. 494 00:28:35,300 --> 00:28:40,200 The idea here is You are approaching data, bring in your 495 00:28:40,200 --> 00:28:43,100 your judgment into it. In fact, we have to bring in 496 00:28:43,100 --> 00:28:45,400 your judgment into it. So on on its surface 497 00:28:45,400 --> 00:28:50,400 quantitative in intuition sound like an oxymoron but in I, in 498 00:28:50,400 --> 00:28:52,900 fact, together with my co-authors for the book, Chris, 499 00:28:52,900 --> 00:28:56,200 Frank and permanently, we believe that not only that, 500 00:28:56,200 --> 00:28:58,200 they're not oxymorons, particularly the level of 501 00:28:58,200 --> 00:28:59,900 leadership. It's the only way, the only way 502 00:28:59,900 --> 00:29:04,700 to make decisions is to combine the data with a good sense of of 503 00:29:04,700 --> 00:29:07,600 judgment. And when I say judgment, I mean, 504 00:29:07,700 --> 00:29:11,000 Intuition or judgment. This could come actually, in in 505 00:29:11,000 --> 00:29:16,200 at least three stages of the decision making process in the 506 00:29:16,200 --> 00:29:19,800 questions we asked in being a very clear about what problem 507 00:29:19,800 --> 00:29:23,400 we're trying to solve more often than not, we are just going on a 508 00:29:23,408 --> 00:29:25,000 straightening. Oh, we've been collecting all of 509 00:29:25,000 --> 00:29:27,200 this data, then got to be something interesting in. 510 00:29:27,200 --> 00:29:29,700 Its right, we need to guide the process. 511 00:29:29,700 --> 00:29:32,800 To me, the one of the biggest Nono in making data-driven 512 00:29:32,800 --> 00:29:35,900 decision, making is we expect the data to provide both a 513 00:29:35,900 --> 00:29:39,000 questions and There's we should ask questions then. 514 00:29:39,000 --> 00:29:41,400 Hopefully data. Can provide answers. 515 00:29:41,400 --> 00:29:45,500 In fact, we've learned it better than ever since November of 516 00:29:45,500 --> 00:29:48,800 twenty twenty with with a 2022. Sorry with GPT. 517 00:29:49,100 --> 00:29:51,600 We need to provide good proms. If you want to get good answers, 518 00:29:51,600 --> 00:29:54,700 we need to ask good questions. Second is something you already 519 00:29:54,700 --> 00:29:57,400 talked about how do we interrogate data and, and 520 00:29:57,400 --> 00:29:59,900 exactly what I mentioned that you do need the context as 521 00:29:59,900 --> 00:30:02,100 humans. We are very good in context and 522 00:30:02,100 --> 00:30:05,200 particularly, if you are an expert in your domain, as Amy is 523 00:30:05,200 --> 00:30:09,100 in, Amy is in in, in marketing, it's much easier for her to look 524 00:30:09,100 --> 00:30:13,200 at the model of very sophisticated analysis and 525 00:30:13,200 --> 00:30:15,700 interrogated. Not from the p-value of from you 526 00:30:15,700 --> 00:30:19,600 didn't use this three-letter acronyms or another, but from 527 00:30:20,200 --> 00:30:22,200 this number doesn't make sense. I don't know what you've done 528 00:30:22,200 --> 00:30:24,600 wrong, but I can tell you that that it doesn't make sense. 529 00:30:25,000 --> 00:30:29,700 And finally, you're in a crucial place for judgment is in the 530 00:30:29,700 --> 00:30:33,900 synthesis of the information generally data and Analysis will 531 00:30:33,900 --> 00:30:36,600 tell you the what it would not tell you the server. 532 00:30:36,800 --> 00:30:38,300 We need to do it. What does it mean? 533 00:30:38,300 --> 00:30:40,100 And then now what are we going to do about it? 534 00:30:40,600 --> 00:30:43,100 And these are different components where we want to 535 00:30:43,100 --> 00:30:47,300 combine the quantitative together with intuition. 536 00:30:47,600 --> 00:30:50,200 I use the so and learn a lot all the time. 537 00:30:50,200 --> 00:30:53,800 After you taught me that it's those two questions for 538 00:30:53,800 --> 00:30:57,400 marketers. Truly will change the way that 539 00:30:57,400 --> 00:31:00,900 you approach your, your data and your strategies. 540 00:31:01,200 --> 00:31:03,300 And in fact we said that one of the reasons we went on this 541 00:31:03,300 --> 00:31:06,400 journey of quantitative intuition was we were simply 542 00:31:06,700 --> 00:31:10,900 tired of meetings of horrible meetings or meetings that go on 543 00:31:10,900 --> 00:31:13,000 the water. And so we had a meeting and 544 00:31:13,000 --> 00:31:16,100 someone shows data and now we're going to spend an hour talking 545 00:31:16,100 --> 00:31:17,800 about the water. In fact, there is a Persona, we 546 00:31:17,800 --> 00:31:21,400 talked about in the book, we call them the seymours, the 547 00:31:21,400 --> 00:31:24,300 seymours in the organization are the ones who in every meeting of 548 00:31:24,300 --> 00:31:26,800 only one comment can, I see more data? 549 00:31:27,600 --> 00:31:30,300 And then we can have meetings over meetings and postpone any 550 00:31:30,300 --> 00:31:35,300 decision Forever by asking for more data by by keep digesting. 551 00:31:35,300 --> 00:31:39,400 And, and, and slicing, and And slicing again the data moving 552 00:31:39,400 --> 00:31:42,900 from from the walk, right from what's in the data to? 553 00:31:43,600 --> 00:31:44,800 What does it mean? Right? 554 00:31:44,800 --> 00:31:47,200 What are the implications? And again this is a place where 555 00:31:47,300 --> 00:31:50,400 you know people are even fearing a lot with chat GPT and so on 556 00:31:50,400 --> 00:31:54,400 with generative AI what's left for us as human humans. 557 00:31:54,600 --> 00:31:58,400 If all you do is the what, yeah, you will be replaced by a 558 00:31:58,400 --> 00:32:02,100 machine but if your focus is on the so, what on What does it 559 00:32:02,100 --> 00:32:04,100 mean on the now? But what are we going to do? 560 00:32:04,700 --> 00:32:06,000 Not yet. I mean, I'm not saying that 561 00:32:06,000 --> 00:32:09,500 machines will Not get there at some point but not at least in 562 00:32:09,500 --> 00:32:14,400 the near future are so and calm comes back and he says, as AI 563 00:32:14,400 --> 00:32:18,200 continues to be used more and more, do we really need 564 00:32:18,200 --> 00:32:20,800 marketing people? If a I can understand the 565 00:32:20,800 --> 00:32:23,400 context around that data. Yes. 566 00:32:24,700 --> 00:32:28,000 Definitely. You do you do and I'm sure your 567 00:32:28,000 --> 00:32:32,600 dad has a lot to say as well. I mean, where we are in terms of 568 00:32:32,600 --> 00:32:38,600 chat GPT right now is I would say it is a Full assistant in 569 00:32:38,600 --> 00:32:43,300 many ways, you know, having used it myself, my team uses it, it 570 00:32:43,300 --> 00:32:46,800 gets you, I'm making this number up but let's say a third of the 571 00:32:46,800 --> 00:32:51,600 way but it's not replacing what we're doing, it's not replacing 572 00:32:51,600 --> 00:32:55,200 the knowledge that we have. And it's not doesn't always get 573 00:32:55,200 --> 00:32:59,900 the context and the nuances that are very specific to humans and 574 00:32:59,900 --> 00:33:06,000 then even more, you know, kind of granular or smaller there 575 00:33:06,000 --> 00:33:09,800 more, Subsets of nuances amongst different audiences. 576 00:33:10,000 --> 00:33:13,800 Now, you could argue all of that is data, and overtime, maybe 577 00:33:13,800 --> 00:33:17,500 those things could be put into a formula and maybe it could be 578 00:33:17,500 --> 00:33:20,900 understood. But I think we're still a bit 579 00:33:20,900 --> 00:33:26,400 far away from saying this will resonate because or from 580 00:33:26,400 --> 00:33:30,600 understanding that a word, that means something in this context 581 00:33:30,600 --> 00:33:35,100 can be taken in an entirely different way in that context. 582 00:33:35,100 --> 00:33:39,300 And that context is, Medic. And so yes, is it helpful for 583 00:33:39,300 --> 00:33:42,600 instance, first summarizing some research that you might have. 584 00:33:42,800 --> 00:33:45,500 Absolutely. Does it give you something to 585 00:33:45,500 --> 00:33:47,300 gut? Check your approach? 586 00:33:47,300 --> 00:33:50,700 Can it provide more information? Yeah, it can. 587 00:33:50,800 --> 00:33:54,100 But we're not at the point where it could run this sophisticated 588 00:33:54,100 --> 00:33:57,700 campaign for you and get equal or greater results. 589 00:33:58,200 --> 00:34:01,000 And I think Michael you you really hit it on the head with 590 00:34:01,000 --> 00:34:04,100 it within the day Ward context. I think this is the key. 591 00:34:04,400 --> 00:34:08,000 The key is truly context. Both in terms of, by the way, 592 00:34:08,000 --> 00:34:11,900 how these methods have improved. So the difference between, you 593 00:34:11,900 --> 00:34:15,300 know, GPT to which was the previous version and the GPT 594 00:34:15,300 --> 00:34:17,500 three and a half, which is Chad upto. 595 00:34:17,500 --> 00:34:21,400 Now we already GPT for is context meaning and what I mean 596 00:34:21,400 --> 00:34:25,300 by that is in order to predict the next word on door in order 597 00:34:25,300 --> 00:34:28,400 to interpret a board understand the world with these tools. 598 00:34:28,400 --> 00:34:32,100 Do they take the previous that many words to understand in the 599 00:34:32,100 --> 00:34:35,699 context of the board and the reason why and it truly was. 600 00:34:35,699 --> 00:34:38,800 I mean Church uppity Was a huge leap in Innovation. 601 00:34:38,900 --> 00:34:42,400 Over the previous versions is, literally because of context 602 00:34:42,400 --> 00:34:46,800 because they had they use usable is called 8000 tokens. 603 00:34:46,800 --> 00:34:49,500 Think about 8,000 tokens, something like 6,000 words 604 00:34:49,500 --> 00:34:54,100 because it includes periods and so on but six the previous six 605 00:34:54,100 --> 00:34:57,300 thousand words to understand a particular word, the previous 606 00:34:57,300 --> 00:35:01,300 versions, for example, GPT to use two thousand words to 607 00:35:01,300 --> 00:35:04,300 understand any particular word. Is humans. 608 00:35:04,300 --> 00:35:07,100 We are tremendous actually in doing that in in context, I'll 609 00:35:07,100 --> 00:35:12,000 give you two examples. The first example is when I say 610 00:35:12,000 --> 00:35:15,300 the word model and in fact, it Michael, when you use the word 611 00:35:15,300 --> 00:35:18,600 model, both of us understood that in the in this context, we 612 00:35:18,600 --> 00:35:19,700 are not talking about the fashion type. 613 00:35:19,700 --> 00:35:21,700 We're talking about the nerdy type, right? 614 00:35:21,700 --> 00:35:24,500 Because we have enough context to understand that in this 615 00:35:24,500 --> 00:35:28,200 conversation, unless I'm going to give you a real clue with my 616 00:35:28,200 --> 00:35:31,100 previous words that we are talking about the fashion type 617 00:35:31,100 --> 00:35:33,000 model. We are talking about the nerdy 618 00:35:33,000 --> 00:35:33,900 type. Type right? 619 00:35:34,300 --> 00:35:39,600 And and in fact, we don't even understand how good or how our 620 00:35:39,600 --> 00:35:42,600 brain actually in understand context. 621 00:35:42,600 --> 00:35:44,500 I'll give you an example for that. 622 00:35:44,500 --> 00:35:47,600 Think about a toddler at one and a half year old does not have 623 00:35:47,600 --> 00:35:51,200 the intelligence to speak yet. And we show them illustration in 624 00:35:51,207 --> 00:35:54,800 a book and the book has an illustration of a cat. 625 00:35:55,500 --> 00:35:57,100 And we tell them. This thing does meow. 626 00:35:57,300 --> 00:35:58,900 And then we show them illustration. 627 00:35:58,900 --> 00:36:02,800 The same book of a dog kind of book about animals as we often 628 00:36:03,000 --> 00:36:05,200 read. You are a one and a half year 629 00:36:05,200 --> 00:36:08,400 olds and he tells them. Well this thing does wolf and 630 00:36:08,400 --> 00:36:11,400 then we show them maybe 45 times illustration, different 631 00:36:11,400 --> 00:36:15,300 illustrations from different books and so on the next day 632 00:36:15,300 --> 00:36:16,300 we're going to take them to the street. 633 00:36:16,300 --> 00:36:19,700 They cannot speak here they're going to see a cat or dog a 634 00:36:19,700 --> 00:36:21,500 real. A real cat and a dog will first 635 00:36:21,500 --> 00:36:24,500 time ever they see a real one and they're actually going to be 636 00:36:24,500 --> 00:36:29,300 up meow and wolf without even again they can speak yet but 637 00:36:29,300 --> 00:36:33,100 they do it with five observations a few years back. 638 00:36:33,200 --> 00:36:37,500 The first time that machines went close to human level, in 639 00:36:37,500 --> 00:36:41,400 detecting cats from dogs from images and they've done it 640 00:36:41,400 --> 00:36:43,900 because the researcher made available to million 641 00:36:44,100 --> 00:36:48,100 observations take by humans million of cats and dogs. 642 00:36:48,500 --> 00:36:52,700 How can a human do it with four five observations and a machine 643 00:36:52,700 --> 00:36:54,900 needs to million. We don't understand how we do it 644 00:36:54,900 --> 00:36:56,200 with only four, five observations. 645 00:36:56,200 --> 00:36:58,800 If we did, we would have trained machines to do it because it 646 00:36:58,800 --> 00:37:01,000 will be very useful to have less data. 647 00:37:01,400 --> 00:37:04,400 And I think that's gives you the the The understanding of why 648 00:37:04,400 --> 00:37:07,600 when it comes to The Sword. And the novel words, we don't 649 00:37:07,600 --> 00:37:11,600 have enough data of similar. So what an hour maybe if you are 650 00:37:12,000 --> 00:37:14,900 a physician and we have enough decisions of the kind given 651 00:37:14,900 --> 00:37:18,700 symptoms, maybe it works. But for typical business 652 00:37:18,700 --> 00:37:21,300 decision, which is doesn't come that often. 653 00:37:22,000 --> 00:37:24,400 It's unlikely. It's unlikely that at least not 654 00:37:24,400 --> 00:37:27,400 with the current tools and where we are today or even in the near 655 00:37:27,400 --> 00:37:32,100 future that machines would have enough context is 8000 context. 656 00:37:32,100 --> 00:37:36,300 But repeatedly, Only with enough data to truly go to to that 657 00:37:36,300 --> 00:37:38,600 step. Again, may happen at some point, 658 00:37:38,800 --> 00:37:40,900 not yet. If you want one place, where it 659 00:37:40,900 --> 00:37:45,300 did happen already and happened before generative, AI online 660 00:37:45,300 --> 00:37:47,700 advertising, the allocation of Heirs of which atoms. 661 00:37:47,700 --> 00:37:52,200 And seeing when I go to Google, or when I go to Amazon or 662 00:37:52,200 --> 00:37:56,800 Facebook is almost fully automated apart from the choice 663 00:37:56,800 --> 00:37:58,400 of the creative of The Advertiser. 664 00:37:58,700 --> 00:38:00,300 Yeah. That's has been already 665 00:38:00,300 --> 00:38:04,200 automated because the rules are fairly clear there isn't a Oh so 666 00:38:04,200 --> 00:38:06,300 what does it mean? Well it means that this end is 667 00:38:06,300 --> 00:38:09,000 likely to lead to a higher click-through rate, then another 668 00:38:09,000 --> 00:38:10,400 ad, that's a predictive model machine. 669 00:38:10,400 --> 00:38:15,800 Learning can do that already, how can marketers avoid being 670 00:38:15,900 --> 00:38:20,100 data myopic, which is to say losing sight of the fact that 671 00:38:20,100 --> 00:38:23,900 business decisions involve people circumstances. 672 00:38:23,900 --> 00:38:29,800 We use that term context and not just numbers when marketing has 673 00:38:29,900 --> 00:38:33,300 a seat at the table and is involved. 674 00:38:33,400 --> 00:38:38,300 And in the Strategic decisions things like growth or new 675 00:38:38,300 --> 00:38:42,400 audiences or understand where value is derived for a company 676 00:38:43,300 --> 00:38:46,300 that's when they can put together the pieces and they're 677 00:38:46,300 --> 00:38:49,100 not only seeing their small part. 678 00:38:49,200 --> 00:38:53,200 So I think being part of having a marketer, truly, truly 679 00:38:53,200 --> 00:38:56,200 involved at the senior level, with the seat at the table, 680 00:38:56,500 --> 00:39:00,600 who's, you know, just as involved as anyone else in the 681 00:39:00,600 --> 00:39:05,900 growth of the company is really important because it then 682 00:39:05,900 --> 00:39:09,200 becomes top-down. You ask your team, not to look 683 00:39:09,200 --> 00:39:11,800 at this slice. You ask your team to think about 684 00:39:11,800 --> 00:39:13,700 how it connects to something larger. 685 00:39:13,900 --> 00:39:17,600 Oh, Dad, how can organizations strike, the right balance 686 00:39:17,600 --> 00:39:22,000 between relying on quantitative marketing techniques, versus 687 00:39:22,500 --> 00:39:26,500 trusting, their own intuition and experience bringing the two 688 00:39:26,500 --> 00:39:28,800 together. Ask yourself as you looking at 689 00:39:28,808 --> 00:39:31,900 data. Ask yourself what surprised you 690 00:39:32,800 --> 00:39:35,300 And it's amazing. How this fairly simple. 691 00:39:35,300 --> 00:39:38,500 Deceptively simple question often cut straight to the 692 00:39:38,500 --> 00:39:42,200 straight to the chase. You're either finding a mistake 693 00:39:42,600 --> 00:39:45,600 or you're finding an Insight either way, you benefit. 694 00:39:46,100 --> 00:39:49,600 And so, in other words, don't trust your gut. 695 00:39:49,600 --> 00:39:52,600 Trust your doubts, look for these days surprises. 696 00:39:52,600 --> 00:39:55,900 In the data, Amy, you're going to get the last word here. 697 00:39:55,900 --> 00:39:58,600 What advice do you have for marketers given everything? 698 00:39:58,600 --> 00:40:01,900 We've just been talking about your best friend, is the 699 00:40:01,900 --> 00:40:04,200 personal? One who can build the models, 700 00:40:04,200 --> 00:40:11,500 your you need to be embedded in business decisions to drive True 701 00:40:11,500 --> 00:40:15,700 Value. And I don't think that chat GPT, 702 00:40:15,700 --> 00:40:19,100 we'll take our jobs, but if you don't know how to use it, to 703 00:40:19,100 --> 00:40:22,500 make yourself more efficient, or more effective as a marketer, it 704 00:40:22,500 --> 00:40:25,900 will so it's not a substitute but it's an important complement 705 00:40:25,900 --> 00:40:29,400 that we all have to be using and working with them on a daily 706 00:40:29,400 --> 00:40:33,700 basis. Can I accurately Phrase that as 707 00:40:33,700 --> 00:40:37,500 saying, be part of the business, understand the tools. 708 00:40:37,500 --> 00:40:40,400 Understand the data and bring all of that together to make the 709 00:40:40,400 --> 00:40:44,000 decisions that rely on both the data and your experience with 710 00:40:44,000 --> 00:40:48,200 the business and with the context even better. 711 00:40:49,500 --> 00:40:55,700 And with that a huge thank you to ODed netzer and Amy. 712 00:40:55,700 --> 00:40:59,900 Jake from the Columbia Business. School Columbia University. 713 00:40:59,900 --> 00:41:02,200 Thank you both for taking the time. 714 00:41:02,200 --> 00:41:05,000 Time to be with us and share your expertise with us today. 715 00:41:05,100 --> 00:41:07,600 Well thank you, it's beneficial. Yeah thank you by going to thank 716 00:41:07,600 --> 00:41:11,100 you for the audience and thank you to the audience. 717 00:41:11,100 --> 00:41:13,900 You guys are awesome and your questions are great. 718 00:41:13,900 --> 00:41:18,600 Now before you go, please subscribe to our YouTube channel 719 00:41:18,800 --> 00:41:22,400 and hit the Subscribe button at the bottom of our website. 720 00:41:22,400 --> 00:41:25,100 So you can get our newsletter and we can keep you up-to-date 721 00:41:25,100 --> 00:41:28,000 on upcoming live shows. Just like this one. 722 00:41:28,200 --> 00:41:29,900 Thanks so much. Everybody, check out cxo 723 00:41:29,900 --> 00:41:31,400 talk.com and we'll see you next time. 724 00:41:31,400 --> 00:41:32,100 Have a great day.