1 00:00:00,040 --> 00:00:02,600 The greatest risk with the eye is that if you don't understand 2 00:00:02,600 --> 00:00:07,080 what it does, then you really sabotage your own understanding 3 00:00:07,200 --> 00:00:10,320 and learning journey. I don't think it makes sense for 4 00:00:10,320 --> 00:00:14,160 someone in 2025 to work even the same as they worked in 2024. 5 00:00:14,200 --> 00:00:17,640 The danger will be even worse, right, if some of these agents 6 00:00:17,640 --> 00:00:19,640 hallucinate and they all hallucinate with each other. 7 00:00:20,080 --> 00:00:22,760 At some point it's going to be hard for a human to understand 8 00:00:22,760 --> 00:00:25,200 what's going on, and that's where I think they'll have to 9 00:00:25,200 --> 00:00:27,400 stop and think. Will AI actually improve our 10 00:00:27,400 --> 00:00:29,840 critical thinking or actually reduce our critical thinking? 11 00:00:30,160 --> 00:00:32,119 They do think that people will get burned a lot because they 12 00:00:32,119 --> 00:00:35,520 will believe what AI sent them and then they'll learn in the 13 00:00:35,520 --> 00:00:36,960 hard way that they shouldn't have. 14 00:00:37,240 --> 00:00:40,600 They should resist the urge to send out anything you don't 15 00:00:40,600 --> 00:00:43,560 completely understand. Before the people were scared or 16 00:00:43,560 --> 00:00:46,120 reluctant to write because they would say OK, no one ever read 17 00:00:46,120 --> 00:00:47,600 it. Now AI read is going to read it. 18 00:00:48,120 --> 00:00:50,400 AI is not lazy. AI is going to read it right. 19 00:00:50,840 --> 00:00:54,200 AI is a super powerful tool and especially in explaining things 20 00:00:54,200 --> 00:00:57,120 to humans and also aligning humans to ask questions in 21 00:00:57,120 --> 00:00:58,400 actual language. It's amazing. 22 00:00:58,720 --> 00:01:01,560 But we shouldn't expect AI to understand the code. 23 00:01:02,080 --> 00:01:05,280 No, AI can look in the code and understand that the way didn't 24 00:01:05,280 --> 00:01:09,040 do 5 things that are not implemented because you tried 25 00:01:09,040 --> 00:01:11,000 them sometime and you know that it's not going to work I. 26 00:01:11,040 --> 00:01:13,360 Think there's some danger here about hallucination. 27 00:01:13,480 --> 00:01:15,440 How much context should you give to AI and? 28 00:01:15,640 --> 00:01:18,800 When you provide it with complex code bases, AI makes mistakes. 29 00:01:19,200 --> 00:01:22,360 It has some very fundamental limitations and that's where you 30 00:01:22,360 --> 00:01:42,120 need other techniques like. Hello everyone, welcome back to 31 00:01:42,120 --> 00:01:44,840 another new episode of the Techni Journal podcast. 32 00:01:44,840 --> 00:01:49,120 Today I have with me the CTO and Co founder of swim, Swim dot IO 33 00:01:49,360 --> 00:01:52,520 right, Omar Rosenbaum. He's here with me today in the 34 00:01:52,520 --> 00:01:55,080 show. So I think we plan to have a 35 00:01:55,080 --> 00:01:59,400 lot, a lot of discussions about, you know, learning research, AI, 36 00:01:59,400 --> 00:02:02,520 of course, and also other technical leadership wisdom that 37 00:02:02,560 --> 00:02:05,680 Omar will share today with us. So, Omar, looking forward for 38 00:02:05,680 --> 00:02:08,360 our conversation today. Me too, Henry. 39 00:02:08,360 --> 00:02:10,440 Thank you for having me, it's great to be here. 40 00:02:11,160 --> 00:02:13,080 Yeah. Omair, I'd like to invite you 41 00:02:13,080 --> 00:02:14,960 maybe share something about yourself first. 42 00:02:15,080 --> 00:02:17,440 Any career turning points that you think we can learn from you? 43 00:02:18,480 --> 00:02:22,960 Of course. So I started my career and I had 44 00:02:22,960 --> 00:02:29,000 an opportunity to be a part of a special technological unit and 45 00:02:29,000 --> 00:02:32,200 actually started with a very special training that kind of 46 00:02:32,200 --> 00:02:35,800 changed my life. It was 13 weeks throughout which 47 00:02:35,800 --> 00:02:40,280 I learned more than I had in all of my years before that this 48 00:02:40,280 --> 00:02:44,800 professionally speaking. And it was really mind blowing, 49 00:02:44,800 --> 00:02:48,600 like how much you can learn in such a short amount of time and 50 00:02:48,600 --> 00:02:49,880 also the different methodologies. 51 00:02:49,880 --> 00:02:53,920 And it really got me into getting interested in teaching 52 00:02:54,000 --> 00:02:57,520 or in learning, which also affected a lot of the things I 53 00:02:57,520 --> 00:03:01,200 did later. Then I went to the university 54 00:03:01,520 --> 00:03:05,440 where I took all kinds of courses in variety of topics 55 00:03:05,440 --> 00:03:12,040 including chemistry, math, linguistics, psychology and 56 00:03:12,040 --> 00:03:15,000 others. And while I was a student that 57 00:03:15,000 --> 00:03:20,120 we also taught a lot, wrote a few books and started a few 58 00:03:20,400 --> 00:03:24,480 training programs, one in Singapore by the way. 59 00:03:24,840 --> 00:03:28,480 So yeah. So I got to teach in different 60 00:03:28,480 --> 00:03:31,880 places and again think about teaching methodology as well as 61 00:03:31,880 --> 00:03:36,280 research methodology. And in 2019 I Co founded SWIM 62 00:03:36,760 --> 00:03:40,480 and ever since then I've been the CTO and the Co founder here. 63 00:03:41,160 --> 00:03:43,280 Well thank you for sharing your interesting journey. 64 00:03:43,280 --> 00:03:46,480 I didn't know that you kind of like started a job first before 65 00:03:46,480 --> 00:03:49,400 you go into university. What do you think is an 66 00:03:49,400 --> 00:03:52,760 advantage of starting the job first before you get into uni? 67 00:03:53,440 --> 00:03:54,800 Yeah. So first of all, you don't have 68 00:03:54,800 --> 00:03:58,000 to start a job per SE. It's just that you basically 69 00:03:58,000 --> 00:04:01,080 have a few years of hands, hands on experience, right, which is 70 00:04:01,080 --> 00:04:03,560 more similar to a job. And then when you go to the 71 00:04:03,560 --> 00:04:08,160 university, either you choose to focus a you study computer 72 00:04:08,160 --> 00:04:10,600 science, right? But you come with understanding 73 00:04:10,600 --> 00:04:13,280 what this looks like an application in the real world. 74 00:04:13,800 --> 00:04:17,360 And you want, you want to learn about the science or about 75 00:04:17,839 --> 00:04:21,000 specific aspects of it like algorithms and so on. 76 00:04:21,800 --> 00:04:25,760 And you can also choose to learn other things and broaden your 77 00:04:25,760 --> 00:04:31,560 horizons while you already have a lot of knowledge and a firm 78 00:04:31,560 --> 00:04:34,040 basis based on the hands on experience you got. 79 00:04:34,680 --> 00:04:38,240 So I think it gives the whole learning experience a very 80 00:04:38,240 --> 00:04:43,400 different atmosphere. And you can approach it not as 81 00:04:43,400 --> 00:04:45,720 like, this is my way to get a job because I already have my 82 00:04:45,720 --> 00:04:47,120 skills, right? I'm here to learn. 83 00:04:47,120 --> 00:04:49,760 I'm here to deepen my knowledge to understand things better. 84 00:04:50,840 --> 00:04:51,960 I think that's really interesting. 85 00:04:51,960 --> 00:04:54,000 OMSO what I think as well, right? 86 00:04:54,000 --> 00:04:56,640 If you started kind of like a job first before you go into 87 00:04:56,640 --> 00:04:59,600 uni, you kind of like have a good understanding of what kind 88 00:04:59,600 --> 00:05:01,560 of roles that you like or don't like. 89 00:05:01,880 --> 00:05:04,400 And probably the hands on experience will be also quite 90 00:05:04,520 --> 00:05:05,760 relatable, right, when you study. 91 00:05:05,760 --> 00:05:09,400 Because sometimes I imagine if we go to uni, we can't even 92 00:05:09,400 --> 00:05:11,080 relate to some of the subjects, right? 93 00:05:11,080 --> 00:05:13,680 So we don't even know it's going to be useful or not. 94 00:05:14,000 --> 00:05:16,640 And I think a lot of challenges in parts of the world as 95 00:05:16,640 --> 00:05:20,200 juniors, right, is actually to understand whether what I'm 96 00:05:20,200 --> 00:05:24,120 learning now is going to be usable or relatable with my job 97 00:05:24,120 --> 00:05:26,240 or not. And especially these days, you 98 00:05:26,240 --> 00:05:28,040 know, the crazies about AI, right? 99 00:05:28,200 --> 00:05:31,360 And there's a lot of fear for juniors about, you know, getting 100 00:05:31,360 --> 00:05:34,760 a job because some people think AI could replace the need for 101 00:05:34,760 --> 00:05:37,000 any kind of juniors. So maybe let's start our 102 00:05:37,000 --> 00:05:39,240 discussion by discussing about this challenge, right? 103 00:05:39,320 --> 00:05:43,680 So what do you think juniors should do in the age of AI these 104 00:05:43,680 --> 00:05:45,640 days? About getting a good job or 105 00:05:45,640 --> 00:05:50,480 about learning new skills? So first of all, I think juniors 106 00:05:50,480 --> 00:05:54,200 need to acknowledge that it's a challenging time for juniors. 107 00:05:54,680 --> 00:05:55,920 First of all, because it's unclear. 108 00:05:56,160 --> 00:05:59,160 I mean, we can postulate about the impact of AI on junior 109 00:05:59,160 --> 00:06:01,080 developers. Different people have different 110 00:06:01,360 --> 00:06:03,680 opinions. We don't know them for a fact, 111 00:06:04,040 --> 00:06:06,440 right? So clearly it's more 112 00:06:06,440 --> 00:06:09,480 challenging. And I think, I mean, as a 113 00:06:09,480 --> 00:06:14,040 junior, you can't impact the fact that now AI is changing the 114 00:06:14,040 --> 00:06:17,720 world, right? What you can do is acknowledge 115 00:06:17,720 --> 00:06:19,440 that and try to find the opportunities. 116 00:06:19,640 --> 00:06:26,560 I think on one hand it's really challenging to justify the need 117 00:06:26,560 --> 00:06:29,280 for a junior developer for some tasks. 118 00:06:29,640 --> 00:06:32,600 If you think of a junior developer as someone who will do 119 00:06:32,600 --> 00:06:36,640 the small tasks right while gradually learning and becoming 120 00:06:36,640 --> 00:06:42,360 better, then now you could let some coding assistants do that 121 00:06:42,520 --> 00:06:45,880 for you, right? On the other hand, I don't think 122 00:06:45,880 --> 00:06:48,600 juniors are going to be extinct from the world, right? 123 00:06:48,600 --> 00:06:53,520 Everyone starts as a junior and we eventually evolved to being 124 00:06:53,520 --> 00:06:55,320 more senior and more experienced. 125 00:06:55,600 --> 00:06:59,080 Everyone go through that. And I think for junior 126 00:06:59,080 --> 00:07:02,400 developers, the important thing is to understand what's 127 00:07:02,400 --> 00:07:04,160 important to learn and what is not. 128 00:07:04,800 --> 00:07:11,800 For example, I think learning with AI and how to use AI 129 00:07:11,800 --> 00:07:15,920 effectively is important and it's one of the advantages they 130 00:07:15,920 --> 00:07:19,040 can get, right? Like if you see, I don't know 131 00:07:19,040 --> 00:07:21,720 our, whenever we see our grandparents operate the 132 00:07:21,720 --> 00:07:24,480 computer, we feel the difference, right? 133 00:07:24,480 --> 00:07:27,960 Even when you see like a 10 year old, he's computer native, 134 00:07:28,160 --> 00:07:30,720 right? They started with a computer, 135 00:07:30,720 --> 00:07:32,680 they know what it's like. So junior developers can 136 00:07:32,680 --> 00:07:35,840 actually start with AI and feel even more comfortable working 137 00:07:35,840 --> 00:07:39,400 with AI tools. Perhaps on the other hand, they 138 00:07:39,400 --> 00:07:42,480 should be super careful about not understanding things. 139 00:07:42,480 --> 00:07:47,560 I think what differentiates experienced, high qualified and 140 00:07:47,560 --> 00:07:51,640 effective engineers or researchers from the rest is the 141 00:07:51,640 --> 00:07:53,760 fact they understand deeply how things work. 142 00:07:54,560 --> 00:07:58,040 And the greatest risk with the eye is that it seems to do, you 143 00:07:58,040 --> 00:08:01,200 know, you're asking it to do something, you get some code 144 00:08:01,200 --> 00:08:03,720 back, it kind of does what you wanted it to do. 145 00:08:03,920 --> 00:08:06,720 And if you don't understand what it does or you don't understand 146 00:08:06,720 --> 00:08:12,600 what the alternative is, then you really sabotage your own 147 00:08:12,720 --> 00:08:15,720 understanding and learning journey. 148 00:08:16,200 --> 00:08:18,280 And as a junior, that's the most important thing, right? 149 00:08:18,280 --> 00:08:20,560 You need to learn and improve everyday. 150 00:08:21,400 --> 00:08:24,520 And also you might introduce critical bugs without knowing 151 00:08:24,520 --> 00:08:28,320 that. So I think the urge to just ship 152 00:08:28,320 --> 00:08:33,159 something, let's say I got a task, OK, I sent it to one of 153 00:08:33,159 --> 00:08:35,840 the AI coding assistants, or I use the completion tool or 154 00:08:35,840 --> 00:08:37,799 whatever. I got a result. 155 00:08:37,799 --> 00:08:40,960 I can just issue the PR. I think of juniors. 156 00:08:41,000 --> 00:08:45,480 You should resist the urge to send out anything you don't 157 00:08:45,480 --> 00:08:48,760 completely understand. On the other hand, there is an 158 00:08:48,760 --> 00:08:53,280 amazing opportunity here before I I on something they could get 159 00:08:53,280 --> 00:08:56,360 stuck and just they need a senior developer to help them. 160 00:08:56,520 --> 00:08:59,640 That's it. Which might be a waste of time 161 00:08:59,640 --> 00:09:01,920 of the senior developer. Maybe I don't feel so 162 00:09:01,920 --> 00:09:04,360 comfortable as a junior developer to approach a senior 163 00:09:04,360 --> 00:09:06,920 to help me and admit that I'm blocked, right? 164 00:09:07,280 --> 00:09:10,240 And AI can actually unblock you from lots of things. 165 00:09:10,520 --> 00:09:14,200 It might be a generic thing about, you know, as it's junior 166 00:09:14,200 --> 00:09:17,880 developers who might not know how to run some tests, which is 167 00:09:17,880 --> 00:09:20,640 something that generic that would take you a long time to 168 00:09:20,640 --> 00:09:23,080 learn. But if you ask Claude or GPT or 169 00:09:23,080 --> 00:09:26,720 whatever, you'd get an answer really quickly, right? 170 00:09:27,200 --> 00:09:29,880 And also if you select a specific part of the code that 171 00:09:29,880 --> 00:09:32,200 is hard for you to understand and you ask AI to walk you 172 00:09:32,200 --> 00:09:36,680 through it step by step, it might be an easier, a faster way 173 00:09:36,680 --> 00:09:41,000 to learn than the non AI way of having to understand yourself. 174 00:09:41,000 --> 00:09:43,520 Or again, if you're stuck asking for help from someone. 175 00:09:43,960 --> 00:09:46,960 So it's like having someone with you. 176 00:09:47,600 --> 00:09:52,760 I think on in a sense the dream of every junior developer to 177 00:09:52,760 --> 00:09:55,760 have a senior developer who doesn't get tired of your 178 00:09:55,760 --> 00:09:57,240 questions. Right? 179 00:09:57,800 --> 00:10:00,520 You just need to realize, not really a senior developer. 180 00:10:01,400 --> 00:10:03,360 They don't really know the code base. 181 00:10:03,560 --> 00:10:06,640 They don't really understand all the pros and cons, right? 182 00:10:07,160 --> 00:10:12,360 They're a limited senior developer with some strength and 183 00:10:12,360 --> 00:10:15,760 some weaknesses, and your goal is to make sure you understand 184 00:10:15,760 --> 00:10:18,520 what's going on. That's the main thing I think of 185 00:10:18,520 --> 00:10:19,760 the junior developer you have to do. 186 00:10:20,760 --> 00:10:24,000 I think you brought up very good points, right, specifically like 187 00:10:24,080 --> 00:10:25,760 your term. You know I'm going to use it 188 00:10:25,760 --> 00:10:27,600 like AI native generation, right. 189 00:10:27,920 --> 00:10:30,960 So because when they go into the workforce these days, right, 190 00:10:30,960 --> 00:10:33,440 they can use AI. It is simply available, right? 191 00:10:33,720 --> 00:10:36,280 And I think looking back at my time back then, right, we 192 00:10:36,520 --> 00:10:39,000 probably the best that we had our books, right? 193 00:10:39,240 --> 00:10:42,640 Maybe Google Stack Overflow was probably also just starting, 194 00:10:42,640 --> 00:10:45,760 right? So like now you have AI natively 195 00:10:45,760 --> 00:10:48,640 that you can use straight away. So I think this gives a lot of 196 00:10:48,640 --> 00:10:51,440 opportunity for sure for anyone to upscale and learn about 197 00:10:51,440 --> 00:10:53,840 something. And especially if the workplace 198 00:10:53,840 --> 00:10:57,080 also allows AI usage, you know, like AI coding assistance, you 199 00:10:57,080 --> 00:11:00,360 can also get up to speak to the code base, asking about specific 200 00:11:00,640 --> 00:11:03,280 parts of the code that you don't understand without bugging the 201 00:11:03,400 --> 00:11:05,000 so-called the senior developers, right? 202 00:11:05,760 --> 00:11:08,560 And I think you brought up a good points about, you know, 203 00:11:08,640 --> 00:11:11,600 understanding your code base, even though it's generated by AI 204 00:11:11,600 --> 00:11:14,760 before you actually submit or even push it to an environment 205 00:11:14,760 --> 00:11:17,840 of production, right? So tell us maybe about this 206 00:11:17,840 --> 00:11:20,160 workflow, right? So for example, a junior, you 207 00:11:20,160 --> 00:11:24,120 know, specifically has a task and it has to run specific, I 208 00:11:24,120 --> 00:11:26,880 don't know, like business logic or something like that, but AI 209 00:11:26,880 --> 00:11:28,880 gives them like a code that they don't understand. 210 00:11:28,880 --> 00:11:31,520 What specifically should a junior developer do? 211 00:11:32,440 --> 00:11:36,360 As a junior developer, I mean, right, let's start from what you 212 00:11:36,360 --> 00:11:39,760 shouldn't do, right? So I gave a specific coding 213 00:11:39,760 --> 00:11:43,600 assistant a small task and I wanted to see what it does. 214 00:11:43,600 --> 00:11:45,800 And I wanted it to help me save me time. 215 00:11:46,120 --> 00:11:50,240 And I told you, OK, write a test, implement the task and 216 00:11:50,440 --> 00:11:52,880 keep iterating until it passes the test, right? 217 00:11:52,880 --> 00:11:56,880 And then it says, OK, great, everything passes, you can work 218 00:11:56,880 --> 00:11:59,280 with it now. Then I look at the code, right? 219 00:11:59,760 --> 00:12:03,360 And I see that it has an if for the specific case of the test 220 00:12:03,440 --> 00:12:06,200 and it does something to pass the test specifically, right? 221 00:12:06,200 --> 00:12:09,440 So it has a string of like the mock data that I used. 222 00:12:09,520 --> 00:12:11,920 So if that returned this OK, the test test great. 223 00:12:11,920 --> 00:12:14,560 Right now it's an extreme example, but it happened to me 224 00:12:14,560 --> 00:12:18,440 right now, right? I mean, very recently and a 225 00:12:18,440 --> 00:12:22,320 junior developer might say, OK, I have it, the test pass and I 226 00:12:22,320 --> 00:12:24,680 will ship it, right? This would create a really bad 227 00:12:24,680 --> 00:12:30,680 impression of course, but also think of a less clear example 228 00:12:30,680 --> 00:12:34,320 where if you actually understand the code, you understand that it 229 00:12:34,320 --> 00:12:40,480 actually solved a very specific case and not the broader scope. 230 00:12:40,520 --> 00:12:43,960 Or maybe there are some other considerations, security 231 00:12:43,960 --> 00:12:46,440 considerations. Maybe there is something in 232 00:12:46,440 --> 00:12:50,120 there which puts a vulnerability, maybe performance 233 00:12:50,120 --> 00:12:53,400 considerations like it works, but it wouldn't work on scale. 234 00:12:54,000 --> 00:12:56,720 And in case this code should operating scale, right? 235 00:12:57,000 --> 00:12:59,560 And performance is important, you must know that. 236 00:13:00,200 --> 00:13:02,800 So I think of the junior developer, you can take the 237 00:13:02,800 --> 00:13:06,800 task, explain it. By the way, the fact that AI 238 00:13:06,800 --> 00:13:10,520 makes you explaining natural language what the task is, is 239 00:13:10,520 --> 00:13:12,360 great. Because whenever I got to teach 240 00:13:12,360 --> 00:13:15,800 people how to program, the first thing I told them is explain to 241 00:13:15,800 --> 00:13:18,520 yourself. I wouldn't say natural language, 242 00:13:18,520 --> 00:13:19,920 right? Just explain to yourself what 243 00:13:19,920 --> 00:13:21,800 you're supposed to do and what the steps are. 244 00:13:22,200 --> 00:13:25,360 Now if you write this to AI, it's actually a good practice. 245 00:13:25,880 --> 00:13:28,240 And then when you get back a response, you need to go over 246 00:13:28,240 --> 00:13:33,040 everything and understand everything and make sure it 247 00:13:33,040 --> 00:13:35,240 makes sense in the projects of the task. 248 00:13:36,040 --> 00:13:38,360 At which point, if you don't understand something, you can 249 00:13:38,440 --> 00:13:39,720 use AI. Ask it. 250 00:13:39,880 --> 00:13:42,360 Why did you do this? Is there another way? 251 00:13:42,560 --> 00:13:45,960 What are the pros and cons? Right, Keep checking that, but 252 00:13:45,960 --> 00:13:48,680 never ship anything you don't completely understand. 253 00:13:49,280 --> 00:13:52,800 I think it would be very challenging for people now. 254 00:13:53,240 --> 00:13:57,040 It would require a lot of discipline not to ship something 255 00:13:57,040 --> 00:13:59,240 you don't completely understand. Because they're just like, why 256 00:13:59,240 --> 00:14:01,440 would I even care, right? People say like, AI will write 257 00:14:01,440 --> 00:14:04,840 code, then AI will fix the code. At least for now. 258 00:14:04,840 --> 00:14:08,840 It's very far from the truth. And again, for a junior, you 259 00:14:08,840 --> 00:14:10,400 need to understand so you can grow. 260 00:14:10,800 --> 00:14:15,880 You won't grow by delegating all of your tasks to AI. 261 00:14:16,920 --> 00:14:19,800 Yeah, I think that's a very good reminder, I would say, right? 262 00:14:19,800 --> 00:14:23,480 Because sometimes I think even seniors using AI, right, if they 263 00:14:23,480 --> 00:14:26,920 think, oh, this code looks OK just by looking at a glance, 264 00:14:26,920 --> 00:14:28,800 right? Sometimes a particular bug could 265 00:14:28,800 --> 00:14:32,040 just appear and before you realize it makes a, you know, 266 00:14:32,040 --> 00:14:33,960 like a production issue or something like that, right? 267 00:14:34,240 --> 00:14:36,880 So I think no matter whether you're senior, junior, right, 268 00:14:36,920 --> 00:14:39,640 always look at the generated code by AI, right? 269 00:14:39,920 --> 00:14:41,760 And make sure that you understand, again, like the key 270 00:14:41,760 --> 00:14:44,880 point that you emphasize is like understand exactly what the code 271 00:14:44,880 --> 00:14:46,360 is doing and maybe also the design, right? 272 00:14:46,360 --> 00:14:49,680 What certain aspects suggested by the AI, I think that's a very 273 00:14:49,680 --> 00:14:52,160 good thing. And I think this brings me to 274 00:14:52,160 --> 00:14:53,840 the next question that I'd like to ask, right? 275 00:14:53,920 --> 00:14:58,480 But simply just by understanding and, and maybe asking back, 276 00:14:58,480 --> 00:15:00,840 right, or maybe being curious about why certain things are 277 00:15:00,840 --> 00:15:03,200 suggested that way, it's actually like a critical 278 00:15:03,200 --> 00:15:04,920 thinking kind of a capability, right? 279 00:15:05,160 --> 00:15:08,120 And there's a recent research saying that, you know, using AIA 280 00:15:08,120 --> 00:15:11,320 lot will actually impact your critical thinking ability. 281 00:15:11,320 --> 00:15:14,600 Maybe it will reduce it or even make you less critical. 282 00:15:14,600 --> 00:15:17,880 So maybe in your point, what's your view about this? 283 00:15:17,880 --> 00:15:20,760 Will AI actually improve our critical thinking or actually 284 00:15:20,960 --> 00:15:25,400 reduce our critical thinking? I think that it's hard to tell, 285 00:15:25,520 --> 00:15:27,120 right? I'm not a prophet. 286 00:15:27,520 --> 00:15:30,240 I do think there is a risk there, but I do think that 287 00:15:30,240 --> 00:15:33,440 people will get burned a lot because they will believe what 288 00:15:33,440 --> 00:15:37,960 AI sent them, like the responses they get, and then they'll learn 289 00:15:37,960 --> 00:15:39,920 in the hard way that they shouldn't have. 290 00:15:40,240 --> 00:15:44,360 So it might make them more willing to check everything they 291 00:15:44,360 --> 00:15:48,840 get as a response. And so I'll be optimistic and 292 00:15:48,840 --> 00:15:53,120 say it would help our critical thinking if some junior 293 00:15:53,120 --> 00:15:57,040 developers are listening. I would advise you to do that 294 00:15:57,680 --> 00:16:01,360 elaborately. Make sure you want to improve on 295 00:16:01,440 --> 00:16:05,800 your critical thinking rather than get burned and then iterate 296 00:16:05,800 --> 00:16:07,520 from there. Yeah. 297 00:16:07,520 --> 00:16:09,960 I think like any kind of technological advancement, 298 00:16:09,960 --> 00:16:11,640 right, it makes your life easier, right? 299 00:16:11,640 --> 00:16:15,040 And by making life easier, sometimes we get lazy, so to 300 00:16:15,040 --> 00:16:17,200 speak, right? So don't forget like you have to 301 00:16:17,200 --> 00:16:19,120 be critical, right? That's the first thing, 302 00:16:19,120 --> 00:16:21,360 understanding fundamentals. And I think the other aspect 303 00:16:21,360 --> 00:16:23,960 that is really important, especially for junior, is 304 00:16:23,960 --> 00:16:26,080 actually understanding the domain knowledge, right? 305 00:16:26,160 --> 00:16:29,600 Maybe it's the business aspects of it, it may be, I don't know, 306 00:16:29,720 --> 00:16:32,080 other aspects of the code base that not necessarily just 307 00:16:32,080 --> 00:16:34,360 technological. So maybe in your view, what's 308 00:16:34,360 --> 00:16:36,960 your take about juniors, you know, upscaling themselves in 309 00:16:36,960 --> 00:16:38,880 domain knowledge? Because yeah, sometimes you can 310 00:16:38,880 --> 00:16:41,760 ask AI maybe if it's like generic domain knowledge, but 311 00:16:41,760 --> 00:16:44,600 there are specific things that are in organization that 312 00:16:44,600 --> 00:16:47,560 probably is not easy for AI to suggest something. 313 00:16:48,400 --> 00:16:51,080 Of course. So I think we're raising 2 areas 314 00:16:51,080 --> 00:16:55,680 where juniors can grow here. 1 is the technological area where 315 00:16:55,680 --> 00:17:00,240 it will also be with you when you switch jobs or switch 316 00:17:00,640 --> 00:17:03,840 business domains, right? And it's the fundamental of 317 00:17:03,840 --> 00:17:06,160 being an engineer, I think. So it's a must. 318 00:17:06,440 --> 00:17:10,440 But you're also raising another very valid point to AI excels as 319 00:17:10,440 --> 00:17:12,480 creating demos from scratch, right? 320 00:17:12,480 --> 00:17:15,280 Because it's something generic. It doesn't XLS much. 321 00:17:15,680 --> 00:17:18,560 It's getting better. We get other tools that enrich 322 00:17:18,560 --> 00:17:22,319 it with context, but AI never knows your company really. 323 00:17:22,560 --> 00:17:24,440 It doesn't understand the broader business logic. 324 00:17:24,440 --> 00:17:27,720 It wasn't in that meeting where the product manager met with the 325 00:17:27,720 --> 00:17:30,640 client, understood the needs, and then on the meeting whether 326 00:17:30,640 --> 00:17:33,840 the product manager told you what they need, right. 327 00:17:34,200 --> 00:17:38,000 So I think one of the responsibilities of every 328 00:17:38,000 --> 00:17:41,080 engineer, not just junior engineers, would be to 329 00:17:41,080 --> 00:17:45,120 communicate to AI and to the code you're writing yourself, 330 00:17:45,800 --> 00:17:50,920 the specific business logic, business rules, constraints, all 331 00:17:50,920 --> 00:17:54,040 of that context that is unique to your organization. 332 00:17:54,840 --> 00:17:57,960 And I think writing it, by the way, is crucial and something 333 00:17:57,960 --> 00:18:01,560 we've been neglecting as humanity for various reason. 334 00:18:02,200 --> 00:18:04,760 But I think with AI it can actually be easier because you 335 00:18:04,760 --> 00:18:07,360 explain to AI why you did something and then you can 336 00:18:07,720 --> 00:18:10,120 hopefully preserve that knowledge alongside the code. 337 00:18:11,320 --> 00:18:14,520 Yeah, so I think writing I've been hearing a lot of times, 338 00:18:14,520 --> 00:18:17,480 right, in so many different episodes, like writing skills, 339 00:18:17,480 --> 00:18:20,480 very crucial, right. Leaders especially, you can't 340 00:18:20,480 --> 00:18:24,120 just lead by talking to people. You have to write more, right? 341 00:18:24,320 --> 00:18:27,800 And writing here also means like some kind of knowledge base, 342 00:18:27,800 --> 00:18:30,200 right? And I know swim dot IO is, you 343 00:18:30,200 --> 00:18:32,320 know, kind of like dealing with knowledge problem, right. 344 00:18:32,600 --> 00:18:36,600 So tell us, how can we actually learn better by, you know, 345 00:18:36,960 --> 00:18:39,680 having a knowledge base within the company or the practice 346 00:18:39,680 --> 00:18:41,480 itself, right? Having more writing more 347 00:18:41,480 --> 00:18:43,640 documentations within the organization. 348 00:18:44,480 --> 00:18:46,560 Right. So I think if you have an 349 00:18:46,560 --> 00:18:51,800 organization that values written knowledge and specifically 350 00:18:51,800 --> 00:18:57,160 documentation, you got lots of benefits before the era of Gen. 351 00:18:57,160 --> 00:19:00,240 AI and coding assistance. You would say it's crucial for 352 00:19:00,240 --> 00:19:02,920 developers and the communication between them, right? 353 00:19:03,600 --> 00:19:07,120 I think one of the extremist cases I've experienced myself 354 00:19:07,120 --> 00:19:11,880 was a crucial real time system that two separate teams worked 355 00:19:11,880 --> 00:19:13,440 on. And there was a queue of 356 00:19:13,440 --> 00:19:18,720 messages where one team assumed that one was the top priority 357 00:19:18,720 --> 00:19:23,840 and 10 was the lowest priority. And the other team assumed 358 00:19:23,840 --> 00:19:27,120 exactly the opposite. And they didn't understand why 359 00:19:27,120 --> 00:19:30,320 they get random sequences of messages and it's not according 360 00:19:30,320 --> 00:19:34,080 to priority, right? So this is just an extreme clear 361 00:19:34,200 --> 00:19:37,680 example of miscommunication. But in general, when you write 362 00:19:37,680 --> 00:19:41,400 code and it accumulates over time, what you lose is the 363 00:19:41,400 --> 00:19:44,760 business logic context and why you did lots of things. 364 00:19:44,760 --> 00:19:47,200 There are things you can't deduce from code. 365 00:19:47,200 --> 00:19:49,920 No way I can look in the code and understand that the way 366 00:19:50,440 --> 00:19:54,240 didn't do 5 things that are not implemented. 367 00:19:55,120 --> 00:19:57,680 They're not there because you tried them sometime and you know 368 00:19:57,680 --> 00:20:00,040 that it's not going to work because of this and that and 369 00:20:00,040 --> 00:20:02,360 they don't know that it's a request by a specific client. 370 00:20:03,080 --> 00:20:09,000 So when you have such unique knowledge, it's critical to 371 00:20:09,480 --> 00:20:14,800 capture it and preserve it. And I think now given that we 372 00:20:14,800 --> 00:20:18,400 provide the context to AI coding assistants to help us with the 373 00:20:18,400 --> 00:20:23,120 coding tasks, it even more clear that you have clear value from 374 00:20:23,120 --> 00:20:26,200 writing. Before the people were scared or 375 00:20:26,200 --> 00:20:29,040 reluctant to write because they would say, OK, no one ever read 376 00:20:29,040 --> 00:20:30,680 it. Now AI read is going to read it. 377 00:20:31,200 --> 00:20:33,720 AI is not lazy. AI is going to read it right. 378 00:20:34,280 --> 00:20:38,360 Your coding assistant will read your docs and use them. 379 00:20:39,160 --> 00:20:41,960 And then one of the biggest challenges is actually keeping 380 00:20:41,960 --> 00:20:44,400 that knowledge up to date with the code as it evolves. 381 00:20:44,800 --> 00:20:47,320 And this is actually one of the things we solved first in SWIM, 382 00:20:47,320 --> 00:20:50,920 even before the era of Gen. AI as it is today when it was 383 00:20:51,120 --> 00:20:55,920 commonly used, we started by allowing developers to write 384 00:20:55,920 --> 00:20:59,000 documentation and make sure automatically that it's kept up 385 00:20:59,000 --> 00:21:00,640 to date with the code as the code evolves. 386 00:21:01,400 --> 00:21:05,360 With AI, it's even more crucial because AI can look at a piece 387 00:21:05,360 --> 00:21:09,760 of documentation, not know that it's outdated, and rely on it 388 00:21:09,760 --> 00:21:13,280 when generating code or other docs or tests and so on. 389 00:21:13,880 --> 00:21:18,440 So this is one of the pillars for having a knowledge base that 390 00:21:18,440 --> 00:21:21,960 is comprehensive, describes everything you need to know, 391 00:21:22,200 --> 00:21:25,320 especially things that you cannot deduce from the code, and 392 00:21:25,320 --> 00:21:29,360 caps up to date as your code evolves and then reachable to 393 00:21:29,360 --> 00:21:33,720 both humans you might need it and AI assistance and agents. 394 00:21:34,640 --> 00:21:37,360 Wow, I think I left when you mentioned that before we were 395 00:21:37,360 --> 00:21:40,240 not so sure anyone would read the documentation, right? 396 00:21:40,240 --> 00:21:42,840 So sometimes we are lazy to write some documentation because 397 00:21:42,840 --> 00:21:44,200 we think nobody is gonna read it. 398 00:21:44,520 --> 00:21:46,560 But now I think you make a fairpoint, right? 399 00:21:46,560 --> 00:21:49,000 The AI will be the first audience, especially if the AI 400 00:21:49,120 --> 00:21:51,200 tool has access to your documentations, right? 401 00:21:51,680 --> 00:21:53,600 So I think you have dealt with this kind of challenge, 402 00:21:53,600 --> 00:21:56,800 knowledge sharing and also understanding documentations 403 00:21:56,840 --> 00:21:59,480 with your company, right? So tell us how maybe some use 404 00:21:59,480 --> 00:22:03,000 cases of how AI actually improves this kind of knowledge 405 00:22:03,000 --> 00:22:06,360 sharing and also documentation. Because all we know, I mean, we 406 00:22:06,360 --> 00:22:08,680 are laymen people, we think, we just feel to AI. 407 00:22:09,080 --> 00:22:11,360 AI will summarize, AI will tell you what to do. 408 00:22:11,520 --> 00:22:14,160 I think there's some danger here about hallucination. 409 00:22:14,360 --> 00:22:17,040 You know the context, right? How much context should you give 410 00:22:17,040 --> 00:22:18,960 to AI? Do you actually also share 411 00:22:18,960 --> 00:22:20,680 everything within the organization to AI? 412 00:22:20,680 --> 00:22:23,040 So tell us a little bit more about this, because these are 413 00:22:23,040 --> 00:22:25,480 the nuances that I think some of us might not understand. 414 00:22:26,280 --> 00:22:28,960 I agree. And I think the difference lies 415 00:22:28,960 --> 00:22:34,720 in the small details sometimes because when you give AI code 416 00:22:34,720 --> 00:22:37,080 piece in isolation and you ask AI please explain what's 417 00:22:37,080 --> 00:22:40,880 happening here, it will do an amazing job most of the time. 418 00:22:41,600 --> 00:22:46,000 However, when you talk about knowledge that you might need 419 00:22:46,000 --> 00:22:47,760 and you think what kind of knowledge is it? 420 00:22:48,160 --> 00:22:51,000 So one type of knowledge is understanding some code in 421 00:22:51,000 --> 00:22:53,360 isolation, right? But there are other things. 422 00:22:53,400 --> 00:22:56,560 The things that are not in the code cannot be deduced from the 423 00:22:56,560 --> 00:22:58,200 code, right? Obviously. 424 00:22:58,560 --> 00:23:02,920 So again, some business logic that is not clearly translated 425 00:23:02,920 --> 00:23:07,720 into the code, requirements that stem from some regulation or 426 00:23:07,720 --> 00:23:11,720 client requirements, things you ended up not implementing for 427 00:23:11,720 --> 00:23:14,920 some reasons, all of that is not something you can deduce from 428 00:23:15,520 --> 00:23:19,120 just reading the code. And I think AI is great at 429 00:23:19,880 --> 00:23:23,280 writing things in a way that humans can understand, right? 430 00:23:23,280 --> 00:23:26,520 So if you provide it with very clear context and all the 431 00:23:26,520 --> 00:23:30,200 contact need, then you say, now create, write it in a way that's 432 00:23:30,200 --> 00:23:33,160 easy to understand. Perhaps translate it to another 433 00:23:33,160 --> 00:23:36,160 language because not all of us are native English speakers and 434 00:23:36,160 --> 00:23:39,120 not everyone wants their documentation or explanations to 435 00:23:39,120 --> 00:23:40,720 be in English. Translate that. 436 00:23:40,720 --> 00:23:44,640 This is amazing for AI, but we also need to understand and 437 00:23:44,640 --> 00:23:48,600 acknowledge the limitations of AI. 1 is not assuming it could 438 00:23:48,600 --> 00:23:51,400 understand things that are not in the code, but another is that 439 00:23:51,400 --> 00:23:56,480 also code is not always easily understood even by AI. 440 00:23:57,200 --> 00:24:01,240 Just for given a few examples here, if you give AI one 441 00:24:01,240 --> 00:24:04,600 function and it's written in a way that is very clear, the flow 442 00:24:04,600 --> 00:24:08,040 is linear, or the function name is clear, it's documented, Of 443 00:24:08,040 --> 00:24:09,320 course, it's easy to understand, right? 444 00:24:10,160 --> 00:24:13,880 You take the same function, you rename it, you remove all the 445 00:24:13,880 --> 00:24:16,560 comments, and you change the variable names to be something a 446 00:24:16,560 --> 00:24:20,760 bit less clear, which might sound like an exercise in bad 447 00:24:20,760 --> 00:24:23,160 coding, but I've seen enough code bases to know that it 448 00:24:23,160 --> 00:24:27,840 happened in many companies. OK, then all of a sudden AI is 449 00:24:27,840 --> 00:24:30,640 starting to make some mistakes, one understanding it. 450 00:24:30,640 --> 00:24:33,320 And then you get to complex flows. 451 00:24:33,560 --> 00:24:36,360 And when you have flows, you have lots of cases that 452 00:24:37,120 --> 00:24:40,280 compilers know to take into account and AI doesn't. 453 00:24:40,280 --> 00:24:42,480 For example, you have ambiguity resolution. 454 00:24:42,480 --> 00:24:46,920 So let's say you have a call to a function called find and you 455 00:24:46,920 --> 00:24:49,760 have the find of, I don't know, no JS find. 456 00:24:49,880 --> 00:24:52,960 But you also have three different find functions in your 457 00:24:52,960 --> 00:24:56,720 code base and you call find. And if AI picks the wrong 458 00:24:56,720 --> 00:25:01,840 implementation of find, it can get the whole flow wrong right? 459 00:25:02,680 --> 00:25:05,080 So ambiguity resolution is 1 case. 460 00:25:05,800 --> 00:25:11,000 And also if the code involves looking at resources that are 461 00:25:11,000 --> 00:25:13,480 external like reading from database that it might not have 462 00:25:13,480 --> 00:25:16,000 access to. So in Jordan, I think AI is a 463 00:25:16,000 --> 00:25:20,080 super super super powerful tool and especially in explaining 464 00:25:20,080 --> 00:25:23,440 things to humans and also aligning humans to ask questions 465 00:25:23,440 --> 00:25:25,040 in actual language. It's amazing. 466 00:25:25,360 --> 00:25:28,480 But we shouldn't expect AI to understand the code. 467 00:25:29,160 --> 00:25:30,840 It doesn't really understand the code. 468 00:25:31,680 --> 00:25:34,520 And when you provide it with complex code bases that are 469 00:25:34,520 --> 00:25:40,000 convoluted with mixed conventions, specific domain 470 00:25:40,000 --> 00:25:44,840 knowledge that it hasn't been trained on, of course, sometimes 471 00:25:44,840 --> 00:25:48,200 even misleading comments that are not up to date with the code 472 00:25:48,840 --> 00:25:52,200 variable names that sometimes are cryptic and sometimes are 473 00:25:52,200 --> 00:25:56,280 even confusing and misleading. AI makes mistakes. 474 00:25:56,920 --> 00:26:00,360 And I've tested it thoroughly, OK, here at SWIM, we also create 475 00:26:00,360 --> 00:26:02,880 documentation automatically from code bases. 476 00:26:03,320 --> 00:26:06,920 And one of the first hypothesis was, OK, let's just use an LLM 477 00:26:07,480 --> 00:26:10,000 and let's try that. And we tried a lot, OK? 478 00:26:10,280 --> 00:26:13,560 And it's made us understand it's an amazing tool, but it has some 479 00:26:13,560 --> 00:26:18,160 very fundamental limitations. And that's where you need other 480 00:26:18,160 --> 00:26:21,280 techniques like static code analysis and other things. 481 00:26:21,280 --> 00:26:25,000 You can do that. When you join with AI, you get 482 00:26:25,200 --> 00:26:28,960 clear coherent documents or other forms or written knowledge 483 00:26:29,600 --> 00:26:34,000 that explain in a way that is useful for humans to understand. 484 00:26:34,960 --> 00:26:38,520 Yeah, I think so for some people who have used AI coding 485 00:26:38,520 --> 00:26:40,920 assistance a lot, right? Especially working with like a 486 00:26:40,920 --> 00:26:44,480 bigger code base, very complex, you know, written by so many 487 00:26:44,480 --> 00:26:46,120 developers. I think that's also one thing, 488 00:26:46,120 --> 00:26:47,880 right? Because you can see the amount 489 00:26:47,880 --> 00:26:51,800 of inconsistency or like not so coherent kind of a code from one 490 00:26:51,800 --> 00:26:53,640 as one module to the others, right? 491 00:26:54,000 --> 00:26:57,200 And variable naming as well as some people like to use a 492 00:26:57,200 --> 00:27:00,440 certain terms, the others use other terms and they can be 493 00:27:00,440 --> 00:27:02,040 duplicates but mean different things. 494 00:27:02,240 --> 00:27:05,040 So I think that there's really a big challenge here if you just 495 00:27:05,040 --> 00:27:07,560 rely on LLM, right? Because LLM will just take it 496 00:27:08,080 --> 00:27:10,080 words by words. And I think you mentioned a very 497 00:27:10,080 --> 00:27:13,080 good point about combining it with, you know, maybe like 498 00:27:13,080 --> 00:27:16,080 static code analysis or other kind of, I don't know, like 499 00:27:16,080 --> 00:27:17,800 compiler's ability or something like that. 500 00:27:17,800 --> 00:27:19,840 Because computers are also good at that, right? 501 00:27:20,000 --> 00:27:22,640 Not AI, right? Maybe tell us how do you 502 00:27:22,640 --> 00:27:26,160 actually combine these results at like for example any specific 503 00:27:26,440 --> 00:27:30,440 study or maybe a customer case that you have solved using these 504 00:27:30,440 --> 00:27:34,040 kind of techniques? I think the most extreme cases 505 00:27:34,040 --> 00:27:37,400 we had in that regard and it's also our focus now is actually 506 00:27:37,400 --> 00:27:41,400 legacy code bases and more specifically mainframe legacy 507 00:27:41,400 --> 00:27:47,440 code bases with COBOL, which is a language that I had never run 508 00:27:47,440 --> 00:27:51,920 into before I started working on this problem for those client. 509 00:27:52,360 --> 00:27:55,920 And since then we doubled down on COBOL, right. 510 00:27:56,440 --> 00:28:01,280 But it's a language where most of the code is not available 511 00:28:01,280 --> 00:28:04,360 online or there is almost no real Kobo code available on 512 00:28:04,440 --> 00:28:06,920 online. The code that you have on GitHub 513 00:28:07,680 --> 00:28:09,640 doesn't look like the code that companies run. 514 00:28:10,200 --> 00:28:12,880 And it stems from multiple reasons, but I think the most 515 00:28:12,880 --> 00:28:16,480 important one is that when GitHub launched, no one was 516 00:28:16,480 --> 00:28:20,360 writing Kobo in an organization that wanted to publish their 517 00:28:20,360 --> 00:28:23,120 code, right? Let's say you start a new 518 00:28:23,640 --> 00:28:26,920 project today and you work even not with an AI assistant. 519 00:28:27,440 --> 00:28:31,120 You start with lots of libraries, you start with 520 00:28:31,120 --> 00:28:33,880 Python, you have your libraries for Python And frameworks, You 521 00:28:33,880 --> 00:28:35,520 start with JavaScript, whatever, right? 522 00:28:36,000 --> 00:28:40,280 You have lots of frameworks. And those frameworks are built 523 00:28:40,280 --> 00:28:45,800 on open source. When people developed big code 524 00:28:45,800 --> 00:28:49,840 bakers in Cobo in the 70s, in the 80s, they didn't have 525 00:28:49,840 --> 00:28:52,160 libraries. They had to re imagine 526 00:28:52,160 --> 00:28:54,240 everything themselves. So every organization looked 527 00:28:54,240 --> 00:28:59,040 very, very, very different. So LLMS don't have access to 528 00:28:59,040 --> 00:29:05,720 real world Kobo code and LLMS don't have access to the 529 00:29:05,920 --> 00:29:08,960 specific code of your organization. 530 00:29:09,920 --> 00:29:15,240 And in Kobo you have cryptic variable names all the time and 531 00:29:15,320 --> 00:29:18,440 the structure is different than other code languages. 532 00:29:19,120 --> 00:29:21,960 So it was I think the most extreme we saw. 533 00:29:22,000 --> 00:29:25,840 And we clients try to send some code to an LLM and ask, OK, what 534 00:29:25,840 --> 00:29:28,760 does he do? You get super generic and 535 00:29:28,760 --> 00:29:31,360 confusing and wrong results. But of the times. 536 00:29:32,120 --> 00:29:36,600 And there what we did was we wrote a COBOL parser that 537 00:29:36,600 --> 00:29:43,600 actually takes the code, parses it syntactically, and connects 538 00:29:43,680 --> 00:29:46,720 the dots together in a way that makes sense for an LLM. 539 00:29:46,720 --> 00:29:50,720 So in the end, for example, we have a variable and we want to 540 00:29:50,720 --> 00:29:53,560 understand what it does. We take the variable name. 541 00:29:54,440 --> 00:29:57,200 The variable name can be reused in lots of different places in 542 00:29:57,200 --> 00:30:00,000 the code base. So we find only the occurrences 543 00:30:00,000 --> 00:30:02,320 that are related to this occurrence of the variable. 544 00:30:02,920 --> 00:30:05,680 And then we send all of that context to an LLM and we ask for 545 00:30:05,680 --> 00:30:09,960 a summary about this variable. So when the LLM gets the right 546 00:30:10,200 --> 00:30:12,680 context and only the right context, it does a great job at 547 00:30:12,680 --> 00:30:14,120 explaining it in natural language. 548 00:30:14,640 --> 00:30:17,400 If you just throw the old code base at it, it starts combining 549 00:30:17,400 --> 00:30:19,320 different variables who have the same name. 550 00:30:19,320 --> 00:30:22,800 It happens a lot in Koble and you know, I'm not blaming it. 551 00:30:22,840 --> 00:30:26,040 It's really hard to parse all that code and understand what 552 00:30:26,040 --> 00:30:27,640 really belongs together and what doesn't. 553 00:30:28,520 --> 00:30:32,320 So what we do is first analyze the code base statically. 554 00:30:32,360 --> 00:30:35,600 We build our own internal representation of the code, how 555 00:30:35,600 --> 00:30:40,120 things relate, what function calls other functions, what 556 00:30:40,120 --> 00:30:44,480 variables are used, where hierarchy of say a flow. 557 00:30:45,000 --> 00:30:50,160 And then we slowly build the knowledge by sending small bits 558 00:30:50,160 --> 00:30:53,240 to an LLM to explain. After we do a lot of work on 559 00:30:53,240 --> 00:30:56,960 cleaning everything. And this is an example I think 560 00:30:56,960 --> 00:31:01,840 of combining static analysis or code that analyzes code in a 561 00:31:01,840 --> 00:31:05,160 deterministic way, not something probabilistic that an LLM 562 00:31:05,520 --> 00:31:08,880 produces right? And then using an LLM for what 563 00:31:08,880 --> 00:31:13,760 it does best, which is taking text or specific parts of code 564 00:31:14,200 --> 00:31:17,480 and explaining it in natural, coherent language. 565 00:31:18,320 --> 00:31:22,120 At the last phase, we also asked the LLM to generate parts of the 566 00:31:22,120 --> 00:31:25,800 documents we show to the user, because again, the LM is great 567 00:31:25,800 --> 00:31:28,280 at formulating that in coherent English. 568 00:31:28,280 --> 00:31:30,840 That is clear. That explains the story after it 569 00:31:30,840 --> 00:31:33,480 already has all the context that we build bit by bit. 570 00:31:34,480 --> 00:31:35,800 Wow. I think it's a very novel 571 00:31:35,800 --> 00:31:38,080 approach, so to speak, right? And especially you dealt with 572 00:31:38,080 --> 00:31:42,280 the most extreme code base available, I think COBOL, I 573 00:31:42,280 --> 00:31:44,960 think I also didn't have experience with COBOL, right? 574 00:31:45,120 --> 00:31:48,040 I can only imagine like the difficulty dealing with such 575 00:31:48,040 --> 00:31:50,600 legacy code base. And I think you brought up like 576 00:31:50,600 --> 00:31:54,120 a very good realization for people because I'm sure a lot of 577 00:31:54,120 --> 00:31:56,840 tech leaders or senior executives think, OK, now we 578 00:31:56,840 --> 00:32:00,240 have AI, any kind of code base it can understand and explain to 579 00:32:00,240 --> 00:32:02,720 us. So we probably don't need to be 580 00:32:02,720 --> 00:32:05,600 concerned so much about losing the ability of understanding the 581 00:32:05,600 --> 00:32:07,720 code base. And we can even probably hire 582 00:32:07,720 --> 00:32:12,320 some maybe less good developers and just use AI to fix all the 583 00:32:12,320 --> 00:32:14,960 problems we have. I think maybe for simple cases, 584 00:32:15,280 --> 00:32:18,160 maybe more up to date libraries and programming languages, you 585 00:32:18,160 --> 00:32:20,920 can do that. But if you look back, we have 586 00:32:20,920 --> 00:32:23,760 some, so many legacy systems, right, written so many years 587 00:32:23,760 --> 00:32:27,160 ago, where with the people also leaving the business, knowledge 588 00:32:27,360 --> 00:32:28,960 probably is also changing a lot, right? 589 00:32:29,160 --> 00:32:32,080 So I think this is 1 task that probably AI would not be able to 590 00:32:32,080 --> 00:32:33,880 do. And maybe combining different 591 00:32:33,880 --> 00:32:36,880 kind of approach like static code analysis would make AI 592 00:32:36,880 --> 00:32:40,120 works much better. So maybe in terms of knowledge 593 00:32:40,200 --> 00:32:43,560 base, right, with the ability of AI these days, any kind of 594 00:32:43,560 --> 00:32:46,840 medium size or maybe large organizations, what will be your 595 00:32:46,840 --> 00:32:51,000 advice of doing some practices or maybe cultural things that 596 00:32:51,000 --> 00:32:54,280 actually can fit into AI and help you know, the knowledge 597 00:32:54,320 --> 00:32:57,600 sharing aspect or knowledge base aspect becomes much more 598 00:32:57,600 --> 00:33:00,640 effective and maybe even like a multiplier effect within the 599 00:33:00,640 --> 00:33:02,640 organization, right? Any kind of practice and 600 00:33:02,960 --> 00:33:04,400 cultural things that you can share? 601 00:33:05,360 --> 00:33:07,720 Yeah. So I think the first thing is to 602 00:33:07,720 --> 00:33:10,440 acknowledge that it's really, really important and for all the 603 00:33:10,440 --> 00:33:15,160 reasons we said before, right? Especially now AI will read the 604 00:33:15,160 --> 00:33:18,080 knowledge base, right? So you want to invest in it and 605 00:33:18,080 --> 00:33:21,640 it's going to be a multiplier effect, as you said, Henry, I 606 00:33:21,640 --> 00:33:26,240 think specifically you want to create a culture as a leader 607 00:33:26,240 --> 00:33:29,640 that values people who capture knowledge. 608 00:33:30,320 --> 00:33:34,440 Some organizations used to look at these people as wasting their 609 00:33:34,440 --> 00:33:35,960 time. We're doing the easy task, but 610 00:33:35,960 --> 00:33:38,120 just explain what the reason instead of creating. 611 00:33:38,800 --> 00:33:40,840 I don't think it's a viable argument anymore. 612 00:33:41,040 --> 00:33:43,360 I didn't think it was a viable argument back then, but let's 613 00:33:43,360 --> 00:33:46,160 say it's arguable. I think now it's not a viable 614 00:33:46,160 --> 00:33:49,800 argument at all. It's clear that if you document 615 00:33:49,800 --> 00:33:53,840 what's happening, it will help AI accomplish anything 616 00:33:53,840 --> 00:33:56,040 afterwards. The other thing is that you 617 00:33:56,240 --> 00:34:01,400 should put effort into finding the tools that will help you 618 00:34:01,800 --> 00:34:05,880 with the task of writing this comprehensive knowledge base 619 00:34:06,200 --> 00:34:09,320 documents with keeping it up to date as the code evolves, 620 00:34:09,600 --> 00:34:12,280 because otherwise you just have misleading and wrong 621 00:34:12,280 --> 00:34:16,000 information. And 3rd, to be able to find that 622 00:34:16,000 --> 00:34:18,639 information again, both humans and AI should be able to find 623 00:34:18,639 --> 00:34:20,120 the information they need when they need it. 624 00:34:20,639 --> 00:34:23,639 So you should invest in those tools to help you with that task 625 00:34:23,719 --> 00:34:27,800 and also create a culture that values the knowledge creation, 626 00:34:27,800 --> 00:34:32,480 preservation and sharing. Well, I like the the emphasis 627 00:34:32,480 --> 00:34:36,159 you put into like valuing people who actually do the so-called. 628 00:34:36,360 --> 00:34:39,000 I would say it's a hard job actually to actually capture 629 00:34:39,000 --> 00:34:41,600 knowledge, distillate, summarize it for other people to 630 00:34:41,600 --> 00:34:44,199 understand. It's actually not an easy job. 631 00:34:44,239 --> 00:34:47,360 I would say it's maybe becoming more valuable now because you 632 00:34:47,360 --> 00:34:51,520 can fit it into AI as a context and everyone can benefit just by 633 00:34:51,520 --> 00:34:53,840 one simple writing, right? And maybe it can be reused 634 00:34:53,840 --> 00:34:56,239 multiple times. So I think the other challenge 635 00:34:56,239 --> 00:34:58,880 about, you know, knowledge based documentation and all that, 636 00:34:58,880 --> 00:35:01,880 keeping it up to date, right? Any documentation that you have 637 00:35:01,880 --> 00:35:05,480 within an organization, I'm sure most of them are still not up to 638 00:35:05,480 --> 00:35:08,080 date, maybe even wrong when you read it again. 639 00:35:08,480 --> 00:35:11,400 So tell us maybe some good practice that we can do to 640 00:35:11,400 --> 00:35:13,680 actually make it up to date. Right. 641 00:35:13,680 --> 00:35:16,720 So I think there are two kind of ways to approach it. 642 00:35:17,520 --> 00:35:21,560 And we at Swim spent a lot of time working on this specific 643 00:35:21,560 --> 00:35:25,200 problem, so very emotionally attached to it I would say. 644 00:35:25,560 --> 00:35:28,360 I would say that nowadays there are basically 2 approaches. 1 is 645 00:35:28,360 --> 00:35:30,760 regenerate the documentation every time you generate 646 00:35:30,760 --> 00:35:33,800 documentation automatically just regenerate it all the time. 647 00:35:34,280 --> 00:35:38,960 If you regenerate now, it will match the codes state right now. 648 00:35:40,000 --> 00:35:42,640 I think it makes sense. In some cases, for example for 649 00:35:42,640 --> 00:35:46,200 API documentation, it could make sense, but what you're going to 650 00:35:46,200 --> 00:35:49,600 lose if you regenerate every time is additional context that 651 00:35:49,600 --> 00:35:52,920 is not there in the code, and you need a way to preserve that, 652 00:35:52,920 --> 00:35:56,040 because that could be the most important piece of knowledge 653 00:35:56,040 --> 00:35:59,320 that is written there. So another approach is to 654 00:35:59,320 --> 00:36:03,320 somehow track the changes made to the code that is referenced 655 00:36:03,360 --> 00:36:07,160 in specific documents and then update those parts of the 656 00:36:07,160 --> 00:36:10,400 documents based on the changes and maybe ask for a human 657 00:36:10,400 --> 00:36:14,120 intervention in case the code change drastically, for example. 658 00:36:14,560 --> 00:36:17,480 So this is actually something we provide with SWIM. 659 00:36:17,520 --> 00:36:21,880 When you create a document with SWIM, we track the changes made 660 00:36:22,040 --> 00:36:25,640 to the code you relied on in the document and we either 661 00:36:25,640 --> 00:36:28,240 automatically update the document or if the change is 662 00:36:28,240 --> 00:36:32,120 drastic, we tell you as a human, please decide what you want to 663 00:36:32,120 --> 00:36:34,680 do from now. If you want to reselect this 664 00:36:34,680 --> 00:36:37,040 part of the code, if you want to rewrite it, maybe this part is 665 00:36:37,040 --> 00:36:40,160 no longer relevant, maybe you need to add some unique 666 00:36:40,160 --> 00:36:42,240 information. But the goal here is to 667 00:36:42,240 --> 00:36:45,640 understand that a lot of the unique knowledge that only 668 00:36:45,640 --> 00:36:49,240 developers have in their minds is what you need to work so hard 669 00:36:49,240 --> 00:36:52,480 to preserve. Therefore you can just rely on 670 00:36:52,480 --> 00:36:55,880 AI generating documents. Yeah. 671 00:36:56,240 --> 00:36:59,800 So I think looking in the past, right, when I have difficulties 672 00:36:59,800 --> 00:37:02,320 finding knowledge, right, maybe sometimes the knowledge is 673 00:37:02,320 --> 00:37:04,320 there, but we just don't know where to find, right? 674 00:37:04,520 --> 00:37:06,680 I think that's the one thing and keeping it up to date, right? 675 00:37:06,680 --> 00:37:09,600 Because sometimes I find this piece of documentation, I read 676 00:37:09,600 --> 00:37:12,760 it, well, if we assume it's correct, but it's wrong, right? 677 00:37:12,760 --> 00:37:14,000 It's also quite dangerous, right? 678 00:37:14,280 --> 00:37:16,520 And I think we have also multiple tools within the 679 00:37:16,520 --> 00:37:18,440 organization, which is in like silo, right? 680 00:37:18,440 --> 00:37:21,040 So for example, some information maybe in our tracking, 681 00:37:21,120 --> 00:37:22,480 ticketing, tracking system, right? 682 00:37:22,840 --> 00:37:26,720 Maybe let's say Nigeria, some Confluence, some in Slack, some 683 00:37:26,720 --> 00:37:29,640 in e-mail. How do you actually build these 684 00:37:29,640 --> 00:37:33,800 kind of linkages, references and again like probing people to 685 00:37:33,800 --> 00:37:36,640 actually, hey, these parts of the knowledge base is not up to 686 00:37:36,640 --> 00:37:38,280 date. I think this is like a real 687 00:37:38,280 --> 00:37:40,280 world challenge if we can solve it right. 688 00:37:41,320 --> 00:37:46,560 Actually, AI has a great promise in the sense of accumulating all 689 00:37:46,560 --> 00:37:50,240 of this information from across the organization, right from, 690 00:37:50,560 --> 00:37:54,360 say, Jira, Slack, documentation tools and others. 691 00:37:54,840 --> 00:37:59,440 And I think it's suddenly possible to just ask a question 692 00:37:59,440 --> 00:38:01,480 and get a response from various resources. 693 00:38:01,960 --> 00:38:05,480 The key here is to understand that some of these resources are 694 00:38:05,640 --> 00:38:10,960 more historic references than provide actual up to date 695 00:38:10,960 --> 00:38:13,240 information, which is also sometimes valuable, right? 696 00:38:13,240 --> 00:38:17,920 Like the Jira ticket can tell you what a product manager 697 00:38:18,000 --> 00:38:20,800 wanted you to accomplish at some point, right? 698 00:38:21,360 --> 00:38:23,760 At least most of the times it won't tell you what's actually 699 00:38:23,760 --> 00:38:26,000 happening right now, right? But you have a code 700 00:38:26,000 --> 00:38:29,760 documentation software that actually explains what happens 701 00:38:29,760 --> 00:38:32,160 in the code and keeps it up to date, then you can relate on it. 702 00:38:32,680 --> 00:38:37,200 So I think for AI coding assistance or AI tools that help 703 00:38:37,200 --> 00:38:40,840 you find information from across your organization, they should 704 00:38:41,000 --> 00:38:45,040 always explain what resources they're using to formulate the 705 00:38:45,040 --> 00:38:50,440 responses and perhaps mark some of those messages or snippets of 706 00:38:50,440 --> 00:38:54,400 knowledge as how likely they are to be up to date. 707 00:38:55,240 --> 00:38:58,200 Yeah. So I think it's very challenging 708 00:38:58,200 --> 00:38:59,920 right in the 1st place accumulating. 709 00:38:59,920 --> 00:39:03,560 So I think what provided that we can give AI the tools capability 710 00:39:03,560 --> 00:39:06,440 to maybe like, I don't know like crawl our knowledge base and you 711 00:39:06,440 --> 00:39:08,360 know, get the contacts and all that. 712 00:39:08,720 --> 00:39:12,200 I think we we all know one danger of AILLM is actually the 713 00:39:12,200 --> 00:39:14,240 hallucination part, right? You mentioned about providing 714 00:39:14,240 --> 00:39:17,680 references and all that, but assuming that we have like large 715 00:39:17,680 --> 00:39:21,120 knowledge base, how do you actually ensure that it is not 716 00:39:21,120 --> 00:39:23,080 hallucinating? Because sometimes the 717 00:39:23,080 --> 00:39:26,280 hallucination could happen in a very small part of the summary 718 00:39:26,280 --> 00:39:28,840 that it generates, Especially now these days, we have like a 719 00:39:28,840 --> 00:39:33,280 deep research tool where it can do on its own in hours, whatever 720 00:39:33,280 --> 00:39:34,840 that is and provide you a summary. 721 00:39:35,280 --> 00:39:37,760 But the always the challenge is like, how do you Fact Check it, 722 00:39:37,760 --> 00:39:39,040 right? How do you know which part is 723 00:39:39,040 --> 00:39:40,800 hallucinating? Or maybe they provide 724 00:39:40,800 --> 00:39:42,720 statistics. How do you know it's actually 725 00:39:42,720 --> 00:39:45,080 correct statistics? So do you have any experience 726 00:39:45,080 --> 00:39:47,920 in, you know, doing this fact checking and preventing 727 00:39:48,080 --> 00:39:51,000 hallucination to actually make your decisions wrong? 728 00:39:51,800 --> 00:39:54,000 Yeah. So we worked a lot on it when 729 00:39:54,200 --> 00:39:57,240 working on SWIM, when we generate documents to make sure 730 00:39:57,240 --> 00:40:01,000 they reflect the accurate state of the code. 731 00:40:01,480 --> 00:40:03,000 And for that we do lots of things. 732 00:40:03,000 --> 00:40:05,680 But I think the most interesting part in terms of what the end 733 00:40:05,680 --> 00:40:10,160 user can get from it is that when we generate the documents, 734 00:40:10,160 --> 00:40:13,760 you see for everything we write what we relied on to provide 735 00:40:13,760 --> 00:40:16,480 that information. We show you that it's grounded 736 00:40:16,480 --> 00:40:19,080 in this part of the code or that part of the document. 737 00:40:19,760 --> 00:40:21,520 And you can validate it yourself. 738 00:40:22,040 --> 00:40:25,280 And in addition to, of course, making sure we do everything we 739 00:40:25,280 --> 00:40:28,920 can to avoid hallucinations by all kinds of techniques that are 740 00:40:28,920 --> 00:40:32,080 available to eliminate or at least decrease hallucinations. 741 00:40:32,560 --> 00:40:35,320 It can always happen, right? There is some hallucination. 742 00:40:35,320 --> 00:40:39,040 And as the end user, you can never tell if the LM made a 743 00:40:39,040 --> 00:40:42,280 mistake or maybe the other providers made a mistake. 744 00:40:42,720 --> 00:40:47,000 So I think you should trust tools that show you what they 745 00:40:47,000 --> 00:40:50,240 relied on. And if you want to incorporate a 746 00:40:50,240 --> 00:40:52,720 tool into your organization that gives you answers based on 747 00:40:52,720 --> 00:40:57,600 questions, based on your own knowledge base, you should 748 00:40:57,680 --> 00:41:02,000 enforce the fact that they give you resources or citations to 749 00:41:02,000 --> 00:41:06,440 everything they output. Yeah, I think it's pretty 750 00:41:06,440 --> 00:41:09,640 dangerous if you do not have the citations, you know the 751 00:41:09,640 --> 00:41:12,400 references, right? And even if these days you do 752 00:41:12,400 --> 00:41:15,120 have the citation, sometimes I from my experience, right, they 753 00:41:15,120 --> 00:41:17,960 give you the citation, but sometimes the summary itself can 754 00:41:17,960 --> 00:41:20,640 still hallucinate a little bit. So I think that's very 755 00:41:20,640 --> 00:41:24,160 interesting experience as well. Yeah, I agree. 756 00:41:24,280 --> 00:41:27,840 The the other thing these days, people are crazy talking about, 757 00:41:27,840 --> 00:41:31,120 you know, this agentic capability of AI and also maybe 758 00:41:31,120 --> 00:41:34,480 the MCP protocol, right? Maybe tell us the next 759 00:41:34,480 --> 00:41:37,200 evolution. Do you think you can see using 760 00:41:37,200 --> 00:41:38,680 AI? You know, with all these cool 761 00:41:38,680 --> 00:41:41,880 things, and especially in the context of documentation and all 762 00:41:41,880 --> 00:41:44,960 the sharing, is there anything that you can see up and coming? 763 00:41:45,680 --> 00:41:50,920 Well, I think MCP will be a game changer in the sense that you 764 00:41:50,920 --> 00:41:53,360 will see lots of information being fed all the time. 765 00:41:53,680 --> 00:41:58,040 And I think the IT will help create a flying wheel effect 766 00:41:58,040 --> 00:42:02,880 where when you put the effort into generating valuable 767 00:42:03,320 --> 00:42:07,040 documents or a knowledge base, then all of the AI assistant 768 00:42:07,480 --> 00:42:12,080 would be able to reach it, find the relevant information and 769 00:42:12,080 --> 00:42:14,080 make use of it. And then we would get some mind 770 00:42:14,080 --> 00:42:17,440 blowing things, right? You can have an AI assistant 771 00:42:17,440 --> 00:42:21,480 that analyzes your Jira tickets and provides a summary and all 772 00:42:21,480 --> 00:42:23,840 of a sudden it knows what's happening in the code base and 773 00:42:23,840 --> 00:42:27,360 it gets the broader contact from a document that was written 774 00:42:27,360 --> 00:42:29,920 partially by AI and partially by a human. 775 00:42:29,920 --> 00:42:32,680 And I think that's what we want to get, right. 776 00:42:33,240 --> 00:42:36,160 But to get there, we need to make sure we create these 777 00:42:36,160 --> 00:42:40,640 explicit knowledge fragments alongside the way and also that 778 00:42:40,640 --> 00:42:44,760 we rely on them. So we guide say the AI coding 779 00:42:44,760 --> 00:42:47,520 assistant to rely on specific resources. 780 00:42:48,520 --> 00:42:51,080 Yeah, I think that providing all these bits of information, 781 00:42:51,080 --> 00:42:53,400 again, coming back to what you said, right, the value of 782 00:42:53,400 --> 00:42:55,760 writing or you know, capturing the knowledge I think will 783 00:42:55,760 --> 00:42:58,160 become a key. And then the next part is 784 00:42:58,160 --> 00:43:01,280 actually to expose that, right? Maybe in a genetic manner, 785 00:43:01,280 --> 00:43:03,240 right, using this MCP protocol, right? 786 00:43:03,440 --> 00:43:05,960 I'm actually really excited about this MCP capability, 787 00:43:05,960 --> 00:43:07,640 especially when doing coding, right? 788 00:43:07,840 --> 00:43:11,040 You can communicate with different tools and ask it to do 789 00:43:11,040 --> 00:43:12,760 certain things just by natural language. 790 00:43:13,080 --> 00:43:14,840 That can be really super powerful, right? 791 00:43:15,120 --> 00:43:19,120 And I think I can be certain that once we see more and more 792 00:43:19,200 --> 00:43:21,680 agentic capability may be provided by different companies 793 00:43:21,680 --> 00:43:24,240 and tools, we can see this multiplier effect. 794 00:43:24,320 --> 00:43:27,080 Although the danger, it would be even worse, right? 795 00:43:27,080 --> 00:43:30,280 Because if let's say some of these agents hallucinate and 796 00:43:30,280 --> 00:43:33,120 they all hallucinate with each other, like we probably lose 797 00:43:33,120 --> 00:43:35,680 track of what kind of things they used to deduce the 798 00:43:35,680 --> 00:43:37,680 decision, right? Any take on this from you? 799 00:43:38,440 --> 00:43:43,360 Now, just this week I saw a friend of mine posted that he 800 00:43:43,360 --> 00:43:49,520 used Claude and I think it was cursor and cursor ran RM minus 801 00:43:49,520 --> 00:43:53,280 RF and deleted lots of his valuable information by mistake, 802 00:43:53,360 --> 00:43:56,200 of course. And you know, it's, it's like a 803 00:43:56,200 --> 00:43:58,280 funny example, but those things will happen. 804 00:43:58,680 --> 00:44:01,160 So I think if we go back to the beginning of our discussion 805 00:44:01,160 --> 00:44:04,320 about junior developers taking code that they don't fully 806 00:44:04,320 --> 00:44:07,400 understand and committing it to the code base, right, If you 807 00:44:07,400 --> 00:44:12,600 have an agent doing that, you have to somehow constraint it 808 00:44:13,360 --> 00:44:16,600 and validate the output that it generates. 809 00:44:16,960 --> 00:44:20,080 And I think of the next step this week with an announcement 810 00:44:20,080 --> 00:44:23,480 of A to a like agent to agent. So we're talking about new 811 00:44:23,480 --> 00:44:26,080 protocols for agents communicating with other agents. 812 00:44:26,080 --> 00:44:29,520 And at some point, it's going to be hard for a human to 813 00:44:29,520 --> 00:44:32,720 understand what's going on. And that's where I think we'll 814 00:44:32,720 --> 00:44:36,360 have to stop and think, right, like what's actually happening 815 00:44:36,360 --> 00:44:38,360 here? Where should we have the human 816 00:44:38,360 --> 00:44:39,560 in the loop? And we're not. 817 00:44:39,560 --> 00:44:43,320 I think it's going to be really exciting times in that sense. 818 00:44:44,360 --> 00:44:47,520 Yeah, I think stop and think will be a point in time where we 819 00:44:47,520 --> 00:44:50,880 all realize, OK, we probably hallucinate ourselves thinking, 820 00:44:51,040 --> 00:44:52,360 yeah, we'll solve a lot of problems. 821 00:44:53,360 --> 00:44:56,080 So yeah, probably one day we will have to build guardrails, 822 00:44:56,120 --> 00:44:59,120 you know, constraints such that AI won't lead us to like a 823 00:44:59,120 --> 00:45:01,560 dangerous, right. So I think these days people 824 00:45:01,560 --> 00:45:04,040 talk about AI. I'm sure every team, every 825 00:45:04,040 --> 00:45:07,320 organization also want to integrate AI somehow, build 826 00:45:07,320 --> 00:45:09,680 capability, you know, build something on top of AI model 827 00:45:09,680 --> 00:45:12,280 LLM, whatever that is. And for that, they need to do 828 00:45:12,280 --> 00:45:14,880 some kind of research, right? Some companies have their 829 00:45:14,880 --> 00:45:17,360 capabilities, but most of the companies, they don't have this 830 00:45:17,400 --> 00:45:20,640 knowledge and capability, right? And doing research is partly 831 00:45:20,640 --> 00:45:22,600 something that, you know, some organization find it 832 00:45:22,760 --> 00:45:25,240 challenging, maybe finding the time, finding the resource. 833 00:45:25,480 --> 00:45:27,960 And you brought up a good point before our discussion thing 834 00:45:27,960 --> 00:45:31,280 here, but you know, as a product company or maybe it's like a 835 00:45:31,280 --> 00:45:34,120 business organization, if you want to do research, what's the 836 00:45:34,120 --> 00:45:36,480 best way to approach it, right? Maybe you can share a little bit 837 00:45:36,680 --> 00:45:39,520 so that people who want to build capability by doing research can 838 00:45:39,520 --> 00:45:41,800 do it more effectively. Sure. 839 00:45:41,880 --> 00:45:45,760 So I think first of all, we need to kind of define what research 840 00:45:45,760 --> 00:45:48,240 is. Engineering organizations are 841 00:45:48,240 --> 00:45:50,960 usually called R&D, right? So it's research and development 842 00:45:50,960 --> 00:45:52,480 and the research piece comes first. 843 00:45:52,880 --> 00:45:56,160 But I think in most teams, there is no pure research. 844 00:45:56,640 --> 00:45:59,760 And that's fine, right? Usually have the research as in 845 00:45:59,760 --> 00:46:03,600 like problem solving and you need to find the best way to do 846 00:46:03,600 --> 00:46:05,520 something and you need to learn alongside the way. 847 00:46:05,560 --> 00:46:07,320 That's fine. That's all fine, but it's part 848 00:46:07,320 --> 00:46:11,680 of development, right? And I think where I draw the 849 00:46:11,680 --> 00:46:18,920 line is if you know the task is achievable and you know the 850 00:46:18,920 --> 00:46:22,400 approach, the right approach to get there, then it's 851 00:46:22,400 --> 00:46:24,440 development. It's research. 852 00:46:24,440 --> 00:46:28,480 When you have a task, you're not sure if it's possible or you're 853 00:46:28,480 --> 00:46:30,840 sure it's possible, but you really don't know how to get 854 00:46:30,840 --> 00:46:33,160 there because there are so many different options and it's 855 00:46:33,160 --> 00:46:35,680 unclear. That's where it's research. 856 00:46:36,240 --> 00:46:40,840 So for example, if I have to, it could be even a really hard 857 00:46:40,840 --> 00:46:42,320 development task, of course, right? 858 00:46:42,320 --> 00:46:45,840 Let's say I would want to implement, I don't know, VS Code 859 00:46:45,840 --> 00:46:48,400 from scratch, right? There are lots of things I don't 860 00:46:48,400 --> 00:46:50,440 know I would need to learn along the way. 861 00:46:50,440 --> 00:46:52,960 I would need to design the architecture. 862 00:46:53,120 --> 00:46:55,440 I'll have to work hard on it, right? 863 00:46:55,440 --> 00:46:57,920 But it's all development. I know it's I know what the 864 00:46:57,920 --> 00:47:02,240 output looks like and I know that it would involve a lot of 865 00:47:02,240 --> 00:47:05,840 engineering, whereas research and say I have a Cobalt code 866 00:47:05,840 --> 00:47:08,240 based and I need to generate useful documents. 867 00:47:08,440 --> 00:47:10,640 I'm not even sure what documents are useful at first, right? 868 00:47:10,640 --> 00:47:13,320 I need to learn that and then I need to find different ways. 869 00:47:13,360 --> 00:47:15,640 Should I go with generative AI all the way? 870 00:47:15,640 --> 00:47:17,000 Should I go with static analysis? 871 00:47:17,000 --> 00:47:19,600 Combine them where? And that's more for research. 872 00:47:20,040 --> 00:47:25,480 So I think when you want to work on a product and research is 873 00:47:25,480 --> 00:47:27,720 something that is blocking, for example, I don't know if it's 874 00:47:27,720 --> 00:47:32,720 possible to achieve this, right? And the the difference between 875 00:47:32,720 --> 00:47:37,000 development in the research is time estimation in development. 876 00:47:37,080 --> 00:47:41,280 I mean, it's notoriously hard to give real time estimations. 877 00:47:41,280 --> 00:47:45,400 But usually when developers say this will take me a week, it 878 00:47:45,400 --> 00:47:49,040 won't take a year, right? Usually like there is something 879 00:47:49,040 --> 00:47:50,880 between the estimate and how long it takes. 880 00:47:51,400 --> 00:47:54,280 With research, you can sometimes just don't know, right? 881 00:47:54,280 --> 00:47:56,400 Like, I don't know, maybe it will take me two days because 882 00:47:56,400 --> 00:48:00,840 there is an easy win and maybe I'll get into a wall which would 883 00:48:00,840 --> 00:48:03,800 be much harder to pass. So I think it's something to 884 00:48:03,800 --> 00:48:06,640 acknowledge. And when you have a research 885 00:48:06,840 --> 00:48:10,000 that is guided by a product, it means you have some problem you 886 00:48:10,000 --> 00:48:12,960 want to solve and you need to do a few things. 887 00:48:13,800 --> 00:48:19,120 One is to lay the entire flow from beginning to end, even 888 00:48:19,120 --> 00:48:22,520 though you can't solve all the intermediate stages. 889 00:48:22,600 --> 00:48:26,600 For example, in the example, if I take a couple repository and I 890 00:48:26,600 --> 00:48:29,560 need to generate useful documents automatically, the 891 00:48:29,560 --> 00:48:33,040 first thing I would do is take a couple repository to play with 892 00:48:33,360 --> 00:48:38,160 and generate the documents by hand manually for myself and get 893 00:48:38,160 --> 00:48:40,480 feedback on them. Are these documents really 894 00:48:40,480 --> 00:48:42,680 valuable? Is this where I'm heading to? 895 00:48:42,920 --> 00:48:45,200 Right. Once I know that's what I want, 896 00:48:45,600 --> 00:48:49,440 say OK, what do I need to do? So I need to say parse the 897 00:48:49,520 --> 00:48:55,040 couple repository and I need to find say a few components, for 898 00:48:55,040 --> 00:48:57,800 example. OK, I don't know how to find the 899 00:48:57,800 --> 00:49:00,520 right components. So for now, I'll wrap that in a 900 00:49:00,520 --> 00:49:03,120 box and I'll keep going to the next step. 901 00:49:03,120 --> 00:49:05,960 Now that I know what components there are, how do I document the 902 00:49:05,960 --> 00:49:09,520 component? And I actually write that on a 903 00:49:09,520 --> 00:49:12,760 white board with boxes and I keep them closed. 904 00:49:13,200 --> 00:49:14,920 I don't want to open the boxes now. 905 00:49:14,920 --> 00:49:18,120 I want to make sure I can understand what the process 906 00:49:18,120 --> 00:49:22,720 would be and then before I open a box, because the most the 907 00:49:22,720 --> 00:49:25,920 thing that makes what you would want to do is open an 908 00:49:25,920 --> 00:49:29,640 interesting box, peek inside, try to solve it right? 909 00:49:29,800 --> 00:49:33,320 But it might be irrelevant. So you need to first make sure 910 00:49:33,320 --> 00:49:36,200 you can achieve everything. If everything works, you say, I 911 00:49:36,200 --> 00:49:38,080 assume all the boxes work. Will this work? 912 00:49:38,640 --> 00:49:41,720 Yes. OK, why don't I know now there 913 00:49:41,720 --> 00:49:43,800 is this specific box. I'm not sure it's possible. 914 00:49:43,800 --> 00:49:48,080 I don't know if an LLM can read a couple program and describe 915 00:49:48,080 --> 00:49:48,920 it. OK. 916 00:49:49,600 --> 00:49:53,680 And then the key thing, and this is I think the crucial thing 917 00:49:53,680 --> 00:49:58,000 when managing research is to deliberately pause and think 918 00:49:58,000 --> 00:50:01,200 about the different directions together, because every 919 00:50:01,200 --> 00:50:03,640 researcher will do the research themselves, right? 920 00:50:03,640 --> 00:50:06,960 Let's say you give someone a task of understanding what a 921 00:50:06,960 --> 00:50:09,920 program does. OK, they will read the code and 922 00:50:09,920 --> 00:50:12,840 try to understand what's happening, but you can help them 923 00:50:13,320 --> 00:50:15,560 by stopping and thinking what their best technique is. 924 00:50:15,560 --> 00:50:19,240 When I LED a cybersecurity course, we taught reverse 925 00:50:19,240 --> 00:50:23,480 engineering and one of the exercises we would give what we 926 00:50:23,480 --> 00:50:26,000 would give as part of this course, right? 927 00:50:26,000 --> 00:50:29,320 So the students would reverse engineer more and more 928 00:50:29,320 --> 00:50:31,520 applications and they, they would get a game. 929 00:50:31,840 --> 00:50:35,320 And the question was, what are the rules of the game? 930 00:50:35,960 --> 00:50:39,800 And after an hour, we would stop them and we would show how to 931 00:50:39,800 --> 00:50:43,520 approach it correctly, which is you open the game, you click on 932 00:50:43,520 --> 00:50:47,360 help explain, and you have a textual description of the 933 00:50:47,360 --> 00:50:50,840 instructions, right? And the lesson learned is you 934 00:50:50,840 --> 00:50:54,400 don't always have to reverse engineering by reading through 935 00:50:54,400 --> 00:50:57,720 the code, right? And what we wanted to teach them 936 00:50:57,720 --> 00:51:01,960 is that before you jump into one way of solving the problem, stop 937 00:51:01,960 --> 00:51:03,520 and consider different solutions. 938 00:51:03,920 --> 00:51:07,440 So what I usually do when I work with people on research tasks is 939 00:51:07,440 --> 00:51:09,840 draw it as kind of a tree, like, OK, we're here. 940 00:51:10,400 --> 00:51:13,040 How can we solve it? We have option 1-2 and three. 941 00:51:13,080 --> 00:51:16,080 OK, we don't have a time estimate because we don't know 942 00:51:16,080 --> 00:51:18,640 what we'll find out. If the LLM can just read an 943 00:51:18,640 --> 00:51:20,960 entire code base and give us great documentation, OK, we're 944 00:51:20,960 --> 00:51:23,360 done here. OK, let's give it a day and see 945 00:51:23,360 --> 00:51:26,360 what happens. And I usually call it time to 946 00:51:26,360 --> 00:51:28,960 leave. Like for how long are we going 947 00:51:28,960 --> 00:51:33,480 to work on this before we stop and re discuss what we found out 948 00:51:33,480 --> 00:51:36,360 and whether we should keep pursuing this specific direction 949 00:51:36,360 --> 00:51:38,680 or change to another direction of the research? 950 00:51:39,520 --> 00:51:42,960 And one of the most important things is to make everyone stop 951 00:51:42,960 --> 00:51:45,280 and think about the various ways to approach a problem. 952 00:51:45,840 --> 00:51:48,880 Sometimes the easy solution is there, but you need to think 953 00:51:48,880 --> 00:51:50,440 about it. Sometimes it's just clicking 954 00:51:50,440 --> 00:51:53,120 help and you have the solution. You don't have to read through 955 00:51:53,120 --> 00:51:55,760 the code, right? And they have many, many, many 956 00:51:55,760 --> 00:51:58,200 examples for that. Also not from courses, right? 957 00:51:58,200 --> 00:52:00,920 From real life where people, you know, in retrospect, they say, 958 00:52:00,920 --> 00:52:04,040 oh, right, we should have done this, right. 959 00:52:04,840 --> 00:52:07,160 And when you work on a product, you don't have all the time in 960 00:52:07,160 --> 00:52:10,720 the world to just do research. You have to make sure you can 961 00:52:10,720 --> 00:52:12,920 provide a product to a user in a timely manner. 962 00:52:13,520 --> 00:52:17,320 So to summarize, I think 1 crucial thing is to understand 963 00:52:17,320 --> 00:52:20,160 the time estimates is hard. What you can do is give it time 964 00:52:20,160 --> 00:52:22,320 to leave. How long am I willing to spend 965 00:52:22,320 --> 00:52:24,000 on it before I stop and re evaluate? 966 00:52:24,720 --> 00:52:28,320 The second thing is to make sure we get the end to end process 967 00:52:28,960 --> 00:52:34,080 from the input to the end output that the user sees or that the 968 00:52:34,360 --> 00:52:38,120 other product takes into account as an input, and so on. 969 00:52:38,960 --> 00:52:42,040 And the third part is pausing and thinking together about the 970 00:52:42,040 --> 00:52:44,760 different ways to approach a research task, because again, 971 00:52:44,760 --> 00:52:49,640 what characterizes research tasks is that it's unclear how 972 00:52:49,640 --> 00:52:51,800 to make progress. So we need to stop and think 973 00:52:51,800 --> 00:52:53,200 together on how to approach this. 974 00:52:54,480 --> 00:52:57,160 Things so many good Nuggets I would say, right, because doing 975 00:52:57,160 --> 00:52:59,880 research by itself is kind of like unpredictable, right So 976 00:52:59,880 --> 00:53:02,240 like you mentioned, right, you don't know the time estimation 977 00:53:02,240 --> 00:53:04,840 required, the effort required. Sometimes it could be easy, 978 00:53:04,840 --> 00:53:07,320 right if let's say some you know, one day you find oh, 979 00:53:07,320 --> 00:53:09,920 there's a library that you can use, but most of the times you 980 00:53:09,920 --> 00:53:12,040 know like you don't have the skills you have to gather a lot 981 00:53:12,040 --> 00:53:14,280 of knowledge, maybe ask expertise and things like that. 982 00:53:14,560 --> 00:53:16,680 So I think you have given some good things. 983 00:53:16,680 --> 00:53:19,000 I would just call out a few things that I could remember, 984 00:53:19,000 --> 00:53:21,320 right? So the first is try to workflows 985 00:53:21,320 --> 00:53:22,920 the direction that you're going into, right? 986 00:53:22,920 --> 00:53:26,480 Because sometimes we can go into rabbit hole easily, especially 987 00:53:26,480 --> 00:53:28,000 playing with technologies, right? 988 00:53:28,000 --> 00:53:30,080 So techies, we all love playing with technologies. 989 00:53:30,080 --> 00:53:32,760 So we keep digging and digging, but maybe we go to the wrong 990 00:53:32,760 --> 00:53:34,720 direction. The time to live, I think it's 991 00:53:34,720 --> 00:53:37,280 also very crucial, right? You can't spend all the time 992 00:53:37,280 --> 00:53:39,000 just doing research that goes nowhere. 993 00:53:39,440 --> 00:53:43,200 And I think many people find it difficult to juggle or even 994 00:53:43,200 --> 00:53:46,200 like, for example, justify the value of doing research. 995 00:53:46,520 --> 00:53:49,400 And we all know these days, especially again, like bringing 996 00:53:49,400 --> 00:53:51,520 up the point of the age of AI, right? 997 00:53:52,160 --> 00:53:55,400 If you don't do research about capability of AI that could help 998 00:53:55,400 --> 00:53:58,040 your business, maybe you will lose out in the next, I don't 999 00:53:58,040 --> 00:54:01,080 know, the time span is can be really short these days, right? 1000 00:54:01,560 --> 00:54:05,480 So how can maybe business leaders or maybe executives have 1001 00:54:05,480 --> 00:54:08,640 this in mind, right, to spend some time doing research even 1002 00:54:08,640 --> 00:54:11,320 though it's difficult, unpredictable, maybe cannot 1003 00:54:11,320 --> 00:54:13,880 justify the profits and the revenue coming out of the 1004 00:54:13,880 --> 00:54:17,640 research, Maybe can you give us some examples here, Right, so I 1005 00:54:17,640 --> 00:54:20,640 think. On a personal level, say if 1006 00:54:20,640 --> 00:54:24,680 you're a CTOI think one of your responsibilities is to know how 1007 00:54:24,680 --> 00:54:27,760 technology can empower your business, and it is also 1008 00:54:27,800 --> 00:54:31,200 internally right? So as a very recent example, 1009 00:54:31,400 --> 00:54:35,520 you're a CTO, you keep yourself informed with what's happening. 1010 00:54:35,520 --> 00:54:39,240 You know that there are great AI coding assistants that can help 1011 00:54:39,240 --> 00:54:43,760 people become more effective. You go to your engineering teams 1012 00:54:43,760 --> 00:54:48,720 and you introduce them to that tool and you help them adjust or 1013 00:54:48,720 --> 00:54:52,200 adopt new tools. Some people will always be a bit 1014 00:54:52,200 --> 00:54:54,000 wary of trying new things, right? 1015 00:54:54,200 --> 00:54:57,600 And I think as leaders, one of our responsibilities is to show 1016 00:54:57,600 --> 00:55:00,760 them, look how easy it is, look how useful it could be. 1017 00:55:01,320 --> 00:55:05,200 So this is like on a personal level and how to incorporate new 1018 00:55:05,200 --> 00:55:09,320 tools or methodologies or techniques to the organization. 1019 00:55:09,920 --> 00:55:15,200 So another great way to drive change is by giving talks. 1020 00:55:15,440 --> 00:55:20,960 So get your company team grouped together and give a live demo of 1021 00:55:20,960 --> 00:55:23,120 using the cool new tool, for example. 1022 00:55:24,080 --> 00:55:26,920 So that's more about incorporating new methodologies, 1023 00:55:27,040 --> 00:55:30,640 techniques, and tools. When we talk about deep 1024 00:55:30,640 --> 00:55:33,760 research, I think it's not for every organization. 1025 00:55:33,880 --> 00:55:36,880 I'm not going to say that every organization needs a research 1026 00:55:37,080 --> 00:55:40,440 person or team, right? But if you do, you need 1027 00:55:40,440 --> 00:55:42,600 technology takes time and a different mindset. 1028 00:55:42,960 --> 00:55:46,040 You can't expect a research team to operate the same way as a 1029 00:55:46,040 --> 00:55:48,960 development team with a clear timeline for every milestone. 1030 00:55:49,800 --> 00:55:54,000 So you can adjust people there, assign them to research tasks, 1031 00:55:54,520 --> 00:56:00,720 Make sure that the value for the product is clear and easy to 1032 00:56:00,720 --> 00:56:04,160 get, relatively easy at least, and get those people to 1033 00:56:04,160 --> 00:56:07,280 understand what research means. Make them be professionals at 1034 00:56:07,280 --> 00:56:12,000 managing research, at assigning time to leave to different 1035 00:56:12,000 --> 00:56:15,760 directions and having brainstorms about what the best 1036 00:56:15,760 --> 00:56:17,960 way to approach a specific issue is. 1037 00:56:18,600 --> 00:56:21,480 If you don't have this expertise, it's fine. 1038 00:56:21,960 --> 00:56:24,400 Consult with others who do. There are people who are 1039 00:56:24,400 --> 00:56:27,080 experienced researchers. They work differently from 1040 00:56:27,080 --> 00:56:29,160 people who are not experienced researchers. 1041 00:56:29,160 --> 00:56:32,120 It's the same with engineers and anything else, right? 1042 00:56:32,120 --> 00:56:35,800 But research is a skill. It's a skill that people can 1043 00:56:35,800 --> 00:56:38,880 improve at. It's a skill you can learn and 1044 00:56:38,880 --> 00:56:42,160 you can get help from others if you don't have the experience. 1045 00:56:42,440 --> 00:56:44,280 Well, I think that those are really, really great. 1046 00:56:44,280 --> 00:56:46,480 Advice, right? I particularly like about the 1047 00:56:46,560 --> 00:56:48,240 aspect of changing your mindset, right? 1048 00:56:48,240 --> 00:56:51,480 Because people think doing research is straightforward, 1049 00:56:51,480 --> 00:56:53,000 right? You do research, you get 1050 00:56:53,000 --> 00:56:55,280 something out of it, and you can use this straight away. 1051 00:56:55,800 --> 00:56:58,520 Especially people think that, OK, now with AI you can even get 1052 00:56:58,520 --> 00:57:00,720 more intelligence, right? You can speed up the research or 1053 00:57:00,720 --> 00:57:03,280 whatever that is, right? But I think especially doing 1054 00:57:03,280 --> 00:57:05,560 something that you are not capable of in terms of 1055 00:57:05,560 --> 00:57:07,640 capability within the organization, it's something 1056 00:57:07,640 --> 00:57:09,840 tricky, right? You can't sometimes justify the 1057 00:57:09,840 --> 00:57:11,880 effort. So hopefully people today learn 1058 00:57:11,880 --> 00:57:14,600 a lot of things you know about knowledge based research, using 1059 00:57:14,640 --> 00:57:16,360 AI for documentations and all that. 1060 00:57:16,720 --> 00:57:19,680 So Amir, as we reach the end of our conversation, I have one 1061 00:57:19,680 --> 00:57:22,480 last question that I'd like to ask you, which I asked to all my 1062 00:57:22,480 --> 00:57:24,400 guests. I call this the tree technical 1063 00:57:24,400 --> 00:57:27,280 leadership is them if you can think of it just like an advice 1064 00:57:27,280 --> 00:57:30,520 to us, what advice do you want to give to us today? 1065 00:57:31,400 --> 00:57:35,240 OK, so I. Think the first one would be put 1066 00:57:35,880 --> 00:57:37,800 your time and effort into your people. 1067 00:57:38,480 --> 00:57:41,160 I mean they make all the difference right? 1068 00:57:41,520 --> 00:57:45,680 And it means talking with them about how they are and what can 1069 00:57:45,680 --> 00:57:49,280 help them and making sure you have them grow in their 1070 00:57:49,280 --> 00:57:51,800 position. Finding them is hard. 1071 00:57:52,440 --> 00:57:56,040 Leading people is sometimes hard, but I think it's also the 1072 00:57:56,040 --> 00:57:59,320 most rewarding part of the leader's job. 1073 00:58:00,040 --> 00:58:05,200 The second thing is, in this era, you must be open minded to 1074 00:58:05,200 --> 00:58:08,640 try new things. I don't think it makes sense for 1075 00:58:08,640 --> 00:58:12,480 someone in 2025 to work even the same as they worked in 2024. 1076 00:58:12,720 --> 00:58:14,680 And it sounds almost childish, right? 1077 00:58:14,720 --> 00:58:17,160 I used to make fun of people saying things like that, but 1078 00:58:17,160 --> 00:58:21,360 nowadays it doesn't make sense. Things change really fast and in 1079 00:58:21,360 --> 00:58:23,840 order not to stay behind, you have to be on top of it. 1080 00:58:24,200 --> 00:58:27,480 So you have to be open minded and keep yourself informed. 1081 00:58:28,040 --> 00:58:31,920 And the third thing, in case it's viable for your 1082 00:58:31,920 --> 00:58:36,560 organization, put the time into research because it can open up 1083 00:58:36,920 --> 00:58:40,520 new directions and it can help you in ways that you haven't 1084 00:58:40,520 --> 00:58:44,200 dreamt of before. Sometimes spending two days on a 1085 00:58:44,200 --> 00:58:47,880 research task can make you change your decisions 1086 00:58:47,880 --> 00:58:55,680 completely, so I think allocate the time in case it's relevant 1087 00:58:55,680 --> 00:58:59,800 for your business case. Yeah, specifically about being 1088 00:58:59,800 --> 00:59:01,720 open. Mind, I myself also quite 1089 00:59:01,720 --> 00:59:04,840 concerned, you know, with some of my habits and skills that I 1090 00:59:04,840 --> 00:59:07,120 learned in the past, right? Whether that it can still be 1091 00:59:07,120 --> 00:59:10,760 relevant, especially the pace of changes these days is so rapid, 1092 00:59:10,760 --> 00:59:12,400 right? Every day you would probably 1093 00:59:12,400 --> 00:59:14,360 hear, oh, there's a new way of doing things. 1094 00:59:14,360 --> 00:59:17,560 There's new tool that can help you do what being open minded 1095 00:59:17,560 --> 00:59:19,880 and willing to try and willing to be challenged. 1096 00:59:19,880 --> 00:59:22,800 I think it's also another thing that I feel sometimes as a 1097 00:59:22,800 --> 00:59:26,200 senior, right, we think we know the problem well, we can solve 1098 00:59:26,200 --> 00:59:29,440 it by heart, but sometimes there are new ways of doing things 1099 00:59:29,440 --> 00:59:31,160 these days. So thank you so much for sharing 1100 00:59:31,160 --> 00:59:33,560 those wisdom. So if people want to connect 1101 00:59:33,560 --> 00:59:36,320 with you or you know, asking you more things, is there a place 1102 00:59:36,320 --> 00:59:38,880 where they can find you online? Sure. 1103 00:59:38,880 --> 00:59:43,840 So you can reach out to me via. E-mail it's Omer Omer at swim 1104 00:59:43,880 --> 00:59:50,520 that's SWIM m.io also on LinkedIn, though I don't really 1105 00:59:50,520 --> 00:59:53,560 use social media as much so if you send me a message and I 1106 00:59:53,560 --> 00:59:56,200 don't get back, I apologize but probably didn't see it. 1107 00:59:56,640 --> 01:00:00,000 I do answer to emails and I'll be happy to stay in touch. 1108 01:00:00,920 --> 01:00:02,760 Thank you again, Omar, for spending the time today. 1109 01:00:02,760 --> 01:00:04,400 I. Think we all learn a lot about 1110 01:00:04,400 --> 01:00:07,840 using AI, building knowledge base and doing research as what 1111 01:00:07,840 --> 01:00:10,960 you advised just now. So thank you again, my pleasure. 1112 01:00:10,960 --> 01:00:12,240 Thank you for having me, Henry.