1 00:00:00,000 --> 00:00:03,720 Today, developers spend a lot of time worrying about their code. 2 00:00:03,810 --> 00:00:07,098 How do they make it reliable? Anywhere between 40% to 60% of 3 00:00:07,098 --> 00:00:10,692 the developer's time is on all of the kind of scaffolding 4 00:00:10,692 --> 00:00:14,198 versus the core business logic. Preeti Somal, Senior VP of 5 00:00:14,198 --> 00:00:16,551 Engineering at Temporal, discusses the challenges of 6 00:00:16,551 --> 00:00:19,020 building and operating large scale reliable systems, drawing 7 00:00:19,020 --> 00:00:22,020 from her experiences at Temporal, HashiCorp and Yahoo. 8 00:00:22,170 --> 00:00:25,470 Temporal recently just released this state of developer report. 9 00:00:25,530 --> 00:00:28,782 What are the key highlights? About 36% of the engineers and 10 00:00:28,782 --> 00:00:31,899 leaders that were surveyed, kind of had reliability and 11 00:00:31,899 --> 00:00:35,869 compliance as their top need, even more than cost or 12 00:00:35,869 --> 00:00:38,287 performance. Our focus at Temporal is around 13 00:00:38,287 --> 00:00:41,965 making it easier for developers to build these reliable mission 14 00:00:41,965 --> 00:00:45,523 critical applications. The thing that is so magical is 15 00:00:45,523 --> 00:00:48,880 that the Temporal platform itself takes care of all of 16 00:00:48,880 --> 00:00:52,447 these reliability concerns. We shield the developer from all 17 00:00:52,447 --> 00:00:56,415 of the complexities of running and building reliable code. 18 00:00:56,415 --> 00:00:59,295 And the developer can focus on their business logic. 19 00:00:59,385 --> 00:01:02,140 It sounds too good to be true. Our two co-founders have been 20 00:01:02,140 --> 00:01:04,595 kinda working on this problem for a lot of years. 21 00:01:04,644 --> 00:01:06,480 So it's not an easy problem to solve. 22 00:01:06,955 --> 00:01:10,435 How we do that is essentially we introduce... 23 00:01:11,425 --> 00:01:15,102 We've had customers tell us that they've been able to get like 6X 24 00:01:15,102 --> 00:01:19,225 developer productivity, but also peace of mind and knowing that 25 00:01:19,225 --> 00:01:23,343 reliability is taken care of. I wish we had Temporal back when 26 00:01:23,343 --> 00:01:25,480 I was at Yahoo. It would've really made my life 27 00:01:25,480 --> 00:01:44,370 so much easier back then. Hello, guys. 28 00:01:44,370 --> 00:01:47,065 Welcome back to another new episode of the Tech Lead Journal 29 00:01:47,065 --> 00:01:49,669 podcast. Today, I have with me SVP of 30 00:01:49,669 --> 00:01:53,440 Temporal, a company that I've been hearing a lot good things 31 00:01:53,440 --> 00:01:55,590 about Temporal in the recent months, right? 32 00:01:55,920 --> 00:01:59,601 Preeti Somal here with me today. I'm really looking forward to 33 00:01:59,601 --> 00:02:02,151 learning from you from your career journey and about 34 00:02:02,151 --> 00:02:04,475 Temporal. And I know that Temporal just 35 00:02:04,475 --> 00:02:07,233 recently launched a state of developer report as well. 36 00:02:07,600 --> 00:02:09,560 So yeah, looking forward for our chat today, Preeti. 37 00:02:10,098 --> 00:02:13,722 Yeah, likewise. Thank you for having me on your 38 00:02:13,722 --> 00:02:17,044 podcast, Henry. And uh, I'm really excited to 39 00:02:17,044 --> 00:02:20,130 talk more about myself and technology and Temporal. 40 00:02:21,090 --> 00:02:22,919 Yeah. So maybe let's start from 41 00:02:22,919 --> 00:02:25,088 yourself first, right? I always love to invite my 42 00:02:25,088 --> 00:02:27,464 guests to share some career highlights or turning points, 43 00:02:27,464 --> 00:02:30,459 learnings that you have throughout your journey that you 44 00:02:30,459 --> 00:02:33,943 think we all can learn from you. Yeah, that sounds great. 45 00:02:33,943 --> 00:02:37,965 I think, for me, uh, you know, I started off my career in 46 00:02:37,965 --> 00:02:40,524 enterprise software. I started in Oracle, spent some 47 00:02:40,524 --> 00:02:44,075 time at VMware. I would say though, that the 48 00:02:44,075 --> 00:02:47,103 turning points were sort of the three roles after. 49 00:02:47,554 --> 00:02:49,294 The first one, of course, was Yahoo. 50 00:02:49,294 --> 00:02:53,382 And, um, Yahoo for me was the first time that I was on the 51 00:02:53,382 --> 00:02:59,327 other side, both building and operating tremendous web scale 52 00:02:59,327 --> 00:03:02,193 infrastructure for mission critical services. 53 00:03:02,613 --> 00:03:07,553 Uh, and so it was my first sort of opportunity to really learn 54 00:03:07,553 --> 00:03:11,823 about running services at scale using open source. 55 00:03:11,823 --> 00:03:16,513 Honestly, Yahoo, as you probably know is a big user and 56 00:03:16,513 --> 00:03:20,753 contributor back to open source. And so that was fantastic 57 00:03:20,753 --> 00:03:23,991 learning. My customers at Yahoo were other 58 00:03:23,991 --> 00:03:27,932 engineering teams, so really the developers within Yahoo that 59 00:03:27,932 --> 00:03:30,892 used all of these platforms I was building. 60 00:03:31,735 --> 00:03:35,191 From there, my next turning point was HashiCorp. 61 00:03:35,191 --> 00:03:41,263 And, uh, I got to HashiCorp early in 2018 when the company 62 00:03:41,263 --> 00:03:47,005 was still 150 people. And really I think it was an 63 00:03:47,005 --> 00:03:51,125 amazing journey, again, open source developer, but also scale 64 00:03:51,125 --> 00:03:56,325 how we scaled the organization, the teams, and continued to 65 00:03:56,325 --> 00:04:00,317 really remain close to the developer community. 66 00:04:00,933 --> 00:04:05,517 So I spent five years there. And when I left HashiCorp, uh, 67 00:04:05,517 --> 00:04:08,893 you know, this opportunity at Temporal came up. 68 00:04:08,893 --> 00:04:13,093 And we at HashiCorp were using Temporal. 69 00:04:13,407 --> 00:04:16,317 And so I was really familiar with the technology and, uh, 70 00:04:16,647 --> 00:04:21,951 another turning point because we run Temporal as a cloud service. 71 00:04:21,951 --> 00:04:27,958 And so mission critical scale reliability is like our number 72 00:04:27,958 --> 00:04:33,378 one value. And just learning to see how our 73 00:04:33,378 --> 00:04:38,799 customers rely on Temporal has really brought me a lot of 74 00:04:38,799 --> 00:04:42,529 perspective on the place we have in that technology stack. 75 00:04:43,550 --> 00:04:47,047 Wow, I think, uh, we can see like quite an illustrious career 76 00:04:47,047 --> 00:04:49,486 so far, right? So I think, and also you joined 77 00:04:49,486 --> 00:04:51,270 at the right time, I would say, right? 78 00:04:51,270 --> 00:04:54,413 So maybe Yahoo back then was still kind of like big, right? 79 00:04:54,713 --> 00:04:57,738 And then moving to HashiCorp, we know where HashiCorp is now, 80 00:04:57,738 --> 00:05:00,506 right? And then lastly, joining new 81 00:05:00,506 --> 00:05:02,666 pretty exciting, uh, companies, right? 82 00:05:02,764 --> 00:05:05,224 I think it's gonna be a big thing as well, Temporal. 83 00:05:05,524 --> 00:05:08,864 So yeah, maybe we can learn a little bit, from these three 84 00:05:08,864 --> 00:05:10,724 companies, uh, in your career, right? 85 00:05:10,724 --> 00:05:14,706 So you are operating and running, you know, a big 86 00:05:14,706 --> 00:05:17,554 infrastructure. Maybe, cloud, uh, engineering 87 00:05:17,554 --> 00:05:20,158 teams. Uh, what are some of the key 88 00:05:20,158 --> 00:05:23,275 learnings, you know, seeing because you do not just handle a 89 00:05:23,275 --> 00:05:26,944 smaller scale, kind of a traffic and kind of infrastructure. 90 00:05:27,184 --> 00:05:29,434 So tell us a little bit more about this journey. 91 00:05:29,534 --> 00:05:32,634 What kind of scale you were running back then, and what are 92 00:05:32,634 --> 00:05:35,514 some of the key learnings? Yeah, yeah, absolutely. 93 00:05:36,214 --> 00:05:42,089 Uh, so at Yahoo, the scale was literally running, uh, you know, 94 00:05:42,089 --> 00:05:44,807 the entire sort of internal cloud services. 95 00:05:44,807 --> 00:05:48,689 And I'll pick on one system, which was the monitoring 96 00:05:48,689 --> 00:05:51,117 service. And, uh, this monitoring 97 00:05:51,117 --> 00:05:53,779 service. And I'm talking about, you know, 98 00:05:53,779 --> 00:05:57,717 2008 to 2013, so it's been a little bit of time. 99 00:05:57,717 --> 00:06:03,687 But it was called yamas and it was consuming billions of sort 100 00:06:03,687 --> 00:06:08,041 of time series data points on a daily basis. 101 00:06:08,041 --> 00:06:12,347 And in fact, in terms of the scale, it was getting to a point 102 00:06:12,347 --> 00:06:15,936 where we were, uh, really looking at perhaps taking this 103 00:06:15,936 --> 00:06:21,211 data and moving it to the Hadoop big data clusters as well. 104 00:06:22,191 --> 00:06:25,872 Uh, so that's a quick idea on scale, but I think from a 105 00:06:25,872 --> 00:06:32,124 learning's point of view, it's really around taking kind of 106 00:06:32,124 --> 00:06:36,711 scale reliability seriously, because these are mission 107 00:06:36,711 --> 00:06:40,719 critical systems and outages are very, very costly. 108 00:06:41,079 --> 00:06:46,146 And when I say seriously, I mean both in your design, build, how 109 00:06:46,146 --> 00:06:51,635 you iterate, how you test, what your incident response sort of 110 00:06:51,635 --> 00:06:55,472 culture is. All of these pieces ultimately 111 00:06:55,472 --> 00:07:00,657 result in a culture of like strong engineering ownership. 112 00:07:01,407 --> 00:07:04,908 And this was one of the things I really learned at Yahoo, that 113 00:07:04,908 --> 00:07:09,227 there was a tremendous amount of ownership and pride in like the 114 00:07:09,227 --> 00:07:12,482 work we did. 'Cause you think about it, a lot 115 00:07:12,482 --> 00:07:15,755 of the infrastructure work is really invisible, right? 116 00:07:15,755 --> 00:07:20,482 Like Yahoo would launch a new consumer set of application. 117 00:07:20,922 --> 00:07:24,610 But, you know, nobody in infrastructure gets any 118 00:07:24,610 --> 00:07:29,900 visibility around that. And so you really need to have a 119 00:07:29,900 --> 00:07:33,338 culture where your engineers are deriving that satisfaction 120 00:07:33,338 --> 00:07:37,603 because of the scale and the reliability that they're able to 121 00:07:37,603 --> 00:07:40,956 hit. Yeah, I was about to say that 122 00:07:40,956 --> 00:07:42,450 when you mentioned about invisible, right? 123 00:07:42,450 --> 00:07:44,400 Yes, pretty much when things are all right. 124 00:07:44,430 --> 00:07:47,477 But when things are not all right, like outages, I think 125 00:07:47,477 --> 00:07:49,740 pretty much infra team is the most visible. 126 00:07:49,980 --> 00:07:53,040 Uh, I had that experience as well back then, um, in my 127 00:07:53,040 --> 00:07:55,115 previous company. And yeah, I think it's uh, quite 128 00:07:55,115 --> 00:07:59,347 funny when you said that, right? So I think back then you kind of 129 00:07:59,347 --> 00:08:01,936 like building this internal engineering platform, I would 130 00:08:01,936 --> 00:08:04,783 say, right, platform engineering is quite famous term these days 131 00:08:04,783 --> 00:08:08,403 and internal developer portals. So you were running it back then 132 00:08:08,403 --> 00:08:11,103 long time ago. So any kind of learnings that 133 00:08:11,103 --> 00:08:14,237 you think are still pretty relevant to what we call 134 00:08:14,237 --> 00:08:16,543 platform engineering now? Absolutely! 135 00:08:16,543 --> 00:08:19,504 And it's actually really interesting because platform 136 00:08:19,504 --> 00:08:23,406 engineering now is becoming like this big sort of pattern. 137 00:08:23,406 --> 00:08:27,870 But, you know, back, you know, 10, 12 years ago, like literally 138 00:08:27,870 --> 00:08:31,326 the team I was part of was called platforms, right? 139 00:08:32,166 --> 00:08:36,206 And so a lot of the learnings I think are very relevant, which 140 00:08:36,206 --> 00:08:41,932 is really around make sure that you are sort of working very 141 00:08:41,932 --> 00:08:44,646 closely with the consumers of your platform. 142 00:08:45,136 --> 00:08:48,926 Pick a few use cases. Make your customers successful. 143 00:08:49,286 --> 00:08:53,366 Nowadays, we would call it like developer relations or you know, 144 00:08:53,366 --> 00:08:56,029 having a product manager on board. 145 00:08:56,449 --> 00:08:59,599 Set some goals and targets around adoption. 146 00:08:59,899 --> 00:09:04,107 Make sure that you are providing visibility into how you are 147 00:09:04,107 --> 00:09:07,944 doing around kind of the objectives that you've set out 148 00:09:07,944 --> 00:09:11,641 for and talk about them. You know, going back to the 149 00:09:11,641 --> 00:09:14,359 invisible, uh, sort of theme we were talking about. 150 00:09:14,779 --> 00:09:19,036 One thing we would do at Yahoo was we would put posters up in 151 00:09:19,036 --> 00:09:23,382 the kitchens about like the work that the platform team had done 152 00:09:23,382 --> 00:09:27,654 and just, you know, create more visibility into the work that 153 00:09:27,654 --> 00:09:31,794 you're doing and the value you are providing as well. 154 00:09:33,156 --> 00:09:35,364 Wow, I think, uh, it's pretty interesting, right? 155 00:09:35,364 --> 00:09:38,614 The putting the posters, uh, for your internal platforms. 156 00:09:38,964 --> 00:09:42,719 I remember Google also has this something they call testing in 157 00:09:42,719 --> 00:09:44,064 the toilet kind of thing. So... 158 00:09:44,147 --> 00:09:45,930 Right. I used to work at Google as 159 00:09:45,930 --> 00:09:48,566 well, so whenever we went to toilet you can see some posters 160 00:09:48,566 --> 00:09:51,400 about testing patterns or these, you know, good design pattern 161 00:09:51,400 --> 00:09:53,668 for testability. So I think it is pretty similar 162 00:09:53,668 --> 00:09:55,382 thing. I guess like you should put 163 00:09:55,382 --> 00:09:58,428 visibility so that people kind of like unexpectedly learn about 164 00:09:58,428 --> 00:10:00,691 your platform. When you mentioned about 165 00:10:00,691 --> 00:10:04,391 reliability as a key thing. Back then Yahoo has a lot of 166 00:10:04,391 --> 00:10:06,453 engineering teams, I suppose, globally, right? 167 00:10:06,453 --> 00:10:08,433 And a lot of internet scale as well. 168 00:10:08,673 --> 00:10:11,637 What kind of key things that you think is the most critical thing 169 00:10:11,637 --> 00:10:14,721 that people should know about reliability and how to kind of 170 00:10:14,721 --> 00:10:17,952 like do a good pattern to ensure that reliability is kind of 171 00:10:17,952 --> 00:10:19,925 high? Yeah. 172 00:10:19,985 --> 00:10:25,414 Um, great question and you know, I think reliability for me has 173 00:10:25,414 --> 00:10:31,401 been, an area that I now spend a lot of time in, given my role at 174 00:10:31,401 --> 00:10:32,967 Temporal. I think. 175 00:10:33,257 --> 00:10:37,265 I wish, first of all, that we had Temporal back when I was at 176 00:10:37,265 --> 00:10:40,621 Yahoo. It would've really made my life 177 00:10:40,621 --> 00:10:45,852 so much easier back then. But really I think the pieces we 178 00:10:45,852 --> 00:10:50,602 would worry about a lot is the patterns that break. 179 00:10:50,962 --> 00:10:54,652 How do we make sure we have like fault tolerant systems? 180 00:10:54,657 --> 00:10:58,372 How do we make sure we've got, you know, all the alerting and 181 00:10:58,372 --> 00:11:02,302 monitoring in place so that we can respond and recover quickly? 182 00:11:03,159 --> 00:11:07,625 Thundering herd problems, you know, making sure that we had 183 00:11:07,625 --> 00:11:12,954 all the right, like rate limits in place so that we weren't 184 00:11:12,954 --> 00:11:16,560 overwhelming downstream systems. All of the patterns that, you 185 00:11:16,560 --> 00:11:20,483 know, have kind of come up around distributed systems and 186 00:11:20,483 --> 00:11:23,977 engineering. And just to segue to my current 187 00:11:23,977 --> 00:11:27,952 role at Temporal, uh, the thing that is so magical about 188 00:11:27,952 --> 00:11:33,394 Temporal is that the Temporal platform itself takes care of 189 00:11:33,394 --> 00:11:35,477 all of these reliability concerns. 190 00:11:35,477 --> 00:11:38,902 And that's sort of why I was saying, you know, this has 191 00:11:38,902 --> 00:11:41,707 followed me to my current role as well. 192 00:11:42,957 --> 00:11:46,429 Yeah, I think it's pretty hard to run a high reliable systems, 193 00:11:46,429 --> 00:11:49,509 especially when you have a distributed systems, many 194 00:11:49,509 --> 00:11:51,177 components running altogether, right? 195 00:11:51,477 --> 00:11:54,942 Just to piecemeal what is actually happening for serving 196 00:11:54,942 --> 00:11:57,567 one request, for example, I think it's also not easy. 197 00:11:57,962 --> 00:12:01,048 Uh, and the nature of, you know, high traffic as well makes it 198 00:12:01,048 --> 00:12:04,170 even more difficult. So maybe let's go back a little 199 00:12:04,170 --> 00:12:05,535 bit to HashiCorp experience, right? 200 00:12:05,585 --> 00:12:09,155 I don't know whether HashiCorp back then runs, uh, like a high, 201 00:12:09,335 --> 00:12:12,514 highly scalable system, right? From my point of view, it's 202 00:12:12,514 --> 00:12:14,641 mainly the open source products that they have, right? 203 00:12:14,641 --> 00:12:18,211 Starting with Terraform, you know, Vault, uh, and all that. 204 00:12:18,613 --> 00:12:20,458 But lately, they have also cloud platform. 205 00:12:20,458 --> 00:12:25,637 So tell us, what was your main highlights when you work in 206 00:12:25,637 --> 00:12:27,256 HashiCorp? What kind of things that you 207 00:12:27,256 --> 00:12:28,438 think are your biggest achievement there? 208 00:12:29,036 --> 00:12:34,736 Yeah, I think for me, the highlights really were around 209 00:12:34,736 --> 00:12:40,700 how strong the HashiCorp community open source presence 210 00:12:40,700 --> 00:12:43,970 was. Especially when you talk about 211 00:12:43,970 --> 00:12:46,450 Terraform. You know, Terraform has sort of 212 00:12:46,450 --> 00:12:49,870 become a verb. It's a standard and everybody 213 00:12:49,870 --> 00:12:55,179 knows Terraform. And so I think the power of like 214 00:12:55,179 --> 00:12:58,286 taking this concept of infrastructure provisioning and 215 00:12:58,286 --> 00:13:04,475 enabling it as code so that it can go in GitHub and, you know, 216 00:13:04,475 --> 00:13:07,734 all of these benefits of code that brings forward, right? 217 00:13:07,734 --> 00:13:12,839 And the, the role that the developer plays versus, you 218 00:13:12,839 --> 00:13:18,101 know, prior to something like Terraform, you had to either go 219 00:13:18,101 --> 00:13:21,389 ask for permission or you had to speak to somebody in IT. 220 00:13:22,139 --> 00:13:28,115 And the interesting piece also is that the technology trend 221 00:13:28,115 --> 00:13:32,059 around cloud becoming more ubiquitous was also happening 222 00:13:32,059 --> 00:13:37,457 around the same time, right? So, um, to me, I think the 223 00:13:37,457 --> 00:13:42,335 highlight really is community, how we worked with them, how we 224 00:13:42,335 --> 00:13:47,253 got contributions, and just the sheer sort of scale of the 225 00:13:47,253 --> 00:13:51,093 standardization that's happened around the HashiCorp products as 226 00:13:51,093 --> 00:13:53,032 well. Yeah. 227 00:13:53,182 --> 00:13:56,392 And how do you see the, you know, the kind of like the 228 00:13:56,392 --> 00:13:58,132 infrastructure as code side, right? 229 00:13:58,132 --> 00:14:00,322 So you were running platform engineering back then. 230 00:14:00,422 --> 00:14:03,122 How did you automate some of these, you know, server 231 00:14:03,122 --> 00:14:06,042 creations and all that? And how do you think tools like 232 00:14:06,042 --> 00:14:08,158 Terraform, you know, like infrastructure as code, maybe 233 00:14:08,158 --> 00:14:10,352 these days is more declarative as well. 234 00:14:10,352 --> 00:14:13,635 Like how do you see some kind of things that uh, you think will 235 00:14:13,635 --> 00:14:16,995 be great to have back then? Oh, absolutely. 236 00:14:17,325 --> 00:14:21,805 I think, uh you know, I think the parts around... 237 00:14:21,805 --> 00:14:25,971 You know, it was definitely great to have like the code part 238 00:14:25,971 --> 00:14:29,167 of it. But the elements of being able 239 00:14:29,167 --> 00:14:33,086 to reason about your infrastructure and to be able to 240 00:14:33,086 --> 00:14:38,433 have those feedback loops as things were working or not 241 00:14:38,433 --> 00:14:43,168 working, scale or not. You know, I think those pieces 242 00:14:43,168 --> 00:14:47,619 are definitely advancing a lot more now than sort of in the 243 00:14:47,619 --> 00:14:51,976 early Terraform days. Fun fact is we are also using 244 00:14:51,976 --> 00:14:56,221 Terraform here at Temporal. But the workflow orchestration 245 00:14:56,221 --> 00:15:00,938 around Terraform is being done by Temporal, the Temporal 246 00:15:00,938 --> 00:15:04,130 workflows in the Temporal platform as well. 247 00:15:04,610 --> 00:15:11,105 And we can do a lot more there in terms of being able to really 248 00:15:11,105 --> 00:15:14,563 see what's happening in the provisioning flows and where we 249 00:15:14,563 --> 00:15:17,481 are running into issues and how we resolve them. 250 00:15:18,706 --> 00:15:21,588 Wow, It's pretty interesting. Uh, so using Terraform in your 251 00:15:21,588 --> 00:15:24,276 workflow as well. So I must assume it's more like 252 00:15:24,276 --> 00:15:27,222 a dynamic infra propositioning and all that, that is part of 253 00:15:27,222 --> 00:15:30,197 the Temporal workflow. So you mentioned about incident 254 00:15:30,197 --> 00:15:33,472 response, I think for someone running infra engineering org, 255 00:15:33,472 --> 00:15:37,473 especially a big scale, right? Definitely you cannot run away 256 00:15:37,473 --> 00:15:40,297 from like outages, issues, incidents, and all that. 257 00:15:40,575 --> 00:15:44,150 So tell us a little bit, how do you build such ownership and, 258 00:15:44,150 --> 00:15:47,472 you know, maybe a great incident response as part of your 259 00:15:47,472 --> 00:15:49,057 learnings in all these organizations? 260 00:15:49,739 --> 00:15:52,547 Yeah. I think my number one principle 261 00:15:52,547 --> 00:15:55,169 there is you have to lead by example. 262 00:15:55,580 --> 00:16:01,052 And so you know, when an incident happens, I get paged as 263 00:16:01,052 --> 00:16:04,328 well. I am joining the incident rooms. 264 00:16:04,328 --> 00:16:09,014 And you really need to show up for your team and help them 265 00:16:09,014 --> 00:16:11,714 understand that, you know, this is really important. 266 00:16:12,074 --> 00:16:14,894 You care. You are there supporting them. 267 00:16:15,261 --> 00:16:19,551 I may not be able to actually help with the details, but I am 268 00:16:19,551 --> 00:16:25,377 there and that sets a pattern. So I'd say lead by example. 269 00:16:25,587 --> 00:16:31,262 Second would be really have strong processes in place, and 270 00:16:31,722 --> 00:16:36,642 just reinforce those processes every time an incident happens. 271 00:16:37,062 --> 00:16:41,664 Uh, so for us, those processes are around, okay, what's the 272 00:16:41,664 --> 00:16:45,587 on-call chain? How quickly, uh, does that 273 00:16:45,587 --> 00:16:49,100 on-call get escalated? Where are the runbooks? 274 00:16:49,508 --> 00:16:52,980 We've put all our runbooks in GitHub as well, so that, you 275 00:16:52,980 --> 00:16:57,087 know, they're available to everybody and we can take sort 276 00:16:57,087 --> 00:17:00,801 of improvements and enhancements from anyone within engineering. 277 00:17:01,221 --> 00:17:04,790 So have a really, really strong sort of process in place. 278 00:17:05,241 --> 00:17:10,771 And then finally, it's around being able to measure, report, 279 00:17:10,771 --> 00:17:14,173 and build a strong culture around this. 280 00:17:14,173 --> 00:17:18,270 So one of the things, for instance, that we do at Temporal 281 00:17:18,270 --> 00:17:22,632 is every two weeks we have a meeting that's open to the whole 282 00:17:22,632 --> 00:17:26,537 company where we pick some incidents and the owner, the 283 00:17:26,537 --> 00:17:31,020 person who was kind of the incident lead basically runs 284 00:17:31,020 --> 00:17:35,585 through the incident timeline, the root cause analysis, what 285 00:17:35,585 --> 00:17:39,092 worked well, what didn't, how we're improving. 286 00:17:39,372 --> 00:17:44,717 And just sort of making it, uh, really normal to talk about this 287 00:17:44,717 --> 00:17:48,821 in a way that is really looking at how we get better. 288 00:17:49,271 --> 00:17:53,321 Uh, so building that culture through not just engineering, 289 00:17:53,321 --> 00:17:57,834 but through the whole company around reliability is really 290 00:17:57,834 --> 00:18:01,128 important in having a strong incident response. 291 00:18:02,330 --> 00:18:05,762 Yeah, so I must also highlight the, you know, the RCA, the root 292 00:18:05,762 --> 00:18:08,150 cause analysis, post-mortem culture part, right? 293 00:18:08,150 --> 00:18:11,609 Because if we don't normalize talking about it, making sure 294 00:18:11,609 --> 00:18:15,376 that, you know, first, we handle it well, so hopefully the 295 00:18:15,376 --> 00:18:18,308 incident got resolved, right? And then what key learnings from 296 00:18:18,308 --> 00:18:21,275 there, how we can iterate and have this feedback loop so that 297 00:18:21,275 --> 00:18:23,801 we can improve the process. And I think the most important 298 00:18:23,801 --> 00:18:26,144 thing is probably the so-called the psychological aspect and 299 00:18:26,144 --> 00:18:30,722 kind of like making sure people are not very, very disappointed 300 00:18:30,722 --> 00:18:33,509 about failures, right? So obviously we don't want to 301 00:18:33,509 --> 00:18:36,541 have failures, but the key thing is how can we learn and improve 302 00:18:36,541 --> 00:18:39,763 the processes from there? So maybe anything about 303 00:18:39,763 --> 00:18:42,649 psychological safety, especially infra team, right? 304 00:18:42,709 --> 00:18:46,365 They always get a lot of scrutiny by everyone when things 305 00:18:46,365 --> 00:18:49,879 are going wrong and maybe people think, oh, you're not doing your 306 00:18:49,879 --> 00:18:52,221 job well. So tell us a little bit more on 307 00:18:52,221 --> 00:18:54,409 this side. Like, how do you ensure the 308 00:18:54,409 --> 00:18:57,319 infra engineers feel safe? And they are okay whenever 309 00:18:57,319 --> 00:18:59,674 incidents happen. Yeah, absolutely. 310 00:18:59,674 --> 00:19:02,846 It's really important as you, as you know, and I can see that 311 00:19:02,846 --> 00:19:05,684 you've kind of been through this and you feel it. 312 00:19:05,684 --> 00:19:10,652 It's very important. The way that we approach 313 00:19:10,652 --> 00:19:14,221 building that sort of psychological safety is around 314 00:19:14,311 --> 00:19:19,104 really, you know, striving towards a blameless culture, 315 00:19:19,104 --> 00:19:24,188 really focusing on, you know, what was the information that 316 00:19:24,188 --> 00:19:27,770 the engineer had and how they made decisions? 317 00:19:27,770 --> 00:19:32,304 And was there sort of, was there a need to do better in the 318 00:19:32,304 --> 00:19:34,337 tooling, the information gathering? 319 00:19:34,915 --> 00:19:39,854 And also really around what is it that we learned from this 320 00:19:39,854 --> 00:19:43,229 incident. And then one thing that's really 321 00:19:43,229 --> 00:19:47,453 interesting that we've started doing is, at the end of the 322 00:19:47,453 --> 00:19:51,509 quarter, we actually send out a survey to all of the on-call 323 00:19:51,509 --> 00:19:54,149 engineers, and it's an anonymous survey. 324 00:19:54,569 --> 00:20:01,369 But it really is a way for us to gauge kind of like, not just how 325 00:20:01,369 --> 00:20:06,219 they feel about on-call and their effectiveness and kind of 326 00:20:06,219 --> 00:20:11,575 their sort of quality of life with on-call, but also the 327 00:20:11,575 --> 00:20:15,328 satisfaction that really comes from feeling empowered and 328 00:20:15,328 --> 00:20:18,814 feeling that they have that psychological safety as well. 329 00:20:19,347 --> 00:20:23,645 And that survey is, has been really well received in that, 330 00:20:23,645 --> 00:20:27,371 okay, you know, we care about that on-call experience and we 331 00:20:27,371 --> 00:20:31,548 wanna hear from people and we wanna make it better for the 332 00:20:31,548 --> 00:20:34,325 engineers. Uh, so I think these are some of 333 00:20:34,325 --> 00:20:37,264 the things that we're doing. And of course, I feel like the 334 00:20:37,264 --> 00:20:40,742 more we talk about this, the more we can share and learn from 335 00:20:40,742 --> 00:20:44,009 others as well. So I'm always open to new ideas 336 00:20:44,009 --> 00:20:48,889 around how we can do better. Yeah, I think the survey part is 337 00:20:48,889 --> 00:20:50,983 really important, right? Especially people may not talk. 338 00:20:51,403 --> 00:20:52,963 Especially this part of the world, right? 339 00:20:52,963 --> 00:20:55,828 I'm in a Asian part. So some people actually don't 340 00:20:55,828 --> 00:20:58,797 like to talk about failures, you know, their frustrations, so 341 00:20:58,797 --> 00:21:00,815 they think it's a part of the job. 342 00:21:01,145 --> 00:21:04,300 And I think anonymous surveys, probably every leaders can do, 343 00:21:04,300 --> 00:21:06,205 right? Just, you know, build that 344 00:21:06,205 --> 00:21:09,354 culture where people can submit their feelings, their feedback, 345 00:21:09,354 --> 00:21:12,434 their frustrations anonymously and maybe, yeah, we can take 346 00:21:12,434 --> 00:21:16,029 actions around that. So let's move on a little bit to 347 00:21:16,029 --> 00:21:18,432 another topic that we would like to discuss today. 348 00:21:18,432 --> 00:21:21,920 I know that Temporal recently just released this state of 349 00:21:21,920 --> 00:21:25,122 developer report, right? Every time I'm seeing in the 350 00:21:25,122 --> 00:21:27,667 industry, there's always this new state of something report, 351 00:21:27,667 --> 00:21:29,137 right? It's very exciting. 352 00:21:29,347 --> 00:21:31,567 But this is, I think the first time Temporal is doing it. 353 00:21:31,777 --> 00:21:34,447 So tell us a little bit more about this report and why are 354 00:21:34,447 --> 00:21:37,500 you guys publishing this report? Yeah, absolutely. 355 00:21:37,500 --> 00:21:40,140 We're super excited to have it out. 356 00:21:40,140 --> 00:21:45,519 And uh, essentially, you know, our focus at Temporal is around 357 00:21:45,519 --> 00:21:51,204 making it easier for developers to build these reliable, uh, 358 00:21:51,204 --> 00:21:55,221 mission critical applications. And we have a really strong 359 00:21:55,221 --> 00:21:57,123 community of developers that is growing. 360 00:21:57,456 --> 00:22:01,892 So what we wanted to do with the survey was really sort of tap 361 00:22:01,892 --> 00:22:05,620 into the knowledge of these engineers and developers and 362 00:22:05,620 --> 00:22:09,640 really try and understand, especially with AI, you know, 363 00:22:09,640 --> 00:22:12,888 what are some of the patterns and trends? 364 00:22:12,948 --> 00:22:16,998 How are they viewing the technology that they have? 365 00:22:16,998 --> 00:22:19,758 What are their challenges? What's important to them? 366 00:22:20,118 --> 00:22:24,388 And so that's was really kind of the impetus of getting the 367 00:22:24,388 --> 00:22:27,870 survey out. And we hope to do this regularly 368 00:22:27,870 --> 00:22:31,863 so we can kind of, you know, hear back in a very data-driven 369 00:22:31,863 --> 00:22:34,977 approach and then publish and share those learnings as well. 370 00:22:35,900 --> 00:22:36,465 Wow. Yeah. 371 00:22:36,465 --> 00:22:38,307 Looking forward to learning from this report. 372 00:22:38,517 --> 00:22:40,815 So... yeah. You mentioned about building, 373 00:22:40,815 --> 00:22:44,173 you know, maybe building and using AI kind of thing, right? 374 00:22:44,203 --> 00:22:46,273 So that's pretty trendy these days, right? 375 00:22:46,543 --> 00:22:49,317 So. Um, do you see, first of all, 376 00:22:49,317 --> 00:22:51,955 the usage of AI? You know, the adoption rate is 377 00:22:51,955 --> 00:22:53,613 really high, as part of your survey? 378 00:22:53,943 --> 00:22:57,153 And secondly, how many teams are actually building AI-related, 379 00:22:57,513 --> 00:23:00,805 you know, systems or solutions in their, you know, engineering 380 00:23:00,805 --> 00:23:03,252 teams? Yeah, great question. 381 00:23:03,252 --> 00:23:07,044 So it's interesting that the data is a little bimodal in the 382 00:23:07,044 --> 00:23:12,540 sense of, it's showing that 94% of the reports, they're using 383 00:23:12,540 --> 00:23:17,889 some kind of AI tools. You know, whether it's like a 384 00:23:17,889 --> 00:23:22,382 Copilot kind tool or ChatGPT. You know, there's some sort of 385 00:23:22,382 --> 00:23:24,530 usage of AI tooling in their workflows. 386 00:23:25,061 --> 00:23:31,877 But only about 39% said that they've got sort of built any 387 00:23:31,877 --> 00:23:36,731 sort of major AI projects themselves at scale. 388 00:23:36,731 --> 00:23:42,003 So the usage is definitely sort of increasing. 389 00:23:42,393 --> 00:23:47,049 But like the infrastructure or to use it more sort of natively 390 00:23:47,049 --> 00:23:50,888 in the work that they're doing or the work the customer 391 00:23:50,888 --> 00:23:54,118 application they're delivering that is still early. 392 00:23:55,092 --> 00:23:57,862 Yeah, so I would say many people would have adopted, you know, 393 00:23:57,862 --> 00:24:00,652 some kind of AI tools, especially the chat part, 394 00:24:00,652 --> 00:24:02,892 obviously it's quite common, ubiquitous, right? 395 00:24:03,095 --> 00:24:06,582 So what's your takeaway as a leader, right? 396 00:24:06,582 --> 00:24:10,602 So do you use also AI often and can we actually use AI reliably 397 00:24:10,602 --> 00:24:12,672 for infrastructure-related stuff? 398 00:24:13,834 --> 00:24:17,134 Oh, I wish. I think, uh. 399 00:24:17,734 --> 00:24:23,236 So within Temporal, we are certainly also doing a lot of 400 00:24:23,236 --> 00:24:28,782 work with some of the tooling. One really interesting thing is 401 00:24:28,782 --> 00:24:32,860 we are seeing a lot of developers, engineers, 402 00:24:32,860 --> 00:24:37,870 companies, building AI tools on Temporal. 403 00:24:37,870 --> 00:24:43,390 And one of the biggest things that we are seeing is that 404 00:24:43,390 --> 00:24:48,280 reliability is really, really important for kind of AI and 405 00:24:48,280 --> 00:24:54,474 agents to really be adopted in any kind of a, sort of, uh, 406 00:24:54,474 --> 00:24:57,799 widespread manner. It's really interesting because 407 00:24:57,799 --> 00:25:02,043 these problems around reliability with AI agents are 408 00:25:02,043 --> 00:25:07,301 all very, very similar patterns to the reliability of 409 00:25:07,301 --> 00:25:09,033 distributed systems applications. 410 00:25:09,453 --> 00:25:14,300 Uh, and so we're seeing a lot of customers really coming too 411 00:25:14,300 --> 00:25:17,583 Temporal for solving these reliability problems. 412 00:25:18,284 --> 00:25:22,073 Now on the other hand for your question, sort of, are we using 413 00:25:22,073 --> 00:25:25,424 any infrastructure AI tooling at the moment? 414 00:25:25,694 --> 00:25:26,714 No. Um. 415 00:25:27,131 --> 00:25:31,545 We may, we may build some ourselves for our own uses 416 00:25:31,545 --> 00:25:35,708 because we haven't seen anything coming out that we think really 417 00:25:35,708 --> 00:25:39,753 fits our needs. Um, but you know, we don't have 418 00:25:39,753 --> 00:25:44,778 anything in production that we use internally at the moment. 419 00:25:45,658 --> 00:25:47,713 Right. So yeah, definitely very 420 00:25:47,713 --> 00:25:51,478 interesting to see this space. I'm sure there will be lots of 421 00:25:51,478 --> 00:25:53,328 new inventions coming. Troubleshooting, debugging and 422 00:25:53,328 --> 00:25:56,715 all that, I think is still pretty much the, maybe the most 423 00:25:56,715 --> 00:25:59,945 use case that people use, right. Especially when you see kind of 424 00:25:59,945 --> 00:26:02,209 like peculiar error messages. Sometimes like we don't know 425 00:26:02,209 --> 00:26:03,869 what is going on with that product, right? 426 00:26:03,869 --> 00:26:06,489 So I think having an AI can help us. 427 00:26:06,869 --> 00:26:10,424 So, um, you mentioned about building, operating AI kind of 428 00:26:10,424 --> 00:26:12,947 like systems, right? Personally, I haven't done it 429 00:26:12,947 --> 00:26:15,624 myself, right? So what do you think are some 430 00:26:15,624 --> 00:26:18,583 complexities that involved in, you know, building AI systems, 431 00:26:18,583 --> 00:26:22,418 maybe agentic systems as well that, you know, requires like 432 00:26:22,418 --> 00:26:26,334 good reliability? Yeah, I think the, you know, 433 00:26:26,334 --> 00:26:31,756 when you think about like at the core of it, what the agentic AI 434 00:26:31,756 --> 00:26:37,785 system is doing, it's really, uh, sort of looking at what is a 435 00:26:37,785 --> 00:26:41,532 sort of customer problem at hand that they want to solve. 436 00:26:41,922 --> 00:26:45,819 You know, whether it's like booking a flight or pick any 437 00:26:45,819 --> 00:26:48,781 example. And then looking at kind of 438 00:26:48,781 --> 00:26:53,763 calling out to one of the models around what are the various 439 00:26:53,763 --> 00:26:57,141 options, what is the recommendation, and then calling 440 00:26:57,141 --> 00:27:01,423 to some tools around making the booking or the changes. 441 00:27:01,423 --> 00:27:05,448 You know, it's a... We think of it as, it's 442 00:27:05,448 --> 00:27:09,876 essentially like a multi-step process where each step can be 443 00:27:09,876 --> 00:27:12,993 error prone. It's very common, for instance, 444 00:27:12,993 --> 00:27:18,315 to call out to a model and get rate limited and not get your 445 00:27:18,315 --> 00:27:21,616 response back, right? So the, each of these steps is 446 00:27:21,616 --> 00:27:24,142 something that you could run into issues. 447 00:27:24,842 --> 00:27:28,102 Some of these steps could be fairly long running, especially, 448 00:27:28,102 --> 00:27:33,072 you know, one of the things that is coming up is like agents that 449 00:27:33,072 --> 00:27:37,768 are learning based on your usage pattern or when you need a human 450 00:27:37,768 --> 00:27:40,372 to do some approvals in that workflow. 451 00:27:40,852 --> 00:27:44,328 And so the big sort of reliability issue here is that 452 00:27:44,328 --> 00:27:49,042 if any one of those steps crashes or returns errors, what 453 00:27:49,042 --> 00:27:52,519 do you do? You know, do you start all over 454 00:27:52,519 --> 00:27:54,929 again? And we know that some of these 455 00:27:54,929 --> 00:27:56,717 steps are really expensive as well. 456 00:27:57,377 --> 00:28:02,103 And so this is where, you know, we kind of have been really 457 00:28:02,103 --> 00:28:06,679 fortunate to have a product that has been running in production 458 00:28:06,679 --> 00:28:10,919 at scale with Temporal, where the core of the problem that 459 00:28:10,919 --> 00:28:14,276 Temporal solves is how do you take like this multi-step 460 00:28:14,276 --> 00:28:19,226 process and you do all of the reliability, state management 461 00:28:19,226 --> 00:28:23,018 recovery pieces within the Temporal platform versus every 462 00:28:23,018 --> 00:28:25,841 engineer having to worry about it. 463 00:28:26,873 --> 00:28:30,834 Yeah, so I think agentic systems definitely requires a lot of 464 00:28:30,834 --> 00:28:32,699 multi-step. Sometimes even the steps are not 465 00:28:32,699 --> 00:28:35,653 well defined in the first place because the model will kind of 466 00:28:35,653 --> 00:28:38,727 like build, you know, what steps that, uh, it needs to, you know, 467 00:28:38,727 --> 00:28:40,249 do in order to solve the problem. 468 00:28:40,579 --> 00:28:42,769 And I can see like the complexities, right? 469 00:28:42,769 --> 00:28:45,739 So there's a undeterministic thing happening as well. 470 00:28:45,739 --> 00:28:49,426 And plus you dunno the kind of failure, I dunno, patterns that 471 00:28:49,426 --> 00:28:51,233 might happen, uh, with all those steps. 472 00:28:51,383 --> 00:28:54,362 So I wanna save the time to talk about Temporal at the end. 473 00:28:54,468 --> 00:28:58,194 But what are the key highlights or key learnings that you think, 474 00:28:58,324 --> 00:29:00,022 we can talk about from the report? 475 00:29:00,825 --> 00:29:06,528 Yeah, I think, one of the kind of key learnings from the report 476 00:29:06,528 --> 00:29:11,332 that actually mirrors a lot of our conversation is how 477 00:29:11,332 --> 00:29:16,167 important reliability is. And so, you know, about 36% of 478 00:29:16,167 --> 00:29:20,882 the engineers and leaders that were surveyed kind of had 479 00:29:20,882 --> 00:29:25,163 reliability and compliance as their top needs, even more than, 480 00:29:25,163 --> 00:29:27,803 for instance, cost or performance. 481 00:29:28,133 --> 00:29:30,773 So that was one finding from the survey. 482 00:29:31,228 --> 00:29:36,772 The other finding is, this is not going to be new to you and 483 00:29:36,772 --> 00:29:41,241 me, but it's around how engineers make tooling decisions 484 00:29:41,241 --> 00:29:45,321 versus, uh, the people who have the budget. 485 00:29:45,321 --> 00:29:48,451 You know, how they make the tooling decisions and that 486 00:29:48,801 --> 00:29:53,568 continues to remain misaligned. I think this is an age old 487 00:29:53,568 --> 00:29:57,028 problem. But that kind of came up in the 488 00:29:57,028 --> 00:30:00,945 survey as well. And, uh, you know, last but not 489 00:30:00,945 --> 00:30:05,844 least is, again, not a surprise, but it's around how failures 490 00:30:05,844 --> 00:30:08,951 represent like real business risk, right? 491 00:30:08,951 --> 00:30:15,343 And we all know that an outage is potentially like a 492 00:30:15,343 --> 00:30:17,694 existential sort of threat for a company. 493 00:30:17,694 --> 00:30:22,778 And so again, the survey validated that these failures 494 00:30:22,778 --> 00:30:27,959 come with like this business risk that is sometimes hard to 495 00:30:27,959 --> 00:30:32,159 kind of quantify. But it's a very, very real, uh, 496 00:30:32,159 --> 00:30:35,822 sort of feeling and, uh, effect of a failure. 497 00:30:36,921 --> 00:30:39,746 Right. I must say that I'm pretty, uh, 498 00:30:39,746 --> 00:30:41,877 kind of like not surprised by this. 499 00:30:41,877 --> 00:30:43,647 You know, reliability is the key thing. 500 00:30:43,647 --> 00:30:45,567 You know, compliance, especially, uh, security, 501 00:30:45,567 --> 00:30:48,273 privacy and all that is definitely top of mind. 502 00:30:48,393 --> 00:30:51,118 And that failures, maybe incidents result in like 503 00:30:51,118 --> 00:30:53,158 business kind of, uh, risk, right? 504 00:30:53,552 --> 00:30:56,412 Especially if you are running like AI agentic system, first of 505 00:30:56,412 --> 00:30:57,931 all, how do you know it's accurate? 506 00:30:57,961 --> 00:30:59,461 How do you know it's reliable and all that? 507 00:30:59,860 --> 00:31:02,710 So definitely those part is, I think it's unsurprising. 508 00:31:03,040 --> 00:31:05,873 But what still surprised me is the choice of developer tooling, 509 00:31:05,873 --> 00:31:07,870 like what you mentioned, the misalignment. 510 00:31:08,200 --> 00:31:11,544 I know that it happens, uh, throughout various different 511 00:31:11,544 --> 00:31:14,065 organizations. But what I can say is like, 512 00:31:14,065 --> 00:31:16,201 typically, like if the developers choose the tooling, 513 00:31:16,201 --> 00:31:19,438 right, it can help a lot in terms of, you know, choosing the 514 00:31:19,438 --> 00:31:22,381 best tool that can serve the purpose really, really well. 515 00:31:22,771 --> 00:31:26,177 Maybe in terms of, uh, adoption rate as well, it will be much, 516 00:31:26,177 --> 00:31:29,354 much higher. So tell us why, you know, the 517 00:31:29,354 --> 00:31:32,196 decision makers or the executives still think they can 518 00:31:32,196 --> 00:31:35,119 choose, you know, the best tools on behalf of the engineering 519 00:31:35,119 --> 00:31:37,022 team. So tell us a little bit more 520 00:31:37,022 --> 00:31:38,989 about this problem, misalignment, how we can solve 521 00:31:38,989 --> 00:31:40,185 this? Yeah. 522 00:31:40,185 --> 00:31:45,032 So first isn't it great when the survey kind of validates how 523 00:31:45,032 --> 00:31:47,325 you're thinking about a problem, right? 524 00:31:47,715 --> 00:31:52,197 But second, on the misalignment, honestly, we are not sure. 525 00:31:52,197 --> 00:31:57,001 Like the survey didn't, besides highlighting the fact that there 526 00:31:57,001 --> 00:31:59,907 is misalignment, didn't give us insight into why. 527 00:32:00,497 --> 00:32:04,632 And I think, you know, I, for instance, a hundred percent 528 00:32:04,632 --> 00:32:07,050 agree with what you're saying, uh, you know. 529 00:32:07,631 --> 00:32:10,331 The key here is to empower the developers. 530 00:32:11,109 --> 00:32:17,414 Our suspicion is that some of this stems from all of the 531 00:32:17,414 --> 00:32:20,063 compliance pieces that organizations sometimes, you 532 00:32:20,063 --> 00:32:25,704 know, a lot of times kind of have to follow, where some of 533 00:32:25,704 --> 00:32:30,599 the tools that developers want to get might not have the SOX 534 00:32:30,599 --> 00:32:34,873 compliance or the HIPAA or, you know, whatever other compliance 535 00:32:34,873 --> 00:32:37,933 pieces are in play for your company. 536 00:32:38,819 --> 00:32:43,885 Yeah, so I think, I think one of the things that is very real in 537 00:32:43,885 --> 00:32:49,082 our industry is in order to really get broader adoption 538 00:32:49,082 --> 00:32:53,556 within these bigger, more regulated companies, how can 539 00:32:53,556 --> 00:33:00,039 some of the sort of new tooling fit into their compliance and 540 00:33:00,039 --> 00:33:04,601 security frameworks as well? So I suspect that some of that 541 00:33:04,601 --> 00:33:08,030 misalignment is coming from some of these requirements that the 542 00:33:08,030 --> 00:33:11,761 developer may not see, but the decision maker has to comply 543 00:33:11,761 --> 00:33:14,850 with. Yeah, so compliance, security, 544 00:33:14,850 --> 00:33:17,984 definitely, uh, typically top off, you know, their concerns. 545 00:33:18,044 --> 00:33:20,324 And the second one is probably cost, right? 546 00:33:20,324 --> 00:33:23,570 Because some of these developer tools may be deemed as 547 00:33:23,570 --> 00:33:24,954 expensive. Even though sometimes it's 548 00:33:24,954 --> 00:33:28,154 really hard to quantify the, you know, the value coming out from 549 00:33:28,154 --> 00:33:30,758 the cost, right? If it can save a lot of, you 550 00:33:30,758 --> 00:33:32,874 know, I dunno, engineering hours, I think that tool is 551 00:33:32,874 --> 00:33:35,538 worth that much, right? Um, but definitely, yeah, these 552 00:33:35,538 --> 00:33:38,492 are some of the things I also experienced myself back then, 553 00:33:38,492 --> 00:33:41,180 right? One concern is always about, you 554 00:33:41,180 --> 00:33:43,886 know, like let's say the management always think, okay, 555 00:33:43,886 --> 00:33:45,650 this tool is not compliant, not secure. 556 00:33:45,868 --> 00:33:48,652 The cost is pretty high. But every time I see new 557 00:33:48,652 --> 00:33:51,028 technologies coming. Obviously they don't put a lot 558 00:33:51,028 --> 00:33:53,776 of focus on this area first. Because like if you think about 559 00:33:53,776 --> 00:33:56,455 open source, it's more about functionality first, features, 560 00:33:56,455 --> 00:33:58,288 developer experience first maybe. 561 00:33:58,528 --> 00:34:02,334 So tell us how for engineers who build these solutions to have 562 00:34:02,334 --> 00:34:05,424 this mindset about compliance, security, and maybe cost as part 563 00:34:05,424 --> 00:34:09,467 of their design as well. Yeah, it's an excellent 564 00:34:09,467 --> 00:34:15,125 observation and I think, you know, I feel strongly that you 565 00:34:15,125 --> 00:34:21,405 really need to be clear about what your monetization strategy 566 00:34:21,405 --> 00:34:27,755 is going to be. And if you are really looking at 567 00:34:27,755 --> 00:34:33,551 sort of an enterprise market, you have to have the security, 568 00:34:33,551 --> 00:34:38,690 compliance pieces in mind. I think, security is like, these 569 00:34:38,690 --> 00:34:44,081 days it is a no-brainer, as in you really need to be thinking 570 00:34:44,081 --> 00:34:47,947 security first, right? Compliance is something that you 571 00:34:47,947 --> 00:34:52,679 could argue, you know, like SOC 2 compliance or these things can 572 00:34:52,679 --> 00:34:56,114 come as you mature. But your architecture, your 573 00:34:56,114 --> 00:35:00,376 design for the platform and the product you're building really 574 00:35:00,376 --> 00:35:05,200 needs to be very, very security aware and security first. 575 00:35:05,954 --> 00:35:10,682 And so for instance, one example I'll give you is Temporal, it is 576 00:35:10,682 --> 00:35:15,389 an open source tool. But we actually don't see any of 577 00:35:15,389 --> 00:35:18,089 the customer's code or customer's data. 578 00:35:18,329 --> 00:35:20,939 That all runs in the customer's infrastructure. 579 00:35:20,939 --> 00:35:25,985 And so that sort of design decision was made very, very 580 00:35:25,985 --> 00:35:30,419 early on, but has really resulted in much faster 581 00:35:30,419 --> 00:35:34,837 adoption, because it took a whole conversation off the table 582 00:35:34,837 --> 00:35:38,761 when it came to a using and buying decision. 583 00:35:39,466 --> 00:35:43,348 So my advice basically would be you have to think about security 584 00:35:43,348 --> 00:35:47,462 from the ground up. You have to make sure you're not 585 00:35:47,462 --> 00:35:51,496 going through any, what I'd call a one-way door on compliance. 586 00:35:51,496 --> 00:35:54,706 But you don't necessarily need to have all of the compliance 587 00:35:54,706 --> 00:36:00,869 pieces lined up early on. And on cost, it's again, really 588 00:36:00,869 --> 00:36:05,784 important to think about just how you're going to monetize. 589 00:36:05,784 --> 00:36:09,309 And, you know, at the end of the day, we know all these tools are 590 00:36:09,309 --> 00:36:11,589 competing for some share of budget. 591 00:36:11,589 --> 00:36:16,067 And being very clear on the value your tool brings versus 592 00:36:16,067 --> 00:36:20,615 the cost and how that lines up. And then, you know, also working 593 00:36:20,615 --> 00:36:23,414 hard towards lowering costs for developers, right? 594 00:36:24,431 --> 00:36:27,072 Yeah, so hopefully, you know, uh, everyone that is building, 595 00:36:27,072 --> 00:36:30,392 you know, this kind of tooling, developer tools and all that can 596 00:36:30,392 --> 00:36:32,084 also learn from this thing, right? 597 00:36:32,324 --> 00:36:35,352 So personally for compliance as well, I think it is, uh, useful 598 00:36:35,352 --> 00:36:38,290 if let's say you already know what are the controls that the, 599 00:36:38,290 --> 00:36:41,225 you know, whatever compliance certifications that you are 600 00:36:41,225 --> 00:36:44,026 aiming for, right? Like what are kind of controls 601 00:36:44,026 --> 00:36:46,481 that are coming. Think maybe how you can bridge 602 00:36:46,481 --> 00:36:48,063 the gap, right, in the future, right? 603 00:36:48,063 --> 00:36:50,862 Rather than having to retrofit, which is always difficult, 604 00:36:50,862 --> 00:36:53,450 especially if you're already running with customers and data 605 00:36:53,450 --> 00:36:56,277 and all that, right? So I think that is also a tip 606 00:36:56,277 --> 00:36:59,334 from me. Let's maybe dive deeper into 607 00:36:59,334 --> 00:37:02,350 Temporal itself, right? So sometimes, every few years, 608 00:37:02,350 --> 00:37:06,217 uh, we will hear about this new paradigm in the engineering 609 00:37:06,217 --> 00:37:10,357 world where it is kind of like new, novel, like the way for 610 00:37:10,357 --> 00:37:12,758 solving problems. So we can think of, I dunno, 611 00:37:12,758 --> 00:37:15,125 like cloud is one, infrastructure as code is one, 612 00:37:15,125 --> 00:37:17,125 right? Service mesh is probably another 613 00:37:17,125 --> 00:37:19,721 one. And I think Temporal is kind of 614 00:37:19,721 --> 00:37:21,793 like somewhere in this space as well. 615 00:37:21,823 --> 00:37:23,503 First of all is there's a paradigm shift. 616 00:37:23,773 --> 00:37:26,845 So maybe let me ask you like what kind of paradigm shift that 617 00:37:26,845 --> 00:37:29,533 peoples need to have whenever they think about Temporal. 618 00:37:30,031 --> 00:37:34,041 Yeah, yeah, absolutely. Uh, so you are absolutely right. 619 00:37:34,071 --> 00:37:39,605 Temporal is a paradigm shift. Essentially, the thing that is 620 00:37:39,605 --> 00:37:44,890 happening is today developers spend a lot of time worrying 621 00:37:44,890 --> 00:37:48,415 about their code, all of the error conditions. 622 00:37:48,535 --> 00:37:52,767 How do they make it reliable? And there are some studies, for 623 00:37:52,767 --> 00:37:56,461 instance, that would say, you know, anywhere between 40% to 624 00:37:56,461 --> 00:38:02,806 60% of the developer's time is on all of the kind of 625 00:38:02,806 --> 00:38:05,244 scaffolding versus the core business logic. 626 00:38:05,904 --> 00:38:10,084 And the paradigm shift with Temporal is that we take care of 627 00:38:10,084 --> 00:38:14,694 all of that for you. So basically we shield the 628 00:38:14,694 --> 00:38:19,804 developer from all of the complexities of running and 629 00:38:19,804 --> 00:38:25,230 building reliable code, and the developer can focus on their 630 00:38:25,230 --> 00:38:30,793 business logic. Now the way that we do this is 631 00:38:30,793 --> 00:38:37,192 that you code to our SDKs and we, the Temporal server, handles 632 00:38:37,192 --> 00:38:44,003 all of the task dispatching, retries, all of the state 633 00:38:44,003 --> 00:38:48,459 management, all these pieces that are often distributed over 634 00:38:48,459 --> 00:38:54,024 lots of systems and very hard to manage and like trace through. 635 00:38:54,418 --> 00:38:57,568 We do all of that for you. Sounds good, right? 636 00:38:57,568 --> 00:39:00,131 Doesn't it sound great? Yeah, I think it sounds too good 637 00:39:00,131 --> 00:39:03,050 to be true because, uh, I don't know like how many developers 638 00:39:03,050 --> 00:39:04,555 have built this kind of workflow. 639 00:39:04,555 --> 00:39:07,498 I would say if you are doing some kind of workflow, 640 00:39:07,498 --> 00:39:09,935 distributed systems, you definitely can relate to this 641 00:39:09,935 --> 00:39:12,379 problem, right? But if you are just building a 642 00:39:12,379 --> 00:39:13,915 CRUD monolith, probably it's less so. 643 00:39:14,228 --> 00:39:17,078 I'm quite curious when you mention about this kind of 644 00:39:17,078 --> 00:39:19,186 complexity, right? Because I actually want to 645 00:39:19,186 --> 00:39:20,992 borrow the model like service mesh. 646 00:39:20,992 --> 00:39:22,943 I think it's pretty good analogy as well. 647 00:39:23,093 --> 00:39:25,781 Because in the past we used to, you know, worry about 648 00:39:25,781 --> 00:39:29,549 networking, how do we call, you know, uh, other services out 649 00:39:29,549 --> 00:39:32,846 there, the retries, and also the back off, you know, exponential 650 00:39:32,846 --> 00:39:35,567 back off, and things like that. And we put that, bake that in 651 00:39:35,567 --> 00:39:37,582 the code, right? But with thing like service 652 00:39:37,582 --> 00:39:39,306 mesh, right? It is all transparent. 653 00:39:39,306 --> 00:39:42,024 You don't actually see it. And I see the same thing with 654 00:39:42,024 --> 00:39:44,547 Temporal as well, right? When you create your business 655 00:39:44,547 --> 00:39:47,972 logic, your workflow, you just focus on the business logic and 656 00:39:47,972 --> 00:39:50,798 you don't actually see all this other concerns, right? 657 00:39:50,948 --> 00:39:54,158 Like, be it the retries, be it the state management, and be the 658 00:39:54,158 --> 00:39:57,376 dispatching of the task. So this is, I think first of all 659 00:39:57,376 --> 00:40:00,483 is the most important thing. How do you actually do that, 660 00:40:00,483 --> 00:40:02,117 right? How do you actually make sure 661 00:40:02,117 --> 00:40:04,481 that developers can really focus on just business logic? 662 00:40:05,539 --> 00:40:09,367 Yeah. So you captured kind of the, the 663 00:40:09,367 --> 00:40:11,187 paradigm shift really, really well. 664 00:40:11,807 --> 00:40:17,072 How we do that is kind of, first I think I wanted to spend a 665 00:40:17,072 --> 00:40:19,845 minute on like the history of Temporal. 666 00:40:19,845 --> 00:40:22,991 So Temporal. Our two co-founders have been 667 00:40:22,991 --> 00:40:26,003 kinda working on this problem for a lot of years. 668 00:40:26,415 --> 00:40:30,689 They got together at Uber and sort of built this open source 669 00:40:30,689 --> 00:40:33,359 project that then became Temporal. 670 00:40:34,289 --> 00:40:38,929 And a lot of their sort of lessons around running 671 00:40:38,929 --> 00:40:43,409 distributed systems at scale are what Temporal solves for. 672 00:40:43,409 --> 00:40:47,470 So the main point here is that they've been thinking about this 673 00:40:47,470 --> 00:40:54,252 problem for about 15, 20 years. So it's not an easy problem to 674 00:40:54,252 --> 00:40:58,936 solve and, um, it's definitely one where it has been production 675 00:40:58,936 --> 00:41:03,476 tested. Now how we do that is 676 00:41:03,476 --> 00:41:08,492 essentially we introduce some abstractions around what we call 677 00:41:08,492 --> 00:41:11,904 a workflow and activity and a task. 678 00:41:12,294 --> 00:41:16,650 And we, in this abstraction, essentially we create a 679 00:41:16,650 --> 00:41:21,666 programming model where the developer is just focusing on 680 00:41:21,666 --> 00:41:25,332 their business logic. They have some rules that they 681 00:41:25,332 --> 00:41:28,744 need to kind of abide with. So for instance, anything that 682 00:41:28,744 --> 00:41:31,944 is error prone, they would put in an activity. 683 00:41:32,620 --> 00:41:37,750 And that these abstractions then let us do things like retries. 684 00:41:37,824 --> 00:41:42,839 And we, we save the state of kind of the execution of that 685 00:41:42,839 --> 00:41:46,589 workflow. So we know at any given point in 686 00:41:46,589 --> 00:41:50,859 time where that code is. And what's like passed and 687 00:41:50,859 --> 00:41:54,142 what's coming next. And so if something were to 688 00:41:54,142 --> 00:41:58,840 crash, we have a concept called replay where we can go in and 689 00:41:58,840 --> 00:42:01,852 run again. And this, you know, I'm 690 00:42:01,852 --> 00:42:06,088 simplifying this down quite a bit, but you know, there's a lot 691 00:42:06,088 --> 00:42:08,651 of power in the programming model. 692 00:42:09,055 --> 00:42:13,300 But this is kind of the core of the paradigm shift is there's a 693 00:42:13,300 --> 00:42:16,371 few abstractions that the developers need to learn and 694 00:42:16,371 --> 00:42:21,805 then we take care of the rest. And very frequently what we hear 695 00:42:21,805 --> 00:42:25,953 developers say is that like understanding that paradigm 696 00:42:25,953 --> 00:42:29,062 shift was a little bit of effort. 697 00:42:29,062 --> 00:42:33,682 Because their natural instinct is, oh, where is my, like, if 698 00:42:33,892 --> 00:42:39,308 error do x kind of block, right? But once they get it, then they 699 00:42:39,308 --> 00:42:41,602 say that they can't ever look back. 700 00:42:41,659 --> 00:42:45,582 It's just such an elegant model that then they are like Temporal 701 00:42:45,582 --> 00:42:51,031 developers for life. Yeah, so I remember back then I, 702 00:42:51,031 --> 00:42:54,972 whenever I dealt with this kind of new programming model, right? 703 00:42:54,972 --> 00:42:58,002 So I've dealt for example, like for example, Hadoop, right? 704 00:42:58,062 --> 00:43:00,906 Dataflow, which is, uh, having also Apache Beam programming 705 00:43:00,906 --> 00:43:04,083 model and also all that. Definitely the first key 706 00:43:04,083 --> 00:43:06,168 difficulties is understanding the model itself. 707 00:43:06,498 --> 00:43:10,218 And it depends on how good documentation is, how intuitive 708 00:43:10,218 --> 00:43:11,988 the, you know, programming model is. 709 00:43:12,259 --> 00:43:13,649 There's always this hurdle, right? 710 00:43:13,889 --> 00:43:17,237 So maybe from your experience looking at your customers or 711 00:43:17,237 --> 00:43:21,017 maybe developer adoption of the open source version, do you 712 00:43:21,017 --> 00:43:24,524 think Temporal programming model and SDK is something that is 713 00:43:24,524 --> 00:43:28,896 difficult to learn? It's not difficult to learn, but 714 00:43:28,896 --> 00:43:33,456 it is difficult to unlearn the patterns from wherever you came 715 00:43:33,456 --> 00:43:37,937 from. And what I mean by that is what 716 00:43:37,937 --> 00:43:42,764 we see is that developers kind of go through like a journey 717 00:43:42,764 --> 00:43:47,331 where at some point they don't believe it and then there's an 718 00:43:47,331 --> 00:43:49,799 aha. And then they're like, oh my 719 00:43:49,799 --> 00:43:51,941 God, do you know, is this for real? 720 00:43:51,941 --> 00:43:56,099 And then they'll try it. And once they really kind of 721 00:43:56,099 --> 00:44:00,612 understand it, then there is a tremendous amount of excitement. 722 00:44:00,612 --> 00:44:05,042 And you know, we've had someone come up to me at a conference 723 00:44:05,042 --> 00:44:07,352 and say, I couldn't sleep that night. 724 00:44:07,702 --> 00:44:11,321 The moment I finally understood Temporal, I was so excited, 725 00:44:11,321 --> 00:44:14,719 right? So the, you know, the, the 726 00:44:14,719 --> 00:44:18,154 programming model itself is actually really simple, really 727 00:44:18,154 --> 00:44:20,879 elegant. But what typically happens is 728 00:44:20,879 --> 00:44:26,927 developers coming in who are new to Temporal expect to kind of 729 00:44:26,927 --> 00:44:32,110 have to do all of this work around sort of reliability and 730 00:44:32,210 --> 00:44:35,000 unlearning those habits, take some time. 731 00:44:35,552 --> 00:44:38,730 One thing that we are trying to do, and we've put a lot of focus 732 00:44:38,730 --> 00:44:43,442 on, is really, do more community education. 733 00:44:43,442 --> 00:44:47,966 We have a Slack channel where our community joins in and 734 00:44:47,966 --> 00:44:51,469 shares what they've built. We've got something called 735 00:44:51,469 --> 00:44:55,119 community exchange where developers can kind of share the 736 00:44:55,119 --> 00:44:58,983 code that they've built and like linked to their GitHub repo, et 737 00:44:58,983 --> 00:45:01,591 cetera. So we're really working towards 738 00:45:01,591 --> 00:45:06,439 just showing more code and doing more sort of education around 739 00:45:06,439 --> 00:45:09,829 the model as well. Right. 740 00:45:09,889 --> 00:45:11,959 That's very interesting insight that you mentioned, right? 741 00:45:11,959 --> 00:45:15,083 It's much more difficult to unlearn, uh, what we know. 742 00:45:15,200 --> 00:45:18,757 Especially for maybe senior, more senior developers who have 743 00:45:18,757 --> 00:45:21,197 had battle scars of building such system, right? 744 00:45:21,197 --> 00:45:24,563 So maybe for people who cannot relate, if I can imagine some of 745 00:45:24,563 --> 00:45:25,913 the problems that we typically have. 746 00:45:26,003 --> 00:45:28,793 Whenever you have like a business workflow with states, 747 00:45:28,793 --> 00:45:31,368 for example, right? It could be very simple, you 748 00:45:31,368 --> 00:45:34,493 know, three states, for example, start doing and stop, right? 749 00:45:34,643 --> 00:45:37,723 And if it involves like kind of like events and multiple 750 00:45:37,723 --> 00:45:40,535 distributed systems, it's always very difficult to build some 751 00:45:40,535 --> 00:45:43,677 kind of reliability between these transitions and calling 752 00:45:43,677 --> 00:45:45,867 different workflows from other things. 753 00:45:46,212 --> 00:45:49,062 We used to have like messaging system with DLQ. 754 00:45:49,542 --> 00:45:51,342 And then the DLQ you need to take care of it. 755 00:45:51,342 --> 00:45:54,202 How do you know that the messages, uh, is there, 756 00:45:54,202 --> 00:45:57,277 somebody's picking up? So all this seems to be 757 00:45:57,277 --> 00:45:58,897 magically handled by Temporal, right? 758 00:45:59,275 --> 00:46:01,002 So I think that's a very big thing. 759 00:46:01,002 --> 00:46:03,029 I haven't actually played with Temporal. 760 00:46:03,059 --> 00:46:05,759 Uh, but I can actually imagine like these things will 761 00:46:05,759 --> 00:46:08,641 definitely be something that is great for developers to work 762 00:46:08,641 --> 00:46:11,951 with because simply because it removes away all these 763 00:46:11,951 --> 00:46:14,903 unnecessary complexities, the boiler plates, the things that 764 00:46:14,903 --> 00:46:17,633 we don't like to do, right? And plus, when you mentioned 765 00:46:17,633 --> 00:46:20,908 about, persisting the states and all that, it's very, I assume 766 00:46:20,908 --> 00:46:24,234 it's very easy to troubleshoot. Um, because if, let's say you 767 00:46:24,234 --> 00:46:28,031 don't have this, you only rely on logs and we know that reading 768 00:46:28,031 --> 00:46:30,493 logs is not fun and it's always, uh, very difficult. 769 00:46:31,330 --> 00:46:33,100 Yes. Absolutely, yes. 770 00:46:33,373 --> 00:46:36,749 Yeah, yeah. So I think, um, if people are 771 00:46:36,749 --> 00:46:39,965 sold to kind of, you know, new programming model and paradigm, 772 00:46:39,965 --> 00:46:42,278 right? So what do you think are some of 773 00:46:42,278 --> 00:46:45,260 the first step that they can do to actually try Temporal. 774 00:46:45,892 --> 00:46:48,437 Yeah. So a few things. 775 00:46:48,531 --> 00:46:51,021 One is definitely join our community. 776 00:46:51,471 --> 00:46:54,771 Uh, second is sign up for Temporal Cloud. 777 00:46:55,216 --> 00:46:59,008 For signup, we have like credits that we give to developers and 778 00:46:59,008 --> 00:47:03,872 we have like a quick start guide that helps you get set up and 779 00:47:03,872 --> 00:47:08,032 run samples, et cetera. And I think the third thing 780 00:47:08,032 --> 00:47:12,528 would be there's a lot of talks that our customers, developers 781 00:47:12,528 --> 00:47:17,112 who have built on Temporal have done at our conferences. 782 00:47:17,112 --> 00:47:21,642 So, you know, just listen to like another developer talking 783 00:47:21,642 --> 00:47:23,982 about how they're using Temporal. 784 00:47:24,372 --> 00:47:27,975 And that really, I think, will help you get started because 785 00:47:27,975 --> 00:47:32,975 then you can start sort of envisioning how you could make, 786 00:47:32,975 --> 00:47:36,449 bring Temporal to the problems you have at hand. 787 00:47:37,257 --> 00:47:41,820 One of the other things that we didn't talk about, but the we 788 00:47:41,820 --> 00:47:46,850 hear a lot is not just that development teams can go faster. 789 00:47:46,884 --> 00:47:50,664 Uh, so for instance, we've had customers tell us that they've 790 00:47:50,664 --> 00:47:54,198 been able to get like 6x developer productivity because 791 00:47:54,198 --> 00:47:57,734 all of that boilerplate, like all of that is gone. 792 00:47:58,254 --> 00:48:05,071 But also this aspect of like peace of mind and knowing that, 793 00:48:05,071 --> 00:48:08,549 you know, reliability is taken care of. 794 00:48:08,579 --> 00:48:11,339 And, uh, you know, failures happen. 795 00:48:11,399 --> 00:48:15,179 We know that in any distributed system, failures will happen. 796 00:48:15,179 --> 00:48:18,944 But recovering from those failures, Temporal will handle 797 00:48:18,944 --> 00:48:23,357 that for you. So that element of like how much 798 00:48:23,357 --> 00:48:29,474 peace of mind this brings to the teams that are on-call is also a 799 00:48:29,474 --> 00:48:33,593 really significant part of the value we believe Temporal 800 00:48:33,593 --> 00:48:36,119 brings. Yeah, thanks for the plug, 801 00:48:36,119 --> 00:48:37,850 right? So definitely, uh, recovering 802 00:48:37,850 --> 00:48:40,522 from failures, again, it's not fun thing. 803 00:48:40,612 --> 00:48:43,828 And I think the term that you guys use is called durable 804 00:48:43,828 --> 00:48:46,792 execution, right? So you just think I submit an 805 00:48:46,792 --> 00:48:50,536 execute, uh, execution, right? And we just think that it will 806 00:48:50,536 --> 00:48:53,564 happen sometime, you know, like, um, especially if the system, 807 00:48:53,564 --> 00:48:55,702 you know, doesn't crash all the time, right? 808 00:48:55,702 --> 00:48:59,244 I assume if the system can recover, I think the execution 809 00:48:59,244 --> 00:49:01,649 will get guarantees that it will be executed, right? 810 00:49:01,806 --> 00:49:03,822 Yes, yes. So maybe a little bit about this 811 00:49:03,822 --> 00:49:06,375 term. How do you come up with this 812 00:49:06,375 --> 00:49:09,027 concept, durable execution? Maybe if I don't explain it 813 00:49:09,027 --> 00:49:11,414 correctly. Yeah, no, uh, you explained it 814 00:49:11,414 --> 00:49:14,948 really well. And I think essentially, what we 815 00:49:14,948 --> 00:49:19,982 were working towards is how do we describe the concept. 816 00:49:19,982 --> 00:49:24,581 Because the, you know, in the early days of Temporal, the 817 00:49:24,581 --> 00:49:29,091 natural instinct for developers is to try and compare you with 818 00:49:29,091 --> 00:49:32,137 technology they know, right? So, are you a message bus? 819 00:49:32,137 --> 00:49:35,958 No. Are you, you know, like a Visual 820 00:49:35,958 --> 00:49:38,236 Studio type thing? No, right? 821 00:49:38,236 --> 00:49:44,131 And so durable execution for us, really sort of encapsulates the 822 00:49:44,131 --> 00:49:49,801 core value that once you've built your code using the 823 00:49:49,801 --> 00:49:53,521 Temporal model, you know, we will execute it for you. 824 00:49:53,521 --> 00:49:58,321 Like that durability guarantee is really, really very strong. 825 00:49:58,718 --> 00:50:02,561 In fact, Henry, there's really fun blog and video on our 826 00:50:02,561 --> 00:50:06,340 website. What we did was we sent our 827 00:50:06,340 --> 00:50:11,026 mascot with a Raspberry Pi running a Temporal workflow to 828 00:50:11,026 --> 00:50:14,656 space. And what you can see is the 829 00:50:14,656 --> 00:50:19,573 workflow is executing and at some point, it stops because the 830 00:50:19,573 --> 00:50:24,206 Pi loses connection. And then after a little bit of 831 00:50:24,206 --> 00:50:28,856 time, it starts again because the Pi has come back into orbit 832 00:50:28,856 --> 00:50:31,166 and has picked up connection again. 833 00:50:31,166 --> 00:50:36,316 So we do these fun things as well around helping, to show 834 00:50:36,316 --> 00:50:41,456 very visually in a very like real way, like how seriously we 835 00:50:41,456 --> 00:50:43,591 take that durable execution guarantee. 836 00:50:44,990 --> 00:50:48,076 Wow, I assume that's a very, uh, fun video, right? 837 00:50:48,076 --> 00:50:49,726 Definitely we'll put that in the show notes. 838 00:50:50,002 --> 00:50:52,814 When you mentioned that, I also remember when I studied Temporal 839 00:50:52,814 --> 00:50:55,804 docs, uh. So you can even imagine like a 840 00:50:55,804 --> 00:50:58,364 scheduled task that is sometime in years ahead. 841 00:50:58,394 --> 00:51:01,364 Let's say you have a contract, multi-years and you have to 842 00:51:01,364 --> 00:51:03,639 schedule something that runs whenever, I dunno, they wanna 843 00:51:03,639 --> 00:51:06,898 renew or something like that. You can also submit it reliably 844 00:51:06,898 --> 00:51:10,622 to Temporal, and it will make sure that at that point in time 845 00:51:10,622 --> 00:51:12,794 it'll execute. So I think again, like building 846 00:51:12,794 --> 00:51:14,714 a scheduling thing is not fun, right? 847 00:51:14,714 --> 00:51:16,034 Especially for developers, right? 848 00:51:16,034 --> 00:51:19,154 How do you ensure it's reliable and you still think about like 849 00:51:19,154 --> 00:51:22,454 when it gets executed or not. So I think that's a pretty 850 00:51:22,454 --> 00:51:23,807 interesting thing. Yeah. 851 00:51:23,834 --> 00:51:27,234 Maybe one concern, yeah, one concern that people have, uh, 852 00:51:27,234 --> 00:51:29,370 whenever they adopt this new programming model, especially 853 00:51:29,370 --> 00:51:32,804 when it comes with a set of infrastructure, is the locked 854 00:51:32,804 --> 00:51:35,884 in, right? So especially when you are 855 00:51:35,884 --> 00:51:39,083 embedded into this programming model, which is probably not so 856 00:51:39,083 --> 00:51:41,753 much common these days. So for example, people talk 857 00:51:41,753 --> 00:51:44,684 about cloud locked in, right? Even though in cloud you have 858 00:51:44,684 --> 00:51:47,188 many different options these days and they're kind of like 859 00:51:47,188 --> 00:51:49,098 interchangeable, right? So tell us about these, 860 00:51:49,098 --> 00:51:51,476 probably, I dunno, concerns that people have about lock in. 861 00:51:51,760 --> 00:51:54,753 How do you ensure that their code can still be portable if 862 00:51:54,753 --> 00:51:57,399 let's say one day Temporal is not the solution for them? 863 00:51:58,002 --> 00:52:00,599 Yeah. It's a very real concern. 864 00:52:00,599 --> 00:52:02,129 And thank you for bringing it up. 865 00:52:02,129 --> 00:52:09,267 I think the way that we sort of really live by our values is 866 00:52:09,267 --> 00:52:11,936 around the open source commitment, right? 867 00:52:11,936 --> 00:52:17,715 So essentially, our guarantee to a developer who's building using 868 00:52:17,715 --> 00:52:23,631 the Temporal SDK is that they can run their code on the open 869 00:52:23,631 --> 00:52:28,159 source sort of server that they could be running in their own 870 00:52:28,159 --> 00:52:31,957 infrastructure or they could bring it to cloud without any 871 00:52:31,957 --> 00:52:36,008 code changes. And so we kind of guarantee that 872 00:52:36,008 --> 00:52:40,408 sort of code compatibility between Temporal Cloud and open 873 00:52:40,408 --> 00:52:44,259 source. And so, you know, from a vendor 874 00:52:44,259 --> 00:52:48,335 lock-in point of view, you are honestly kind of, the 875 00:52:48,335 --> 00:52:50,080 programming model is sticky, right? 876 00:52:50,220 --> 00:52:53,647 So if you wanna use a different programming model, you will have 877 00:52:53,647 --> 00:52:58,163 to go change your code. But you don't have to rely on 878 00:52:58,163 --> 00:53:02,308 Temporal the company, because the server is available in open 879 00:53:02,308 --> 00:53:05,524 source. And, you know, we have a very 880 00:53:05,524 --> 00:53:09,008 strong sort of commitment to code compatibility between cloud 881 00:53:09,008 --> 00:53:12,963 and the open source server. Yeah, that sounds really great, 882 00:53:12,963 --> 00:53:14,735 right? Especially like in the past I 883 00:53:14,735 --> 00:53:16,445 also have this experience with Apache Beam. 884 00:53:16,445 --> 00:53:17,795 I dunno how much you know about it. 885 00:53:17,795 --> 00:53:21,929 Like it's a programming model, uh, runs first class on Google 886 00:53:21,929 --> 00:53:24,629 Cloud Dataflow, right, the product for this data pipelines. 887 00:53:24,869 --> 00:53:27,487 But they also encourage other, you know, they call it runners 888 00:53:27,487 --> 00:53:30,024 back then, right? Other open source project like 889 00:53:30,024 --> 00:53:32,575 Spark and few other, Flink, for example. 890 00:53:32,575 --> 00:53:35,169 A few other things can also run the same code with the same 891 00:53:35,169 --> 00:53:37,648 programming model and it can still be run, right? 892 00:53:37,648 --> 00:53:41,300 So people don't have a fear of being locked in and especially 893 00:53:41,300 --> 00:53:44,569 if there's an open source solution that, uh, you know, 894 00:53:44,569 --> 00:53:47,257 people can use, right? So definitely it's something 895 00:53:47,257 --> 00:53:49,977 that is good that you, uh, uncover, right? 896 00:53:49,977 --> 00:53:52,639 I guess because people always have this concern about locked 897 00:53:52,639 --> 00:53:54,753 in. And what I'm concerned most is 898 00:53:54,753 --> 00:53:57,573 for those people who have unlearned that, you know, their 899 00:53:57,573 --> 00:54:00,099 previous habits. They got this aha moment. 900 00:54:00,413 --> 00:54:02,795 They kind of like locked into that paradigm shift, right? 901 00:54:02,795 --> 00:54:04,625 So I think that is more concerning for me. 902 00:54:04,715 --> 00:54:07,823 Uh, especially for developers, we don't like to go back and do 903 00:54:07,823 --> 00:54:09,852 all these mundane tasks. Yeah, absolutely. 904 00:54:09,875 --> 00:54:14,196 So maybe as we, yeah, as we go out, uh, go to the our, almost 905 00:54:14,196 --> 00:54:16,380 wrapping up the conversation, is there anything else about 906 00:54:16,380 --> 00:54:19,329 Temporal or about the developer report that you think, uh, you 907 00:54:19,329 --> 00:54:23,862 wanna cover as well? Yeah, I think mainly, you know, 908 00:54:23,862 --> 00:54:28,740 I wanna just reiterate our commitment is really to make 909 00:54:28,740 --> 00:54:33,065 developers' lives easier. To help them just focus on their 910 00:54:33,065 --> 00:54:38,105 business logic and not have to worry about all of this sort of 911 00:54:38,105 --> 00:54:42,243 really complex, messy, sort of pieces we've been talking about. 912 00:54:42,790 --> 00:54:48,118 We're committed to open source. We are available under the MIT 913 00:54:48,118 --> 00:54:50,518 license. We have a really strong 914 00:54:50,518 --> 00:54:53,333 community that is growing quite a bit. 915 00:54:53,663 --> 00:54:57,457 One thing we didn't touch is that our focus on developers is 916 00:54:57,457 --> 00:55:03,207 so incredibly strong that we build our SDKs across multiple 917 00:55:03,207 --> 00:55:06,970 languages. So we have Python, TypeScript, 918 00:55:06,970 --> 00:55:12,050 Java, Go, really sort of, .NET as well. 919 00:55:12,130 --> 00:55:15,990 Our focus is, the way we think about it is we wanna meet the 920 00:55:15,990 --> 00:55:20,812 developers where they are and we're truly driven by making 921 00:55:20,812 --> 00:55:24,010 their lives better and more productive. 922 00:55:24,690 --> 00:55:29,340 On the Temporal Cloud side, we are available in like multiple 923 00:55:29,340 --> 00:55:33,770 regions across the globe. Uh, so we have a number of 924 00:55:33,770 --> 00:55:36,330 regions in the Asia Pacific area as well. 925 00:55:36,902 --> 00:55:42,662 And our goal with Temporal Cloud is to run the Temporal service 926 00:55:42,662 --> 00:55:47,135 so that you don't have to worry about the reliability of the 927 00:55:47,135 --> 00:55:50,688 Temporal service itself, right? And so again, you know, go check 928 00:55:50,688 --> 00:55:53,000 it out. Become a part of the community. 929 00:55:53,000 --> 00:55:57,698 And the best way to learn is by just sort of looking at samples 930 00:55:57,698 --> 00:56:00,968 and listening to others. 'Cause, you know, Henry, you and 931 00:56:00,968 --> 00:56:04,643 I know I can stand here and I can talk about Temporal, but 932 00:56:04,643 --> 00:56:09,176 having someone who's used it from outside the company is 933 00:56:09,176 --> 00:56:14,420 always a much stronger indicator of the strength of the platform, 934 00:56:14,420 --> 00:56:16,353 right? Right. 935 00:56:16,713 --> 00:56:19,645 And also you just, uh, whenever you, when you explain that, you 936 00:56:19,645 --> 00:56:21,778 just remind me about the compliance piece that you 937 00:56:21,778 --> 00:56:22,953 mentioned earlier as well, right? 938 00:56:22,953 --> 00:56:26,282 How do you ensure Temporal, especially the Cloud part, is 939 00:56:26,282 --> 00:56:29,426 compliant and secure? And I can see actually banks are 940 00:56:29,426 --> 00:56:30,641 starting to adopt Temporal, right? 941 00:56:30,641 --> 00:56:34,043 I know like it's always very difficult to have banks adopting 942 00:56:34,043 --> 00:56:35,438 Temporal Cloud. Right. 943 00:56:35,711 --> 00:56:38,795 So tell us about this important piece of compliance and security 944 00:56:38,795 --> 00:56:40,610 aspect that people don't know about, right? 945 00:56:40,610 --> 00:56:42,647 It's always unintuitive, yeah. Yeah. 946 00:56:43,107 --> 00:56:46,760 So I think the first piece to understand is that when you 947 00:56:46,760 --> 00:56:52,296 build your code using our SDKs, you are building what we call a 948 00:56:52,296 --> 00:56:56,044 worker, and the worker is running in your infrastructure. 949 00:56:56,571 --> 00:57:01,419 We don't see your code. What we are doing on the 950 00:57:01,419 --> 00:57:05,019 Temporal Cloud side is essentially dispatching the 951 00:57:05,019 --> 00:57:09,759 tasks and the retries. One way I like to talk about it 952 00:57:09,759 --> 00:57:14,123 is think about a puppet master who is sort of, you know, 953 00:57:14,123 --> 00:57:18,138 directing all of the puppets, but we aren't running the 954 00:57:18,138 --> 00:57:21,077 puppets in our environment, that's in yours. 955 00:57:21,077 --> 00:57:25,115 And we are sort of the brains that is orchestrating all of 956 00:57:25,115 --> 00:57:28,037 these pieces. So that immediately kind of 957 00:57:28,037 --> 00:57:33,073 overcomes a lot of the questions around, do we see your data, do 958 00:57:33,073 --> 00:57:35,441 we see your code? And the answer is no. 959 00:57:36,191 --> 00:57:39,858 And then the sort of task dispatching pieces obviously 960 00:57:39,858 --> 00:57:43,603 need connection that would typically go over something like 961 00:57:43,603 --> 00:57:47,309 a private link. And then on the Temporal Cloud 962 00:57:47,309 --> 00:57:49,861 side, you know, we are SOC 2 certified. 963 00:57:50,217 --> 00:57:54,510 We have had some requirements from customers around having 964 00:57:54,510 --> 00:57:59,106 kind of their disaster recovery and primary region in the same 965 00:57:59,106 --> 00:58:02,797 country. We can sort of build a region 966 00:58:02,797 --> 00:58:04,847 wherever the cloud providers exist. 967 00:58:04,847 --> 00:58:06,967 So that's been pretty easy for us. 968 00:58:07,640 --> 00:58:11,851 And I think this is the kind of the elegance of the model around 969 00:58:11,851 --> 00:58:17,051 security that has really enabled us to move much faster and make 970 00:58:17,051 --> 00:58:21,651 inroads in banks using Temporal Cloud pretty quickly as well. 971 00:58:22,700 --> 00:58:25,240 Yeah, pretty exciting. So I think, uh, if people hear 972 00:58:25,240 --> 00:58:27,734 about this conversation, maybe they think it's too good to be 973 00:58:27,734 --> 00:58:30,614 true, but I would invite people to just learn from the links, 974 00:58:30,614 --> 00:58:32,595 the community that Preeti has mentioned, right? 975 00:58:32,895 --> 00:58:35,894 Because, uh, I'm sure this requires a much dive deep, kind 976 00:58:35,894 --> 00:58:38,318 of like discussions, maybe the study, right? 977 00:58:38,325 --> 00:58:40,788 Before you actually understand, what Temporal is. 978 00:58:41,568 --> 00:58:44,238 So Preeti, I think it's a pretty exciting talk, right? 979 00:58:44,238 --> 00:58:47,234 Uh, unfortunately we have to wrap up the call pretty soon. 980 00:58:47,534 --> 00:58:50,144 But before I let you go, I would like to ask you one question. 981 00:58:50,144 --> 00:58:51,944 This is like a tradition in my podcast. 982 00:58:52,244 --> 00:58:54,674 I call this the 3 Technical Leadership Wisdom. 983 00:58:54,974 --> 00:58:57,794 You can think of it just like advice you want to give to us. 984 00:58:57,884 --> 00:58:59,974 Maybe if you can share your version today that will be 985 00:58:59,974 --> 00:59:01,802 great. Absolutely. 986 00:59:01,862 --> 00:59:04,811 I'd love to. And you said three, right, 987 00:59:04,811 --> 00:59:05,840 Henry? Yeah. 988 00:59:06,162 --> 00:59:07,610 That's correct, yeah. Okay. 989 00:59:07,820 --> 00:59:12,734 So I think my first one would be, you know, really being clear 990 00:59:12,734 --> 00:59:16,166 about the problem you're trying to solve. 991 00:59:16,936 --> 00:59:22,186 And think deeply and from a first principal's viewpoint. 992 00:59:22,640 --> 00:59:29,090 You are really sort of solving the problem in a way that's very 993 00:59:29,090 --> 00:59:32,586 impactful. And really hold that high bar 994 00:59:32,586 --> 00:59:35,978 around the problem and how you solve it. 995 00:59:35,978 --> 00:59:40,751 I think that would be one. I think the second piece of 996 00:59:40,751 --> 00:59:44,431 wisdom would be don't shy away from hard things. 997 00:59:44,431 --> 00:59:48,521 You know, sometimes it's easy to take shortcuts, but, you know, 998 00:59:49,171 --> 00:59:54,314 we're in this for the long term. Solve the hard problems and 999 00:59:54,314 --> 01:00:00,238 really don't sort of lose track of that long term piece that 1000 01:00:00,238 --> 01:00:03,336 you're building. And then I think the third 1001 01:00:03,336 --> 01:00:06,444 thing, and I know you said technical, but I think this is 1002 01:00:06,444 --> 01:00:10,969 very related, is the team that you have and the people you work 1003 01:00:10,969 --> 01:00:15,559 with, they are critical in solving any technology problem. 1004 01:00:16,009 --> 01:00:21,559 Your engineering team, the talent, is your biggest sort of 1005 01:00:21,559 --> 01:00:24,436 asset. It's how you can solve these 1006 01:00:24,436 --> 01:00:26,911 problems. So make sure that you are hiring 1007 01:00:26,911 --> 01:00:30,633 well, you're taking care of your team and your engineers, and 1008 01:00:30,633 --> 01:00:35,313 you're motivating them and helping them understand the big 1009 01:00:35,313 --> 01:00:38,383 picture and empowering them to have an impact. 1010 01:00:39,352 --> 01:00:41,182 Yeah, I would say it's pretty relevant, right? 1011 01:00:41,182 --> 01:00:43,218 The last one, right? So I think it's pretty beautiful 1012 01:00:43,218 --> 01:00:44,722 especially. When these days, when people 1013 01:00:44,722 --> 01:00:47,412 think about, you know, maybe replacing more people with the 1014 01:00:47,412 --> 01:00:49,006 AI. But I think engineering team is 1015 01:00:49,006 --> 01:00:51,974 still kind of like, you know, requires a lot of people, uh, a 1016 01:00:51,974 --> 01:00:54,514 lot of innovation, creativity, and ownership from people as 1017 01:00:54,514 --> 01:00:55,994 well, right? So to get it right. 1018 01:00:56,604 --> 01:00:59,554 So if people, uh, love this conversation, they would like to 1019 01:00:59,554 --> 01:01:00,994 reach out to you, learn from you. 1020 01:01:01,054 --> 01:01:03,304 Is there a place where they can find you online, Preeti? 1021 01:01:04,157 --> 01:01:06,779 Yeah. Uh, in our community Slack, I am 1022 01:01:06,779 --> 01:01:09,227 in there if you could find me there. 1023 01:01:09,587 --> 01:01:13,697 Also on LinkedIn and Twitter. Uh, so absolutely come find me. 1024 01:01:13,697 --> 01:01:15,912 I'd love to hear from you. All right. 1025 01:01:16,272 --> 01:01:18,231 Okay. Thank you so much for the, you 1026 01:01:18,231 --> 01:01:20,612 know, sharing today. Uh, I learned a lot of insights 1027 01:01:20,612 --> 01:01:23,062 from you, from your career, from the developer report, and 1028 01:01:23,062 --> 01:01:24,337 especially about Temporal as well. 1029 01:01:24,337 --> 01:01:26,007 So thank you so much for sharing today, Preeti. 1030 01:01:26,730 --> 01:01:29,802 Thank you, Henry. And I hope that your listeners 1031 01:01:29,802 --> 01:01:31,290 enjoy the discussion today.