1
00:00:00,090 --> 00:00:01,820
Welcome back to the Architecture
Corner. 

2
00:00:01,890 --> 00:00:05,400
This is part three of our five 
part series looking at improving

3
00:00:05,410 --> 00:00:07,540
application availability. 
That's right. 

4
00:00:07,710 --> 00:00:10,600
We've talked about redundancy in
compute and persistence. 

5
00:00:10,610 --> 00:00:14,780
Before and today we're shifting 
gears a bit, exploring 

6
00:00:14,790 --> 00:00:18,360
strategies that are maybe less 
complex, potentially more cost 

7
00:00:18,370 --> 00:00:20,220
effective. 
Exactly. 

8
00:00:20,290 --> 00:00:24,960
We're diving into graceful 
degradation and asynchronous 

9
00:00:24,970 --> 00:00:27,560
processing. 
The goal is understanding how 

10
00:00:27,570 --> 00:00:29,520
these can really boost 
availability. 

11
00:00:29,670 --> 00:00:32,110
A lot of these ideas come from 
some great work by Mario 

12
00:00:32,119 --> 00:00:33,810
Bittencourt. 
OK, so most people focus on 

13
00:00:33,820 --> 00:00:36,540
preventing failures, right? 
Yeah, throwing more hardware at 

14
00:00:36,550 --> 00:00:39,360
the problem maybe. 
Pretty much, but today it's 

15
00:00:39,370 --> 00:00:43,690
about a different angle, almost 
counterintuitive, how planning 

16
00:00:43,700 --> 00:00:46,110
for failure can make things more
resilient. 

17
00:00:46,160 --> 00:00:48,910
So how do we actually do that? 
How do we move beyond just 

18
00:00:48,960 --> 00:00:51,810
adding more servers and, you 
know, account for things 

19
00:00:51,820 --> 00:00:54,010
potentially going wrong in a 
smarter way? 

20
00:00:54,140 --> 00:00:55,810
Well, graceful degradation is 
key here. 

21
00:00:55,820 --> 00:00:57,190
It's a fundamentally different 
approach. 

22
00:00:57,200 --> 00:00:59,370
You see different hacks. 
Instead of trying to prevent 

23
00:00:59,440 --> 00:01:03,110
every single failure, which is 
often impossible or just too 

24
00:01:03,120 --> 00:01:06,990
expensive, you consciously plan 
for certain parts to fail. 

25
00:01:07,060 --> 00:01:10,790
OK, plan for failure. 
How does that help availability?

26
00:01:10,980 --> 00:01:13,630
It allows the application to 
keep running, maybe not 

27
00:01:13,640 --> 00:01:15,450
perfectly, but running 
nonetheless. 

28
00:01:15,840 --> 00:01:19,690
It continues operating, perhaps 
in a less optimal mode, by 

29
00:01:19,700 --> 00:01:22,020
prioritizing the really core 
functions. 

30
00:01:22,260 --> 00:01:25,080
OK, can you give an example? 
Sure. 

31
00:01:25,150 --> 00:01:27,740
Think about an ecommerce site. 
Yeah, let's say it relies on an 

32
00:01:27,750 --> 00:01:30,880
external fraud check service. 
Right retty, Common dependency. 

33
00:01:30,950 --> 00:01:33,600
Exactly. 
Now what if that fraud service 

34
00:01:33,610 --> 00:01:37,480
goes down temporarily? 
The default might be to just 

35
00:01:37,490 --> 00:01:41,080
stop the purchase, full stop. 
Which is a terrible customer 

36
00:01:41,090 --> 00:01:43,730
experience. 
Awful but with graceful 

37
00:01:43,740 --> 00:01:45,450
degradation. 
You might say OK the fraud 

38
00:01:45,460 --> 00:01:47,700
service is down. 
Instead of blocking the sale, 

39
00:01:47,710 --> 00:01:49,650
maybe you offer a sort of pre 
approval. 

40
00:01:49,720 --> 00:01:52,610
Ah, like let the purchase go 
through but flag it for a closer

41
00:01:52,620 --> 00:01:54,170
look later. 
Precisely. 

42
00:01:54,440 --> 00:01:58,290
Or maybe you apply a simpler 
internal risk rule temporarily. 

43
00:01:58,620 --> 00:02:01,870
It's not the ideal flow perhaps,
but the customer completes their

44
00:02:01,880 --> 00:02:04,170
purchase. 
It's way better than a complete 

45
00:02:04,180 --> 00:02:07,210
failure and you achieve this 
without massive over engineering

46
00:02:07,220 --> 00:02:09,020
costs. 
That makes a lot of sense. 

47
00:02:09,070 --> 00:02:11,090
So how would you actually 
implement that? 

48
00:02:11,280 --> 00:02:14,290
It takes a few steps really. 
First you've got to identify 

49
00:02:14,300 --> 00:02:17,360
your applications dependencies. 
What external or internal 

50
00:02:17,370 --> 00:02:20,560
services does it rely on? 
OK, and crucially, what happens 

51
00:02:20,570 --> 00:02:23,280
if they fail? 
What's the impact on specific 

52
00:02:23,290 --> 00:02:26,180
user journeys? 
You know, tools like BPMN 

53
00:02:26,190 --> 00:02:30,000
diagrams or even just good APM 
data can help map this out. 

54
00:02:30,450 --> 00:02:33,260
Visualize the connections and 
failure points. 

55
00:02:33,730 --> 00:02:37,640
Then Step 2, you need to talk to
the business side, the product 

56
00:02:37,650 --> 00:02:41,220
owners, the stakeholders to 
define what an acceptable 

57
00:02:41,230 --> 00:02:44,280
degraded experience actually is.
What's the fall back? 

58
00:02:44,400 --> 00:02:48,290
Is it OK to show slightly older 
data, use a default value, skip 

59
00:02:48,300 --> 00:02:49,190
an optional? 
Step, right? 

60
00:02:49,200 --> 00:02:51,370
So it's a business decision as 
much as a technical one. 

61
00:02:51,380 --> 00:02:53,650
Absolutely. 
And finally you implement that 

62
00:02:53,660 --> 00:02:57,210
fall back option and importantly
you protect it with something 

63
00:02:57,220 --> 00:03:00,260
like a circuit breaker pattern. 
Ah, the circuit breaker. 

64
00:03:00,270 --> 00:03:02,490
So that stops you from 
constantly hammering a service 

65
00:03:02,500 --> 00:03:04,030
that's already down. 
Exactly. 

66
00:03:04,040 --> 00:03:07,270
It detects the failure, trips 
open and directs traffic to your

67
00:03:07,280 --> 00:03:10,080
fall back logic. 
Then it periodically checks if 

68
00:03:10,090 --> 00:03:11,350
the main service is back. 
Up. 

69
00:03:11,500 --> 00:03:14,310
So what this all means is you're
essentially creating a safety 

70
00:03:14,320 --> 00:03:16,580
net. 
Yeah, allowing your application 

71
00:03:16,590 --> 00:03:21,670
to stay functional, keep serving
users even if some component is 

72
00:03:21,680 --> 00:03:23,990
having a bad day. 
That's a perfect way to put it, 

73
00:03:24,040 --> 00:03:26,650
a safety net. 
OK, that covers graceful 

74
00:03:26,660 --> 00:03:28,630
degradation. 
It gives us that safety net, 

75
00:03:28,640 --> 00:03:30,950
keeps things moving. 
But error? 

76
00:03:30,960 --> 00:03:34,650
What if that degraded experience
that fall back just isn't good 

77
00:03:34,660 --> 00:03:37,890
enough? 
What if using a default value or

78
00:03:37,900 --> 00:03:39,990
skipping a step is simply too 
risky? 

79
00:03:40,060 --> 00:03:43,090
That's a really important point.
Sometimes a default behavior 

80
00:03:43,100 --> 00:03:44,670
just isn't acceptable, full 
stop. 

81
00:03:44,680 --> 00:03:47,280
Right? 
And this is where asynchronous 

82
00:03:47,290 --> 00:03:49,220
execution often comes into the 
picture. 

83
00:03:49,290 --> 00:03:52,080
It's a different pattern. 
Asynchronous, so doing things 

84
00:03:52,090 --> 00:03:54,640
later, not right away. 
Essentially, yes. 

85
00:03:54,710 --> 00:03:57,500
It works by removing what we 
call temporal coupling. 

86
00:03:57,510 --> 00:04:00,200
Temporal coupling, Yeah, it's 
when two parts of a system are 

87
00:04:00,210 --> 00:04:03,560
rigidly tied together by time 
one process has to wait for 

88
00:04:03,570 --> 00:04:06,000
another to finish. 
Right now, asynchronous 

89
00:04:06,010 --> 00:04:09,420
processing breaks that 
dependency, often by using 

90
00:04:09,430 --> 00:04:12,350
messaging infrastructure. 
Things like message queues. 

91
00:04:12,710 --> 00:04:16,370
A task or request is placed on a
queue and some other process 

92
00:04:16,380 --> 00:04:18,950
picks it up and handles it later
at its own pace. 

93
00:04:19,000 --> 00:04:20,910
OK, let's tie this back to the 
fraud example. 

94
00:04:21,000 --> 00:04:24,600
If the business says Nope, we 
absolutely cannot pre approve 

95
00:04:24,610 --> 00:04:27,950
without a proper check. 
How does async help? 

96
00:04:28,010 --> 00:04:32,210
Well, if a default rule is too 
risky, you could make that fraud

97
00:04:32,220 --> 00:04:36,090
assessment asynchronous. 
So the customer clicks buy. 

98
00:04:36,140 --> 00:04:37,770
Right. 
The order is accepted 

99
00:04:37,780 --> 00:04:39,670
immediately. 
May be placed in a pending 

100
00:04:39,680 --> 00:04:42,090
review state. 
The request for the fraud check 

101
00:04:42,100 --> 00:04:44,370
is put onto a queue. 
And the customer gets 

102
00:04:44,380 --> 00:04:47,060
confirmation their order is 
placed blocked. 

103
00:04:47,110 --> 00:04:49,080
Exactly. 
The customer experience is 

104
00:04:49,090 --> 00:04:51,260
smooth. 
Even if the fraud service is 

105
00:04:51,270 --> 00:04:53,860
down at that exact moment, the 
order goes through from their 

106
00:04:53,870 --> 00:04:55,580
perspective. 
The actual fraud check happens 

107
00:04:55,590 --> 00:04:58,260
later when the service is back U
or a background worker picks it 

108
00:04:58,270 --> 00:05:00,400
up. 
So the high value operation, the

109
00:05:00,410 --> 00:05:03,580
check itself might be delayed 
internally, but it doesn't stop 

110
00:05:03,590 --> 00:05:08,120
the customer in their tracks. 
Precisely for those dependencies

111
00:05:08,130 --> 00:05:11,980
where you simply can't accept a 
default or degraded behavior, 

112
00:05:12,190 --> 00:05:16,060
making the interaction 
asynchronous can virtually 

113
00:05:16,070 --> 00:05:19,700
eliminate its failure from 
impacting the immediate user 

114
00:05:19,710 --> 00:05:21,560
experience. 
That's powerful. 

115
00:05:21,890 --> 00:05:23,820
It does raise a question though,
doesn't it? 

116
00:05:24,090 --> 00:05:27,000
For those really critical 
dependencies, you need to ask, 

117
00:05:27,390 --> 00:05:30,760
can we make this asynchronous? 
Can we defer its execution 

118
00:05:30,770 --> 00:05:32,800
without messing up the core 
business need? 

119
00:05:33,090 --> 00:05:34,640
That's the crucial question to 
ask. 

120
00:05:34,650 --> 00:05:39,060
Yes, it requires careful design.
So wrapping this up, graceful 

121
00:05:39,070 --> 00:05:41,890
degradation and asynchronous 
processing, they're like these 

122
00:05:41,900 --> 00:05:44,780
powerful levers you can pull. 
They let you maintain 

123
00:05:44,790 --> 00:05:48,200
availability, keep that customer
experience smooth, essentially 

124
00:05:48,270 --> 00:05:50,690
get planning for failure. 
Not just, you know, crossing 

125
00:05:50,700 --> 00:05:52,490
your fingers and hoping it 
doesn't happen. 

126
00:05:52,500 --> 00:05:55,030
You're really underscores that 
improving availability is a 

127
00:05:55,040 --> 00:05:58,130
complex journey, isn't it? 
These strategies are vital for 

128
00:05:58,140 --> 00:06:00,590
resilience, especially when 
you're dealing with dependencies

129
00:06:00,600 --> 00:06:03,670
you don't fully control. 
But yeah, it does make you 

130
00:06:03,680 --> 00:06:07,090
think, what if your application 
is truly mission critical? 

131
00:06:07,180 --> 00:06:09,820
I mean, where even these minor 
hiccups or delays we've 

132
00:06:09,830 --> 00:06:14,560
discussed are just unacceptable.
Ooh, that is a provocative 

133
00:06:14,570 --> 00:06:16,030
thought. 
Something for you all to Mull 

134
00:06:16,040 --> 00:06:18,360
over, and it's exactly what 
we'll be exploring next. 

135
00:06:18,370 --> 00:06:20,670
Time indeed. 
We'll be diving into disaster 

136
00:06:20,680 --> 00:06:24,130
recovery strategies and the 
whole concept of active active 

137
00:06:24,140 --> 00:06:25,970
solutions. 
Looking forward to that one. 

138
00:06:26,180 --> 00:06:28,850
For more information on today's 
topic and to keep up with all 

139
00:06:28,860 --> 00:06:31,370
our insights, please do check 
the description for details. 

140
00:06:31,500 --> 00:06:34,670
And don't forget to subscribe 
for free to the Architecture 

141
00:06:34,680 --> 00:06:37,970
Corner newsletter over at 
architecturecorner.substack.com.

142
00:06:38,040 --> 00:06:39,200
Thanks. 
For joining us, see you next 

143
00:06:39,210 --> 00:06:39,450
time.