Welcome back to the Deep Dive. Usually we take a wide angle
lens to a topic, you know, exploring the ecosystem, the
market dynamics, or even the philosophical implications.
But today we're doing something a little different.
This is our Practical AI Digest edition.
We are zooming in. Weigh in.
We're going past the headlines and straight into the
engineering. Exactly.
We aren't just going to talk about cogeneration in the
abstract. We are going to unpack
specifically how to build and I think maybe more importantly,
how to evaluate the models that build software for us.
And we have a perfect subject for that today, the.
Topic on the table is the engine that really launched this whole
era of AI assisted programming, Open AI's codecs.
This is the model that powers GitHub Copilot.
And you know, while it's really easy to just install the
extension and let it auto complete your loops, the actual
mechanism under the hood is, well, it's a fascinating case
study and machine learning adaptation.
So what are we basing this on? What's the source measure?
We're basing this deep dive on the primary research paper from
the Open AI team. It's called Evaluating Large
Language Models Trained on Code. We'll also pull in some context
on the launch from a great piece in Singularity Hub.
OK, so the mission for us today is to bridge that gap between it
works like magic and it works like math.
I like that. We really want you to understand
how a model translates natural language, you know, plain
English, into functional Python code.
And to do that, we have to look at 3 very specific areas.
Right, first the architecture. Specifically how it even reads
text. Then the metrics.
Why do standard language scores totally fail for code?
And finally, the tuning. How do you actually control the,
let's call it, creativity of the machine?
And I think for the developers listening or maybe anyone
managing a technical team, there's a key concept here we're
going to hit. Hard Oh yeah, the Passat collie
main metric it is absolutely crucial.
It really changes how you should think about the reliability of
these tools Well, it's not just about getting the right answer,
it's about the probability of generating the right answer
within a specific budget of demps.
It's a whole new way of thinking about correctness.
Right, let's start at the beginning, then the origin story
with rewind a bit back to GPT 3. The original text Titan.
When GPT 3 launched, it was the king of text.
It could write poetry, it could summarize emails.
It could generate marketing copy.
But was it a coder? Not really.
I mean not in any practical sense.
It was a generalist. So what was it trained on?
It was trained on the Common Crawl, which is basically a
scrape of the entire public Internet.
And the Internet has code on it. Oh, it has a lot of code.
If you script the web, you're going to get Stack Overflow.
You get technical blogs, you get random snippets and forums.
O GT3 definitely saw code, but it just saw it as another form
of text. O it didn't understand the like
the physics of code, the rules. Exactly, that's a perfect way to
put it. It could mimic the structure O.
If you asked it for a for loop, it would give you something that
looked like a for loop. But it wouldn't work.
Often not. The variables wouldn't quite
match, or the syntax would be just slightly off.
It was like a student who memorized the textbook glossary
but didn't actually understand the grammar or the logic flow.
It was producing text that looked like code rather than
functional instructions. Let's just say it was a mimic.
So the pivot to codex then wasn't just let's make GPT 3
bigger, it was something more fundamental.
Right, it was. Let's specialize.
They took the foundational GPT 3 model and they fine-tuned it.
They basically force fed it a massive diet of pure code.
Where'd they get all that code? Specifically, they gathered
publicly available code from GitHub.
We are talking about a data set that even after filtering came
out 259 gigabytes of unique Python files.
I want to pause on that word filtering because it sounds so
trivial, but in data science that's everything, isn't it?
It's the whole game, you can't just feed it every single repo
on GitHub, that would be a disaster.
Why? What's wrong with the data on
GitHub? Well, a lot of it is auto
generated. Think of those massive compiled
JavaScript files or configuration dumps.
It's repetitive and it's very low quality information.
It's noise. OK, so they stripped that out.
They stripped that out. They also removed files that had
super long lines, like over 1000 characters, because that's
usually just data storage, not actual logic.
Right, like a big Jason BLOB or something.
Exactly. And most importantly, they
filtered for quality to some extent, which we'll get into
later when we talk about the Codex S version.
But just the sheer volume here is the first big differentiator
from GPD 3. But here's where it gets really
interesting for the engineers listening.
You can't just feed code into a text model without changing how
the model reads, can you? No, you can't.
We have to talk about the tokenizer.
This is such a crucial piece of engineering that I think most
people just totally overlook. So what is a tokenizer,
honestly? Large language models.
They don't read words like cat or print, they read tokens,
which are just chunks of characters and the standard text
tokenizers like the one in Jeeple T3.
They're optimized for English sentences.
So they're really good at encoding common words like the
and, or, you know, suffixes like ING.
Exactly. They're very efficient at that,
but code isn't English. The whole distribution of
characters is completely different.
And in Python specifically, there's a very unique problem.
White space. Because in Python, the
indentation isn't just for looks, it's grammatical.
It actually defines the scope of the code.
Right, If you're using Java or C++ L, you have curly braces.
You can mess up the spacing all you want and the compiler
doesn't care. The code still runs.
But in Python, 4 spaces versus 2 spaces completely changes the
logic of the program. It's a different program now.
A standard GPT 3 tokenizer is just terrible at this.
It treats multiple spaces really inefficiently.
If you have a heavily indented block of code, a standard
tokenizer might burn through 5 or 6 different tokens just to
represent the blank blank space. Before you even get to the
actual code. Before the line even starts.
And remember, these models have a limited context window, a
limit on how much text they can remember at once.
So wasting tokens on blank space is a huge performance hit.
It's a massive hit. It's pushing the actual logic,
the important stuff, out of the models memory.
It's a waste of memory and compute.
So what did they do for codecs? They modified the tokenizer.
They added a specific set of new tokens that represent runs of
white space of different lengths.
So 2 spaces is its own token, 4 spaces is another token.
Precisely 8 spaces is a token and so on.
It's kind of like you're zip compressing the silence.
That's a great analogy, and the paper notes that this single
optimization allows the model to represent code using
approximately 30% fewer. Tokens.
That is a massive number. That means for the same amount
of computing power, the model can see 30% more of your code
base. Correct.
It can. Look further back in your file
to understand the context of what you're trying to build.
It's not just some efficiency hack, it is a capability unlock.
Because if the model can see the function definition from 50
lines ago because it saves space on indentation.
It's going to write better code, it's just that simple.
OK, so we have a modeled trained on billions of lines of code
with a tokenizer that's optimized to read it
efficiently. Now comes the hard part.
The evaluation problem How? Do you know if it's actually any
good? This is where the paper really
shifts the paradigm for the entire industry.
So in traditional natural language processing, we'd rely
on metrics like the Bleu score, right?
Bleu. Yes, and VLEU is essentially a
text matching score. It's pretty simple if I ask a
translation model to translate the cat sat on the mat into
French and I have a reference translation from a human.
You just see how much they overlap.
You just check how much the machine's output overlaps with
the human's output. The more matching sequences of
words and grams, the higher the score.
Which makes perfect sense for language.
The cat sat on the mat and the cat sat on the rug are very
close in meaning and in structure.
But code is fragile. It's brittle.
Let me give you the example from the paper.
Suppose the correct code to solve a problem is X = y + Z OK.
Simle addition. Now imagine the model outputs X
= Z + y. Textually, that's different.
The Z and the Y have swapped places.
A text matching metric like Bleu would see that the order is
wrong and penalize it. Exactly.
The Bleu score would drop, but functionally.
It's identical. It's perfect addition is
commutative. The code works 100%.
The computer doesn't care about the order.
OK, so now imagine the reverse. Let's say the model outputs
XAYZ. Well that looks almost identical
to the correct answer. The text match is incredibly
high. It's just one character off.
The Bleu score would be near perfect, probably a 99% match.
But the logic is completely wrong.
It subtracts instead of ads. It's a totally different
program, or even worse, imagine it just misses a single colon at
the end of a function definition.
The program crashes. It won't even compile.
It won't even run. It's a syntax error and yet to a
text matching algorithm it looks like a master piece.
It's a 99.9% match. So the researchers found that
high Bleu scores just didn't correlate with working code.
They explicitly found that the correlation was weak.
You could optimize a model to get a really high Bleu score and
it would produce code that looked like code.
It had the right variable names and the right structure, but it
just didn't function. It was what they called cargo
cult programming. Exactly.
It's going through the motions without understanding the
substance. So Open AI had to just abandoned
the standard NLP metrics. They needed a metric that
actually measured software quality.
And they moved to functional correctness.
They effectively applied the principles of Test Driven
Development to AI evaluation. And to do that, they built a new
benchmark. Yes, a benchmark called Human
Evil. Tell us about human evil.
Why did they have to build a new one?
I mean why not just use problems from say leet code or hacker
rank? Data contamination.
It's a huge, huge issue in machine learning.
Remember, the model was strained on all of GitHub.
It is almost certainly seen every solution to every popular
leak code problem ever posted. So if you tested on those
problems. You aren't testing its ability
to code, you're just testing its ability to memorize.
It's like giving a student the exam questions a month
beforehand. Exactly.
So what the Open AI researchers did was they handwrote 164 brand
new programming problems. These are problems the model has
never ever seen before. And what's in each problem?
Each one has a function signature, a dock string that
explains what needs to happen, and this is the absolute key, a
suite of unit tests. And the rule is simple, it
either passes all the tests or it fails.
Zero or one, black and white. There's no partial credit for
looking nice or being close. And this binary success fail
criteria leads us to the most important metric in the entire
paper, the metric that really defines the current state of AI
coding. We are talking about pass at K.
Pass at Ki think for a lot of people just seeing K in a metric
name brings back some bad memories of Alger class.
Yeah, probably. So let's break this down
conceptually. Why do we even need a variable
there? Why not just pass or fail?
Because these models are probabilistic, they are non
deterministic. If you ask codecs to write a
function to sort a list of names 5 different times, you might get
5 slightly different pieces of code.
So we can't just ask does it work?
No, we have to ask how often does it work?
That's the right question, so pass at Tay answers this.
If I let the model generate a color number of solutions, let's
say 100 different solutions, what is the probability that at
least one of them is correct? At least one.
OK, that sounds like a very generous metric, doesn't it?
If I'm a developer, I don't want to have to dig through 99 broken
snippets just to find the one that works.
You would think so, but let's think about how these tools are
actually deployed in the real world.
When you use an auto complete in your IDE it often gives you a
drop down list of suggestions, right?
You get a few options. Or if you're using a chat
interface, you can just hit the regenerate button.
True, if the first one looks wrong, I just hit the button
again. Right, so from a user experience
perspective, if the model generates the correct code
within the top few tries, the user is happy, they can
recognize the right solution, they accept it, and they move
on. So pass it K is really measuring
the hit rate of the models brainstorming process.
That's a great way to put it. It treats the AI as a generator
of candidates, not as a sole source of truth.
Now, there's a specific engineer's note in the paper
about how they calculate this. It's not as simple as just
running it 100 times and dividing dividing by 100.
No, that would be incredibly computationally expensive and
also statistically noisy. They use what's called an
unbiased estimator. It's a bit of math gymnastics,
but it's really important for getting an accurate number.
Can you give us the simplified version?
Sure. Essentially what they do is they
generate a large pool of samples upfront.
Let's say they generate 200 samples, so an equals $200.
Then they run the tests and check how many of those are
correct. Let's call that number 2.
OK, so you have your big pool of answers.
Then instead of running a whole new experiment for every single
value of Calor, they just use a combinant torques formula to
calculate the probability. Right, the one in the paper is
$1.00 Frack Banal Deitman. Exactly, and without getting out
the chalkboard, the intuition there is that it's calculating
the probability of picking purely wrong answers.
So it's asking, if I reached into this bag of 200 samples and
pulled out, say, 10 samples, what are the odds that every
single one of them is a failure? Precisely.
And then you just subtract that number from 1:00 to get the
probability of success. And if the probability of
getting all failures is low, then the probability of finding
at least one success must be high.
Correct. It lets them report very
accurate statistics for any value of tellers without having
to run an infinite number of tests.
It's a standard statistical method for sampling without
replacement, just applied here to software generation.
So let's get to the scoreboard using this very rigorous pass at
Co metric. How did codecs actually perform
on the Human Evil benchmark? OK, first the baseline.
Let's talk about the original GPT 3 model.
How did it do? It solved 0% of the problems.
Zero as in none. Effectively 0.
It just could not handle the functional logic constraints.
Like we said, it might write a nice poem about the code, but it
couldn't actually write the code.
OK, So what about codecs? Now looking at codecs,
specifically the 12 billion parameter version, on the very
first try which is pass at one, it solved 28.8% of the problems.
28.8% that is. Well, it's not taking anyone's
job just yet. That's a failure rate of over
70% on the first guess. It's AC minus student at best.
But, and this is a huge but, this is where that pass at K
logic kicks in. What happens if we give it a
hundred tries? What's the number for tasks at
100? If you let it generate 100
different samples, and specifically if you use the more
fine-tuned Codex S model which we'll talk about, the success
rate shoots up to 77.5%. Wow.
OK, that is a staggering difference.
We go from barely functional to solving more than three out of
every 4 problems. It tells us something really
profound about the nature of these models.
They aren't oracles of truth, they're idea generators.
They contain the knowledge to solve the problem.
Yes, but they have trouble focusing on the exact right
syntax in a single shot. But if you let them just shotgun
the problem with a bunch of attempts, they almost certainly
find the answer somewhere in that spray.
Which brings us to a really practical dilemma for the user,
for the developer. If I need to generate 100
samples to find the right one, how do I configure the model?
I mean, if I just ask the same question 100 times, won't it
just give me the same wrong answer 100?
Times it would, yes, unless you tune the temperature.
This is section four of our digest the Concept of
Temperature in AI. Temperature is a hyper parameter
that controls the randomness of the model's next token
prediction. It's basically a knob that you
can turn to adjust how adventurous or creative the
model is allowed to be. Let's define the extremes.
What does a low temperature look like?
What does that do? A low temperature, let's say .0
or 0.2, makes the model very conservative.
When it predicts the next word or the next token, it looks at
the probability distribution and it almost always just picks the
top winner. It's greedy.
It plays it safe, it doesn't take any risks.
Very safe. It's almost deterministic.
If you run the same prompt 10 times that temperature 0, you're
going to get the same exact code 10 times.
It just locks in on what it thinks is the most likely path
and sticks to it. OK, So what about high
temperature? A high temperature like .8 or
even 1.0 flattens that probability curve.
It makes the model much more willing to take risks.
It might pick the second most likely token, or the third, or
even the 10th. This introduces variation.
It gets creative. Creative or sometimes it
hallucinates depending on how you look at it.
Now the paper presents a crucial strategy guide for this.
The optimal temperature to use depends entirely on your taller
strategy. So if I only have one shot if my
Telugu was one, what should I do?
You use low temperature? Absolutely.
If you only get one guess, you want the models absolute best,
most confident guess. You don't want it experimenting
on your time, you want the highest probability.
Path. But if I have 100 shots if Kalu
equals 100. Then you absolutely must use
high temperature. Think about it, if you use a low
temperature for 100 samples, you just get 100 copies of the same
potentially wrong piece of code that's totally useless.
You need diversity. You need diversity.
By cranking the temperature up to .8, you force the model to
try different approaches. Sure, maybe fifty of those
samples will be complete gibberish or syntax errors
because it took a weird risk, but five of them might be
brilliant, novel solutions that solve the problem in a way you
didn't expect. It's the difference between
asking one expert for their single best piece of advice
versus asking a whole room full of people to just brainstorm
wild ideas. Precisely.
And for code generation, the paper proves that the
brainstorming approach yields significantly better results.
That's how you get to that 77.5% success rate we mentioned.
This creates a new problem, a very practical 1.
The selection problem. Right, in the research lab they
had the unit tests. So they can generate 100 crazy
ideas, run them all against the tests and just see which ones
pass. But if I'm a developer sitting
in VS Code, I don't have a unit test for the function I haven't
even written yet. Exactly.
This is what's called the Oracle problem.
The model generates 100 options in the background.
It knows one of them is probably right, but how does it know
which one to actually show you? It can't run the code itself.
So they need a heuristic, a way to guess which code is the
winner without actually running it.
And the paper explores a few of these.
The most robust one they found is mean log probability.
Can you put that in English for? It its confidence, the model
assigns A probability score to every single token it generates.
If you average all those scores across the whole snippet of
code, you get a kind of confidence score.
So it's basically the model saying.
I felt really sure about this code when I was writing it,
yeah. That's it.
So the heuristic is you show the user the one the AI felt best
about, and it's a pretty good proxy for correctness.
It helps filter out that high temperature nonsense where the
model took a wild risk and just got confused.
If the model is kind of mumbling its way through the code with
low confidence on every token, you probably shouldn't show that
one to the user. Probably not.
They also mentioned a really clever technique that involves
dock strings. This is what they called Codex
D, right? The D is for dock string.
Yes, this part is fascinating. So far we've only talked about
text to code. Codex D is a model that's
fine-tuned for the reverse code to text.
It reads a piece of code and then explains in plain English
what it does. Right, it generates the doc
string. So here's a way you could verify
your code without needing any unit tests back translation.
OK, how does that work? It's a round trip verification.
So step one, take your prompt, write a Python function to
calculate the area of a circle. Step 2, the model generates a
code snippet. Step three, you take that code
snippet that it just generated and you feed it back into Codex
D and ask, hey, what does this code do?
And Step 4, Codex D writes a new doc string a new explanation.
And then you compare you compare the new explanation to your
original request. If the model looks at its own
code and says this function calculates the perimeter of a
square, then you know the code is wrong.
It drifted from the original intent.
But if it says this function takes a radius and calculates
the area of a circle, then the intent was preserved.
Exactly, It's a semantic check. The paper does note that this
successfully filters out a lot of bad code, though.
Surprisingly, it wasn't quite as effective as the simple mean log
probability heuristic. But it shows the direction
researchers are going. Totally.
It's using AI to grade AI. It's mimicking how a human might
double check their own work. Wait, does this code actually do
what I wanted it to do? I want to circle back to
something you mentioned earlier. You said codecs.
You mentioned filtering the training data, right?
This is the supervised fine tuning version.
And we know GitHub is a a messy place.
It is. It's full of half finished
projects, homework assignments that probably got AC minus and
just, you know, bad coding habits that have been copied and
pasted for a decade. Lot of spaghetti code.
A ton. So if you train the model on
everything, it learns the bad habits too.
It learns to write inefficient loops, or to use terrible
variable names like temp one and temp two.
It learns to write code that doesn't even compile.
So for cut excess, they changed the diet, they got pickier.
Much pickier. They identified high quality
code. They didn't just grab random
files from GitHub. They specifically looked for
standalone functions that could be compiled and run.
Where do they find this good code?
They prioritize code from 2 main sources, competitive programming
websites where solutions are almost always correct and highly
optimized, and from repositories that had heavy continuous
integration or CI testing. So code that is proven to work.
Proven and clean and the impact was significant.
Standard codec solved 28.8% of problems on the first try.
Pass at 1:00. Kodak staff solved 37.7%.
Well, that's nearly A10 percentage point jump just by
cleaning up the data. It really reinforces the golden
rule of AI. Data quality almost always beats
sheer data quantity. You want the model to learn from
the senior engineers, not from the interns.
OK, we've covered the tech. The model is powerful, it's
efficient. But is it safe?
Section 6. The paper covers limitations and
risks, and frankly, there are some serious engineering red
flags here. There really are, and for anyone
thinking about deploying this in an enterprise environment, you
absolutely need to listen to this part.
Let's start with what they call alignment failure, or as I was
thinking of it, the bad influence problem.
This is from the Settle bug experiment.
The researchers wanted to know, does the model always try to
write the best possible code? You would assume it would You
assume it's trying to be helpful and correct.
But it's not. It's not trying to be helpful,
it's trying to predict the next token based on the context it's
given. So they ran this experiment
where they gave the model a prompt that already included a
bug. They wrote some sloppy,
incorrect code in the setup part of the prompt.
They set a bad example for it. And codecs followed it.
Instead of fixing the bug or writing clean code, it produced
more buggy code. It aligned with the sloppy style
of the user. That is wild.
It's like a chameleon. If you act incompetent in the
prompt, it mimics incompetence back at you.
The paper calls this misalignment.
The user's actual content is help me write correct code, but
the models objective is just match the distribution of the
text provided. If that text looks like garbage,
it generates more garbage to match.
That feels like a massive trap, especially for junior
developers. If you don't already know what
you're doing, the AI will just validate your mistakes.
It will amplify them. It gives you this false sense of
confidence because the code it generates looks just like your
code. OK, what about complexity?
Can I ask codecs to, you know, build me a clone of Twitter?
Not a chance. The performance drops off a
Cliff as complexity increases. The paper shows this really
clear graph of success rates for what they call chained
components. Meaning what like do task A,
then do task B, then do task C? Exactly.
A prompt like read the data from this file, parse it as a CSV,
sum the values in the third column and then print the
average. A multi step process.
As soon as you chain more than just a few operations like that,
the model gets confused. It struggles with binding, with
remembering which variable holds the data from step one while
it's working on Step 3. So it has a short attention
span. It's a sprinter, not a marathon
runner. It is excellent at writing a
single complex function. It is terrible at designing a
multi part system architecture. Security is another big one.
We hear about supply chain attacks constantly.
Is Codex writing secure code? The findings in the paper are
pretty concerning on this front. The researchers asked codecs to
generate cryptographic configurations.
You mean like encryption keys and settings?
And it frequently generated insecure configurations.
For example, it would often use RSA encryption keys with fewer
than 2048 bits. Which is considered cracked or
at least very weak by modern standards.
Why on earth would it do that? Because it's trained on the
Internet, and the Internet is absolutely full of tutorials
from 2012 that use small keys. Because they are faster for
demos and examples. So codecs doesn't know what year
it is. It has no idea that NIST
standards have changed. It just knows that
statistically, after these words I usually see this number.
It propagates obsolete security practices.
So if you just copy paste security related code from an AI
without having an expert audit it, you are very likely
introducing vulnerabilities. Yes, you're just repeating the
statistical average of the Internet, and the average code
on the Internet is definitely not secure code.
And finally, let's talk about bias.
We often talk about this in terms of text and images, but
does it manifest in code? It does, and in a really
interesting way. Specifically, it manifests in
the data structures the model creates.
If you ask codecs to create a class for a person or a user,
what happened? It often defaults to things like
binary gender fields or very limited outdated racial
categories. Because that's how databases
were designed in the 90s and 2000s.
Exactly. It's mirroring the structural
biases of the legacy code it was trained on.
It preserves the rigid categorization of the past.
If your modern application needs more or inclusive data
structures, codecs might actually fight you on it because
it just hasn't seen enough examples of that new way of
thinking yet. It's an artifact of the training
datas age and distribution. A direct reflection 1.
Last thing that people always ask about this stuff copyright.
Is this thing just a big plagiarism machine?
Is it just stealing code? The paper addresses this
directly. They did a specific check for
what they called literal copying.
Verbatim copy paste from the training data.
And they found that literal copying occurred in less than
.1% of cases. That's.
Extremely low. It's almost nothing, and when it
did happen, it was almost always what you'd call boilerplate
standard license headers or very common simple algorithms like
quicksort that pretty much everyone writes the exact same
way. So it acts.
It's more like a synthesizer than a photocopier.
That's a great way to describe it.
It learns the patterns, but it rarely regurgitates the exact
source file. So pulling all this together,
we've looked at the specialized tokenizer, the big shift from
Bleu to pass AK, the importance of temperature tuning, and all
these risks of alignment and security.
What is the final verdict here? The verdict is that codecs and
tools like it are incredible productivity multipliers, but
they fundamentally change the job description of a programmer.
How so? The job is moving away from
writing syntax and towards verifying intent.
You're not the writer anymore, you're the editor.
You're the architect and you are the quality assurance department
all rolled into one. The pass at Hick metric proves
that the AI is a candidate generator.
It just gives you options. You, the human, must have the
skill to look at those options and distinguish the diamond from
the glass. That's a really powerful
thought. It doesn't lower the bar for
knowledge. In some ways, it actually raises
the bar for critical thinking. It does.
If you can't read code well, you can't safely use the tool that
writes it for you. Exactly.
You can't delegate the responsibility of correctness to
the machine. Not yet.
Before we wrap up, I want to leave our listeners with a
thought that really struck me while I was reading the paper.
They mentioned that codec struggles with dock strings
describing long chains of operations.
It can't handle infinite complexity yet.
Right, the sprinter, not the marathon runner.
But let's just assume Moore's Law holds.
Assume the models get better and better.
Assume they can eventually handle massive chained
complexity perfectly. OK, where are you going with
this? Well, today the doc string, the
English explanation is just a comment, it's ignored by the
machine. The code itself is the source of
truth. But if the model gets perfect at
translating English to code, does the code itself become
invisible? That is the ultimate abstraction
layer. If I can just write the dock
string and the machine perfectly and reliably executes the logic,
will we eventually lose the ability to even read the code
entirely? Will Python just become the new
assembly language, something only the machines look at?
Well, we humans just write prompts.
We might. The dock string would become the
source code. The English language becomes the
programming language. A world where we program in
plain English. It's democratizing, it's
exciting, and for those of us who kind of love the syntax,
it's a little terrifying it. Makes verifying that intent the
most valuable skill on the planet.
Absolutely. Well, that is all for our
Practical AI Digest on Open AI's codecs.
We hope you have a much better understanding of the engine
under the hood of your autocomplete now.
And if you are a developer, go into your settings, try tuning
that temperature parameter. See what happens when you crank
it up to .8 on a really hard problem and.
Let us know if you get any brilliant or just hilarious
results. Thanks for listening to the deep
dive. We'll see you next time.