This meeting is being recorded.
Good evening folks. This is the February 2820 s 2024 meeting of the St.
Louis Linux users group. This is Lee. I'm helping out here. Gary is still on the way.
James is not here and Stan's microphone isn't working. So I am going to. Start.
How about now?
This is Stan. Can you hear me?
Who is that? Yeah, Stan.
Had to change headsets.
You want to do your openings feel? I just started the recording.
Okay.
Are you going to be bashful tonight? Should we go ahead and start a meeting?
Stand your speaker working. You hear me? Okay.
Yeah, it's. Something else died on me.
Yeah.
Well, since Alex is ready to jump into it, why don't we just go ahead and start the meeting?
Sure. I guess I'll share my screen then.
Yeah, yeah.
Okay. Well, at least let me introduce you. Okay. We have a very special presentation tonight.
We got anonymous tip here a couple of months ago that there was a fanatic about large language models right here in the area.
Chap by name of Alex Tao and he agreed to do a presentation this month. The actual topic is demystifying large language models on Linux from theory to application.
Alex, over to you.
Thank you. As you said, my name is Alex. I. The masters in computer science and mathematics both from SIUE.
Not yet.
You can reach me at Lex. Right, can you see my screen? Okay, How about now?
There you go. You got it.
You can reach me at the lex. matterp.com. My blogs here. Metacr.
Dot com. Have a good help at Quillies. Twitter and you can actually see the all the material for this presentation.
At this GitHub.
Let me maximize this. Okay. And yeah, I'll be presenting large language models and also just.
This new age of AI. And here's the, the online, I'm gonna go over basic theory.
I'm going to go over a very simple. Probably oversimplified language model. It's called the in ground model.
Okay.
I have a Jupiter notebook for that and you can if you want Play along with me. And I think maybe it will help us understand certain things about about these things.
If you can get your head around the, the simple language model. It's not very good, but.
It's interesting, I think. And, then I'll show an application. Okay. To try to make, well, I make a database, search API.
Int, I get a natural language understanding. By just using LLMs to Do try to take in the data.
Process it and figure out what the user wants and return the right kind of data. It works okay on the small models.
But really right now you need like the large. GPP 4 level models. Where even 3.5 that really shine So we're not quite there yet.
I just getting there and there's also various techniques that I can I can discuss. That will probably help it too.
So. And finally, we'll end with an open discussion.
So, you know, in their early days, like Alan Turing, all those founding fathers of computer science.
They, they. Did quite a lot of research in AI. They were pretty optimistic too.
You know, they There was one guy, I can't remember his name, but. Yeah, this idea that he could Good together a bunch of smart people.
And knock out the whole AI problem over a summer or at least make large progress on it. Turns out that, it's a lot harder than we thought.
So, you know, the first age, the good old fashioned AI. You know, in that case we try to symbolically represent the problem.
And then use logic or rules. To solve it. You know, these rules are things we can understand with our small little brains.
You know, this is programming, rule based systems, forced order logic. My favorite programming language. This will probably prolong.
But that's a pretty obscure language, although. I take it may make come back. I don't know.
Cause, all these things. Good old passion. Yeah, I don't think it's going away.
I think In fact, they could see a resurgence because these LLM, these LLM models are pretty good at tool use.
They're good at language, they're understanding, they're seeing plenty of I'm training data about how to use these tools.
And everything else. So I think the That might be an interesting outcome of all this.
So. Good old fashioned AI. It works for a lot of problems. Programming on the whole is based on Yes, taking the problem.
Now filtering it through our small, small working memory. And, then bringing it down. And, smaller problems that our brand can understand.
We keep doing this until the problem small enough. Now we can solve those. Then we combine those solutions in some way to solve the big problem.
No, our small working memory is kind of what we call inductive bias. And make certain assumptions about the world.
And in our case, in our universe, those assumptions seem to help. Us reason about the world, that generalized outer distribution like we can see new, we can make new observations and still be able to come to reasonable conclusions, even though we've never seen that situation before.
Our university so happens to work. On some of these simple rules like physics and things like that. That kind of inductive bias works.
But not always.
Many problems are hard to break down into separate parts. You know, the whole expression, the whole is greater than the sum of his parts.
This is also. Talking about it as maybe emergent phenomenon. No things we can't quite rock.
Maybe even consciousness, just something that we can't. Understand because this is too complicated and it can't be easily reduced to a simpler system.
So, some problems are just too complex to. So, deductively. And, and if that's the case, then we can't program computers to do it.
We can't even identify a cast in the picture very well.
Now, I think it, Stephen Pinker said the hard problems are easy and the easy problems are hard.
And by that he meant that. The things that we consider to be like higher level. Problems that really take a high IQ and all that.
Those things are easy for computers to do for the most part, like differential equations and and linear algebra.
Although things And Chess, they can do those problems pretty easy. But the easy problems like playing with Legos.
Or you know identifying cats in a picture that's that's hard.
So how do our brains do this? Well, our brands are programmed by evolution to survive in a very complex world and it's been it's been going through this process for billions of years.
Now is this this our brains are still more massive than, you know, GPT 4 level models.
You know, our brains are a very capable projection engine that learns to predict the world. We want us to predict.
It's future observations.
But the conscious mind, the unconscious mind is not limited by a small working memory. They can do things we can't understand.
And because of that, the brand is a black box. That's a big area of research right now.
You know, and, and interpretable machine learning and. The alignment problem and all that. I'm not gonna go into that too much.
We can talk about it later on if you want. At the end, but, I don't have a lot to say about that right now.
So if we can't solve it reductively or using our conscious. Mine to try to reduce these hard problems.
The simpler problems. Let's have computers learn from data the same way we learn from experience.
And there are 3 main types of learning. There's supervised learning. Unsupervised learning and reinforcement learning.
LMs use a kind of, and RL, HF, but. RL in general, reinforced learning in general will be possible.
Especially on synthetic data and things like that.
And, we'll get bored in that later, but, supervise learning is kind of interesting.
Little ground. So it's almost like supervised learning. That is, but ultimately it learns unsupervised.
Problem in. You know, it models the data, the data generating process. Of whatever data we're looking at.
That's So, supervised learning. You know, this learning from label data. We have some input and output data.
And we want to learn how to map the input to the output.
Maybe that technical with the cats and dogs. Now given an unknown function, F. And instead of input output pairs.
Denoted by X. Comment f of x. Learn to approximate that function. No f hat.
On those input operators, you know, classification, for instance. No, we. Have some unknown function.
It takes a cat or a dog and maybe images or whatever. And it identifies that either a cat or dog.
So if we can estimate that function well enough, then we can use that to predict. Is it a cat or is a dog?
This is probably the easiest problem to solve in machine learning, but it's very limited by data.
When we fine tune, This is a kind of supervised learning approach. We're fine tuning it.
On labeled data essentially. It's not exactly the same sense label but It's going to improve it on on certain kinds of tasks.
And of course.
It's kind of interesting that, like GPT 4 level models. They can actually generate this kind of data for us.
Like synthetic data. It's been shown that a lot of small models. Or just trained on data from large like GPT 4 level models.
There's also other ways to train it. To more efficiently replicate the behavior of these large models.
Scroll like distillation and things like that. I won't go into that right now, but there's a lot of research.
You know, if you look at our, arcs of preprint. There's this.
Too many papers to keep up.
The next type of learning is unsupervised learning. No label data. Now we're trying to learn something about the underlying structure of of the data.
A classic example is clustering. We take a bunch of data points and some higher dimensional space or whatever.
They try to group them in various ways and all kinds of different clustering algorithms. And it's interesting though that Like, rag, retrieval, augmented generation is a.
Is uses clustering. To you know It will use the. Representations learned by these large language models.
You know, it'll cluster them. Then when you see like new data and say, what is this new data most like?
Then they can look in that database, find the nearest examples or example ours. Then they can find out, okay, what data that does that map to is it's much more context, semantically aware kind of way of doing these.
Database lookups.
Alex, what's drag an acronym for?
Retrieval augmented generation. This is all the rage right now, I guess.
It hasn't, worked out as well as people hope, but. It's, it's still working progress.
And they're, they're very interesting examples of it. But yeah, is this the idea is that you take a I mean, the highest halibut is that you just.
Put a bunch of data in the context. Of the, model that's relevant. To that context so that you like maybe you have some Maybe has some Macmath question.
And you want to add. A bunch of facts about math or ways of solving math problems that might help the language model.
So that, you know, it's like team of thought prompting where you do this step by step reasoning.
And there's all kinds of things like that.
Other kind of, unsupervised learning is, dimensionality, dimensionality reduction.
And that's about learning efficient representations of the data. This is the very hard one and one of the most important problems in machine learning.
We'll talk more about that later.
This is the estimation. This is just, a stochastic estimate. Of the process that.
Made or generated or the data we're observing. You know, say the process generates pairs x and y.
And we estimate its destiny with. This probability function. That's the PR stands for probability and then we're just taking the probability of direction y as a joint distribution over x and y.
And classification is. Crossification is this day. Hey, special case of this larger.
Technique was my
This is the most general form of machine learning. If you if you learn this in some sense, you learned all there is to know about the data.
Of course you can't learn all there is no about data, cause that, that, and tractable, but you can learn a lot of, you can learn a lot about it.
And when you do, all the other kinds of. Of techniques reduced to just different variations of using this.
Probability joint distribution. To do things like cause it to. Few new kinds of tasks.
Classification is to provide running approach. Is this the conditional distribution? Of why I give an X.
And that's not easy, but is this, is this, It's proportional to this joint distribution.
So if you know that, then. You can answer that kind of task as a special case. Of this larger model.
The pre trading stage of LLMs. Is like unsupervised learning.
You learn the distribution of raw text, whatever, all the crazy text out there on the web. And a lot of this very toxic and and so on, but you learned that raw distribution.
And use that. Using self, learning, we're talking, we're talking more about that later.
Basically, is the same. Give him this context, giving all the tokens or words that came before.
Predict the next token. That's actually a very powerful way of doing this.
The final type of machine learning is reinforcement learning.
This is an agentic approach.
Hmm.
So conceptually, an agent. No, interact with the environment and also can take in perceptions from the environment has sensors.
Okay.
Whether virtual sensors or other kinds of sensors like physical worldby has cameras or whatever. Takes in the environment.
And learns from just a simple reward signal.
And the goal in this case is to maximize. The expected sum of rewards.
The, right now there are a lot of agentic frameworks. Try to include large language models as a prediction component.
As a as a any important. Piece of that framework. And we kind of see that. If you can predict.
And you can do some kind of search procedure over. Possible futures, kind of factual reasoning. Then you can do planning like long term planning.
There's still much, that's very much a work in progress, but that's what kind of everyone's.
Moving towards finding ways on like that. We're not there yet. Maybe we'll have to change architectures.
My guess is. Scaling up will enable a lot of these. These click.is. But you probably will have to run it in loop.
Like say, okay. Have you heard of like the true your thoughts? approach. Where you just, you prompt the model.
To various ways to generate different possibilities. And then you evaluate those possibilities in some way, which may just be prompting the model again.
Okay.
To select branches to explore further, then you can kinda have it explore a tree of thoughts. And hasn't planned out too well yet but That's kind of the idea.
Combining prediction plus search. To enable like system to thinking right now in some sense. These large language models have system one thinking.
They're very Intuition driven, they're immediate. And look at the contacts and come up with a very quick.
Very quick output. Now, whether you're thinking about. The Raven hypothesis.
Or about what to eat for lunch, you know, it doesn't matter. Of course, there are ways to get these things to think longer about it.
Through like, like tuna thought prompting and to your thoughts and whatever else.
So, this is actually a big reason why people are so excited about Sora. Has, has, has, everyone, seen, I mean, let me open up.
It's Scott had given us a little. Little hint, a little taste of that last week.
Oh yeah. Hey, this is, It's pretty incredible.
I'm on Twitter, so if anything pops up crazy.
Look at this one.
Let's go back.
Yeah, generated generate. Wow.
Yeah. Yeah, you just tell it to, 2 partnerships. That's 2 partnerships battling and a coffee cup.
Yeah.
And there's so much more of this. This is not even the most impressive one. Is able to.
To do this kind of, Yeah, like I thought this was gonna be like a year down the line at least but open open open open, open AI comes out and releases this, not releases that but they show it.
Be a while before we get access to it. Cause you can imagine this could be all kinds of safety training for this.
Like red teaming and all that just to make sure it's safe because The election's coming up.
It's going to be crazy. All the all the it's gonna be hard to tell. Reality from fiction.
No.
Oh, no, nobody would try and screw with the election. Never mind.
Let's hope you're right. But,
Let's look at a couple more because If you haven't seen it, I used to get more videos.
Awesome.
Oh yeah.
They had that one with the girl in the street, on, on GMA and they show that the right leg turns into the left leg.
Oh yeah, all kinds of All kinds of things like that that happens. You know, get some little things wrong.
Okay, it often gets a lot of things wrong. It takes a while to notice.
Yeah, the one with the, I think it was an older woman, an older lady, she was kind of mixing a bowl.
And like the spoon or the 4 comes in and out and the smiles it was pretty creepy.
Okay. Yeah. Yeah, you often can be kind of creepy because it's Oh no, it's like doing things that You can't really tell the first blush, but then you get looking closer and you know more and more anomalies.
Yeah.
Let me, it's able to do this. You know, it had to learn quite a bit about the, the about reality about the world to be able to do this kind of stuff.
You can't just generate wall pixels. Without also having this level without also having some representation of the world, some model of the world.
This is where. You have to have something. We think that's going on. Where it's able to.
So kind of reason about the world understands things about it.
Can you, stop hovering over the thumbnail? Maybe that might. Who's the, speed there?
Alright, sorry.
Or any of these bitmaps or they just pure images? Cause I've seen for a while there people doing.
Blender generation not completely through but some assets and things where they'll do like the movement part or I just wonder if these are pure video if there's any actual asset generation manipulation going on.
No, is it? They claimed that this is pure video. Just. Text, prompt to video or they also can do images.
Like they can actually, Give it the image and say, okay. Imagine some world where this image is the final one.
So I can do all these weird things like that. And it kind of has what you might think of as a.
Intuitive physics engine like within the weights of the neural network. This model something like a physics engine.
Kind of like a world model and maybe in the future. You can use this. This capability to have an AGI or an intelligent system think about the future.
Think about future possibilities and plan in a very complicated environment because language is nice, but it's not near as complex as reality.
Languages by. Bye. By design, very information rich, very, very low, very low noise, most, of language is just very high signal.
Very low dimensional. But in cafes, all the things we care about and think about. So it's still important, but it doesn't require as much representation learning.
So that's Kind of a new capability that. Has got people wondering.
I'm going to close this down.
Okay.
Okay, we coming off the Another tangent, but, I'm gonna go back to talking about the.
The history of machine learning and and whatland is up to where we're at. There were early failures in machine learning because reality is very complicated.
You know the number of the sequence length How many? Things that are we have to consider once. The, the, the dimensionality of each object that we're looking at, all that is extremely.
Complicated and Early efforts were not very successful. They just weren't large enough. They weren't They have enough data.
Just. Enough compute this didn't work out It's time wasn't yet. Just.
You know, a lot of the reason progress isn't based on new insights. A lot of it is based off scaling up things.
Just making the systems larger, making the data more, more intensive. And, that seems to work.
There's actually Richard suddenly, you know, he wrote like the founding father of reinforcement learning, rework the book.
You talk about this and I
He said the big lesson. Is that general methods that leverage computation? Or ultimately the most effective.
So in their early days, we had all kinds of very advanced clever systems, that were machine learning models like like support vector machines and.
All that they were they were all the rage for a time because they were able to. Do a lot with the very limited.
Training data and very limited, computation. But they didn't scale up once we got more data.
More competition and this couldn't compete with neural networks. So, NOR, came back on the scene and now of course.
Or now they dominate everything.
Sure.
Back on the slide, please. So the term over fitting in this case. Well, how could how would you define overfitting?
And,
Okay. So like, let me draw something.
Let's do a simple example. So you have a Some say you make some observations. So we have this.
Let's say we make some observations, you know, from, from something. You see that dot.
And receive that dot dot.
No. This one too, not the best. The best, step to the train data. Might be something like this.
Let's suppose we, let's suppose then that we see a lot more data. Like this one goes off like this.
This goes off like this because the polynomial in the end just go off to infinity.
All sudden, see this. So. We trained on the Training data, that's what we saw.
And then we wanted to use that data to make predictions about the future. Well, here's the future, these points.
And this thing doesn't say anything about those points.
Just line if you see it, it's probably like this. And then of course, this line maybe like y equals like a.
1.1 X. Plus some noise. Where like often no worries is like. Gaussian.
If they'll do something like that or whatever.
But this straight line. Helps you take the data. The future data much better. Just this line is just over.
Cool.
Hmm.
Yeah, this is a well known problem in machine learning and statistics. We do a lot of things to combat that.
No, there are things you can do. You can, Let me try you some of the data.
Hello, some for verification or testing. Okay, I'm gonna fit it. And see how it does.
And you can kinda tweak the hyper parameters. To do something a little more clever. But it's it's a problem and there's also what we call a bias fairness turn off the more complicated the model.
The best bias you have. Because you know we have this. But the more variance you have, cause It's much more sensitive to, train data because this thing I hear.
If you had, of seeing.
See this? It's okay like this. This point.
Do you see new points? Okay, here, here's what it is.
So what 1 point added to it completely changes the shape of that function.
And.
So there's all these trade-offs and. Is, to machine learning. And, it's interesting that, these large members models and large neural networks.
They have more parameters and they have observations. They have tunes and observations. Not well, I guess not more parameters, but they have.
Way more parameters than you think they would need. And, you know, There's many theories about what's happening there.
Cause it seems to violate Classical stats. And we had all this theory about about how much data you need.
How much, how much, When you when you will just be too complicated and over fit to the data.
And then all of a sudden we have these. These buildings and parameter models and they're doing better than ever.
And this happens that, you know, one way to. Make the model simple is to reduce the number of parameters.
Another way is to Apply some. Some procedure that PIN penalizes complication. Like, You know, it's very sarcastic.
And in the operation, batches of data. So as you get more data, as you get a new batch, it will.
It will, randomize things and you can't, it can never fit too much to that, to data because it's always seeing different substance of it.
But the other things you can do too is like, you know, you can try to. Impose sparsity and and Of course, the many layers of the machine, the min and layers of the No, that works and these like these deep learning.
Neural networks, they also. You have these. Inductive biases where you know one layer learns some representation.
Another, another one, another one. They kind of learned these, so, And then not fitting the data, but they kind of learn certain programs that generate the data.
But you know, all this, still very much unknown. A lot of times we're just kind of doing things.
Doing experiments and just kind of engineering things and there's no formal theory for all this yet.
Maybe one day we'll have it. Maybe the future AI system can help us develop it.
For sure for. It's almost like the point there is any time you all have the the y-axis, whether it's time or some measurement or whatever, but once you go beyond the bounds of where you've actually collected data.
Oh boy, it gets more and more that predictive curve there.
Yeah, yeah, this whole thing here, like, make a good point. This all this like here for this model is Not it's good to stuff within it within its Training data and it can it can also do pretty good at at stuff this within his window kind of like it like interpolation besides rather than extrapolation, like you said, unless it goes outside the bounds of it.
Take off and be pretty good at interpolating between values. But yeah, once you get off the bounds, all things go off, go off tracks quickly.
And,
The cursive dimensionality is that related concept or
Oh, that's that's this. Yes, this how When you increase number of, parameters of the more like like in the, model.
If you have a You have this. No. To learn this. To learn the function that has like in.
In like a vector in size in. The number of possibilities. Like, let's say you have zeros and ones.
Each XJ is just 0 and one. Okay. And just say, in their sequence.
How many functions?
Does that represent? Cause we're, we're doing a machine learning. We're trying to learn the function that.
Generous. That data. Let's say there's no noise or anything else. It's just.
Is an exact function and we want to figure out what it is.
Now. If you don't see it all, that's even more problematic, but let's say you see all that there are.
2 to the 2 and. Functions. Just on this, We also, not just that, but no, it takes.
Takes a No, they were asking okay. What's accident given this? Good. That might be the function.
And in this case, X of in is also binary. And that's the case and we have this many.
This many possible functions. I have one that. But like, you can like to do a simple table like that, like an and table, like.
Bye.
Right.
X one x 2 right
How many functions are there for this? Well, and will, you know, take 0, 0. 0 and it'll take a 0 1 0.
1 0 1 1 0. 1 1 one as this basic logical and.
But there's all the ones too, right? So this thing are here. Could have been 0 or one.
So there's 2 possibilities there. For any, to their to their. To their when you most play together you get to the 4 or sixteenth.
Or 16 possible binary functions. And is one, or there's X or. And so on.
Now, of course, the the, just means If you had another one, now I'll send instead of 2 to 4.
We have 2 to the 2 to the 5. 2 5 is. Right.
3. So 2, 3 of those 2. 2 to 3. That's 8. We're 2 8.
Now is 2 56. So it's exploding. Is, Exponential world.
Right.
Okay.
And, and this is the simplest case you can imagine, just a simple binary function. Reality of course is, you know.
Much more complicated.
Okay.
Yeah.
You were talking about the 3 types of learning before and you were talking about overfitting. Do you think that one of the learning types is more susceptible to overfitting than the others or do you think they're all pretty much the same?
I don't know, what do you think?
Yeah.
I imagine you can over fit all of them. But maybe the supervised learning is more prone to it.
Maybe. I really thought about that. That's an interesting question. Let's see.
I mean, you can definitely cherry pick things, right? We, when you're doing, so, revised learning, your, your, You're picking the data, you you've often.
Well, I don't know if you're picking it, but. There may be a tendency to.
To do things in a way that 63. Good, that's some neural domain of task that you design it for, but it can quickly get outside the domain outside of that domain and very easily.
Just not do good on anything outside that neural task, which is often okay because because sometimes you just wanted to be good at that task but But, yeah, what do you think?
Yeah, another question I had too was, so when people are saying like a model, I feel like that's a very overloaded term, right?
Huh.
So, can you, Like for example, can you have one model that uses multiple there was trained on multiple learning styles like this like could you have one model that was supervise and reinforcement or would that really be like 2 models that are kind of combined together in some way?
I mean, well, just.
Historically, they were separate things. But, you know, if you train like classification, for instance, I mean, if you train, if you train the function.
Or a distribution.
And you want you wanna you wanna classify X given what cost by Y given X. That's classification running, right?
So, if you learn.
Tomorrow with the data. And you also learn to classify the data. So that's kind of a classification learning.
So I guess there are no hard boundaries. It's often difficult to say one thing is this and that's.
Another thing, like even the way we train large, language models. We use self-supervised learning.
And so on the one hand, you want to say that's, Superbike learning because you know, you're predicting.
X and given all the other stuff.
Right.
Some, distribution.
Give me the context.
And.
So on one hand, when I say, well, this output, this is a labeled data.
Cause we're Looking at this. Data, we say we want this.
It's not in a sense, supervised learning. But it's but it comes for free though, right?
Cause we're just looking at reality or words like the brain. Learns to model it's observations it gets from all the sensors.
And it doesn't, you know, we're, we're not, we're not label things.
We just, is this. So, okay, I guess it was a, it calls a label, say, okay, this thing.
It's probably dependent upon all this stuff.
So.
So on one hand, it may, you wanna call, self, I'm gonna call it, to provide learning, but other hand you wanna call it, supervise because Now there's, you don't have to do anything special.
They don't have to label the data or anything else. And in the end is also learning. This function, that made this.
That's the most powerful thing you can, It's very crude, it's very crude approximation of it because no usually doesn't.
Doesn't see like this thing. The context, what is conditioning on often noises like the last.
Kate Hopkins.
When, when in rowdy, or maybe like.
Well, maybe infinity or just like maybe in much larger than K. Of course, you know, some recent advances are all about making that pay larger and larger.
Making the context from the larger. And of course, there's other things you can do too, like our brains don't just model.
That don't do this way either our brains have some way of recalling having memory and recalling information.
Stored in some other kind of way and then it kind of able to augment his contacts. With that information that it learned previously.
But, it's interesting that many things and these language models and in general, what kind of kind of doing the same things or benefiting from the same things that that we do.
Like so. I'm not sure where all this ends up, but it's fascinating. And you have any more questions about this slide or?
I'm gonna go to next slide.
And the final ones is, This is like a An agent, this gives a
I think I were talking about the Z. Yeah. Okay.
And, that's where everything's happening right now. And of course, as you as we all, well, there was AI winter.
And, and these things just. Went off the radar for a while. People like, Jeffrey Hinton and and you're on the turn they they I think I said that wrong but even.
They've been working on this on this stuff. For for decades now. Marking up security for a while, but now of course.
They're the, of the, modern AI. And. And your network is universal function approximator.
They can learn to represent any function. Earlier we were talking about F. And probabilities. These are both functions.
And they can learn to. This is the, I think this is the functions probability distribution, but that is actually a function because it's just learning to map.
You know, The input. Learning to, the context. To a bunch of numbers.
All of which, some to one. So this is what it's learning.
And. This is why, so, Survivor, only work so well for, for these, models, cause, cause, cause now when you have a probability, I don't talk about this in the slides anywhere else, but.
When you, When you learn this.
Then you can use, to see our neural networks parameterized by the theta. These are the weights.
And we have data.
Well, no, I gotta do. To learn these weights is to take the You can look at all the data you have.
Okay.
Then maximize it. Find the parameters. That maximize this thing. This is this is this maximum likelihood estimation.
We were pre training the models. Like just learning to represent to model the data that we're looking at the raw data.
That's this maximum likelihood, maximum likelihood estimation. It was invented by Fisher 100 years ago. It works very well.
Is very universal. You know you can you can you can You know, it's a very powerful tool.
And so that's what these things are doing at this stage. And the development. And use this to.
Fine, parameters that Maximize. This thing. And of course, each one of these things is you know, one of these things.
Like this thing. Xjay, maybe it's a This one.
Does it decide to hear that when we sample these models? No, we, we use this distribution.
I guess that one.
We actually sample xj.
From
Given the context, right? This is not actually what we want to do. This is a greedy algorithm.
We're just finding the next token. That, that maximize it. Really what we want to do is Maximize.
You know, overall possibilities.
That's computation expensive. And when also when we're sampling at first time. You might wanna do that too, you know, I don't wanna just take the next token, cause maybe that gets you stuck.
And in a weird situation where you said something that's been correct. You can't backtrack because it's just an AR model.
And, So now it just keeps going through. That sequence even though at this point is very unlocked to occur.
They did a greedy algorithm. They didn't look very far ahead. What's the next best token?
Of course, you don't just pick the next next token, you sample over that, over that distribution.
But that's. That's the idea. And you, you actually wanna like maybe sample like.
Just sample the next.
Maybe something like this. You don't do too much, but. Maybe you wanna sample the next.
Does 5 tokens. You know, given the context. I should label these different.
And that's also known, but it's, you know, we can compute that kind of with this, this function.
But it's a. You have to explore the space of possibilities. And once again, that's that's the curse of.
Dimensional. Dimensional. Dimensional. Dimensionality.
Dimensional. That's not wrong, but so I think I said enough about that.
You want to say anything about this?
The the term network in neural network. Where's the network part come in?
Yes. Oh, it's called that because You have, you imagine these nodes.
We have, you know, some impact.
Is it is it ethernet or fiber?
I'm going with the fire workers.
Okay. Yeah. As long as it's not ATT.
No. And some of Now these are the inputs going into these what you call these nodes. And they output something called the activation.
Based off some nonlinear. Combination of the weighted inputs like you know you probably all seen this kind of notation Kim or out goes exactly but but
You know, you have, And, and you just, you just. Doing these nonlinear functions.
Over all the inputs. Just do that over and over again. You just had a bunch of layers and it's able to model all kinds of complicated complicated patterns in the data.
If you don't have this thing for now, I think I think this math is wrong here. But But if you don't have this thing, nonlinear component, then all it is is matrix multiplication.
And that's this.
So you want to do something a little more, a little more advanced bill. Yes, and though this thing can just be this.
This function. It can just be this.
Straight lines, but at this maybe 0, sort of, going straight, this goes up. And then I think they call those.
RELU functions. Regularized, linear, These turn out to be very good before before that they had other ones like sigmoids and and other ones and and You know, has a problem with the vanishing gradients and exploding gradients and and First Turks have been designed since then to get around all those problems and we're also very accurate at producing.
Now, when we, when we all learn the parameters of a network.
We're using that propagation.
Which is just. Let's say you have a function. Of that.
That propagation is this.
If you want to learn the derivative, partial derivative of this function. With respect to the drinks component.
That's what you do. You wanna learn learners for this this function.
Okay, can see that but
This, you know, G. This is the partial.
Times
In fact, prop is this way of. Doing this This computation very efficiently so it can back propagate through thousands or whatever.
Layers and only do these computation at one time and do very precisely too. And, well, back problems doesn't do that.
No survey. We have like things like computational graphs. That, help do all the, all these very, very intensive numerical computations, but they're very accurate now.
And, that's what. That's a scale up to these massive neural networks.
There's a question in the chat. As anyone touched on, training, a way to get the model to forget data, it was previously trained on.
That's from PAS.
I haven't did too much research on that. I know it's a very important topic. I look at humans, obviously a big part of our.
Of our where we operate is we forget things. That aren't relevant to what we're doing now.
You know, we have this way of, for getting things that aren't important. And, and remembering things that are, that are important to what we're doing.
Well, I, I was more so asking, you know, if somebody There's a different training and LLM right on.
Versus adding knowledge embeddings to it after it's already been trained is that correct?
Am I using the right terminology here? I don't even know the proper terminology.
To use or would that be?
Okay, can you repeat that?
Yeah.
Training and LLM on your data. Or training, and LLM.
On knowledge-based embeddings, I guess you could say. So the difference being you're adding knowledge to the LLM or out of the box prior to you giving it any other information by using human language.
It's it's prepared with a knowledge set of your data versus the default LLM and it's, you know, one of the default versions that has no, ability it doesn't know anything about you or your data and then you afterwards train it on knowledge.
Specific, you know, knowledge embeddings either through raw text or Excel sheets. I guess the question I'm trying to ask is in either use case, Can you untrain it if you accidentally feed it wrong information or is this a restart from scratch every time?
Does anybody have a solution for For the
Right, I guess it depends on like if you're just talking about like rag retrieval augmented generation.
If you're, you're, you've mentioned, that kind of applies that your idea is that, You're storing data external to the network external to the, neural network.
If not, not in the weights.
If you
Correct, correct. Yes, that's what I'm, that's what I'm afraid.
So the 2, there's well you could do it 2 ways. So I was actually asking about each way separately.
The
Right. Yeah, when, Do, you, triple like rag.
Now, you can definitely forget that. Just don't use it. I remember from the database. And you can add things to it too and then you can do that as much as you want.
Of course. These language bottles only have a finite working memory. The context. Can be easily overwhelmed, then it's not useful to just throw much stuff in there.
The most effective way to give them Hey, and reasoning about things is to embed it into weights, you know, train it, you know, train it, pre train it on that vast data set.
And, and, we do that and then, They don't have this problem with the context, cause not using the context necessarily in that way.
I mean, they still do the context and they're still learning their their behavior based off that context, but they can much more capable.
They don't get overwhelmed by all these.
Perhaps uninformed details they can they can just learn from the raw data and train on that And that's probably pretty hard, hard to forget.
Hard for to. To undo or whatever. You know, that's a setting the weights.
And you can give a more data to change this behavior. And we train it and you can you can adapt to.
This new data. But it's a new data is like, bad data and all of sudden it finds itself in that in this different where state is no longer able to do good on the previous data that is more important.
Then yeah, it's probably gonna, you know, it's not gonna be all forget. Oh, that stuff.
And this is gonna be
Different.
Is there a situation where you'd recommend?
Pre-training the LLM with weighted scores correctly. Into the, knowledge base.
For Prior to, I guess you could say, I don't really not ask a question, correctly.
But, I guess just trying to understand if. Everything about your data should come post. Training through those knowledge embeddings or if there are use cases you've been able to identify.
To apply it to prior to the alone being trained.
Yeah, I'm talking about fine tuning, right? Like, yeah. Yeah, yeah, you have some, some data.
Right, that's it.
That you want to make the the model better at. Doing some tasks like mathematical reasoning or whatever, give it a bunch of examples for that.
You know, ideally you will give it examples like if it's like. But whatever task one, do better at you can just give it a much more data for that and it'll get better at it.
It may forget how to do other things. That was it was previously good that too, but there's ways of.
Or it could get worse, right?
Yeah, yeah, yeah, and get worth that other things. There's a way to try to counteract that, you know, by.
Bye, not only maximizing the likelihood of the likelihood of the new training data, but also. When you get too far from the original.
Distribution. Yeah, you can use like, KL divergence and things like that.
But yeah, I mean, you can definitely, definitely, have it do worse. On the previous data when when you find tuna for this new data.
And, but, lot of times it's worth it, you know, and of course, you know, You have most, you have a bunch of models, though, you know, like when you find 2, one, you, you still have the version one somewhere around there.
And maybe the fine tuning is just adding a. And there's a layer of some neural network.
And they can switch those out in and out as you as you please and might be various ways of choosing that too, you know, but But they're very composable in some sense.
Interesting. I hadn't considered that. Yeah, thank you.
And do, do you recommend, if you're If you are making an attempt to.
The interesting fine tuning and knowledge embeddings is what could you explain it real quick to me one more time.
But, well, yeah, well, knowledge of betting is this, where. You just.
In some way you have some, some data that's external to the network, like maybe a database or a vector database or whatever, it doesn't matter.
And you want to find a way? To put and within the context of the language model. Some facts about that so that it can they can.
Yeah.
Do the continuation, you can do prompt continuation based off this in rich context. It's not just based off what it's seen before, but.
And the media past, but maybe you would have seen a long time ago or other things, other people have seen or done like various tools, whatever.
And you just want to put that into the context. Trying to make it do better at some task.
And that's a very different mechanism. Then the waste of the neural network, 2 different things.
Then I think about the. The like the rag, take this external data and put it into the context window of the language model is it's very adaptable.
You can make it you can have as much data as you want. And, doesn't overwhelm anything else.
Is this independently? Parameterized and And sounds a nice feature about it. Even this in a sense, train the system, nothing, work in the system as a whole.
To do better at certain tasks over time. They can do learning. There's not just learning that's, done by addressing the waste of the neural network.
I think probably a lot of our learnings same way, you know, we have in this memory, but it's probably not.
Exactly like a weight like weights in a neural network is probably some other mechanism like The hippocampus has almost a different way of doing things, but it's, well, remember it comes from, I'm not, I'm not a neuroscientist.
So don't listen to me on any of that, but. But, Yeah, I mean, it's all very fascinating and there's also ways of not just You can also train these things in to end.
Okay.
You don't just have to have this. Separate retrieval mechanism, you can actually, why why you're by pre training this model.
Or fine-tuning or whatever. You can give it access to this. Retrieval augmentation.
And you can have it learn how to do that too. That can be part of the extended mind, you know, and it can train those 2 and learn how to use that tool better too.
And maybe prioritize, things that it should memorize and weights because maybe, when those learns in the weights, they can see more patterns and do more compositions.
And same things that they should just learn. To, wrote memorization because maybe it's like like Paris's and in France and or maybe things that There's this kind of disconnected facts.
And you just, look it up, And that may make the, the, network much smaller and more computationally efficient.
It's still a lot of resource in that in that, in that direction, but, Yeah, right now the way most people do rag isn't doing any of that.
And, and of course the resource scientists, that's what they're doing. They're trying to get that to work better.
Well, fair enough. Thank you for that. And is there a third party open source service you'd recommend for that.
I haven't actually did any of that. Yeah. I haven't messed with with rug or anything.
I actually just got a GPU. Like 20 days ago. This is
Oh, you get what? 49 or would you go? Consumer. Okay, very cool.
3,600. What do you have?
Yeah.
30 60. But I'm hoping for, you know, I want 24 gigs of ERAM personally.
Yeah, I mean, you can. Rundy's very large models if you have enough system memory, just running very slowly, but that's not very fun.
Right.
There's, Enterprise HPM 2 GPUs on eBay.
A few people are talking. I think you get them at 16 gigs and 24 gigs. Pretty cheap like a hundred bucks a pop.
Hello, that's not bad.
Really?
Yeah, yeah, they're headless technically so they're for they're good for running AI you can't really run anything else out of them but yeah they work I can't remember the specific model number but there's Some big data center got cleared out and switched over so they're selling all of them and there's a bunch of enterprise series ones.
Do you know who the seller is? Do you know who the?
Oh yeah, pretty cheap on eBay. It's more of just search, HBM 2.
They still say.
GPU or the Nvidia HPM 2 GPUs. And there, yeah, they pop up.
There was some video ones. There used to be some AMD ones before that I came or what they were called.
They constantly pop up on eBay. There's quite a few enterprise GPUs of 1624 gigs and even higher for not very expensive because they have no real use for a lot of people in this main world.
Huh. Wow. That's unbelievable.
All, I'll search around and post some links in the chat if I come across them here shortly.
Okay. And is anybody familiar with LinkedIn and had any integration with LL L's or no, right?
I have some. I guess earlier I was messing with it. Although I when I was trying to get my My application to work.
Oh, okay.
I ended up just this not using link chain. I could get getting in the way. And maybe, maybe is this my, as maybe this is a new technology and I didn't know, have time to.
Really protect my use of it, but I end up, my intention was to actually use it link chain to develop the application.
But the end I just I just dropped in to start sort of this hand crafting the raw strings.
You know, they take taking these f strings and this manipulating those. There's gotta be a better way, but, I had like a data mess with it to get it ready for this presentation.
And that's what I did. So.
But.
Yeah. Yep.
Okay. Well, thank you for putting this together. I this is definitely an area of interest to mine and I've actually got an LLM running with speech attacks, text to speech, runs all locally on my machine and I'm trying to add now memory to it every time I do.
It's not except maybe my dad is not being, input correctly and I was wondering if there was anybody that thinks that they might be able to help with something like that.
I mean, Give me time. Let me look at it. I'm sure I can, talk, talk about it and probably do something with it.
Okay.
Cause, talking about using the, or whatever to help you like talk to your PDF or whatever.
Well, PDFs, or Rottex files, or sells, that doesn't matter.
Yeah, yeah, whatever. Just anything. Yeah. Yeah, I mean, I'm sure we can.
I mean, I think I have, I think I have, OLAMA web service running that does that, but I'm not sure.
I've missed with a much.
Yeah. And one last question, is everybody familiar with any, like cheap, if I don't have a 40 90 and I'm tired.
Is there any online? Platforms I can use. To train an LLM on my data set instead of having to purchase a 40 90 something that's a somewhat reasonable costs.
I felt I thought all cloud computing rentals were expensive. I haven't found a cheap one yet.
A reasonable one.
I mean, should it be costing me 50 bucks? Each time I want to retrain on my own data.
Attention. Yeah, fine tuning it.
Oh, you're training on it? Are you fine training? Oh yeah, that'll cost the money.
Right. So is anybody familiar with like any? What anybody recommend any third party service?
I have to use them because they seem too expensive.
Yeah, okay. All right.
Oh my god.
But plus I just I was just playing with moment toys. I was using it. I have started to do a fine tuning project on GitHub.
But. I have I haven't I haven't looked at it in a couple of weeks, but, but, but my intention is to get back to it and fine tune.
Something that's good at the application. In a bit, I'm gonna show you an application I'm working on.
Okay.
And it didn't work as well as I hoped. So I'm taking I can fine tune it.
Take a very small language model like tiny lama and fine tune it for some task. Getting very high quality training data from GPT for and other kinds of, approaches and, then, training on that.
But. And we also have any experience with, fine tuning.
Not fine tuning, but I put I put a link in the chat for it's an OLAMA tutorial, but they they do something sound similar to what you're kind of talking about where They, they feed in the Odyssey by Homer.
And then they're able to ask, one of the language models about. That text.
You know, and it's able to respond pretty well. Looks like from this example, so.
Good check out the tutorial.
Yeah, what? Yeah, what to do. I'm sorry. Go on.
Awesome. Thank you. Thank you.
Yeah, what they're doing there and I have looked at it but I would guess It's not fine-tuning what they're looking at chunks of data over time.
And, Like you can just do a search on it. And that's very crudely asked the model, is this, this took a text relevant?
That's what I want and just keep doing that. And the other approach that's commonly used is like, you know, using these embeddings that you were talking about.
But, that's not fine tuning. The client tuning is a much more intensive process for you're adjusting the weights usually not all the weights.
At the, the neural network, but like some subset of them. They specialize it for a different task.
Oh, no.
Understood. Thank you. I've taken up More questions and I deserve so thank you.
I found those links once for the Nvidia or sorry the AMD instinct and then the second one is the Nvidia Tesla.
You can get a Tesla right now, 32 gigs of GDR 5 for 1 20 on eBay.
Be careful with those because they may be the. It's so nice, but there may be the one with the 2.
GPUs, so each one only has 16. I'm not sure.
Oh.
I don't know, I know what you're referencing. I'm not sure. Sure, I mean, because there's other Tesla models too.
Yeah.
There's M ten's P forties. That's why I just searched by Tesla.
Okay.
And so I'm not sure model specific, but none of them are very expensive. And I can't you use more than one GPU on an LM?
Sometimes you can, but sometimes like the bandwidth between them is not enough to do. To really benefit the.
Okay, I mean, I guess it depends. I don't have enough experience with it yet, so I, I can't speak very confidently about it, but I know there are problems with it, but I know there are problems with it and they're always trying to find ways to make the bandwidth very confidently about it.
But I know there are problems with it and they're always trying to find ways to make the bandwidth But, I don't know, but my guess is you're running the problems when you try to transfer data between GPUs.
If you try to transfer data between GPUs, if it's not over super fast. Link.
Oh yeah, that makes sense. I was wondering when somebody was gonna release some version I could. You know, string a few GPUs together because You're right, they're all isolated, you know, they're all
I mean
I mean, nobody's done it. Probably, a good reason for it too.
Yeah, I don't know. I mean, it seems like in theory it should be possible, right?
Course the devil's in the details. I mean it seems like At least like neural networks are very distributed by design.
Like you can run one layer. No, somewhere in California, another one in Montreal and that's fine, I think.
Yep.
But I guess it depends upon how much data sharing. Please be done. And if you're if you're doing inference like end end.
Like going from the input to the output. That probably needs to communicate. Across the entire, it's probably very connected.
You need to communicate there. That these values across the entire network and if they spread out that may be slow, but I don't really know.
Yeah.
Fair enough. You know, just I would say everybody is working on this. Be careful letting it make its own decisions.
Yeah.
Rather, let it provide information, but allowing it to make the decisions is a little bit of a problem.
They can it's gonna result in a major mess up So just be careful, I guess.
Fair enough.
Thank you guys. Thanks.
Thank you.
Quick references from the chat college mentioned there's a 5 min video I'm building a PDF reader with a Q&A front end Okay check the chat if you wanna pull up that URL.
See there was also comment about this started learning language chain. It's a bit more briefy.
Than a semantic kernel.
And those comments actually go back a good, 10 min. So, sorry I didn't get them in earlier.
So anyway, turning it back over to you, Alex.
Yes. And so. Where's I at?
Maybe I'll skip, skip this stuff, cause I, But, well, quickly, And inductive bias is.
No, any time to look at something. Like, like your observations or training data. Usually there's a many different models or parameters in your network.
That are equally good. That, the data are compatible with the data. And so you need some way.
Oh, choosing one. Model over another one. And this is usually where. You know, indirect advises come into play.
No, some sort of assumption, some model makes about the data. But I think O's razor is a classic example of this.
All things else being equal choose the simplest model. I've been a long supporter of long fan of slopping off induction.
And this kind of the digression, but I do slom off induction as kind of a. Kind of the Yeah, the endpoint of all this, you know, like these Laura's neural networks, these LLMs.
They're kind of doing an approximation of slom off induction. That's long enough induction of course if you know anything about it is it's not a problem.
But of course you can get to it.
The better off you are and that kinda slow off induction represents kind of upper bound on performance.
So what is we want to generalize out of distribution from inputs not in the training data? So that we can think about the future and Do things that are sensible.
And new environments. But, cause we're almost always in. These new environments were always added distribution, especially these high-dimensional spaces.
So good inductive biases are necessary for, but there's a well known theorem called the no free lunch theorem.
No model is optimal for all tasks.
Just wanna be good for the set of tasks that you find yourself doing a lot. A lot of tasks are very important.
You can imagine a task for anything. Most of the things, don't even make sense to us.
This is just not, it's just not something that would be relevant to. To what we wanna do.
So you wanna find, find. The way of. Slotting those.
Biases that help you do what you need to do better.
Yes. I like the idea.
So now, like, probably. Well, 12 years ago, 10 years ago? You know, the era of deep learning kind of.
Kind of like came on to the scene.
One of the hardest problems is learning what you call sample efficient representations of the data.
And, you need to do this because
And this is actually just, Back to inductive biases again. But you want to learn sample official representations so you don't have to see all the data.
You just want to be able to see a small set of data. And then generalized from that.
It's on deep learning the layers. We're on progressively higher level representations.
That is a very good. I kind of bias kind of representation learning. And Alex Net. Was introduced in 2,012.
And it just totally dominated image in the competition. I don't know where. When, when Alex Net.
Enter that competition. I think they were the only one. Using a neural network. They won by last slide and, change people's expectations about what's possible.
The next year, like almost all of them were using neural networks. So that was just a flip.
And a lot of people think that these deep neural networks are learning tiny little circuit programs. That can generate parts of the creating data.
Hey, I, some reason I'm not seeing. Yeah, actually those images here too. You see, the images here.
I think, I think.
Does anyone see an image right here?
Nope.
No, we're not seeing images.
I've noticed that's a bunch of BUSINESS. Maybe I can reload it.
We see you drawing, but.
Yeah. Let's try this one.
Okay, yeah.
Yeah, there's your images.
Yeah, I'm going back to this through a markdown.
Okay, and, and close that.
Okay.
Yeah, that looks a little different. Okay.
Oh yeah. Yeah. So, okay, anyway.
Yeah, I didn't hear what's in there. It was released on this on this, round 2,012.
Here was the previous state of the art. We came on the scene. And this 50% was say there are.
After decades of research, people handcrafted all these feature representations. And then deep backing on this, it just learned.
Have to represent the data by itself. Representation learning. There's no longer something humans did. There was something.
The model then. Like. Before research or hand tuning. Like ways of Did detecting edges and things like that.
Ways of seeing where There were there were boundaries on objects and it was all very technical and we weren't very good at it because it's very hard to do that.
But the, that comes on the sand and or just Yes, Alex, and I think the people doing diddogs, that never even, Mess with the image classification before.
They just knew about neural networks. They applied it to that problem and it And it did. 30.3% better than the previous.
There there are. And as you see. It's just been going up and up and since then that now is 90.8 8%.
That was around 2,021. I got 222. And that's apparently better than humans.
And what is this y axis?
This is the accuracy. Like a portion of things that they classify correctly. Like, say you show a picture of a leopard.
Okay.
And you're saying, what is this? Yeah, this seems like an easy problem nowadays because we have these, these, image transformers that can, that can, like, multi model.
Models that can reason about images and do all kinds of things with them. But But yeah, is this how many how many of the images do you quickly classify?
What's what's the legend on that y-axis looks like top something.
Oh, that does. On the vertical?
I'm sorry, the, the vertical axis. Okay. Yeah, what's that left hand margin?
That's top one top accuracy. I can't see what says me.
Okay, alright.
But is this the RCC on it? Like, maybe it's saying like,
Not too familiar with it. I just know that this is a . The. Of course, some measure of accuracy whatever they they do in the.
And the, Rosanna was just the year.
So. Now, with deeper deep, deep learning, you know, we can do all these kind of problems and take a.
An image and classify it. We can take a audio waveform. Okay, what is that saying?
Our quarters with outside and we can do machine translation.
I think I could cut off there, but and we could and we can also do it like image captioning.
And now we're at this. Doing a label like leopard, What's happening in the image?
And, it seems like these deeper deep, Deep neural networks. They have, they have so many layers. It's a fixed amount.
They don't have any recursion, but they have so many layers. The very good at learning representations and and How to do these tile tasks.
And it seems like Anything a human can do maybe like half a second? Without this thinking is kind of using our intuition in our pattern recognition brain.
Maybe the case that these deep neural networks can do that too. So we kind of solved the space of problems that.
Take half second to solve. Not, now we want to solve the problem to take. You know, minutes or days or hours.
And that's where all the research is right now.
And now we have. You know, the Do you know, Jennifer, Jennifer, who's my voice, sorry.
So that's more right now with all these multimodal models. Everything's kinda going crazy right now.
And so instead of going like, you know, Actually, sold these slides from Jeff Dean. He did a recent talk and he talked about this.
And, We went from these. These things to these things. And now with.
Oops. Sorry about that.
Goodness.
And now with the. But I get rid of that.
I'm gonna close it out because I think it's gonna be the little time.
Did, does anyone know how to get rid of that?
Oh, I see now. Oops. There's a, Alright.
It's, zoom's kind of new to me. So, Not too much about it.
You mentioned that neural networks kind of fell out of favor for a while. Any similar driving reason for that.
Yeah. They just. They're very, they have a, very capable models. But they're hard to train, they acquire a lot of data.
And so back then we was trending on very small datasets. On very small computers. And now we scaled up to massive data sets.
And massive computers and suddenly they're doing better than everything else.
And, some people call that like, Richard Sutton, the father of reinforcement learning, kind of kind of predicted this would happen.
Okay.
The easy. Easy. Approaches. That's scale up.
Very easy to parallelize and all that. We'll in the long run dominate or everything else because the most important thing is not something else.
Cause the most important thing, it's not some sort of clever algorithm, but just being able. To to commercial numbers.
And look at massive data sets.
So that's the main reason. Just we didn't have enough computational pet power yet. And back then we had the computers that were as powerful as a.
As the cockroach, I guess.
And now, of course, we have much more powerful computers. Still not as powerful as a brand, cause the brand says.
More. Parameters or whatever wanna call that but
What do you?
At least some brains do, but my brain. Not so much, but.
I lost my place.
Okay, I'm way forward. Sorry about that.
Okay, yeah, so. So now we're in this this new age and we're kind of reversed the input the output now we're gonna take a leopard and we can generate we take a a description of a picture like leopard and you can generate leopard or We can say how cold does that sign?
I mean, Texas beaches always been possible.
You're gonna move.
Texas piece has always been possible, but it's got much better lately. Tried any of those new models.
They're pretty good. You can also, if you have your own, your own, GPU, try running work.
Bye, They think that's what it's called. I can do very impressive. It's not just the speech generator.
It can do any audio. So you can make it laugh and things like that. It's kind of cool.
So, so now we can, we can even take a description of something like, and then. We can generate an image that.
That it, that, that, that, that, that, that, that, that, that, that, that, s, that, that, that, that, s, that, that, that, s, that, that, that, s, that, that, that, s, that, that, those are called Sonos.
What's that one video, video? Thing called We can make a videos.
Alright.
Asora?
Yeah, so now we also see sore and that's this whole new level.
So now we have computers that can see here, understand and generate all these things.
So.
We're gonna do that again. We're aware of this war, but many people think that's or is basically like a world simulator.
Some people don't, some people don't think so. They think this is a dead end.
And just be cool for making videos and stuff and won't enable. Like type 2 or system 2 reasoning because it won't be able to be grounded in in reality or doesn't have a world model.
I think they're wrong probably, but we'll see.
Well, isn't there some sense, cause like I kinda touching on what you guys asked earlier. It's hard to get them to forget what they've know or further retrain them or it takes a lot of effort so it's almost easier to a point to start over.
So even if sore isn't necessarily a dead end. Somebody new innovation will come along to where some training part or something will be more efficient.
So thus all old models are almost out of date in a sense or will be.
Oh yeah. Yeah, my guys are every 6 months or a year. We're gonna have a whole new generation of capabilities and Yeah, I agree.
So the the amazing thing is, what we call the scaling laws. And it seems to hold. That as you increase the number of parameters of these systems in the data.
You get almost predictable. You can't predict the exact. Okay, bodies, capabilities, but you can predict like the perplexity, like how good does it predict?
The next token or the next word. The DFO is like a power law curve very well so you can make predictions about well I'm not sure what it will do as far as like competencies and skills, but it will be able to predict.
The next word with this level of accuracy. And usually when that when we're gonna get a prediction, we're also good at everything else because prediction is.
The right kind of thing to optimize. If you if you can look at some Texas. Based on all the evidence I Conclude that x.
Did the crime. If you can do that well, predict that x verbal. And that means you, in some sense, reasonable.
About the fast. Reason about all the evidence that you were exposed to and came to an informed decision. That's kind of what reasoning understanding is.
Of course, a lot of debate about that. I mean, right now we're learning so much about the brain too, just based off.
Learning about these large language models. Is a very fascinating time to be live right now.
So one major hypothesis is, you know, prediction equals compression, equals intelligence, compression in the sense that we're finding these very
And.
Compression to predict it and vice versa. No, you can use like a an arithmetic encoder.
Based off your predictions and do a very good compression. And if you can do compression or prediction, then you can also.
Generate intelligent behavior by sampling from those predictions. I think you probably also need to do a search.
In a more advanced way. Or like long term planning, but they're all kind of related in some way.
And if you look at look up small enough induction or even like see. Bye, Marcus Hudter.
Hutter uses sloping off induction to. To pose a model. That is in some sense at optimal intelligence.
You can't do better than it.
But of course, that's also in computable, but this is like a. Like a hypothetical upper bound on how smart somebody can be.
So now, talk about this language models.
And these, mostly these models, what we call. A auto regressive models because they learn. Hey, probably, distribution over training data by using self-supervised learning.
And that's just, you know, That's just, you know.
Good to learn this joint distribution.
We just look at all the training data.
The condition on that. And, project project that. And then we just find the parameters of the neural network.
That maximize this. I said earlier this is maximum likely to maximum likelihood estimation.
But this is hard problems.
Unfortunately, right now we have so much data. So much compute power is starting to become possible. And that's what great rise to the modern LLM.
And also, there's nice advantage to language because earlier I mentioned that Language has a very efficient.
A very high, to noise ratio. So the representation learning part is a lot easier.
But it doesn't matter with representations like we're the back and all these embeddings.
What that is is representation. And it does, this is a pretty good thing too, like with work to back.
Like if you look at the, this the first layer, doesn't look at all the context and just look at the entire context, just like the first layer, where it's looking at tokens.
Now, we may have this space right here. You may put dog right there. And K 9 right there.
And Maybe a cat right there, but like a chair. Right there. So these embeddings are some sense.
Capturing the the relationship between these these things, these tokens or words. So it's been showing that often the case is that if you take Okay.
If you take the token for King, I can't write it out. And then you subtract, these are vectors.
And you subtract the token for mail. Man, that more or less ends up in the same space.
As if you take Queen and you subtract women. So you kind of get, and maybe they're both kind of close to.
Dictator, which is maybe, less more neutral on gender. So they're learning all these representations.
And. And the deeper into the layers you get, the more advanced, the more subtle.
The more contextual, the more global representations are. Yeah, those final areas are where we use like, our generation and all that.
I need to delete this thing again.
That kind of losing my voice.
Right.
And, like I said, it is a very powerful. Tool we have. The humans are tool users and language is one of our best tools.
And we more or less use it to discuss and think about all the problems we care about, which is why.
Learning how to generate and predict language is so powerful. If you do that well enough, in some sense you have something that can use tools that can.
Reason about the world larger the same way as we can. Especially with like chain of thought prompting and we really still out all the missing details.
And, so now we have like these sparks of AGI. Hey, JI stands for artificial or general intelligence.
That's kind of moving target. No one quite knows what we're talking about when they say that, but We know that things are getting more intelligent.
More like more human like and probably in the end much more advanced because it'd be hard to imagine that human, cognition is like the upper bound, what's possible, very likely that.
Will be far surpassed. I'm not sure where the implications of that are, but even like Alan Turing, one time ago he says, once the, machines No, become smarter than us.
We should have to expect them to take control. That's a very controversial statement and I don't know where I stand on all that.
Right now I'm just, right now they're so weak. By comparison, that, like I'm messing with these small models.
They don't seem that intelligent, so I'm not too worried about these small models and I feel I feel okay exploring their full capabilities and releasing them open source and all that.
Maybe we'll revisit all this with GPT 5. I don't know. Does anyone have thoughts about safety and alignment and all that?
That sentence right at the top of the screen right now about the language is designed to have low dimensionality.
Mean there, but language. Low dimensionality.
Yeah. Well, like. Like, are you familiar with the expression a picture is worth a thousand words?
Yeah.
So kind of like that. You know, a picture has like a thousand words in it. But, But it takes a thousand words to represent that picture.
Okay.
So each word has a lot less information. And a picture and of course when we look at the world with our eyes we're seeing these pictures this not these frames over and over again.
And and they're just from they just stick to raw data process from reality. It's not it's not crafted to suit our capabilities.
Is this? Is this reality coming into our sensors? But, we use language to communicate. Clear, we use it to be very precise with the language to communicate things that we care about, nothing things that this happened in nature.
So it crafted and fine tuned. To deal with the problems we think about a lot. And language is also what you call symbolic, you know, a tokens.
Has a distinct meaning. There's only a finite number of tokens, words. And even.
If we think about all the words that are possible, sequences of them, that's still countably infinite.
So there's a the sense that these things are much much less. Complicated and But they have much richer signal.
They carry a lot of meaning. That's not able to us. But you know, the damage though has all this all this stuff in that noise, like a lot of noise that, We'll fluctuate based off the time of day.
Know the angle of looking at it this much more complicated data, data structure and But language is this? No, this discreet set of symbols that we use.
Take to communicate.
As far as you were at a question or you mentioned. Legality or how we're gonna regulate in the future or like AI sentience.
I wasn't sure. I don't every model the longer it runs the more incorrect it gets over time open AI has admitted it I think that's kind of the secret why Tesla's having so much trouble with driving cars is The longer it runs, the more, cause everything has to be reinforced or it has to have the weighted values reevaluated and it's going to constantly hallucinate
Yes, you're saying.
at a greater and greater percent the longer it's been running so even Yeah, so even if there is some hyper scary AI that takes over the world It's only, I mean, there I've watched watch video.
Webinars by the Aspen Institute and NIST and they're definitely talking about regulating and regulating it and stuff and trying to and being ahead of the curve but Even if it did it would be borderline incoherent in a couple days because it is so smart It's gonna get so off on some tangent.
On so many it's all at once on so many different things that it will be hard for it to get back in its wheelhouse.
Cause I mean, it's even happened with It's happened with all the AIs I've come in contact with even yeah even the expensive ones
You make it make some good points. Maybe though, think about this. No, when, when humans make predictions about the world and we, We do that.
We're just, you know, modeling the world, making predictions about the future. We're constantly being.
Getting feedback from the world. And we're constantly trying to, hey. I thought this would happen, but this happened instead.
So I'm gonna, I'm gonna take this information. The account, you know, minimize my error and I'm gonna, we calibrate based off what I've seen, what I'm seeing from the, from my, from my sensors.
If we were just running a, model by itself. Yeah, it may just get more and more out of distribution and more and more coherent.
But it's a sense in which. That you can also brown them. Like when you wrote compile, like when you run like a Python program, it makes a Python program.
Then it, in it. And this is okay. Then it's back the error messages. Okay, this is what's wrong with it.
Okay, right. And then it'll it won't be worse over time. We'll get better over time.
It'll make the program more and more correct. It will counteract that hallucination. I mean, who most nation is actually good. That's what we want.
We wanna be able to entertain possible worlds, possible futures. But at times and we also want it to be to be grounded in reality what was happening.
I think this is a grabbing problem. We just need to take in. External data as feedback to these systems and and and find ways for them to recalibrate and in some ways they can already do that but yeah I mean hallucination is a problem and they do get while they off course.
If you love them just talk about themselves, cause you know, they're just kinda doing their own thing.
If you're ever at Autogen, and they can do some crazy things, you know, but, It also may just be prompting like, is it doing this?
And those are not. Yeah.
We have another language model that's That's But I guess my point is that, I'm not sure that's a long term problem.
May just be a very, very, A big problem right now, but. I think it's gonna be solved quickly as far as like getting reliable outputs from these things.
But to a certain extent. The, aren't all of the, the models somewhat biased?
I mean, like for example, the, like the, the, with Google's Gemini AI with the whole image generation as to how it seemed it was biased to producing people of color.
Or like refusing to produce people, like that were like white or Caucasian, even when That was the accurate like descriptor view provided.
So it kind of seems like. Each model or whoever is behind the model is applying. Implicit or explicit biases.
As to how the the model should run. So are we really gonna get something that is fully accurate or capable of being accurate if these guardrails are are kind of always in place.
I doubt it.
Yeah.
Software always reflects the bias of whoever wrote it.
Even the even the datasets they've mentioned, there's bias and data and statistics and it's unintentional too.
Okay.
It's just the way math works out or the way that because of this formula or and So the data sets that are fed to these things, let alone the rails themselves bias can be put in and unintentionally in 2 separate ways.
More than one, I guess, indirectly.
Yeah, they're a function of their data.
Okay. Your comment.
I guess, I guess, so we're humans, I guess.
Your comments about the AI basically going crazy after a while. Do you ever watch large weather models?
They go nuts after about 72 h.
Good day.
Yeah. They don't predict anything useful. That's why they're always in a weather bureau when they're doing this.
You know, as fast as they can turn it around, they. They they run it for I don't know how many hours and then they shut it down started over with you know, whatever their initial conditions are.
Really?
And that's actually the truth with AI. They, it does the same things happens with Open AI and with the.
Elon's cars they have to basically reset the neural model every so often. I've and I've even noticed this with Open AI.
Sometimes I talk to it and it's very coherent can do OCR just fine. It can Digest documents very well.
It can produce documents very well and then all of a sudden 2, 3 weeks later it's struggling.
To do the exact same thing it could do and then you another 2 3 weeks it's back on point they just yeah it's a known fact that the longer these things run the more off the rails they get it's yeah
Well, with that case, you know, If you have like a like a long history, I could long history.
Cause like each chat session is a separate thing. The the weights right now are are they're under static.
Oh no, I'll start new chats. It kinda goes into what you were saying is It has to be constantly reinforced and us as users are necessarily reinforcing it when it goes off the rails, we'll just start a new chat.
Well then, the, the model thinks, okay, I did a good job. If we didn't do the thumbs up or thumbs down, they're not necessarily doing it internally.
That the reinforcement because the model is producing so much. It's an exponential of how much we then have to reinforce because it's producing hypotheticalically more and more questionable things that we just ignore.
And move on.
I mean, the, like open AIs open it that they're collecting data like our thumbs up and comes down for like our RLHF and that data is being collected and maybe they're training it every now and then like doing like a I run through the, the layers.
I don't know, but it doesn't learn all time. The waiter static, and then this doesn't, this runs through the Yeah, just run through the numbers.
So they're not like humans where they're constantly taking in data. Update their model. And then getting better at that or being more, more confident or whatever they're doing.
Just kind of fixed static. And, but maybe there's something to that. I mean, I'm sure that it's hard to control these things, especially if they are every now and then training it and updating the weights.
Mess things up a bit
It is learning off us a little bit to a point because LG and Samsung both got in trouble.
They were engineers were questioning it about things that they were designing and people outside of LG and Samsung were able to get them model to regurgitate the info to them.
So it's not I wouldn't it I don't know on what level it is learning but it is taking what we're asking it about and applying it to other conversations because there was a big classified thing the info that came out was technically classified and those engineers got in trouble there was a big news article about it.
Oh, that's like a new way of hacking.
And well, that's why now if you ask it to give you your phone number or something, it'll it'll it's specifically known not to.
So they had to put a thing in place because it was giving other people's info. That it was told by that personal to you upload your resume.
Well, it knows your address and it didn't know not to share that. And so it was for a while.
Wow.
Could this be actually? Could this actually be like a search where There's information on the web and it has a web search feature and is looking at looking up the data from that.
It's no the articles were referencing data that it was specifically given by engineers that was classified and it turned around and gave that info to other people when they were asking things that were similar that weren't.
They it was let's say it's a plane that you upload classified plane stuff and these other people asking about a plane, the AI references the data you uploaded.
Without even it's been done a couple of times they had a kind of change it so it doesn't do that anymore.
So I don't know if what you call that. I don't think it is quite learning because it's not changing the model.
But the data we're giving it is interacting with something.
Well, my guess would be that, I don't know too much about that, but it could just be that.
You know, they are they are clicking this data from us. All the data will be used in the future to retrain them.
And they could just be doing retraining, they could be retraining all the time. And so if you give it information about something, maybe it learns about it and the next the next generation of the model, 4.1.
Don't give them a version number, but the next update. May have some of that knowledge incorporated into it.
I don't know. I don't know too much about that. I know that training, so, so expensive and and computationally intensive that they're not doing the costly.
Like humans are just constantly retraining, but these models, where they operate is, it is these large batch training runs.
And that's how it's more efficient to do it. They also like to like to take a date and kind of curate it, make it make it higher quality data so that they don't learn a bunch of, out there that.
Yeah, of course the pre train model. It's all, it's all the crap out there, but.
I don't know much about that. You have to show me some links to it.
Here's a question that.
What do you think about? What do you think about benchmarks? Cause I keep seeing like, Oh, Gemini is.
Better than GPT 4 and these benchmarks or something like that. But that makes you wonder like how do you evaluate how good an AI is compared to another one at certain tasks and what do you think about these benchmarks?
Do you know anything about this and are they legit or they just marketing stuff you think?
Does this, the expression that when,
When the benchmark becomes target and it's no longer a benchmark. So it's often happening is some benchmark comes out.
Okay, we want to get good at doing that. We want to get good at doing that. So they get go doing that, but once you actually specialize it or in some way target your performance to be good at doing that is no longer a benchmark.
It's something else. So it was often happening is they're kind of just, ing the models to get good doing those tasks that give them the best bang for the buck or the best PR.
And or just simplicity, they just think. They're using that as their measure of progress. So let's get going to do this get good at doing that.
So that's one that's one dimension of the problem. Another one is, yeah, I mean, it's not clear how to define the benchmarks.
No, no, how to go about doing all that. It's, sometimes you get a high benchmark.
Or something and it doesn't mean anything in the world like doesn't help its reasoning. It's not clear what all that means.
I know it's being worked on a lot and people trying to come up with good big marks because you want benchmarks to be able to evaluate progress and and know what to do next but
I'm gonna skip this section. This is all. Cool stuff, but.
We can't talk about it already, so.
Actually, I, in, in, in, in, in, in, naive, this is the in.
Here's the, on my, GitHub, you can, Quillies for sl slog talk.
Slug desk, talk, dash, LLM, you can just go to pre and look at it.
QUEE Li US. There's a collab link there if you're interested.
But I'm not sure it's kinda getting late. I think I'll just explain what I show.
And.
Are you? Are you saying that last line? That's the slide deck that we're looking at right now.
No, this, and, this has the slide deck, but also has various other things like links to, Links to the collab.
Okay.
Yeah. And, other things. And, and just, well, maybe I'll click on that.
If you could just copy and paste those in the chat real quick and that would be helpful. Thank you.
Okay. Good point.
And I don't wanna spend too much time looking at this because. It's kind of trivial.
But it's in. Do you guys know what in ground models are?
No.
So to, you know, sort of deal with, like GPT forward, stuff does, I take the, towards possible.
Hi. Take the context like maybe the last in words. Or invites. In this case, I just said, roll by model.
Take the last invites. I use that as a dictionary look up. That's a key in a Python dictionary.
And I map that, that key to the value, which is another dictionary. Which has a key.
And the keys are the tokens, the next tokens that. That the next tokens.
The values are the The probability is the probabilities of accounts of those, token in the training data.
And I'll just sample from that.
This is a, Markcard chain, actually. But has quite a few number of states.
Luckily the data itself is pretty sparse, so you don't have to, so even though I have a pretty large, large context windows.
It doesn't blow up too much. So I have recently small, model that, that can do okay job.
At replicating this kind of a
And run things just do like ship enter.
And I have this basically this. Synthetic data or algorithmic data. I generate expression trees. Like, so.
Let me, let me get.
This is an expression tree. I have a picture of it somewhere.
And I just, I generate brand of expression trees. Which have deterministic values, but the The prefix, this the prop, the context, is this random.
I just do whatever.
Yeah, at some point I'll generate an equal sign. And then what comes next is this is this is deterministic.
So you're good at doing this and you've really modeled the We really model the the data process.
What do you mean by being deterministic? What are you determining?
Oh, so, oops.
So when you, give it this.
How's that?
Good. Could you kick the screen size up just a little bit?
Oh, yeah.
Maybe Control Plus would do it.
Yeah, you know.
Okay.
Yeah, there we go. Nice. Okay, there we go.
So when I.
So when we give this as as context. Like in the, You feed a context and then. Simples the next token.
That's random because the data is so complicated. In my case, this this is all random. Not that complicated, but it's random, a random expression tree.
But it has a well-defined meeting. Like some of 4 3 7. Max of 3 5 and 2 is 5.
If you sort. That.
Okay.
That's wrong. But when you start that, you should get
3, 3 and 5.
It's not that, ignore that wrong output. That was my fault, but. So.
5 and 7, right?
Yeah, it's probably this old. Like I didn't update the values and I was writing it because this is this can coded.
But, but yeah, the idea is all it's like a math or program. Doesn't have, doesn't have brand new to it, necessarily.
Input and you map that to a definite output.
So my data is a lot simpler than. Language, natural language. And that's why I chose it because this town data can be more easily modeled.
By a very simple, DUM, inGRAM model that. Did I expand earlier? Just a dictionary dictionaries that maps to counts.
I'm just.
Okay.
So, and here's the call, you want to run it, you can mess around with it.
Wait a minute. Okay.
I posted the articles of it leaking company do. It was Samsung and then Google's attorneys have come out and said that It's leaked internal Google documents things that are very eerily similar and it's happened so many times that Chat GPT is even had to put in patches so it would help stop doing that.
Oh.
Thank you. I'll look at it later.
Wonder what those patches do to the model.
Okay, so you, you can sample from this. It's like, And.
That's very simple. Let's make it a little more complex. So, Do this.
Just make a more complex expression tree. Now you get all this crazy stuff. Any other you can ask it to?
That's pi 2 complicated and make a little simpler.
So you generate this data, these expression trees, and then you ask the, ingram model.
2 to train on it. If he gets, if he's not data. It will eventually be able to learn all these kinds of algorithms.
But, and this is the most inefficient way. To write a program to do these kind of problems.
But is this there for don't experiment? But of course, you wouldn't train it, you wanna do like it.
This might be kind of slow under under under on their notebook, but.
Okay.
Yeah, let me do it. Usually for in, model to work on these complicated kinds of expressions, you need a lot of data because it doesn't have the, it's not very.
This doesn't learn good representations of the data. This stores it. This this, No, you're giving a big enough lookup table.
Bigger than universe. And that thing can be intelligent, but It's not a very official way to generate to do intelligence.
And my Markov model is kinda the same boat. It just stores the raw data and uses that to make predictions about the future.
There's only one way in which this model
On the computer if you can run it this is too slow. And it's not worth opening up on my computer but If you run it, you can do various predictions like you do some.
You know, it'll predict, okay, when you do a, it predicts,
I can't see my Oh, there you go.
So,
Let me do a. You end up protecting this whole thing.
Oh no.
So this prompt completion. This top set up here. You give him a prompt and it'll complete that prompt.
And in a way that resembles the training data. This is what this is doing. If I give it this.
It is really closed. This sum of one is one. You give it this and you then do the equals sign.
This is also old, but we output 6.
Let's, this, I've never seen this. I've never seen this input. The train data, it may not have.
In my model, what it does is. This, okay, I've never seen this before. Many didn't see this.
Maybe it's too, maybe the context is too long. Only sees the last like 5 Byte or whatever.
So it throws away this. So that doesn't know what it is. You know what? I think this is the maximum.
So it's okay. I'm gonna imagine this is a max. Take the backs of 1, 2, 3, and it gets 3.
So the, the context, of course, having a member of the far past. Sometimes it's a member of the very distant past.
Be able to produce intelligent outputs. My name is model. If it doesn't, if it doesn't have the whole thing in context, it doesn't mean very stupid and just drops.
The oldest tokens kind of inducted by us is the older the token is. The older the Observations were seen in the past, the less relevant it is.
But this case is probably the most relevant thing. The sum is more relevant than anything in this.
But my model pose it away. And that's the inductive bias. It's not even, it's not a good inductive bias to this algorithm because because You know, it often ends up throwing away the operation.
But kind of gets gets the point across that the context window, the inductive biases that exist in the model.
These things are very important.
So, I'm talking about this. The guy kind of pitch in all this so The bottom line is this things not very good.
And we can do a lot better with large language models and and so I think next I'm gonna So my application.
I'm not.
Yeah, just talk about tool. Now, now I wanna make a tool that can.
Process natural language queries and map those queries. To complicated elastic search. Queries, which is kind of The JSON.
And. Let's see how it does.
Where's that?
Oh.
On this one, I'll in the end, I'll give you access to it so you can play around with it.
But, For now, I'm just gonna go ahead and. Hello.
So I have this.
Actually, I have a . 1 s
Hello.
If you go, you can read about what's going on with the thing. So the idea is that with these language models.
It'd be nice if we made every API endpoint, every user interface, if we made it.
And if we gave it natural language understanding. But of course, since it's everywhere on every endpoint and every interaction, be on the phone.
We probably want to use smaller models. Oh, faster, lower latency. You can talk with it real time, maybe get back to you and have a conversation.
And also the small models, but maybe these 0 demand, maybe these, PASS, or more narrow domain, they don't need the big model like GPT for.
So we can just fine tune it or specialize it for this model, smaller tasks. So the trial run, I'm gonna.
I'll use Elasticsearch as a back end. To enable natural language queries.
You know, and that's it's kind of a copy. It has complicated structure like say you wanna look for.
We have articles and we have articles index and we want to find, we, we wanna, account of the authors.
One account of the articles by offer. Well, accounted the authors by a number of articles they wrote.
And this would be the.
This one, this would be the. Can you see a mouse cursor.
Yeah.
Right. This would be the, the JSON query that. It's like, search expects.
You want to do this and here's what you need to need to write. And you can get longer complicated too.
You can, you know, you can, you can imagine, bullying queries and, and fuzzy approximate searches and All that is a very powerful engine.
But it's hard to use and I'm sure there are better APIs for it but I just wanted to use the JSON.
And just map natural image queries to the right. JSON query.
Now, you, the fast API engine to, write the. API front end. Hey, I used to write it for me more or less.
Before this, I didn't do much front end work, but GPT kind of makes it easy.
So we expose a single endpoint. It has a lot more endpoints on it, but when we care about is this.
Now, we say the index name like articles or books. Let me say a natural natural language query.
And hopefully the limit model converse that to some meaningful JSON. And we're good.
So I probably for as a test case, I populate the, index with 2. 2 indexes or the articles.
And, I just download a bunch of books from the Good Board website, all those free free books. And I just populated with just some relevant, feel like author, publication day, whatever.
And now let's look at some code.
Okay.
So I just. This is, Here's where all the magic happens. It's not so, but this is where the.
The, attempt was made.
So we give an index. It'll be given a a natural language request, just a string. But they also have some other parameters like which, do you want to use?
Along with, 13.1 or. Or even open, open AI, whatever phase is.
I have a local model running. On this end point. It can swap in different models.
And then this one, I'll explain, but it's basically. I use GPT for.
Okay.
To populate the. Did to complete a database with examples, high quality examples, or given that language queries on that index.
And the hope is that we can in context train. The smaller model based off this teacher model. And then we can use a smaller model to do all the, most of the computations.
Very quickly.
So I hear my system, from this. This function. And so let's look at this one.
Okay
So generate system message takes in the. Just stuff is. This first ignore the use examples. Well, let's say use examples as false.
So it goes through. It gets the mapping of the index. Let me show what that looks like.
So I guess the mapping of the index.
This is what that looks like as Arthur field. Did they say get very complicated? As publication day.
This is this thing. A one layer list is not nested, not doesn't have anything going on already, but.
And title. And based off these, this metadata, the names of the fields. The hope is that the language model can do a lot of.
Prediction about what these things mean and then be able to take natural language queries and construct reasonable reasonable search queries on your behalf.
Let's go back to the code.
So here's the system system prompt. No, no examples in this case. This one shot, one shot learning or 0 shot learning.
Just asking the model to come up with an answer without any. Other context except listen here. You know, it's kinda funny how you, how you have to prompt these things.
Kind of. Puffed them up, your world class. Natural language.
To natural language to elastic. Your task is, blah, blah, blah.
I had to do this. Deeds. Field keyword because the small models and then not GPD 4 but the small models kept getting confused and adding.
Mixing in old and old. Old, syntax induced syntax.
And this wasn't. Like wasn't being very coherent. But this kind of solves into that.
And I say, okay, here's the index I showed you before. Here's what that looks like.
Based off which end which which index you choose. It'll be a different mapping.
And then I just, I do that and then I. Same generate and that's a query based on the following.
That the recurrence of content and then we go back. To the main. Here's the system.
We pop this is a measure with that data. And for the user message.
Okay, users turn. That's what Grammer's query is, request, and then the query that the user submitted.
Nice, and I submit that to
Opening not necessarily because the basic if you change the base your your URL you can you can query A local model that does have to be.
Open AI. So in this case, in the end, I also try my alumni models and along with CPP models, but.
But GPT, 4, to provide will vastly better at this task. So I started with them.
And then I get a completion from the model. Does anyone have any questions?
I'm kind of running through this quickly because it's
It seems like you made a bit of an AGI in a sense.
Well, is not that smart, but I appreciate you.
Not quite, but like in a basic form, cause it's doing, yeah.
Well, I want to say that, but I appreciate the I think I think it's pretty dumb still, but like GPT 4 is pretty good.
But this is just a simple task of. Of doing search queries. So I don't know about AGI, but.
Is this way of
Well, I mean more of it's going out finding something and then bring the knowledge back to aggregate it.
Oh yeah, yeah.
Which kind of then falls into the category of an AGI a little bit.
Maybe you're thinking, A, A, maybe.
I mean, yeah, maybe because I just know I've seen people Use that to aggregate data in a sense.
Yeah.
And they were referencing as it as that, but yeah.
Okay, yeah, I mean Yeah, this is nice way to use these models is you know they're already kind of intelligent they understand language.
And then you, you wanna prompt them, do you wanna give, that you wanna pull on data? Specialized for whatever you need to do you can kind of you can you can find tuna but not through the weights you can fine tune it by prompting it by conditioning it.
I give you the And so, you know, in this case, I'm just giving you a data about, The database is structure and things like the more data you give it in some cases for the big models, the better will do the small models, they kind of get confused by the context.
So. You have to be careful. These small models aren't quite up to par yet. Maybe we can find 10 or something like that, the way something can do better.
But. That's definitely something that needs to improve. And they are improving very quickly too. No, 6 months ago, the best open source model was terrible.
But I know I didn't even bother with them because they, so that.
So finally, I just, I submit the. The completion, I say, okay. Okay, Well, predict.
The Sword Query. And this response format this means I want it in JSON as a Jason object.
There's actually a lot of interesting resource in this area where you. This doesn't do it.
I think Open AI, for instance. Just fine tuned or model. To output JSON better like to stick to the JSON.
Grammar but there's there's another you can do something else all and they don't do this you can actually Sample, ogrammer.
So, so it can sample from all these words, but you can say, okay, don't sample from everything.
Here's the things that I know where valid next. You know maybe if in JSON accordingly bracket must be next because I saw that So, or, you know, the things that the syntax.
That's down and there was stuff you can throw away. So you sample from that subset. Of the distribution.
So it's a conditional distribution and you can enforce it. Precisely. But This is actually a problem with that though because You have to use things like beams.
You have to be able to explore multiple paths simultaneously because it's very easy to get stuck in.
And, and to get stuck in some. In some place where it's not able to complete the request.
So you have to kind of explore multiple passport, multiple paths. You can't just do a 1, one through, but I'm not sure what's happening with.
Here, but I think Open AI just fine tunes the model. I'll put JSON.
And it works pretty well in fast, I guess.
And I just, I do the standard notation.
As for. Calling the model and then I,
Another response this time. I have this query. Predicted what the Elasticsearch JSON query looks like, given that natural Assess it to that.
And then I search, I use the elastic search engine. Defined whatever the user is looking for in that index.
No, I just process it and I add more data to it. Somebody can see what's going on.
The response. Now we can, we can try it real quick.
Let's see.
So we hope.
Can you block that font 2 or 3 times?
Yes. Yeah, yeah, yeah.
That's better. Thank you.
David. Okay.
So let's look at.
We'll just kinda do, look up Gutenberg. That's the index full of books.
Good. Good busy is this one. I know this one works pretty well, so I used that first.
Fine articles by link in the contain, getting his brother or something like that. So. We're gonna do kind of a fuzzy search that the user may not know how to do that.
I'm using GPT 3 3.5. GPT forward does much, it nails these queries all time.
GP 2.5 less so but it can do it too. That's pretty good too.
Walmart. It's not so great.
But it's a much smaller model. You know, that's how it is, I guess.
These things get better with scale. Let's ask you. Hmm. So.
The phone, make is getting birth address. By, LIKE, in contains Gettysburg in it.
Here's the query made. You see the query is pretty complicated. So like, North American content, Kittsburgh, bullshit, doing all this, is making a pretty complicated source for.
From this thing right there.
Then I also say for diagnostic purposes, I show what the. What the system message was.
So let's go back. We've been trying to different one though, cause like I said, it does terrible sometimes.
Just, let's, let's, research for.
Okay, like a hundred books. You guys have any recommendations?
I know, but actually it's just 2 articles.
I'll be, I'll be this depends in an example these days. Show me how many?
Too many.
How many angels fit on the head of PIN?
We try that.
That's a good question. I don't think this
That's a good one.
Hey, if it comes up with the answer tonight, a lot of philosophers over the century will be happy.
But you must remember though, it's only a generating, the elastic search, JSON query.
So if they, if they do a good job with the query, but, but, I'm sure the Elasticsearch database doesn't have that information in it, so.
Let's see, I'm kinda curious. So for this question.
That's not very.
Didn't know do it apparently. Is that okay? Give him the.
Given the given what the article looks that the database looks like I have no idea. Do this thing. So I'm just gonna.
Make it the title. And magic. Yeah, and does it does it doesn't like the Is that this using matching?
Is using like B and 25. Using it, information retrieval matching mechanism.
So it doesn't require this exact phrase. Is finding documents that contain. These words the most? Like maybe it continues any, like maybe blockchain contains the word.
And that's these, returned for whatever reason. Hey, that's not seems relevant. Exploring the universe.
Yes.
But that's We did. Let's go to, back to the code. Okay.
Okay, so Actually, I'm gonna show how the.
The the small models don't do good. Thank you very well. This used 13. One or 2, 13.
And then we also use the local host.
Let's see how, let's do, let's do better. Let's see, and more, easy query this time.
We all the by.
This one doesn't do well this one, so I'll do that one.
If you get bombs this question for whatever reason, so.
That's not good. Is doing a large request.
Oh, didn't let me see that happened there.
Yeah, I didn't. I think it didn't even construct valid JSON. So I'm just, it just totally bombed.
I didn't even, didn't even do.
I think that's what happened. I'm not gonna read that though. Yeah.
Hey Alex. This is great by the way. Thank you. Very much.
I was. Trying to follow along and I was wondering where. Where are we providing the context?
Of the library of of these of these books. Like I, you know, I was seeing directly where you're.
You're sending sort of the Bootstrap statement to open AI and the actual. The actual prompt.
But then where is the context being provided?
Get the context. So the system message contains the context and the The, the, the, the, and the user query.
The system message contains the information. About the search index. So when you Do a search on this index.
That's a database. Elasticsearch knows how to retrieve information about that database, like the structure of it, which with, I had, what, feels that hasn't it, the type of those fields, you know, they can also have metadata.
I didn't use metadata. But it will populate the. The context with information.
About that. And then the whole the hope is that the language model itself will see all that. Hold that the structure of the database and for what is for Okay, and then, okay, given this.
When, when, a user asks. This natural query.
A proper JSON. You know, the proper JSON query. Should include these keywords.
It should include this, If it's like choose an Arthur, if they find Arthur's whose name starts with Ben or whatever is gonna do a search.
On the author's offer to they can do much more complicated things.
So it's just the system message is just the metadata. It's not the actual.
Yeah.
Publications themselves.
No, no. I mean, you can do that, but no, this is this. Very efficient.
I just wanted to use Elasticsearch to do it quick. Search on the database. But I wanted I wanted to enable the wanted to use the large language model of this to map.
Map, Natal language to to The right kind of, kind of queries. So, Yeah.
So it's basically just a translation, right? Like you're typing in natural language and you're using open AI to translate that into an elastic search.
Yep.
Query which then you're sending separately to Elasticsearch running on your laptop or somewhere.
Yep. Yep.
Nope. Yep.
And really that part of it has nothing to do with the AI, right? Once the AI generates the query, and you just sending that to Elasticsearch.
Yeah.
Understood. Thank you.
Hmm. You want me, well, also one other thing I guess is showing you because Because these small models do so bad at this task, that's actually the first time it failed at making JSON though, maybe the buck of my code.
Usually, bit, well, JSON, even longer 2, 7 B. Even, It was making, but it just wasn't making a good query.
So to combat that, well, maybe, if I feed. In this context window. These small models, good examples of what a query might look like.
Maybe we'll do better. So. I asked you between 3.4, 4.0.
Give me Given this database. And this query, what's a good? A good, mapping.
It was a good way to. No, I did that because, you know, obviously I can't ask GPT for each time.
If I did that, I may as well use GPT for. Which I wanted to provide the small models with context.
About how to construct good queries and maybe it can generalize from there. It's all about using in context learning.
It's okay. I kinda see the pattern. Here's where you, here's what you should do.
And it's a pattern based on. The existing index. No, this index has This kind of stuff in it.
And that's where it's going to be for that index. And I thought, well, maybe this is gonna.
Maybe a few examples will tell the small model. A lot of rich information about how to construct these queries.
This is Tribal
Let's first look at the examples. So here's. And every time to make a proper database.
So I just, my database is just a dictionary.
But obviously you can expand this and make it more more. Sophisticated. So I had, here's a, an example, here's a, I have.
Example of natural language query.
Show me books. That one query we did. And here's the, here's what the career looks like.
For articles, Here's what that looks like. Number number number of articles by offer. But you can imagine having a bunch of these things.
And you can, you know, maybe 5, examples or 3 or whatever is good for that model. Sometimes they get over, they can easily get overwhelmed by it.
Do you vectorize those before you send them and or do you just send them as text?
Yeah, that'll be the next step.
Yeah, so what kind of explain?
So now we're now here teaching the smaller models.
So we feed examples and yadda yadda.
And so my finding was. They don't do very good at it even with examples. And like maybe is this polluting the context window?
They just didn't. Have enough relevant examples. And these queries are pretty, well, they map to pretty complicated objects.
These Jason objects weren't very straightforward lot of times. I mean, it didn't do bad.
Usually it produce, on that can be enforced too, but this wasn't the best. You can imagine how I can be enforced too, but this wasn't the best.
Okay.
You can imagine how I can do much easier, much better. So I thought, okay, let's use what you said.
Retrieval, augmented generation. Maybe the smaller models. Need to be said with more relevant examples.
So we can use, to find. Examples that are relevant for that search query. Yeah, so user looks for a certain for a certain natural language query.
We can use rag to retrieve other queries. That would like to add the past for that index.
And then feed those as examples. And to the. Into the context window of the small language model and see how it does there.
I haven't did this yet, but my guess is this would improve it even more because if you have a You know, get as many examples as you can from GPT 4 over time.
You know, Maybe you sample from GPU T 4 every now and then. And over time it just learns to do it, it'll learn to do it better and better.
For the for the small models too. Because I have a bigger database. Of these vector, this vector.
Another option is to find tenant. You can see my my get help for this. This is probably one of the best ways to train them because they are they are pretty small models.
And if you can just train them in the weights. We train their neural network. Then that's probably a lot better.
And you can probably fine tune them for that task. And it's easy to, in this case, it's very easy to get the data you need.
You can just generate it either with GPT 4 or even synthetically. Or using algorithms, but in my case, I just.
Sample from GP 4 or 3.5. Kind of as an expert trainer to get the raw data for training for fine-tuning and then better do it.
Okay, so.
Did you guys want to see an example of how it incorporates examples into its context?
Sure.
And we're going to use examples true.
We're using the small model again.
I think I must have a bug somewhere because we wasn't doing this before.
Yeah, I did. I need a Python package. And, you know, I've said, I've, about it.
But then again, It was working for, and TPP to, 5. I don't know.
Anyway, the main thing I will look at is the
Response, sorry.
That's ignore this. Here's the.
The example, including in a
You are world class, natural language. Okay, okay. Okay. Now here's an example.
Now here's why I' you an example.
Yep, and so.
Again. Again, I hate to suggest this, but up the font size again, please.
Oh yeah, yeah, yeah.
Maybe even one or 2 more clicks.
Oh.
Yeah, Gary's got a teeny tiny screen over here.
Is it controller enter or plus?
Control plus.
There you go, there you go. There we go.
Okay. You control shift.
So find articles about, okay, so here's an example of NLQ in this corresponding elastic query.
For this. Index articles.
So shows the That's a language query. Yeah, so was the The JSON query that's an example for it.
And then hope is that we feed this into his context. They can learn how to do a better job itself at Do you really?
These things.
So with the small models, they didn't do, they didn't do very well at this.
That even with those examples. But we talked, we talked briefly about how maybe improved using the vector stores or fine tuning.
So that would be future work. I've already started fine tuning them. Yeah, you have the data collected.
I just, I actually sampled GPT. Like 100 times. This actually I'm a cheesecake. I just did it in chat window.
And then from that I actually. Then used.
A local model, but, longer. Like the next biggest one. And it did okay.
Did you know any more examples? And then my intention is to use all those examples to fine tune like tiny lama, like a very small model.
They can just run lightning fast. You know, if you have like the time long, sorry, how fast it is.
I'm sure you guys know.
It's, it's literally the first time, but this first time, it takes a while.
But yeah, but it, it can, I mean, it's not always the best text. But it can generate very quickly.
And if you find 10 it may be good enough. So then you can put these things at all kinds of endpoints.
Like on APIs and phones, whatever. And then have these things. Running at lightning speed.
And it may also be good for. For large, language models to use. Like to outsource them for Tribute tasks or for doing that.
Before exploring all kinds of possibilities like a lot times doesn't need. Good outputs. It just needs a lot of different outputs to sample from and explore with.
So that's kinda. My thoughts on that and I think that's. Yeah, let me see where I'm at with my.
My talk.
So, Any discussions or you want to see anything else like when I see Autogen or anything?
Although it is kinda late. I don't think, but I think maybe I'll use the.
Passing very fast.
Maybe have some final questions and discuss those points.
Get in my open one. Yeah. Yeah, well, Alex, it's a Great, review of broad area there.
So yeah, that kind of neat. So, certainly if anybody has any. Questions kind of here.
Now's a good time to wrap it up. And Alex, I meant to ask earlier, my fault I should have inquired earlier, is your last name pronounced toll or towel or?
TA second.
And I wasn't, I was just getting setting down at the beginning tonight and I neglected to do.
An introduction to Alex Alex. Oh recently got a second master's degree from SIU Edwardsville.
So he both has a master's in cops I and also in math slash statistics.
And, He's obviously. I've been applying it very well to the field of artificial intelligence and Got a real tour to force here tonight.
So. Any, any questions? Well, first off, any question specifically for tonight's talk.
Or comments, suggestions.
I've got kind of a question.
Yes, Steve.
Back when I worked at Boeing back in the early nineties. We were experimenting around with.
Neural nets. To do some things. And.
We would train them on. Some data sets, training sets, I guess. And they had. You know, certain parameters.
You know, they went from, you know, various parameters. Went from, you know, one in to the other.
And we got. Networks that Did a really good job of answering your questions. As long as you stayed within the boundaries of the training sets.
Yeah.
If you went outside that boundary. The further you got outside that boundary. You know, you were, it was just no way to guess what it was going to answer.
You know, so wherever you were going to use this thing. You had to make very carefully sure that your training set encompassed everything that it that you could ask it or that you would ask it I guess would be the proper way to say it.
Is has anybody? I've been completely away from this subject since About 1996. And so I have no idea.
Has anybody? Is any of that changed since then? Can they operate outside the boundaries that are training sets now?
I'm just curious because I've been away from it. You know, for so many years.
I just have no idea what the current state of the art is. No idea at all.
Okay.
The Zoom, wanna take that?
Yeah, anybody.
There was one that like taught itself Hindi. Or something it did it on the fly there's a couple write ups about, they queried it.
And in about some book or something in The books now, yeah, that wasn't necessarily the title wasn't in English.
And it came back. And with an answer and they were all perplexed because the book it didn't have an English translation and they looked at it and it turns out that it What?
I think they queried it even further like to say reference something in this book or something like that and then it gave the answer and then it referenced Hindi language.
That it didn't know that it wasn't shown. I don't know, I don't think it was Chat GPT, I think it might have been a local model.
I'm not sure exactly, but there are there are a few write ups about that.
Hmm. Could you, Yeah, if you can find one of those write ups. Could you post it on the One salute.
I'd love to read some of that.
Yeah, that's, let me see here. Okay.
Please do.
I could show something that I think might. Answer the question an interesting way. Maybe. Alright, I'm sharing my screen real quick.
So hopefully you can see it.
Yeah.
Yeah, we can see it.
Alrighty, so I'm gonna run L 2 locally. On my local computer and I'm gonna ask it.
What is the St. Louis Linux users group? And it knows about us. It knows what we do.
And I guess it's got scraped our website here so it knows all about us. And then I can ask it.
Something like this. So I say here's an abstract from an upcoming presentation for the next St.
Louis Linux user group. Please summarize this abstract into an enthusiastic tweet, okay? And then I'm gonna paste in.
The abstract and send that. And this is the abstract from the talk I gave a few weeks ago about system D timers.
And here at generated like what looks like probably a bit too long for a tweet, but pretty close.
And Sean will reveal 3 game-changing features blah blah blah that's got like emojis and it's got hashtags And so I think this is kind of an interesting maybe use case for how we could use these local language models as a user group here.
If we wanted to. And we could, you know, maybe do some other things here like we could.
Ask it like, hey, what are the main technologies discussed in this abstract? And then it could, you know, it does a decent job here where it's like, hey, system D timers, Chron tabs.
Shell code, you know, some of this isn't the greatest, but You know, it's better than having to like type all this out yourself or think about it yourself and then you can ask follow-up questions like hey what topics would be good to follow up after this presentation.
Okay.
And it could maybe give some suggestions like oh people like this one here's some other type of topics that that might be good for future talks like integration with other tools or technologies like Ansible and Puppet, you know, I think that's right in line with.
System D timers and stuff like that. And so you know, you're saying outside of, you know, you ask the question about outside of this data set.
Well, I think that these models were trained trained on basically the whole internet. So it's kind of, it's kind of hard to answer like what's outside of the data set.
But you know, anyway, I think we could use this as a user group to kind of Make things easier or you know do some of this kind of stuff and really use this to our advantage if we if we put a little work into it.
That's kind of stuff and really use this to our advantage if we if we put a little work into it.
So anyway, just some ideas I had that I wanted to share. Thanks for indulging me.
I have put the article link to it was Google AI teaching itself Bengali.
Oh, you really?
I put I put that in the group chat. Yeah.
My zoom is acting a little weird.
That is, what's going on there?
And it frequently acts weird.
TBS News.
Oh yeah, that one. Cause they, yeah.
Hey, Alex with your masters in computer science and now statistics and your familiarity with making local models or trying to train them.
I'm surprised you thought about getting into trading. In the future with based on the model. I've been cruising the idea and I have not nearly the knowledge base you do.
So I'm interested. Your thoughts on training a model that would do a bit of trend analysis, market sentiment, cause it could ping the internet.
It can. Get the sentiment of the Yahoo news. It can read it all faster than we can like.
Just the example given a second ago, he showed it. Going through just the previous.
Presentation and thinking critically about that so It seems well within the realm of possibility to run locally and do something like that.
No, you know, by will house, but, but now I'm just trying to get the job with.
Like matter or opening eye but of course they're highly competitive so Yeah, I haven't really thought about doing that.
But yeah, you know, I mean, this seems interesting.
I mean, because every exchange has APIs and API calls that can be done. There's even meditator 4 and 5 that can, there are exchanges specifically for interpreting code and and Pine script and I feel like you might be surprised how far you'd make it if you put a weekend into it.
Yeah.
Yeah. Well, think about it. Thank you.
Yeah, I see I found your article here. Interesting.
And you never know how truthful they're being, you know, it's hard for me to believe that Google scraped the whole internet and didn't run into Bengali before, so I don't.
You know, but it has been claimed. So I'm not, yeah.
They scrape the whole internet.
Well, I'm assuming that's what Google did. I mean, that's their Yeah, I'm assuming that's what they had it trained on.
So.
Kind of find that a little hard. To believe, but you know.
Yeah.
Okay.
I mean that's what their search engine does to to an extent. So I yeah, I don't know it's hard for me to believe they did it without it knowing of Bengali
But that's just looking, a search edge is just looking for keywords and key phrases.
I think. That's interesting.
There was an Aspen Institute webinar recently and the guy one of the guys that works is like the NSA or department defense I can't think of his name he said it only it only takes 45 min.
To scrape through the whole internet. I don't know what he was using, but he says it's a 45 min thing and it's done.
Wow, you're no kidding.
Yeah
I'm not sure I believe that.
And then he said, you know, it's just a series of tubes.
Yeah.
Yeah, no, I mean he works for he works for DOD or White House. I mean it was it was a very prestigious cybersecurity conference.
Yeah. He was, yeah, I was surprised when he said it too, but he rattled it off like it was not a big deal.
So I don't know. I don't know how long it does take Google to index everything or even showed in or any of the.
I don't think you could ping every computer on the internet in 45 min.
I mean, you know how many? I can't imagine how many 1 million are out there. Yeah.
No, I, yeah, I agree. I agree. Let alone the bandwidth. You would need going in and out to make all that, yeah.
The calls, let alone the indexing of said info that came back.
Yeah, that sounds to me like I'm not sure I believe that guy.
But,
Hey, ask it. What's the biggest problem with converting? 30% of our cars to battery electric.
I'm kind of curious what it would how it would answer that.
Cause that's all over the internet.
And that's kind of another reason why AI is super relevant is because there's so much SEO around.
Positives and negatives of both. It's hard to get. I find myself relying on AI potentially.
To give me the new story that's not so
Just drowned out in all the Google links or whatever because it's the other so much just SEO out there that it kind of ruins Google search in the sense or whatever search engine you're using to try and get info.
What do you mean.
Hello.
Search engine optimization. It's what people put in like websites and stuff so they rank higher.
On the, yeah, on the Google like page or whatever. And it's the reason why when you search things now it's not nearly It's honestly like Google search has been getting worse because SCHOOL is built in into so many web development platforms that.
Yeah, they all they make so much noise and clutter that it It was better to get info directly from AI who had.
Kind of it can read through the links that oh that's the same that's the same that's the same these are different and it aggregates the info more efficiently instead of me having to scroll through.
2 to 5 of the first pages to find something that's different or an actual. First hand account of a new story.
Oh, I'm sure. It's better than any any of us can.
Yeah.
But
How would I pose that question about electric cars to one of these? Ai engines.
Okay.
You gotta start with, you are a world class. Climatologist or something like that.
It's weird how like that like these AIs are so good at like impersonation, you know, that's what I've always found weird like even the system prompt that, was showing I started off with you are a world class such and such.
It's like you gotta, you gotta get the AI to role play if you want a good answer.
Yeah, well, I mean, the reason I pose that question is because I already know the answer.
It's not super simple, but. Not that complicated really.
I've found. Getting it to sometimes the
Good prompting helps, but or is a good way, but I've kind of found good results with just jogging its memory beforehand.
Being like, tell me what you know about the climate. Tell me what you know about car production. Tell me what you know about emissions.
And I don't know why I get way better answers on stuff like that. And then I kindly slowly lead it into.
Okay, well then with everything that you know, that you know, what is your thoughts about this? And then I can get away more elaborate coherent.
Conversation like response instead of just generic copy. Of some other article it found.
Yeah. It's it seems like you're bringing some of that knowledge to the surface or. Bringing it higher up so we Season never.
I'm sorry.
Actually, I think it sends the whole chat history each time. So every time you send the new chat response, I think it includes all the previous.
Chat history, right? Because it's not actually, it's my understanding that all your, Open AI is a rest based system.
So it's stateless between the calls. So if you ask it, hey, summarize this and then you ask it in another call.
So what was the main character's name? The only way that it knows like the context of the your previous question was if it sends the entire history again.
So by you kind of doing those pre prompts when you do the final question or whatever, it's probably sending all of that both the question and the responses every time.
To, you know, just building and building and building on that context.
That almost sounds like a bandwidth and a memory problem.
But it stays locally on their servers. It only, I mean, it's honestly, cause it's not sending the whole chat to you.
Hmm.
It's only sending it's more recent response but that I mean that's It's possible, cause I've even noticed it when I do trading stuff or let's say I'm having it write me cover letters or helping me build a resume or trying to just I what are the what are the top known books for for financial trading or options trading or what are the top known books for cover letters or information pertaining to
it. And it seems to, yeah, I, and it would make sense. That it's sending the whole thing again.
I'm surprised it doesn't increase chances of hallucination, but I've wondered why it gets better.
So it makes sense. Yeah.
Sure.
Hey Steve. What's your question? I can I can ask Chat UPT and share my screen.
Oh, okay. Well, let's see.
What's the biggest problem with? Converting 30% of all automobiles to battery electric.
You type like I do.
Yeah.
Okay.
Let me know when you're done reading.
Alright, go ahead.
I'm not a fast reader anymore.
Okay, there more?
Mmm. Second line under number 4 is not correct.
This one.
Yeah.
Hello, I've never really seen a case where the total cost of ownership was lower.
And on the price of electric probably.
Well, yeah. There's a lot more to it. Those suckers are heavy. And the insurance companies don't want it.
Or if you ever, if you get an offender vendor, yeah.
You have to replace a $20,000 battery. Ad nauseam at infantitum.
Yeah.
I don't know, it seems like it just gave me a list of all the possible problems. There's a few hard ones in there.
Like right now there's
Not enough rare earth elements to make all the batteries.
There's not enough power in their in our grid. To charge him. IS just not there. It's not that it faces problems.
It's just you can't. You can't do it as fast as a government wants you to.
I have no doubt that in a hundred years they'll all be electric. But.
Okay.
That's actually a solid question. Ask it if there's enough electricity in the grid for all of us to drive electric cars.
Yeah.
Ask it if you humans will be here in a hundred years.
Okay.
That's a better question.
Fraser more accurately, You should have asked us is there enough generating capacity?
By the way, I have it. During these long winded, outputs because I haven't thinking like step by step.
I'll show you what my
Cool principles. I haven't doing all that stuff. So it produces long outputs, but it's good to like math research and stuff.
Okay.
Bye. It's pretty long context when it real long prompts. And I always tweak it, but.
It seems to work pretty well for a lot of any kind of step by step thinking.
But yeah, it also produces very long prompts. So, you gotta tweak it.
So you said you use this to help you write Python. And I know I've heard the chat GPT actually can run.
Python code and then. Like it can actually run it and I guess if there's what like a compiler error it will know not to show you that one or something.
This apply make it Friday prime, time number algorithm. And I'll put it.
So now is writing code. To solve the problem I asked it. He's gonna run that call too.
Look at the output, and it will tell me.
And that's probably, that's probably correct. Is this, is this code looks correct? This prime is, I mean, this is correct.
So, of course, you know, it's seeing plenty of data for prime number. For prime number of finding algorithms.
So it's not. That's amazing. It's pretty good at any kind of border plate code.
It doesn't do too well when the code gets long or it's kind of new or novel, but it's getting better.
And, but the main thing is they can use anything that's been like boilerplate, or play code, solve problems.
I don't know that that are published. And even though it's a combined existing solutions and in trivial ways, sometimes in novel ways, but.
But you can rely upon it to generate code and, and then. I'll put it incorporated into his answers and give you a response.
That's pretty reasonable. To answer this question. And of course, you know, this could be hidden normally.
You don't have to even see it, you know. All you see is this. 2,000 prime number is this.
Yeah.
That's interesting. So analyzed your prompt realize the best way to give you the correct answer would be to write some Python code and then run it.
Yeah, you made comments to yourself to find the, the 2,000 prime number, we can use a simple prime funding algorithm.
It does that. I find it writing Python for a lot of things that I don't even ask for.
Yeah.
I've done document OCR and it writes Python to run a Python program.
Yes.
Yeah, it actually I find it defaulting into Python quite a bit.
Yeah, it does Python. Python is a very powerful. Tool to have available to you.
Yeah. Well, lot times if I wanna like, if I want to install the application, I'll just tell Why a program to convert this to PNG or you know some stupid little program that I don't want to download and install.
You can just ask Chat to write the program for you and then run it.
That's awesome.
Yeah, you can even imagine the future like more and more of this like most applications, if the one off things that Chat GB for you.
Runs it, then you throw it away.
You just made me think of something. Can it write Pong?
Like pong the old games the beeboop 8 bit game
Right, Palm does that. Oh.
That would be interesting.
That's even, it's even installing. The library. This may be kind of easy for it because it's using.
Nice library to do that, but yeah.
That's really interesting.
Oh yeah, I mean, it's very powerful.
This is by take a while. I can either. Could you interrupt it or stop it?
Yeah.
No, no, I don't want you to stop. I just wonder if you can.
Yeah, you either. Just the, stop button. I'm always stopping that because like They set up and Just tell me what I wanna know.
Which will run it?
Sure.
Can you run?
Yeah.
Oh yeah, please do.
Okay, just copy code out.
Yeah.
Good night.
Good evening.
I can't talk.
Okay.
Type better than me, so it's fine.
So in this case, I need to install this module.
So it didn't do right, but you can actually feed this back into it.
Yeah.
We'll try. Oh, this is okay. The driver. I'm not sure I have that.
See if it can run within the instance of Chat GPT Tel-tat PGPT just to like put in the command to run in Chat GPT and see if it all.
But it'll run like a things that don't produce like graphic outputs, but that one if I want wrong, we can try it.
But First, I'm gonna I'm gonna produce the outputs that it gave me the the error message.
See what it says about it. This, the whole thing and copy and paste it.
Let's stop it.
We're just having to produce a simple ASCII game.
We can't either, got it, got it right.
Not a cold.
It will.
Next I gotta ask you to use assembly. I'm kidding Okay.
72 lines.
What?
Let's just run it as root.
Oh yeah, here we go. Yeah.
Bye.
Yeah.
Is they wonder how AI is going to take over. Yeah!
Okay.
Well, I haven't running my computer sometimes. This program called Open Interpreter. They'll type like LS.
They'll say, oh, I think you're or somebody, type, show me my files.
See, I think you want to run LS, shall I run it? So sure, you know, you can have a bit more less controlling your computer.
Mr. Wise stand up. I didn't saw it.
It's PIP, right? Not apps, but you saw that as your user with PIP, right?
Yeah. Yeah.
And then you're running script as route. So. You're gonna install it globally, but maybe that's not worth it for this little game.
I don't know.
Yeah, yeah, I think you're right. I can do it.
It's all good. That's still impressive though that it was, cause yeah, and like in, you know, how long before is able to do like rid racer.
And then 5 more years. It'll, I mean, yeah.
Yeah.
If the
That's what I, yeah, that's what, that's what, this is my, my conspiracy.
I'll go ahead, sorry.
Well, But it even says, please note the keyboard module may require. So even, to, okay, you told me to, okay, yeah, it said to do that.
Okay.
Hello.
Oh, so you must be rude to use this library. So if you just run the run it as sudo, I could be wrong.
Yeah, even mentioned it.
Yeah, I think you need to do like pip install global, right? Yeah.
Okay.
But it totally, yeah, in okay, so that's the 8 bit era, you know, if you can nail Mario and then it's the 16 bit era.
You're
Then the 32 bit. I wonder how far how long that is away.
I can't wait, man, cause I think VR is so cool, but the problem is like creating those 3D environments is so labor intensive right like it's just so hard to create really good threed environments by hand it takes forever but if we could get AI to generate really really good threed environments we could have like the most immersive VR games, I think, or it'd be a
lot cheaper to develop the VR games, I think.
Well, some of that. Does exist, cause like there's no, there's procedural generation and I know you can play no man Sky in VR.
So it kind of is the thing that's also I asked earlier on the if it was making assets or if these just videos because they're That is it's played with a little bit
Of AI generating assets.
That could then be used proceduralurally to populate a whole world.
One last time, and we're done. With this,
Oh man, you spelled the file, right? We're gonna get it this time. I feel it.
No, no.
So can't talk at all.
Okay.
Hi.
Ghost!
Yeah.
You're playing.
Oh, dude. It did it.
Yeah. Huh?
And
So are you up and downing or is it up and downing too?
It's a bit walking, but it's doing it. Kinda. That's kind of it's paying itself, so.
Okay.
I can't even. This like a rewarding
Only when he move is not to play.
You want to play thermal nuclear war?
I do that.
Yeah.
Yeah. Yeah.
Okay.
Looks like the guy in the rise meeting big time though.
That's pretty good
Also does things that mean is not perfect but it's More than you might expect, well.
More than I could do.
Yeah.
Yeah, really.
In like a 10 min span. It's yeah.
Yeah, he wrote like, for, G, Games, this, some of them didn't work for my system because they require certain libraries.
What about towers of Hanoi?
Yeah, I can do that one pretty easily. I'm sure because it's, solve problem.
But anyway, I am I am poked.
Good presentation. Thank you. Yeah.
Alex, thank you very, very, very much. It's been a fascinating evening. We appreciate it.
Thanks for attending. I know I'm kinda long winded at times.
Covered a lot of good material. So.
Well, thanks for coming. I appreciate it. Thanks for your questions.
Alright, you want to book into Gary? And this has been the, February.
A day, yeah, don't forget anybody is here to tell them about the ballots they're available on the main page.
Oh shit. About that. Yes, this is election month anybody who attended 2 meetings. Any of our meetings in the year, 2023.
If you attended 2 meetings, you're qualified to vote. We're voting for the board of directors.
Hey, let's see. Where's the link, Stan?
Okay.
The on each web page. There's a paragraph that as the ballot link there is WWW.
Sluug.org slash ballot 2024 dot PDF.
Okay.
Go ahead, Sergei Shibakran.
Yep. Let's see, this is the, yeah, send your ballot in.
Okay.
You can print it in. You can print it and buy US mail or you can Do the PDF and email it in.
I've got a Steve, excuse me. And thanks to Stan, it actually is works interactively.
I have never seen anything like that.
I wasn't sure it was working right. Anyway, vote early, vote often.
Looks like it to me. And that's why we have the balance time stamp. So we take whatever the most recent ballot somebody sent in.
So. Alright, this was Thursday the 20 s of February 2024 it's the St.
Louis Linux user group sent to his log. Alex Towell was our speaker this evening. And he was telling us all about large language models, LLMs in Linux.
And so we really appreciate