Hey, don't have. Hold on. Please turning on the recording.
This is the, St. Louis Linux users group. Date is Thursday, the 20 first.
Of March, 2024. Our speaker tonight is Sean Tweethouse and he's going to talk to us about large language models and rag.
So we're looking forward to that. And, see what else they put on here.
Oh, retrieval augmented generation is what Greg stands for. Cool. And, let's see, I guess that's the important part for the book.
Let's see. We usually have announcements and then a very brief, QA at the beginning of this.
One thing we're always curious about is if somebody has not been at one of our meetings for a long time or if this is the first time you come to one of our meetings, we'd love to find out how you wound up the finding out about us or what motivated you to come tonight.
So if you want to just flip on your mic and tell us locally, we're more than happy if you want to just type it into the chat.
You can do that too. So. But yeah, let us know. We're just curious.
Hi, came tonight because James Conroy reminded me to check my email. My name's, James Moore.
Great. Welcome, James.
Who are you talking about, James? What? Who? I don't know that person.
James Conroy.
Yeah.
Oh yeah, you never shows up anymore. And he's just giving me a hard time.
Okay.
Let's see, other. See if anybody else wants to jump in, you're more than welcome to.
Otherwise, put it to us in the chat.
Hey, my name is Pranab. I got, this link through Robert Siteek.
And I'm interested in LLMs and rags. So just. Perfect.
Excellent. Excellent. Welcome. You're welcome back any time.
For anybody who's not familiar with the group. We actually have about 4 meetings a month that are oriented towards.
General membership. 2 of those are more presentation oriented like tonight where there is one or 2 formal speakers on a particular topic.
The other 2, the one is called the new users, log. And basically that's for anyone who feels like they're new to you new to Linux or maybe they're not new, but we're finding that, hey, a lot of times the best questions and discussions come up in that type of a setting.
So it's always good to have some more familiar hands who are around at those meetings. To tell the newer users, oh, you could do it this way.
You could do it that way. The usual rule of that meeting rather than having a formal topic is whoever considers themselves the the newest user.
The can claim the right to ask the first question and basically pound that question home until they get an answer to it.
And and then we go on to the next person and seniority so The other group that doesn't have a set topic is the slack, St.
Louis, computer club. And that is one of the oldest groups in St. Louis and it kind of is more oriented towards the hardware itself.
And so if you got hardware problems or Wondering how to get a particular copy of Linux on there will even talk about how Microsoft runs on a computer.
Yeah. So, sorry, that would be a Slack meeting. They're on our calendar.
Let's see. Perfect looking up there.
Okay.
I'd like to I'd like to point out the disembodied voice. It's talking is Gary Meyer, our MC for this evening.
Okay. I should have.
Thanks. You should introduce yourself every meeting.
You're right, you're right. And actually, turn this over to James if you want it to be the.
Well, you know, if this This is chairman of the group. I'm Stan Reichardt.
I do a lot of the saw she's behind the screen kind of stuff.
You can blame me if we screw up the announcements or the, the timing and the. How to connect and all that.
I've done it. I've screwed it up in the past at least twice. Whatever.
Those are the, URLs through our meetings. There are other meetings. The new Linux group is a beginner workshop.
The Slack meeting is a configuration workshop. Hardware and software. Even Apple machines on occasion. If we have somebody in the room that knows that.
A lot of people that we haven't been shown up recently. But that's just the way things are.
We have a major calendar. On our site that lists up in the Metro area and also remote stuff as we find out about remote, and so forth.
We have a mailing list. We sent out the announced list and our discussion lists of general questions because can be asked on that discussion list.
We don't. Talk about something at the end of this meeting. You can.
And can you, the discussion. Our other topics can be brought up on our discussion list. And this meeting is being recording, so it'll be posted up on our archives, which can be reached from.
Our home web page.
I've been showing the. Please still board items here in the front. So I'm gonna stop sharing stuff here.
If you could flip over and share the calendars and that would help.
We have a calendar.
Hey, how about that?
Hmm.
Yep.
Hmm.
But by way of announcements, I just want to point out 2 things on the calendar, which are a bit unusual.
And I apologize to those of you who are from out of town because they're both local. Events here in St.
Louis that don't really have a video or an online portions. The one is going on right now.
It started today actually and it runs through Saturday and that's the high school first robotics competition it's the regional competition so it's basically for, most of Missouri and, the First, I forget what that acronym is.
It's an organization sort of by Dean Kamen. I will be about 15 years ago.
And so they they basically high school teams are Every year they change what the game is going to be and your high school has to build a robot or more to perform in that particular competition.
They make it so it's not battle bots it's in fact you actually lose points if you damage somebody else's role but but, every year it's a different game.
You know, sometimes it looks like basketball or soccer or building a structure and that way the students all have to you know apply their their own thinking power that year to figure out what type of a rope up would do that.
And, so tomorrow is, the, I believe the first round and then Saturday is the finals.
It's being held down on the campus. St. Louis University. And the information is there on our calendar.
So the first robotics.
Stan, I gotta put on my other glasses to find that.
You wear glasses?
Okay.
Yeah, how about that? Hey.
There you were and
I wonder what those things on top of your head were.
Yes. Anyway. So that's 1 thing on the calendar and. Don Ellis, one of our longtime members, he's down there.
He's, one of the mentors for high school team. David forest you know one of our longtime members i don't see him on the on the roster here tonight.
List of attendees, we've had a number of guys who volunteered to actually be mentors at different high schools over the years and it's a rewarding thing.
It's fun to do. Tell you one quick story of the the one year. The robots one of the things they had to do is pull themselves up to the top bar on.
Whatever the goalpost was at the end and everybody interpreted that is what you have to build like 2 arms on the robot so it can reach up to kind of like do pull-ups to pull itself up you know from one bar to the next.
And, some school came up with a completely different concept. They basically made it so the front 2 wheels of the robot.
Towed in and the wheels grabbed the bar. That was the vertical bar and it just inched its way up the bar with its wheels and and then went up and touched the the top bar.
And the neatest thing was before the competition even began you could tell this kid was going to have a future of going to law school probably for being an intellectual property lawyer I guess because he came up to the judges table to say okay before you all disqualify us for saying this is illegal I just like to go through the rules with you and he literally read through the rules for that year and explained how they had, you know, every one of the points is something their robot was doing
correctly. And so it really is interesting seeing how the high school kids actually accomplish this. So that's 1 of the neat things about this weekend.
The other thing is the audio engineering society, Central States Conference. It starts Friday. Friday afternoon about 3 o'clock and it runs Saturday and Sunday as well.
And Stan, if you could click on, yeah, and yeah some of you may be going audio engineering well that's not really computers why are we talking about that well as many of you have noticed in the age of postpandemic when Everything is remote meetings.
We're all using Zoom and, how many times you've heard somebody say my microphone isn't working.
My my earphones are I can't hear you on my speakers. So yeah, we gotta know a little something about audio and basically this conference is all about audio engineering so there are people there talking about how microphones work how speakers work yeah the neat thing a few years ago is they want to be to sit in on one of the panels because they desperately needed somebody who needed somebody who New about
networking. And it turned out they really didn't need me because the professional engineers they brought in from different companies.
The whole point was that Now, rather than running speaker wires and microphone cables. A speaker has an IP address.
A microphone has an IP address. And the whole thing was being done as a networking exercise. And it was interesting between the old timers who had to learn this and the new timers.
And just last week I was talking to a guy who teaches, audio at the junior college level and you were saying that they literally are going to throw out their old analog mixers and analog amplifiers because yeah, they really just use it to to teach the kids how it used to be.
And, he said, nobody uses analog equipment anymore. Everything is being done. Digital now.
So, yeah, it has more and more to view with our area. So they've been doing this for 18 years.
Like I say, it covers the central states, but they also bring in professionals and academics around the country and sometimes in international.
And so it's it's a bunch of good talks. So you're all welcome to to come.
It's a preview.
Let's see with that. Something here and chat. Let me see what it is.
Okay.
It's a very good one. Oh.
Oh, somebody, Phil was nice enough to put up the definition of first. And,
And no, I do not know how Android audio. What if there's a, you know, what causes delay and that would be interesting, but I bet there'd be somebody at this conference who could probably explain it.
Let's see, Stan or Lee, is our calendar considered an embedded? Google Calendar.
Yes, an iframe.
Okay.
Okay. Alright. Alright. And one, One last good comment about, the first robotics.
When St. Louis was actually the international finals. It's down at the football stadium for 7 years, I think.
And what we got that award it to St. Louis. Dean Cayman himself came to do the official announcement.
And, so it's fun hearing him talk about it because he pointed out that Kids are not stupid.
They pick up on these little clues from adults, you know, what things have value. And so, you know, what do people talk about?
They talk about, you know, sports figures and, you know, and all, yeah, you guys get paid millions and millions of dollars playing sports.
And there people who get paid millions and millions of dollars for being you know rock musicians and and he goes but How many people actually make a living at that?
You know, it's like the number of people who are making millions of dollars is very few. And so, something that society really needs and benefits from is STEM.
And so, you know, something that society really needs and benefits from is STEM. And so, you know, we should be giving kids the clues in high school that no you want to study science, technology, engineering.
You know, learn how to do that. It contribute to society. There's good money in it and there's a lot of jobs that, you know, will be needed.
And, so I do that and, It was very interesting. He also said the high school is damn well better send cheerleaders and the high school pep band to come to a robotics.
They do. They are cheerleaders and there's, music bands who got this about.
Anyway, enough joining on about that. Questions Q&A, you can ask any question at all, but it'll have a lot better chance of an answer if it has something to do with computers.
Does anyone have any? Kind of numerical, comparison between the kinds of search speeds you'd get with plain text versus something that you would ask a large language model to summarize your documentation.
Yeah, really.
Sorry, not summarize. Access and, yeah, I think the word is access.
I have not. Go ahead.
I would I would expect that if the question is which one can retrieve the data faster like a search engine type of appliance or a large language model.
It's gonna be a search engine because the large language model is going to have to inest that document store and to an awful lot of processing on it to get it ready, whereas a search engine is just gonna have to index the keywords and if few things like that.
So my thinking would be it would take a lot longer for A I type LLM 2. Be able to give results.
But it's quite interesting the results that they do give because, I, kind of like what Bing was doing with Chat GPT there for a while.
You could ask a question that you might ask a search engine. And it would summarize, let's say the first 10 results.
And then importantly, footnote its summary. And I thought that was great. So instead of having to click through 10 different links, it would just sort of summarize them.
The reason I ask is I've noticed having to use work, Windows at work for many years that the indexing service when you search is excruciatingly slow compared to simply doing DIR slash a slash S.
And redirecting that to a file and then loading in its memory and doing control F to search for it.
It's like an order one operation compared to however many I node reads from the hard drive so That's why I'm interested to know if it's.
Okay.
Worth the squeeze, I guess you could say, if you wanted to. Index all of your corporations documentation that way.
Yeah, do you have the actual index or you running the search without the index? Because the indexer is turned off by default.
Last I looked.
Yeah, the
I am not fully aware of that. I assumed that it was on.
It seems extremely counterintuitive that an indexed search would be slower than an actual file system crawling.
What? Think about if.
I can't speak to it specifically, but I mean, think of a database you you definitely want indices on on things to make to make it faster.
Right, but how would it perform without its index? Like what would it speed be? That's what I'm suggesting is It's trying to do the search assuming the index is there and the index isn't there.
Yeah. I think that's correct.
So it's having to do it on the fly. There's an NTFS file system parameter.
Relating to indexing.
God, it's been for ever since I've messed with this and it's turned off.
Because I think the indexing service is on by default and it basically is excluding everything and you include the things you want.
It's weird because it takes CP resources when it rebuilds. The index much like update dB takes resources on Unix when it rebuilt its index.
So it's there. Most people and I think by default It's not indexing anything.
There's a program called. Everything on Windows. That does it very quickly and far better than what Microsoft does.
It's called Everything on Windows.
It's called everything.
Oh, I just love those marketing guys at Microsoft.
So I guess the advantage of, RA. For LLM would be.
That it's kind of has an inference on what you want and then could point you in the direction of some group of files and specifically one file or more.
Whereas an indexing service would have to You know, use actual keywords. And then find through those.
Beetree nodes, I guess.
All right, well, unless there's more follow up on that question, by the way, We usually, you know, make a little bit of time for Q&A here on the front, but certainly after our speaker finishes, although I'm sure they'll be some questions at the end.
There's also an opportunity for more just general questions that you know idle not idle chit chat but you know just more informal discussions of stuff after the actual talk so for those of you like to stay around for a while afterwards but with that unless anybody has anything they really really need to ask before we get into the main talk.
I think we're going to turn it over to Sean. Oh, Sean is, done a speaker a number of times.
To some really nice presentations for us. And so really happy to have him speak again. And, Sean's been a sysadmin for a decade and he's Guru, shell scripting and Ansible.
And he's warned us that, you wanna ask anything about dot files with MRC, or modern CLI tools, be prepared to stay for the rest of the evening because you can.
Talk about all for a while. But right now, he's going to tell us all about large language models and rag.
So, that I'm going to turn over to Sean.
On you.
Thanks a lot. Appreciated. Sorry, what was that, Stan?
How's it gonna say you're muted but you've unmuted yourself. I'm sorry.
Yep, I'm just getting everything lined up here. Make sure I can see the chat. I will respond to the chat if you put something in there.
So feel free and also feel free to. Interrupt me at any point if you if I say something incorrectly or if you want to add something.
Feel free to interrupt me at any point. The presentation tonight. I'm gonna talk about using local LMS and RAC for content promotion.
I'm gonna start with my normal disclaimer that I am not an expert and anything that I'm about to talk about.
Please think of this more as I got this cool thing to work on my computer and I'm excited about it and so I'm sharing it with you.
This is very much outside of my depth. This whole thing and so You know, if you ask a tricky question, I'm gonna say I don't know, but feel free to ask it anyway.
And also, if you can't tell, I asked chat GPT to generate this image over here.
And of course chat GPT couldn't spell disclaimer. So I had to like edit it in Gimp and I'm really bad at that but I think it works given the content of the slide.
I'm sure James would agree.
Looks fun.
I know, I know.
Actually, Shawn, that just looks perfect.
Yes. Yeah, Chatty PC can't spell. So whatever. Anyway.
So, some of you all might not really realize this, but we have the lug has a YouTube page.
And Twitter, profile. And Stan has been doing awesome work and uploading some of our presentation archives.
To youtube which i'm really appreciative of because i think this could be a really you know we have the presentation on our website archives on our website but YouTube is just very convenient and it can also be somewhat good to for.
Extensibility or for sorry content promotion and people discovering our lug.
So I definitely like uploading these here. This is great. Stan, but when he's been uploading them, he's just been putting the group name and the date.
And so if you're coming here trying to find the one about like get or whatever, it'd be very challenging to find it here.
But, so it'd be better if they had like, titles, you know, and also the thumbnail seem to be kinda just a random frame but you know this is all volunteer driven right so complaining is not gonna do any good just like an open source.
Project it's all volunteer driven so complaining won't do any good but what can do some good is you know coming up with ways to make with technology that makes The volunteers lives easier.
And so that's what I want to investigate tonight. And here's our Twitter page. You can, and this is, right around the time Elon Musk drove Twitter into the ground is when I stopped posting on here.
But perhaps we could get a mastodon page going or something like that. If we could find some tooling that might make it easier.
To come up with tweets that are relevant to our upcoming presentations. And so that's really what This presentation is going to be about.
Using local LMs and rag to think of ways that we can do better content promotion here in the lug.
And I think it was a month ago. We had a LLM talk that was much more, you know, theory and background and theoretical.
And this one's going to be much more hands-on and demos.
This is Stan. I'd like to point out that I uploaded the year. 2019 and I wanna do the rest of the days to bring them up to date.
Shit.
Okay, let's go back to Telegram.
As I was putting them up there, it allowed. It allowed. It allowed, me to put in.
Somewhere the title to the topics I put the list of the on for the units group the base topic and the the main topic.
And the title for for what was in there other than that date. And I just don't know where it where it went.
As far as a, a standardized front image. That would have required as far as I know.
Editing the entire video at least my skill level that's the best I know about it so Hopefully this will help.
Yeah. Yeah, I completely agree with you. And like I said, definitely not complaining. Okay.
Oh yeah.
You know, all you do more than anybody else. So I'm certainly not complaining. I'm just, you know, hypothesizing about some ideas that could make us have some better promotion to outside outsiders when they're investigating our longer when they're looking around.
And, and I'm trying to come up with ways to make all this stuff. In the middle too, so if you think this stuff might be fun to play with or you want to help us out.
We could use some more volunteers too so for sure agree with all that
Yeah, and a volunteer would include an observer because sometimes somebody just asking questions helps everybody else get work done.
Believe it or not.
Totally.
So, so the way that I like to do things is I like to do things locally. On my machine, I, you know, so the way that I like to do things locally on my machine, I, you know, because, especially with these LLMs, they actually kind of push your hardware a little bit, you know, because, especially with these LLMs, they actually kind of push your hardware a little bit, you
know, especially with these LLMs, they actually kind of push your hardware a little bit, you know, like I have a 2070 or something GPU.
In this machine that I'm on right now. And when I'm running these local models, it'll it'll max it out for a little while.
And I just kinda like, I don't know, something about that. It's just like, wow, I'm actually using the hardware that I paid for.
The to the fullest extent. That's kind of fun. Whereas, you know, if you're using a cloud service and all of a sudden it starts using it too much, you start getting scared like, oh no, how much is this gonna cost me, you know?
Oh, I can't. Oh, I better stop it. You know, it's like the exact opposite in the cloud as it is locally.
And the models that I'm going to be using tonight are they're not open source, but they're like openly available, I guess you could say.
The models like a large language model is not code. So you can't really apply, I don't think a software license to it.
At least I wouldn't think so because as far as I'm aware and I could, you know, definitely could be wrong, but I think a large language model, like when you download it on your computer, it's just a bunch of parametric weights that are stored, I think, as numbers.
So I think a large language model is just a file with a ton of you know gigabytes worth of numbers inside I don't know but it's not code I know that and so you can't really say this large language model is open source.
But they have made these models available, for people to use freely. And there's a lot of great open source software around how to run these models.
And one piece of software is O Lama, which is the one on the left here, which is what I'm showing on screen right now.
O'lama is is, a way to run these models on your, local machine very easily.
On my machine very easily. On my machine very easily. On my machine, I'm using the I'm using it on Windows and it's only in So it will crash.
The, will crash at least once during this demo. So we'll get to rag on Windows when it crashes.
I did use it on Linux, pretty for quite a bit before and it was rock solid on Linux.
It's just the windows. Is still in beta and it crashes all the time. But on Windows, the software is rock solid.
But O Lama helps you run large language models. Locally. And O Lama is not the only open source software that I'm going to be using in my demo tonight.
I'm gonna be using a Jupiter notebook, which is open source. Running Python code, which is open source.
My Python code is going to be calling, using the Lang chain library, which is open source.
Lang chain is going to be calling O Lama, which is open source. And O Lama is going to be running a large language model.
Such as mistral AI or llama 2 which are openly available and so you know this is not just chat stuff anymore.
These LLMs in this, this is stuff that is open and we can use it on our own hardware and we can really play around with it and have fun.
And it's not just all locked up and cloud vendors and corporations. So I like that. So let's see this and action here, this kind of stack right here.
Okay. So I'm gonna pull up BS code. This is where I've been running my notebooks from.
And we're just gonna call, remember the problem we're trying to solve is the YouTube titles, the tweets, you know, generating those and making that kind of stuff easier for our volunteer so they can spend less time or use the time that you spend on more effectively.
Okay. And so this is my Python notebook where I'm gonna be doing this first little bit.
And so you'll see the first bit of code here, we're gonna choose a model. So.
You know, we could choose llama 2, which is from meta, or we could choose mistral, which is from a company called Mistral AI.
I'm gonna go ahead and start with Mistral. We can try the other one if we feel like playing around.
But I'm gonna start with Miss Joel, so I've ran that block. And now, down here, I'm gonna call LLM invoke, cause I save that, OLama into this LLM variable.
And now I'm gonna call LLM and Vogue and I'm going to ask the LLM what is the St.
Louis Unix users group? So let's see if the LLM knows who we are, right?
Cause we're trying to see, can these things help us with our YouTube channel and our Twitter? Okay, so let's just ask it.
Do you know who we are? And if we look, we'll see my GPU spiked up there.
While it was processing that. Okay, I'm gonna be reading a lot of AI generated text tonight.
So I hope my monotone drone doesn't. Put anybody to sleep. If anybody would like to volunteer to read a couple of these prompts for me, I would be I would be happy but I'm gonna start reading this.
So when I asked it, what is the St. Louis Unix users group? It responded by saying the St.
Louis Unix users group STL UUG is a local chapter of the International Unix and Open Source Community.
Unix is a family of computer operating systems that was developed in the 19 seventys at the University of California, Berkeley and became widely used in academic research institutions as well as industry due to its powerful features and flexibility.
The St. Louis Unix users group provides a forum for users of Unix and related systems to exchange information.
Learn new skills and network with other professionals in the field and and it goes on. The group holds regular meetings featuring presentations on various topics, okay?
And so it, you know, I'm not sure if it knows exactly who we are, but it gets the gist of it, right?
And so it seems to somewhat know at least a little bit or at least it can act like it knows who we are.
Now in this next prompt, I'm gonna ask the LLM, what was the base presentation to the St.
Louis Unix users group on February 2021 and the LLM responds by saying, I apologize for any confusion, but there seems to be a misunderstanding here.
I did not give a base presentation to the St. Louis Unix users group on February, 1021 or at any other time.
So.
Yeah.
Hell! Oh.
That is extremely unhelpful. Let me rent it again and see if it gives us something a little bit more helpful.
No, yeah, it's still completely wrong. Okay, so it has no idea. What presentation was given on February tenth and frankly it's being a little bit surly, okay?
It's saying it didn't give the presentation.
Yeah.
Hello, with an attitude.
Alright. Yeah.
What, what did you feed it, Sean?
Nothing, I've had it nothing, I'm just prompting it. So this, this file here, we're just prompting the LLM.
This is very similar to opening up chat GPT and just typing in this in chat GPT.
Obviously we're not using chat GPT we're using a different model but the technique that we're using is the same as just opening up a chat and and typing this in to an AI.
So we're not feeding it anything yet. And so it's, Okay, so it's this is not helpful, right?
This is not gonna solve our YouTube problem. This is not gonna help us with our Twitter. This is like not good, right?
And so the problem here is that, you know, large language models are trained on a very broad spectrum of data.
But that training set doesn't include any new anything new or proprietary data or domain specific information, right?
And so especially if you think, you know, if you think about it, we're going to want to use this to generate tweets about presentations that haven't even happened yet that are gonna happen next month, right?
And so we can't rely on the data that the large language model was trained on to help us at all.
And so that's kind of what we just showed there. And so. What we need to do is we need to do a little bit of prompt engineering or maybe something that I'm gonna call augmented.
Generation. We basically need to give the large language model kind of an open book test. So we need to tell it, we need to give it the answer and then ask it the question and ask it to generate those tweets for us.
So let's take a look.
At what that looks like. And I'm just reading, Alright, go ahead.
Maybe a silly question. But what is the value? Of giving somebody something the answer to turn around to give it the question.
It seems like the circular loop you're better off just doing it yourself. Or is it time spent?
So you or somebody else can ask it that question or a permutation of it. In the future. And see having trouble seeing the value.
Yeah, and I'm gonna demonstrate the value. Very clearly in a minute. But I will give you a preview.
So I'm giving it. The title of the talk, the abstract of the talk, the person who's giving the talk.
And it's gonna take that information and generate YouTube titles and tweets. So I'm not giving it YouTube titles and tweets.
I'm giving it what the speaker is giving to Slug. You know, and then it's giving us YouTube titles and tweets from that.
Oops, spoilers. Okay. So let's go to the next one. Now I am calling this, let me clear this.
I am calling this augmented prompting or augment to generation. I don't think it's a real term.
I think it's just something I made up. Okay. But But we're gonna do this.
So this part here is the same thing where we're just selecting a model using LANG chain, Python library.
And now what I'm gonna do is I have this file called system prompt.md. Let's take a look at this file called system prompt.
Md and in this file I'm doing what is you know referred to with prompt engineering called a system prompt.
So the first thing I doing is I'm telling the LLM how I want it to respond.
Okay, so I'm telling the LM and LMs are very good at role play. I don't understand why that's something way over my head, but they're very good at role play for some reason.
So I'm telling LLM, you are the head of marketing play for some reason. So I'm telling the LLM, you are the head of marketing and content strategy for the St.
Louis Unix users group. Use the following pieces of context to answer the question at the end. If you don't know the answer, just say you don't know, don't try to make up the answer.
So I'm telling it how I want it to respond and then down here I'm just giving it 2 paragraphs about who we are, So I'm gonna skip the first paragraph.
It's just some really basic information about us. And the second paragraph, I say the St. Louis Unix users group have met continuously since we incorporated in July, 1992.
The meetings consist of 2 presentations. The base presentation is a short presentation of an introductory topic.
The main presentation is a longer, more detailed presentation of a more complex topic. All of our meetings are free and open to the public.
Okay, and so I'm just telling. So this is like a system prompt. I'm telling you, I'm how I wanted to respond to me and I'm giving it some really basic information.
Cause all of the questions that I'm about to ask it pertain to our group. And so this is kinda just like a baseline that I'm gonna send with every prompt.
Okay, back to the code.
Okay, so I am going to read this system prompt using Python code into a string. The string is just called system prompt because I'm not very original And here I just had it print out for some reason.
So this is now read in to a variable. Now, the next thing that I'm gonna do is I'm gonna do the same thing, but I'm gonna load information about February tenth.
2021 presentation. Okay, so let's go here again. And if we look at this markdown file I have, I have basically all the information from the February tenth, 2021.
Website what was on our website at that time because Stan is awesome and he saves those And so what we have here at the top, we have h 1 header that says the name of the group and the date of the presentation.
And then I have 2 level 2 headers. One is for the base presentation and this one down here is for the main presentation.
And on February, the tenth, 2,021, the base presentation was given by our very own Steve Stagman. Out of applause.
Yay!
Thank you.
All right. He was given a presentation on them.
Ens.
Yeah, I was, he must be asleep. I figured that would get in the pipe up with Okay.
Yeah.
Alrighty. Anyway, yeah, so I'm gonna read it out. So the base presentation on February, 12,021 was titled Speedbar the presenter was Steve Stegman The presentation abstract.
Was as follows. Emacs Speedbar for fast file access. Speedbar is a tool written in Emacs for organizing or listing files and directories.
Speedbar is an integral part of Emacs which has been in use since version 23 dot 2 and 2,010.
And it goes on and on, right? And it also has, and I also put the information about the main topic under the main heading.
So this says the main presentation on February, the tenth, 2021 was titled, A Gentle Introduction to Get.
The presenter was Robert Seetec. The presentation abstract was as follows. From its inception in 2,005, get his gone from a version tracking tool used by Linux kernel developers to a corporate mainstream for collaborating and teams on projects.
And on and on so you'll see that that I have basically taken what was on the website back in February, 2,021 and I've put it into this markdown document in a specific format.
Okay? And now in our code. I am going to. Read this into a variable called I don't know, I'm a bad at naming.
And I printed it out below just so you could all see it. The total length of the document plus the system prompt is 2,924 that's going to be important later.
Okay, and now I'm gonna prompt. The LLM again. So I'm gonna do the same.
Llm invoke. That we did before.
But this time, before I pass in the question, I'm gonna pass in that system. Prompt and the doc string.
So the system prompt is, you know, how I wanted to respond to me and some information about our group.
The Dockster is that markdown file that I showed you that has the base in the main presentation information.
The question that I'm going to ask is, what was the base presentation to the St. Louis Unix users group on February tenth, 2021?
Alright, I'm gonna run that. Once again, we should see my GPU. Spike.
Okay, there it goes. Alright, it says the base the base presentation on February tenth 2021 at the St.
Louis UX users group was titled Speedbar and was presented by Steve Stegman. The presentation was about Emacs Speedbar for fast file access.
Speedbar is a tool within Emacs that has been in use since version 23.2 and 2,010 which allows users to organize or list files and directories and traverse upwards and downwards from the current position and it goes on and on.
Okay, so got it. I got it right that time. When I feed it, the information and then I ask it to answer.
A question from that information. It gets it right. So we're given the AI an open book test.
We're saying here's the book. And here's a question, give me the answer.
And it got it right. And so now on my next question here, I'm going to, once again, I'm passing in the system prompt.
I'm passing in the Markdown document and I'm saying generate several short and enthusiastic tweets that summarize the base presentation to the St.
Louis Unix users group on February, 2021. Use future tense. Alright, let's see what it came up with.
Anybody volunteer? . To read these
Exciting news, on February tenth, T. Steadman will introduce us to Emacs game changing speed bar tool.
Get ready to access files. Faster than ever before. 2, Emax, Speedbar. Prepare for a productive Bush with the upcoming base presentation that slew on February tenth.
These Deadwin will show us how Egnac Speedbar can revolutionize file exploration. Food productivity hacks and Emacs.
Alright, thank you, Stan. Appreciate that. So I think these are pretty good. You know, these ain't half bad, to be honest.
I got emojis. You know, it's got hashtags.
What are the hashtags used for? I know.
Okay.
Yeah, so on Twitter, that's a way to group tweets together, right? So, so if If I use the hashtag slug and you use the hashtag, then somebody else can go in and search hashtag slew and they'll see both of our tweets.
That's really all it is.
Okay.
I'm, the, MASSED, it on and fed a verse has something somewhere.
So yeah, those are pretty good, right? I think those are not bad. Like those, I mean, better than nothing. And for free, right?
We didn't have to do much work at all to get that out of the LM, right?
And that's dinosaurs. You're not good at marketing. They really are not.
Great. And now you don't have to be. So, okay, let's keep going.
So let's ask it. What was the main presentation on February tenth, 2021 and let's see what it knows.
So it says the main presentation to the St. Louis Unix users group on February, 1020, 21 was titled, A Gentle Introduction to Get.
Alright, so it got that one right as well. And now once again, let's ask it for the tweets.
And the whole time I'm still passing in the system prompt and the dock string. And we're gonna ask it to generate some tweets for the main presentation that was about get.
Alright, so would you besides anybody besides, and I want to volunteer for this one?
Exciting introduction to get coming up at Slube. Discover how This version tracking tool transforms into a collection powerhouse for teams.
Learn about the 3 main parts of the get project working folder, staging area and repository.
Get ready to create and manage projects from scratch using get. Hashtag sloop, get Okay. Join us at Slug as Robert Sytech takes us on a gentle journey into the world of get, unleash your inner developer and gain a solid foundation and using this essential version control system for your project.
Get ready to branch out into advanced get users. So, get and tap.
Oh golly.
Oh man, use the pun. You see that? Put get here. Alright, thanks, Dan.
Thank you again. Appreciate it. Alright, now I'm gonna do the same thing. Oh, Yes, sir.
Hey, Sean. Could you do me a favor? Could you scroll up? From that tweet a little up.
Sorry.
When you've printed. Yeah, yeah, when you printed the main presentations. Docsring.
Oh, right here.
Oh.
No, even further up in your the doc STR. When did you print the doc STR?
Oh, yes, sorry. You want this doc?
Okay, interesting. Okay, okay. It was there the working folder, staging, and repository was there in the.
In your dog stream train. Okay, okay cool cool cool cool. Thanks.
Yeah. Yeah, I didn't read out the full thing, but yeah, it goes into a bit of details here.
So yeah, I kind of, didn't invent that.
Okay.
Hey Sean. John, while you're there, where did that come from? Did you build that?
Your doctrine. Yeah.
One, the abstract. No, no. So Stan has, let me show you. So Stan is doing awesome.
He kept. Sorry, this one. He kept all of the old versions of the website on, And so this is literally the website how it looked from 2021 And so all I did was copied this.
And paste it into that markdown document. Just copy and paste the stuff right off here. I did it all manually.
And paste it into that Markdown document. That's all I did.
So, This name is the announcement.
Okay, so basically your doc strings were the prompts from the website of those meeting.
Okay.
Yes, that's right. So I took those and I put them into this markdown. I just copy and paste it I'm right into here so the base information is under this base presentation header and then the main information is under the main presentation header and I just copy and paste it from what the website look like and I put a, I put a little bit of pros around it, right?
Like the website doesn't say. This exact, sentence like this, but I'm given the LM a little helping hand here.
And those are the abstracts generally that we got from the presenters. Some times I word Smith in a little bit, sometimes Gary, wordsmithed in a little bit.
But mostly it's information from the presenter.
Right. So it shouldn't be. Much or any work from the volunteers to copy and paste it over like if we wanna start doing this more regularly.
But yeah, totally agree. And, and I just wanna be super clear, right? So, so the technique that I'm using here is exactly this.
Yeah. Let me see.
Okay, so let's say I have the system prompt here. Let's say I copy this.
And I go to chat GPT and I paste it in. Okay, and then I hit, and then I make a space.
And then I go here and I take this. This, marked down with all these abstracts and I go here to chat GPT and I paste it in.
And now I type in my question.
Okay, doing this with chat GPT is like the same as what I'm doing in the code, right?
This is literally low tech basically dummy, you know Really low tech. That's why I'm just calling it like prompt engineering or Augmented generation or something.
You know, it's very low tech.
Right. A question. You were you were saying that the chat of the TPT is good at role playing.
And apparently, in that role playing as a marketing guy, it, it knows how to, and you gave it a prompt of.
Give it a tweet or and some other things like that. That's built into the the chat.
Make it.
Yeah, I don't understand it, but yeah, if you tell it like, hey, you're this, and you'll see that all over the web if you Google like how to write a good prompt.
It'll tell you to do that.
Is there some kind of API or whatever it says that? That gives you these, praises to use to talk to the GPT.
There starting to like do some of those like. Like, GitHub for prompts and stuff like that.
I'm sure there's tons of those out there right now. I don't, I just been kind of making it all up as I go and try and trial and error.
Yeah.
Did I see a hand up?
Thank you.
You're a little quiet.
Yeah, my I am Okay. Just check out. So, so I was just, you know, doing all that annual labor money.
You just feed it. Same, It is.
Can you move closer to the microphone, please?
Right. And we're gonna, I think we're gonna get to what you're talking about and it's yeah.
I'm asking you about, instead of feeling all that information, You, You should.
Yeah, there, and we'll get there, but,
Could you're basically saying, can we just get it to read in the HTML file from the old website?
And the answer is yes. And the answer is Even more, yes, if we slightly tweak the website.
A little bit. Oh, I'm sorry. That's not what you're saying. You're very quiet.
I don't know if you got your mic. Sounds like you have your mic like up on your head or something.
Yeah.
You don't know a species where's his mouth.
How's this, Mike?
There you go, that's a little better.
So the same information is in the announced archive.
Oh, you know, it's okay. Okay, I didn't know that.
So instead of, you know, doing all that manually, you can just feed it the whole announced archive.
And have to look it up itself.
No. Where? I don't believe.
Okay, gotcha. Yeah.
It doesn't believe that. All the all the announced emails are in the archive. Reach each meeting.
Why are you thinking that doesn't work, Stan?
Stan, why, why are you?
I don't, I don't, see that it's in the.
Not as I understood the question. No.
Well, the thought was each announced email has both the topic and the.
Description for
Yeah, it it has that but it doesn't have the information. That's, that's on the web page, the HTML.
Okay, I get you.
Oh, anyway, let's keep it. Let's keep going here. So, okay. So the last thing before we, Got off those good tangents was we asked it to generate some YouTube titles.
Once again, we're feeding it the prompt. We're feeding at the markdown with the abstract and all that.
It's your, Emacs navigation experience.
Exploring Speed Bar, a powerful Emacs file explorer. Mastering get a gentle introduction for beginners.
Get basics. Version control with get from scratch. Open systems tools Speed bar and get February 2021 slew meetings
The the general problem I have with that kind of thing is that I think that anything beyond 5 words gets away from what the actual technology is and it's hard to find.
If you're interested in any particular technologies, some of these for tiles. Sound like the marketing title that you get, you know?
Like that number 5. You know.
It's just for me, it just seems too long to find what you're looking for. If you if you come to our site and you look at our presentation.
Archives listings or even on the web pages they put up if Sometimes I'll list what the presenter gives as a title as a subtitle and try to boil it down the technology down into just 5 words or less above that is the main title.
Because we've had some titles in the past that. You look at and you know, 5 years later, what the hell is that?
It doesn't really point out that that particular technology. I mean, it's a nice catchy praise.
You know, it talks about the difficulty. Or, or the goodness or the flavor. The yumminess of of some New slick thing.
But. I think it some things get after time you kind of lose. What it's talking about.
You know, shorter ones.
Yeah, that's better. I mean, clearly you can change it. You just, this is we're still at the base training stage at this point, right?
Right, right, okay.
Yeah, so you can keep giving it more prompts. It's, we're at the stage we're still training the model.
Once we get past it, then we can interact with it more.
Okay.
Yeah, we're not actually, we're not, I don't know. I wouldn't, I definitely would not call this training, right?
Like I said, we're literally just like copying, pasting stuff, extra stuff into the chat prompt down here.
Right.
I wouldn't call that. There is a thing called fine-tuning and that is where you basically retrain the model, but you include, some additional data, but that's very expensive, right?
Right.
Because you have to retrain, the model. Whereas this is just Literally copy and pasting a bit of extra information into the chat prompt, you know, so not definitely not training, but yeah, so Stan I just I just modified this prompt to say use less than 5 words in each title and you know these are much shorter now I think it pretty much stuck to that rule.
Right, right.
So, you know, it, it is possible.
Well, I can't count.
Yeah, so the, the, every, these instances of you putting an additional layer each one of these promises a layer that you're putting on after the after the model.
The model's already been trained, all the weights have been said. So you're just using this as what we were calling a DAG at this point to give it another present.
Present layer, right? A current layer for it to be.
And our time in our context.
Cool. What, what are you calling to act? Was, tag?
Well, no, no, I'm just saying everything we're doing at this point. I'm not entirely sure exactly what DAG is in this convent right now, but.
What is the scope of DAG?
I don't know, I don't know, I don't know, that is. Oh, rag.
Okay, no, right. Oh, brag, sorry.
Oh, okay. Right. Yeah, no, we're not. We're not doing that yet.
Okay, cool.
This is not rack. We're, we're doing rag. So rag stands for retrieval augmented generation and we're doing All of the parts except the retrieval.
We're not doing any retrieval right now. We're just, and so I just called it augmented generation, because we're doing the same thing without the retrieval part.
But that's just something I made up. So don't go expecting other people to know that crazy term that I just made up.
Okay. Anything else before we?
I'm thinking that what you're doing so far helped us understand the process. Of what we need to throw at it.
Yeah. Yeah, this is this is as good as it gets. I'm gonna show you rag later and it's gonna be worse.
Yeah.
It's gonna be much much worse. Okay. So, so this is it. But I mean, I think that this has proven.
That not only, our LMs useful in general, but specifically for our lug.
We can use LLMs, local LMs, openly available LLMs and open source software to to benefit our luck, right?
To help us promote our content, get our, get our words out there, get our name out there, get our message out there.
And what we need is some volunteers, right? I consider what I'm showing you right now to be just, you know, a proof of concept, right?
But if we could get a couple of volunteers, you know, somebody who wants to code, somebody likes to code, somebody who just wants to hang out.
We could turn this proof of concept into some kind of a realized workflow, right? Like I'm thinking about like just a real basic MVP at first, right?
So we could basically get the abstract, you know, maybe even just get it from the announce like you guys are saying.
And then we could just have a little script that generates these tweets and maybe we just.
Send the generated tweets out to one of our mailing lists. And then anybody who, you know, wants to can just copy and paste those tweets into their Twitter, into their mastodon, you know, and just we'll have ready-made tweets for anybody who wants to send it out can send it out, you know, and then whoever is running, you know, if anybody wants to volunteer to like do
the actual Twitter like of the slew Twitter, you know, they have ready made tweets there that they can just copy and paste.
So instead of them trying to draft a tweet and come up with something, they just need to read, you know, read through 4 or 5 of them and pick the best one or pick the best one or 2.
You know what I mean?
I find it amazing that this thing can generate. These hashtags and that are acceptable or we would be useful in with the various things like mastodon and and twitter and so forth I mean, not being familiar with those environments.
This all that that structure is is all unfamiliar to me. And. Totally.
Yeah, and yeah, those were good tweets, you know, those were way better than anything I was coming up with when I was doing it manually. You know what I mean?
And, And you know, so anyway, this is my call to actually get some volunteers if we can.
Of course, anything we come up with, we're gonna make open source, right? So we're kinda all about that.
And you know, this is something we can continue to enhance as we go. Like once we get the MVP working, you know, the sky's the limit for how far we could take something like this.
So, so I just want to do a little call for action here in the middle. You know, if anybody, at all, is interested in playing around this AI stuff, please just send an email to, as interested in playing around with this AI stuff.
Please just send an email to steer comedian with AI stuff, please just send an email to steercom sloop.org or message me directly or message me directly or message me directly or anything like that and we can we can get it we can get it going.
And while people are thinking about that, I want to, talk about some other ideas.
That I had. So, you know.
Hey, Sean, I have some questions. About your setup. The LLM that you just.
Instead, It is only running within that Jupyter Notebook, correct?
It's running. So.
In the sense that if you shut it down, it goes away or does it saving state?
Okay. So it is not saving state with this. So what I'm doing.
Is I'm doing.
Llm dot invoke. Sorry, I scrolled past it. Right here.
I'm just doing. Invoke from LANG chain and this is a brand new, invocation each time.
It's not remembering anything. That's why. Pass in the system prompt and the doc string every time.
Nice.
There are different chat or prompt templates that I'm not gonna go into but you know you could look it up on the Lang Chain website.
There's something called a chat prompt template. And if you use that, it actually will remember.
From one prompt to the next.
I see, I see. Okay. So it's like kind of in memory at this point. So the moment.
E, you turn off it, you lose that state of your 2. Start from the beginning.
Yeah, once it processes the prompt. It's gone.
It's good. Okay, okay. So that's why you keep sending the problems. Okay, good.
Thank you.
Okay. So I'm gonna go over like 2 more ideas that I have for ways that we could use an LLM to benefit the log and I don't really have proof of concepts for these other 2 like I did this first one.
And then I'm gonna, I need to go get some water and I'm gonna ask for anybody else to tell me any ideas they might have or any just comments questions or concerns they might have with any of the stuff.
So. One thing that I thought. Was we could actually Get the LLM to generate design ideas.
For the YouTube thumbnail. Okay, and then once the LLM generates the design ideas.
We could use that as a prompt to stable diffusion, okay? Stable diffusion. Is another LM that you can run locally on your machine.
Oh, I don't like how long this is taken. Stable diffusion is another LM that you can run locally on your machine, that will generate images from prop, from text prompts.
So basically, I asked this L, like what are some design ideas for a YouTube thumbnail?
It gave me back some ideas. We could pass each one of these ideas into a stable diffusion prompt.
The stable diffusion will generate an image and then we could have a human pick which image they like the best and use that as the YouTube thumbnail.
And so that could be cool, you know, or another idea I had. Was, we could actually like take the last like 3 months or whatever worth of emails on the discuss mailing list.
And we could feed those into the LLM and we could ask the LLM, you know, hey, given our members have been talking about these things.
What would be some good? Future presentation ideas. And, and we would get presentation ideas from the LM that are customized to exactly our people, right?
Because if you look on our website We have this section on our website for possible future topics. But I don't you know it doesn't have anything but if we could have LLM analyze our discuss emails and then suggest topics relevant to what we've been talking about on discuss, that'd be pretty cool, right?
So, you know, that's just another idea I have for how we could probably use LLMs to benefit, the lug.
I need to go get some water. Does anybody else have any ideas for something we could do?
John, Mr. Stan, would we dare pump in the planning schedule report? Data. Is it's got a Oh, topics that.
In the past people have asked about.
I'm wondering if we, wouldn't wanna build a state full. Version where you just pumped in the entire Discuss archive and start asking the questions.
In other words, that way. You know, you're not looking at just a particular presentation. You're looking at an entire discussion history.
Which will give it a lot more context.
Of course, discuss emails or more. Impromptu, more Lucy Goosey.
Could be more pro to error.
So this is something you're gonna put on the website for people to like pose questions to it or Okay.
I don't know about on the website, but we'd certainly make it available to.
You know, to share common. You know, cause on the website, if your robot runs across, it'll, Wipe us out as far as traffic.
I see it all the time on some of the websites I manage. Web servers I manage I should say.
Sean, Burnab, just, posted a question in chat, or comment.
If I want to read it yourself or I could, questions basically how does One determine the minimum system requirements to run.
LMs locally. I went to a llama website and there is no
Yeah, so I, I don't know. The the bottom line up front answers, I don't know.
But what I think is that you can run these. On pretty much any like laptop or desktop.
But it's gonna be slow, really slow. Like you're gonna ask it something and you're gonna get something back 2 min later, something like that.
You really need a GPU or an Apple, M. Something, you know, M 1, 2 or 3.
Or an Nvidia GPU or actually hot off the presses as of last week. These things actually support some AMD GPUs.
I don't know which ones, but some. So yeah, you definitely, if you're, if this is something you're going to play with a lot.
You certainly want a GPU. If it's something you're gonna play with and then you probably just can use chat GPT.
Hmm.
You know then maybe you can get by with just having a CPU but I think You get the same answer, I think.
No matter what hardware you're running on, I think it's just generates that answer much more slowly.
That's my hunch. I don't know that with any certainty.
The more RAM the better.
Alright.
The more, you need specifically video RAM on your video card. Yeah.
I don't know how many people build their own machines, but if you take a look at a lot of the Newer chips that are coming out many of them have multiple GPUs built into them already.
So if you have a new chip like say, I think, 5,600 chips got like 4, I believe it's 4 GPUs and 2.
Main, our standard.
CPUs.
So having the the hardware available is really not that out of line.
The place you need really a lot of compute power. Very, very high compute power is when you're actually training the models.
We're just using them. It doesn't take a lot of compute power.
Right, I agree with that totally. What I'm running here is I have like a this desktop that I'm on right now is 2016 but it does have an RTX 2090.
Yeah, no, 2070. In video 2070 from a with all the rest of the parts of this computer from 2016 so and you see how fast it is on mine but that's because it's only using GPU mostly so Okay.
Yeah, so what you're seeing is there if you got a machine that's Less than 4 5 years old You might have enough GPUs already, so I don't think that's a big problem.
Yeah, well, it's okay. So installing, I didn't go over in this talk, but installing OLAMA is very straightforward.
And, And, but if you have trouble with it, we have that new lug, you know, so.
Feel free to stop by the new log if you have trouble getting it installed, but it is very straightforward.
I've done it on Linux and Windows, very straightforward, and it's free and it's, you know, doesn't take a whole lot of time.
So certainly just try it and see how it goes for you. You know, there's Very little reason not to.
IS next Tuesday
I have I do have a newer desktop but it's upstairs. My gaming machine.
Okay. Alright, so, the rest of this talk is gonna be about retrieval, augmented generation.
And, it's, I'm just gonna give you a spoiler alert. I don't think it's gonna work as well as we all hope.
And it's certainly the best that retrieval augmented generation can do is. Match what I've shown you so far.
Okay.
Okay, that's the best case scenario. Of what it can do. Is match what we've been showing so far.
And it's gonna get complicated and there's gonna be all sorts of edge cases and I'm not going to explain it super well okay but I don't want if you so if you don't understand this next part that's my fault not your fault okay I'm not explaining it well Also, if you don't understand this next part, that doesn't mean you can't volunteer to help us with any
of the stuff if you want to build something cool, you still totally can without doing this rag. I've shown you that even just using augmented generation with no retrieval, we get really cool, really good results and actually it's going to be the best way for us here, Now, I'm going to show you rag because it's all the hotness right now and it's in
the title of the talk so I feel like I have to. But it's not gonna work as well So let's get started.
Okay, so. Oh, I see messages in chat.
So, RAG is the solution to a problem. Sorry, Rich, we're gonna say something.
Okay, let me keep going. So, rag is a solution to a problem. And so before I show you the solution, I want to show you what the problem is.
Okay, so I'm going to move to this next notebook that I have here. And, this is going to be very similar to what we just saw.
Okay. So first I'm gonna load up a model on LM. And then I'm gonna read in.
The system prompt just like we did before, no change. Now here's where the differences, okay, instead of loading one file that has a base and a main presentation, I'm gonna load up 6 months.
Worth of base and main presentations, okay? So. Excuse me if we go through we have February here We have arch.
Go ahead.
Yeah. Yeah.
Sorry, I thought I heard something. We have April.
We have May. We have June. And we have July and I have the you know, Paul D straight out of the website archive just like I did the others so you know this one in July was about you know, very own stand, right cart.
Data entry form in Calc and then we had window barber talk about pre execution boot environment or something like that.
So these are all legit here straight from the website archive and so I've loaded I'm gonna load up 6 months worth of talks.
And I'm printing it out I guess. So it's gonna scroll and scroll.
Actually it's truncated. But it's 10,000, the length is 10,258.
So I've loaded up 6 months worth of talks. And I'm gonna pass the system prompt and the 6 months worth of talks.
Into the LLM, just like I did before, I'm going to ask it the same question.
What was the base presentation on February, tenth, 2,021? And it's gonna get it wrong.
Well, likely, right? LMs are very inconsistent. And non-deterministic so you never know what you're gonna get from one to the next but yeah it got it wrong so I asked it what was the base presentation and it's saying a gentle introduction to get which is the main presentation okay so it got it wrong.
And the reason why I got it wrong is because we've loaded too much data. We flooded the context window that the LLM can handle, okay?
We have overloaded it. By sending in 6 months worth of talks.
And so.
Is, is that overload based on? Running up against the limitation of your hardware, your RAM.
No, this is a hard limit with this model. The Mistral model and all of the models that you can run on locally, they all have this limit and it is a hard limit.
As far as I know, there's no way around this limit, you know, like we're at the very edges of you know, of large language models here.
We're at the very edge of what, you know, we, how we know to deal with this, you know, as an industry.
As far as what I'm concerned. So here I'm gonna ask.
How many kilobytes or megabytes of text did you give it?
So not much. Okay, but it doesn't go based on that as well. Okay.
Yeah.
I'm speaking out of school here. It's based on tokens. Okay, it's how many tokens.
Now what is the token? I don't know. Is a token a word or is it a character?
I don't know. Is it 4 Byte? I don't know. But it's based on tokens and I, you know, and the limit I think is different for every different model.
And I think the limit on this one is like 4,000 or 8,000 tokens. But I could be wrong.
I don't know. But I know from trial and error that passing it in 10,000 characters will overload it.
And I don't know why this is taking so long. I was not expecting this.
What's your GPU look like?
Well, only 80.
So again, let's give it another minute, see if it finishes. I might have to.
Cancel it.
What do we do if it becomes self-aware?
Okay.
Yeah. Br and
Terminated.
Okay.
If it can become self-aware in 8,000 tokens. I would like to know the meaning of life for sure.
Yeah.
Okay.
That's what the automatic power switch is for.
Right.
Let the smoke out of the machine.
Yeah.
Oh!
I've always got this.
Yeah.
I thought there was supposed to be like a stop button somewhere up here. Oh, it finished. Wow.
Wow.
Okay, that's a 2 min. Holy crowd. Okay, so, so once again, I asked it to generate short and enthusiastic tweets about the base presentation on February tenth, 2021.
And it's gonna give us all kinds of stuff, right? So here is talking about stands, entry form, which is the July talk.
Here it's talking about. Some. Some person I've never heard of before.
Giving a talk about bash functions or something like that. I think that was in June of 2021.
It's talking about this other thing in July. Like it's just completely confused, right? It just can't handle this much context.
Okay. It can't handle this much data.
So anyway, I think this illustrates the problem. That I had generated a lot of these. How to do?
Well, let's look. This is why it took 2 min because it just went ham. It's like, oh, y'all want some tweets?
Oh my.
Like how many? Holy cow. Alright, anyway.
Well, you can put some links in here that probably don't exist.
Okay. Anyway, it went totally.
It's the, Shawn.
What's that?
I don't think it was a relevant link if it if it was too Etsy.
Okay. Right, so this is the problem, right? We flooded the context window. We gave it too much data.
It can't handle it. It's getting confused. It's hallucinating. It's doing whatever.
And Billy asked would narrowing down the question. Help the situation. Yeah.
So this is. Pretty much what we're gonna do. So what I did was I asked the LM about the February tenth, 2021 meeting.
But then I gave it a bunch of data about, you know, not only February, but like March, April, May, June, July, right?
I gave it a bunch of data that doesn't matter at all. To the question not relevant at all to the question that I'm asking, right?
And that is the problem. So if this context window is a hard constraint, then the only thing we can do is work around it, right?
And so what rag does, retrieval augmented generation, is it tries to, break?
The text into different chunks. And then it tries to retrieve the chunks that are relevant to the question.
And it'll send only the relevant chunks. Into the prompt. And if you're only sending the relevant chunks, that should fit within the context window, okay?
And so that's all rag is. Is basically doing what we were just doing manually, but doing it much more complicated and much more, harder, right?
And so when I read those 4 steps, you probably had some thoughts going in your mind. It's like, wait, how do I break it into chunks?
Like, do I do a certain number of characters before I break it into a chunk. Do I break every sentence into its own chunk?
You know, do I do it based on, you know, A marked down header, you know, spoiler alert.
Do I do it based on a markdown header? You know? And then once they're broken down, it says retrieve the chunks that are relevant to the question.
Well, how do I know what's relevant? How do I figure out what this chunk means?
You know, means and then once I figure out what it means, how do I figure out what the question is asking for and then how do I tell if those those 2 things are relevant.
You know, like that's a hard problem. And so is the problem of the best way to chunk up this text.
And the answer to all those questions is of course it depends, right? It depends on the question, the prompt you're asking.
It depends on your source data. It depends on the model that you're using. It just depends on a lot of stuff.
And there's no right answer. And what you're gonna see when I do my demo is that, you know, I've clearly chosen the wrong.
The wrong, thing to answer these questions because it's not gonna work great, but there's a lot of techniques here and this is a very deep area and it's a very useful tool. Okay.
I don't wanna make it sound like It's completely worthless. Okay, I think it is a very useful tool and I think the people that know how to really do it well are gonna get really good results okay but it is it's complicated there's a lot of knobs that you need to tweak and get just right to get good results and stuff like that.
But this is really what it boils down to is I can only fit so much stuff in the prompt so I need to make sure the stuff that I'm putting is relevant to the question and not just garbage.
Okay, let's give it a shot. Let's give it a try.
This one Let's give it a try. So here I am loading up the model and you could probably tell by the number of import statements that we're gonna do some more coding now.
Okay. Okay, so now I'm gonna load up the system prompt just like we did before, but this time I'm gonna use one of Lang Chain's prompt templates, okay?
And this prompt template here doesn't save the state from call to call, I don't think.
I think it just is basically like what we were doing before with the invoke. So I'm gonna go ahead and run this that'll load in the system prompt into the prompt template.
And if you remember before, I was basically just pasting in the markdown file after the system prompt and then I was putting the question at the very end.
And that's all this is going to do, right? So I'm doing this prompt template.
And I'm gonna pass in to the prompt template. Context and question and it's just gonna place it in where these placeholders are.
So this is very similar. Frankly, the exact same thing is what we were doing.
Alright, now here I'm gonna use 6. Here's our 6 months of talks. I'm gonna load those in.
And now here's the new thing. Okay, here's the really new thing. One of the new things, I need to split.
The I know I need to break these 6 months of talks into chunks. Okay and I have structured these markdown files by the presentation.
Okay, so I have a heading here for the base presentation. I have a heading here for the main presentation.
I have a heading here for the main presentation. And so we could chunk these on these headings and so we could chunk these on these headings and that's exactly what I want to do.
So.
So looking back here, I'm using something from Lang Chain again called the Markdown Header Text Splitter.
And I'm passing in this list of headers to split on. So the top level header, I'm calling the meeting type or meeting title.
And the second level header, I'm calling the presentation type, okay? And I'm gonna use these headers to split that into chunks.
And I'm gonna print the chunks out. So we'll see them right now. Okay, so these are all the different chunks and I have them kind of a space separated here.
I have a space between them. So we can see that we have a chunk. For the base on February.
We have a chunk for the main on February. We have a chunk for the base. In March, we have a chunk for the main.
In March. Okay, so I've split each presentation up.
I've chunked it out based on the talk based on the the month and the presentation okay and if we look here we also have some metadata And the metadata says.
It has one field called meeting title and this is where it takes the h 1 you know the very top of the document it puts that in the meeting title metadata and then it also has a presentation type metadata and here's where it puts the second level H 2 heading in there.
So for the base presentations it'll be base presentation type and for the main presentations it'll be main presentation type.
And that's really all this is doing is chunking it up into the content and with a little bit of metadata and I'm chunking it up by presentation.
And there's a lot of them. And actually this output is truncated, right? But they'd be 12, right?
There'd be 12 of them. If we saw the full output. Okay, so we've done step one.
We've chunked up our document. Now we just need to do the small little simple thing of figuring out what those chunks mean.
Alright.
No big deal, right? No problem. So, so the way that I'm gonna try to do that is by using this thing called an embedding model.
Okay, and this is another AI model. Is not like Llama or chat GPT or any of these.
It's older than that, right? And this one I'm gonna use is called Nomic Embed Text and this is based on Bert, which I'm not gonna go in. Don't ask me.
I don't know. But it's based on BERT and this is what I'm going to use to try to figure out what these chunks mean.
Okay and it's going to create embeddings. And then I'm going to pass those embeddings along with all of the chunks.
To a vector store. Okay, and the vector store that I'm using here is called FAIS s, which stands for Facebook, AI Similarity Search.
So this is a library, a Python library, I think, built. By Facebook, to do this kind of thing.
So The way that you can think of a vector store is kind of like this. Okay, a vector store is a multi-dimensional space.
And Each embedding is. Several points. Within that multi-dimensional space. And you can see in this kind of example here that I just kind of grabbed off the web.
You see apple and banana are kind of grouped over here close together on this plane because they're both fruit, right?
They have similarity. They have semantic similarity, so they're grouped close together. Wolf, dog and cat also have semantic similarity.
So they're grouped close together on this other plane over here. Okay, and so So when we take a chunk of text and we create embeddings.
We are creating several, you know, hundreds of dimensions. Plots on it points on a grid and then we're and then we're doing the same thing with the prompt right we're taking the prompt and we're figuring out its coordinates on a grid and we're saying, okay, if the prompts here, what is next to it?
What is the thing that is closest around next to it on the grid? Oh, those things must mean the same thing semantically if they're close to each other.
In the vector store. Okay, and if that sounds confusing, it's because it is and I'm not going to be able to explain it much better than that.
Okay. And so here we're taking the vector store and we're turning it into a retriever.
And the retriever is what we're going to call to do the actual searching, right? So we're going to say, hey, retriever, search the vector store and find me things that are relevant or similar to my prompt, okay?
And we're going to use a similarity search type. And we're gonna say give me a maximum of 3 give me give me 3 chunks I only want 3 chunks
That's what this one says. And last thing is we just create the chain. Okay, oh, I've got to run this.
Let me run this. I run this thing, yeah. Okay. So this is gonna take, you know, at least a minute or 30 s here to generate these embeddings and get the semantic meaning of all of all this stuff.
If anybody has a question, I'll try to answer it, but this is this stuff is going way out of my depth really quick.
Quick clarification that prompt at the second variable. Is that the that the big text? Right above the cell.
So.
The last chain equals create stuff documents chain LLM comma prompt.
Yeah, so the LM is declared at the very top. Right here.
The prop is right here.
Correct. And then the prompt is the one okay, the prompt is the prompt template.
Yup, yup.
Okay, okay. Thanks.
Okay, and then this create stuff documents. This is a link chain thing
It's d1. 0, it's a 25 s. Okay. So now we should have our vector store loaded up with the meaning of those chunks.
So hopefully we have, you know, stuff like this.
Okay, so now I'm gonna ask this thing. So I'm gonna ask our retriever, right?
I'm gonna ask the vector store, Hey, give me things that are near this prompt. And the prompt is the same one we've been asking.
What was the base presentation given to the St. Louis Unix users group on February, 1020, 21.
And I want to see what this retriever brings back because I think it's going to be wrong.
So I'm just this code right here is just only printing out what it's retrieved.
And then down here, I'm asking the LLM. I'm asking the chain, okay.
To invoke the prompt and I'm passing in what the retriever brought back as the context, okay?
And so this before this was me. Giving the markdown file. Now we have this retriever, you know, trying to find the right markdown file and putting it in here for us.
Okay. And then the question is just, you know, the prompt, the actual prompt. Okay, so let's run this.
And I don't think it's gonna work. And I think I have a hunch as to why.
Okay, so here are the chunks that it retrieved. One is from April. One is from June and one is from July.
Okay. So it did not retrieve. The chunk from February like I asked it to okay and so we're giving the LLM the wrong book on this open book test okay so it's not going to be able to answer See, it says I'm sorry for any confusion, but there seems to be a mistake in the provided context.
As there is no mention of a base presentation, giving to the St. Louis Unix users group on February, 1020, 21.
The information provided only mentions the main presentations on April, June, and July. Okay, so the LLM can't answer because we gave it the wrong book.
And the reason why I think we didn't retrieve the right stuff is because my prompt is asking.
For February, tenth, 21. And I think when it was trying to figure out what does this chunk mean, I don't think that it was.
Emphasizing the date of the meeting as important, okay, because here's why I think that, okay, so if I go down here.
And you know, we know the base presentation on that day was about Emacs speed bar. Okay.
From our, what we did earlier. So if I asked the LLM. When was a presentation about E back speed bar given and by who it's going to be able to answer this one, okay, because I think when it was trying to semantically figure out what is important about this chunk, it said, oh, speed bar, the title, that's important.
The date not as important, but the title is important. Right, so here I asked it about speed bar and it pulled back the correct chunk right away first try.
Yeah.
And for the answer, it says a presentation about Emacs Speedbar was given on February, the tenth, 2, 21 by Steve Stegman.
Okay, and then here I'm gonna ask a last one a last question about What presentations were related to source code versioning.
Okay, I'm not mentioning yet. I'm just saying source code versioning. Just see if it can figure that out.
And I think it will be able to. Yeah.
IT was great.
Okay. Sean, do you think this thing does not know the difference between?
Okay.
21 day show 2 and February. 2021.
I tried that. I I had the same thought as you did and so I tried it. I spelled it out everywhere.
Yeah.
It didn't matter. I just think it's doesn't think the data is as important.
So it throws back the metadata to you, but it doesn't. Know how to use the metadata.
Yeah, it's just not not quite linking it up, right? So. So here I'm asking about source code versioning on this one and there's actually 2 talks about get, in this data set.
The first one is on, or one of them is on July ninth titled, peering behind the curtain and then for some reason it returns this one about the pre execution environment.
I don't understand that. But then down here for the third one, it does find the gentle introduction to get and if we ask the and the LLM is smart enough to know to throw this one away.
So the actual, you know, LLM says the presentations given by Robert Seitech on February tenth called a gentle introduction to get.
And June ninth titled get peering behind the curtain were related to source code versioning as good as a popular source code versioning system.
So it got that 100%. Right. And so. Yeah, that's really it.
For this. But, there are different search types like there's another search type called MMR and don't ask me what actually I know I'm not gonna say let me get it up here
Okay, I don't have it up anymore, but there's 1 called MMR And if we use this one, it's like maximal most.
Relevant or I don't know MMR and if we use this one it usually gets this date question right somehow.
So there's 2 different search types, similarity and MMR. And let's see.
So I'm, I switched it over to MMR. I'm gonna ask it this question about February tenth again and let's see what it retrieves.
Okay, so it retrieved April, June, and then it did retrieve the February one as the third one.
And so I think it'll get the answer right.
Yeah, it's in. It got the answer correct. So, but it'll probably get these other 2, Rob.
Let's see.
Okay.
It's like it's reading the entire string and then it then when it gets to the right answer that then stops reading.
This, is what it looks like when a llama crashes. So let me reload it real quick.
I knew it was gonna happen at least once, right? I'm glad I was at the very end.
Cause I'm not after this is Pretty much all I have prepared. So we can keep playing. I'm happy to keep playing around with this if we want to.
Keep asking more questions, keep filling around, but this is all I had. Prepared at this time so let me go back to
Slides real quick just to wrap it up. So, okay. If you so this is obviously hobby grade I've said the whole time I don't know what I'm doing if you want to see enterprise grade this is where you go right here I want to put this in the chat.
You know, this is enterprise grade rag. It uses, hybrid rag, which is a mixture of the semantic search along with the keyword search define the right chunks.
It uses a hybrid of both of those. It uses, you know, at least half a dozen.
Azure cloud services. That's how you know it's enterprise grade, you know, it's enterprise grade, you know, and so if you want to go deep, that's where you go. You know, and so if you want to go deep, that's where you go.
This resource right here. And then, yeah, if anybody wants to play with this stuff and help out the log a little bit, there's plenty of opportunities here.
I can, I attach it to the chat. I can I'll put up on GitHub.
That's fine.
Well.
And this presentation will be up on our archive. In a day or 2. Recording.
And, Stan, will we also put the, Slide deck up there.
Is there a slide deck?
President. Yes, that's right. You just got a notebook.
Yeah.
Now.
I have a question. So I'm wondering if you take the date or portion out of your search.
And just left in the month and year. And see if it would find it. Because the commas are a delimiter.
And, I don't know if it sees it as a delimiter.
I'm just wondering if you just left in the month and year?
Yes, let's try that. That's a good point. Let me change this to what was again similarity.
Yeah.
Let me change this back real quick. Let's definitely try that. I wanna try that for sure.
I'm gonna change this back to similarity because it'll get it. That's what it needs to be to get it wrong.
And then we'll, you saying we're gonna take this out and we're just gonna do February, 2021.
And we'll try that. Oh, I need to do it down here too.
There we go. We will give that a shot.
Okay, it finished so now I'm gonna run this one and let's see what it retrieves April, June, and July.
Nope.
Oh, I don't think it that,'t seem to do it. I guess we could try to figure out when would it eventually do February.
Let's change this to 12. Because there's only 12 there should only be 12 chunks So let's just see, this is probably gonna crash, but that's okay.
We'll, see the chunks.
Oh, I didn't run that. Did I? Oops, okay, so I changed that to 12, but I didn't run it.
So I need, I need to run this again. And then we'll do it.
Sean, another question while that's running. In your presentation, did you cover what was these, It, it was it O'lama was the only thing you installed on your system before this I'm assuming you already have like visual studio code and all of that but and and Python and all of that but was it olama only over the other libraries?
Yes.
Okay, yes, so there's a lot of libraries here. Let me this is what I'm gonna put on
On sorry What I'm gonna put on GitHub. Hi project. So I don't know.
I'm not gonna Python. Okay. I don't know anything about Python.
But I heard right is like the big hotness. And so I used dry. And so I have all of these dependencies in my Python project.
Right here.
I would.
Oh, and then this is also you just push this to your GitHub, right? So I can pick it.
I'm gonna push this to GitHub. Yes.
Okay, awesome. Awesome. Thank you.
Let's see.
Good.
Sorry, is there any other questions while I push this to GitHub?
I'm just trying to do this now so I can give people the link.
So it seems from whatever you said so far, the engineering aspect is. Is is coming up with your like data ingestion.
Right, like you chose to create those. Marked down headers and you chose to. Determine what is important, what is not and that's why you're saying is that there is a level of.
Try, see, and then, you know, update based on how the models behaves, right?
So you can. Pick more mark down headers if you want. Like for example even split out the dates if that's a sticking point or.
Even split down the individual. like the abstracts further if necessary. Right.
Yeah.
Yep, okay, interesting. Yeah, because the thing that I've been having having a hard time.
To kind of get my head around is that there is so much conversation around these 2 topics.
That it was always hard to understand where the where the challenge
Existed Right. Because you always, you always see chat GPT or Gemini or whatever, right?
As the example, but there are companies trying to build these things. And there's always talk about like the complexity behind it like it was not until this presentation that the Complex it not not apart from you coming up with all the libraries.
It was just the way you chunk your data Which was very interesting I think that made the big difference.
Yeah, I, I agree too. And if we look at the link chain. Suitters.
See, there's HTML, which adds metadata and then there's marked down, which adds metadata.
And I don't know how big of a difference this. Metadata makes, but I figured I'd try it out.
Was the HTML one like if on our website we put like an ID on the element.
Hmm.
Bye, I can find wherever the, yeah, here. So like on this H 2, if we put like an ID that said like, you know, base title.
We put like base abstract, you know, then we could easily extract this from the website the HTML similar to how I'm doing the markdown headers.
You know, since these are all from 2021, it'd be. And the reason why I did 2021 is because those are.
The YouTube videos that we have up right now. And so I was thinking about that.
Interesting. Thank you. This was really this was This is quite amazing to watch.
Yeah, it's limited, right? Like you think like, when Lee was saying, oh yeah, let's just throw our entire, Let's just throw our entire, you know, email the whole emails at it, you know, it's like, You know, I wish it was, but man.
Not quite.
It almost seems Like, we're expecting there to be. New fangled very complex and tricky things that need to be considered.
And it seems like with problems actually are are very old-fashioned simple things like how big the dang variable field is.
How much data can we cram in and one bite or one chunk?
Yeah, and then figuring out like what does this chunk mean and how do I compare it to the prompt, you know, to know which ones are relevant to send in or not.
Yeah.
That's the really, that's the other really hard part. And but you know, I don't think it's gonna take innovation to, move this forward, like from a industry perspective, I think it's just gonna take iteration at this point, right?
We just need to iterate on the way that we compare our chunks to our prompts. We don't need like new paradigm.
We just need to get slowly better at it, right? Cause I think what would be great is if we could write JSON, right?
And then or structured text like that right now these LLMs work very well with pros And I think that's because.
The semantic searching stuff works a lot better with pros and with kind of than it does with structured data.
But I think if we could get like a structured. If we could get like a vector store type of thing that could work on like an entire JSON object or something, that would be really cool.
And I might already exist and I just don't know about it.
Did you do data conditioning to the unsuccessful data dump you gave it for instance? I'm just reading the chat here.
Yeah, so it looks like what you're talking about there, JM is what I did at the very beginning.
And what we had the success with basically. And what that's doing is doing basically rag without the retrieval, right?
Because you're not like, you're not like, here's a whole pile of data.
Let me try to retrieve what's relevant. You're saying I as a human have picked this chunk.
I know it is relevant. And I'm passing it along with my prompt. And that is like the best way.
But obviously it's the most labor intensive way as well because you need a human to do the manual retrieval.
So yeah.
Alright, well I'm gonna rest my voice and I'm gonna pass it back to So Gary. Alright, so thanks everybody.
Hope you enjoyed.
Thank you very much. I think my brain hurts. Yeah, definitely.
Learning a lot of new things. So yeah, this has been great. Anybody have any particular, insights or things they found fascinating to tonight's, talk?
Yeah, it's hard to just kind of focus on. But anything in particular almost.
Now, I think it just demystified a lot of the jargon. That we see.
Not just generic news but even within tech news. It was really helpful to see this.
Thank you, Prano. That's a good, good description.
I was amazed at the ability of it to. Do the hashtags that would be appropriate for Twitter and You know, take it.
I think that would actually be the most useful right now if we can use generate. Twitter feeds or Facebook posts or whatever.
And Meetup posts.
Yeah.
Because that actually the simplest to use, you feed it one set of data for the meeting. And ask it to generate the marketing information.
As opposed to searching through 20 years of meetings.
Meetup sucks.
Yeah, but Gary likes it. And I think I think I've actually counted more than 10 at this point after 15 years.
Of people for meet up that actually showed up in a meeting. Yeah, no, I think we have had some fresh blood if I can refer to it as that.
Come in from. So.
The problem is that there are 24 day deodorant quits and they never come back.
Yeah.
Okay.
Maybe that's yours, 24 day deodorant. Yeah.
Yeah.
That is true. That is true.
Yeah.
A couple epic failures over the years.
Okay, and I'll try not to make this sound like a hard sales pitch, but Sean was very good about saying that, you know, many, many fingers make the work, light.
We do have a kind of a task force of volunteers who are working on. The upgrade of the operating system on our server, Bach.
But, yeah, same thing. Although Shawn has kind of point at us, point at us in the right direction of things that are possible here.
Obviously, it needs some time and effort, to, to come up with some practical results that would be useful for our potential.
Visitors members And, so yeah, if, I, for one, I'm looking forward to maybe, being part of a little group here to work on this project.
But I know I would certainly learn a lot. From it. So, anybody else who would be, interested.
It's 1 of those things as time allows, you know, I mean, anybody can only commit the time that they They have to spare.
So, but, Yeah, Sean said, we as a group would appreciate anyone's Contribution of efforts.
And I think we'd all learned something. Trying to do something practical. With this information.
So. Hey Sean, said he to contact him. Or send an email to Yeah, sure comes probably the best way we can keep track, you know, see what's going on and everybody can kind of watch the progress.
That's a good idea.
It's still kind of a small group anyway. It's not like you're throwing it out to That's St.
COM at Slug. Got.
That's short.
I would suggest maybe that we close out the recording of the meeting so far and.
Good point. Booking, Gary? Already? This was the St.
Louis Linux User Group meeting for Thursday. 20 first of March, 2024, our speakers tonight with Sean Twe House and he was talking to us about Okay.
Utilizing LLM, large language models. With rag, retrieval augmented generation.
And thank you all for coming. And hope to see you again.
The