Are we recording?
Shit. No.
Not yet, what I understand. Okay, now we get a recording.
Now we got a recording.
Okay.
This is the December thirteenth, 2,023 meeting of the St. Louis Unix users group.
We have a base and a tutorial. And I shall let who's doing the base?
Stan?
Sean is doing the base.
John, that's right. Shawn's doing the base with All kinds of Python magic.
And the meantime, no, Stan has our introductory segment. Stan Closers.
Oh.
Well, you confused me. Anyway, what else is new? This is the St. Louis Unixus group.
We focus on Unix all kinds. Second Wednesday of each month. It's a.
IRS, 501 c 6 designated not for profit professional association. Dedicated education. Among computer users.
We have many Linux, unique professionals, network experts, see the system experts, obvious students, etc.
We are interested in Unix, Unix like operating systems, Linux VST, 3 open source.
Software applications, products, projects, services. We've met continuously since we incorporated in July of 1992.
All of our meetings are free and open to the public. There's no individual membership fee. Hey, agenda is being, since we meet here.
Oh, the time changed again. Oh my. The central daylight time from 6 pm to 6 30 central daylight time.
It's time for testing your chat. Microphone sound, screen sharing and so forth. And then about 6 30.
We start with the basic topic. Any questions? Then from 7 on. The middle introductions and so forth, announcements, call for help, etc.
Calendar, other stuff. And then we go under the main topic. After the meeting further discussion is possible on a mailing list, recorded chat, slide work video.
Build our archives within. Next day or so.
Based topic. Try to do something simple. Thanks, Sean.
I'm here.
That's right. Okay. Go ahead and take over from here.
Sounds good. Let me share my screen.
Oh boy.
Hello, my name is Sean. I'm a software developer. I used to be a system administrator back when that was more of a job and I mostly do software development.
And even nowadays I'm Not even getting to do much of that anymore. Can everybody hear me and see the screen?
Are we good?
Be your screen, yes. She's fine.
I can see, yeah, I can hear you pretty good. To see your screen fine.
Excellent. Yeah, for some reason this in the middle here is supposed to be the Skeena logo.
But it's not showing up, so I don't know, it's weird. But yeah, tonight I'll be giving a short presentation on Skinemo, which lets you record, share, and replay your terminal.
I think a month or so ago, Stan gave a little base presentation on script and this is a very similar thing but I think it's got some nifty capabilities that script might not have and so I thought we could explore together.
So let's just, so this is the,
Just the screenshot from the homepage and let's just go there now. Let's just go to the web and see this.
In action. So this is ASCII, Skinema's homepage. And if you look right here, this is a recording of a terminal made with the schema.
And it looks like a video player, right? I can hit play. And it'll start playing, but this is actually not really a video.
I mean, it is, but I can. Like copy text out of here, right? And copy and paste it.
Into my terminal or wherever I need to copy it from and I think that's so cool. And I really enjoy that and.
I'm not sure if you can tell based on this, but there's like a an element of timing that we don't get with as far as I know you don't get with something like script or script replay.
Or something like that. And so it really puts you in good control over over your presentation.
So let me.
Go to the next slide.
Shawn, could you elaborate on that? What do you mean by the element of timing?
Yeah, I will. I will go into it, much, much more, here.
Yeah.
Okay, so first we gotta get this thing installed. I don't have it installed right now, but we're gonna install it and then we're gonna go through it.
And play around with it. So it has, installation for a bunch of different Linux distros.
It's also installable via PIP. It's also, you can install it with brew if you're on a Mac.
So tons of ways to install this thing. So let's go ahead and do that now. So here I'm on.
WSL and I'm doing. What am I on here? Okay.
Looks like, A bunch of 2204. And a skin was actually in the base repo if I do
And that's, big enough, right? Everybody's good.
Yeah, looks good.
Okay, and in the 2204 repo we have a schema version 2 dot one dot 0.
But if we look at their, docs for the installation. For a We'll see they recommend this PPA.
Yeah.
So I'm gonna go ahead and, install this PPA here. I thought it would.
So I'm gonna add the PPA. Enter to accept. It automatically does an update now, which is a nice feature, so I don't have to do that manually.
And now let's see what version the PPA has. The PPA has 2 dot 2 dot 0.
So it's really only one. Minor version ahead but whatever I guess we'll use it anyway so let's install Ask a, and I looked up the name, their FAQ has like a how to pronounce it.
And it's apparently a mix between. Askey and some like ancient Latin something something that means like movement or something like that.
And so ASCII movement, I guess, is supposed to be the thing I thought it was a play on cinema but Maybe I'm just, yeah, it was just a bad assumption, I guess.
Okay. So now we have installed this pretty quick and easy to install it on a bunch of here.
And now if we do a scheme. Dash H, will see the help. And this is pretty much what the man page looks like also, very similar.
And you can see here there are several sub commands record play cat, etc, etc.
And this, program has the good kind of, help where you can do. A sub command, dash H and it'll show you the help just for that sub-.
I really like it when CLI tools do this. So you can see here the record sub-command takes all these options here.
But for starting out, let's just keep it simple. Let's just do a Rec and I'll do first.
Dot cast and what this is gonna do when I run this, this is going to open up a sub shell and it's gonna connect some pseudo terminals to it.
And the pseudo terminals are gonna record the input and output and that's what allows you to like replay it back and all that kind of stuff.
So here we go. And. And so now I'm in this sub shelves and I can do echo.
Let's see. One.
And.
Yes.
Hi. Hi. Hmm. Okay, and so all I've done now is I've ran one command which did some input and output.
So we got both streams there and then I hit Control D to exit out of that sub shell which stops the recording.
So I'm no longer recording right now. So that's our first recording. Let's go ahead and play it back with the schema play.
First. Cast and the first. Dot cast is the name of the file. And you see here this first part I type kind of quickly.
And then the second part I typed extremely excruciatingly slowly and you can see that when we're playing it back, it's playing back extremely excruciatingly slowly.
And I can hit space bar to pause. Okay. So it's kinda hard to tell.
There's no. Indicator, but it is paused right now because I hit space bar and then you can hit dot.
The period on your keyboard to go frame by frame so each you see everything dot dot Yeah, that, that, okay, like, it can hit play to resume.
And that's when I exited and that's the whole thing. So that's what I meant by some element of timing.
So, you know, it's not just gonna spit out, you know, your thing, your cast.
You know, as fast as, you know, standard out will spit it out. You can definitely do some timing stuff.
So if you're you know, the way I think this could be useful is if you're sending, if you do something in the terminal or if somebody needs help doing something in a terminal and they don't ask chat GPT.
And they ask you instead, you could record some kind of cast like this and send it to somebody. Or if you did something fancy in the shell and you want to remember how you did that in the future like 6 months from now.
Then you could record that for yourself and you'd know like, oh, this is how I got to that, whatever I did.
And so being able to control the timing, I think can help, You know, help you just be in more control.
So now, yep, go ahead.
John? you mentioned the dot and go frame by frame. Can you go backward?
Not that I'm aware of, no.
Can you start, or I guess you could start hope.
Yeah, you hit control C and that'll just cancel out of it.
And start over. There's another one too called the skin of a cat. And if we do this cat, it's just gonna.
Basically play the whole thing instantly kind of thing. I think this is like what script and script replay and stuff like that do.
They just do the cat functionality. Alright, let's do another command with. So let me go down a little bit.
Okay, let's do. Another record, will do second.
Okay. And this time I'm gonna put a bunch of spin. I'm gonna do 2 commands, but I'm gonna put like a 10 s pause in between them.
So. Echo one.
And then I'm gonna wait like, I don't know, 5 or 10 s, just a little bit, a little bit of time here.
And then I'll do echo to. And then I'll stop. And now when I play this back.
It's gonna have literally that like 5 s pause. In between the command. So 1, 2, 3 4 I apparently did longer than 5 s.
There it goes. And so this is like, this could be bad, right? If you're willing to record something, And maybe you wanna be careful, maybe you're running the bands at a different, you know, maybe you wanna be very careful, you wanna make sure you don't do any typos, you wanna go slow, or you want to take your time between commands, you don't want that to show
up in the recording. So what you can do is you can use this dashi flag here. And I'm gonna put dash I one and what this means is the maximum time between commands can only be 1 s.
So here we'll see the first command that I did. And then the maximum of 1 s delay, even if I took a half hour break, it would just be a 1 s delay and then it would play the second command.
And you could also do that when you record. So I, I could do ask any of my record, I'd I one or 10 or 5 or whatever I want.
And then it would record with the maximum being that time you specified as well. So that dashi flag works the same for play or record either one.
And there's another cool flag called S. And S is like the speed multiplier, right? So you can do 2 and this is gonna play it twice as fast.
So it's going twice the speed here, which means if it was like a 10 s pause before it's only gonna be like a 5 s pause now or whatever because it's going twice as fast.
Let's see if we can do a hundred times as fast. Okay, that's Very similar to just doing the cat, right?
Okay. Oh, and I got, okay, I got a slide for this.
And so if you don't wanna put your, if you don't wanna do that, I think every time, if you want that I like 2 every time, I think that's a pretty good default.
You can do a config file here in your home directory. And then you don't have to.
Specify on the command line every time. And here, I think this is all the possible configurations that you can do.
This is straight out of their documentation. Which will just go to that now if I can
Yeah, so this idle time limit, that's what we've been doing with the I.
And they have some more.
Some more options here as well.
Okay, let's see. I don't think I've I don't think I'm quite done yet.
I, I guess I should mention, this tool is open source written in Python. Obviously, that's why I can start with PIP.
There's also a server, a self hostable server. Because what you can do is you can upload.
Your clips. Like if we go back to the, main, Skin on my website.
You can see these are a bunch of clips that people have uploaded. And I guess they just Choose to make them public or they display them for some reason but, you can upload your own clips and then you can share the link to someone.
And so basically if this, you know, and this could be easier than sending them the file, right?
Because When you record your recording to a file, you could just send that file to somebody else. But if that's like inconvenient for some reason, you could upload it to schema.
Dot org or you could self host your own. The skin of my server and
And I'm gonna stop playing. I hope this video doesn't, show anything inappropriate.
So I've never watched that one before. Okay. But yeah, you can self host your own and then you could share the links that way or you could just send the file.
It's whatever is easier. Okay, I think.
John does it does it play in the browser itself?
Yeah, and you can if you upload it to a skin.org you can actually embed that like on your GitHub like if you have a CLI tool.
Or you can actually embed that like on your GitHub. Like if you have a CLI tool, that would be that's like the slick way to display it on your GitHub is like NuShell should probably do that.
Yeah.
I know Ed, I know you love NuShell. You should probably get some ask, casts of new shadow going on somewhere.
If you've got me going, I'm gonna do that.
There you go. Nice. Okay, so does anybody, does anybody, oh, here's another cool thing that I wanna show.
So I'm gonna do a scheme. Correct. Dash I 2. And I'm gonna do, Paul, mute.
Dot cast okay so okay so now I'm recording the maximum time between commands is 2 s so I don't need to rush there's no rush here I can take my time.
What I'm gonna do now is I'm going to echo, first. And now what I'm gonna do is I'm gonna hit control backslash.
Okay, and the backslash is the one above the enter above your inter key. So I hit control, you can't see it, but I hit control backslash right there.
I'm gonna do echo second. Okay. And I'm gonna hit control backslash again to unmute and then I do echo third.
Okay. And now I'm gonna exit. Okay, now let's play this back.
And what we're gonna see is we're gonna see first third. And the reason why is control backslash is like mute.
It's like, hey, ignore, take everything I'm about to do out of the recording.
Until I hit control backslash again. So let's make sure this worked. I think it's pretty cool.
And then third, see first and third. So skip second because that was muted. Okay, here's another thing.
You can do. Let's do. Break.
Hey, Sean, it looks like you're mute is if you're recording something where you entered a password or something like that.
So by default. I want to play with this later too. I've not tested this myself.
We can test it here in a sec if you want. But by default, it doesn't record what you type.
Here's it doesn't record what you type it only records what's displayed on the screen kind of thing, somewhat.
Anyway, it doesn't do passwords. As long as the program you're typing your password into masks it either by not showing it or showing the stars or whatever it won't be in your recording according to their FAQ.
We should test this out. But let me do one more thing here. So what's really weird is, if I scroll up through my history, this is actually gonna be in the recording.
Let's see this. So let me do,
So I'm playing that one I just did and if you saw I like scrolled up through my history.
See this is me scrolling up through my history in the recording. Which I'm not sure if that's good or bad. I don't know.
It's kinda, I mean, it's cool that it can do like that high fidelity, you know what I mean?
But, I don't know if that's a feature or a bug. But anyway.
Okay.
But it'll do that when you when you hit the up arrow. Okay, here's another thing that I don't think script can do.
They can't record and maybe it can, but I don't remember it being able to.
It can't record the greatest application. And, ever invented, which is, so if I do, If I go and to nail them and I'm in the best text And it's or.
And I save and close that. And I stopped my cast. And I play that back.
I'm pretty sure this will work.
And you don't know this all your time pose until you plan it until you're literally watching it back and like horribly slow speed.
And also, like, I really type that slow? I thought it was a fast hyper, but this seems very slow.
Anyway, that's cool that it worked, right? It's cool that it worked. And you know, we could like.
Does the I 2 while you're recording record at double time?
So like if you wanted to create a slow move video type thing.
You wanna let's see. So the eye, the eye is idle time limit. So it just limits the number of Seconds.
So, and the dash S that I showed you before, that's on play. And it plays at a double speed.
Yeah.
Or yeah, how it's how much you wanna speed up the playback. So. I don't know if there's a way to record faster or slower.
Typing faster.
Yeah, recording with no dash I. And just type the quick brown fox type thing. Close it, play it.
And then redo that recording with like an eye of 10. Pipe the same thing. Close it and See if it affects timing.
Yes. So this is test no I. And you want me to type something.
Just a catchphrase you use. I don't care.
And then I'll do hash i 2. And we'll do.
Hello, and now we'll play.
The No, I.
I'll play the with I
I made the same type of both times. I just wanted to be consistent. I want to give us the consistent experience.
I think it's the same.
Okay.
Yeah, I think it is too. It's certainly not double or half.
But you can, I mean, I could do dash S 100. It will really see me type fast.
Okay. You wanna try a password thing? What's that?
Hold on. Well, let's take a manage number. Make it a number.
Hmm. For the S, let's see.
Nope. Oh, maybe it thinks it's another argument. No.
Or maybe a point 5.2 5.
Yeah. It's said it could be fractal.
Okay, yeah point 2 5 that's painful
Okay.
Okay, alright, let's, let's, okay. Alright. I've went for quite a while.
We could keep trying to break it. Like we could try, you know, it works within, does it work with SSH, doesn't work with.
Do password. Oh, I guess I can show you the. Let's cat one of these, just to see.
So basically what this is, is they, so he calls it line. He calls it, new line, the limited JSON.
So I guess each one of these is a valid JSON thing. So I guess this is an array and stuff like that.
And so, and I guess this is something to do with the timings here. As the first, element in the array and stuff like that.
And so Yeah, really cool. Oh, but you know it doesn't work though. Do you remember how I showed you that the new of M one works?
It doesn't work. Which I don't even know how you would even do that, right?
But if I cat the, It doesn't show the, window, right? Because And I don't even know how you would do that, you know?
I feel like this is recording. What is considered echo from the system. And when V exits. It's actually probably either clearing the screen or changing.
Screen mode like from alternate screen back to regular screen. And so when you cat it, you jump straight.
To the end state, which is no window. But when you do a play. You actually see the progression.
That seems to be what's happening. Yeah.
So you could edit the JSON if you made a mistake like you. You could go change the as a text.
I mean, the text is right there.
Yeah, totally. You could just like delete a whole, I don't know, let's do it.
Let's delete a couple of these. Let's see, so here's where I typed in VIMA dot TXT.
Let's see. And then it gets kinda crazy. Wow, it's really crazy.
I think this must be like all my colors and new of them like my oh yeah, cause here's my status line.
Here's where it says. Insert down here at the bottom. And all that. Let's do a different one.
And then one's probably not a good example. Let's do one of the simple.
Yeah, the one you did 3 echoes.
Oh, mute, yeah. Okay.
Is this better?
So it's every, it's every, keystroke. So.
It'd be tough to edit, but you could possibly.
You technically could, yeah, technically. Could do that. So if I take this eye out. Then it's gonna.
It's gonna be. Yeah, I took the eye out. So there you go.
Okay.
Well, you know, when you have nice, you know, open formats. That's good.
Yeah. Oh, yeah, if, you, so that I don't know if anybody likes, oops.
Podcast or whatever but there the work got me onto this was Linux downtime, episode 83, they talk, this is where they interview.
Marston.
And he's the, I guess one of the main authors, Skinner. So if you're interested at all, Linux downtime, episode 83, will go into much more depth about this.
But I think I've done Quite a bit. Or why I could keep going, we can keep messing with it.
But while you're on that page, could you post that link?
Like I'd like to see if it works in Docker. Yeah, certainly. But I've also taken quite a bit of time already, so we could just move on.
If that's what we want to do. Why don't we go ahead and. Just do that actually.
I have 3 quick questions though.
Can you make it for? Cause I feel like 3, a, no, go ahead. What's up?
You just ask the fourth.
Okay.
Yeah.
Yeah, you're right about that. Okay, so the first kind first 2 had to do with a web page again.
Can you copy and paste or cut copy and whatever? Can you copy off of the web page?
The text like you can when it's in the terminal.
Yeah.
So it's regular just text on the web page. Okay, second question is on the web page.
You said you could upload it to the org site, right? Can I download it from the website?
In other words, I don't want to play it in my browser. I want to play down my terminal.
I don't know. We could take the time to investigate. I've never, so I've never uploaded. I logged.
Okay, so when you I got an account or whatever on their website and all it requires is an email address and a username that you provide.
No password, no other information. So I feel like it's very privacy respecting if it only takes your email.
And it's one of those things where like you put your email in and it sends you a link and then you click that link and then you're kind of like logged in.
Which I feel like it's fine for something like this because if you're if you're dumb enough to upload secrets to.
Dot org server then you kinda deserve what you're gonna get but yeah, but I did not play around with it too much other than to just be like, Hey, it works.
Okay, I'm done.
Okay, I'll check that out later. Second question, so we don't have much time is as somebody mentioned that he thinks it's like recording the echo like what's being echoed right that was the thing about the so my question is what I like to do a lot when I'm typing in the in the in the terminal is I'll actually add a you know a comment with the pound
sign. Right? And that shows up when I hit up arrow, right? Right, so I can see what did what did I what did I do here, right?
And you can, I can like make a little note to myself. I learned that from Craig Buchek many many years ago.
So that's just echoing it, right? In other words, it's not. Something that the program echoed.
It's something that you typed, but the typing itself was I'm assuming that that will work.
Can you try that real quick?
Good thing.
While he's doing that, what I was talking about, anybody that's messed with terminal.
Physical terminals or terminal emulators and cereal ports. Local echo on versus local echo off.
Yeah, kinda like half duplex and duplex, right? Or serial terminals.
Oh, yes. I am suspecting a schema is recording. What would be the echo what is coming from.
The host. So when you're typing your command and space space space pound sign, this is my super special time saving command.
And hit Enter to execute it. That is part of what comes from the host. So I would expect.
That Escenima would see those command comments and
How much?
Yeah, and that would really make it helpful for, if you're doing tutorials or whatever that could see that.
Yeah.
Yes. I really like the idea of being able to edit the files or even create new.
Huh. All right, exactly.
Good job for JQ.
That was gonna be one of my examples, right? Like, cause I could totally, cause you know how sometimes when you're, or like when you're writing a shell script, right?
It's like. You come across this line that's like pipe to cut, piped off, piped to this, piped to said, pipes, and like, you have no idea what the result of that is without like running it, right?
But it'd be cool if you recorded a cast that was like every step of that process and you could actually see yourself building that pipeline in the cast, you know, or like which AQ if you start off by like, okay, give me all the such and such and then filter down to the first thing like oh crap I need to add the last name too let me get the last name and you could like record a cast of that
and so that way the end product if you want to know like how did I figure this out again you could you know use the recording and so that's something that I think could be a good use case for this.
Thank you.
I had put in the chat earlier, any idea if it messes with the terminal type at all?
You know, screen and a few other things do which can be annoying when you do fancy things with special terminals.
Interesting. Yeah, I, I don't use any fancy terminal stuff. So, I, yeah, we'll have, we'll have to do some homework on that one.
I will.
But they have a whole big page on how it works though. And they talked about their pseudo terminals and master enslave and more details and I really read all the way through.
So maybe that would answer.
Let's see. Did you get a chance to glance at all of the chaps?
We covered everything there.
Yep, looks like it.
Alrighty. Okay. I think probably we need to start heading on, but, gee, Sean, this was really interesting i mean it a great counterpoint or follow-up on, on Stan's presentation last month.
So thank you. And, Let's see. Something I was going to ask.
And it scapes me. You'll come back. All right. Oh, I know.
Sean, just by way of introduction. I don't think I ever told people where you work.
So where do you work down there?
Or not want to say.
Yes, I, oh no, it's okay. I work for Southwest Airlines.
There's no big secret. I work for Southwest Airlines. I work in their training department so we kind of schedule and manage the pilots training to make sure they have all their initial and annual training so they're safe to apply the aircraft.
Okay. So. Of course, I think the biggest airline here in St. Louis by number of planes and gates since.
Oh great. Well, thank you once again, Sean. This is an absolutely exquisite presentation.
Really enjoyed it. And I'm sure we're gonna appreciate it. So. With that, this is the GUI middle of our Oreo cookie design of meetings.
This is where we do. The announcements the calendar and brief call for help And then, then we go to our second topic of the night.
And that'll be, Robert Sitek and, we'll get to that in just a few minutes.
So, Let's see. Well, I'm Gary Meyer. I'm the president.
Stan, whose voice you've heard a number of times here is our vice president. And Lee is our treasure and yes we have a vacancy for secretary if anyone We care to help out a little.
We'd certainly appreciate it. So contact. Oh, any of the other 3 people there, me, Stan or Lee.
And we'll see if we can utilize your talents. So let's see.
We also just need a whole bunch of volunteers for one thing where we, we maintain our own server.
But, and the words, name is Bach. And we want to upgrade it. Right now it's sitting on Debian 10.
We want to get up W and 12 and so we're just looking for a team of volunteers. You don't have to have a you know, an amazing background.
Basically, he's wanted to get together a team of guys who all wanna. You know, do an upgrade of Debian over a couple of releases.
It'll kind of be a captures catch can, you know, maybe somebody will say, okay, let's have a zoom.
Meeting and talk about this. But it kind of has to be planned out as to you know how you all get together and do it and then the actual planning of how to do it.
So, anybody's interested in. Kind of experiencing being part of a team doing an upgrade.
This is your chance so Coming up next week, 8 days from now is the St. Louis Linux meeting.
That log meeting the the speaker is some guy named Sean Twe house i don't know we've heard him before Oh yeah, yes the Sean who was just speaking to us.
Oh and he's going to talk on the ZFS file system. 3 2 one backups and bit rot and the abstract is out there on the calendar I think already And I believe I don't think I have it out on meetup yet, but.
I'll definitely get it out to meet up tomorrow if I don't have it out there already.
So, but anyway, it looks like a really interesting topic. Those of you who don't remember and they're scratching your head with something that Sun brought out.
As part of Solaris. And it really was an amazing upgrade in file systems. And then of course, on sold itself to Oracle.
And eventually Oracle open sourced it and it kind of At least I'm, if they kind of lied fall over a few years, but Sean will probably give us.
Give us a good update next Thursday about how this all came about. 5 days after that Thursday meeting.
On the fourth Tuesday is the next new Linux meeting. New Linux meetings are basically The day after Christmas. It's the day after Christmas.
So, certainly everybody will be talking about that new computer they got in their Christmas stocking. So.
Of course the newslet, new Linux meeting also is the chance for the rule is whoever whoever considers themselves to be the most neophyte, asking the simplest question, that gets addressed first.
And, then they move up to more complex questions from. From the crowd. So, it's a chance for people who who, don't want to call themselves experts yet, but still would like to have a good technical conversation, so.
Other announcements. It is holiday weeks and, so I know it's difficult to get to some of these things.
Anybody have any other announcements?
Now for those of you who are
Not really, I mean.
Okay, not really. Okay. With that, let's see. Oh, I guess I could announce.
For those of you who are looking for the secret Air Force launch tonight, it was. Delayed because of whether they rolled it back into the hangar down at Kate Kennedy.
So when the weather clears up all Florida. There may be another neat. Satellite launch or secret satellite.
Oh, it's weird. They show pictures of it. We just don't know what it does.
It goes up in our, but it stays up in orbit for like a year and then it comes back down.
It's like a miniature shuttle, unmanned. And the Air Force sends it up. It's called the X.
37 B and they've never. Told us what it does. But, it goes up, spends a year, so up in orbit, then comes back down and I put it back up again.
Let's see. Other than that, Oh, it will be elections coming up in, February.
So, It's a great opportunity for anybody who wants to Wikipedia has a nice page on it.
Anybody who wants to volunteer, we'd love to have you. It doesn't have to be with the elections.
I mean, we've got so many roles that could be filled. Robert Loverts been doing some publicity work and There's calendar to take care of and just all kinds of things so yep, we need more fingers, more people working on things.
Let's see. I don't remember anything particularly on the calendar we need to point out.
There are some conferences and stuff if anybody wants to say anything about conferences they've been to or going to.
More than welcome. And, see, the announced lists, that comes out.
We used to get it out once a month, then it became once a week. Now it's kind of just very intermittent, but very low, low volume.
The discuss list. Oh boy, you can ask any sort of technical question. Sometimes amazing and sometimes not technical and sometimes not technical.
The steering committee, we try and not bother everybody with how we make the sausage.
So, The steering committee is just gets together to talk about who we can get as speakers and what things this organization can do so it's not a secret committee you're certainly welcome to come.
And And like I say, we just try and not bore people with the sausage making, but.
And let's see, St. Louis Linux user group, the St. Louis log. Good part of organization.
Slack actually started as a separate group and then kind of became part of us. It, It's the as far as we know it's the oldest computer group in St.
Louis. And so at the Slack meeting they talk about hardware, they'll talk about Microsoft, they'll talk about all kinds of different stuff.
And then the new log, I just described that a minute ago. So it's kind of more for More from beginners, but boy, it sure helps to have people with the technical.
Expertise come and so yeah, don't don't think you have to be a new new user to be there.
It really helps. It's amazing sometimes a wonderful technical conversations. I've learned a lot. And let's see, next slide, Sam.
Okay, next coming up tonight is going to be our main topic and that usually goes into a little more depth, a little more technical aspect.
And so we're gonna do that here next. Is there another slide?
Okay. This is the general Q&A period. It is also after after the main topic. It is also after after the main topic, we can go back and do more general Q&A.
We can go back and do more general Q&A, but. If anybody feels they need to get in their question now because they may not be here later tonight.
You know, now's your chance. ask a question. Maybe somebody here may have an answer.
Okay. Not hearing anything. Which is okay. That means everybody is.
So, our main topic tonight, is going to be done. By Robert Sateck.
Robert has been Robert's been troubled without a move big data sets or big amounts of data. Efficiently get them into programs and stuff and so He's gonna tell us all about Python's panda data brains.
And every time we have Robert, it's always a good, good talk, so. I'm going to turn it over to Robert.
And we're not. Okay, there he is. Yeah. Okay.
Alright. There I am. Greetings everyone. Greetings, greeting.
Let me see if I can fire up my video as well. Let's see if this works.
Yeah. And Robert, you've been such a familiar face. I neglected to really give much of an induction, Robert lived here in St.
Hmm.
Louis for a number of years. He was a vice president of the organization for a while i think and a a great contributor even we were meeting face to face and now he lives in, is it New Mexico?
It is, Albuquerque, New Mexico. Yeah.
Help. And he's teaching and so we're lucky to get him tonight. Finally got done with all those students this semester.
Yeah, literally. I just turned in grades this morning, so it's been, it's been interesting.
And, yeah, the next, the next group is Oh boy, they start tomorrow with their with their pre work and and then that starts soon as soon as the holidays are over so yeah exciting stuff.
Alright, let me see if I can figure out how to share my screen. Yeah.
Alright.
We are sharing your screen. Yes.
That working there we go excellent all right so as Gary said yeah many many years ago I lived in St.
Louis, Missouri. And excuse me boy, what it's been about 12 years, maybe, Maybe 15 years now.
Well, no, not quite 15, then about 12 years since I left. I went to Michigan for a while.
Ed went with me. Well, sort of kind of. and, then I ended up going to Portland and meeting some other folks and then, and then down here in Albuquerque, New Mexico.
They have been in health care and biotech for the past 20 some years. As well as in research.
Genomic stuff dealing with lots of amount of larger amounts of data and you know got a chance to retire for a while during COVID.
I enjoyed that. Wife wasn't so keen on it. And so she said, hey, you gotta get back to work.
I said, okay, okay, I guess I guess so. And she ended up applying for a bunch of jobs without me knowing it and one day I got a phone call and they said, hey, how would you like to do blah, blah, blah?
And I've got a phone call and they said, Hey, how would you like to do blah, blah, blah? And I kind of went, who are you?
And ended up, ended up interviewing at this position and did it up getting the job. So yeah, I am the lead instructor here.
In the data science course, working with actually a few other really, really talented instructors.
We have a number of programs, obviously data science, which is what I teach, but we also have, Java and, and Android.
So they're, folks that build, Java apps, for the Android.
We also have an, IoT, group, a, digital, community college which is how you say basically community college so you get a accreditation you can then go on and do trade or go on to higher education and at University of Mexico or you know any other four-year university.
Our focus is actually on upscaling individuals who have already been in the workforce and now are trying to do something new or People who just want to change in in careers.
They want to do something different and you know they've been doing something for a while they say you know this is not what I want to do anymore we're gonna move I want to move on and do something else.
So. The program that I mentioned including the data science one is a 12 week program. It's 8 to 5 and literally I'm just blah blah blah blah blah the entire time.
Well, not entirely true. They ask questions and I answer them as well. And but yeah, it's it's intense, right?
It's 8 HA day. It is not, you know, 1 h of a lecture 3 day of a week.
It is 8 HA day, every day for 4 days a week. That fifth days actually spent trying to telly catch up and and relax from breathing or drinking from that fire hose.
So this can't talk him about because, had some conversations with a few people who, are at Slug and we were talking about data.
And stuff like that. You know, hey, how do you do this and that? And I go, wow, you know, in panters that'd be really easy. You just do blah, blah, blah, blah, you're done.
And You know, and later on another conversation will come along and it's like, oh, you know, how do you do blah blah blah?
And it's like. Oh, you just do that in paint as it's, you know, just a couple lines of code.
And after a while, it came down to, you know, you keep talking about Panas, you know.
Why don't you, why don't you actually show something about pandas that is actually useful?
And so that's how this talk kind of came about. It's literally just bore out.
It was born out of some conversations that I had with some slug members as well as some other folks.
And I said, well, I got this material in my lectures. I'll see if I can.
Stitch pieces so we don't go too deep because if literally if I were to talk about pandas with any true substance we would we would have a a good you know 4 or 5 h if not a full day lecture where we can do you know, 4 or 5 h, if not a full day lecture where we can do, you know, tons of stuff.
But we only have, I'm gonna try to keep it short. I'm shooting for 30 min and I know I'm gonna go over that.
It's gonna be 45 and probably longer. But, but we'll try it. And see how it goes.
So I want to start off with a couple of examples. Just of what you can do with pandas before we actually get into what pandas actually is.
Little preamble, it is Python. We are going to work in with some data structures.
So if, if anything is, I would say kind of new to you or unfamiliar. Please let me know.
I'm happy to answer questions. So please do interrupt me and, and ask those questions.
So, Let's fire off and the first one I'm actually yeah question
Oh, alright. So. The first part came about with talking with with Ed and I and I heard him earlier.
And so he was talking about.
Did you give us a couple of on the font, Robert?
So say that again.
Can you bump the font at least one or 2 sizes?
Thank you, sir.
Oh, absolutely. 1 s. How's that? Moore?
One more. There you go. Perfect.
Well.
Yeah, oh that's, yeah.
That's better.
Very good, good.
Alright, cool. Yeah, no worries. So And had some things, he's working on, he's, you know, had this, he's got this call record, if you, he's got a phone book and he's got a call record that, keeps track of his calls.
And he wanted an easy way of putting this stuff together. And of course, The immediate thing that comes to mind is what you always wanted to just upload it in Sequel Light, do a join and you know, hey, Bob's your uncle, right?
But I thought it would be a good nice example of demonstrating what some of the stuff that you can actually do in pandas as well.
Nice example of demonstrating what some of the stuff that you can actually do in pandas as well.
The environment that I'm showing on my, I'll create a GitHub repo and put it in there and you guys can play with this stuff as well.
Google collab is is great for that kind of stuff. You literally just click a button boom and you have a full environment ready to rock and roll, right?
So the environment is very much like a Jupiter notebook. I think it's Google's implementation of a Jupiter notebook.
We have a combination of cells or a it's a collection of cells what they call cells very similar to Excel except it's just one column and these cells are either text or code.
What you're looking at here is this happens to be a code cell. We're basically creating some fake data and creating a file called phone book.
CSV. Right, so I'm gonna I'm gonna run this. And it's going to go chuck a chug at Chuga Chuga.
And create this phone, right? So it wrote the phone book and now we're gonna, I created a bunch of fake data for, creating a fake CSV file for the call record.
And so we'll run that as well. And it wrote that. So I'm going to just do an L.
That is the standard way for listing. The long format with everything, on the file system.
And there they are. There's the detailed record and there's the phone book. If we look actually at the CSV.
Files, we can see there they are. Right? So, Standard CSV stuff, right?
Comma separated values. Everything is in quotes. Yeah, I probably could have left the quotes off the numbers, but it was easier if I didn't.
So,
What the questions that that Ed had, as he wants to know, would you as I've got a bunch of people in my phone book and if I get a call from there them I want to know you know who you know when they made a call and I also want to know if there's people that made a call that are not in my phone book, who are they?
What are their numbers? In other words, how often do I get unsolicited phone calls from people that I don't know and how often am I making a call.
Right? So let's, and, and how much are those calls, right? That actually would, it didn't ask that, but I'm throwing that in there.
So this is actually a fairly simple to do in pandas, which is a module. and in python so we're gonna we're gonna load this modules in And this is how you read in a CSV plot.
You simply say PD, which is the. Alias for pandas, read CSV, give it a CSV file, and then we'll display the actual, what is called the data frame.
And this is a data frame. And so we notice this looks very similar to. Any other kind of tabular data that we've ever worked with, right?
This could be a spreadsheet, it could be a database, it could be a data frame from R.
Or any a number of cases right so we got rows Right? And we have column speed dial name phone number and type.
And that's all there is to importing a CSV file into a Python data frame.
Right? We can also do some interrogation. We'll get into that in a little bit, but basically if we ask it what is the shape attribute it tells us well there's 10 rows and 4 columns.
Well, the same thing with the detailed record. Right? And we'll display the first 5 rows.
Right, this is the nomenclature, the syntax for displaying the first 5 rows.
One of the ways I should say. But that gives us the first 5 rows. Again, we can look at the shape.
We can tearrogate the, now we know we have 40 rows and 9 columns. We can also do some, I would just say ask it about the metadata, right?
So we can do this info method and it tells us about the data frame, right? It tells us that it is a data frame.
It tells it that there are 40 entries, number 0 through 39. There are 9 columns. These are the names of the columns.
How many are null or I should say how many or not null and and then what kind of data type is it.
In this case, we see we have a number of objects and integers and floats. And this gives us a nice little bit of objects and temperatures and floats.
And this gives us a nice little, summary at the bottom. We've got 3 floats, one integer and 5 objects.
And so the question is, How many outgoing? Let's let's look at the outgoing calls, right?
And in order to do that, we want to actually do what would be equivalent to an inner j, excuse me, a left join between.
The call records and our phone book, right? And this is how you can do that using pandas.
You basically say, great, take, it's in 10 as they call it a merge instead of a join.
And there is a join, but we won't talk about that. But we take our initial.
Call record or call call log, we take our Data frame our phone book data frame and we say on the left side use the calling number.
This is the column name on the right side uses a phone number out. We're adding this little suffix out and we'll see why that in the menu in a minute.
And I want you to do a left join. And when we do that. Here are the first 10 calls.
And one of the nice things that we can do is there's actually a slightly different format to read this.
We can click on that and it's a slightly Cleaner format if you will. I partner my apologies for the wrapping on this.
It's just because of the font size. But here what we can see is we've now done a left join.
We now have not only the call, Who called? Excuse me. Yes, what number placed the call?
In other words, what was the outgoing number? What is the incoming? number.
How long it lasted, how much it cost, the tax, the total, and so forth. And on the right hand side, we can see that it was John DOE who made that outgoing call.
We've now joined it. Based on the phone number, right? That's what this is, right?
The phone number out is coming in with the calling. So we have all the information on the right hand side from the phone book.
And we have all the information on the right hand side from the phone book and we have all the information about the about the call from the phone book and we have all the information about the about the call from the call log on the left hand side.
Standard stuff if you've ever done SQL stuff, right? To have. Database queries and joins, right?
And we also, we have, let me display all of them. If we scroll down, whoops, scroll.
I say only, I'm only showing the first 10. Let me show all of them. 2, 2, 2.
Do that.
I notice we have a lot more. Oh, come on. There go, let's make a little bit pretty.
Here we go.
If we scroll down. We notice that, yes, we have all these joints with the, with the phone book and then we have these that say NA.
And in essence, this is Python's way or Panda's way of saying, these are nulls, right?
It's not a number, is what N and stands for. And in essence, this is a call that a originated from someplace else.
And for an outgoing call, it did not come from someone in our in our phone book. So.
One of the other things we can do is we can say, well, gee whiz who Who, or excuse me, how many calls did John DOE make or can, can we just see the calls that John DOE made, right?
So we could do something called creating a filter and if we do that, it comes back. And says, oh, name is doesn't it isn't defined.
Let me. Fix this real quick.
Here we go. And the way Pandas works is selection is kind of a 2 step process. The first step is you go and you create an expression, basically a wear clause for those of you from SQL, that looks something like this.
It's a it's some kind of a Boolean expression. We want outgoing calls equal to John DOE.
We can create a filter and what that filter is is literally a list of more correctly a data series of which rows match that and which ones didn't.
We can then filter that and say, okay, which ones actually did match? And what we see is rows 0 6 and 16 matched.
If we use that filter against the actual data frame of outgoing calls. We get all the calls that John domain, right?
And those are 3. Here's 0 6 and 16. And we see that. John DOE made those calls.
What about incoming calls? Or we can do the same thing. Alright, we can merge it and this time.
Oh! Go back up. Come on. This time, we're adding on the suffix.
That's just for readability to say in, but this time we're just saying the called number and stuff is a calling number and we're gonna go and, merge it with the phone number in. Right?
So now these are all the all the calls that Jane Smith received and Avra Brown received and Charles Davis received.
Right, so these are incoming calls instead of alcohol calls. Well, what if we want both of them?
Well, we can actually do that. We can take the outgoing calls and merge the with the incoming calls.
And we can sort it based on date. Or dig time more correctly. And so now this is all the calls.
Right. And so we see that. Here Charlie Davis made a call.
On October first And. Jane Smith received the call. Also, excuse me, not October first and January, January thirteenth and on January fourteenth of this year.
Right, so we now have both incoming and outgoing calls. With just a few lines of code and pandas.
And so, let's see what our call shape looks like, our data frame. We have 40 rows and 17 columns.
Right. One of the nice things that Pandas has. Automatically is that it can do graphing.
So we can actually now say, alright.
Create a histogram of the call amount. And give me 10 bins and show me how many calls were made at different price points, right?
So we have our amounts on the bottom. Right, so these were the how expensive a call was and how many calls were made at each of those at each of those price points.
So literally one line of code I just broke it up into 3 just for readability, but along one line of code we now have a histogram.
We can all still. You do things like, well, how many calls were there? In this case, we're going to do a value counts.
Basically says, okay, let's look at the person's name and how many calls did they make.
For those of you in Sequel, this is equivalent to a count and a group by.
Not exactly the same, but close.
We can combine them into a single data frame. Right, so Alice Williams made 2, had 2 incoming calls and 2 outgoing calls.
Bob had 2 and 2. Charlie had 4 in one out and so forth. Let's create a quick graph of what that looks like.
And here we have a bar chart that shows. What we have. So literally with just a few lines of code, obviously I wrote a lot of code there in the Snowbook to explain what's going on.
But in practice, you would probably write 4 or 5 lines of code and you would have a graph that quickly shows you who's making the calls, where are they at, how much they are, and so forth.
So that is one example of of how Pandas can help. Any questions on what I've covered so far?
That one line we said. Filter, bracket filter bracket. So is. Is.
Yeah, where's that?
What was?
Here we go. Right here.
Yeah. Yeah, yeah, yeah. Okay.
Alright, let me break this out a little. Yeah, go ahead.
So. Yeah, is the one filter that a command and the other filter what you had just defined above as the filter.
Is that the one?
Hold that thought because I'm going to talk about how you work with data frames and series and things like that in a little bit.
Okay.
But in essence, this is this syntax says. Generate this is my conditional. And this will generate a filter with, you know, I just assign it to the variable called filter because it's convenient.
But in essence, this becomes as as a data series of You know, row one is true or row one in this case would be false or 0 would be true.
One is false, 2 is false, 3 is false, 4 is false, and so forth. It's just a Boolean list of which ones are true and which ones are false.
Got it.
In other words, meaning which ones match that condition and which ones don't. Right. And you can take that list and give it to a data frame and it will use that list.
To filter out the ones, the ones you want. So anything that's true.
Gets comes through everything that's false gets filtered out.
That makes sense? Alright.
Okay. Okay. Okay.
See that again?
It's now with, with the area formulas in Excel, that sounds like an It sounds like an area formula in Excel.
Were you dealing with an array formula?
Yes.
Okay, can, there's a little bit of an echo. Can you spell the word.
A RAY.
Okay. Okay.
Yeah.
Alright, that's all.
Yep.
Yes. Huh.
On the question on the merge. Do a full outer join. Is that the cross option or is that?
Something else
A full outer join would be a left or right and, not anything in between. And can it be done?
Yes. And in other words.
Yes. It can be done. There's a syntax for it.
You can also do a full cart, a full cross join if you want to as well or Cartesian join however you want to call it but basically it takes all combinations between 2 tables.
And there's, there's lots of other things that you can do with this. With, with data frames.
So yes, any, anything, any kind of. Manipulation that you can imagine you can likely do with a data frame.
Okay, cause I have I've figured out how to temple.
My list of people on my list of accounts. And I'm trying to figure out what accounts don't have people and what people don't have accounts.
Yes.
So, and I just need to do 2 different joins, but if I could do that with a Cartesian join.
Huh?
And then say, okay, I wanna know what people don't have accounts and then. Jump at that as a list and what counts don't have people.
And dump that as a list, then
So, so effectively in this particular case, I just did a full outer join, right? I've got incoming calls, I've got outgoing calls. They're all in the same data set, but I call them 2 different things.
And if we come down here where I have both. Right. Let me just make this a little, let me just make this a little smaller for a little bit just to.
Just a highlight. Yeah, not that small. Come on, come back up.
And I'm gonna get rid of the sort by real quick. Just to just to show.
And let's see if that clean it up a little bit.
Come on, all the way up. Here we go. And what we see is we got everything that matched at the top.
Get rid of this. Right.
And everything that had an incoming call. Right. That didn't match has this NA.
Alright, thank you.
Okay. That.
Yeah. So yes, he. Ed, far away.
Yeah, yeah, I got a couple quick questions. One because I couldn't see the screen. The first time the first thing you typed in, you slurped in.
The phone book and you use PD, right?
Yeah, which is an alias for Pandas.
Okay, and. Was that like in the equivalent of a Python repo?
And what I'm asking is did you have, did you put brackets around the file name and then quoted the file name?
Yeah, I put parentheses around the file name. Oh, excuse me. It's a, it's a, it's a method.
So you got parentheses and then inside that, inside the parentheses, I've got double quotes.
For the file name.
Yes.
Okay, thank you. Alright, second question. And this comes up, this comes up a lot whenever I do it and it came up with these exact call detail records that you're talking about.
And especially with new show. So new shells got like a built in data type, right?
And you can read in. This is so there's like a data type literal and it based upon like some version of ISO daytime string format, right?
Okay, and The ones that come out of our VoIP provider here. They were quite not quite right it didn't quite exactly match So a new show couldn't couldn't handle it.
It handled it as a string, but couldn't handle it as a day time so I couldn't ask questions like give me everything from this week to this week, right?
Okay.
Which is possible a new show if you, if you had it, right? But, but without the dates without reformatting the date, right, I had a problem.
So my question is, and does. Except like these sort of weird dates that they had like I don't I didn't give you an example of what the date look like, right?
Yep. Yep. Yep.
So I assume you just made those up. And your fake data. So. Right, so I don't think, I don't think they look normal, but they look pretty straightforward, but they don't don't look like.
So date
Or day times I should say. Yeah.
The date time. Yeah, the date times that I have are, good question. They don't look like ISO, 8601.
I think you're you have you need to have a time zone as well as a T in between the 2 to be ISO 86 or one but that's coming off my memory I could I could I could be wrong That said, Pandas has a very extensive library, for dealing with dates and times.
You can do all kinds of stuff where if the format is different than what you expect it, you can give it a format string.
It will read the string and extract the information in in the right way. So, you know, if you give it the right format screen, you can get pull out the the year month day, hour minute second and so forth, right?
Time zone, what have you, right? And convert it into a real daytime object.
Okay, okay. And that would be like a, like a Python daytime object.
Correct. Correct. Yes. Yes.
Internally. Okay. Okay. Okay, and the other question, which, you and I talked about prior to the tonight.
Was about the headers that I got. Okay. So you mentioned that you could use like, I don't curl or something.
And go directly to the web page because all this data, right, if you're logged in, right, is available on the web page and a table format like an HTML table.
And it looks identical to. In this comes separated. Value.
And the first, row. You know like the TH row in the table right Is the is the header row right?
Okay.
And when you download the call detail records, you get This first row in the CSV is just that.
But it was weird because like it had like. Spaces and pound signs and things like that. And they're like, say, call number, you know, like call, phone side, right?
Or call outside or something, right? And I don't know whether that would like it would accept the spaces or not.
So I put quotes around it for your ease of use. The question I have is, well, Pandas like handle that.
Or again, is there like, you know, like filters and things like that you can do. to slurp that stuff in better.
And we say.
In other words, did that when you get it in the CSP for it doesn't have the quotations around the headers.
Right.
But how, what is it using as a delimiter?
The comma.
Oh, okay. So if it's common, the limited, it doesn't need the quotes.
Right? Doesn't absolutely need the quotes, right? That's your delimiter. If quotes are there, it'll use the quotes.
But if it's not, it won't. That's in the important point of the quotes is if you have a comma inside.
Whatever your data is or your headers are, right? That's that's why you need the quotes.
Otherwise the quotes are optional and then it'll it'll parse it just fine.
Okay, and I'm sorry to keep harping on this, but going to the phone book example, and this was the cool thing about about SQL light.
You'll notice the first column on the very left hand side is speed underscore dial I think or something.
That's what you gave me.
Right? And I I gave you that because I've typed that in by hand and the again the boy provider does not provide any heading or any heading.
Information on the CSV for that so I made them up out of whole cloth. But,
The interesting thing is like if you don't have any speed dial information, right, which we don't.
That's in the data portion. The first row after the header. Right.
We'll start in every row after that. We'll start with a nothing and then a comma.
Huh?
Right? So you just basically happen like an empty cell, which is an empty string, right? And Nicely secret light.
Accepted that. But, I don't know whether Pandas would or other things.
So.
I'm pretty sure it would. I don't want to say it will because as if I don't know if you All of the conversation between Ron and myself, but.
About some things with pandas. Every now and then you come across across some quirks.
So, but we can try it out later on and see what happens.
Okay, so the stuff you did when you created the, fake data, did you put, Is put like an empty string in there or did you put like a like a space or an actual speed dial number?
Yeah, I have numbers and everything one through 10 and then fake names, fake phone numbers, fake type.
Oh, okay, okay. Alright, well then this is my question. Okay, thanks.
Yeah. Yeah. But we can. Yeah, go ahead.
Have a The other question, okay, I just wanna point out is. If you wanted to read that straight from the web page.
What you'd probably do is use the Python requests module. To extract. The HTML from the web page.
And then I think you throw that to Panda's read HTML method. To parse that table into a data frame.
That is one way to do it. That's not the only way to do it. And that's what's cool about pandas is it's gotten, Pandas is turning into Pearl.
It used to be, you know, Python, excuse me, Pandas, Python is turning into Pearl.
Python used to be the, what was they saying? Something like, you know, there should be only one obvious way to do something because Pearls model was there's always more one way to do something.
Yeah.
But, but Python.
Yeah, Tim, there's more than one way to do it.
Exactly. And, you know, Python is turning into that. And it's got You know, Pearl has its, what do you want to call it, idiosyncrasies and Python is now getting more and more idiosyncratic as well.
It's it does some stuff. In inconsistent and weird ways, at least in my opinion. But that's, that's what it is, right?
Alright.
It's, it's, it's, it's, it's a cool language, but it does those things.
So.
Awesome. Like any other system where you have 90. Or thousands of people contributing to, I think it should be done this way.
Yeah. Right. Yeah. And there's, and I'm trying to show you just, just the minimum.
On there are. So many books on this stuff and videos and you tubes and what have you that get into the gory details of some of the really cool things you can do.
This is just the, hey, this is a way to do it, not the way to do it.
So, alright. Let's hold off on questions. I want to go to another quick example talking about reading things from a website.
I just wanted to show you something from the Sec's Edgar Database. This does indeed use the request module.
It makes a request out to. Out to Edgar and what it's doing is it's using the Edgar API.
To get some information about McDonald's. So McDonald's is a CIK number is 63908 I'm looking for their tenk for from 2,021 and so what I can do is I can go to their API URL or construct the API URL for that JSON file.
I have to give it a. A user agent? And I include that into the headers when I make the request, right?
So I make the request, I get a response back of 200. This is what the response JSON looks like.
These are the keys, right? So it reads the JSON, turns it into a dictionary which is Hions version of a hash and associative array, key value lookup.
You map, whatever you want to call it, right? It's basically key values. And these are the list of the keys.
I can see or interrogate that data structure a little bit so I can see CI K is a string, name is a string, filings is a dictionary.
I can play around with filings. It happens to have a key called recent. I can see what's in there.
One of them. Is something I'm interested in. So let's read that into a data frame.
What we've done here is we've gotten a JSON file, pulled it into, pant or excuse me, pulled it into Python as a dictionary and then we're now going to read that dictionary and turn it into a data frame.
And when we do that, this is what our data frame looks like, right? It's this big long thing, lots of columns.
We got a. Thousand rows and 14 columns. Personally, I like to flip that around sometimes, especially if I want to get just one of these.
So, and I'll maybe I want to plot it. So what I can do is I can take that data frame, which is this here.
Oh, let me bring the, the font back up. By the way, if I do that stuff, please, please interrupt me and let me know that, hey, increase the font.
We've got the data frame. I want to change, and this kind of goes to what Ed was saying about data types.
I want to change this filing date which is originally a string. Here's the following. Here's the filing date.
This is just a string. I'm gonna turn that into an actual date object. Act more correctly, a date time object.
I'm then going to extract the gear. I'm going to calculate the count. In other words, how many?
Entities are there per year. Sort it by the year and then plot it as a bar plot so For McDonald's.
Oh, G, how do I make this full screen?
Okay. Okay.
Yeah.
Yes, F 11. That's a lot of things. Anyway, Hmm. Yeah, as you can tell, I don't, I'm on a Windows machine and I don't do Windows very often.
But basically we've
That still works. Sure.
Okay.
Is, is it full screen now for you guys?
No, nothing change.
No.
Nothing's changed. Oh, did my
Did my zoom just die?
Taken out of full screen, it may actually make a new virtual screen that is full screen and display that.
While we're still looking at the other non full screen.
I worry that my zoom just died on my Windows machine. Says meeting disconnected. Hold on.
This is why I connect to audio via my phone.
And share the screen on my laptop.
Well, if you're using Linux for a host, you probably wouldn't die.
You're probably right.
I'll put in that request. I'm viewing it on my Chromebook here on the left hand side too.
So yeah, so I'm connected via my phone. My work laptop and my own personal Chromebook.
I'm still not seeing full screen.
So, all right, let's hit that F 11 again. There we go.
There you go.
There you go. I see it now. Thank you.
Alright, so here's that one-liner. I know Python is not supposed to have one liners.
It's supposed, you know, that's a pearl thing. Well, anyway, here's a way to do it in one line and in Python.
You, in essence, you're doing method chaining going from left to right. You get a data frame, you convert that one font that one column into a date, you then operate on that date, you calculate the number of counts in there, you sort it, you then plot it and you specify that you want a bar plot.
And so this is basically the number of forms or number of filings that McDonald's has done for every single year.
Right. We can also look at What kind of forms they filed over the past 20 years and their accounts.
Chuck Chuck chug. There we go. So there's a form for they have most of them, which are most of them, over 500, and then it kind of trickles down from there.
Right? We can also find all the 10 queues. And we can look at the 10, there's you go, there are the 10 queues, we've got 30 of them.
And here we go again with the filtering. Basically I am saying I want the form or I want
I want all the records where the form equals 10 Q. That is my filter. And it doesn't show it as clearly as well.
This is now the entire filter, right? From 0 to 1,000. And a majority of them are false, right, except for 30 of them.
If I filter on itself and say size, I get the 30. If I then actually use that filter on the data frame, it shows me those 30.
Right? So here, here they are. So these are all 30, 10, queues.
I forgot what we're looking for already. Yeah, thank you. I can also. Do some bullying, operations, right?
I can do an and. So not only do I want forms that are 10 Q, but I also want it where the date starts with 2,023.
Notice in this case I'm not doing a date operation. I'm just basically saying, hey, does it begin with 2,023?
I'm treating as a string. Again, I'm looking at the filters. This time I only have 3.
And since what I get back when I do a filter on a data frame is also a data frame. I can do other things.
I could do other data frame type of operations on. So in this case I'm doing a transpose.
So I'm actually flipping things around so that the rows become columns and columns become rows.
So now I have the accession number the filing date, the report date, and so on and so forth for each of the records which are now numbered as columns.
So, We see, this is where I wanted to see just the, instead of a tenk, I wanted a, excuse me, a ten-Q, I wanted a 10 K of leave.
Here we go. Here's the data frame for just a 10 K. Makes sense. 10 K or annual reports.
I should only expect one. And here in this case, I'm saying, okay, give me the tenk for 2,023.
I transpose it again. I do something where I say, well, instead of as a data frame, give it to me as a data series.
And I'll talk about that in a second. And I can then go out and grab some extra information.
I can say, hey, give me the accession number and the primary document. It pulls those out from the data frame.
I can go out to the I can then use that information to go back out to the SEC website and pull out the actual tenk report as an HTML.
Document, right? And here it is. Whoops. I need to actually run it 1 s.
Boom. Do I have it? Yes. Get it.
Great. Run it. Fetch it. Display it.
Oh whoops, didn't grab it. But it's it's in the it's I need to do some other stuff to actually get the contents of the HTML file.
So that's a quick example of going through and getting JSON, it into a data frame, using it to query to get other information and then going back out to the website and pulling information.
Alright, so what are we done so far? Well, in essence, what Pandas is great for is for, how would you say the ETL process, right?
So extract, transform, and load. Etl has been around for a long time. That's a very common acronym.
But my preference is is to use. Well, how would you say more common terms, if you will.
Extract is nothing more than a fetch or a pole or a get or somehow getting data from someplace else and bringing it in, right?
That is that is the extract part. The transform part is, well, what would now be considered data wrangling or aggregation or modification or somehow tweaking the data, coercing it into the form that you actually want.
And the last part load is a push or a save or a right or putting it someplace else so that it's what you say more easily workable right so that that is the ETL.
So I want to talk a little bit about the transform part within Python, right, or within pandas, right?
So. That's good. I've got too many sessions going. Let me kill off the other sessions.
Goodbye.
And they say Windows doesn't complain about too many sessions.
Check that.
Yeah. This is actually Google complaining because, Yeah, it's, I would just say all of this is running on their,
Generosity of a line leader use their systems, right? So.
So what is what is the Google co-lab cost? I mean, our, our average peons like us.
Able to use it for free.
Yeah. Yeah. Yeah. You just go to, I don't know if this URL is readable, but it's basically says collab.
Research. Google. Com right now.
Given that it's Google and given that you know there's a bunch of other stuff going on with it, how would you say it doesn't always work?
You there are definitely restrictions. There's no guaranteed up time, dot, dot, dot, right?
But for majority of the time, you know, 99% of the time it works perfectly for my needs.
When it does it, I've got my Chromebook and my Windows machine and I've always got a Google code or excuse me, GitHub code spaces where I can fire stuff up on.
And at worst, I go to Amazon or any of the cloud providers and just fire up a a VM and I'm good to go.
So, here again, what we're doing is we are loading in pandas and This is how you create a blank data frame.
So let's let's create a data frame and then start. Putting some fake data in it and playing around with it, right?
And the easiest way to do this is to create a blank data frame. There's nothing in here.
No rows, no columns. We can see the data type is indeed a data frame. Let's add some data.
And what I'm doing here is I'm basically saying Create a column A and add in this random.
Data using Numpy. Numpy has a methods that will generate numbers or a series of numbers.
And in essence, what I'm doing is I'm saying I want the mean to be 10.
I want the standard deviation to be one and I want a million of these. Right? And Far it up and boom here we go we got a million rows in what was that probably half a second right we could look at the shape That's one of the things we can do within Pandas we can interrogate the metadata of the data frame.
So here we have a meta, 10,000. So So, let's make it bigger.
So what I'm going to do is I'm going to iterate. I'm going to create a loop over all the letters in the alphabet.
And, basically.
Create a million rows for each one of them And boom, there we go. I now have a data frame that is 1 million rows and 26 columns.
Right? So some of the things we can interrogate, we have already seen shaped. We've already seen size.
Shape gives you the number of rows and the number of columns. Size gives you the total number of entities that exist in the entire data frame, right?
So in essence, it's the number of rows times the number of columns, right? If we want to see it nicely, we can use what is called FSF formatting.
So we have 26 million. We can do an info which we've seen before. Again, info shows you how many entries there are, what data type it is, the name of the columns and how many are non-null and so forth, right?
We can also ask it, hey, data frame. What are the names of your columns? And when we comes back, it says ABC, D, EFG, HI, etc.
Right? We could also ask it what are the indices of the roads. And it tells us that it starts at 0, it stops at a million, and its steps is one.
And yes, you can re-index things. Both on the columns and on the rows. We can also run some quick summary statistics.
Right, so I just say, hey, Run, describe. It goes through all 26 million and quickly gives me account a mean, a standard deviation, the min and max, and then in any different quartiles, right?
2515, 75%. So we now have the median here, which is the 15%, as well as a mean.
Alright, since the output of a describe is also a data frame, I've already think I've mentioned this before, I can do a transpose.
For me, I like this method. Not, earnest, the way it displays better. If I transpose it, I have all my statistics at the top.
And all the columns from my data frame on the left hand side. And when I do that, I can see the the output as well.
Right, same thing. I've got the count that mean the standard deviation, so forth. What's also nice about this, and if we get to it, I'll show it.
You can then do calculations on this. So you can actually calculate, for example, the coefficient of variation where you can simply take the mean and divided by the standard deviation added to this data frame and boom you've got that in one shot as well.
Right. One thing I find annoying About Pandas is that when it gives me account, it gives me account of dot 0.
I don't know why. Why would you count with a floating point number? I don't know.
That's what they do. But what's cool is I can simply say, you know what, recast that as an integer and when I do it it does that happily right so I calculated and now I actually get a number.
All right, any questions so far in creating a data frame and interrogating it?
Looks almost like it's true.
Alright. But let's go into.
Oh, I couldn't hear that. Say that again.
Sorry, Robert, it almost looks too good to be true.
It gets betterly.
Yeah.
And so. What else? Can you do? Well, obviously if you've got a table and you've got rows, you've got columns, what do you want to do?
Well, You want to select the columns, right? You don't, sometimes you don't need all the columns or you want to do things with the columns.
And so therefore you want to select them first and then work with them, right? And the way you do that, one of many ways, and I'm not gonna cover them all because, I already realize I'm running short of time.
I can simply give it a list of the names of the columns. So for example, if I want just From my data frame just columns A and BI can say here data frame within square brackets.
We're actually more correctly 2 square brackets because the inside of the square brackets is a list.
It now gives me back a data frame and B. Right. And the order matters in which it in which you put it, right?
So if I say BA. It comes back in that order. The data frame now shows me B and A instead of A and B, right?
I can also have multiple This the duplicates of column names so I could have a BBA And if I run that, I now have those 4 columns, right?
We see 10. The, the row A, or excuse me, the column A here on the left is the same as the column and the right and same with the B, right?
I
So it's always it's always referencing the first data frame you created, right? The 26 million roles.
Yeah, because because I've signed it, I've assigned that to the to the variable DF.
Oh, okay. Okay. Yeah.
And yes, I can have multiple data frames, right? Which is, which is what we saw with, with the call center, right?
I've, call center with the phone book and the call logs, right? I can have 2 different variable names.
Yeah, 2 different data frames with 2 different variable names.
Okay.
Yeah, so that's that's how I'm referencing it. Right. And of course, since the inside of this is a list, I can create a variable that's a list.
I'm creating this variable called calls and I'm taking this ABBA, turning it into a list and using that.
As my, what I want for my columns. And when I do that, I get the same result.
Right. I have A BBA. Another way to select rows is I can simply select a random sample.
I can come in and say, you know what, here's my data frame. I can come in and say, you know what? Here's my data frame.
There's a sample method and I specify, specify I want 5 columns. The access, which is You know, talking about these strange terms and in pandas, they, they use axis.
I would prefer dimensions, but. Is what it is. Hi, specify columns and I can give it a seed.
This is the random. They call it random state. This is your traditional seed number for when you for pseudo random number generators.
And when I do this, it comes back and gives me 5 randomly picked columns, in this case IQ A YL.
Right. So pick those up. Randomly. An important note.
If you want a data frame back, you have to give it a list. If you give it a single column.
That is a scalar instead of a list, you will get back a series instead of a data frame.
Right, so there is a series. With single single square brackets. Here is a data frame with 2 square brackets.
Other advantages to using to getting back to series, I'm not going to get into it now, but.
Keep that in mind when, when, when you Don't put 2 square braces.
Next to your data frame variable. Right? Just like you can select columns, you can select rows.
Right, again. How does it know that you want a row versus a column? You actually have to use this extra method called I look.
Just the way it is with pandas. I wish it was different, but it is what it is.
And that stands for index locator. However, when you do that and you give it a list of numbers, It will use that as the index of what you want.
And just like in columns, you can specify this more the same road more than once. So in this case I'm looking for 1 2 2 one and it comes back as 1 2 2 one so this 1 2 2 one are the indices for the rows Just like with columns, I can give it a list.
And assign that to a variable and use that variable for selecting things. So here I'm selecting, I'm using the range operator to generate the numbers one and 2.
And it comes back and selects only rows one and 2. And again, just like in columns, I can specify a random sample.
This time instead of specifying the access as columns, I specify the axis as rows. And it comes back and randomly picks the 5 rows.
All right. Other things that you can do with rows, you can slice them. This is the nomenclature for slicing.
It's actually 2 numbers to be explicit. You give it a start, you give it an end.
If you are familiar with Python, lists, this is exactly the same. The start is inclusive, the end is exclusive and it it gives you whatever's in between the 2, right?
The
So if you If you specify 0 colon 5, you will get the numbers 0 through 4. And it works like that.
And also just like in LIST, you can count backwards. You can say, I want, I want to start from the left side.
I want the last, the last row in the data frame and you can get that as well. Right.
For you Unix folks, this is head 5. This is tail dash n minus one.
Or no. Yeah. Or tail end my tail and one. Right.
Just like in columns if you you know, we've been giving it lists If you give it. A single value, a scalar that is, you will get back a data series.
So if you specify DFI log 0, you will get back the first row. Has a data series.
Right? In order to get it back at the data frame. Do you square braces?
There you go. Alright, now. Expanding on what what Gary was asking about for filtering.
Here's my Boolean. I want a conditional. I'm basically saying, hey, we're column A, wherever the value is less than 6, I'm going to sign that to my variable called filter. It can be anything.
I like filter because well it tells me what it's doing. And if I look at that, I get back a data series of true and false values.
In this case, most of them are false. If I use that. On itself. This if I use the the variable and square brace itself, it gives me back all the rows or all the entries where that value is true.
So in this case, we got a bunch of true values. Not the hardware store.
I can then use that filter on the data frame.
And because it's a data frame. In other words, since this left part here that I'm highlighting is a data frame.
I can then use square braces and give it a list of, list of columns. And I can specify just those columns.
Oops. What happened?
Oh, I don't think I ran my filter. Hold on. Do, do, do. There we go.
For speed, I ran this notebook. Several times and I forgot to clear it. So we've this, we've got a data frame and now we just want these columns.
So this is a way to select just select both rows and columns for what I want. That mean my criterion.
That is one example. Any questions on selecting rows?
All right, we'll do a quick, we'll do a quick one of Rose and columns.
Again. I can switch that around. I can specify the columns that I want and then use ILOC and give it a list of rows and it will return just those rows and columns.
They'll do that, be quick. Any questions on any of that?
Will this be on the test?
Yes, it will be on the test.
Okay.
Alright. Now let's play with some other fun stuff. This is So far, everything that I've caught, I've talked about is stuff that you can, you can probably do with Sequel Light.
Or any sequel languages and and play with stuff, right? But I wanted to show you a couple of other cool things.
That you probably can do with some versions of SQL but not others. And this is where you can do things called exploding and nesting and unnesting and and things like that.
So Do do. Let's bring in pandas. I don't remember why I have marked down, bought in, but I have it.
I'm gonna create a quick data frame. This is another way to create a data frame. You can create it using a data frame.
You can create it using a dictionary where the key is your column. You can create it using a dictionary where the key is your column name and your key is your column name and your value is a list of the values that you want.
It has to be a list of the values that you want. It has to be a single item for the key and a list of the values that you want.
It has to be a single item for the for the key and a list or some kind of collection for the for the key and a list or some kind of collection for the for the key and a list or some kind of collection.
And a list or some kind of collection for the For example, a Numpy array as as the value and when you do that and say hey turn it into a data frame or turn that dictionary into a data frame, it will do that happily.
Now notice what we have in this data frame. We've got an ID. Randomly generated and we have fruit that actually has Multiple values in it, right?
This is. This is not good normalized database design, right? But you see this all the time.
Right. And so how can we break this up? Well, one way to do that is there's a function called get dummies.
Again, why did they choose that name? I don't know. In statistics, they often call it one hot encoding.
Which is a way to do this. And, and in essence what it does is it takes that column, splits it out, and turns those into individual columns.
So before up here where we had for fruit, apple, orange, apple, and dragon fruit and orange.
When we say get dummies and specify comma, comma space, it breaks those out into individual columns now. Right?
So we now have Apple one, Dragon Fruit 0, Orange One, and so on and so forth. Right?
So that's one way to break things out. And what you can then do is you can then say, okay, great.
Break it out and give me the sum. Right? And by doing this, we can now see that the app, there's one Apple, one Dragon F, 2 orges and so forth, right?
So this is the advantage of breaking it out or breaking it out this way. we can then go through and, How would you say do operations on it that we normally otherwise would be able to do, right?
So in this kind of data set, it would be kind of, would be more challenging to figure out, well, how many, how many oranges are there, right?
How many, how many, you know, we've got 2 of them, how many apples are there, right, and so on and so forth.
By exploding it out or more correctly, one hot encoding it, we can get that information.
But that's not the only way to do it. And I'll show you that in a minute.
Once we've exploded it out, we, do the One Hot encoding or the get dummies, we can then join it back in.
Now, earlier when I was talking about the example with the with the call logs, I use the merge function.
And here I'm using the joint or the merge method and now I'm using the join method.
They basically do the same thing, but they do it in different ways. The merge operation is much more.
Would you say flexible? You can specify what columns you want to join on and how they get joined.
As in, you know, you want it to be greater than less than equal, you know, left to one right through and all that kind of fun, equal, you know, left to one right you and all that kind of fun stuff.
With the joint operation, it uses the indices. Of the 2 data frames. Right, so the it only uses indices and matches on indices.
So the advantage of using join, it's much faster, at least in my experience, than doing a merge.
But, but in essence, that's how they're, they both combined. Data sets together or data frames together, they just do it in different ways.
So here we're doing the join. Here's the original data frame with the ID and the fruit with the fruit values being nested.
Here it is, on, and here on the right side we have it where all the fruit has been one hot and coated.
Right? And lastly, what we can do is we can then drop the fruit now that we've broken it out.
Let me,
I see that works. That doesn't work on. I forgot to run these things as we're going along.
Let me just run again. Wow. Alright. So in this case, what we've done is we've dropped the fruit column.
This is one way to do it. Another way to do it is to drop it right away and then do the join.
Because it's joining off the off the axes.
That's one way to get these things out. Another way to do this
If we look at fruit.
If we do a split, a split function returns a list. Right. And so instead of, Now instead of text, we actually have a list which actually has individual elements inside it.
Right? And what we can do with that. Let's add it back in. There's our fruit.
We could do something called an explode, right? So we can. Take that, take the fruit.
Let's talk.
Split it out and we can explode the fruit and what that does is instead of taking, instead of creating columns, it creates duplicate rows.
So now notice we have ID here with Apple. The same ID has orange. And the same ID has Apple again.
Right. Past 13 has dragon fruit and orange and so forth. Right. So we now have a series of rows.
That have been duplicated. Right. Or at least part of the row has been duplicated, with, with that information.
So one of the things we can do is we can then Drop that. We could do a crosstab.
Which in essence what it does now does is it takes that fruit column and breaks it out. And by default what it does is it adds things up together
So we get the same information again.
Actually, we get almost the same information.
I know. Hey, what's up?
Alright, what it does by default is it adds things up so we almost get the same answers and I'll show you that.
Let's go back in a second and show you that.
And then what we do is we rename, we, how would you say creates a bunch of indices, we clean those up.
We do our join again to put it back in with our original column. So the end result If we were to drop this fruit column would be the same.
We basically exploded out the the fruits into their own individual columns. Now one thing that is different in this case from what from before is we're
I'm trying to find it. Oh yes, here we go. When we do our get dummies, Notice this one here.
We actually have 2 apples in that first row. But we only get one when we do the get dummies.
And the reason why is because the one hot encoding is simply a binary. Do you have it or do you don't?
Very much like a set operation. Duplicates are not counted. So this is different ways of, as you say, doing almost the same thing and getting slightly different, slightly different results.
But what is cool is that you can take data that's columnar and turn it into rows and vice versa.
Right? You can explode things out.
Alright, any questions on that on that stuff of how to manipulate data from rows to columns and columns to roads?
Basically pivots and cross tabs.
I'm curious how you would, you have used this. In production. The music, often use it to clean up data or what?
Absolutely. So for example, A common thing is, you know, here we've got apples, Dragon Fruits, orange, others, watermelon, right?
They're each their own individual columns. It would be nice to go through and say, you how many
Gosh,
How many fruits are there?
Right? If you could break this out.
Right. Individually, right? We could then count how many fruits there are. Let me just go back up here.
That, where's my, here we go. Right. By breaking it out like this, by doing an explode, we know now how many fruits there are in total.
Which is something we didn't know before.
Right. Or we, we didn't have an easy way of calculating. Right? We know how many rows there are, but some rows have 3, summers have 2, Some rows have one, some rows may have none.
Right? There's no easy way to get that. The only way to get that would be to explode it out.
And then count that. So that's a reason why you'd want to, explode things out.
Yeah, it's fine. Thank you.
Does that make sense? Stan? Yeah.
Very common in financial records where you'll have something where it has like some information and then it'll have like year one year 2 year 3 or 4 year 5 and That's nice for display purposes, but for actual database kind of stuff.
You actually want those each. As, as individual. entities, if you will.
You'd like to have a year and then whatever they're, whatever they're measuring.
So just different ways of transforming data. There is a ton of other stuff that we can do with this.
There's type conversions. There's Another, we can go from, from integers to flows from floats to integers from dates or from strings to dates from strings to other stuff we can use regular expressions to parse things out.
All kinds of fun stuff. You can do with with pandas operations. Right. So that's that's a quick thing of of the transformation part.
Any questions on transformation before we start talking about ingesting or getting data?
Probably not a distinction, but this might be a good. You've used the term one hot encoding.
Yes.
Is there a definition for that term?
Yeah.
There probably is. It's a common term used by statisticians for basically saying if I've got a bunch of categorical variables, I want to turn them into their own individual features.
And then I can use that to do statistics on it. But good question. Well, look it up later.
Okay.
I'll make a note of it. Somehow make a note of it and we'll come back to it.
For what exactly one hot code it is. A bigger question that I have is is why why is Pantas calling it get dummies?
So you know, what, you know, if one hot is such a common thing in statistics.
Who came up with Get Dummies and why? But, yeah, that's, that's probably a, you know, conversation to have over beer at a Python conference.
They good? Microsoft?
Or a Pandas conference or pi data conference or something like that. So. Yeah.
Robert, I know who did it. What large company, it always wants to do things pass backwards. It must have been Microsoft.
And entirely possible. And entirely possible.
Oh, that's hilarious.
So.
Doesn't, Guido work at, Microsoft?
He works it, he worked at Google. I don't know where Guito works now. That said, Guido is not responsible for pandas.
West, West McKinney is. And yeah, just a quick brief history about, about pandas.
Data frames evolved from statistical software. Originally, the S programming language apparently was one of the first that had it, which came out of Bell Labs.
And, surely afterwards. I'll shoot the guy's name slips in my mind, but he wanted an open source version of the package of S.
As we look to see if we are speaker from the
So he created the language R. Go figure. Could have called it anything, but I guess our, was at the top of his list.
And, And so it started having. You say what was initially, they weren't called data frame, they were called something else.
But it basically had some kind of tabular, structure so that you can work with things.
And rows and columns. And then the early 2,000, Wes McKinney was working at a financial services company.
He wanted to do a bunch of manipulations. He really liked Pandas, excuse me, he really like Python.
And so he looked at our and said, hey, these data frames are really nice. Let's do the same thing in Pandas.
And you could you could kind of say the rest from there is history. There's a lot of things you can do with with with data frames.
Including ingesting material from a lot of other places, right? So we already started talking a little bit about some of the stuff.
So a little bit of repetition here, but I do want to click on this guy.
And we'll get into that a little bit. So. Someone was talking about CSV files that are on a website.
You do not have to download it first and then read it in. You can literally give pandas. A URL.
And say go out there, fetch it. And reed in. And here's an example of Reading in the, some data from the NOAA.
National oceanic and atmospheric administration. They got a nice. Short small data set.
Literally what this just went out fetched the data read it into a panda's data frame and we're looking at the data frame now, right?
We can look at the shape, right? It's 3, 3 rows and 9 columns. That's not the only, and you know, we could do it with that.
I've also got a data set called the Home Data, which has a bunch of home prices in it.
Again, it's on Amazon. Go out, go fetch it, bring it back. We're looking at the first 5 rows in this case.
And it's got 81 columns in it, right? The entire data set is 1,460 rows and 81 columns.
And what's nice about getting it as a data set, doubt from the web is we've pulled it in as a CSV.
You know, the simple thing is to say, well, GM is why don't you just save it as the CSV and then load it in, you can do that.
But if you've already loaded into a Pandas data frame, you can then say, hey, dump this as a CSV file.
And there we go. Okay, and if you do LS. Voila? There it is.
Home. Some information about the home, our data frame. We've got all this stuff in here.
It's objects and integers and blah blah blah. Right, so it's a fairly big.
We can create a histogram out of this. Oops. Maybe we can't because I didn't do something right.
Wholesale. Oh, I must have, I must have deleted that.
Shoot, I deleted something inadvertently.
Whoops. All right. Well, anyway. I don't know what I deleted, but, oh, there was another data set that I brought in that I, sorry about that.
Yeah. In addition to bringing things in as CSV files, we can also bring things in from Excel.
So I've got a data set out there in Excel. I can read a data set or Excel files are actually called workbooks.
Workbooks can have worksheets and pannas can can read an individual worksheet and load it in as long as it's formatted in the right way, right?
So. Rows and columns and kind of stuff, right? So here's a simple example. I selected sheet one.
I've got animal age in color. What I can also do is in a given file, it could have multiple sheets.
Here I'm reading in the other. the other sheets, if you will, assigning them to other data frames.
So here I've got animal age color. I've also got animal name and sound. Right.
I could also do it as a loop. Right. Where what I do is I say, okay, great.
My sheets are one and 2. And then with with the Excel file, I can read it into a Addictionary in this case and I can then say, okay, great, select sheet 2.
Which is a dictionary, my data, the data variable is actually a dictionary, but the key points now to that is the value of that of that key is an actual data frame.
So here I actually have a collection. Of data frames. Right? If we look at the keys.
There we go, sheet one and sheet 2 and there's the data from sheet one.
Any questions about reading in Excel files?
Alright, next one. Slightly different version. There are tables.
In websites, right? And those tables often have interesting information in them. Right? For example, here is the URL to the Wikipedia page for New Mexico.
Right? Pandas can go out to a website. Grab the HTML and get a list of all the tables.
It will create literally a list.
And what's inside those lists? Well, the values of all the elements in the list are data frames.
Right, so in this case I went out and grabbed all the all the tables in New Mexico or excuse me, I went to the New Mexico Wikipedia site.
Eat as in pandas went through found all the tables and brought it into a list. Of tables.
I can look at table number 5 as a data frame and there it is.
That is literally all I needed to do. Go to the Wikipedia page. And boom, I've got I've got a bunch of I've now got a collection of data frames.
Each one of those data frames has columns. So I can say, okay, great, go through the list and for every single table, print me a list of the columns.
Now, This is not very readable, but nevertheless, if there's something in particular that you're looking for, you can quickly go through and probably find what you're looking for to narrow down some of the tables that you're interested in.
For example, I may be interested in knowing what the language is spoken in a particular that are in New Mexico and there happens to be, let me see if I can find it now.
One of these says, oh, here we go. Here is a table where one of the columns says, language is spoken.
It has this text in it. And it will basically, this particular piece will basically say look for any columns that match the pattern language spoken and only bring those in.
So again, what this is doing, the read HTML is going out to the HTML page.
Sloping it in, finding all the tables, looking for those that match and create a list of those.
And when I do that, I only get one table and so therefore Here's the list of the table.
And I can now say, great, add that to a New Mexico data frame. And here is the New Mexico data frame.
Right, so these are the languages spoken and the percentages, right? English only, Spanish, Navajo, and others.
So pretty neat that all you need to do is point it at a website that has tables and it will pull out those those tables into a list of data frames.
Any questions on ingesting or what is called extraction, the extraction part of getting data into a data frame.
Does it deal with so? There is a character sets. But they, yes.
Yes Yes, there's an option where you can put it in when you say read. You can specify the character set that you want.
Yes.
By default, it's UTF 8, but if you want it to be something else, you can say, I forgot what it's called.
Almost wanna say it's a charge set or something like that. But yes, it can deal with multiple, you know, different kinds of character sets.
And by the way, I just showed you a quick thing of ingesting. It will read CSVs.
It will read JSON. It will read HTML. We covered about reading Excel.
But it will read a bunch of other different types of file formats or or connectors I guess would maybe even a better format.
An interesting one, interesting story, is this one here, read SPSS. SPSS is the statistical package for the social sciences.
One of my students was working on a project for his capstone and he met up with a research group that was doing a bunch of stuff on traumatic brain injury and he thought that well that would be cool to look at and so he said hey can I get some of your data and they said sure here you go and they give them to it as an SPSS file.
And, that's not something we actually cover in the class, but. It wasn't hard at all.
I mean, literally instead of changing, you know, read CSV, he put in read SPSS and it literally just read the entire data set right into a data frame and he was, you know, rocking and rolling from there.
So lots of things that that can be read directly into a data frame. Other things can be done indirectly.
For example, Pandas doesn't read in PDF files, but Python has some really nice tools.
For extracting data out of PDF files. And so you can get it from a PDF into text of some format and then from there you can then read it into pandas and extract the information.
So and on the flip side is it can do a lot of exporting stuff, right? So that that last part that load or I'd like to call a push or export or write it can write out to all these different formats as well.
HDF Feather, Park K or X Data. Look at first JSON. Yeah, there go JSON as well.
Right. So, and as an example of that, I'll show you how it works with with Sequel Light since we mentioned that earlier.
Don't install SQL Light on Google Kolab. I've got too many sessions running again, Boom.
Rock and roll. The first part is just it basically gets sequel light set up in, in how you say installs or, create some tables and what have you.
And so I'm gonna kind of skip through that further quickly and then show you how.
You Once you've once you've read the data from a SQL like database into pandas you can then make some changes and then send it back to Sequel Light as well.
So this is just showing that Sequel Light is installed. I've also got the insane dictionary.
Of words. I've grab a few of those, parse it out. And bring it into directly loaded into SQL Light.
So. Here we go. So this is basically a SQL query.
Select everything from words to show me the first 10 words. Here they are. I can view the schema.
There it is. I can now take that and create another. Another one where I have a key.
Do a few other modifications, boom, simple lights, equal light, sequel light. All right, so now let's use it with pandas.
So I'm gonna import the Sequel light module as a dB. I'm gonna create a connection.
That's what this line does. I'm gonna create a query.
I then send that query. To the database connection, which in this case is Sequel Light.
And read the results. Right? And put it into a data frame called words. And when I do that, Voila.
The same information that I have in Sequel Light, I've now pulled it into pandas and I can work with it.
Do other things. We could look at what the what the data set looks like. Notice that it is everything is an object.
And by the way, in pandas, it doesn't have a string data type. If it's not a integer or a float.
It's an object. Regardless of whether it's a string or a list or whatever, it just says object.
Right. So that was one table. The other table has a slight change.
It actually has an index on it. And it has a different data type. The data type first word one is actually a number.
And so now if we look at the info that we pulled in of this data that we pulled in from Sequel Light, notice that the data type is preserved.
In other words, the data type that was in the database actually made it across into Pandas and says, aah, I know that's an integer, you will be an integer.
Now, how many of that actually works? I haven't done an extensive test. I do know Int work floats work.
Objects, everything else is pretty much an object. And, and that is also, that is very much true for Sequel Light.
I haven't tried it with some other databases of what what data types are supported for bringing over the wire and which aren't.
But something something in my queue of things to do. Right. So.
Here we go. There's our data frame. And what we can do, these are just the first 10 words.
I'm looking for something. Oh, I guess they don't make the change. Oh!
I don't make the change. All I've done is I've read the data from a sequel light table into data into the data frame and now I'm going to send it back out.
To sequel light.
And if that table, this is some of the options you can put in, this is the name of the table that you want to create.
No, that's not what I want to do. This is the database. Connection and what to do if that table already exists and in this case I'm going to replace it.
Boom, it does that. It works and it tells me how many records were sent over. If I look at the schema now in SQL Light 3.
This is the original data set. That I had. I then modified it within Sequel Light to create, 2 things.
Number one is I turned it to turn this field into an integer. Also made it a primary key.
And I also added an extra index on the word 7. Or see me added an index. Yes, added an extra index to the to the column called word 7.
This is the one that came from. from Pandas. Notice we saved the integer.
But we no longer have it as a primary key. That number one. Number 2 is we have this extra thing called an index.
And I'll talk about that in a second. And we also have this thing that says create an index.
On World Real 2. Excuse me, on.
On index. That's what I'm looking for. Right. So it's creating a new index.
So basically, we lost this as an index and we lost this as an index when we brought it over into into Pandas.
So this pan is perfect, absolutely not, but you can do a lot of cool things with it. And there are ways to, there are workarounds for dealing with some of the stuff that doesn't work perfectly.
Any questions on exporting this stuff? Out of pandas into other formats.
Now to just get Django to send me the file.
I might be using pandas to to display and render. Reports inside of the
And people wanted to be able to download it as an Excel file.
And.
Yes. And Panda isn't the only way, right? There there's a native Python package for creating Excel files, which offers a lot more flexibility if you need it, right?
So if you need to worry about things like fonts and, you know, you know, all the other, what you want to call it, attributes that that a cell may have.
Or a worksheet may have, you would want to use that. Pandas is great for just reading stuff in and generating stuff.
Fairly quickly and easily.
Alright.
Robert, I think you've, more previous conversations. You've done quite a bit of stuff and are.
And as you were saying, this Seems like a lot of things that I've done in R.
Do a Is this an improvement over RA. Quick and dirty version of what things can be done from our What's your comparison?
I mean, our language.
Our language, yeah.
I how to how to phrase this.
Our is a wonderful language. For doing statistics. And machine learning and those kinds of things. The the challenge with our is That's what it's for, right?
On the flip side, pandas is a much more general purpose. Programming language that is easily extensible.
Via modules, right? So I can take the data from pandas now and and do things like machine learning and if I want to I could also have it go through and, you know.
I I could use modules to have it do. Infrastructure for me where I may want to run my machine learning modules, right?
Doing that in our Not that it can't be done. It's, that's not what it's.
Purpose is for our right so Yeah, that's it's it's almost like ours is a bit domain specific.
In the sense of it's it's just you know it was originally designed by statisticians for statisticians right and that's that's pretty much It's niche.
That said, there there are things that you can do in R that you At least that I'm not aware of the how that you can do in Pandas, right?
There's some, you know, some special statistics that It's not that you can't do it in Python yet.
It's I haven't seen those things done in in Python, right? So, but I wouldn't be surprised if someone, you know, along the lines who actually needs a module that does something that our does, we'll look at and go, what do you, is I can just write that in Python and create a module, put it on Python and boom, everybody's got it.
right? So, yeah. Absolutely. If you want to do this in our, yeah, we can we can do this in our.
From a personal perspective, I'm not a big fan of our syntax. But that's just, that's just me with our, So.
In your language.
It's another language. That's all.
Thank you. Thank you. Good distinctions. I like it.
Like this like the description.
10 min warning.
10 min warning. Alright, cool. Let me just do something real quick. I just I just wanted to show.
Let me see if I can fire this up real quick. You know, I promised, you know, one of the things you can do is fire up.
Where are you? There we go. Is that you can use a data frame to do some machine learning.
So let me just walk you through real quick an example of using a data frame for machine learning. This is a data set called the Boston Housing data set.
It's got a bunch of information about houses in Boston and Hey, we are loading that data into a data frame.
So here's the CSV file. It reads it. Yeah, one of the things you can do is you can specify in a data frame.
The index to use and I'm specifying a 0 is the index. Like I said, there's a bunch of options that you can pass into into a read CSV, including, you know, the character set, whether how many columns to skip, whether it's got headers or not, blah, blah, blah, right.
So here's the, here's the first 5 from the data set. We've got a shape here.
490 rows by 13 columns, the total size, 6,000 some. This is what the data looks like.
Notice everything is a float. Right, it's it's floating point. So that makes things really nice Especially for data science.
Cause, excuse me, from machine learning because every, oh, here, talk, actually going back to your point, Gary.
One of the nice features of our is many of its machine learning models that I'm aware of. There's probably many others that I'm not aware of.
Most of the ones that I've used in our You don't have to worry about some of the data data types that you give it.
And Pandas on the other hand, everything has to be a number. So if you've got categorical data that's written as text, you need to convert it into a number.
One hot encoding, ranking something to convert it into a number. So in this particular case all of our number all of our columns are numbers we've got the count mean and so forth some descriptive statistics so let's create a random force out of this thing.
A random force is a decision tree. I should say a forest, so many decision trees, that allow you to do both regression as in, you know, some kind of a linear kind of combination of things as well as categorizing or more correctly classifying different things.
So it's literally a tree. You're saying, hey, given this, ask it a question is a bigger or less than some value.
If it's greater than goes this way, if it's less, then it goes this way, and then it asks another question, and that's how it tries to segregate things out.
What we're going to do here is we're going to take that data frame and split out the actual what is called the target and just look at the features of the things and we're going to try to predict what the meeting home value is, with this, particular decision tree.
Okay.
We're gonna do that with the 5 basic steps of machine learning. We're going to first break.
Or what is called a train test split, we're going to split the data set into a training set and a testing set.
We're going to use the 800, and 20, so 80% will be training. 20% will be testing testing.
We're going to do a random forest model. These are just. Values that I picked to get it going.
I'm, yeah, great. 1010, 10 levels, if you will.
We're going to fit the model using the training set. So we're gonna, this is what's called training.
We're fitting the model. We're then going to use that model. And along with the test set features to create a prediction and then we're going to see, okay, great, we know what the answer is, what is the prediction?
How close are they? Right? That didn't work. We're going to look for what is called the mean squared of the error.
And then lastly, we're going to square root that to get the the root mean squared, which is a measure of the air.
So here we go. Our RMS in this particular case was 2,854.
What does our model look like? Let's bring in some graphing things. Whoops.
The, model is not defined. Oh no.
Let's get rid of RF. Oh, that can't be right. Yes, it can.
Okay.
There we go. We have 10. Let's display this.
Boom. Here is our decision tree.
Thank you.
Come on, you can go faster than that.
Hey, let's say it's a pretty big decision tree.
Okay.
Boom boom boom boom and one day I'll figure out how to make these graphs so you can actually see them. Actually, you know what?
I'll just do this real quick. Doesn't make it. There we go. That's really good.
So in essence, what we do is we start here. So in essence, what we do is we start here, we ask this question, and if it's true, it goes this way if it's false it goes this way we then ask another question.
And if it's true, it goes down. If it's not false, it goes to the right and so on and so forth.
We so we traverse these trees with a series of questions. This entire tree was basically based on the data that we had in the data frame. Right?
So load the data, do this. Build the tree. And, and go from there, right?
So, One of the things we can do with the decision tree is we can look at, okay, in this, in tree, what are the actual important features?
And it turns out in this particular model, the whatever this column is, perk lower status and num rooms were the biggest predictors for the price of these particular houses.
So just a quick machine learning model to show you how data frames can be used. If you have data in a data frame, you can get your data into data frame, you can then use it for some, some machine learning models.
Okay. Any questions on that? Oops.
Alright.
One last thing. Ron was talking about that Panda has bugs. Absolutely is bugs.
It's a matter of finding them and then, and then, figuring out how to work around them and fix them.
He was kind enough to actually, he actually went in and did a fair amount of work to to deduce where the bug is and provide a not only a workaround but also a solution to it.
This is one of those subtle things of where is the bug is the bug in the documentation.
Where it's says, Hey, delimiter, blah, blah, blah, when there shouldn't really be a delimiter.
It should probably be called something along the lines of field. Strip characters because what it does is if you're given a character if you're giving it a class of characters it will strip those characters off the left and right side of a of a field.
But it's labeled as delimiter. And if you don't specify anything by default, it removes all white space, which is not what is what would be expected behavior in this particular module.
So, the workaround, I think I posted in Slug, but I'm not sure.
In essence, what you can do is you can specify the, the delimiter. Here we go.
Ron's workaround is to use a question mark. My workaround suggested workaround is to use a an end of line character like a backslash and and and the reason why is because you will never have a end of line character in your data set for a record.
It assumes here that every line is a record and so therefore it's already splitting at the end of line so you won't have an end of line in the middle because That's where it gets split.
But yes, there are bugs and pandas. Find them, figure out workarounds, follow bug reports.
And for that run, I think if I know you were on earlier, so. Thanks. Alright, I knew I did a horrible job.
I am I apologize. I was shooting for 30 min. I was expecting maybe I go over to 45 and here we are over an hour later.
My apologies. But thank you for sticking around and listening to me blabber on about about this stuff.
Nothing wrong with ours. Nothing wrong with that.
If you guys have any questions. If you have any questions, please feel free to throw them on discuss.
My emails out there if you want to send them to me directly or or throw them on discussing we can chat about it more there.
So thank you very much for your for your time and patience with me. Any questions?
Many, many.
Oh, fun.
Yeah.
Oh, I literally tried to run that cross merge on this data set that I've been fighting with trying to merge together.
The air is cannot unable to allocate 54.7 GB for an array with this shape of 12.
Bye.
Type object. I'm like Okay, I guess it's the wrong way of doing this.
Yeah.
Yeah, I was good. That was quite question I was gonna ask, but it's about the memory, usage.
Is it like ingesting it all into RAM or what happens there?
Yes.
They based on that result, I would have to say yes.
And I say that with how to say a little bit of a caveat. There is more than one way to run pandas.
In other words, by default, Pandas uses Numpy and you know, how would you say the standard Python features to run?
There are newer tools so that how would you say you don't have to slip everything into memory?
You can use, and I haven't played with any of these, well, I shouldn't say I haven't played with any of them.
I've played with them, but that's about it very minimally. There's pullers, there's arrow.
There's a couple others that I'm slipping that in my mind right now. But in essence, what they do is they, in essence, distribute this stuff over a bunch of different machines and only call it in essence what they do is they in essence distribute this stuff over a bunch of different machines and only call it as needed.
That said, it doesn't necessarily mean that Pandas is doing the most efficient kind of work.
If you do a cross join and then a and then a filter it may be doing exactly that. It's it may be doing all possible combinations and then filtering as opposed to being smart about it.
And doing some common optimization. And if that's the case, I highly recommend doing this in SQL.
Yeah.
And I noticed Grant T mentioned, you know, you know, multi-line cells are a thing. In a lot of the data that I work with.
And yes, there are ways around that. I should have clarified what I was saying. In the in the fixed wheel the fixed width format as far as I can tell it assumes that the entries are one line per record, but there are other formats that that you can slope data into pan is that don't make that assumption and you can specify what those what those.
Other delimiters are for both records as well as fields are. That's one way the other way is to Obviously you can create some kind of a looping structure where what you do is you read in things, you know, file open.
Do stuff with it. Correct, collect it as a record. Then turn that record into a the data frame and then work it work with it from there.
A good example is that that was something that I was working with in sequencing. There's a file format called a fast queue, which where what you have is it's a multi-line record.
You basically have to bring it in first in one data structure and then slope it into a, a data frame.
But it can be done.
So you have to export it, transform it to load it into your data frame before you do other things with it.
Okay.
In that particular case, I believe so there may be better ways to do it, but that requires maybe a little bit of thought and time that I didn't have.
So
Fair enough and consider this more a heckle than an actual criticism.
About multi-line cells.
Well, there's truth in it, but the comment about, you have to do ETL before you can do the ETL.
Okay, fair enough. Yeah. But once you do, and again, for example, using sequencing stuff is if I were to bring that into a data frame first, I can then do all kinds of things like calculate GC content, look for restriction sites, look for all kinds of other features.
That would not be as easily done. Outside of a data frame, right? There are things that I can do that.
How would you say having it organized in a data frame? It simplifies things. It may not be optimal, but, but it does simplify things.
Fair enough.
Huh.
Okay, questions about error. In didn't really, I don't think you talked about that.
Nope, lot of things I didn't talk about. Okay.
Yeah.
But anyway, the There's a certain. Well, a certain city in New Mexico that has a bunch of air quality centers around town and this data happens to be available on the web in a weird file format or weird.
Like tape or I don't know exactly like maybe key values but in a tape tabular format. I'm not quite sure.
Yeah.
But anyway, point being, so one question I have about that is like how easy is it to write one of these connectors, right?
That could, let's say, that in and then turn it into. I think what we did was didn't we put it into like a Python dictionary or something dictionary list of dictionaries or, dictionaries of dictionaries or something.
Something like that.
Yeah, it was actually extremely simple to stream it into a YAML format. Then converted into JSON and then pull it into a data frame.
So.
Right, right. Yeah, we just, okay. But my question, what I was going for was that the, There's dirty data sometimes, right?
Sometimes.
Yeah.
So there's missing fields, right? Or, or, or like the commas. Should be there, but they're not there or you get 2 comes when you didn't really want 2 commas right or this is the bad part is about I don't know mid you know, 2,014 or something like that, they decided to change the damn format, right?
I need to tell you, this is version one is 2. Right? It literally changed the format a little bit.
Right. And my question is, well, what happens when you do get dirty data and does it? I just bomb out or something or what happens?
The hard break or does it do like HTML and say, hey, I don't know what that is, but I know this other subset will just ignore it.
It depends. One of the things that, that I do in my classes, I actually give my students dirty data.
Not really dirty data as dated, you know. It could be much dirtier. Always can be dirty.
But for example, I've got columns that have a text in them that should actually have numbers, right?
So for example, you know, it shows you the Boston dataset that has housing price. I've got a data set that has something like square footage.
And they're every now and then there's text, right? Something like, you know, unknown or something like that instead of instead of leaving a blank so that it can be interpreted as a null.
Someone decided to put some text in there and so what happens in that particular case what Pandas does is it tries to go through and say, OK, great, this looks like a number and and when it hits something that does not meet that, that data type, it goes in and says, oh, you know what?
We're gonna make everything a string and then all of a sudden all your numbers become strings. So that's one thing that Pandas will do with automatically with you.
I'm trying to remember some other things that we're Pandas doesn't go so great.
I think when you have, I'm trying to remember the right term for it, if you have a CSV file that is expecting, let's say, 20 columns and you all shouldn't have a field that has fewer than that or more than that.
I don't remember exactly how it handles it. I think I don't remember the term is is sawtooth or jagged or something like that but basically it doesn't have as many columns as you should have.
And off the top of my head, I don't remember what Pandas does. Yeah, I'll have to look into it.
But yeah, there's all different kinds of dirty data. And sometimes Pandos will will try to coerce it into something that it understands and other times it it doesn't.
Right, it comes back and comes up with an error saying, sorry, I can't read the file or what have you.
There was a question. Help go ahead.
A good example. No, I, I was just gonna add on to that. Actually, this, this happened, this was something that happened recently in, with my last class is for whatever reason the file was a UTF 8 file and every now and then it had just some random characters you know that are like out in left field you know just bizarre stuff And, Yeah.
Python bought it that or excuse me, Pandas booked at that. The quick and dirty way for me to do it was to was to take that data and pipe it through TR to get rid of all the stuff that wasn't, would you say UTF 8 and then, and then read that data in.
I think I was looking into reading into, well, Julius, can I just sell, tell, Pandas to skip it?
And that turned into a bit of a rabbit hole. And so I said, you know what? I'll look into this some other time and and I just looked at it I had a working solution and so I went with it.
Best way to deal with dirty data is to clean the data.
Yeah, don't don't create dirty data to begin with, which is, yeah, lot easier said than done.
It was a question in chat.
Well, that actually kicked out your skill.
Oh, sorry.
Sorry, go ahead.
Yep.
There was a question in chat, which I thought was rather interesting about the using generators. And Python has generators which can be used for streaming meaning like it doesn't actually load everything into memory.
It just like, you know, as you, every time you grab it, it gets, you know, it computes it the next thing.
Oh.
Right? Or it pulls the next thing from the file or the list or whatever. So, yeah, so they, so, but I don't know if Pandas and generators are.
To be used together.
But well, Pandas itself loads everything in the memory, right? So, In other words, there's no.
As far as I know, there's no. There's no construct in pandas. To create a generator, right?
So in other words, if you say, give me the first 5 rows, it has those first 5 rows in memory.
It's already been allocated. It it displays us for first by rows. However, what it does have or where it can use generators is in the in the creation of the of the data frame.
So if you've got something that is streaming data from someplace and you need to make those changes.
Before you pull it into Pandas. It could use a generator there, right? So for example, let's say you've got a really big list of something where what you need to do is you need to I don't know Turn it into upper case, for example, right?
You can create a list comprehension generator, right, so you use parentheses around it instead of a instead of square braces that reads the data from someplace, let's say from a file and and creates this the generator and then use that generator inside pandas to pull it into pandas.
So you're not duplicating the data. You're not. Upper casing at first and then bringing it into into pandas.
You're doing it on the fly, right? So it streams it in. That does work.
But Pandas itself, I think it. It allocates all the memory. So if you've got, you know, Something that generates 50 gigs and you don't have 50 gigs of RAM.
You're you're kind of Stuck.
As far as I know. Anybody else knows otherwise I'd love to hear about it.
You were gonna say something, Lee?
Oh yeah, clean data requires a little bit of knowledge and skill. Seldom is that available.
Okay.
Well, you know, I made a career out of it, so I totally agree with you.
And careers are still to be made out of it. One of the challenges talking about, you know, garbage data is, My students always come up with projects of, hey, let's get this data from this side and the site, right?
And, a lot of those sites don't have API, so they end up having to go into, so they end up having to go into, you know, get the HTML and then scrape the data.
Out of that which is challenging, but that is not nearly as challenging as getting data out of a PDF.
PDFs where data goes to die.
Yeah, and
Huh.
Yeah, it's a PDF so the root of all evil.
There's a there's a great book actually on data wrangling. It's, data mainly with Python or data ringling with Pandas or something like that.
It's an Riley published book and they've got a chapter on panda or excuse me on on PDF and literally right after the, you know, it's like, I don't know, chapter 7 PDFs right after that it says avoid it all possible or something something along those lines you know or don't do it you know and yeah it's It's yeah, it's
it's kind of funny except when that's the only form your data is in. It's.
Yeah, you get the joy of dealing with it.
Thank you.
And Robert, let me pull your attention over to the chat. The last entry in the chat from Matt Mabby.
Yeah. I was a generator question.
So Right, so we've already addressed that there.
Yep.
Yes, I think we have.
Right.
Yeah. Needless to say, you know, and even though I'm, again, I apologize for going over, this was really a skim of of the stuff.
There's so much more that I wanted to talk to, talk to you about of what you can do with pandas in a data frame.
But a good one of the things that I had originally in the lecture was working with geographic data. There's a there's an extension to Japan is called Geopanus where you can now do stuff with basically geolocation data, right?
So you. You can have points and lines, right? So you, you can have points and lines and line segments and polygons and you you can have points and lines and line segments and polygons and you can do all kinds of things with that, line segments and polygons. And you can do all kinds of things with that, right?
You can do things like, you know, how many, How many, you know, if I've got a point, is it inside an area?
And if it's not inside an area, how close is it to it? What's its central a, a bunch of stuff.
So if you're into geolocation kind of stuff, Yeah, it's there, knock yourself out.
It's a blast. The other thing in related to that was I wanted to show an example of extracting XF information out of images.
Most people don't realize it, but whenever they take a picture with their camera, they're not just taking a picture.
It's capturing as in it is your camera is also often capturing a bunch of other information including information about your camera.
So you've got like a serial number, some kind of an identifier about the camera in that XIF information.
It has got the make, the model, the date, the time. The latitude, the longitude, and in some cases it even has the direction that you're facing when you when you've taken that picture.
All of that is really rich, wonderful data that is right for just pulling into a data frame, cleaning it up, and then stuffing it into a database.
To do other work on. And of course, Depending on the camera you have, you could have multiple.
Multiple channels for for the image your standard camera is usually RGB right so 3 channels but I've heard of people having more sophisticated cameras that are as high as 8 channels to do do things.
So they're capturing very specific ranges of frequencies or intensities or something like that.
And that data is getting pulled in. And you know, gee whiz, you can, you, there's, there are libraries in, in, in Python to extract that information as well.
So lots of cool stuff that can be done with it.
So, alright, any other questions? Anybody?
Oh yeah.
Okay.
Then. And naturally when you put those pictures up into TikTok, when it all gets down to the communist Chinese servers, none of that personalized data from your cameras, no they don't store any of that.
That was my whole question I thought you were gonna be this Chinese. That was my whole question. I thought you were gonna address tonight is why the pandas have those nice pretty black and white outfits and why why did the communist Chinese take them out of our national zoo and take them back home.
Sorry Next, part 2.
Good. Part 2 or 3 or 4. Hey, I, I, this has been a wonderful, wonderful talk and I really, really enjoyed it.
If there's anything, wonderful talk and I've really, really enjoyed it. If there's anything that, you know, really, really, really, really, really, really, really, really, really, really right now, go ahead and ask the question.
Otherwise, I'm gonna suggest. Robert's always been great about picking up things in our email discuss.
So I'm sure if we asking some questions tomorrow. He'll come up with some answers for him, but anybody anything that really has to be talked about yet tonight?
No, but I'm gonna take my call detail records in my phone book and put them into NuShell.
Which, by the way, completely supports data frames.
Cool, cool.
So the follow-up to Pandas is going to be data frames in NuShell given by Ed.
Okay.
Oh.
I did not. You know, I joined the Navy because it said never again volunteer yourself that was the background volunteer yourself that was the background.
Yeah.
I'm following, I'm keeping to that, trying to keep the
Hello.
Yeah. Well if, Ed, if you put it together.
No, it would do you think that is that like the hello world of data frames like something simple like that?
I mean, cause I don't know what the whole world of data frames would be.
Oh, good question.
I was like, I like to start there.
Like read in a very simple CSP or something.
Yeah, probably. I mean, I the example that I gave, I mean, I use your data record, but, There's the one that I had where I just started from, oh, that's Edgar.
Here we go, transform.
Yeah, that's a good question. I should. I don't. In my class.
I the example that I'm using is a variation of the example that I gave where I say, here's a blank data frame.
And here's how you can get a, you know, a series of numbers into a column. Right.
And you now have a data frame with a single column with a bunch of numbers. Right? And you, and I didn't do any of the stuff that would, you know, that we could also talk about, right?
It's once you have these numbers, you can actually then do all kinds of math things, right?
I just use describe, but there's stuff where you can do, well, what's the men, you know, you can actually calculate yourself, you know, all kinds of stuff, with, with data frame.
So, but I didn't get into that. But yeah, I would imagine this, the simple start would be, hey, here's a blank data frame and here's how you create a column with some stuff in it.
And then, and then display the data frame. Right, so yeah, maybe I should I should do that, you know DFA is a list hello comma world and boom Let me do that real quick.
Oh, this is gonna work. Oh, I'm already starting off on a bad foot.
Nice to hear.
Well, that's how I picked up. Okay, go ahead.
No, no, go ahead. I'm just playing here.
I was gonna say that's the reason I'd like to start there because that's exactly where they started and okay in our book And that's exactly how we start every new programming language that I learned.
Is how do I do hello world. Right. And, so everything I do, I start with a little world.
And I just wondered, well, what's the whole world for data friends?
Okay.
There you go. Credit blank data frame. And then create a column with a list with 2 words hello and world and then display the data frame and it says hello world.
So I'll send that to you, Ed.
Thanks. Okay.
I mean, what other people think? Does that sound like a legitimate hello world kind of thing with data frames?
Yeah, it even looks like it.
Even says hello world. I imagine that.
No, no, I'm, and even it even looks like a data frame.
Oh yeah, yeah, yeah, yeah. One thing I forgot is the exclamation point. One sec.
There we go. Oh shoot, then I got rid of the D.
Yeah. Okay. Yeah. Wordal.
Yeah, exactly.
Yeah.
It's still not right.
I don't remember that they did, you know, they've done
Oh, there it is. You got it.
Wait a minute, it's not right, What's wrong?
Hey, got it so many times in so many places. I forgot if the original K in our book.
I mean the original, right? If it actually had the explanation point at the end. Did it? I don't remember.
Oh god. I'm not home I know, otherwise I'd look.
Well, I was gonna I was gonna ask Bill Bunch because he had KAY in our I don't know if he had, I don't think.
Yeah, who doesn't?
He had it like in a in a PDF or text file or something. Yeah.
Where's mine? Thanks to having a Somewhere. I said, why should it be a dot from you?
Yeah.
So, all right, folks. It is getting late. Any other questions before we bring this over to the, the online, mailing list?
You bet.
Hey Robert. Hey Robert, if your boss.
Thank you very, very, very much, Robert. We've really appreciated it. Again, Robertson from New Mexico.
Your boss ever wants to do a like a day long boot camp. I'd be more than willing to pay for it.
Oh wow. That is, believe it or not, that's actually one of the things that is in discussion is taking the 12 week program and breaking it up into smaller pieces for I would say we get a lot of people that already say, well, I already know.
XY, and Z, all I need is the last part. You know, the, a little bit in this or a little bit in that.
And so yeah, we're definitely looking into it. So if anybody's interested in that kind of stuff, yes, please send me an email, because, I can then go to my boss and say.
People are interested. We should do this. So yeah. And, and by the way, the course that I teach right now is, is a hybrid course.
Meaning it is both in person as well as online. So if any of you guys are interested in, we take anybody from anywhere in the United States.
I actually talked about to my boss I said, what do you wiz? You know, I'm going down Argentina in a couple days.
You know, if I talk people there and they're interested, would, would you be, would, you know, is there any reason why we would say no to them taking the course?
And her response was, absolutely not. So if you know of anybody elsewhere in the world that's interested, let them know as well.
Okay, well, yeah, just If we could dump a link to your course description and Just discuss. I might actually.
Cause I
Absolutely. It was here. What did I do with it? Oh, here it is.
Hmm.
There we go.
I'll put it in the chat. And I'll also I'll also send it out on on email as well, but that's that's the that's the URL.
Oh, for in. I looked up in the original K and R. And the phrase in the original KNR is hello comma space world and a new line.
No capitals. No. Explanation point.
Not even a capital for the H.
That's correct.
Yeah.
Yeah, there's been, I like maybe they ought to write a book about the variations of a little world over the years.
So, so just to so I get it right it was All our case, hello, comma, space, and then world.
Correct.
Wow, that's
Modified. There we go. Look at that. Updated.
Okay.
Bye, my very first C program. I did, you know, did I get to the printf statement and I wrote, I changed it to goodbye, cruel world.
Yeah. Oh, and just can't follow directions.
Yeah.
I figured that I figured No, no, I just figured you had to, you know, they had to be clever.
Cool.
That was it was a good it was a real good talk. Didn't think I would be interested in data frames, but I got me curious.
They're cool data structures, but like anything else, it's a data structure, right? If you have use for that kind of data structure, they're great.
If if you don't then yeah it's not so useful, right? One thing, I mean, talking about that, yeah, don't think it's a queue, right?
Don't treat it like a cue. Don't try to put things on the bottom and and put things on the top and constantly add things on and off.
Do not do that. Data frames are horrible for doing that kind of stuff. Even worse, don't treat it like a deck, right?
You know, double-ended queue. Don't do that either. They're, they're absolutely terrible for that.
So. But yeah. Use the data structure for what it's for and for a lot of the stuff that I have to work with, it's it is indispensable.
It does, it does a lot of cool stuff. For how would you say rankling the data? All right, folks.
With that, I'm, I actually have to need to, I actually have to take off because I got a pack for a flight and got a bunch of stuff to do tomorrow before that flight takes off.
So.
My goodness, you go into Argentina?
I am.
Have fun.
Oh, alright. Well, with apologies to Kern again, Richie, we have to say goodbye world.
Yeah.
And this is the book end for the recording. So this was the Wednesday, thirteenth, December, 2023, general meeting of slew, the St.
Louis Unix Linux Unix users group. Our speakers tonight, we're Sean Tweet house on I'm gonna slaughter that.
We were mispronouncing it as ASCII cinema and hopefully you can find it.
That way. And, Robert Citak did a wonderful presentation on, Panda data frames, Python, the data frames.
And with that, thank you all for coming and please come next Thursday night to hear Sean Twe House on ZFS 3 2 one backups.
Thank you and good night. Goodbye world
Yeah