What happened to his new generator set?
And… I don't know. Probably a 25 cent fuse.
Oh, there we go. Okay. Okay, recording in progress.
This is the 20 march 2025 edition of the St. Louis Linux users group session tonight is going to be by Robert Sittick, a topic a desub on Google Cloud Platform.
I assume that Robert is still here. He was here earlier.
Before… Hi there.
Hi, man. Greetings, everyone.
Themes that the meeting is being recorded now so that we can go ahead with that.
Oh, man.
I'd like to go ahead and
Quickly cover a few things here. Come on, machine.
Oh, that's terrible. Well, that button didn't work.
God. No, it doesn't surprise me at all.
And that surprises you?
Jeez.
If only we had a… group we could ask for help for things.
You would think so.
I don't know if lee has… mains power back, but he had been running on generator After last Friday night's storm.
Oh, since then?
I don't know if he got it back sometime in the meantime, but I know he was as on generator Late Sunday.
Ooh.
Yeah, I mean, it was… working when gary got to the bat cave and then when all of a sudden after all of a sudden Lee had started up the meeting and about 10 people were in there and he got a phone call from somebody about something and
Was away from his desk and then all of a sudden the power went out Boom.
Did he not pay his gas bill? That would do it.
Probably not, you know. That would do it, I guess.
I bet that thing sucks a lot of natural gas.
I don't know.
The one my neighbor has has a fairly small line going to it.
What's the pressure in the line? Excuse me.
Oh, it's quite… two to three bar, I think. It's not very much.
Two to three bar two bar is 30 psi.
Yeah. All right.
In the grand scheme of things, it's Compared to car tires and other things, it's quite a bit lower.
No.
Well, natural gas in general for residential is served at about three inches of water column pressure.
Maybe that's what I meant, not bar. It's been too long.
Very low.
Yeah, that makes more sense.
Where are we?
Gotta cheat here. Anyway, this is St. Louis Linux users group meeting we're usually… It's held eight days after the general meeting, sometimes 12 days, depends on how the month is laid out. That's either the third or fourth Thursday of each month.
All of our meetings are free and open to the public. There's no individual membership fee.
It's for… Our website is the stlouisww.stlouis stlinux.org.
We will either use JSY or Zoom, but not both. Right now we're using the zoom presentation archives are on our web page.
We have PDF handouts available and assorted other links. From 6 to 6 30 arrival time is for testing your chat your microphones.
A web camera. Things like that. And then the meeting is supposed to start at 630.
If you don't have any… bugs or things like that happen.
Uh and we can start out with the introductions and announcements and mention of other sponsored meetings and our archives and calendar and so forth, and then we'll have a call for help. And then we'll have the presentation. After the meeting, further discussion is possible.
Plus, people can ask questions on our mailing list. Recorded chat, slide work and video or go to our archives page in the next few days.
Chairman is James Conroy. Is James here tonight or is he too… on his project.
James got married recently, and I think that's a major project.
Anyway, Gary is normally the the emcee of the meeting.
Our host, Lee Lamerck at Omnitech. Runs the facilities and everything.
I do some production and recording and move things around and send out announcements and things like that.
During the presentation, we'll need somebody to actively monitor the chat room and monitor the chat room Usually the speaker is too absorbed.
You follow the chat and if questions are asked there. It's nice if somebody's nice if somebody verbally responds and verbally responds and lets the speaker know.
Okay.
We also need other people to help with getting
Additional presentations throughout the year. Our next meeting will be on April 17th.
Five days from now on the fourth Tuesday, the monthly new Linux meeting.
Stan, we can't.
If you are our new Slug website built by Sean Thuihaus and uh I'd just like to remind everybody that Microsoft Windows 10 End of Life is in November this year.
Does anybody else have any Painted like to announce.
I'm monitoring the chat. And I've seen a few times within the last week that Amazon has made a change to Echoes.
They're doing away with on-device processing and anything and everything they hear goes back to Amazon for processing. And I want to say that was like as of the 25th or the 28th.
Of March.
I was listening to Windows Weekly yesterday and was revealed in a very subtle announcement by Microsoft that they're… discover, not discover, they're antivirus facility, whatever, no longer uses rule systems. It's just all for artificial intelligence.
They don't really know what it's doing now to filter out viruses.
I thought that was hilarious.
Anyway. Go ahead.
Don't you do it? I was going to submit I want to read an interesting paper.
God, it comes from McDonnell Douglas back in the 90s. I'm pretty sure the title is Children of the Magenta Line.
Oh, yeah. It's a good movie. It is. It's a movie.
I didn't think it was a movie. Really?
Yeah. Well, a video
I didn't know. Yeah, okay, so it wasn't a Hollywood movie or anything.
No, Mac made it. Because their pilots so focused on this new thing called a magenta line on the screen that they stop looking at other instruments.
Yep.
By the way, I've just been notified by my system that my uh internet connectivity is unstable. So I might disappear any minute now.
I'll try not to… vaporize.
We'd just like to mention that we have other affiliated special interest groups. This is one of the affiliated SIGs of the main Unix usage group that in five days from now, the new Linux user group is going to be a beginner workshop.
And then on the First Thursday of the following month, the Slack configuration workshop for hardware and software kind of questions.
Operating system agnostic, hardware agnostic It's a very nice group.
Fortunately, Phil Bunch and a number of other people show up to answer a lot of the questions.
We have a calendar. On our calendar, wwwslub.org slash calendar.
It lists all kinds of groups in the St. Louis area. It used to be strictly face-to-face listings of Anywhere from 150 to 180 Monthly uh meetings during them every month here in St. Louis.
Now we've started incorporating remote group meetings because of the ability of Zoom and Jitsi and things like that.
We have a mailing list. You can get to our website. It's available.
They have an announced list with very limited traffic. Usually it's just the meeting instructions on how to get on the meeting and on the meeting discuss for general discussions, opinions, questions, reviews, things like that.
If you have a system administration question, you can ask it The sysadmin group shown lower down is for our volunteer support staff.
People that actually do work to keep the organization going. And then we have a steering committee group Every opinion counts. We appreciate people to come there. Ideas welcome, but a lot of people don't like to see how the sausage is made.
It's open. The general meeting, the slug meetings and this meeting, the St. Louis Linux meetings are recorded and are archived up on our website and we're starting to put them up on YouTube.
Recorded chats, the slide works and videos They'll be available in a few days.
And this archive presentation is listed at the bottom of each of our websites.
Okay. Privacy and security. Like I said, only those two meetings are recorded Well, they're recorded and posted and I, for one, will be recording this meeting for my use to track attendance.
Anyone here might be recording. If you want to do your own recording.
I did a presentation on base presentation in October 22. On simple screen recorder, very easy to use.
Call for help before we go on. This is a call for help time. If you got a question, you can ask it now.
Anybody have anything that they'd like to talk about? Did they break something recently?
We might be able to help. If you don't ask… I got it. Yes.
Everybody get that tone like I did?
Yeah. Yep. That was my fault.
Oh, I thought maybe my speakers or my amplifier was getting ready to blow up or something.
Go ahead, Gary.
But I'm going to suggest we travel on from the call for help because we've got kind of limited time from uh Robert tonight and so robert tonight There will be time after he speaks, both the questions on the topic that he's speaking about tonight and then also we can go off into other general questions after that.
Take over from here, Gary.
Okay, on the St. Louis Linux user groups the St. Louis Linux user group excuse me We try and keep the presentations more linux focused.
And uh the presentations like tonight tend to be hopefully a little more advanced and detailed.
Other times we do more. Basic or tutorial types. We usually do those on the front end of the Wednesday night meeting the general meeting.
But so by all means, watch our calendar, our website. And hopefully we'll see you all again.
And the other things there are instructions to the presenter. We're always looking for presenters so if If you don't want to present, but there's a topic that you know of that you'd love to hear talk.
Spoken about, let us know. If you know somebody who might be a good presenter.
Let us know and we'll try and follow up and find these people.
Send your suggestions to editor at slug.org. Or you could mention them locally here in the meeting.
Robert, Robert Sittick, do you want people to ask you questions during your presentation or you want them to hold it to the end?
Absolutely. They can ask during the presentation, not a problem.
Fantastic. Okay, so let's see. Next slide.
Well, that was toward the end. I did everything else, Gary.
Yep. Okay. So Robert Sitek is a longtime member of this organization.
I used to live here in St. Louis, but he and his wife have moved off to the northwest part of the country.
And so he's been a great resource for us, a great speaker.
And we're really happy to have him here tonight. With that, I think I will turn it over to Robert and let him Start his presentation today.
Awesome, Gary. I'm trying to start my video. There we go.
Okay.
Hello, everybody. It's been a while since I've seen a lot of you. Hope everybody's doing okay. I heard you had some interesting weather recently.
Hope everything worked out okay. Not too much damage, nothing irreparable. Excuse me.
I'm going to try to share my slide and then turn my video off because nobody needs to look at me. I'm not that interested. What's interesting is what's going to happen on the screen.
Let's see if I can share my screen.
I just want that one.
Let's see, that looks like we're sharing. Awesome. I'm going to turn my video off.
I'm going to try to There we go.
Awesome. There we go. I posted links to the presentation as well as to
As well as some demos that I have on GitHub. So all this material is available online as well. You're welcome to try it or go with me.
Or walk through with me as I go through the presentation. What I'm hoping to do here Oh.
There we go. The flow of the work or the flow of this talk, I'm going to walk you through a case study that happened to me or that I was working on a few years ago and how we used various technologies to get a How would you say a insurmountable task done in a in a very short period of time with, how would you say with
Unbelievably reduced cost. Using a tool called DESA.
I'll then go through and actually show you. I took some, how would you say, some That's some screenshots, but copy them into the presentation, both for local as well as on Google Cloud Platform, so you can actually see what it looks like.
If we get through all of that and answer questions, I also have then a live demo to do so you can actually see it running. I've got a GitHub Codespaces running with everything ready to go so you can see that.
I ran into a little bit of trouble with the live demo of GCP, of actually getting Google Cloud Platform to work. But knock on wood, if I try again during the presentation, maybe it'll work.
We'll see how it goes. And yeah, unfortunately, I will have to leave in a little more than an hour from now. So we'll see how everything goes. But obviously, any questions you have during the presentation, please feel free to interrupt me. If you post in the chat, I will likely not see it. So whoever does see it, Gary, Stan, whoever, please just interrupt me and ask me. I'm more than happy to.
To stop the presentation and answer questions. So a little bit of background about myself. Yes, as Gary mentioned, I used to be in St. Louis, used to work at Wash U, then worked for a startup company, and then ended up working at Sigma Aldrich.
Which is now i heard was acquired by somebody else. Unfortunately, I don't remember who. I think it was Merck, but I'm not sure. Since the years I have left left, then went off to Detroit for a while.
Then ended up in the Pacific Northwest for a number of years And now currently, since the pandemic, I'm actually down in Albuquerque, New Mexico.
So, and here I am an instructor for our data science program.
And I teach data science. We have funding, if you will. So if you folks are interested in taking a data science course with me, we do have remote options. There is funding available. If you qualify, just reach out to me and I'll post a message on the SLU mailing list later on as well. We cover a number of materials.
We use Python. We cover everything from getting data, cleaning it up, storing it, querying it.
Doing a bunch of machine learning algorithms, both supervised and unsupervised. We get a little bit into deep learning.
And of course, visualizations and all that kind of fun stuff.
And this is actually a lecture that I gave actually a few years ago, initially to where I worked.
And then to my students, in fact, I gave this to my students this morning, a slightly revised version of it.
I actually gave it to him. I gave him an intro to it yesterday and then I give them the full version today. And then I'm going to actually be talking to a larger group of students in about a week and a half
Ish about the same topic. So if you guys have any questions, any feedback, I'm always receptive to that and appreciate that.
So this is the flow case study, local example, Google Cloud Platform example using D7.
So let's talk about the case study. So a number of years ago, I was working at a healthcare organization in their pathology department and Basically, we would get these pathology slides and whenever we would get these slides, we would scan them using a fancy slide scanner, which generated
Proprietary images, which the pathologist would then look at markup and what have you, zoom in, zoom out, put all kinds of annotations on the image and then save.
We would then take those images and push them off to Google Cloud Storage.
We've been doing that for a number of years and years One day I happened to be running I would say I walked into, I was walking the halls and ran into a postdoc and we just struck up a conversation there
And she's actually a postdoc in the research side. I was on the clinical side. She was on the research side. And she was telling me about this project where they want to take all those slides and use them for machine learning.
To work with a partner basically to analyze these images, right?
A couple of key features of that project. Number one, there were about 10,000 pathology slide scans, right?
They were at a resolution that made them pretty big, roughly 100 megabytes an image.
The images, however, were in a proprietary format, which unfortunately meant they couldn't be used in a standard software that they were using for machine learning, right? And so the postdec had gone through and took a sample of the images and ran them through a conversion program that was provided by the vendor. And on average, it took about 30 minutes
To convert a single image. And perhaps the most important part about this project was that they had a milestone that had to be met within two weeks. That is, they needed the 10,000 images converted and initially do some initial work on within two weeks.
So we sat down and did some back of the envelope calculations.
To answer the question, can this be done? So we have so we have about 100 images, excuse me, about 10,000 images, each image about 100 megabytes each.
Roughly the size of this entire bolus of images is about a terabyte. That's not too bad.
Child's play for Google Cloud stuff. But we would have to get those locally onto our local computer to work on it And it would take about 30 minutes. An image on the computer that they had in the lab.
So how much total time would that take? We'll simply multiply 30 by 10,000, you get 300,000 minutes. That works out to about 5,000 hours 200 days about 30 weeks which is about seven months. Last time I checked.
30 weeks is much longer than two weeks. Which is what their milestone deadline was.
So what do we do? But we looked at our options and we came up with three options Right off the bat. The first one was admit defeat.
Go back to leadership and say, hey, guys. There's no way we can do this. We estimate it'll take about seven months to get all these images, right? So that's one thing.
That I'm sure would not have gone over well. The second option was to get much bigger computers or computers as in plural.
We estimated that this would cost somewhere between $30,000 and $80,000.
The computer that they had in the lab At the time that would take about 30 minutes to process an image, cost roughly $2,000.
If you were to take 30 weeks and divide it by two weeks, that's 15.
15 times two, well, that's $30,000. So if you spend $30,000, you can get it done in two weeks.
The problem is that's if everything worked fine For those of you who have ever worked on a project, a complex project, and everything worked fine.
Good for you. That is a rarity, right? There's always something that doesn't work. And keep in mind, the $30,000 to buy these 15 computers, that's just for the computers.
That's not networking. That's not network switches. That's not finding the power or the racks or anything else that you need.
It's just the computers, right? And that gets you to two weeks.
If you wanted to give yourself some breathing room, let's say a week, well, you can double that number, right? 60,000 throw everything together, maybe $80,000, right? And that's assuming that you can get everything through procurement, through requisition, all that fun stuff in a large organization, right?
Good luck. The third option that we explored was, well, gee whiz, all these images are already in Google Cloud Storage.
Why don't we just use the cloud? And so that's what we did.
We use Google Cloud Platform. That was the first obvious thing. The second obvious thing was how do we go through and crank through all these images?
And that's where DCEP comes in, right? It's a simple batch.
Processing tool that uses docker as it's, how would you say as its core technology
The cool thing about this is this was If you've ever done worked in any queuing system such as SunGrid Engine or LSF or any of those, they have a tool called QSub, which was for the QSubmission tool, right? You would take your jobs and you can submit it to the queue.
That is basically what DSub is based on. Right.
The components to that for this particular project is we needed to have Google Cloud Storage. So we had that. That's where all the pathology images were.
We needed to have Python. Well, that was easy to get.
We needed to have Docker, but we were already very familiar with docker.
And we needed to have a place to put that image, a container registry. And fortunately, Google has a container registry service, a GCR.
And then tying all those things together was deceive itself. The way it works is basically we push the images to Google Cloud Storage, which we already had done. So that's taken care of. Checkmark, done.
The second part was to wrap the Python script in Docker. Well, that was relatively easy to do. We had worked with Docker files in the past.
Took our script, put it together in a Docker image. We could test it out local It worked just fine. So we're good to go there.
We then pushed it to the Google Cloud Registry. And then we created what is called a desub template file.
As well as a few other small how would you say options, if you will.
The way this technology works is, in essence, think of it as every image or excuse me Yeah, every image gets processed on a single Docker instance within a single VM. So for instance.
There's a Docker container and that Docker container resides on a single virtual machine.
With these subs, you can specify the virtual machine characteristics, right?
The size and type of CPUs and the number of CPUs, the RAM, the size of the RAM, local storage whether it's preemptible or not, and we'll talk about that in a little bit All of these features you can specify on what your VM should look like.
We also had a little script that basically told it how to process. In essence, we said, hey, once you're in the Docker container, run this Python script, literally one line.
We had to tell it where the input was that is where is the proprietary image Which was on Google Cloud Storage and where we want the output to go, which would also be in Google Cloud Storage. And the output was actually
Two things. Number one, it was the converted image. In this case, we use TIFF.
But we also got metadata out of it, right? So remember, the physicians would write a bunch of annotations on the image. We wanted to capture that as well. And so we did. We pulled that out.
From the image as well. We also put that into Google Cloud Storage.
Once we had all these pieces assembled, we then basically said, hey, DISA, do your thing.
So in graphic form, this is what it looks like. All the slide images on the left-hand side end up in Google Storage on the right-hand side.
We take our Python script and any libraries that we might need bundle it together into a Docker image. We take that Docker image and put it into google Container Registry, and then we construct this DSub template, which is basically one line per file.
And it specifies where is the file located in Google Cloud?
And where do you want the output to go when it's finished?
That's what the DSAP template does. Once we have those pieces together, we then launch DSUB.
And what DSUP does is these six steps. Number one, it launches a VM.
In Google Cloud, it pulls the image that is the pathology image from Google Cloud Storage into the VM.
It pulls the Docker container from the Google Cloud registry into the virtual machine.
And it then launches the Docker container to convert the image.
Once it has that converted image as well as the metadata, it takes both of those and pushes them out to Google Cloud Storage.
And then destroys the VM. And it does that for every single image.
Launch your VM, pull the pathology image. Pull the Docker container, run the conversion push the output to Google Cloud Storage, destroy the VM.
Except it does it in a very particular way. It does it in parallel.
We can launch all 10,000 images in one shot. So again.
The decent workflow. Step number one, Google. Step number one, you create the VM.
We then take the image from Google Cloud Storage the pathology image, put it in the virtual machine.
Pull the Docker image into the virtual machine, run the container, which takes this image and puts it into or extracts it and creates these two entities right one converted image in a TIFF file, and the other one happened to be a json file
Those then get pushed to Docker storage in two separate buckets.
And then we blow it away. It all dies. Well, the VM dies.
Right. We repeat, we do it for the next one.
Again, all at the same time. Right?
Some metrics about the process. We ran a maximum of 4,000 virtual machines all at the same time.
The total running time to process all 10,000 images, or approximately 10,000 images, was about three hours.
Per image True to form took about 30 minutes per image.
But the total cost was only $300.
So the cost per image was only three cents. So what does it look like if we were to actually get 4,000 machines, right? 4,000 CPUs? And I'm going to assume one CPU per machine, which is not quite correct, but
Well, for demonstration, we'll use that. What we're looking at here is a row inside a data center. On the left-hand side and right-hand side, we've got roughly 10 racks.
Each rack holds up to roughly 40 computers. So this is roughly 40 machines.
Excuse me, 800 machines. If we wanted 4,000 of these We would need to have five rows. And this is roughly what five rows of computers looks like.
There is no way we could have gone out and bought five rows of computers to run this processing. We just didn't have the room or the budget for that.
But fortunately with Google, we can just say, hey. Fire this up and when you're done, blow it away.
And the way Google's pricing works is you literally only pay For the virtual machine while it's running, right? So if you create a machine Boom, the meter starts running.
As soon as it's done, DSA would automatically kill that virtual machine which meant the clock stopped running.
That was a huge savings. Instead of spending $30,000 or more, maybe as much as $90,000, we only spent $300. And instead of taking seven months or 30 weeks to complete this, we got it done in a few hours.
Last I checked, three hours is much less than two weeks.
The lab now had plenty of time to go through and look through the images to see, hey, did it work? We can test it, all kinds of fun stuff.
At their leisure, if you will, instead of panicking of how to get this done.
Instead of taking 5,000 hours, it only took three hours. That's about a thousand fold increase in efficiency and speed.
Instead of costing $30,000, and I'm picking the low number there, we only spent $300. Literally, I put it on my credit card and then asked for reimbursement.
No problem. That's a hundredfold. Savings right there. The net result we were able to go through and get all these images done faster better and cheaper. And what I mean by better is that we actually had enough time to go through and quality control the images to make sure everything worked appropriately. If we would have got this done in two weeks, we would have never had the time
Even one week would have been probably pushing it.
We met her in lost him. We got funded.
That's a nice feeling, right? To hit something and come to a vendor or a partner and say, yep, here you go. Here are the images.
What's the next step? Hey guys you know this is who you can make the checkout to.
It's kind of a nice accomplish so How do you get started? Where to start? Obviously, there's a GitHub site for DSub. You can read through the instruction, get started there. I've gone through it, and the way I like to start, of course, is in a Docker container.
All the instructions for how to set it up All the requirements, the packages, all that fun stuff. I've already prepackaged into a Docker container. And this is how you can run it.
You literally just say docker container run and pass in a few options And you're up and running. You're ready to go.
It creates a temporary directory in which to store things. But once you've got this up and running, you're ready to go.
Once you've got it running, just real quick, it runs as a service It runs in what is called detached mode.
You can then exec into it. So you get an interactive session, and then you can start typing doctor commands.
Or excuse me, DCEAP commands. So for example, we can pull a generic image Let's imagine this is the image that has everything we want in it already.
If it doesn't, if it doesn't we could use this as a base image and then add more things to it, which is what we did. But for demonstration purposes, we can literally pick any image we want.
Download it and play with it. We're going to pick you 2404.
To launch the V sub we simply type d sub and give it all these options, right? So in this case We're not using Google Cloud Platform. We're going to run it locally, literally on our local machine.
And the image we're going to use is the Ubuntu D sub. We tagged the Ubuntu image with the label D sub.
I specify we're logging and output goes as well as what command to write to run.
In this case, I just want to say, hey, hello world, just run that.
And if we run that We'll get some output, right? Basically some job information. Hey, are we going to run this job. It has this ID.
If you want to check the status, you can use the dstat command, which is another command that comes with D sub.
If it is a long running process and something doesn't seem right or you realize, oh, I forgot something you can kill the process. You can cancel it. You can use the ddl So delete or d sub delete if everything works well.
You get this nice success at the bottom. Yay, things work.
If we check the directory that we specified where we want output to go, this is what it looks like. We have all the logging here for this particular job, right? Standard error, standard out, et cetera.
And we have the actual output from our command. In this case, we put it in an output directory and it's going here. And all of this is local.
If we want to see the contents of that file, we simply cat it and there it is. It says hello world.
What if we want to run more than one? Very similar to what we did with the 10,000 files. We can create a tab separated value file, a TSV, We specify where the input is, where we want the output to go. In this demo, I'll just give you an example input.
I'm going to create a mock input file. Here's one that just says hello world. And there's my input file.
And the TSV file will just have the inputs and the outputs. On the left-hand column.
Is the inputs on the right-hand side is the outputs For this demo, everything is the same input, but I have three different output files, out one, out two.
And O3.
So I've got the input file. I've got a TSV file to say, hey, this is where we want to read the input. This is where we want to put the output. But what do we actually want to do?
And so this is a script that actually generates the output from the input. In this particular case, I'm just running a set command. I'm going to change that hello to greetings.
Add it to the output file, read the input, send it to the output. And then I'm going to add the date at the end of the output file.
And then I'm going to launch it. I'm going to run the D sub. Again, I just type d sub use a local specify a docker container that I want to use or a docker image.
Specify where logging goes, what script to run. And the tasks file. This is the one that has the three different tasks.
When I run it, it comes back with job information. I'm launching this job.
If you want to check the status, you can look at the stat If you want to delete it, you can delete it.
And again, we're looking for this success. Which we got there.
Again, if we look at the output directory We have the logs of all three tasks, right? So task one Task two and task three And we have the outputs from all three of the different tasks.
If you want to see the outfits themselves, here's a tail that grabs all three of them. Notice now, instead of saying hello world, it says greetings world.
In all three files. And notice that the time is the same for all three of them.
This wasn't sequential. We don't see 37, 38, 39 or 37 in some later time.
All three of them are at the same time. It runs everything in parallel.
We can do the same thing in Google Cloud Platform. It's a little bit more involved, partly because, well.
Google has lots of security permissions and all kinds of fun stuff all around it. I've got some notes on how to set it up.
This is… not quite complete, but it's pretty close.
You need to do all the setups for permissions and things and enable APIs and all kinds of fun stuff in advance. Once you do.
It's in essence the same thing. In this particular case, I'm specifying a bucket This is where my data resides.
And now that I had to put a few little Google Cloud administrative things, right? I can specify what project I want to be in. That's a Google Cloud thing. They have these things called projects.
Right. I can specify the region where I want my virtual machines to run.
But pretty much everything else is the same, right? My provider now, instead of being local, it's Google.
My logging, instead of being local, it's on Google Cloud in their storage service. Their output also goes to storage service.
The image I use, again, still Ubuntu, in this case, 2204. I forgot to change that to 2404, but I can specify any Docker image that I'd like. And in this case, again, I specify a single command, right? So this is exactly like what I had on the local system where I run
A simple command. And when I run it again, I get the same kind of job output, right?
Does it look any different other than, oh, look, I'm using Google instead of the local provider.
Again, I've got success. If I look at the output files, now within the Docker container, I have the gsutil command there. It's already there ready for you to use.
You can list the files in your buckets. In this case, I'm looking at my RWC data bucket I go through, again, I see my standard error, standard out, and standard logging I also see my output.
With the gsutil command, I can display the contents of the output file. And there it is hello world.
Same as before. I don't have an example with the TSV file in Google Cloud, but it's in essence the same thing as what we've seen in the local demo.
So what's next? There are tons of options in DESA, lots and lots of options, right? Do d sub dash dash help.
Literally pages of things that you can do with it. Some of the cool ones are the ones I mentioned here. After is kind of nice.
So AFRA allows you to have some conditional flow control, if you will. So you can launch a job and then you can say, hey, after run this other job.
So you can have two different batches depend on each other, right? So maybe you want to make sure one job completely fills before you start the next job. After we'll help you with that Preemptible.
I mentioned this earlier. One of the nice things about Google, as well as other cloud providers.
Is that they have this thing called a preemptible instance. If you're familiar with Amazon, Amazon calls these spot instances.
I actually think even Google now, they've begun to call them spot instances as well, but I'm not sure on that.
In essence, what this is is When you normally start a VM, you get what is called on-demand pricing, right?
You look it up and you say, aha, great. If I want to start a VM that has these kind of features, it's going to cost me 10 cents. If I want more advanced features, it's going to cost me 25 cents per hour.
If I want the latest, greatest whiz bang thing, I can get machines that will cost me several hundreds, if not thousands of dollars an hour, right?
So that's the on-demand pricing. Preemptible or spot instances Those go at market rate.
In other words, if Google happens to have a bunch of machines just lying around, a bunch of VMs lying around, not doing anything.
It has a significant reduced price. Per hour to actually utilize those systems. So for example, maybe something that cost I don't know, a dollar an hour would now cost only 10 cents an hour, right? Or less.
So that's the advantage of using preemptible. You basically say, hey, if you've got these kinds of machines that I need and nobody else is using them.
I'll pay a reduced price to use those machines. There is a caveat with that. The caveat is if all of a sudden demand goes up For on-demand instances.
Google couldn't pull the plug out of your job, out of your VM.
They can destroy it.
For us, yeah, far away.
I've got a question. Okay, so let's say you need I don't know, a thousand machines to do some kind of processing like this and you want to run at that reduced rate, can you tell them schedule it whenever the schedule
Cheap rate is available. Don't schedule it for prime time. You have control like that?
Oh, good question. It's basically if you say preemptible, it goes at the low price.
If no machines are available. D sub just sits there and waits until They've got machines available.
Think about it as like standby for airline ticket for a plane. If the plane's full.
And somebody no-shows people on standby can use it.
And do they pay a reduced fare?
For the airline example, is that also with a reduced fare?
Pardon?
I don't know about that. But for the sake of discussion, yeah.
Okay. Yeah, let's imagine it is, right? So let's say a normal ticket costs, I don't know, a hundred bucks or let's say 500 bucks to go to i don't know.
Hawaii, right? And you're on standby and you'll say, hey, if there's a flight available, I'm willing to fly there for 50 bucks.
No one's on the plane. The plane says, you know what, we'll take your 50 bucks here. Grab a seat.
If that kind of process happens with airlines, yeah, it's the same kind of thing here, right? If there are virtual machines available and nobody's using them.
Basically, you get to use them at a reduced cost.
The only difference is they don't kick you off the flight mid-flight, right?
That can happen to you here, right? You could be almost done processing whatever you're processing And if all of a sudden the demand goes up for machines and people are willing to pay the normal on-demand prices.
Boom, your VM will get killed.
Well, you know, if the price break is worth it, then it's probably not bad.
Excuse me.
Exactly. And if you've got the time So what, right? You just, hey.
Yeah, got it, man.
Maybe I don't have 10,000 instances right away available to me. But you know what? I'm willing to wait three days. I mean, remember, we had a two week deadline. More realistically, probably a one week, 10 day deadline Because we wanted to make sure we had some time to review things.
Who cares if it would take three or four days? We didn't care.
So yeah, throw it on a preemptible. If it takes three or four days, great.
The fact that it got
I don't think it's really fair to compare that to running out and buying machines and setting them up and stuff like that because That kind of thing, yeah, we're going to sit here and we're going to estimate we can do all that in a week. And you know that it's going to take a month.
That's just a way of the world. You know, it's never going to go the way you want it to go. It just never does.
Or am I the last guy that has that problem? Sure.
So the worst case scenario, let's imagine the worst case scenario.
We don't get any preemptible machines after let's say three or four days And we have to pay on-demand pricing.
Yeah.
Right. Boom, we pay on-demand pricing. We get it done in a couple of days we're still much, much less expensive than if we went out and bought all these machines.
Right? Right.
Oh, God, and think of the trouble. You know if how many people would you have to hire to set up those machines?
And where would you put them? You're not going to put them in a lab.
You'd have to find a data center. Right?
No.
Where would you put out a form? Where would you get?
And again, the other thing that's special about this project, it's a one-shot thing, right? We literally only needed to do the conversion once.
So this is not an ongoing every week we got to do this kind of processing kind of thing.
So that's the other thing that's really important to keep in mind, right? We needed to get a lot of things done in a very short amount of time That is a beautiful use case for D sub and using preemptible images on a cloud provider. We happen to use Google.
And mostly because we were familiar with DSUB and DSUB supports Google.
There are… how would you say desires maybe that's the best way to say it to have it also run on Amazon and Azure and Oracle and others, but those haven't materialized yet.
Right. But yeah.
Yeah, it sounds like a hell of a deal.
It worked out well for us. And the other thing about Oh, go ahead.
You know? Do you know if any of your instances were preempted.
Ooh, we'll save that for a couple minutes. How's that? Yeah.
Okay.
The other thing that's cool about preemptible is that you can tell it to repeat a fixed number of times.
So if it does get preempted. It will wait a certain amount of time and then try again on a preemptible instance. And you can say, you know what retry three times or four times or however many times you want
And if it doesn't work by, let's say, the fifth time.
Go ahead and use an on-demand. So that we're guaranteed to get it done.
So we can try preemptible a number of times And if those number of times fail each time, we can then specify on demand.
So that we're, how would you say ensured that it completes
A couple other, yeah, go ahead.
No, I'll just clear my throat. My allergies are making me crazy today.
Oh. I hear you. Same thing down here. We got, yeah, not fun.
Robert, quick question. I don't know if I missed it or not.
But what exactly is d sub is it a program Python script.
Yeah, yeah, it's a Python script. Well, I think it's a collection of Python scripts.
Python. Okay.
So basically, you run it from the command line, right? So you say d sub and then give it a bunch of options.
Is it a GitHub project or is it normally part Okay.
It is. It is. Yeah.
That data biosphere slash d subsystem
I have a link in the presentation at the end. Yeah.
Okay, thanks.
We've got other options. We can mount things. So if you have a persistent storage or a distributed storage, something like an NFS or something like that.
And I don't remember what the terms are in Google Cloud, but they have something like an NFS or a shared file system.
You can mount those things. You can use the end option to pass in secrets. That's the most common usage. You don't want to store your secrets inside your image, right? And so D sub has a way to basically pass it in via environment variables so that the container can read those from the environment. So you don't have to have them in the container itself. There's tons of other options.
Including VM options. In other words, the size of the RAM, the CPU, if you want a GPU, God knows whatever else, right? All that fun stuff is configurable.
In addition to the D sub command, there are other commands. We saw dstat and dsub. Dstat is the one that we use most commonly just to query how would you say the state of where things were There's also an option called dash ssh
The way that works is if you fire it off with DAS SSH, which is what we did when we ran it.
What actually happens is you end up having two containers running on the same VM. One container is the one that is processing your job. The other container is literally just for SSH.
And DCEP then provides you with a, how would you say, a command line that allows you to SSH into that second d sub or excuse me, that second Docker container.
And that second Docker container has connections into the other containers so that you can see and probe what's going on.
So it's a nice way to troubleshoot if you see that your Docker container, your DSAP job is taking a long time. Is it because something went wrong and it's in an infinite loop or something or what have you?
So it's a nice way to troubleshoot things. So in summary.
Visa provides a really nice solution if you've got a lot of tasks or a lot of jobs that you want to run in parallel, right? So you got these things, they're all independent of each other.
Hey, run them all at once. Spread them across a bunch of different machines, right?
And what's also nice is all you really need to get familiar with D sub is you just need to have Docker installed. I've already got a Docker container for you to use. You just fire up that Docker container.
And you can start playing with DSub locally to get familiar with it.
If you then want to go do more advanced stuff, more sophisticated stuff of actually using it in Google Cloud to launch many instances to do a particular task, then you'll, of course, you'll need to have a a GCP account and be familiar enough with it so that you set all the permissions correctly using IM and service accounts and enable the APIs and God knows whatever else is in there. I had a real fun time playing around with it the past few days and just banging my head.
But once it works. It's pretty nice.
Here are the links. Basically for DSAB, I posted a link to this presentation. You can then get the links and see what it is. But yes.
There's a GitHub site for DSub, which has very nice documentation. Obviously, we use Docker or Docker is a necessary component about it. There's a link to how to use Docker.
The Google Container Registry, it's just like every other registry. It's just, you know, you've got to have your authentication, all that fun stuff through Google. Google Cloud Storage.
In our particular case, we had a Python script that did the conversion, which made it nice. And of course, we used Git and GitHub for version control and all kinds of other fun stuff. Including this presentation, this is actually a GitHub pages presentation.
Right. All right. Any questions before we go into a live demo?
Fire.
We've got one. Why one image per VM? Why not?
Process multiple Even if it's an upper bound
Does that make sense?
It does. I think it's mostly for simplicity, right? If you've got a task that doesn't require much resources.
When you launch that when you launch that job, just give it a small virtual machine right you can You control the size for doctor.
No, no. Not parallel us.
Why one sequentially not like process image one, image two, image three, image four in sequence and then kill the VM. Why only… Process one killed a man.
Yes.
Oh, for our use case. For us, we wanted to get everything all done at once. How would you say the startup uh The amount of… The amount of brainpower to use to figure out how to get a single instance to process more than one image.
Was more than the amount of time it would take for us to simply say.
Here's 10 000 Run.
Okay, that makes perfect sense. Thank you.
Yeah. But to your point.
Absolutely, we could have, right? A good use case for something like that is something that goes really quickly, right? In our case, it takes 30 minutes to process an image.
Your startup teardown overhead is order of seconds compared to order of 30 minute processing time.
Exactly.
If it was much closer you would start benefiting from doing more jobs sequentially in a single VM.
To avoid startup shutdown thrashing.
Big up. Yep. And there's
Has anybody has… Another question kind of off topic Has anybody investigated doing genetic algorithm work?
With a setup like this. Because the uh you've got God, unlimited amount of problems to be run and they are data independent They don't depend on each other.
So you could, you know, if you had a machine that could handle it, you could set thousand of them up to run in parallel. They don't interact. When they're all done.
You look at the output from the group and decide what you're going to do next.
It sounds to me like if you're doing that kind of work this would be great.
Anybody out there doing that kind of work?
I know some people who are and uh See if I can ask him what he thinks.
So I had asked, did any of your systems get preempted And I also, it sounds like you had allow preemption a certain number of times before going non preemptible higher rate.
Was the non-preemptible higher rate your error recovery? Or did you have some other logic or does DSub have some logic that if something that it fails, can it resubmit or give a report So you can circle back and mop up what failed.
Absolutely. Right. And in fact, let me go back real quick.
Remember, whenever you run a job, you also get this output, right? So we have a Let me go to the tree real quick. Whoops.
Yeah. Actually, let me go to the other one, the internal one.
So whenever you run a task, so every single line, this is task one, task two, task three, we get this output.
If a job fails, if any of those fail, you'll see it in the output.
When it gets resubmitted, it gets a new identifier. So we will have a chance to go through these logs and basically Yeah, literally the simple thing is, is to go through the logs and grep for errors.
And see why Yes.
Did you have to resubmit or did d sub automatically resubmit?
Yep. Both. And for a couple of different reasons.
Right.
If something got preempted. We did not have to resubmit because we had that preemptible option We basically, we set it, say, hey, try three times and then use a um Here's an on-demand instance. And I think only a handful
Of jobs actually got resubmitted uh resubmitted to the point that they had went to a on-demand instance.
Most of them were completely running on preemptible. In other words, even those that got preempted eventually finished on a preemptible instance.
Right.
But a few didn't.
All right, a couple questions for you guys then.
If I go back to my stats
Or where are my metrics? Here we go. I had a maximum of 4,000 virtual machines.
I had 10,000 images. Why didn't I have 10,000 VMs?
I was wondering if you put an upper bound.
The answer is we did not put an upper bound. But Google did.
Okay.
So absolutely, we ran this. Like I said, we did some test runs, of course, first in the morning. And by shortly before noon, we said, hey, we are confident enough that this is going to work. Let's let's go, right? And we submitted all the rest of the jobs
Made sure everything looked good and was running fine. And literally we went out to lunch.
When we came back. We expected everything to be done right i mean half an hour, hey, everything should be done. If we give it an hour, for sure, everything should be done. Even the ones that got preempted.
Let's go back and check it out. Well, when we got back from lunch, we checked it out and no, not everything was finished. In fact, when we looked at it, we noticed we had 4,000 running at a time.
Yes, it turns out that Google sets limits And they've got tons of limits, by the way, on all kinds of things right they have limits on how much networking comes in and out and how many CPUs you can run and where you can run them and what regions you can run it. You name it, they can put limits or quotas on things.
Turns out we had a limit of 4,000. Of course, you don't know that, or at least we didn't think to check.
Until we fired this off. At the time, we were fortunate enough to have weekly meetings with Google engineers. And later in the week, we actually talked to the the Google folks and told them what we were doing And their response was, well, shoot, why didn't you give us a call? We would have cranked that up to 20,000 in a heartbeat.
Right. And we said, yeah, you know, by the time we would have gotten one of you guys on the phone, it probably would have finished anyway. So, you know, not a big deal. But yeah, if you've got a good working relationship, which I strongly recommend, if anybody is doing cloud computing, have a good working relationship with your
With your account executives and your engineering team, your tech folks um because In this particular case, yeah, we could have called them up and cranked it up to 20,000 if we needed to. We didn't need to, worked out fine. But they're also really helpful from the perspective of
They know things that's coming down the pipe long before anything gets announced, right? And we were running into some funky issues for a few things.
And we talk with them and they go, yeah, yeah, we know about that.
Give it about a week. We should have that resolved by then. We'll make sure you're the first to know. So it was kind of nice to have that kind of a relationship with them.
They also get, you know, we also had early access to a few features Which is really interesting, at least for Google, to have early access because The joke is kind of, well, everything on Google is in beta anyway. So what does it mean to be in early access? What it meant is that we got to see the bugs of their current stuff or their new stuff even earlier and see how things can break really quickly and really massively. But it was kind of cool when
You have to keep internal dog food You got to eat the internal dog food.
What's that? Absolutely. Absolutely. We did. And of course we gave feedback on what happened, what worked, what didn't work. When things worked, it was pretty awesome. When things didn't work, well, hey, we got early access to cool things. And yeah, we kind of let them know of what worked and what didn't work. So it was a nice relationship.
So that's one thing that happened.
Oh, whoops. Also about these metrics, let me go back real quick. Here we go. So we had 4,000 VMs.
Why did it take three hours, right? I mean, if you think about it.
One month.
4,000 VMs, great. You get 4,000 done in the first half hour, 8,000 done by the hour.
You should be done in an hour and a half. Why did it take us three hours?
And the answer is kind of the next line down, right, where I say.
Approximately 30 minutes per image.
That's 30 minutes of success, but you may get preempted a few times at 20 some minutes into it.
Bingo, right? And also, 30 minutes is the mean time, right? So for you statistics folks, remember that bell-shaped curve, right? 30 minutes is smack in the middle, if you will, on a perfect bell-shaped curve. We did have images that were significantly bigger than 30 minutes, right? And so you put them all together, plus you have the ones that got preempted.
Bigger images tended to get preempted because, well, gee whiz, they were running longer, right? So yeah.
In essence, that's what happened. We had the bigger images that got preempted. We also had some images that were just crap, right?
Images that were zero size. We had images that weren't really images. They were literally somehow the error output ended up inside the image and that's what got saved. So we had a few of those, not many, but a few.
So those are always interesting to find. So yeah, yeah.
Yeah, fire away, Stan.
This is Stan. I've got a question. Those Once it got preempted, did you have to pay for that time when they threw you out?
That's a really good that's a good question. I don't remember. I think the answer is no.
You still have to…
In other words, you only pay, if I remember correctly, don't quote me on this, but you only pay if it actually completes. In other words, if it goes through the process and it finishes. In other words.
If you get preempted, in other words, if you turn off the machine before you get preempted, yes, you get charged for it. But if you get preempted, in other words, if Google kicks you off.
If I remember correctly, you don't get charged, but don't quote me on it.
The reason I say that is because Even if we did get charged.
We would get charged three cents for the image that got preempted.
Even if it was a thousand images that got preempted.
That's, how would you say, in the grand scheme of our budget, that was not a heck of a lot.
Last time I checked. Three cents times a thousand is three times What, 30 bucks?
Great. Yeah, everybody.
Am I close? If it's 100 image right if it's 100 images, it'd be three bucks 1,000 images 30 bucks We probably spent more than that between the two of us over lunch.
So, um. Not a huge concern.
Excellent question. The true answer is I don't know. But if I remember correctly, you do not get charged if you get preempted.
I could be wrong.
Hey, Robert. Is there any chance you're going to stumble on the answer to that in the next week or so?
Yeah.
I'll look it up. Yeah. I'll ask ChatGPT, see what she says.
Yeah. I mean, obviously it's not a critical thing, but if you do get a definitive answer, drop it into discuss and just let us know.
Absolutely. All right. I had one more question, pre-canned question at the end.
These are the easy answers, things I already know the answers to.
Let me see what they are. Here we go. Oh yeah, which kind of asked, hey, we're all 10,000 images successfully converted. No, the answer is no is because some of them were zero size. In other words, there were things wrong with the images. And so therefore they did not get converted.
All right. All valid images were indeed successfully converted, yes.
Were all valid images successfully converted.
Yes. And that we did verify. Remember that whole thing about, hey, we want to have some time to do QC on it. That's one of the things that we did, right?
One of the things they don't have in the presentation is all that metadata that ended up in another bucket in Google Cloud Storage. We took that metadata and fed it into Google BigQuery.
So we could basically have a database, if you will, a SQL database that we can go through and query everything, all the annotations and things about an image, which was another really nice thing about extracting all the information out of the images.
So we can go, for example, hey, we want all the images that are lung cancer. We only want all the ones that are lung cancer that have, I don't know, a certain kind of lung cancer or, hey, we want this and this or this and all the fun stuff that you can do with SQL.
And with links to the actual images as well.
Any other questions before I get into a demo?
Good. All right. Let's… Let's do a quick demo then.
Demo time.
All right. This is on Google. If I have time, I will actually do it on Google. But for sure.
I'm going to do it in GitHub Codespaces.
Let me see. I am not sharing my screen. Hold on.
Here we go.
All right, cool. Let me just…
Oh, is that your GitHub application you're talking about?
Yeah, this isn't mine, or excuse me, this is mine. This is what I put together. If you want a link to DSub, it's also in here.
So if you click on that Here's actual d sub How to get started and all that stuff.
I basically took all their instructions and put them together inside a Docker container.
And that's what's, these are the instructions for that. The reason I'm showing this is because I'm showing this is because Let me get rid of this. One of the cool things you can do in GitHub is they have this new tool called Codespaces. If you go to any repository such as this one.
Click on code, the big green code button. And if you're logged in, you have to be logged in.
You get this big green button that says create a code space on this branch, right? I happen to be on the GCP branch. And if you click on it, basically you get something that looks like this.
This is a VS Code environment. So you have your left pane, your top pane for editing, and your bottom right pane is a terminal.
I'm going to expand on that, all of those. We get a nice big terminal. This is a Linux.
So if we actually look at the OS release, we're running Ubuntu 2004, right? So it's a few years old, but that's okay.
The cool thing about GitHub Codespaces is that it comes with docker ready to go.
So you can run Docker containers inside GitHub code spaces and so Given that, I'm going to copy my code from the presentation.
Let me get that over here. Where are you at? Go around.
Here we go. You can't see my presentation code now, but you will in a second. So here we are.
I'm going to fire this up. This is taken directly out of the presentation, right? So a desub create a temporary folder I'm going to run a Docker container run this thing. This is my DSub container.
There it is. I'm going to run that. I've already run this before. That's okay didn't go out and fetch it. Normally would go out and fetch it, uncompress it, and all that kind of fun stuff. It is now running.
So if I go docker container list dash a there it is. It's been up for 14 seconds I'm now going to exec into it, right? So I'm going to attach to it Excuse me. I'm not going to attach to it. I'm going to exactly do it. And there is a difference between the two.
So if I exec into it, I get a bash cell. There it is.
I'm going to clear the screen to get things to the top.
If we look at it. It is running Ubuntu 2404.
And this has all the d sub commands as well as the google command so gsutil, Google Cloud, and a bunch of other really nice commands ready to go to help you work With DSub as well as work with Google itself.
So you've exact into the instance. We've already pulled the image. Let me verify that real quick.
Let's pull the image. It's already got it. Let's tag it.
Just in case. I want to tag it with d sub And let's make sure everything is there.
All right, so we have the same image tagged in two different ways. One of them is called DISA. And the reason why is because then I can just use this generic If later on I want to use, let's say, I don't know, Debian or Ubuntu 2604 when it comes out and so on and so forth, it's already going to be tagged with DSub and the rest of the commands will just
So let's actually run a job. I'm just going to do a simple hello world.
So here's so Here's the D sub command. I'm saying dub My provider is local. The image I'm going to use is the Ubuntu D sub, which we just got here. My logging directory is going to be a temporary Excuse me, it's going to be this testing directory within the temp directory. I'm going to specify my output to go here. And this is the command I'm going to run. I'm just going to run this one command.
And I'm going to wait. Waiting is optional, right? If I run this, it literally, it's like running it in the foreground. It sits there and waits for the process to finish before it moves on. If you don't want to
In other words, you don't put in the dash dash wait option, it will run in the background.
It'll spit this out and then you can check the status using the dstats command.
All right, so let's check the output. Let's go ahead and run a tree on our directory.
There's our tree. There's our outlet. There's our logs and there's the actual output. If we want to, we can look at a d sub test I'll look at the output command.
There's our hello world. So we just ran d sub run this command and dumps the output to a file.
Let's go through and do the multi-stage one. Or the multi-task one. Let's go ahead and create some synthetic data and so there's our input file We can cat it. Let's numbered it as well.
There's our input file. Let's go ahead and create a TSV file that describes what we want Not so much what we want done, but where the input comes from and where it's going to go So let's catacash n run.tsv
So we're telling hey The first column is the input. The second column is the output. And these are all the different outputs. Again, the input's the same in this example. The input is the same for each of the inputs, but the output is going to be different. We've got out one, out two, and out three.
Let's go ahead and actually create the job, the task that we wanted to do here's our task. If we cat this Hey, there's a bash script.
We're going to use said to change hello to greeting. In the input file and we're going to take that output and put it in the output file We're then going to take the date and stuff it onto the end of the output file.
We now have all the components in place ready to run it. We now just need to run DESOM.
Let me just clear the screen real quick. Here's our actual d sub command Again, the provider is local, the image is the Ubuntu image our logging is going to go This is our script that we just created.
This is the one that runs said and date. And it's going to do that for every single line that's in the tasks file, which is our run tsv.
And again, we're going to wait. Here's the job info. And they're all running now at the same time.
That finished and here's our output. Here's our original output.
Here's our original logs This is the one that we just ran, the one with the three tasks, right? So task one, task two, task three there's the output for each of them. If you want to see what they look like.
I'm just going to run a quick tail command on those There it is. Again, notice the time This one, this one, and okay, this one hit the boundary, right?
So they all ran at basically the same time. They all ran in parallel.
All right, let's try this once more.
Someone asked the question, why use an Ubuntu doctor image versus something much smaller than is Outline? In our particular case, convenience.
If this was something that we wanted to, you know, that we wanted to squeeze every single performance inch out of it, everything out of it, yeah, we probably would have used Alvine.
But we wanted to get this done quick and dirty. And so we used ubuntu It's really easy to just fire up an Ubuntu instance, say app get update, app get do this, do that, and boom, all the packages are up and running, ready to go.
If you've already got an alpine image and it's ready to rock and roll.
By all means, use Alpine. Or I don't know, maybe you're using SUSE or Red Hat, or I don't know, any of the other images out there. You can do whatever you want.
It doesn't care, right? And this is what is nice. If I remember correctly, one of the things that was really interesting about about some of the queuing systems like SunGrid and LSF that we were using 20 some years ago
Is that you had to make sure that your software was installed on those compute nodes that you wanted to run.
And if it wasn't, you had to request it from your sysadmins. What's nice about here is you just create the Docker image.
The VMs themselves are minimal. They're literally an operating system and Docker and that's it.
Whatever you want to run, you put it in your Docker image.
Robert. Let me expand on that. Let me ask you to expand on that for a second.
And it runs that. Yeah.
Yeah.
It seems to me that if you've got a server with 48 cores and i'm hyperthreaded and all that kinds of poop What's the difference between spawning a VM to run a container versus running 1,000 containers on that server?
Does that make sense? Okay.
Yeah, it does. And that's actually what's happening in this case. If you were to run this locally.
It is just doing that. It's not spawning a VM. It's spawning a Docker container for each of these tasks that you want to run.
Okay, so the actual vm Was that more of a logical construct that you used because it's easier for Google to handle it?
Type of thing? Okay. Okay.
Yeah, exactly. Yeah.
I mean, it brings up the question is, well, gee whiz why couldn't you run this on something like a Kubernetes cluster or something like that and just say, hey, fire this up across all these things and just run it. And when you're done, the Docker container goes away.
Absolutely. The differences, and I don't know enough about Kubernetes to say this, but it assumes that you already have a Kubernetes infrastructure already built up.
The closest thing that I can imagine is you have some kind of an IAC script, right? So infrastructure is code that says, you know what, I want to run 10,000 images will fire up 10,000 Kubernetes pods on all of them
Fire off these doctor containers when they finish blow away the Kubernetes pods.
I don't know how easy or hard that is to do. I've only set up, how would you say, static Kubernetes pods I don't know how dynamic Or how easy is to run those dynamically?
Yeah, you've got two or three more levels of control on a Kubernetes cluster so Yeah, that'd be an interesting comparison to look at.
Yeah, it'd be interesting to try. It also brings up the question that I have is Cloud Functions are getting, how would you say, more and more powerful. It used to be that you could only run a cloud function for a couple minutes. I've seen now Cloud Functions that
At least the last I heard, and this may be already outdated, the last I saw was you can have a Cloud Function running for 15 minutes. Well, shoot, that's that's almost as long as it takes for us to do these image conversions. So if you've got a short job that takes 10 minutes.
You maybe don't even need to do that. You just have a cloud function that runs The only thing is you have to make sure that everything in the cloud function, how would you say, all the packages and everything, all the software that you need for that cloud function is in that cloud function
I don't know if you can actually say, hey, run this doctor container, right? That's what makes it nice.
If you can create a Docker container with anything in it, it just runs that Docker container.
Okay, thanks.
Yeah. All right, let me Oh, dang.
I would love to show this. Let me just try this real quick.
Let me see. So this this is Come on.
Be nice to meet you, doctor. Come on. All right, let's try this.
I'm going to try running this within running this
Google.
Did you switch screens again?
I did. Sorry. One second.
Okay.
Thanks for letting me know. Hold on one second.
Team effort, not the first time that's happened.
What I was trying to do was trying to get it so that
Come on.
Oh, cool. Yeah, I'm not able to get this to work. Sorry. Hey, I'll do it as a lightning talk in a future talk. How's that?
Okay.
It's not that big a deal. It's literally the same as what you saw, but instead of saying local, we say Google Cloud and a few other things, but we run the same stuff. So yeah, Tari.
My Google is just not cooperating with me or my Google Cloud is not cooperating with me.
So cool. Any questions, folks, before I got to take off?
A big, big thank you.
Thank you, guys. Appreciate it. It's always great talking to you guys and answering questions and doing these things. So if you do have more questions or think of things, please post them to Slug, discuss list. I'm happy to answer them there.
And yeah, yeah. All right, folks.
Sure.
I've got a question before you disappear. Would it be fair to say that this is almost or that D sub is almost stripped down job scheduling without much redundancy like you might use in high performance compute.
But a minimalized version for running mostly on a single system that may throw jobs out Because I feel like it's in that vein
Yeah, you basically think of it as having a head node and you farm all these things out.
But that head node doesn't have to stay up even. That head node can disappear.
If you then, let's say, create a new head node and run your d sub command or your dstat command.
D sub has, and to be honest, I don't know where it is Excuse me, it's somehow stored somewhere in the Google Cloud ecosystem so that you query it. I wouldn't be surprised if it throws it on to a GCS bucket of its own or does some other kind of magic.
But yes, but yes It is fairly stripped down.
With the advantage, of course, is that You don't have to set up the underlying infrastructure for an HPC environment, right?
Great. Cool. Thank you.
It sets it up for you. Yeah. Any other questions? I got to jump out.
All right, guys. Take care. We'll see you guys next time.
Thank you, Robert. Thank you, Robert.
Bye.
Yeah, nice presentation.
Thank you much. We'll see you.
God, I wish I had that back in the 90s when I was doing the genetic algorithm. Oh, my God.
Yeah, yeah.
Yeah, that's a CPU melter.
Well, this has definitely been something neat. So.
What, Gary?
It's just so neat. I mean, it's something that costs Les ran faster.
Oh, it doesn't seem to, does it?
Something definitely wrong here. It never works that way. Nope. Well, think of the old timesharing system here you're buying a 10,000 square foot room of computers for half an hour yeah yeah Oh.
You know, if Google can keep them busy their prices you know, pretty small.
Oh, let's see. I'll give preference. Anybody who wants to ask questions about D-sub in the talk tonight But let me also just open it up to general questions at this point, since we didn't have time at the beginning to
Pursue just general questions. Anybody got? Any questions you'd like us to chit chat about?
Who's going to win the Blues game? Let's see. We got to do a D-sub submission. It's got to be preemptible.
Oh, boy. I can't believe they're not going to televise them.
That was amazing.
Okay, let me rephrase it. Who did something interesting this week?
I had…
I fixed my UPS a couple hours ago. And unfortunately, I broke it.
Lee, are you stable on power now? Okay.
We're getting there.
What happened, Lee?
Oh, it's a long story. I mean, you want to talk about UPSs or generator
Well, I take it your power from UE quit.
Well, Amarin died Friday night. Oh yeah, I came back Monday afternoon.
Is it back yet? Okay.
Which is good because the freaking generator was crapping out every half an hour monday morning That was a pain in the butt.
I just… No, it just decided to quit.
Was it overheating?
It was throwing a low speed fault like it was running out of gas, but it can't be running out of gas there's a 750,000 BTU regulator eight feet from the freaking generator And it's putting out 11 inches.
Which is like double what you need for the generator So the regulator at the generator shouldn't have a problem at all.
Sounds like maybe you need to investigate that.
Well, I am, except I got to do it carefully because it's going to be a warranty claim.
I still got a year left on the warranty
And any other problem, unbeknownst to me I have two UPSs feeding the in the network room which also also feeds my desk.
Yeah.
They run a new transfer switch. So you got the primary if it goes bye-bye it switches to the secondary.
Well, one of those UPSs died.
It showed proper status. Show it online. But if the AC power went away it just shut off.
Oh, God.
So now I got four UPS in there.
Didn't you have some fun where the power coming out of the generator was upsetting some UPSs?
It wasn't as stable as I would have expected. Because you can actually hear him clunk in the next room when they switch online.
I should say the i should say the 3K ones. I got two 3ks And I got four 1500s.
The two 3Ks go in the server room, run everything there on an ats And the four 1500s cascaded You know, they got two cyber powers and then two trip lights behind them.
Create two power sources for the ATS and the network network room.
Network closet.
What happened tonight when you lost power?
The power cord, the temporary power cord I made for the 3K so I could plug it into the UPS.
Oh, shit.
I think a connection came loose in it. Well, I mean, I wasn't exactly taking my time doing it.
I have it down for like five or six hours getting that squared away.
I hear you.
Yeah.
We tripped over the course.
Oh, we'll tell them the truth. Stan wanted to be a co-host and so i have the cursor all set and I push the button and on fan, everything went black.
Sounds like fun.
It is when you get it working.
Fun was a request I worked on today. Somebody asked me to report all the versions of Java installed on their servers.
Let's just call it 40 servers.
Find exacting all versions of Java and creating a spreadsheet. Turned out to be about 450 instances of java almost more than 10 to 1.
Oh, God.
Some machines had two. Other machines had 20.
Turning it into a spreadsheet to send to the person that asked was not fun.
Those are all billable time though, right?
I'm salary.
Well, the company's making billable time, right?
Um… it's customer, I don't think it's billable time i think it's just yearly contract type thing.
Oh, okay.
But in a situation like you doing it, yes, it would be billable time.
Well, people are thinking of any other questions and i i regret I didn't say this a few minutes ago.
So coming up next month.
I've got a couple.
Do you know? I mean, you've been considering and and phishing and and hounding people.
But nobody that I know of has said, yeah, I'll do it.
Look at the time. Got to go.
Let's see what we got here. One thing.
Hi, Grant.
Gary, if you asked me last minute.
Let's see. So we got Grant Taylor doing a presentation on… Oh.
Suited rmr Tild G. Meyer.
The dash preemptible, apparently. Okay. Next month, the Wednesday night meeting, the tutorial topic.
We think is going to be on bash variable difficulties. It'll be being done by Ed Howland, a long time member.
And the… The main either Wednesday night or it may get flipped over to Thursday night.
But one of those nights We expect to have migrating from migrating back to Linux from BSD.
One of our longtime members of our long-term Brian Bilgery.
Has uh He's always admitted that he likes BSD better than Linux so kind of stopped using BSD a few years ago.
And I've been concentrating. I'm sorry, it stopped using Linux a few years ago been concentrating using on bsd And so he's building himself a new box and figured, hey.
I should get back and try some Linux stuff again and uh So this is kind of a going to be the experience of Oh, yeah. I forgot there's that difference.
It was so much better on BSD. Oh, this is a lot easier than BSD.
It's just going to be a whole bunch of Observations, realizations of realizations migrating back and forth between the two.
So I think it'll be Wednesday night. It may be Thursday night, but don't have that nailed down yet.
Oh, remind me, I got a good tutorial on there. Oh, good. How do you rescue your firewall when it grabs out?
Not that we've had experience of that.
Any other funny things that's happened to anybody?
Nothing I care to admit to.
Yes, Judge. No, Judge. I didn't do that. Just ask my lawyer.
He'll tell you I didn't do that. I don't care what the recording says.
Absolutely.
Speaking of which, should we go ahead and kill the recording?
I think so.
I guess so. Okay, Gary. Okay, this was the uh monthly St. Louis Linux user group meeting This is Thursday.
March the 20th. 2025. Our speaker tonight was Robert Sutick.
And his topic was using d sub to submit Linux jobs to the Google Cloud Platform.
It was a very good talk. Thank you all for coming.