Again, Gary. Okay. Welcome everyone. This is the monthly St.
Eighteenth of April.
Louis Linux user group meeting. It is, Thursday. The today Okay. Eighteenth of April.
2024. This is a monthly meeting. It's part of SLUUG, the St.
Louis Linux Unix users group. And, see as usual with our meetings on for this we Let me start with announcements and.
Then we'll have our feature speaker in just a few minutes. We have one speaker on Thursday nights usually.
With that, by way of announcements, on the calendar, I think our next meeting, standing off hand when that is.
5 days from now, it's the new. The new Linux meeting on the fourth Tuesday of the month.
New Linux meeting basically is for, anyone who considers themselves the newest person in Linux.
And that meeting, the way our meeting that meeting works for us is that person gets to start the questioning.
And name the topic he'd like to talk about we talk about that so he's kind of satisfied with the answer and then we go to the next least senior person in the room.
And we still want more senior experience people to come because yeah, you are the ones who provide the good answers.
It's amazing. You don't have to be an expert. A lot of time people just really benefit from hearing how you did things.
What you've done before so for new users and more experienced ones please come the slack meeting comes up on the first Thursday of each month and that is the St.
Louis, computer club. And basically it's it's More hardware oriented will even talk about Microsoft Windows systems.
So in the old days people drag their hardware in. It's amazing. Somehow we've been doing this remotely now for 4 or 5 years.
And we still manage to diagnose people's hardware and make recommendations. Without our hands on it. So, Let's see.
I don't think there are any major changes in the dates or, timing. Stan Lee, am I correct on that?
Just regular dates.
Yeah. No.
Okay, yeah, I think so. Yeah. He, see, oh, there's a couple of just odds and ends coming up on the calendar.
I think it's the seventieth anniversary of okay. Cells, photoelectric cells.
Which surprised me. I wouldn't have thought they were that old, but, that's on our calendar in about a week or so.
And really odd one, it applies to our group. Is there something called the Bell Labs Prize?
I think it may have started or at least been promoted. Became the new owner of Bell Labs.
And usually it was about this time of year that the applications were open and anybody could nominate somebody.
And amazingly enough. There's no indication this year of applications process. And it looks like Nokia is now part of a consortium for something called the Millennium Prize.
And, but there's, I haven't found anything saying that the Bell Labs Prize has ceased to exist.
So I'm just curious if anybody stumbles on it. I just like to know.
It's just kind of, it was nice to still have something named Bell Labs, since that's where Unix started.
But it may now be out of existence. So.
I think only as old farts know what Bell Labs was.
Yeah, yeah. Well, no, Kia did try, they put the name back on it and, it was called Nokia's Bell Labs Research Department.
But, that's the way it had been for a number of years now. And I don't know if they're still doing that or not.
I guess I haven't. I haven't found any. Any press release or anything like that about it.
So.
A few years ago they had the fiftieth anniversary of Unix. And that was held at Nokia Bell Labs.
And they had I think Ken Thompson was there and Brian Carnehan and of course Dennis Richie has passed away.
So they had quite a number of the luminaries from the, from the old group. That were there and they gave they gave talks and things like that.
It's all on YouTube. So you can search it down, look for fiftieth anniversary of Unix.
Oh nice, nice. Very cool. See any other announcements a lot of times, we just use through the calendar or anybody.
See anything interesting coming up they'd like to point out.
But, when I enlarged that font, a little bit stand. Kind of hard to read.
I thought it need new glasses.
By the way, on our calendar here, we try and, put our own meetings in, red.
And that way, you know, I can see this stuff is actually run by us. But,
Okay.
There we go. And then we've got it divided up. We're still kind of not doing it very well.
We're trying to make it so that there's separate thing you can click there, which is what, the cursor is doing right now that'll show you.
What meetings are local here in St. Louis. And then you can also click on a remote. So.
Meetings that seem to be of things that are usually interest to the membership of this group but maybe being done from silicon Valley from London.
From Australia, any place else in the world. And, the nice thing about post pandemic era is so much of that's now available online.
So, we figured we'd advertise it when we can. Course National Beer Day is the most important thing on our calendar.
So.
What day is that?
That's that was the seventh of April.
Oh.
Personally, every year I celebrate the repeal of prohibition.
Yes! Yes!
That's a great thing to celebrate.
Is there's actually a bar on Long Island, called, what does it call the 2020 first amendment.
Is that the right amendment number? I think that's the repeal. And, and that's, the theme of the bar.
Celebrate the repeal. So.
Yeah, that's funny.
Alright. Okay, any if nothing else on the calendar, QA, we usually throw open, kind of a A short Q&A if anybody wants to ask a question or make a comment here on the front of the meeting.
There will also be an opportunity at the end to, to follow up and do. Longer or More random Q&A at the end.
It's top. It can be, but anything you want to ask. Helps a heck of a lot of has something to do with computers.
If you want to answer this.
Okay.
While people were thinking, people who need to be introduced. Let's see, our chairman of the St.
Louis log is James Conroy. Okay, take a bow, James. Yeah.
Here, he, he, I am a, merciful master.
Yeah. I'm Gary Meyer. I'm seeing, this evening.
Lee Lammert is also here in the Bat Cave and We use our treasurer and.
Okay, principal at Omni Tech. Comedy Tech is nice enough to host our our computer system and.
Our Zoom Meeting Stan Records or vice president and, he's been doing all heck of a lot of work on production doing things like keeping our archives straight and making the website and, actually done a lot of good things for us there.
Let's see. What else we got here? Volunteers need it.
We're Bach is our mail server mailing list server. It's also we keep our archives of previous meetings and so we're going to do an upgrade, from Debian 10 to Debian, 11 or 12.
So getting together a group of volunteers to, figure out, okay, we do this, we do that.
How do we, how do we do this so we don't lose anything and do it in the best way possible.
I know everybody kind of does this for their own system at home. And it's 1 thing when, you know, you can turn it off and on and if you do something wrong, it's just your skin in the game.
For those of you who don't, in your regular day job, do the the OS upgrades on, on the machine you're using.
This is your chance to kind of participate, how to plan doing a multi-user system that has multiple web pages, multiple email accounts, multiple everything.
And how not to get in your own way on something like that. So if you like to volunteer, it's a chance to get your fingers a little dirty and work with the gang here and you and hopefully how not to break it, hopefully how not to break it.
So, so if you're interested, drop an email to, There we go.
There's the email info@absolute.org. And then we'll add you to the system in mail list.
That's where all our discussions are taking place on the technical stuff.
Okay. And let's see. I mentioned they were in red, okay. And See we hold a separate steering committee meeting.
It's not to keep anything secret. anybody who wants to come is invited. It's held monthly and you'll find it on our calendar.
The only reason we do it separately is the concept of some people just don't like to see the sausage being made.
So, you know, it makes them hungry. Yeah, makes them hungry. So if you like seeing saucers being made, come to the
Okay, we do record tonight's meeting the Linux meeting and the general slug meeting. Those who record it and we put them in our archives so people can go back and listen to them.
Watch them later. So if you object to being recorded tonight, either don't say anything or ask us to cut you off the recording.
So, But most of the time nobody's made and made an objection to that.
So. There is like I said an archive and you can go in and find a lot of our old presentations and we always appreciate Okay.
It's also shown on every one of our web pages in lower right hand corner.
Excellent. Yes.
Okay. Anybody think of a question they want to ask?
Okay. We're just a couple of minutes early. So it sounds like I'm solving. I'm just trying to hold off in case somebody didn't plan to join us until forward to till.
Goodly number of folks, I guess.
So I'm gonna do a slew introduction here. We had Noticed, this, hack this back door some of our members thought in and Put it on our discuss email list.
The day it showed up out there kind of guess that was the first day it became public. It was about 20 days ago.
And so we've been talking about it we thought okay Sounds like an appropriate topic for a meeting this week.
I was trying to get some of our people, I should say that the guy who revealed it, is a software engineer for Microsoft.
I believe he works out Silicon Valley and, so I figured that'd be good chance.
We've had speakers from Microsoft back last fall. And, I thought maybe those guys, might have somebody in their organization since it was their organization.
They revealed it. And but they wound up on a big project this week and they couldn't help us out so at the last minute when I was about to give up.
I, thought of talking to Andrew Denner up at the CIA log. And, so I literally called him at the last minute.
He was actually in the process of delivering his first version of this talk. When I called him on the phone and so thankfully he was already started putting this together for the CIA.
Oh, CIA. You know, there's a rumor that CIA isn't what you think that it's actually Central Illinois.
But that's just a cover. I'm sure. Yeah. So, so anyway, Andrew, was nice enough to, polish up the edges, brush it up and have it for us this evening.
So we really appreciate. Him putting it together and redoing it again here twice in a week for us.
And, Andrew, when he's not, being the president of the CIA lug, on his day job, he does a lot of Python, a lot of Linux and a lot of big data wrestling and, I'm trying to remember the name of your company that you work for up there, Andrew.
I'm gonna let you, tell me you work far here later. So, But with that, I guess we're up to time.
It's quarter till. And, I'm gonna turn it over to Andrew.
Alright, well, St. Louis.
Okay.
But, you're all sure my screen here. And hit play on the slide deck. And so, as, yeah, your instruction, pointed out.
We're going to be talking a little bit about that. Xv vulnerability. What it means to, the community as a whole.
And then also why it scares the hell out of me. Because this really is not a great thing on what it means for the.
Sort of be the entire Linux world and free free and open software in general.
First off, I want to apologize. The slide deck is not nearly as polished as what I'd like.
This month basically for the, Centre, I will Linux users group, I, to find someone else to give a talk and yet totally slip my mind.
So like Monday. We ended up on this topic. I mean, I was totally distracted by stuff like taxes, the state science and technology fair of Iowa.
A little that seem to happen and then also of course a little 16 month old. Who just likes to eat up a lot of time as well.
And of course, here's some photos from, the set eclipse.
I went down a little bit further south than you guys into southern Arkansas. Or in the northern Arkansas and, we were, smack dab in the middle of totality.
So it was a great experience, but of course. A little bit distracting. As was indicated, this presentation was given to the Central Iowa Linux users group.
Last night and it We do meet every third Wednesday. Each month and Most meetings are a hybrid approach.
So if people want to jump in, just go to CIA log.org and, you'll see the announcements and all of the fun stuff that we do here a little bit north of you guys.
My general, intro is that by day I sling code for a small North American seed company called Corteva.
It's the old, pioneer. And I deal with a lot of bigish data in scientific computing, a little bit of HPC, a little bit of cloud, Just a little bit of everything.
I do have some social media connectors. Depending on if, Elon ends up burning down Twitter.
I'm there. I also have some other backups as well. And of course, we wouldn't be in a Good presentation unless I have the standard disclaimer that, this is completely my own, presentation and not my employer or anything like that.
Nor, the commissioner of baseball, he hasn't said anything about it either. So. We're on our own here.
Okay.
Yeah.
So, I do want to think, one of the CIA log members, Sean Flattery for, most of the information, in this, slide deck, he gave a brain dump in our, slash IRC channel.
And I have to say that this is one of the only presentations I've ever. Basically made most of my slides out of one tweets infographic.
So it was a very, info packed, slide. So. We will go from there, but, all of the links that he provided as well as a couple of others are provided at the end of slide deck.
So if you're curious afterwards. We can dive into that a lot further and deeper. Then what I'm going to go into tonight.
So basically, if you get nothing else.
If I could interject for just a second, Andrew. You yourself put into the chat. A URL that would let people just click on that and bring up the slide deck that you're currently going through.
So if somebody wants to look back a couple of sides, They can actually click on it right now and have it have it in front of them so Just wanted to point that up.
In the chat.
Yes, it's in the chat.
I'm just looking at it and I don't see anything in there.
You might have to scroll back, And you.
It is also on the website in the. What's it called? The description of everything.
The abstract.
They're got to fill up. Now they show up.
It is posted again. Yeah, and as. It was just pointed out it is also on our website.
Cool.
And let me get the meeting chat out of the way and go back, slide here. But so anyway, if you get nothing else out of this presentation, this is the TLDR.
You can go grab a beer. Have a hamburger afterwards. This is the stuff that really you need to know about so the xz utilities is a compression utility Somebody snuck a back door into it.
You don't really know who. Based on the way that they did it and how long of a time this was.
Everyone bets that it's a nation state. Which one? Flip a coin. If you have versions 5.6 dot o or 5.6 dot one On your machine, you probably should roll it back to a previous version than that.
In theory you're still not in a great deal of trouble unless you also have SSHD running.
Because that's the vulnerability however the experts that read the stuff that was snuck into it You have admitted themselves that they're not even sure exactly what the heck all of it does.
So. It's best if you just kill it with fire and get rid of it. But is.
Oh, how popular, how popular is he? Is that like in the standard Linux install or is that like part of?
Yes.
Is that a requirement for other packages or?
So it is used rather heavily, because it's also the, with the LCMA, package.
So anything that's using the LDMA to compression. Pretty much everyone's pulling directly from that.
But, the.
Yeah, I got it on my system and it's an old
Yep, so the good news is, here in a few slides, will be going through who is vulnerable for it.
And, it was caught. Thankfully in the nick of time that only a couple distributions and mostly the ones that are famous for being bleeding edge sort of stuff.
We're actually, had it rolled into their packages. But it's 1 of those things that this really It least from where I'm looking at it.
This is a true near miss. And it almost made it into the production world. And was really bad.
And basically, this guy who worked at it works at Microsoft. I mean, he's famous for more than that.
He's also one of the, maintainer for Postgres. Noticed that there was an issue when he was logging onto his machine.
And. It ended up being that his SSH session took a half a second longer than normal to log in and he started digging in why and notice that something didn't smell right and raised the alarm.
And like I said, unless you're a bleeding edge sort of, distribution like, Oh, sit on stable.
The Debian unstable got hit by a couple of the betas were. Stuff upstream from Red Hat.
How are you Linux, the beta?
It got hit by it as well as, Oh. Yeah, KOLY and, Arch.
I got burned by it, but.
But it was in beta and it never went to the.
Yes. Yep, so the good news is unless you're running dangerously on the edge. Which we should probably have a talk later.
On whether or not you should be doing that in production anyway. Odds are This is really a near miss.
So, and the cure for it, like I said, is to roll backwards to, previous version of XV and it'll be okay.
Yeah, mine's got Actually, my, went to the current 22 0 4 long term.
That's just got 5 2 on it for XZ.
So I'm.
Is it?
Yeah, shows my current Susie, it's 5 2, 5 2 dot 3. Yeah.
It's 1 of those, thankless little libraries that people don't really think to update very often.
So the good news is, like I said, if you're slow to the party, you're still okay.
Yeah, I just looked at my.
Susie.
Oh, what's that operating system we like so much? . Yeah, mine's sitting at 5 2 3
Yep, the bigger concern comes on what this actually means for, the community. And how this hack was done.
And what it means for, thrust and. Wondering, okay, we found this one. What else has been snuck into where else?
And can we trust anyone?
So.
That if you can't trust your maintainer. And contributors who can you trust? So. But we'll dive into those concerns here in a little bit.
Like I said, this was just the too long didn't read, part of it.
So thanks to this. Twitter post. Because it has basically almost everything you need to know about it in here.
So this is basically what most of the rest of my presentation is based off of. But we'll go ahead and zoom in here, so that people can actually read it.
If you're like, Stan and haven't been to the eye doctor here in the last little bit.
Yeah, you're talking about me, son.
Okay, Steve, sorry, it's hard to tell when it's a faceless, crowd here.
My apologies, Steve.
But so anyway though.
That's all right. Put my things up there. I thought that just might upset people
Yeah.
I mean, I haven't broken my camera yet, so it's probably okay. But, so the, XV, outbreak, which is also known as CVE.
That's
2024 30 94. What is xv you tells it's a set of compression tools that create it's big claim to fame is the dot x, the, file format.
That is just like GZIP or all that stuff. It's just using LCMA 2.
As well as a bunch of other stuff. Uses the, the LVMA, library for, compression in it.
Notably one of them being SSH and SSHD. So. It's famous for its Uber-high compression ratios.
It doesn't end up being a little heavier and abusive on your CPU, performance wise versus some of the other ones like Gzip or But if you want to make small stuff, it's really, a handy tool as a 1 call out is that you should not trust that it's long term.
Sort of reliability for that they're not gonna change the format or do any other shenanigans like that.
So you shouldn't trust it for archival purposes. Plus, it's just somewhat unsuited for that just for the fact that it doesn't have, nearly as much caching or other stuff that.
Jesap has or stuff like that but if you want to do like across a wire transfer of files and it's, you need to get from here to there.
XV is a great one if you have bandwidth constraints or something like that. There's also been talk about bringing LGMA to into like chrome is one of the the compression standards that you can do across wire.
Now, whether or not it's a good idea for you on your web server, enable that eventually here as soon as that rolls out.
That's a whole other discussion, depends on how much traffic you have as it's gonna. Jack up your CPU usage and all stuff like that.
But especially if you are. Trying to transfer data across the.
String and a couple of tin cans. It's great for the fact that it makes a really small cross wire sort of.
Payload. On 20 ninth of March. Andreas friend, who is a recently hired principal software engineer at Microsoft.
Also known for the the fact that he's 1 of the maintainers for Postgres. Noticed and notified the.
Security community that there was an issue. In XD slash lib LCMA versions 5.
6 to and 5 6 1. And, it was his email that he, sent out basically saying hey there's some odd symptoms.
Using on Debian SID, over the last few weeks. I saw my login with SSH was taking a lot of time.
I got Bell grind errors. So the impressive things there when he says it was taking a lot of.
Cpu in time, it was a half a second. And just anyone who can actually reverse engineer that something was bad.
In Bell grind other than the fact that you have dangling memory. Pointers is impressive to me because I mean.
I just can tile a normal program in Fella Gr's probably gonna yell at me in one way or another, for those that aren't familiar with it.
Bell grind is a memory debugger that looks at software to see do you have like a pointers that, leave memory hanging or stuff like that.
And. So we, it's a great for if you're trying to figure out, do, do I, did I not, in a pull my memory or stuff like that in C plus plus or C programming.
And so he was able to identify what the problem was, where it was and how it happened. And called out that there was an issue there.
Andrew. Andrew, could you kick up the font on that please?
I can try here.
And oops.
It is just an image here. I know in,
I want the PowerPoint slide and take it up yourself. He gave us the link.
This is true. This is true.
Yeah, so I know, in your, while you're viewing the screen, you can actually zoom in.
On the screen itself.
Really the intent of this was not necessarily to have you read the full text it was more the shout out of Here's what the email looked like and it was a.
Sort of boring, post. So, we'll go to then even more boring and even more dry thing that is the, actual cv itself and basically it's a full write-up of everything that went wrong with it.
If you want to, you can, follow the link and read it. It's even more boring.
Which you should be.
But basically, in the dry, call out that there was malicious code, it was from a rebuilt object.
That was hidden in a test file. And modified. Functions inside of the code. And then was snuck into the the tarball itself not in the actual source code.
So we'll dive into how that works here. By jumping back to. The the GitHub summary here.
So who did this?
Okay. Maybe you'll go into it more, but I would. Like a full stop moment of.
Okay.
It's not in the upstream source code. It's modified downstream.
That's correct as part of the build process. Yeah, well, so, it's in the, actual.
Heyload is hidden in some unit tests. That were. Added to the, live LCMA.
Build process and then as part of that build process. It modifies. Itself, and injects itself into the end artifact.
Of the build process.
So if you
Sounds fairly sophisticated.
Yeah.
Yes. As we'll dive into here in a minute, it is, a very clever, very long con sort of game.
And that's part of why people have said this is probably a nation-state sort of. Hacker, just because.
Nobody else has that kind of patience for this. Because. Essentially this, user, JIET.
Okay.
Back in 2021 created a GitHub account. And then slowly. Worked his way up to doing bad things And as you can see over the space of several years.
East slowly snuck in the bits and pieces of the. Process and eventually then brought about the bug in the end.
So. This is. Impressive just on how long it took him to get there. The other thing of note is that he was a maintainer.
He was granted maintainer privileges. Of this repo. And so that's 1 of the big shout outs of the fact that He did a few things.
And, basically then said, Hey, I want to be a maintainer and. Then being the normal open source people the system worked as expected.
There's someone willing to do free work for us. Of course we'll let him, help out.
Why wouldn't we? So that's 1 of the big shout out to. Why this has big implications for, getting people to be able to commit and work in these.
Free and open source, tools.
So this, only a problem in open source or is it, is this possible? For somebody to be this committed in.
Full source or commercial.
Methodology could probably be done in anything.
Yep, so, the biggest, shout out there is the fact that if it takes an extra half a second for you to log in to your Windows box or your Mac OS You're probably just gonna curse at your antivirus or your corporate, nannies that are installed on it or whatever.
And you're not going to be able to reverse engineer that there's a problem here.
So it these issues could be anywhere. It's just going.
So this something that's gonna be usually found or usually not bound?
That's the concern. It's not going to be found until it's too late. Unless there's somebody nice at Microsoft who's willing to dive into their Windows, build and say, well, why is it taking a second to log in?
This isn't cool. Or why does it take so long for Microsoft Word to open or something like that?
Which I mean that's just another day that ends in a Y for most of those.
I think one interesting thing. Where open source is especially vulnerable is there's no background checks for the maintainer, right?
If this person wanted to do this to some proprietary software, he'd have to be hired at whatever company is producing that proprietary software first, right?
And so there may be some background checks and stuff there. So I think that is one door, at whatever company is producing that proprietary software first, right? And so there may be some background checks and stuff there.
So I think that is one door where open source is more vulnerable than proprietary is just the fact that Literally anybody with a GitHub account can start making contributions and as long as they're you know, as long as the maintainer accepts them and they come in, right?
Yep.
I wouldn't consider that more than a white picket fence though.
Yeah, I was gonna say it's a fairly low barrier. All you have to do is find a intern who gets an internship at Microsoft working on a core level.
Osc sort of things and hands them a bat stack a cache and You you probably can get through a lot of these.
But yes, it does make open source even more vulnerable than. Other places but it's also easier to discover.
And here in the future, I'm willing to bet that it's going to take more effort.
To be able to. We're here. This person did just the bare minimum to be able to gain, the trust and become a.
Committer and maintainer you're going to have to do a lot more work to get that maintainer status and we will talk here in a couple of minutes about that.
For another project. People have already tried this same trick. And got rejected. But essentially, Is this vulnerability?
There's a modification. That during the build process. They they modified this M 4 macro build the host.
And what is it doing? There's some said, shenanigans to, essentially.
Take this test file that is bad 3 corrupt to LCMA 2 dot x. Which is supposed to be a bad, file anyway.
So the irony there is kind of impressive. But it ends up playing some, said golf on it to, Basically strip out and, crypt.
The payload into something that, can be used. And, then from there. There ends up being some stuff in the.
The first build stage, which is a, bash file that ends up decompressing that good large LGMA compressed file that's again in the test.
It then strips away some junk data using a bunch of head tool calls. And then.
Rose away, part of the file. And then does a substitution cipher to get the data out of it.
And then, that ends up being a bash script that then gets run again. And what that ends up doing is.
Deciphers this and other stuff from this good large, compressed LDMA.
And then manipulates the output again with a said script. Then does some more sort of magic. And decompresses it again and stores it as an object.
And then it adds it to. The linking process as basically a dot o file which is just a artifact of the decompression.
And, also jazz up the, linker to make sure that it ends up being LinkedIn and ends up pointing at.
Basically the end result is that if you end up with a SSL certificate that has a special signature in it.
It will cause it to run. Whatever you passed in in there that certificate and will run it, which
SSL or the page certificate.
Sorry, an SSH. I misspoke. Apologies. But, essentially, from that certificate if you, sign.
It with a known. Magic key, it will end up, executing. Your.
Payload that was snuck in there, which of course is really bad. The good news is no one's seen.
Anything in the wild that actually uses this. How so it's we're sort of dealing with it as a near miss right now.
That they didn't have a chance they were waiting for it to become more widespread before they pulled the trigger on it.
But of course. We don't know if it was just a 1 z 2 z sort of thing that they, compromise a specific high value target or anything like that.
But, this, with dot I/O blog post has a little bit better description of how things work.
And basically how things ended up happening and basically here we are back with that here's Eve the bad user who snuck into the build process.
This build the host M 4 script that injected code into the configuration script Then that configuration script. Executed with that bad code.
It then, jazz up the, LLLL, And, adds in some, compiler flags that, into the make file that interferes with the simple resolution process so that it grabs the bad.
Simple instead of the the good symbol that was intended. And then as a part of the make file.
It then. Pauses this bad symbol to or this symbol to point towards the malicious code rather than known good code that that was written in the source code.
After that gets generated into the LCMA artifact and is. Per balled up and I compressed and sent off to all of the different OSs.
And distros to use as. In their packaging process. Then, this bad user comes along with a public key that has a payload.
It, as part of your SSHD, it ends up calling that bad. Malicious token or function which then hits the bad code.
It then is extracted and verifies that it had the the payload. And then it runs it, with system.
And then he has the Houston Kingdom there basically. Which is. Fairly elegant thing the biggest thing to call out here is that one of the possible ways that at least this particular vulnerability could be eliminated would be if you.
End up not building your test artifacts and eliminating that when you're building the actual production.
Distribution code and if you you break apart your build process into the 2 steps where you build everything.
And you run the unit test on it then you throw everything away and build the the actual end artifacts again.
And if you got rid of all that test code, well then none of this would have actually worked and it would have gotten rid of the fact that there's these mystery meet, binaries that.
Or part of the test cases. And end up having this vulnerability.
So is it safe to say that SSHD is dependent on lib. You can infect it and then.
I get into SSHD.
Yes. Yep, so it's 1. One of the ways that you can.
Hang on.
Can either run compression and also you for it's used somewhat heavily throughout many different places including SSH just for the the fact that LDMA is a really great, way to make, your data a lot smaller across wire.
Hang on, flag on play.
My understanding. Take it for what it's worth. Is SSHD. From openness, the Open BSC project proper.
Does not support Lib L. In any way, shape or form. That is pulled in as a dependency.
By various distros. To integrate with system D.
Really?
That would make sense. Yeah, that's where I'm, Not clear on that.
Personally, but it would make sense and it does appear to be pretty much all the main line.
Who?
What's what's worse? Hello.
Yeah, guys, there was an interview with Andreas. That I listened to and that's how he described it.
The reason why that it happening with system D is because system D. Uses, for compression of its log files.
So for example if you use like a journal control you know to to you know search through your logs or whatever right it will might uncompress them right to get to something.
So system D has it as a dependency. And then your system D initialization process, if you have SSHD running on your system, you as a system control or rather I'm sorry system D Right, as part of it startup.
I will launch. And because of that the. Just little vulnerability right because it was embedded in the LCMA would notice that and then and do the injection into the SSHD.
Okay, that.
To go on from there. So that's If you listen to that podcast, you'll get, you'll get the, the full story.
There.
Yeah, I'd be interested in seeing that if you have the link to it.
I don't have the link I did put in discuss a list of podcasts for last week when we did the security related podcasts.
It was a link with oxide in friends the most or this. Not the most recent episode, but the one before that.
Because they just posted one. But I'll see if I can find it and I'll post it into chat.
Yeah.
There's also another component to this. Upstream SSHD and upstream LCMA.
If you compile them on a non system D. System or just stock source code. It's fine.
It's part of. The build process. That those distros use as producing their distribution. That applies the patches to hook.
SSHD with that introduces its SSH. Bad XZ version. System D specific distro build scripts.
It's a many factor problem. Like I could take a compromised system.
Do this compile of SSHD from just check it out with get or pull it down from open BSD.
And it will be fine and it will not include LCMA. It's things that the distribution added as part of their Solution.
That bring in the vulnerability.
That makes a lot of sense. And I do know also that it's clever in the fact that it adds itself to the get ignore.
As well so that even if you did build it on a local machine with. Of the stuff to make it actually happen.
And be triggered. You, and you tried to do a get commit afterwards, it wouldn't end up showing up.
So they were clever in hiding it as well.
Tony just posted the link in the in the chat and that might be the one. If you can ask him.
If that's the one that. That was for that episode.
Appreciate it and my intent of this slide deck. Sorry, I didn't say this earlier, but this is one of those things where this is the moving target and there's a lot to take in.
So I am more than welcome this being a round table discussion.
Yeah, and it's very obvious the bad actor or bad actors. Went way out of their way to hide stuff.
This was a long game. And to me, we're seeing another thing. Like Stuxnet complexity.
Yeah, and the
No, come on. This is dead simple. Okay, it's just modifying a script that modifies the script that modifies the script and a test file on the build.
I mean clearly the the maintainer who's getting paid $0. Obviously should have recognized this, you know, like, no, this is absolutely insane, you know, you know how on the ball you'd have to be to like spot this in like a PR, especially from somebody who you, especially if you don't a lot of time to work on and stuff like that.
But.
Like this is a Yeah, I hope nobody's blaming the maintainer here because this is crazy
No.
And the problem was this person actually, was one of the maintainer of the project.
And I think I had heard the old maintainer. Might have bowed out. Either life problems, you know, we had a pandemic going on and various other things and GIA T stepped up.
So even an old maintainer wasn't necessarily around. Is something I had read multiple times.
Well, and this particular thing that, this, packages doing that, exposes its vulnerability and all this stuff, to execute code.
We're even still not a hundred percent sure that there's not something else. In that payload, that was like a longer game.
Sort of. Thing as well so the people who looked at this, have said, yeah, we're not even sure what all this does.
So it's. This is why it's kind of scary. And why even if you don't have.
Sshd on your machine yet still have all the other stuff there. It's best just to get this thing off.
Your, machine and roll back to a happier version of it.
But.
My when you do that roll back. What about things that were compiled against? 5 6 over 5 6 1 will they also have to be rolled back.
Yes.
Okay.
I mean, I wouldn't trust him.
Yeah.
No, I mean, if you compiled it against a bad library, well. You've got to go undo that.
Yeah.
Yeah, it would it would depend on how you linked it, but I mean, I wouldn't trust it in any way, shape or form.
I say that is it's not just roll back one package.
Oh, there could be dozens.
Or hundreds.
And the
Yeah, exactly. I mean, how many reference this XC? I wasn't even aware of it that it existed, you know.
It's in, it's in old, it's a safe version, but it's in old code that I have.
Yeah, XZ has been around. I think it has become the default compression method. It is usurped busy 2.
Correct.
For kernel source code. And that, so it's been around and in use for at least a decade.
Yeah.
Oh, one question. If I look at a distro and I see that the library or the X.
Module in my distro is 5.2. I mean, that means we're safe, right?
Yes.
Yeah, so.
I would say.
Okay, and well, no, the second half of that is. Since I'm using current And it's 4 point versions older than where that exploit exists.
Doesn't that mean that most, standard distros, again, not bleeding edge? You know, should be should be safe.
With that version.
Yep. Yeah, so. That's a great link here into the, the next slide here.
Who's impacted? And first off the the folks that can sort of breathe clear.
Alpine, Amazon Linux, Gen 2, which really surprises me that they weren't more on the ball.
The bleeding edge. Crazy guys that Okay.
I actually questioned into for the reasons you're describing, the saving grace is system D is West I knew still an optional in its system.
It's still defaulting. To run our C.
Yeah, and that's probably what ends up saving them. But Mint, which of course is just, in a trench coat.
I also question. Since it was dependent on distro build. Modifications. Even if G 2 was compiling the 5 6 over 5 6 1 i don't think it would have the same build modifications and I doubt it would get introduced.
That's, yeah, that would make a lot of sense. Now the, the folks who are who do have an issue.
If you had, from 5 5 1 alpha one to 5 6 1 dash one. Though those have issues currently.
Cali Linux, if you, took the updates between Mark, twenty-sixth through 29.
The open source tumble weed or micro OS rolling release versions from March seventh through 28.
And then of course, Arch Linux being famous for being on top of things. If you had those particular versions of stuff.
Which of course this brings up the even more scarier thing that if you're, using either virtualization or Docker containers.
If you built your, Docker container, off of an arch Linux version and compiled it.
Between 2 24 and 3 28. You need to rebuild your Docker image because it's a based off of a flawed.
Version of arch. But of course the saving grace there would be the fact that the odds of you having a Docker container that also has SSHD running on it.
Okay.
And Andrew, take a look at the chat real quickly.
Yes.
Yep.
I'm I'm looking at a current tumbleweed system. And it updated yesterday. And I believe that means that it reverted to 5, 5.4 from 5.6, right?
That's what I'm seeing there, yes.
Yeah, sorry, I was still was trying to parse the, the first part of your, message there.
But yes, I understand what you're asking there. And yes, I, My, would be that.
They, they rolled it back as soon as it was announced. And that's why if you took your updates between March seventh through 20 eighth.
Was the period that it was an issue.
Got it.
Okay, so anyway, for other folks, this is a current tumble weed and it just put It just pushed and out of sequence update 2 days ago.
So it appears that, you know, they have addressed the issue.
Yep, and most. Most distros. Jumped on this really, really fast because It's this is bad.
And then for red hat, I just like because there's a guy in the central I will, love that is a red hat.
I put them on the impacted list since, Fedora. They had a couple, versions that were, impacted by it.
Okay. Okay.
So red hat itself isn't impacted, but the upstream. To it got hit.
Yes.
Andrew? What exactly? I mean, it was a good idea sneaking it into. You know something is benign as a compression utility.
That was a brilliant idea. But does anybody know what this thing actually does? I for sure don't.
Okay.
So my understanding is the part that we do understand of it. Here when I jump back here to the actual, this, chart.
Basically, if you come in with your public key, that has the, the magic payload in it.
It will end up, taking your payload, stripping out the executable in it that your is the payload.
And then it will run it on the, compromise machine, basically executing your payload with a system call.
So in the end in theory you can do whatever you want
Full route level access RCE, remote command execution.
Yeah, this is pretty much as bad of an RCE as you get.
I just wonder what it's for.
Whatever you want it for.
Yeah, basically. Everything from give me every file on the machine. Virus on the machine.
Exfiltrate all of the data from change healthcare. I mean, take your pick.
Hmm.
Yeah, ransomware is probably the primary target. Is there a making money in ransomware?
Yeah.
If this is nation state like they're talking, it is ransomware is not the target.
This is It's.
The back stores. Back door.
North Korea, gets a significant amount of its money through cryptocurrency scams. So ransomware is definitely an option.
Okay, fair enough.
Yeah, I was gonna say North Korea is definitely one of the suspects. China is another good one.
I mean, NSA. Probably isn't, but I mean, I don't wanna say it definitely isn't because.
This is something that would be a fun one for that. Russia is a good target. I mean, take your pick.
Basically it's someone big enough and clever enough and patient enough to pull shenanigans like this
Are they tracking down that user? Is that like an FBI alert now or something?
I'm sure they'd like to have a nice word with them with a rubber hose to.
I have those conversations but Right now.
Well, I mean, this is, this rise to the level of where it's, you know, a security, you know, US, you know, security threat or something.
Not just, you know, Linux.
I Yeah, It's a very serious one and yeah, it has the the, attention of, pretty much anybody who's anybody.
And I'm pretty sure if they had their their chance they'd love to make the person disappear in a nice dark.
Cell where they had some nice words and maybe hired them for, working for them.
Okay.
Oh, that's the thing. All this happened on GitHub, right? So Microsoft may have that data about like what IP addresses this account was logged into and stuff like that.
But since it happened on GitHub, it'd be Microsoft that would have those logs, right?
Yeah, but if we're talking. Hacker hackers with sophistication that we are. You're not gonna trust those source IPs.
That is gonna be bounced through so many different things.
Yeah, so I mean. Now of course it was over a long enough time that who knows maybe they let their guard slip a couple of times and there is data in there that can be used.
But yeah, that's far more clever people than me are working on that, I'm sure.
I wish I had your face.
Okay.
Or at least far more people with bigger guns than I have.
Yeah.
But, yeah, so, Basically, If you want to be double sure and built in suspenders check to make sure that you you don't, you're not, impacted.
There's 2 different easy there's a harder way that you don't have to install anything of even what you currently have.
Or there's a nice easy tool from the nice folks at jfrog that can tell you if you're vulnerable or not.
There's a typo on that line, the LSF line. The lib, the last lib, LZMA has as a back tech instead of an apostrophe at the end.
Oh, yeah. That's a good call out.
I only do that because I try to run it. I had a debug it.
And look. Unless you're trying to do some, interesting command things on your slaughter.
No, although, it's always tempting.
Yeah.
You pass the test! Yeah.
Yeah, but anyway, basically, if you run this line here, it will tell you which version of the xd you have piping it through rep to see if it matches the reject golf of the vulnerable versions.
If you don't get anything out, you're in the clear. And then even if you did have that, if you run, and basically it's checking to see if.
You have, SSHD. And if, the, . library is linked into it.
If either of these end up being a negative result you're technically not vulnerable but I would strongly encourage you if you did have the, the wrong version of, still on your machine.
Fix it because. Who knows what else bad things are happening. The J frogs tool is.
Kinda cute. I mean it It's been a while since I've seen a good
And it will basically tell you line by line that. No, you don't have the vulnerable version.
No, you don't have SSHD. And no it's not linked with
Can I ask a question?
Yes.
Does this involve Windows?
Does.
The best of my knowledge, it doesn't.
Does Windows services for Linux? Have system D anywhere in it.
Yes.
Yes, it does. Yeah, he has system D and I heard that it was vulnerable, but that was from a podcast.
I don't know if that's true, but yeah, WSL does have system D.
We have.
Yep, because, especially WSL 2 is now essentially, just a virtual machine ish.
Implementation and Yeah, so if you have Debian installed on it or something like that. Yeah, you could be implicated.
And, if it was a couple of years ago, you, you'd be, fine because it doesn't have just be on it.
But now they've since improved it to now having 60 I believe.
I wouldn't call that an improvement.
Sorry, improved.
Right. Well, you go back up to your. Previous slide where you have the command.
I'm just looking at The LSOF first part of the line. I'm interpreting that as you're going to get a list of files.
And then your
But I don't see to me that LSO F the first part of that line is going to get.
The process name. For the system D binary. I don't see how you're taking that file name.
And using it anywhere. So do you have a screenshot of that? Or some permutation of that command running.
I do not in front of me here and I'm on a Mac machine so I'm not sure how it's going to behave but we can definitely give it a try and see what happens.
Yeah, I'm somebody else correct me if I'm wrong, but the LSOF dollar per end through the close paren.
I am expecting to return the name of the SSH D. So, DAY, DAYMAN,
That was my impression of how it's work would be that. That it would include the
It's gonna return a PID. It's searching it's printing too. Octopren too is gonna return the the second column, so it's PID.
Searching and searching ox for SHSD and for listener And then if that's found, it's gonna, you know, strip out the PID.
Yep.
Okay.
Alright, so.
So the LSO F is the dash P is for PID.
Yep, so you'll get the PID of that and then in that it will list the. the, been linked against it.
Thank you. I stand corrected.
And I'm sorry, the
Yeah.
So basically If I remember correctly, this LSOF will list all of the different libraries that it's using.
The first type of grip will narrow that down the which are those specific libraries. That is like are dynamically linked if I remember right And then the final grip is to see if, Lib LGMA is in there.
Oh.
But we can definitely. Pop open a, a terminal and see just exactly what ends up happening here.
In a couple of minutes if someone's brave enough to give it a go
I'll run it. Give me a moment.
Oh, he's putting that together. Yeah.
And we can definitely. Yeah, we can definitely, screen share over to that. Like I said, I'm more than happily welcome, input because I'm willing to admit I am a software developer.
I am not a security spook. And I am in no way shape or form a true expert in any of this other than reading some stuff and Yeah, sort of using this as a learning experience.
Yes.
Hey, Andrew. Did, did you guys had to step away from it? Did you? Talk about the 500 ms?
Like where did that come from?
Andreas? Yeah.
Yeah, so basically the reporter, that reported the, issue in the first place. Was that, yep, Andres was a SSH into his machine and notice that there was a I delay in logging in.
My understanding is that it was essentially the vulnerability was doing some like yeses and other stuff like that it on top of the basically just
The part that, the easy quick, picture showing how the vulnerability works. With doing and it was not not the best of code.
That's why Val grind was finding issues with the the way that things were compiled as well and stuff like that.
So You notice that it took a little longer to log in and his CPU usage spiked. You said that this isn't right and then started digging into what was going on.
Yeah.
Yeah, you're essentially right. And again, I highly recommend giving him a list of Andrews Andreas himself and it's in that it's in that URL that I posted in the chat.
The but what's really fascinating about this to me is the fact that the back story is what's really interesting, right?
So They knew that they had to get into And the problem was Well, system D is gonna launch SSHT.
Excuse me. Yeah, system D is gonna launch SSHD and The system D guys knew. That system D is way too complicated.
So they said, hey, we need to like cut down on this. Attack surface area. So they decided that they were gonna make a PR to system D.
And that was gonna really reduce down the things that that were like what were linked in statically or rather like in dynamically I should say to system D and just to the bare minimum to get things up and running.
And so this PR was scheduled to come into. Focus like in March or April or something that this year that's going to be merged into the main mainstream system decore.
And the The, hacker this, gitan guy. Realized he had a very at a very closing window right of time where he could actually implement this back door.
So he went ahead and just sort of pushed out what he had so far. But he didn't actually have time to sit there and optimize his code.
And one of the things that was going on was that in the observation process so that you couldn't detect that there was actually anything going on.
If you just looked at the raw binaries, for example, if you've got a raw binary and you run strings on that binary, right?
You can see like, oh, hey, there's some strings in here. And those don't look normal.
They shouldn't be in this binary. So they had this like observation process going on. So that it look like raw data like raw binary data.
And then when the code was activated. It's sort of, unobsticated it.
And that code was very poorly done. And this was so poorly done that it was causing SSHD to go into the connections and causing a lot of CPU overhead and that's what led to the 500 ms.
So. For further explanation, you know, go listen to that episode. That's really good. Stuff so I thought that was just fascinating.
How? We might not have been able to detect it. If this PR wasn't imminent.
And I'm sorry, which what are you saying we should go back and listen to?
The podcast.
Okay, I posted in the chat like to oxide in friends and it's the interview with Andreas himself.
Okay.
Right? He describes intimate detail for an hour and 50, sorry, hour and 26 min.
How this whole thing was discovered
Good.
So you want the horse's mouth. He's the guy and all that information I just, you know, downloaded on you was my memory of what he said.
Gotcha. Gotcha.
Yeah, and he's a far more clever man than I am. So yeah, I completely agree with go listen to him.
And of course there's much nuance that you don't get in a Twitter infographic.
But I went with this one because it made it fairly easy to understand. In a small bite, essentially.
But yeah, so. Anyway, and going through the. Way that they were on, office, office skating, the, data and.
Doing goofy things with. Said in Doc and. Decompression of ex and all those sort of things.
Just as impressive the the rejects golf that they were performing in the first place.
But so anyway though. So we've gone over who was impacted. How to check and be sure that you're in the clear And then of course there's a deeper dive here at this.
Github link, that will actually take you through, actually how, the payload was handled.
They have an example honey pot that you can run it see if other people are trying to exploit you.
As well as a proof of concept showing how. You could actually
How you could actually, trigger the, RCE by, stopping in your own, private key.
So it's an interesting, deep dive into how, to actually make things work. And I'll leave that as an exercise if people wanna dive into that, yeah, just exactly how.
How all of this work.
And then the last, real big, thought of. What does this end up meaning for the future?
And, as we sort of discussed before. This is why we can't have nice things.
You have to sort of like how banks know their customers. We've got it now do a lot more work to know our committers and know our maintainer.
And there's even been talk of well you shouldn't trust someone as a maintainer unless you've actually been in met them in person at a conference and they're like some big impressive person that D, and all that stuff.
Okay.
Yeah, but that's.
Realistically, that's impossible and is. Yeah, it's 1 of those things that I don't know what the right answer is and.
It's if you're not losing sleep over this, you're probably. Hiding your head in the sand somewhat.
In this case, we got lucky. And these sort of problems are probably gonna keep happening. In fact, there's a, already a report out that there was a near missed, incredible takeover, attempt that was like the XZ, back door incident.
Openjs Foundation. Essentially they had someone who sent a suspicious series of emails trying to get them to give them a committer status to fixed.
Okay.
Security vulnerabilities that they found. Thankfully those guys were this was fresh on the heels of the issues so they said, wait, wait a minute guys.
Really? No, no, come on. Don't do that. So, this sort of accounts as a near miss.
But, sort of is a call out that if you are, in governance around a project, might want to be a little more careful in who you hand out.
Access to stuff with.
And of course, this further illustrates the old XKCD cartoon. Of, that the whole internet relies on this one little tiny junky thing that some guy in Nebraska has been, thanklessly maintaining since 2,003.
The big example that always comes to mind for me is NTPD. The network time flow to call Damon.
There's like one or 2 guys in the whole world that actually care enough about it to be maintainers of it and yet the whole world runs on it.
The main maintainer of ATP is a He now lives in Colorado, but. He actually lived here in St.
Louis for, many years and, used to be a member of the community here in town.
Yeah. But of course, see every issue in the world, if you can't find a next KCD that, somewhat applies to it, you're probably not looking hard enough.
Okay.
So. Anyway though, that really was the end of my, stack of stuff. Here's the links to, the various different sources that.
Sort of were that info dump I'll happily add on that link to.
Hello?
I think we lost you, Andrew.
You just went.
We heard somebody say hello.
That was me.
That was Steve.
Oh!
Apologies my headset just died Yeah, let me transition out here. Hopefully people can hear me again.
Yes, we can.
Awesome. That's what I get for not plugging it in ahead of time. Lessons learned.
But yeah, so that was, the, end of my slide deck here before I was, the, learned.
But yeah, so that was, the, end of my, slide deck here before, I supply giving out here, but Yeah, so anyway though, that's questions, heckling.
And all else are I happily welcomed.
I just have to say that that was very, very well done. I learned a lot and I'd actually listened to the entire Andreas a talk and there's stuff on there that I didn't know So, good job on the research.
Thank you. And. The only reason why I've seen this far is I've stood on the shoulders of giants that did the yeoman's work here i like i said this is the only slide deck i've ever made where 2 thirds of it was from one, Twitter infographic and that that one infographic basically summed everything else up is about as good as you could ask
for.
Okay.
Yeah, Andreas himself mentioned that, other security researchers are doing even deeper dives than he did.
There is.
On this and I think that's where we're getting this this new information.
There's been some analysis of commits both in the Ex the repository. Kinda.
Current to figure out where the malicious activities might have been committed from. To attribute but also pattern discovery and then expanding out to see if those patterns have been done elsewhere.
And I think you mentioned Open JS was one, but there have been some questionable patterns. Showing up in more places.
And I'm currently searching my Frederick's timeline for the follow-up graphic where that loan project from the maintainer in Nebraska has failed and the pile is collapsed.
Okay. The other interesting thing I found out in this whole thing was this concept of something I'd never heard of before which was called sock puppets.
Right? And so apparently this Jayten fella Feller. He didn't just have him.
Doing the social engineering to get himself maintainer privileges on this on this sexy project. There was the sock puppets now Maybe there's real people. I don't know.
It, you know, if this was a long con by a nation state actor, it probably was real people.
But there was these extra accounts. That we're sort of harassing the this this you know our our quote unquote Nebraska guy.
Who was who was having his own you know, mental health issues. And he didn't wanna maintain this project anymore.
And they were saying oh you need to get in there you need to like do these updates and all this kind of stuff and It is just under so much pressure and finally caved into pressure and and allowed this, Jay Tan guy to become a committer.
And then everything was working fine for a while because the, the gitan committer was just committing straight up, you know, good commits.
For a while and then afterwards you know when this whole time crunch window started happening that's when This guy introduced, I think.
So it was a long con and the the sock puppets helped.
Yeah, I think the really telling thing is how they slowly snuck in. The bits and pieces over.
Building up the infrastructure to do it over a such a long period of time while they were also doing other good work as well.
I would argue. That it goes back further than that. Because to be able to implement the pieces they did.
They would almost need to have started or at least have the finish line and do the obfuscation and work backwards.
Okay.
To where they started doing the commits. This is I'm speculating. A 3 to 5 year thing.
Makes perfect sense, but it's terrifying the amount of effort that went into this.
I was not joking when I said stuck step.
No, we know.
And Stuxnet terrified me. This terrifies me.
Yeah.
Well, if you think about it, we're getting ready to have an election. And. Everybody says, oh, you can't do it with paper ballast.
There's a lot of countries in the world that are going back to paper palace just for this kind of a thing.
And, you know, it's. You know, it's not all little, Tiny places in Africa.
I think People just have no clue. How you can get screwed. On the internet. I mean, you know.
And this last time around it was proven that a lot of these ballot machines were logged into the internet occasionally, not all the time.
20.
So how how do you know it was done fairly?
Well, that's why due process you have to allow manual counting. And multiple times over.
Yeah.
It's
Okay.
And then even day-to-day operations look at the, how much, a major health care, company getting compromised caused for the entire world.
Yes
Yes.
Oh god, yeah. I mean. You know, you need, well, I wish there were some people that Could look at this, but you know when you say the guy noticed at logging in took a half a second longer.
Good grief.
My understanding was this person noticed because they were doing performance optimizations and instrumentation on, I think it was Postgres.
And trying it on different systems and it's like. Why is this one so much different than the others?
And it was digging on that. That they found this like wise. Just SSH logging in.
Take this time. It was luck.
Yeah, that's why I called it a near miss because This really was Just pure dumb look that they got caught.
Oh.
I don't know, I think that's
I'm near, this is more like a broken arrow incident.
The keys were in the bomb. The bomb was set to go up with flying over and got accidentally dropped by the And for whatever reason, we locked out and everything that.
To prevent the bomb from going off. Didn't happen.
Let, let me put a different span. Maybe it's just, a, advocate, but let me put, so, you know, Linus has always said that, you know, many eyes makes bugs shallow or something like that.
And what software has more eyes than SSHD, right? So everybody saying we got lucky that this guy just happened to be investigating this.
But how do we know that once this went into R or a bunch of that it wouldn't have been, Also, another thing that we could look at.
For a positive sign here. Is that, you know, this guy had to do this extremely complicated build, you know, attack lip, ZMA in order to get you know access to SSHD like he had to go this extremely convoluted route.
And that's because going in a more direct way, what has a much higher likelihood of being discovered, right?
And so when the bad guys are having to do this many backflips and literally take several years to So, you know, for their plot to come to fruition.
Oh yeah.
You know, I, you know, you could look at that as a positive sign of like, well, it wasn't easy and it takes serious resources and planning to to pull this off because you know, the normal methods just won't work.
The easy ways won't work anymore.
Yeah, when you were talking about vulnerabilities, we had a vulnerability in the printer spooler in Windows.
Which was on every Windows server. In existence, including the domain controllers.
So it's not that close source is the solution. It's. The fact that running things with the
Minimum amount of code reduces. And you were saying that they were doing that PR that was closing the window to this for they were trying to.
Shore up SSHD so that it was smaller. To reduce its its footprint.
This deep.
System D smaller, okay. Bye bye.
Okay.
No, they were trying to make system D. Yeah. Or make it have less, cause it's sort of like a grab bag of all kinds of utilities and libraries and things like that, right?
Because you never know like the system D is gonna want to launch something right well if it's gonna wanna launch this thing or whatever it needs to be able to to pull it into existence and it might need to have that library, it into existence and it might need to have that library, right?
So they were gonna they were trying to reduce its overall footprint.
Good.
Why doesn't get system have Good spot to alter. SSHD and arguably very security sensitive thing.
I'm not sure that it did exactly. I think this hack in the Elsie and, library was able to get into memory so it was so system D was running it's running is root right and it's launching a sub process so SSH is its child process, right?
Okay.
And it you know it has It had the ability to. Get to the memory of that SSHD and and inject its payload.
That is not my understanding.
I think that's what Andrea said. But. Oh, that's not your understanding.
Okay.
My understanding is. That the vulnerable version of SSHD. And it is a vulnerable binary or a compromise binary, not in memory copy.
Okay.
Only get it. When you use the distribution, namely, Fedora and Debian. Build scripts.
Which apply distribution patches. In this case, patches to add system D support. And system D pulls in, with it.
So it's Only. When you build with distribution built scripts. To create a distribution.
And using system D. Cooling lib XZ. Yeah. It is all okay.
Yeah.
System D pulled in lib. Which it used for. Things like log rotate.
Compressing log files and stuff with the journal. So, I mean, system D wasn't really.
Compromising SSHD, it's a shared object kind of a library like a DL, so.
It's compiled to load a system library and then notify. System D that okay, hey, I'm up and running.
Okay.
And system D takes that and it's got. L, which is built with the corrupted.
X-.
I agree that system D is not. The vulnerability itself. I am questioning why. And init script.
System alters. Critical security. Subsist.
Well, because first of all, system D is not really an in it system. It is a services lifecycle management system.
So to think of it as an inner system, an inner system just basically starts up processes. Where is?
Parts and stops and pre starts it Now that we can fight about that another day.
So it. It's really inaccurate to call it a it needs to know when processes or services have started and stuff like that.
So system D reports that I've started so if you got a process that, that requires system.
Ssh to be running. I don't know, perhaps. Some sort of an SSH file system or something like that.
You might need the server, the SSH service running. Then system D can say, oh, okay, you're up and running so I can either kill Telnet now or I can start the SSH file system whatever sort of a thing.
That's that sort of stuff.
Yeah. How and why do every Unix that runs SSH that doesn't use system B not have a problem.
System D is an integral component of this. It is not the malicious one. But it is an .
One
And it's the fact that basically this. Equivalent of a DL is loaded up and the simple resolution when SSH calls for RSA public decrypt.
It ends up being the one that, wins essentially.
Because it was the first one in or something like that.
Yeah, that was the part that I didn't know.
Yeah, the guys. Yeah, the guys that did this. Attack would have come up with they would have found some other way had it not been that oh look if we can Do some highly obscure hack on a small project that tends to get bundled in the lib.
LA, which then gets used by system D, which can then be used to talk to SSHD.
I mean. Guys at this level, like if a nation state is after you, you are coast.
They're gonna get in eventually and so it's It is, it is kind of like.
Yeah.
It's hard to throw any blame at anything in this scenario because You're just dealing with somebody on a level that you cannot defend against.
Yeah, but we can make it That's the whole point of encryption. Sure, if they've got enough resources, they can break it eventually, but it might be worthless by the time they break it.
The X.
That's the whole point of security. Make it take long enough that nobody cares anymore.
Or at least long enough to detect.
Yeah.
Yeah, I think I think the whole concept of blame in this scenario is not right, right? Because the first news should be blamed is she, you know, the person who committed the bad code.
But they're you know likely not going to be caught so blaming anybody except them feels like a you know just look like a witch hunt to me I I don't like it.
Yeah, and, I, none of my presentation was meant to blame any of the committers.
Or anything other than the bad actor. I really believe in a blameless, sort of, root cause, the examination.
I agree.
Yeah, this is a blameless post more then.
Yes.
And in most, I'm asking why. Does the package you're using to start stop and manage processes?
Fundamentally alter the behavior of a very security sensitive. Application.
Logs.
Say again.
Library was being used. To compress the logs. For services.
Have a problem with system decompressing its own logs. Why does it doing its own log compression altered?
Yes.
My guess is that it was also compressing the logs for SSHD.
What? So.
Yes.
Well, I think it was just that it needed system D was waiting for SSHD to report back, hey, I've started if there's anything that is sitting waiting to start up for me to finish.
I am now finished. Starting up. And with shared objects, dynamically loaded libraries, if you will.
The having having it having something that depends on something that depends on X. Means that it just it's just a chain of things that like Yeah, I mean, that's yeah It's so complicated, but.
System D is doing his job. It's like let me know when you started up so I can Tell everybody else.
And how many other operating systems use PS or something, watch the process tree to see if something is started up.
Hair weight.
That's, they all do, but they're . S. Does it just says, hey, here's a list of stuff in their pits That's what you might get from in a system, but not what you It's pretty kind of almost useless if you're trying to Do dependency managements and stuff like that.
I disagree.
Well, then how come?
Both or 2 of the 3 largest
Companies behind Linux distros so canonical and red hat Both were looking. For some sort of a services management system.
Red Hat actually went with canonicals. What do they call that? Upstart.
That was, that was. In Red Hat for a while. And then I guess everybody decided to go with System D.
So These guys are not paying extremely expensive developers. To come up with a system lifecycle management system.
For shits and giggles. There must have been a reason for this. And competing companies were both coming up with their own system for it.
In the end 1 1 out so
And how much. System D was to me. What most people consider to be the lesser of the evils.
And I also suspect system D. 10 years ago. Was vastly different than it is now. And the central.
Emacs versus VI joke is, Emacs, operating system if only it had a good editor.
Conversely, VI is a great editor if only it had an operating system.
Yeah.
That's why we have Neil then.
What?
It does all the things.
That's why we have Neil. It does all the things.
I obviously not a system, but I feel like system D. Is going the way of master control program.
The original one
Yeah, but even if, but this is about. Like the symbol lookup, right? And so even if system D wasn't used.
There's gotta still be a way to do that. So. I don't think that not using system D can prevent like this whole class of vulnerability right
It cannot prevent the clients. System D was the integral link. That pulled. Excellent.
Yeah. For this particular one.
Right.
For this particular one, but X. Was also used by stuff like CPIO, which is used by in at RAM FS or whatever it's called, which is included in the kernel as I understand it.
So. Other there would have been a different a different door or a different window to come through if Not for this one.
This.
Yeah, this was just a particular vector that they chose. There were others.
This broad class. But the code as I understand it.
Looked specifically or or the stack for it. Specifically for the build systems. System D. And SSH.
And it is apparently I've read multiple places. Only x. 86. It is not, is not risk 5, it is not other.
Architectures.
Yep, x 86. GCC and a Debian based or you know Fedora slash red hat sort of a system.
Yeah, and it was.
I think it was restricted to those somebody paid in chat that it may have showed up in the arch A you are but I don't know that that actually The version of Vexed might have made it in there, but I don't think it would have.
The build script would have. Opened up a vulnerability. In an art system.
Agreed.
Hey, Archie is system D. But it's because of yeah, they were looking for certain source things in the make process.
I mean, I kind of look at it and say I'm I'm kind of weak that with C language and make files.
I don't know anything about them, but when I look at them, I'm like. This is a tangled mess of crap.
How many? No wonder people keep shooting themselves in the foot was C and these make file scripts and stuff.
Well.
I, I don't like them. I find them too complicated. You're very, but I can point to those and kind of say if there is the root of the problem right there.
Make is not make as the tool is not the problem. What was done with make. Maybe a contributing factor.
In some ways make as a shell non-interactive one.
Yeah, it's
3 languages inside so it's extra fun.
Let's throw in there.
Yeah, and see that just means you're sloppy.
See.
And if you're writing code is so complicated, you can't keep track of your own code.
Maybe you ought to consider what you're doing.
Yeah, but people just go for the, hey, it works for me, ship it.
Oh yeah.
Yes.
Yeah.
I mean, especially if your solution is playing rex golf with. It's said in the awk to, expel the data out of it.
That's the point that someone really should have swatted hands and said. What the heck man?
But again, small projects.
Tapa is said and Auck. Shenanigans were part of the build process. So that was of obfuscation.
As well. Yeah.
And, on the systems doing the build. It's not, my understanding is it's not part of.
The vulnerability or compromised system.
Correct, it was part of the build script.
Yeah, so the Debian maintainer or the Fedora maintainers, right, would have.
Created the, the poison pill. Not the, not the, you know, if you hold it down from source and you build it yourself.
As possible you might have gotten infected but I think you probably would be safe.
My understanding is if you did a get clone.
And built everything. Just in a clean directory on a stock system. You would not have a compromised binary.
Because the way the script. Before the build process worked. Looked for and were dependent on. Debian and Fedora.
Build stacks.
And also my understanding is that a lot of these distros will grab the tarball artifact from the actual project itself.
And not necessarily rebuild it on. They, their machine necessarily for when they're building the packages.
There's not a custom build for each time. It's, they're provided by the maintainer of that, package.
One thing that occurred to me, just now was that, There's a possibility.
Oh.
Yep.
That there's some compromise Docker images out there. Because Like you've got these images, right, that are layered on top of each other, right?
And so Those images usually get pulled from Docker Hub. So somebody had done a a build of Debian.
Let's say for example and push it up to Docker Hub and it had this particular.
App, or dot dev package in there or something like that. And then when you built your Docker image based upon this Debian version.
You know you might have a you might have a docker in there and if you had if your doctor thing had SSH in it or something like that might be compromised.
Right.
Yeah, so the 2 big saving graces are the fact that. Like say for example if you were building a Debian image the odds of you using the unstable versions that were compromised are fairly low in the view having SSHD running inside of your DACA image is even lower than that.
Yes.
Unless that was the intent of your darker. Let's just have, you know, SSH running in a Docker, but like for K Kubernetes cluster or something.
And if you running in a Docker container.
What is the level of system D integration inside of? That container.
I don't think so. Yeah.
You wouldn't have system D running at all. Inside the container by my understanding. Because for the most part, you end up running, PID 0, whatever you're executing when you start up the container.
Right, and I think normal containers were in complete agreement. My Question is more if you have a container that is an entire virtual machine.
Analogous to a jail or a slurous zone. You can get into some weird things, but even then I don't know that.
System D would be involved.
And most of that I'd swap your hand and say, what the hell are you doing?
I agree.
I'll tell that about half for the applications on true charts.
Now that you mentioned it, I've got, I've rent a VPS out there and I've got some services running on there, which I've created unit files like system D unit files.
And you know I can start stop do all these different things with it and It's just a VPS.
It's I forget what it's actually running because I never created it. I just rent it but That's got me wondering now.
There is some sort of system D inside a virtual machine.
Yeah, and of course there's a difference between, V VPS and a docker image.
Vps really in the end of things behaves far more like a actual full full on virtual machine in many ways.
Okay.
But yeah, it's an interesting question.
I think it's purely by convention. But I also believe if you wanted to run. Like a VPS type environment in file of a container.
Good. It gets normally not done.
In convention.
Well, does this, change the system? Are they gonna modify their build process or? Are they gonna modify any kind of process to try to connect it.
Catch these things now or?
That's hard. That's You remember how, I want to draw an analogy. You remember how.
When TNT, CIA, whoever prism was. Discussed in the late nineties or early 2,000.
And people like Google started encrypting land links, how that was. Massive flag day. And an industry changer.
This is gonna be one of those types of flag day industry changers.
It's also a very compelling argument for S bombs as well.
Or worse.
Stuff or bill of materials.
I would like to think so. I am not entirely sure. I think there'll be talk about it, but moving forwards, instead of looking at the prism example, I look more at the remember way back when we had this global pandemic and people were dropping dead like flies and people's all of a sudden if you were in a crowd and somebody coughed you kind of like Whoa, get away.
And we wore masks and we washed our hands and And if you're sick, stay at home.
You've you plague rats stay away from me well where is that now we don't necessarily learn and the lessons to.
Come out of this to make to defeat something at this level. I don't know if much will change.
I mean that XKCD cartoon that was probably I heard it mentioned it was probably linked to earlier with like everything built upon a to tiny sliver of support.
That's been around for a while and it reflected at the time it was created something to do around for quite a long time.
And it hasn't, nothing's changed.
Well, after the solo wins back door, you're now required to generate S bombs if you're selling.
Software to the federal government at least. That is definitely something that has changed.
Nice.
Right, but even with the S bomb. You still have just because the label says it's using this doesn't mean what it's using.
Isn't secretly compromise or swapped out.
Oh yeah, but it's easier to check if software recording is using the same version that we now know is compromise.
Where is malicious?
Right. The
Is the F spawn have a hash?
I think there's.
Yeah.
Yes. Hi, I have not looked too much into the format, but yeah, you can get Like there's 3 different versions of it. You can sign it.
You can get in just JSON or Cyclebex or . That goes in those
No, just, say, okay, I have. Library A and its hash is this.
Right.
Well, it's hashes this on architecture. Why? Right. You know there's like armor and tell or something.
Yeah, then you get into hash collisions.
Which are becoming more of a thing every day.
I don't understand how ESPAN would really help here. What I was thinking is like, what is our, you know, like system D and SSHD, those are the ones that are installed everywhere, right?
Those are the ones that if you can get those, you win, right? So, so really isn't the the most critical kind of.
Okay.
Things that we should be protecting like anything that those use like their dependencies and the things that they they link to and that kind of stuff.
If you
Yeah, so an S. In this case would, really the only value is. In being able to look across your entire system and say, hey, this is, this particular thing uses this vulnerable package.
And so you know what needs to be mitigated.
Yes, correct.
Right, well that's after the vulnerability has been discovered, right? And, and so if this would have gone undiscovered, then S.
Correct.
This would somebody else said it earlier before I had a chance to. And it might be you Shawn this would have been detected.
It was a matter of time. We just got lucky that it was found while it was still in alpha and beta channels of distros so it could be taken out.
Had it not been found and we were a year down the road and this was released. We're looking at another shell shock type thing, but a whole lot worse.
And, and that's something too. I wonder how bad, cause it was one, cause this was limited to one key, right?
One. And would have had basically free reign. On anybody with SSHD and system be installed.
But you know shell shock and heart leave than these other ones. They weren't nearly so restricted, right?
One
It was literally anybody who knows how to exploit this can exploit it, right? So. I think comparing this if it would have got out to those.
I don't know. Doesn't feel the same to me.
Fair. One key, but you know, double zab kit, let's say it was the dark arm of the CIA.
Or dark armor fresh. I don't care. Pick your nation state. And it's the nation state.
And the entire Cyber Army. Has access to that key. That key is going to be abused and it is going to get out.
It is a matter of time. One of those people is going to sell that key. Pair for money.
And then it's just gonna leak everywhere.
9 more.
Yeah, but, true.
That's very Rumsfeldian. You know, but you said the they known unknowns and the unknown unknowns.
Yeah.
Cool.
And all I want to do is play a the new St. Mike. Thank God.
You could play that.
Yeah. You're steam back is compromised.
So.
Install. There just to transfer files and romps.
Right, but for most people. Like look at we were somebody mentioned log fourj a few minutes ago.
If this had been installed. And vulnerable systems installed. Next year. Those systems would have operated perfectly fine.
And they would have just chalked up the extra half a second to a network latency or CPU spike on the target system.
It would have run just fine. It would, your game would play just fine. On a compromise system.
Yep. And that's how created.
Or at least it would play just
How many routers did they use to take down? The Xbox network.
Who is talking now?
If you elaborate on that.
Oh, I'm gonna say somewhere between 2010, 2015. Some Third party decided that everybody would not get the PlayStation network or The Xbox Network on Christmas.
So they took and went out and compromised a whole bunch of
I don't know.
Yeah, ISP. Routers.
Freaky type for
Devices. I don't know if it was the compromised. They did it, they had created a botnet and sold access to it.
Well, the in the end what they had done is they had use the update process on all these devices that we're supposed to be watched over by their ISPs.
And nobody knew that there's a whole update process. And they took them over. The next thing you know, they're all pointing themselves.
Microsoft and Sony. And taking down their Game network on. Extensively with the one day that they don't want it to go down.
Yes.
Well, I'm not gonna cry any tears for Sony. With their root kit thing.
So
Yeah.
There's there's pain piss off the wrong person at Google and you'll get every CDN.
You're not gonna be able to stay online. Okay.
Gizontite.
A couple of questions we've had. Actually starting earlier in the day. Somebody asked just a little bit ago.
Does this have any effect on a MS Windows box?
Okay.
Only if you're running Windows subsystem for Linux.
And does it affect the DSD system?
I checked my the notes on my firewall and they said that they didn't pull from that.
Repository. Plus BSD so far behind it's not funny.
Yeah. And even at that, BSD, it's not. System D, so it's not also.
It doesn't meet the requirements.
And by that.
Yay for us being different and opinion the ass to manage
And by that same token then, a Mac OS system. Wouldn't be a problem for this.
One thing to note about macos is that. A lot of developers and lot of people like myself have a own brew installed on mac OS and The version of XZ.
That home brew by the way homebrew is just it's like an app to get installed type of a thing or whatever so you just they brew install and give it some open source package and it does the trick.
So I wasn't on mine, but there was many reports in the Mac OS community, the Bella community, right, that they, they had the 5.6 or 5.6 point 1 version of XC.
Again, I don't think it would have. Compromise their SSHD. Cause one thing is that SSHD is not the SSA, it's not open SSH.
Of that.
I do not believe. I think it's comes directly from Apple. At least on my system. So I don't think it would have been compromised the same way.
But still not having, you know, you don't want that on there. So there's a way to get rid of it.
If you've got 5.6 or 5.6 point 1.
Yeah. Like Grant was saying.
I'm sitting at Contemporary Mac and it's SSHD dash capital V for version is Open SSH 9.6 patch one.
Oh, it is? Okay. All right. So they, they must have gone to that day.
But it still would, I don't think it would have launched it because again, no system.
Yeah, they have launched the instead of system D and the tools to build for Mac would not include GCC and the the source files or folders that they were looking for that indicated either a Debian based or.
You know, a fedora based system would have. Failed so it wouldn't have compiled in the malware on a, on a Mac, but yeah, the the bad version of Exed Utils did make it through Homebrew onto Mac systems.
But it would have had no effect, fortunately.
We call that a carrier. As in somebody who carries a virus, but it's not. Sick themselves.
Well, except for the fact that he.
No. But still, you don't want that happening on your system.
Yeah, except for the fact that it wouldn't have been a carrier in the sense that it wouldn't infect others.
In theory.
Okay. Yeah.
Yeah, I don't think the, would have actually been compiled in to SSH because It was looking for certain folders that indicated a x.
86 and Debian slash fedora and GCC and building for a Mac would include none of those conditions.
So you would have had a version of XED that had the same version number built from the same source code, but the final binaries would not have had the payload in it or the malicious bit of code in it.
Thank you, Ryan. Good explanation.
I think one. I think one takeaway that we will. Start to see hopefully from this. And we're kind of seeing it with identity theft.
You can't prevent it. Your best bet is to detect it early. And go to strength or links to recover from it.
Okay.
So if this had made it into primary releases, say we're down the road and it hadn't been detected yet.
And it was installed all over the place.
What would the recovery be? At that point.
You get me recovering 2, at least. Packages.
I don't think so. Given that we would be another year down the road. I think it would be a race to push forward.
As in. Maybe revert the version or identify. Of Lebex.
But we wouldn't be restoring backups. We would be patching to move forward. Like somebody with the Windows.
Yeah.
Prince Fooler fiasco for Microsoft a few years ago. You didn't go to an older version.
You updated.
Oh, you turned off the prints.
You know, I think it would be. I think it would have been, had it gotten through and been in the wild so to speak on production systems for 6 months or a year it would have been a complete pave of half the internet.
Because this was SSH that was open. Probably the first thing they're gonna do is find some good juicy targets.
And plant some actual malware on there so that if you close the SSH they've got like a back door.
Yeah, persistence.
Or reverse shell or something. Right. You basically if you have a server on the internet with SSH Then you have pretty much had your door wide open in a neighborhood full of thieves and you don't know what's happened.
You need to nuke and repay even. Oh, how long would it take for somebody in the wild?
I mean, the fact that Andreas was just happened to be doing some performance testing and you know a half a second of time and a little boost of CPU during the failed SSH connections.
Just happened to get him to look into this further despite it not really being his job or field of interest.
I.
So damn lucky because None of us would have noticed this. We would have if this was on our machine and it took an extra half a second to actually establish an SSH connector.
None of us would notice.
I disagree vehemently with that because I have done things with loops where I time like I wanna run a command across 40 systems.
And the timing and It is not a diamond in the rough to notice it. I am not qualified to do the digging.
But there would have been people. Go, what's going on here? And I'm also not, I concede.
That. And repating is considered safe. I do not subscribe to it is required. Especially if you have backups with.
Version control watching files on your system for changes. Going back far enough. Or all the way back to release.
I think you can establish. Chain of trust. For 95% and yes I'm pulling that number out of somewhere.
That 95% of what's on your system. Is fine. There I was also a person that believed you could remove viruses and I was capable of doing it on many systems Sometimes you do have to nuke and paint it.
Not all the time in the days of their cattle and just destroy the container and rebuild it. Yeah, sure.
Go ahead. My day job is taking care of pets. There are people, I would like to consider myself one of them.
That can. Roll back or safely roll forward. If you have enough data.
Would you? describe the pet versus cattle. No, it's scenario.
The cattle scenario is you want a generic container to serve your birthday card web page so your grandma can see it and you download a Docker container and create the HTML file and hey, looky here, Grandma.
That the only thing of value is the one HTML file. That container can be blown away and replaced by something.
A thousand other options. It could even be replaced with a simple Python.
I support. A 100 some. Systems that are old school Unix. Database.
Multiple versions of what started as a Curses based Utelnet to the Unix server and run the interactive curses program.
Which is backed by multiple Oracle programs and that is. The point of sales application. And each one is customized for each customer.
Be it. And we're not just talking logos on what gets printed out. It's like different reports, different queries.
Different screens for entering different things. It is. Each one is bespoke. From every other one.
Those are pets. That help.
Yeah, it's fluffy versus, you know, your, Big Mac.
Yeah, the way I hear it described is with the pets versus a herd. If you have a if you have pets as servers.
Each one has its own name. Each one has its own quote unquote personality. You know each one intimately.
If one gets sick, you fix it. And if you have a herd of servers, so you're up in at least double digits or more, it's more like, oh.
Server number 123 is not behaving, shoot it in the head and fire up a new one, boom, I don't care.
It's server 1 2 3. I have no name for it. I have no relation with it.
I don't care. I will shoot it in the head without a second thought and spin up a fresh Thank you.
And you have to make things so that you can do that. Easily.
That's what
Thank you.
So if. Part of my day job, which is not impacted by this, if it was.
We're a year down the road from now. I have backups. Offline backups. Going far enough back.
That are prior. To the incident. I could restore oldest backup. Restore the newest backup on 2 separate systems.
That are completely removed from the potentially infected one. And do version comparison comparisons of all the files in the backups.
The oldest and newest. And tell you what has changed. And in that list of files that have changed.
Or been renamed or moved or permissions or whatever. Is that a sis admin doing a job?
Do I have a change control that blesses this operation or? There's this new library and a few other files.
That nothing knows about.
The paranoid in me says that all that also means that you can trust your infrastructure, on the machine that Yeah, they haven't, compromised it in a way that, hides the, the new library from the monitoring system or anything like that.
Here.
Yeah, with rootkits, root kits will. Modify the PS output to hide themselves from the list of running processes, that kind of a thing.
On the ever since it's processed.
There's no way that anybody that got something this sophisticated on my system, that system is toast.
I can't trust that they I can't trust that they haven't modified data surreptitiously that they haven't extracted data that they haven't put a rootkit on there which will not allow me to upgrade.
To restore from backups. Probably give the hardware away as well and start fresh with fresh hardware because At this level, I don't know.
I'm gonna attack your hardware and I have it please.
Yeah.
You know, right? But you know, for there's, you know, if they flash the firmware or something, what was the wasn't there a recent bias or you have UEFI sort of a thing?
Alright.
Okay.
You're not the one that reached and got the logo for so window whose could put your System logo up while I was booting.
Yeah, I mean that's not devastating but the guys that pulled this off who knows how many 0 days they're sitting on just waiting like if they're putting this much skill like there's a lot of skilled people behind this hack with a lot of time on their hands What else have they got in their back pocket?
Yeah. Those same people if they're targeting you and you're ordering brand new hardware.
They're modifying it after either before it leaves the factory or between the factory in your door.
Maybe.
But the other horrifying thing is Intel, actually has every Intel chip has Linux running on it as a ring negative 3.
Operating system
Now that. Not everyone.
What? Yes, but all of the modern ones with the
Okay.
Vipro.
Yeah.
Yeah
That's a pain.
Yep, unfortunately guys, I'm going to have to step off and, put a little, to sleep here.
He's apparently refusing to go to sleep without having daddy tell him good night.
Alright, take care.
Bye.
Okay.
Make him listen to us for a while. He'll be out like a light.
There you go. Bye.
I Yeah, I have done that in the past.
Okay.
Yeah, they've got a couple of people who. Left their thanks in the comments as they departed a little earlier and I'm sure I speak for everyone who is here tonight really really appreciated you putting together this presentation and repeating it and polishing up for us tonight.
We're extremely informative and wow a lot more than we we knew so thank you
No.
And thank you to the rest of you as well. This has been a educational round table for myself as well.
Hello.
Gary, do you wanna book in this and then continue on after we shut off the recording or what?
Oh, Andrew is leaving. I'll add one more tidbit and Andrew can leave.
I mean, I'm not chasing you out, but I'm not trying to hold you on.
There we're talking about, you know, gee, what What terrible things can happen on the internet, what else lurking out there.
Last night from, I think 8 to 11 central time roughly. There were 6 cities in 4 different states.
Who all lost 9 1 1 service. All right, the explanation tonight is that there was a somebody putting up a light pole contractor somehow a single light pole installation managed to wipe out 6 different cities.
Okay.
I'm not sure how that happens. Also there are some reports, 6 different, but it was another report that said a similar multi-city outage happened a week or 2 ago.
So, it's gotta make it kind of start to wonder, okay, is somebody planted something in the 9 11 system to see if they can knock it out when they want it to be knocked out.
Bigger national attack by some multi state player or external state player so yeah there may be other nasty things living out the internet that we don't know about.
Maybe.
Baby.
Yes, I'd say more like his.
Alrighty. Okay, before we get off this topic, even though our speaker has left us, anyone have any other comments?
Points to make. On this topic.
If I have to go find X. See on a robot. I'm gonna screen.
Why is it matter if it's on a robot? You can play your game on the machine. The robot as long as it's not dealing with SSH.
Or if it's not. X 86 processor So what if it's the vulnerable version?
Wait, maybe that's why they, appreciated or made legacy, Atlas as a Boston Dynamics.
Hey.
Yeah.
It could be.
Alright. Let's see. Yeah, what's the other crazy thing? The F.
22 is running in Kubernetes. So who knows how complicated that is.
35 is running Windows.
Compromise already
Oh God.
What's wrong in Windows?
35
Now you're pulling my leg, right?
Nope. I got it from a credible source that works on them.
Oh my god.
Okay, all hope.
Okay, probably still has.
Alright. And we wonder why that one would.
Oh.
I'm just gonna check the pilot
Yeah, the pilot enabled Windows 95 compatibility mode.
Can you imagine what the pilot is going to say if his Heads up, display suddenly goes blue screen.
Worse.
Yeah.
She wanted to play space in Vaders or whatever that thing called game was.
I wouldn't, you know, that's just dumb enough to be. Reasonable.
Yeah.
If you're in the middle of mission, I gotta install updates.
Oh God.
Yeah.
You joke, but I guarantee the US government has. Cool. Absolute. Cool.
Oh.
Access to Windows source code. And the they can actually have their malicious actors look at things and figure out and What have you, but.
They're also.
But can they help face Microsoft's malicious actors?
If you cannot get to them.
You know, one of the, an old trope is the super secure computer that is powered on.
It's in a room with nothing connected to it. I know that the fighters are part of a network, but If that network.
Okay.
Is deep enough.
You have a known quantity that you can see the source code. You can analyze everything.
Oh yeah. The biggest. Thing is that those new fighters are supposed to be built to talk to each other and to troops on the ground and all that kind of stuff.
For situational awareness. So.
That's where I was talking about the secure network that is. Many, many, many layers deep.
Yeah.
Let's hope. Cause all I know is there's about 3 ships. That went that steam for.
Palestine. Last week or the week before and one of them came back on fire.
Yeah.
Yes, sir.
Yeah, very. I suggest book end and shutdown reporting.
Couple more things first.
Okay.
See, Andrew before he got off the line, suggested that the ATT outage well few weeks ago was supposedly due to a certain error.
You put that in the meeting chat.
And.
I missed what you said, Gary.
Andrew hearing, my comments about, the 9 11, Air last night was preceded by a similar error a couple of weeks ago.
Andrew put in that it's his understanding that it's a search error. A certificate error from that's gonna cause the ATT 9 11 outage a couple of weeks ago.
No more details of that though.
Let's see if we've covered everything related to this that we know of or care to talk about.
Anybody have any other general Q&A that they'd like to? Still attack tonight
That's good.
Quickets.
Just Sean, Shawn still with us?
Okay, well you missed that earlier on we were talking about the the new prototype website. So.
Tony was on with me and and and and. We had a good suggestion, which was, can we move it over to, to Bach instead of hosting it on Netflix, GitHub.
Yeah, certainly. The only thing that, the thing I like about NET is that when I push to master, it automatically deploys it.
But if we can get something like somewhat equivalent to that, then the yeah, let's do it.
What if we used to get lab on a box? Would, would that give us the same sort of thing?
Yes, that will give you a pages. Thank you. Actually, I use that for work every day, so I should probably put on my hand and marrying to say I'll help out.
Cool.
Okay, great. We need more help.
Go.
Yeah, we kill that. Gilabs huge though, isn't it? Like that container, forget lab.
And then, you know, get lab releases every 6 weeks. That's a big huge burden to keep that thing up to date.
Are you sure you want to run our own get labs? I've done that and it is a pain.
Hey, Sean, what about? Having the get repo. On one directory on Bach and commit hooks.
That when you push. Another piece on B does a pull into the document route
Yeah, I think I like having like the repo on GitHub, right? Cause then we can make like a slew organization and we won't have like one person who's in charge, would put multiple people as maintainers and stuff and I think GitHub has a good like social network effect thing but I think you know you could set up a mirror right where anytime something is pushed to GitHub
then Bach will notice that. The change in the repo it will clone it because it's open source right so it'll clone it down and run a build and do that stuff like that would totally be.
I think that was totally work.
So the files you're checking in to get. Are they more source or more compiled code?
Yeah, they're just plain text files.
HTML or CSS or
Well, Marked down and dot astro files and yeah, their code and marked down.
Okay, so it actually needs to go through a compile phase of some sort. And I use the term compile loosely.
Yeah, you gotta have node, installed to do the build. Yeah, which. Netles all that too.
I think the week we we could get that art. The part that pulls it that gets it up and running the SSHG as sorry the static site generator SSH on my mind, sorry.
The static site generator, right? The astro part, we could get that running on.
I believe.
Does that just running Debian?
Yeah, you just need a, you just need a version of node. I forget what, like, 16 or later or something.
So,
And the nice thing about that is that the, you can run it locally. You could just say node run dev.
Or NPN Rob Dev, whichever I can't remember, but yeah, you can just run it locally at local host and test it for a you know, looks good.
Okay, good, great. Then, then we could. You know maybe tag it or something and then and then pull it clone it from that get repo on back over to like you said we're doc root is but basically where but the current SSH is sorry, the current slug website.
Yeah, yeah, cause once you pull the files on BACK, oh, yeah, you run the node build and then just copy the disk folder.
Or you know we can even go one step farther right we could have like a GitHub action or GitHub CI workflow that does the build on GitHub and post the, the disc folder.
To the releases thing the little artifact repository and then you know we could just have something on box that's like checking that every day or whatever and then would pull that and copy it to, you know, the document route for a past year or whatever, you know, whatever, the site route, whatever.
And you ever working like, prototype for template for? What, how you're setting this up or is it all?
Open right now.
Yeah, no, it's not. What I've set up was the Netflix thing.
That was like 5 clicks. So anything, but yeah, this needs to happen for sure.
But, for now, nothing, there's nothing. Yeah.
Okay. Anything else? Before we shut down the recording?
Okay.
Yeah, can I have commit access?
Oh.
Yeah.
Who do you do?
Yeah, do we do we know who you really are? And we met you in person. And we shook your hands.
Yeah! I like it.
We was complimenting me. Okay.
How do we need to go take you to the place to place and tie you to a cherry?
Hmm.
Yeah, as long as they're saying.
Okay.
Yeah. Oh, right. Okay, the book end for the recording. This was Thursday the eighteenth of April correct date good 2,024 0 this was the monthly St.
Louis Linux St. Louis Linux users group meeting. St. Louis Linux users group meeting.
Our speaker tonight was, Okay. Cia lug he was doing the presentation on the XZ libraries back door hack which has become the item of interest in the last 20 days.
So very current topic. And as we discussed here this evening could, could make people wonder just what are the possible back doors are there lurking around What are we going to do about it?
So, thank you all for coming. It was a very, very interesting talk tonight and we've run probably about 2 h and 20 min talking about it.
So. Oh, thank you and good night.