Okay, where'd it go?
Evening, folks. This is the February meeting of the STL Linux users group.
We have a couple of presenters tonight. Grant is up first talking about SSH keys.
And then we'll have Andrew Denner talking about system D login.
And as usual, in between those two presentations, we will have the gooey middle, which includes things like upcoming events and discussions.
So without further ado, I will turn the floor over to Grant.
Oh, don't forget we have to have the annual meeting in the gooey middle. That should take us every bit of 30 seconds, right?
If Gary gets your, it might take a minute if somebody else does it.
Wait, do we have the micro machine man on retainer?
All right. This is a super complex slide.
All right.
Quote unquote deck. It is a single slide.
Ssh keys in three easy steps.
Literally run one command on your client. Ssh-key gen.
All one word. In contemporary systems, the defaults should be fine.
That will create some files for you. And… When you have done that and you're ready to use it.
Run command number two command number to point it at the target system. So like I was on my computer helping Lee with something I would do SSH-copy dash ID.
192.0.2.3. The server at Lee's house.
That command abstracts things, copies the public key over. And sets things up for you.
Pretty simple. And then all I need to do to utilize that key is SSH with a default config ssh space, remote system, 192.
0.2.whatever. And there is a lot happening under the hood.
But it is pretty simple. And most things You don't need to tweak any values.
Should just work.
And yes, I was asked to fill a time slot. That's why it's kind of short, somewhat repeat.
But this is the simplest bare minimum you need to do to start leveraging keys.
And I think that's important, especially for new people When I first looked at it, I thought, oh my, that's pretty too short. But then I realized.
No, that covers everything you have to do to get started to be safe with SSH keys.
Yep. There is a whole bunch you can do beyond that.
But…
Right. And I'm going to be posting some links in the chat room to uh YouTube videos that talk about SSH keys that are beyond what we're talking about here.
Yep, they are. Immensely valuable. It can streamline anything that uses SSH.
Rsync. Get. Scp, SCP, SSH.
You name it.
I was having trouble with that copy. I've used it before.
But when I brought Kepler into the system. I couldn't get the copy ID to work so i just took glee's advice and did it manually and worked fine.
Okay. If you want to… deep dive on that.
Either later during a call for help or discuss it on the mailing list.
I'm happy to help figure things out. Ssh copy ID.
Is a shell script. It handles most use cases, but it does have some assumptions baked in.
But it should work 90 plus percent of the time.
And if it doesn't, you can easily manually copy the keys over.
Yeah, that was pretty easy manually copying.
I remember years ago when before copy ID existed, you had to do it manually and set things up and get all the permissions right.
Yeah, but some of us like to see what's happening.
Yeah, I still drive a manual transmission when I can get my hands on one.
I have one in my driveway.
Yeah, but you have a habit of blowing them up, Steve.
And no shit.
Well, don't blow up keys.
Any other questions? For now or feel free to send them to the discuss mailing list too.
You want to talk about any of the advanced features or configuring it, you know, problems you might run into or something like that?
I can. I was purposely trying to keep the slide minimum viable product to make it easy for people to go that's like three paragraphs And it'll work and encourage people to get started.
Yes. I can remember when we first Years ago when we first heard about SSH keys We had, I think… four or five presentations on it before people actually kind of, oh, I get it.
Part of the problem was the presentations were too complex And it just kind of scared people.
Mm-hmm. And I was trying to specifically avoid exactly that.
You know? Yeah. And, um.
That and i that reminded me about that problem. The purpose of the base session is supposed to be for beginners kind of stuff and we often get too complex with it.
And I often think that often think You know, three or five slides really is all somebody might need you know the They don't have to be like me and create something with 50 slides.
Yeah. Aldo, I don't know if I pronounced your name correctly. I hope I did.
Which…
Had a comment in the chat What if we debate about how to restrict authentication to only using keys?
From the server side, also how to automate the deployment of the keys to multiple servers.
Great questions. Yes, there is an option inside of the SSH daemon on the server you're logging into.
To no longer allow authentication with passwords. I would actually have to look it up. It's been a while since I've said it. It's like one line config change.
Hook the SSH daemon, and it just tells it no longer accept passwords.
Only allow keys. And that does improve the security of the system because all the bots out there sending the internet background radiation, trying to brute force passwords they're knocking on a door that says, we don't use passwords, go away.
Oh, good. I did not butcher the name. Glad to hear that.
And as far as deployment goes. That can kind of get tricky. It depends on your environment and your capabilities. If you have something like Ansible Puppet Chef Salt.
Pick your poison you can usually get them to deploy keys for you. And in fact, they're probably using keys themselves to automate interaction.
However, the environment I worked in a few years ago I did not have that luxury.
So there's a utility called SSH pass I'm putting it in the chat.
That is specifically meant to wrap an SSH command.
So, and I think I could even do like SSH pass a parameter of where the password is.
Either an environment variable or a file, or it has a few different options.
Space, SSH-copy dash id And it could just wrap around SSH copy ID or any other SSH command you want So I could kind of bootstrap myself with a little loop hypothetically like SSH pass password ssh copy id target host. And I could put that in a loop and bootstrap a whole bunch of servers if I didn't have the luxury of automation.
All right. Okay. Recording paused and resumed.
Arp?
I didn't do it.
I see stanley. Has posted a Couple of… resources. I presume those are prior recordings from us.
No, no, those are stuff that's out on the internet. There's stuff from… Veronica says, and Joe Collins and uh uh LearnLinux.tv and a couple other outfits.
And I find that they're very good resources. They are more advanced and a little more extensive.
But these three steps you got here are how to get really how to get started.
And… Yes.
Yeah, and that was the goal of I've been using SSH keys for longer than I can remember and once a quarter to once a month, I run into people that aren't using them and it's like Why not?
Usually, well, it's too complicated or tutorials are too long or just not worth it.
Okay, so I'm trying to shoot. Very low. Three.
And those are the people that have their password written on a yellow sticky.
Yeah. Yeah.
God.
Well… You know what? The yellow sticky on my desk means I won't forget it.
And it also means somebody not in my house can't see it.
I don't have to worry about a developer of a password vault releasing the keys to the kingdom and impacting all of their customers.
Oh, gosh. Too frequently.
That happen frequently.
Once is too often, and I know it's happened at least once
That would be moderately embarrassing, wouldn't it?
Yeah, it's a password company that kind of has a bad reputation that a lot of people have been trying to get away from.
What's the name of the company?
Oh, without naming and shaming, I'll say it's The last choice some people want to make, wink, wink, nudge, nudge.
And their name ends with password.
Maybe.
Yeah. Yeah. They were a… subject of topic for Steve Gibson's presentation.
Security Now, a number of years ago.
Yep. And they actually, I was just thinking about Mr. Gibson, and they failed with policy problems.
Steve likes to say accidents happen. Mistakes happen, you try to fix them.
He's kind. He's very kind. Over-kind.
But… But when you continually make bad decisions it's not an oops it's a We don't care.
Kind of like some other company more recently Everybody wants AI in their AI code repository that we just purchased from somebody here, have AI. Say ah, because it's coming.
Yeah, I just wonder how many of the people would say, oh, we do this with AI. I always wonder what percentage of those really do.
To me, it's too many.
I got a sneaking suspicion. Much less than all of them.
Well, that's like the sticky note problem. Most of the people using AI just know how to spell AI.
Yeah, exactly right.
Indeed.
Then some people like me take the step-by-step approach to solving and yellow sticky problem No, I don't use yellow stickies anymore.
I use blue post-it notes.
There you go.
But you also have to consider the attack vector and risk profile for much of what I do personally there isn't any risk.
Inside my house. I trust my wife. I trust friends that would come over.
And, you know, if it's on the bottom of the keyboard, not visible walking past the desk.
Nobody knows it's there.
All right. Any other questions or is it time for the fastest vote in the world?
Well, I'd like to point out the new people here that when we These meetings are being recorded along with the they'll be on our presentation archive page and along with that will be the chat notes and anything that we have in the chat notes
Will we be put up there also? So.
Okay, actually, before we do, Aldo has another question. Usually when we create a key, we're asked about a passphrase.
What about if we need to automate the creation of the keys how to skip the step about the passphrase.
That's a good question. That's not where I thought you were headed.
I would have to check the man page and see if there are some options to turn the passphrase prompt off.
Or provide a passphrase somehow. So the passphrase Indian, Texas is saying or asking is the passphrase in place of using a password So what the passphrase does is it actually encrypts.
Parts of the key inside the file on your system. So when you go to use it when the files opened in red, you will be prompted for that passphrase.
What I personally do on my systems is I have a passphrase set I didn't include it in this because it's like the next level.
And I wanted to make it easy for people. And I would rather people just use keys And then we can make keys better later.
Keys are better than passwords to me. But what I have on my system is I run an SSH agent SSH-agent comes with OpenSSH. Gnome key ring can be an SSH agent.
Seahorse is one. There's a whole bunch of them. But what they do is they read the encrypted password key prompt for the passphrase decrypt it and hold it in memory.
And then your SSH client or anything that uses the SSH client asks the agent, hey.
Will you do the crypto piece on this? The agent does it on your behalf.
And you're not prompted for a password or passphrase because it's preloaded into the agent.
Passwords. Are on the target system you're logging into.
Past phrases in the context of SSH keys is actually encrypting the key on your client workstation.
Really?
Yes. And that's why… you can use, I think it's use SSH KeyGen has a command line option to change the existing passphrase on an existing key.
And you can also use that. To add a key I'm sorry, add a passphrase to a key that doesn't have one or if it's something used for automation.
You can remove the passphrase on a key that is encrypted.
Hitting return.
I'm sorry, say again?
So if you're using SSH gen on the command line in an automated fashion.
You can use dash capital N. And then just quotes.
Open and shut quotes. And that's setting your passphrase on the command line.
And it's setting it as nothing.
Oh, thank you very much, Andrew. Today I learned.
Okay. Any more questions or helpful comments?
Going once.
That's probably the key takeaway some people today that today I learned. I think even even more experienced users and they come to meetings find something that you either didn't know or gee, I forgot that
Yep.
Cool. All right.
Going twice.
And a third time. All right. Thank you very, very much, Grant. This was very, very… instructional. Really?
You're welcome. Thank you for co-hosting and posting the slides.
All right. This is the gooey middle of our meeting, our gooey middle always is where we do our announcements, introductions And then a very brief general Q&A.
Before we go to our main speaker of the evening, our main speaker of the evening will be Andrew Denner. And we'll get to that in just a moment.
We also have a special treat tonight. We have our annual official meeting, which many of you would go, oh, God, no, that sounds so terribly boring. And we try to make it uh very short and hopefully very short and
A bit comical. So moving on Stan, do you have… The slide you want to put up?
I do here in a second, but I had a… system creation. It's taking me about another 30 seconds if you don't mind.
Okie doke. Well, while Stan's working on that, I'm Gary Meyer. I'm the mc and i'm also the president of Slug.
St. Louis. Linux and Unix user group.
This is our monthly general meeting. Lee Lamert is The one at Omnitech. He's our treasurer.
And he also is hosting our system called bach dotslube.org where we keep our mailing lists and a number of other such things.
The other voice you hear you. Frequently hear is Stan Reichert. Stan Reichert's our executive vice president Which means he does everything and keeps things running.
So thank you very much to to Lee, to Stan.
Ken Johnson has been our secretary for the last year and our last year A big thanks to him. Always good to be able to go back and check the notes.
Uh so uh We always appreciate it if somebody volunteers to be the chat monitor. So if anybody would care to volunteer to keep an eye on the chat room during the main presentation.
You can point out to the speaker if there's something that came up in chat that he needs to pay attention to quickly.
Any volunteers tonight?
I got your back until probably… 8.30 or so.
Who is that speaking just now? Is that… Great. Glad I recognize the voice.
That was Grant.
Okay. Topic wranglers. It always helps to have someone else uh trying to find speakers, presentations Even if it's just to come up with the ideas for them.
So if anyone didn't care to volunteer for that, by all means, you can either speak up now Or if you want to give it some thought, you can contact me.
Or Stan later and let us know and we'll help work you into get things done. Announcements eight days from now is the St. Louis Linux user group meeting.
It's a similar format to tonight, only there's not a tutorial on the front of it.
It's just these announcements and general call for help. And then we go right into a a feature presentation.
We don't have that locked in yet. Hopefully we'll be announcing it tomorrow or Friday.
So boom days later, on the fourth Tuesday is the new linux user meeting. The neat thing about that meeting is We say the way there's no set presentation that's going to be done that night. There's nothing announced in advance.
Reformat.
It's just who, yeah, free format. So whoever arrives at night whoever says that they want to claim that they're the most novice, the newest user of Linux They get to ask the first question.
How do you spell Lennox?
And there we go. So it can be incredibly simple or sometimes it gets into incredible complex things for a new user but So no reason for new people to not come to the meeting because, hey, you get to control. We're already at the conversation the way you want.
And experienced users, you come to the meeting too, because as I say, sometimes even though there are new user questions, it's amazing sometimes the things you kind of walk away from go i didn't realize it did it that way. So it's a good process for experienced people too so
I would call it one of our lives. Well, when we were still doing live meetings A gal came and she asked a question and she came back another 11 meetings and asked the same question.
And we answered it as best we could at that time and and covered in different aspects.
So after about 12 meetings, she really understood what she had asked about.
And she didn't feel shy about coming back. So that was great.
Yeah. Excellent. All right.
Okay, I think we are ready to do our annual required public meeting.
Lee said I didn't miss it. We kind of look at this as a joke, but our attorney, when we incorporated this group 30 years ago, pointed out that a lot of groups just kind of fall apart because they never have an official meeting. So we put into our bylaws, we have to have an annual
You're required, actually.
Official meeting If anyone would like to hold their breath, I think you'll manage holding your breath the entire time. Calling the media to order.
Do I hear a motion?
I move, we call it meeting to order.
Move to adjourn. Move to adjourn is on the floor. Is there a second?
All of everything to have the usual sign. The repents know. The eyes have got its own character.
I second it.
Second.
Nice.
We're adjourned. There we go. I don't know if anybody timed it. Those of you who are holding your breath, you can breathe now. This has become an annual joke to see if we can fake previous year's records. I don't remember what the all-time sharp.
Close it.
If you honestly have something that needs to be brought up, like amending their charter minutes and bylaws and things like that.
Talk to somebody on a steering committee and we will work it into that and keep it under Probably 35 seconds.
Or so.
I'll be honest, we'll address all the concerns on steercome and have everything taken care of before we get here.
Right? That's right. That's right.
We usually try and make it so the people who want to be here for the technical meeting can be here for the technical meeting and not see the sausage being made.
Yeah, you don't want to scare the horses either. That's for sure.
So. Moving on, there is some other sausage to be made. This is our annual election time. There's a five-person board of directors who's uh who exists to keep this place going.
And this year, one of the five directors is up for election.
The current incumbent is Steve Stagman, who's in the audience here tonight. Steve, say hi.
Bye.
There he is. So he's the incumbent. There is a ballot and uh We have basically the entire month of February until the steering committee meeting held in the beginning of March, and that's on March the 3rd.
So you can vote by mail. The US Postal Mail, or you can go on the system and do it online.
Well, I'd like to point out if you do it by U.S. Postal system, that ballot is secret the way we do it because we gather them all together And we open the envelopes and we pull them out and we shuffle them up so nobody knows where it came from. You have to follow the instructions on the ballot.
Online, there's some difficulty right now. We've had some… Well, let's put it this way. The way it works… If you print the ballot out and use a pen and pencil kind of thing and fill it out and mail it, that's
That works fine. The online thing, I've got a problem with some fonts so far.
And I'm in the process of trying to figure that out.
But at least there's a ballot out there. And it'll be on… Currently, it's about the middle of the main page.
There's a link to it and there's also a link to the link to Libra office writer document if you want to deal with it that way. That works pretty handy. You can fill in the blank. It's kind of an editable PDF kind of thing.
And click the buttons for your vote, so forth. Um… that you know that so that that will work.
All right.
And I'm in the process of trying to make it work a little better, but I get conflicting information on all the documents that I had from last year.
So.
Okay. There's… somewhat oddly, there's the saying, vote early and vote often works for us.
Basically, there's room for a date on your ballot. And so whatever the last dated ballot that comes from you.
That's the one that gets counted. So yeah, if you suddenly realize, oops, I meant to vote from such a way, you can change it.
And just vote with the later date. So let's see.
Stan will accept bribes for keeping your name off the ballot. I mean…
I didn't get any $20 bills. So there's a lot of people on the ballot.
There we go.
Accept bigger bribes to add somebody's name despite paying to keep it off.
That's it. There we go. A new business model. Yep.
Is just real politics, isn't it?
Okay. Yep, yep, yep so Does anyone have any campaign speeches they want to make?
Okay, hearing none, there's also room on the ballot to vote for the officers.
President, Vice President, Treasurer, executive vice president, lug chairman So by all means, vote right in anybody for those slots that you wish.
And technically the board points all the officers, but the board's always interested in who got the votes.
Let's see. Next item. Let's see.
I'm going to bring up the ballot. Where is it?
Oh, Stan's shaking. No, Stan.
There's the ballot. Seems like it sort of works online.
Okay, so… you know and you can
All right, need to put my name on there. Who paid to put my name on there?
No, you didn't pay enough to keep it off.
That's it.
Oh, God.
Now this sort of works that uh you know that It's the button goes away.
Somehow with the fonts, when you actually print the thing that the buttons go away. So you got to print it out and then use anchor or pencil and fill in the buttons kind of thing.
And… Yeah.
Nice.
Yeah, but… Wood.
By the way, for those, especially if you're relatively new to the organization.
There are no dues to be in this organization. The criteria for voting is if you have attended two meetings in the last calendar year of any of our slug run events. That would be this general meeting.
The next Thursday's St. Louis log meeting the new users Linux meeting, the Slack meeting, which is hardware oriented leaving it once a month. The steering committee meeting, any of the things that we run as a group.
Count if you have been at two of them in the last calendar year You have a vote.
Anything else we need to cover on voting?
No, I don't think so.
Okay. Vote early, vote often.
Let's see. Other things, any calendar changes that you know of lee Not offhand. Stan, any calendar changes we should be aware of?
8meetup.com. Okay. Okay, sure.
We hate meetup.com. Yes, we do. I mean… People show up from there, sort of, kinda But they ask email questions there and they ask questions I never go back and look at any questions posted on meetup.com. If you want, you know.
Come to a meeting or join our… our discussion mailing list, or you can even go to our website and see the archives of our discussions on our mailing list.
We've done a lot.
Okay. Why don't you throw that outline back, slide back up on the screen.
If I can remember where I put the darn thing.
Just thinking we could remember commands, but the meeting procedure we need the notes.
Oh, mm-hmm. Where is it?
While Stan is looking. Let's see.
Where the heck is it? Oh. Hmm.
Always grateful for all from people through the year through the year Ken is… not only been being our secretary, he's also been underwriting our uh our account to keep Keep meetup going so
Where'd I put it?
A little.
You just had it there. Anyway, anything else coming up this month that I haven't mentioned that we should be mentioning?
Not necessarily anything for our group, but anything technology that we should be keeping track of.
Okay.
Well, Valentine's Day is Friday. Oh, yes. Valentine's Day is Friday.
What?
We have to point out that to geeks because they tend to forget, right?
So, yeah. As much as we like to focus on our pleasures in life, sometimes there are other people involved. Yes.
Thank God this time or this year, Valentine's Day didn't fall on a meeting night.
You lose a lot of people to that. Andrew just posted, I knew I was forgetting something.
I have the slides up, Gary. What did you want?
All right. Uh… Okay, let's see. If there are no other announcements, we can just go to a general call for help. There'll be time after the main presentation to sit around and chit chat and discuss things. But especially if you feel that maybe your question is short or you want to hit the
The larger crowd who's here right now Now's the time. Speak up and say what you've been working on or what's been going wrong or what you might like to hear somebody else's view of how to fix.
Call for help. Somebody ask.
Well, all I can say is my call for help was answered yesterday. Thanks to Grant.
Die.
If nobody else has a question, I'll do a quick one. Normally I post this kind of thing and discuss, but I'm lazy and don't like to type.
I've run into a problem with PDFs. I'm noticing the PDFs, it'll say that that there's a font that's not embedded in the PDF file.
And I have kind of two questions. If it's not embedded.
I assume that if somebody goes to print the thing, if they've got that font on their PC that the PDF will pick it up.
But if they don't have it, then it'll be blank. And I've run into that before.
My real question is if the font is not embedded. Can you force the embedded fonts into the PDF into the document.
So if somebody goes to print a PDF and they do not have the font.
Some form of font substitution will happen. And good luck. Who knows how that'll work out.
Yes, you can force the fonts to be embedded. I believe it's called archive mode archive mode So if you're exporting a PDF from, say, Libra office uh There's a whole bunch of options and one of them, I believe has archive
In the option name.
Thank you for that, pointing me to those phrases because that really helps searching the documentation that's available on Libra office. Thank you very much.
Thank you, Ron.
Before moving on, I think there was a question or starting a discussion on an NFS issue.
Before the meeting, were there any more questions about that?
Doesn't NFS stand for no freaking security?
No, it does not. Nfs version 3 and version 4 support Kerberos which can include encryption and key-based authentication.
So…
As an old master sergeant, I've learned to ask stupid questions because it promotes learning.
I absolutely agree. And you may know the answer, but there may be other people in the room you know need to hear the answer.
And if I knew the answer, I've probably forgotten it by now.
And apparently Andrew Dinner has what's to wrangle with the three-headed hound.
All right. Well, I don't see any comments about that or hear anything about
Any other questions? Okay, as I said, there'll be time after the main speaker to continue chatting so With that, our main speaker of the evening is Andrew Dinner.
And Andrew's topic tonight is uh logging in System D.
Of course been some resistance. People didn't want to move to a new way and uh So Andrew's going to tell us all about it. Andrew is the president or chairman of the Central Iowa.
Lug. And so we're always happy to have him come down and do a presentation for us.
So we appreciate it. And with that, I'll turn it over to Andrew.
All right. Hello, everybody. First off, I want to apologize for any sort of train wrecks that happened during our presentation here. Let me hit share screen so you guys can see.
Screen two.
Before getting into it, would you prefer questions be held or raise hands during the meeting or what?
Go ahead and chime in with questions tonight, remarks, whatever you like.
Or save them to the end. I mean, I… I tend to be a little bit… lacks and easy on stuff here.
And worst case… The best answer you'll get is I don't know, but hopefully there are lots of other experts in the call and we can get stuff sorted out here.
If you want to raise your hand. I'm sure there's a way of doing that on Zoom.
And I'm sure also there's a way of lowering your hand after you're done, but I'm horrible at Zooming or teaming or jitsying or all those things so Feel free to use the tools as you see fit and hopefully I'll learn something.
Cool. Thank you.
Anyway, though, hopefully people can see. The slide deck here.
We see your slides, yes.
Okay, awesome. And also, apologies for the utter cursedness of some of the images here.
We use AI. Some of these images were generated with the help of AI.
They went to the pet cemetery and came back a little bit wrong.
But tonight we're going to talk about enterprise logging And sort of some of it I was trying to play off of a joke of lumberjacking because, of course, logging and et cetera.
Hopefully Tux will be able to guide us. Through the forest here and get the logs in order.
First off, just a little standard intro. Because I forgot you guys introduced ahead of time.
Yeah, I have a website and uh I guess actually my title's changed since I stole the slide. I'm now a scientific… Yeah, he… scientific computation scientist at a large North American seed corn company that isn't based out of St. Louis. So there's only a few that you can figure out there.
And then also, yeah, I have your standard X Facebook link and all those things.
And yeah, I occasionally even post there. But why do we log?
Really, the big thing that I want to do when I'm logging is something went wrong.
I need to figure it out. I need to find out why Someone can't log in.
Why the computer's responding slowly, etc. And… Traditionally in the Linux.
The log, it's been a whole bunch of different logs. You have a forest of logs, basically.
And… There's a whole bunch of different files and you have to look at each one.
And… The issue is traditionally You have the biggest issue of, especially once you start and stop having one computer and have multiple different computers.
Now, if you have a distributed web server where you're load balancing between like five servers Oh no, something's wrong and people are reporting that occasionally they can't see the website.
Which of the servers are they on? I don't know. I'm going to have to go to five different logs and spelunk through all of them and try and figure out what's going on.
Well, with System D journal logging, all of your logs are in one spot so I can see You know, NFS is throwing errors here at the same time that the web server was failing.
Maybe they're related, although you shouldn't really host your web off of NFS, web server off of NFS. That's a whole other discussion.
But just as an example, since we were talking NFS. It's also nice that you can do indexing.
And it's baked into the system. And also you can have more than just your traditional text logging.
Old school paper text logs have a place But there are better ways to handle it than what the old way of doing it are.
So what are the core parts of the core parts System D logging. Now, of course, depending on how you want to pronounce it sort of the The world seems to like the CTL is being short for cuddle.
So if I mention journal cuddle. Or journal control, I'm talking about journal CTO.
And… You can also, the whole system is System D Journal. That's the daemon that's running in the background.
And then there also is a great API. You can send messages to your log via command line Or via… other tools like Python.
And we'll even go through and show you how to have your Python app chuck things into the log and then you can also filter them out and find them.
So now that we've got our tools and sharpened them up.
Really, what are the basic things that we're going to show you?
Reading blogs, you can filter down from messages. You can follow a log and basically it's like the old uh watch or tail. And you can also say, hey, give me all the logs that happened between 4 p.m. And 8 p.m. On Monday.
And that way when the user comes and says, hey, at 6 a.m. Yesterday, I was having a trouble getting in it. You don't have to try scrolling through or grepping or doing all sorts of voodoo magic to try and find your spot in a log file.
It's really easy.
He started the sentences by saying. If it hasn't cleared.
It hasn't cleaned.
Uh… Sorry.
Now, just 30 seconds ago, it was on that slide he said If you have a situation where it hasn't cleaned.
Yep. Oh, so sorry. So if someone comes to you and says, hey, I couldn't get into the website at 6 a.m.
On a Tuesday, on Monday or something like that.
Yes. Stan is right on the messages there. So… Yeah, basically… You can say, hey, give me all the logs that happened between this spot and that spot.
Honestly, not sure what I was talking about when I said it hasn't cleaned.
Other than I need to clean off my desk. Apologies there.
I must have been looking at something while I was talking.
Like I said, my goal is always to have the same quality as WKRP in Cincinnati here.
Again, apologies. But yeah, so we'll go through and have examples of doing all of these things.
Here during the demo and also thank you, Stan, for dropping them in the chat.
I have a comment slash concern kind of speaks to what Lee was asking about. I think that was Lee.
Or Gary. I have seen some default installs where Systemd logs to tempfs so if the system reboots the logs from before the reboot.
Are cleared, which is not helpful when you're trying to figure out why something is crashing.
And requires communication.
Yes, that would be a bad thing. And that's very ungood. Do you know which distro that is?
Red Hat.
Oh, God. Okay. No wonder I haven't run into that. I'm more of a Debian Ubuntu kind of guy myself That would explain why I haven't… had that curse yet.
Boy, that that's…
That does not sound like a default install.
This is Stan. A lot of the distros only show or do that to temporary. You have to actually create a, you have to tell the system And a lot of them that it's permanent.
Or persistent, actually, I think is the key word. And in the journal Control in Etsy… journaling, I think.
Yeah, we'll go ahead and dive into that config. You can set how big your logs can be, how long they are.
What the… all of the different attributes there.
But yeah, so anyway, though, yes, of course, like all things with System D, It's immensely configurable and Also, there are many reasons to hate it.
But unfortunately. It is sort of the way that the world is going and we might as well use the tools that are in front of us, I guess.
Is the philosophy I take on it.
So…
I posted in the chat my notes something about editing the Etsy System D.
Journalconfig.com file and you You have to say storage equals persistent.
That seems to properly create it with the proper permissions, I think.
If you do it manually and don't do it that way, sometimes you can get the permissions wrong.
That's all I know.
Yeah. So there are sort of the one nice thing about Using… The Systemd logging and journaling here is that it isn't just a text field anymore. So you can have metadata, which will look at it. You also can set priority levels.
And then provide other contextual data as well. We'll dive into how to provide that.
And then all, yeah. You can also forward your logs on. We'll quick go through how that in our demo.
And you can aggregate them all into one machine so that case where I was talking about of five servers all providing web services together, you can Fool them.
And then you can also do optimization and clean up and handling of stuff.
The good news is it you have options to do security access controls, all of those permissions sort of stuff.
How to mask your data. We'll dive through. And then various other stuff like that.
Which we can go ahead and we can go ahead and look how to pull stuff together. The biggest thing is the biggest thing there's this idea or this tool called System D Journal remote And if you're, depending on if you're in the Fedora world versus Debian.
Those are the commands you'll want to use. And then we just edit this journal upload and add in where you want to send it.
And then generate up the keys. Again, sort of going with the whole idea of The SSH keys that we were talking about, these are using SSL.
But you just basically generate your keys update where they are, store them basically and So this is the server that you want to upload from.
The server that's remote that's listening And then you just restart.
You're logging using system CTL enable And… You tell it to upload receiver is the remote one. Sender is upload. It's real easy. We'll show how to do that here in a couple minutes.
But I threw the… Yes.
Andrew? You're saying the left column is… Which system?
Yes. So starting out here, column one, or this slide one here We've got how to basically think of them as This is… goes up to here. So these are the commands that you run This is where you edit the journal upload in Etsy Systemd.
For the uploading server and you want to add in these And then on the remote one, you want to edit it so that it reads like this.
D.
Do you have to use… TLS encryption or can you do it clear channel because for some encryption complexity makes this become a non-starter.
I believe you can do it non-tla or you can do it non-encrypted.
However… Um… I mean, the cost of encrypting is very… light comparatively. And even if you just want to share uh lead these files out across all your machines.
Having the world encrypted by default is… a very good idea.
I absolutely agree. It's more the gray matter technical debt. Introduced by encryption.
I understand completely and even… I'd argue that leaving out how long it's good and just having it indefinitely is better than it not being encrypted.
But I'd argue that you probably should encrypt all the things no matter what. But I do get your point.
Okay, so again jumping back. The left column is what the right column is what here?
So the left column is column one, the right column is column two.
And basically in this case This is what you edit or what you put in your uh Etsy.
What was it? Etsy systemd journal uh remote.conf file.
Okay.
And then this is the commands that you'd then use to generate you're… OpenSSL certificate For the server key file and server certificate files.
Okay, so the right hand is the right hand commands you're typing in the left hand is the content of the file column one content to the file okay
Correct. Yep, this was basically the output of my console as I was typing things out.
And so it scrolled over essentially. If that makes sense.
So from the previous slide. Just scrolled over okay
On this slide, column one has bullets in front of it.
So this…
Column two does not.
Yes, that was a copy paste failure on my part. I will correct that before I send out the slides.
Well, thank you.
That's Microsoft being overly helpful in trying to format things for me.
It kind of…
Thank you, Microsoft.
I kind of get the impression that this slide is what I'll call the client or the log source.
And it seemed like your next slide was the server or the log destination.
Is that? Okay. Does that help clear anything, Gary?
Correct.
I think so, yes. Thank you.
Okay, any other questions on that?
Okay, hearing none So you can also log in.
Performance metrics like the server took X amount of time to run The… running your nightly backups took X amount of time. The CPU sat at this amount.
You can have that in your application where You can say function x took x X amount of time to run.
Or stuff like that. And this is one of the examples of the cursed uh AI, as you can see.
That tape measure just is wrong on so many levels. But… And then also just health monitoring Again, around performance monitoring that We're getting traffic. Things were getting pings. Everything's okay.
And then really the best practices that you want to care about Structured logging is better than just texting.
Be consistent. If you have some of your error messages formatted in one way.
Format all your error messages that way. And then also make sure that you catch and you handle errors and that you log them out so that you can find them.
Rather than just try catch that you just swallow them and just keep going on.
I think we, at least anyone who's a software developer probably has been burned by those sort of things in the past.
And the biggest thing is we'll cover log rotation and cleanup The worst thing ever is all of the times that I've had an a server or an application die because the logs filled up the disk And now your application's down because the logging
Ate it. The good news is with System D, we can set up and do vacuuming and all those things.
So the meat of the demo here We're just going to dive here into This is on the wrong screen.
So I'll send out the links to my uh get lab repo here where I'll have all these files at.
But basically, this is an example of how you can log basically just straight up from python.
And chuck into your logging. As you can see, we have just your standard blogger And we're calling it Demo Basic is the name of the channel that we're going to be putting out.
Hey, Andrew, can we get that size expanded a bit? Thank you.
Apologies. I realize not everyone has the big monitor that I'm currently looking at here.
We've also got old eyes.
You know, I just crossed the 40 threshold this year. So it's all downhill from here, I think.
You called that? Look at Kid. You got lots of time left.
But so as you can see here, we're… What we're doing in here is we're logging some basic things CPU percentage, how much memory I'm using And you can see that I'm throwing, this is your message here of that system stats were collected. That's the old fashioned text message that you see and love.
And then we're also going to be throwing in that there's a monitor type how much memory, all of these things into this extra field.
And that allows us to do queries and see data and handle that here, as we'll see in just a minute.
For these.
We can throw in some error messages here that CPU percentage is over
Are these logs that you're generating System B specific.
I couldn't tell if you were using standard syslog interface and having Systemd handle that.
Are interfacing more directly with System D.
Yeah, so we've got a sink essentially here, I believe. So we're doing the journal handler.
And so we're adding a handler to our system logger I believe you can do these other things in normal your normal logging loggers.
Just it's up to your… logging sync or your logging handler, how it handles those.
So I'm not sure what you'd get if you set it to console.
No, I wasn't thinking console. I was thinking existing code that's using syslog what kind of changes would need to be made to have it leverage System D logging.
It's convenient.
Okay.
Yeah, so for standard logging Just using the normal Python logging All you'd have to do is just add a handler and you'd have to pip install your systemd.journal So you'd need these two things right here import
So by bringing those on board. And then basically you'd just be logging out your normal text based logging.
If that makes sense.
Yeah.
Yes. Yep. And we'll hit that here next.
As I recall, the command line logger is like an echo to your journal… And it captures it okay
Since I live in the world of Python, that's where I started.
But then just looking We're going to… dump some logging information that we started And that it's version 1.0.
So… Just to show you here, we've got nothing big and fancy here.
And no, I'm not going to install my updates while I'm presenting. I'm not nearly that brave.
And one.
Are you sure brave's the word you want to use?
Brave, dumb, I mean, mad. Take your pick.
I was going to say, I mean, we used to do live demos and Yeah, right.
And apparently the demo gods are hating me here. I know that this work Last week…
You have no idea how many times I had that exact problem in front of an audience.
Okay, what did we do wrong here?
Oh, excuse me.
So…
What are we doing there? That's crazy. Huh. Okay. Well… Let's just go ahead and… This is… Yeah.
Oh, about those updates.
Um do just for giggles, does your next demo also fail. I'm wondering if it's something specific in this demo or larger systemd python is unhappy.
Like I presume to advanced.py will it run?
Let's find out.
Let's try…
Install
Okay, I know that it's there, huh? What if we just go…
Anyway…
In theory, this works. I will get this corrected, but we can go ahead with the rest of the commands that are the desktop side commands.
Because those I do know that actually works. So, for example.
Logger works perfectly fine here. And that is the command line side of being able to send logs.
So as you can see here. If we log just normal text And then do…
And so dash n is get me the last five. It's like if you do tail But as you can see.
Right there is one of my messages. Now is the time for all good men to come to the aid of their party.
You can also log, you can change what level of uh error this is.
As you can see. As you can see here, priority is what level we're sending.
T is what tag you're putting it on. So if we look, you can see that that was an error level.
I think it's a fatal error, if I remember right for priority zero.
Is it always the end or could you put it like… in between something to like annotate your logs.
Tell you one.
So you mean like…
You know, like put a note in my logs so I can you know something happened two days ago. I want to put the note in two days ago, say, you know, this is what happened, how I fixed it.
Oh. You know, that's a very good question. Let's ask man what all of our options are for that because I haven't used it in that way. I've always treated logs as a static thing that you don't go adding stuff to out of time.
But…
What would be the point of doing a logger then? I mean, that's the only point I see.
So Logger, you could put it into like a bash shell script or something And add logs.
I'll just accept that.
To the system log from your script, essentially.
Okay. I get it. Very cool.
Is a current entry. It's right now. Boom. Right now, put this in the log file.
Yeah.
In the journal.
One thing I might be inclined to do if I wanted to annotate Like I'm searching through mail logs And I find something from 18 hours ago.
I would be inclined to do the logger and on the command line include the message id for the message I was looking at.
So if I come back later and I grip through the logs or use System D's ability to search.
For a certain message ID, my log or my note to myself, user had an expired password or whatever.
Would have that search key. And would pop up later.
So you use it in the way that I'm suggesting? You've done that before
Well, sort of. I took your question to be how can I insert a log today in yesterday's log file. And I don't know a way to do that.
Right.
Yeah.
But I could add a current log with the same message id for the email message I was looking at.
Yesterday or whenever it was. So when I search the logs again, you know, I answer my manager and he comes back or the ticket's updated and comes back.
Tomorrow. Have, or another help desk agent needs to search for it.
They search for the message ID like ABC12345 whatever email message IDs.
But Andrew suggested that it wasn't common practice or perhaps not suggested to perhaps suggested annotate the large files.
Well, annotating as in like going back in history and adding it… Like, say, if your message that you're looking at is five days ago saying, hey, right alongside it, I think what Grant is uh arguing would be that you added as a message right now, but if you put the ID
You can search for it and you can find…
Right. Yeah, that's the annotating though. It's still annotating though, right?
To me, annotating is… adding a comment.
Yes.
Like I can take a PDF and I can highlight something and I can save and that highlight stays there.
Could you put…
I'm talking about… Having a new log entry that has the same pattern that if I search for, I'll see the original log entries plus my new one later. Does that make sense?
Could you include the timestamp of the previous entry and so that they line up by some kind of journal control option.
With what I'm thinking, no, that would not be. A possibility, but I may be thinking too small.
Oh, yes. Yes, I could include it in the text of the law.
Well.
I mean, include it in your logger entry.
I mean, it's kind of a security issue to like edit logs, I guess.
Yeah.
Yeah, that would be my understanding is that you can't Just go through and start changing the log files In that way. But yeah, adding a current log message at the end of the log makes perfect sense.
And I know one of the questions that did come up when I presented this in.
Des Moines Central Iowa lug was… what format are these log files in?
And they're binary. Hand waving. They're system readable, but like all fun things in Systemd, they're not meant for you to read.
Except for using the tools.
But so other commands here that we can look at, this journal cuddle alert level, you can say, okay, I want to only see the bad or the info.
Remember that i think it's
Okay. Let's try.
Is it a number? Is it a number?
Well, I think it's because my demo was supposed to be injecting alert status in there.
And since it isn't, that's where it's not Because if you remember… In here, we were setting alert level critical. So it's searching for that metadata tag.
And since my Python script is utterly failing at life. That's why that isn't… showing up.
Let me just try one more thing here before I give up on everything.
Could you use logger with the proper flags to inject the different levels so that they would appear for Your journal control searches.
So Logger, I…
So I thought you could specify the level on that.
Yes, you can. But in my case I'm filtering on alert level if we're looking at critical So that's extra metadata that I'm attaching to this message.
Versus.
Yeah, but I thought the logger that's part of System D JournalD supported that.
From the command line.
Yes, it does.
It was…
I don't see alert. What is it alert level that you were looking for? I don't see that in the man page.
Are you sure not maybe priority?
Yes. Yeah, so alert level, in my example that I was looking at was actually metadata that I was injecting in.
So that's why… that wasn't working.
So anyway, though, here, sorry. Basically, I was trying to filter on one of those metadata tags that I was injecting.
But we can see, I believe, journal control dash n.
10 key of zero. There you go. So when you do dash P, that's setting the filtering for log level.
So if we do like four Now you'll see.
And there you can see the log messages that I was sending last night when I was testing it out.
So it was working just apparently not today. So yeah, and the dash When you set the priority.
The bigger the number, the lower the priority log is. So you can see these yellows are warnings, reds are
Actual full errors and if we went uh And up that to like say Seven, now you're in… Like… information and debugging error.
Here, which, as you can see. This is on a virtual machine using WSL.
There's all sorts of weird clock issues because time is fuzzy when you're virtual system.
I just want to shout out how nice it is to have the color-coded status messages it just makes like they're right there there's your there's your critical stuff in red. I like that so much.
Yes, that is a beautiful thing. And it's one of the best features of how the messages are logged here.
But one of the things that you can do, I believe this should work because I had the logging working earlier.
But let's try… where you can set you want the would help if I wasn't.
You can set the output to JSON pretty. Uh… which let's just go ahead and get all of my errors So dash n of dash n of So.
And let's set a priority of zero so we get the big errors and then output equals
I'm not understanding that like where you put and typed in monitor type is that something that you're filtering on like
Yes, so that I was trying to filter my previous messages that I'd inserted into the log, but they've already been reaped and cleaned up.
So… I was trying to find my message that I'd inserted using that Python script.
So instead here, what I'm doing is I'm catching uh the last, let's go with the last one message.
So where's that concept of… keyword pairs in the man's page, for instance, like what is that Because it's not a command line option. It's something else right it's
Yes. So basically. Those are additional attributes that you can insert using the journaling API.
So as an example here, in my example, had it worked.
You were setting, here, let me scroll it back down so you can actually see it.
I'm actually setting… attributes that are, if I scroll… It's not the right one. Sorry. Here we go. So I'm setting these extra attributes.
And so one of the things that I wanted to filter on was CPU percent equals 81%.
So… when you're writing your Python program You can set these additional attributes and you can actually filter on those attributes. So like say I could author.
Andrew Denner. And so I can say, okay, give me all the logs that have that attribute attached to them.
Yeah, okay, thanks.
Does that make sense? So… back into this pretty printing as I was talking about here.
So…
So we're wanting to get the last one message that was a priority zero And we want to output it as pretty JSON.
And so as you can see, we get what is the machine ID, which is this machine, which if you're aggregating your logs together, you care about that.
It was a priority zero message. We're using the command logger to get it.
We added an identifier of demo And the message is how now brown cow.
And there's your timestamp. And a whole bunch of other data attached to it as well.
And so as part of your logging, as you can see with that extra you can actually insert additional log messages And that allows you to… Now, if I wanted to take this and do analysis on it in Like, say…
A Jupyter notebook or something, I could take all my logs and graph them and if one of the attributes was time that the program was running or something like that.
That allows me to basically graph and chart and do all that advanced analysis on it.
If that makes sense.
So other things that we can do
Let's see here.
You can…
Do complex queries where you say, I want to find this attribute and this attribute.
And if the container name that you were running it in for Docker was this and you want to search since today and you want to output it as JSON, you can advance things along. So let's take our error message where it's priority zero. Let's go ahead and say
Sin.
Since one…
And you need to double dash there.
Since…
Today I know works.
There you can see it. Let's take away the JSON pretty If we go yesterday
Again, we're only seeing the one message because I think we have that history issue that we were running into. Let's go ahead and dump the priority down So, like… Oh, because we only have one message.
That's why we…
Okay, so the last 13 messages since yesterday There you can see that there you can see that last night when I was testing out my examples, there was a message As well as now. And they were all priority zero.
If that makes sense. So if we decrease that.
You can see there's where we're going to dive into using logging from inside of a a pod man notebook or a pod man container and getting HTTPD logs in there as well so Yeah, so… Let's see here. We're going to jump across all of those because
So again, you can see the, give me the last In lines, you can also follow the logs.
So let's just go ahead and follow with the dash F. It's like doing tail essentially So as we're looking and over here in this other window I can go ahead and If I do my… What was the command here? Sorry.
If I do the logger command.
There you can see as more logs come in they actually will appear and this is just watching in real time, essentially.
So you can also and track request ID if you have it watched in there.
And there's what I was looking for, the sense five minutes ago.
So you can say, okay, give me everything that just happened in the last five minutes.
So some gotchas here. Like I mentioned about disc… running out of disk space.
You… see and ask, okay, how much disk am I currently using?
And I'm only currently taking up 119 megs of file. And as we're talking about how to actually set those and edit them.
I believe I'm going to need a pseudo there.
Okay, so here we can see. This is what your system is what your system your… journal conf in etsy System D looks like.
And so all of the default values Of course, you want compression Storage is auto
You can limit how many messages Because, of course, the worst thing is when a log goes really bad, you have a whole bunch of messages all dump in At the same time, you can limit and make it so that you don't have the thundering herd come through.
You can set how many files worth of logging you can have.
Because basically, as it decides it needs to, as one file gets too big.
It will actually flip to a new file or stuff like that.
Are you… Yes.
So you said you want compression?
Good.
In general, compression is not a bad thing. If your logging is intensive and crazy enough that Compression is bringing your system down.
Now, of course, like say if you have a potato for a chip and this is an embedded system.
These are all common these are all commoned out, though. These are not effective right now. These are not active.
Sure, compression probably is less of a good idea. But… Yes. So usually in most of the log files, it's been my experience that when their default version of config file is they'll give you, here's what the default is So by default, it has it in compression. By default.
It's set to auto, etc. So if we wanted to and we said no, then yes, that would be something.
Where this would be nothing actually changed because that's already the default. They're just showing you this is what the values you could put in are.
If that makes sense.
And so…
What I've seen is these defaults will have the percent sign and people will just copy the line And below, take the default the hash off and then change it from a no to a yes.
That way they have a reference to what the default previously was.
And then they can see that they changed it. By having two lines instead of just one.
Yeah, that makes perfect sense. But yeah, so this is where you can tweak your uh messages. You can even have it forwarded on to The console, etc.
And you can even turn on auditing and all of those things. So every time you run a command, let's just go ahead and turn that on.
Well, okay, never mind. It is auditing. Yeah. By default. Sorry about that.
But yeah, so… You don't want to save anything. Once you change this, of course.
You're going to want to run that
Where do you see them all? Where they see all the defaults.
So the defaults
The defaults basically are what's set right here as the default the config uh basically As you see here.
They're actually… Talking about how You can add in drop-in files to this journal confd directory So you can actually change it just by dropping in additional files.
Or if you just delete the in the main configuration file and any drop-ins the so like say if you you had you want to go back to the default And this was compression was set to no The moment you delete it.
You fall back to the default, which is yes. If that answers your question, sorry.
Well, that's how to reset them.
Otherwise, I believe they're… Let's see what else is in that directory. Usually there is a… I config.
In other words, you have to go through all your etc. Files to find all the defaults.
You can't just type a command and say, you know, give me all the defaults that are currently running on.
Yes. Yep. Yeah. Sorry. Again, it's not ideal.
But… System D is still a work in progress somewhat. I found sometimes And they never really asked me for what options they should add into it.
So sometimes it is lacking. So this… Rotate is… will actually, as we'd mentioned before, and I think I just deleted everything here.
There we go. So rotate, as I'd mentioned, there were a bunch of different files that are different files were basically the journals.
And that's what you get searched. So if you have one file that's way too big.
That can cause issues. Rotate. It happens automatically but Also, if you know that you've had a whole bunch of logs come in, you can run the command.
And that just basically starts you out with a new fresh one.
If you know that you've had a whole bunch of logs like say you're backing up a system and you know that there were a bunch of error messages and a bunch of bad things that happened.
You can… run this okay let's only keep the things that were in the last two days.
And that allows. But of course, as you know, you have to have permissions to be able to do stuff.
And in order to protect, just as a normal user, I don't have that ability.
So as you can see. My permission is denied because it can't go through and delete these files.
Because I'm just a normal user until I say that I want to be a super user.
I don't understand that with what Dash, dash, vacuum.
Option does.
Yep. So vacuum is… The easy way to think of the journalist is as if it's a glorified database.
And if you've ever dealt with like, say, Postgres or uh SQLite or something like that. Vacuuming goes through and deletes anything that has been either marked for deletion or In this case, we're saying, okay, just go ahead and clean up everything that's older than two days.
So that erases you anything older than the last two days.
So it goes through and it goes through
Yep. So let's say that I had a server and there's a whole bunch of messages that have come in and I'm out of disk space. I didn't manage things very well.
I know that… let's just say the last 30 days are what I actually care about.
So this would get rid of anything that's older than 30 days.
If you don't trust that if you want to control when that happens, because of course it We'll be chewing through some reading disk space and thinking and doing a bunch of stuff.
If I know that at 3 a.m. Nobody's on my machine And it's a good time to do some maintenance. I can schedule that as a cron job.
I have a…
And so I can say, okay, clean up all my logs that are older than 30 days. Set it to run once a week on Sundays when nobody's around doing anything.
Okay, great. Go for it.
I have a question that's almost the antithesis of this. Is there a way if I know there's log files or log entries within the files that are on the system.
And I have a… e-discovery or something come in and I need to retain Logs, is there a way built into System D, JournalD, that I can basically create an archive that I can then access at some other point in the future, like standard text-based logging, I would just copy the log files.
Is there an answer or solution for that with System D JournalD?
You know, that's a very good question and one that I'm not entirely sure of.
Fair enough.
Because they, thankfully, I haven't had to… fight that good fight. The first spot that i'd uh look and think. As you can see, all of our journals are actually being stored in this var log journal So if you needed to either back them up
Mm-hmm.
Or… retain them. The first thing I'd want to do is just grab all of them.
Right. My concern becomes at some point in the future, say, okay, I got the lit hold. I have saved the logs. Lawyers do their thing and it's now five months down the road And they need to see those logs.
And I know I can point system D JournalD at the copy But I don't want to go, oh, hey, these are more than 30 days old. Delete. No, no, no. Keep those. I want you to print what's in milk.
Yeah, so the other thing that you could do would be… sort of the easiest route that another direction I'd take on it would be When you want to save for retention You could have system D output everything to JSON or
One of those other file formats. Output all the logs And then you have all of your data as JSON.
Okay, please tell me you see the irony and say, use this new logging system to dump your logs to something else so you can retain them to look at the logs.
Yes. No, that's and I'm sure there's a more elegant way to do that
Right. And I'm not heckling you. It's just I was kind of hoping you had an answer and I don't fault you for not.
It's just… run into.
Yeah, no, no, I completely get it. And that's one of those things where… Thankfully, I've not had the black helicopters land on me in quite that way.
I've had other fun discussions where, hey, this piece of data needs to be deleted from the database by end of day today and I'm going to sit here and watch you do that.
Sort of level stuff but Not anything where I've been subject to legal hold or anything like that.
Yeah, I mean…
And that's just the most obvious example but there's multiple reasons why you might want to retain logs past the rotation or size restrictions.
Yes. Yeah. No.
Yeah, it's nice. With journal control you can control like you said, extract them as text, whether it's JSON and or as simple text and You're right where you were if you were using like syslog or something.
How would you do it with syslog? You'd have to kind of manually copy and paste a bunch of stuff maybe across a couple of files concatenated together to get the full range that you're looking for to archive.
And at least with Journal D, if it persists across a few days, you can extract those easily with one command and then Yeah, you'll have to store them as text. I'm going to look now and see if I can find a way to do it with a
With journal CTL itself and store them into binary format, that'd be nice.
Yeah.
Yeah, I have to imagine that that's a use case someone's thought of Just thankfully i'm Neither that paranoid… that visible that I've had to deal with.
The lawyers in the black helicopters yet.
Cool. Thank you.
So the other vacuuming thing is vacuuming based on size.
Well, doesn't the rotate do that? Like, does it rotate it out after it gets to one gig and then you could just save that one gig file?
That old one.
Yes. So rotate will save it out. But this is in the case, again, let's imagine that uh I have a web server and My application's dead on the ground.
Because we ran out of disk space because my var log is on the same spot that my application is saving files to.
And so I need to get the log down in size right now. I need to delete everything but the last gig of logging messages.
That would be where this would come into place where… You could say, hey, just go ahead and Get rid of everything but the last gig of log files.
So I'm saying I don't care in what time that came in. I might have had a whole bunch of log messages come in the last five minutes and that's 10 gigs worth of data whatever, let's just keep the last gig.
If that makes sense.
And…
What command or structure explains how to archive this stuff? I mean, that's confusing to me.
Yeah, that's good.
Ron said he was looking up.
Yeah, Ron's looking it up right now.
Stan, my understanding is that System D, JournalD D it's think of it as an opaque database that you ask questions of.
Logs go into but it inherently manages the history for you And older things roll off.
And go bye-bye. But it is expecting you to leave its files alone.
Yeah, the files that are saved Let's just go ahead and a head of… this one journal file right here.
I'd do a strings, not a head.
Or strings.
And I need a seven there because i need a seven
Okay, let's just grab.
So as you can see, there is actually text in there.
But… It's not meant for human consumption.
Outside of using the tools. Because, like, let's just…
As you can see, it's not for human use.
And just to… If I can just toss in one thing there, Andrew.
But by using the tool
A lot of logs, once they get rotated they become like gzip files, which are also not for human use. So they start out as nice easy to read textiles well, easy-ish to read.
Readable, let's say, but they quickly become like you can't read a gzip file but less will automatically turn a gzip file into a readable text file again And I think of journal CTL the same way.
Although it doesn't necessarily have that plain text initial step Once it's been journalized, you can access it with the tools available.
And it also has the nice benefit that you can do a lot more fine-grained searching compared to using like less.
Um… I hear you. I don't want to discredit what you're saying.
But I've done a lot of log mining and mining and I don't know if it's the way I do it or what.
But I've always found awk or something just more capable than grip or less.
To be able to do a lot of different things And when I've looked previously.
Just about everything that you could do as search criteria with journal control I could also do a search criteria with ARC, but that might be 25 years of Unix talking.
Yes.
That'll be part of it. And also, I'm almost afraid to see what kind of code it would take to show me all log entries from unit post fix from three boots ago because there was a problem and I rebooted twice or three times like
Trying to parse that out with any command line tools is going to be just a nightmare. And you might be able to do But nobody's going to be able to make sense of what you've done. It's going to be very difficult. Whereas with journal
Control you can just like dash B minus three enter and pretty much get right to what you're looking for.
Maybe.
Just because you can use set an awk and you wear the magic robes And the wizard hat.
Most of us mere mortals struggle doing that level of dark magic.
So here, let's see here, how are we doing on Because I don't want to So we're still doing good here. I do have one more demo of Using System D and the journaling to actually capture my Apache logs.
So there are two ways of going about it. If I had… just normal HTTPD installed locally If I set my error log and… custom log to have the the The logging type of journal d it will actually just straight up submit my logs
Directly into the journaling.
And they become a… usable since I didn't want to actually install Apache on my WSL instance because That's just something I don't really want to have to uninstall.
The great thing about containerization is that I can containerize all the things.
And so in this case. I want to output In my Docker file here I'm just doing some simple ECR, so I'm grabbing the latest version of HTBD.
And I'm saying, okay, give me… the… Just let's go ahead and make a Docker container out of it.
I want to make sure that my config file that I'm going to show you here in a second go ahead and… put it in the right spot.
And then also I'm just knocking out the security tokens just to make it a little bit more hardened.
That's just from copying and pasting here. But my config file, as I showed you here.
We're just going ahead and just dumping them straight out to the console.
And so let's go ahead and cd to…
So just to show you, we are here. And advice.
Scroll up here. I should have it.
Sorry, I've been doing a lot of stuff here. My history
Okay, is it?
Sorry here, one moment while i pull that.
Oh jesus is what happens when i leave my note with how i did this.
Okay, here we go. So the command that I care about here is… So the first thing I'm going to do is just run Podman build.
I'm going to give it a tag of HTTP. And I'm saying build this the current working directory that I'm in.
So I'm going to want to grab this command because this is what I'm going to want to run And… I'm actually going to have to cd to…
Okay, now I can run. Odd man.
Build.
And so this would be the same command if you're using Docker. Let's make this a little bigger so people can see it.
I'm saying okay build the Docker file that's in current working directory.
As you can see, I already had it built so I don't have to really do much. I'm getting a weird warn here, which again.
Updates are bad when you're giving presentations. So what am I doing here? Podman run I do want to add the command.
So…
I want to clean up afterwards. I want to run it in daemon mode.
So basically without having it outputting to the console I'm running this command or running the container named HTTPD Or I'm giving it the name HTTBD.
We could name it something else like…
You're saying, hey. Use the log driver journal D. So that means all of the stuff that happens inside of this pod will be logged out to journal D.
And I also want to put the log option of a tag of This, which we'll see here in a moment.
And I'm exposing port 8080. To the world and having it connect to port 80.
And I'm wanting to run HTBD latest, which is what I just built.
We go ahead and hit return and there you can see it's running Now, if I go ahead and pull up a web browser here.
Let's just go ahead and hit.
Local host 8080. Just to prove that it works.
And then… Let's go ahead and just look at journal control and see the last five messages and there you can see I got from There's that tag that we talked about of HTBD and the name of my pod, which is Web Server.
And you can see there's my git of favorite icon And… Let's go.
Yeah, there you can see additional logging Let's go ahead and generate some additional Traffic there.
And there you can see the gits. And they were status 304 because nothing had changed there.
Now let's go ahead and generate an error message just because we can.
And as we look… And so this is the point that we could point What was the… Tag equals tag equals that so if we go ahead and say, okay, give me all of the the journal for tag equals HTTPD slash or dash web server.
And this is where I have to look back at my notes because i'm
Okay, so we want
This is where… alternative.
So let's generate a couple more here so I can find it.
And then…
And this is where we can find out what it was called And it was…
Message. Container ID, hostname.
And I am failing at life here. Sorry, looking at the other
So Rob, to answer the question of how to export things. Basically, you just save it outside, you export it out and then you can point journal control at where it's at.
Hopefully…
Yeah, looking out here, yeah, it looks like Rob has better information.
Looking here, we are bumping up against the 8.30 hour. Any other questions or corrections or snide remarks or heckling is more than welcome.
This is Stan. My question is… Am I correct in understanding that if you had five machines with running journal journaling, you can rsync that to a single site to collect and combine all these journals.
Yes. So rsync probably would work, but there is actually that tool as i had talked about
Oh, that's better. That's better. That's better. I… Okay. But that's what that does, kind of the same kind of thing that rsync does.
Yes.
I don't think they're doing the same thing to me Remember, the System D, JournalD, it's functionally a database of sorts.
So when you're taking database files from multiple systems And are syncing them to a central system now you have pieces of multiple databases. They're not necessarily going to combine and certainly not in any sensible manner.
Whereas if you do the… method that Andrew showed for having it forward logs, you could have all your disparate systems forward logs back to a central system And you would have a single you would have each little database on the outer systems, but on the central system.
You would have a larger database that contains log entries that are copies from the other system.
Does that make sense?
Yes, yes.
Yes. And additionally, how this works is like, let's say there was some sort of net split between the two machines, say you're running a CDN.
Content delivery network and content For some reason, the internet cable between here and Djibouti is cut for a while.
And your logs can't aggregate up. Basically, it will… figure it out as the network comes back online And… get things caught back up to where it's at.
That's part of what this…
Oh, the, that's basically it will… it will actually find what is the high watermark and send only the stuff after that high water mark, essentially.
Contemporary versions of our syslogD also have a way to store and forward post net split.
It's not enabled by default, but it is there.
And then, of course, you can also then export this data out and long-term put it in like a flume or a flume or a any of those bigger log analysis because This example of being able to pool your messages together works great for
Like say if you have 100 But the moment you hit like say a thousand or two thousand servers or something like that.
The idea would probably be that you'd have sort of regional aggregators And then those would then, using the System D functionality and then from there uh you'd want to bring them all together into a more sane way of being able to search things.
There's irony there.
Next question.
At least being able to search writ large.
Next question. Considering that a lot of the data in the journal, the logging is possibly sensitive.
Is there a method where just individual information might kept for the individual that's not a sysadmin?
Or maybe that doesn't even there's no logical reason to do that. I don't know. I just, just a blue sky kind of thought.
So…
So you can set permissions on the actual logs themselves. And so if you like exclude user access from the logs, they won't be able to get to them.
There's no way to segregate that for individuals to certain
You can encrypt data. So, uh.
Like, say, for example… over here would be a…
So you have a user ID that does a certain job and you only want the journal to to show that information for that job that that idea is doing.
So you can encrypt certain fields and then if the person knows how to encrypt it, de-encrypt it.
They'd be able to see it that way. As for the Yeah.
Good answer. Good answer. Good answer. Thank you. Good answer.
So as a Related to Stan's question, so you're running journal control without sudo So how much information are you getting that So you can get logs from that other people's processes.
Okay, thanks.
Yes. As my, by default, my system is set up. You can set additional permissions and stuff like that.
Just on a standard single user machine that that's
Going above and beyond.
At least that's my understanding. I haven't had to… I really don't admin many multi-user machines So that's where if you've already cleared the hurdle of being able to SSH onto the machine.
You're already… At the point, you should probably be able to see the logs.
And it's only stopping you from doing… destructive things like deleting the logs where I'm concerned about it.
If that makes sense.
I have a final question. Have you built a… personal cheat sheet for all the commands that you've been using with journal control or Well, it turned into 12 cheat sheets or do you know the stuff that you've been using by the like the back of your hand i mean how
How extensive has it been a learning process for you?
It's been kind of painful, as you can. It's one of those things where I admit an awful lot of it. I Google when I need to do one of the, okay, find me this thing in this way and et cetera.
And also occasionally sort of full circle here, the ai route of, hey, chat GPT, how do I do this?
Thank you for that answer.
But yes, the man pages are great to uh Henry's uh… message here.
Are there info files on there? System D or is it all just man pages?
Do you know?
It looks like there's uh an info file. Now, of course, if it
No, no, no, no. That's a man page. If info doesn't have a man page, it throws you the man page.
Gotcha.
It looks like it does.
That's the man page. Okay.
Yeah. But I don't know that I necessarily have all of the info files installed on this instance.
Yeah, the thing was that the training videos that I'd been watching that talk about journal control And System D and all that.
They got up to the point a couple of years ago that they didn't have a good way to combine the database is on five machines and ship them off to a remote site but that was five, eight years ago.
I would just, that's why I'd ask the question. And now that you'd mentioned it.
Earlier, you did have something on that slide earlier that said that there was a method to do that. There was some long Yeah, that… Right.
Yep, system D dash journalist dash remote is the name of the package.
And I had never seen that in the, in the man pages, so… Is there a man page for that particular command?
You know what? Let's find out i i'm… imagining it would have to be, I'm betting I have to install it though.
It would help if I… Okay.
No, ma'am.
Well, you won't have a man page if he doesn't have that option installed.
Try that man systemd journalDD. Promote in a web browser and you'll probably get a man page.
Or no.
Excellent. Excellent. Thank you. I've been looking for that for months.
Yep, there you go.
Thank you.
Yeah, no problem. And yeah, it has the availability of sending Yeah.
And so, yeah, it looks like it can even do The other direction and pull… Yeah, it's just using like curl and yeah one of those things where everything is all turtles all the way down.
The entire world runs off of curl apparently
I've been very happy with what you presented here.
Thank you. Yeah.
One of those things where the best way to learn is by presenting.
And obviously I did not sacrifice nearly enough blood to the demo god so
We don't have to end here. Other questions from other people. I hope there are.
And the second best
I have to heckle a little bit. So how do I get my Solaris or AIX servers to do this?
Or Windows servers. Without USL.
Ooh.
Ask the system developers of those platforms.
I was going to say none of those have system d
Ding, ding, ding, ding. I did say I had to head.
But some of those do have service management utilities. Like… So yeah, I mean, again, it's kind of like ask them Do you want… Yeah.
Yeah, ask them.
I do know that there is a… I'm drawing a blank on the name of the program But there is actually… something that can pick up logs and chuck them into System D.
So that might be your… poor man's approach to it.
In a more serious non-heckling manner. Does System D Journal D?
Have the ability to… I would say route certain types of messages to different files I could also have it configure our syslog when it saw a certain log entry matching a specific pattern either severity and facility or text string or what have you.
It can actually be configured to execute a command And that can be a page or send an email or any script I care to write on the system.
Is there a way to get System D, journal D to do that type of function.
That's a good question.
And I don't know is a perfectly valid answer.
Yeah, I don't know necessarily. I can imagine how I'd script it up. Like say I'd want to say give me all the… SEV1 errors in the last five minutes. Set cron to run it every five minutes and If it's an empty JSON, then don't do anything on it but
Yeah, but that's polling and that's scraping. I can actually configure our syslog that when the log entry comes in, it matches this pattern, take this action now as a push basis instead of periodic polling.
Yeah, that's very much an I don't know.
A trigger, like a trigger. I get triggered.
Say again, Tommy. Yeah.
Yeah, that's a good question.
While Andrew's thinking there, I'm going to point out that Hopefully you're all aware of the chat button at the bottom of the screen. There's a whole series of comments that were made here during the meeting.
Henry has been making a number of interesting observations and suggestions.
Along with some other people. So make sure you're At least peaking in the trap to see what comments are being made.
Questions.
Also, the chat will be part of the presentation archive webpage entry through this this meeting.
So in a day or so, those will be posted up on our presentation archive page.
Thank you, Andrew. It's been informative.
Are we done?
And Andrew, in reference to your comment earlier about the best way to learn something is to do a presentation on it.
The second best way of learning something is to come and listen to one of Andrew's presentations on it.
So yes, definitely a very, very big thank you, Andrew. So just covered a lot of territory.
Thank you. By trade, I'm a software developer usually i I leave the… the sysadminning to the actual professionals
Don't sysadmin at home. Okay.
Let's see. As I said. Open up again just for general Q&A if anybody just wants to chit-chat about some other topic. But before we do that any other comments, questions, specific to Andrew's presentation here tonight.
I take it as a no.
Okay, hearing. That's going to be a no. So if you think of anything as we finish up with some general Q&A, you can jump in there and ask it again if Andrew stays with us.
In that case, I'll open it up and go back to general Q&A uh just Anybody got anything they want to talk about?
Problems they've been working on solutions they've come up with. Things that they wish happened.
For those of you who are new to the group haven't heard this in a while.
If there's anything that's anything you would like to point out topics that we should talk about.
Suggestions for speakers. If you email it to editor at slug.org.
That'll get to a number of us And we'll try and follow up on your comment or suggestion or question in there.
So editor. Absolute.org. We need volume.
We need volunteers. We need volunteers.
We need volunteers to stand with saying uh so uh It would help to have more people who We refer to them as presentation wranglers, people that go out find these potential speakers Chad them up and get them to come and do a presentation.
We need more simple presentations also. You know.
Mm-hmm. Yep, they don't all be groundbreakers like Andrew just did.
You've just…
And in fact, a perfect example is as Grant said.
During the tutorial basic at the beginning of this You know, it's like, hey, here's just the beginning steps.
And it was a nice job. It gave a good introduction to how to think of using SSH.
Ease. As we said before, for those of you who are too two meetings in the last calendar year. Vote early, vote often.
Anybody thinking of what they've done? What they could talk about and say you know oh hey i did this last week and it didn't work. Why not?
I think Lee needs to do a presentation on doing emergency uh PFSense build.
But I need to bow out for the evening. Thank you for your presentation, Andrew. I found it very informative.
And Stan Gary, Lee, can somebody else monitor the chat?
Fantastic. Thank you, sir. Good evening. Good evening.
Have a good evening.
Very quiet. We could end the meeting.
Great. We could. We could. Unless somebody comes up with a remarkable piece of commentary or even just something one step above and board.
Well, I got a question. Cool. I've… don't know where I saw it, but it may have been Scott was talking about using a presentation tool that would generate the presentation slides from a markdown file.
We've got some presentations in the archive about Markdown itself. But not doing it but not doing that.
And I was wondering if anybody recalls the discussion one of our speakers was talking about that you know in sometime in the last 10 years.
This is like a neat, useful tool.
And I think you're right. I think it was, Scott. I'm trying to remember, but…
Yeah, he had a tool. It may have been JavaScript if i think Correctly, that would just take the markdown file and present it based on a template that you define.
And, you know, if you want to put a presentation together to me that sounds a heck of a lot simpler than trying to fetch with the outline and live your opposite breath.
Yeah.
Well, I have a question. Nobody else has got anything.
I have a motherboard here. That has with the CPU that both support integrated graphics I've also got an external graphics card in their discrete gpu And I am trying to get them to work together. I want output from output
All the GPUs, but I'm really having a hard time with it.
Anybody have any experiences? Is this even doable?
It is. You have good hardware drivers. I've got a my main workstation here has a four four-port card in it plus one on the motherboard.
And it worked fine until I upgraded to SUSE 15 four i think it was a couple years ago And it did not support the video card.
But it cost me like 70 bucks to get another card that was supported.
You just have to do your homework with that. Distro if i remember correctly, was it AMD that was better supported than Nvidia and some of the others?
Yeah, generally, yeah.
Bye. You know but but basically if you Does it see your onboard?
Video.
Well, it sees… How does this work now? I think if i remove the discrete one, then the onboard will light up a light up via HDMI, light up a monitor.
But I only have one HDMI monitor and it's in use right now. It's my center monitor here.
So when I go back over there to the other there's an external, like the discrete drive which will work um So it doesn't seem to see them both at once like there's just nothing I can do well make them both work. I flashed the
Motherboard firmware BIOS update to the latest version and I've gone into the BIOS and I've set a couple of settings So one of the options is, if I have it here so I don't totally mess up what I'm saying.
One of the settings was for integrated graphics was to… When booting, what is your primary you know video device and you have peg or like i IGD.
Pcie Express… graphics or integrated graphics and I've got it set to PCIe is the primary one.
Okay.
And then I've also got the option for hybrid set to something and it's just not working.
Let me back up a second. You said you have one monitor.
I had one monitor currently, but I have plugged two monitors in at once into that machine, one through the integrated via HDMI and one to the discrete via display, not DisplayPort, DVI.
Okay and they well the point is if they're not there at boot time you're never going to it's never going to be initialized You have to have the monitor connected and available at boot time.
Yes. Right. Yeah, I'm… pretty sure that I did do that and uh Yeah, no, good.
I just want to make that point. So that you knew it.
Yeah, thanks.
Well, if…
I ain't excited as soon as you just run INXI with the graphics option or the F option We'll see if it's coming up. We'll see if it's showing on.
You know, I can probably… do something like that.
Sounds like a Sunday question.
Yeah.
Okay, I do have a display unclaimed amd That would just be the internal one.
And it says it's unclaimed. So drivers not loaded, I guess.
Yeah.
But if it's So yeah, somehow it's just deciding like not to load drivers for that, even though If I disconnect the discrete one it will just come up with that one no problem.
So something I need to look into.
Maybe the internal one is not really an internal one. Like maybe it's like…
This is just a… It's just panning over. It's just like a bridge to the processor or something.
Yeah, it's something that's right on the like right on the processor.
Then when you plug it into three cards, maybe disabling the internal one
Yeah. Although LSHW shows that it's there so it's… presented to the OS I need to maybe do an insult.
Yeah, but you said it was working without the discrete one installed, right?
I forget how many… Yeah, yeah.
So you have a driver installed. Maybe it's unfortunate.
Somewhere, yeah.
Or blacklisting it or something. Yeah, it's definitely a Sunday question.
Right. Yeah.
What's the Sunday question? That's what I figured, but I thought I'd ask the dumb question.
For the ALE group. I don't want to… Yeah.
And Andrew, there's two U's in Slug.
I'm being picky as he names the file. Sorry about that, guys. You were having a good conversation there.
He's still sharing. Does he know he's still sharing?
I'm sorry I didn't understand that.
Oh, never mind. I thought he was still, Andrew was still sharing.
Yeah, sorry, I forgot to drop off there, but not like you were looking at anything that was overly… Amazing.
Well, no, I just comment. There's two U's in Slug, so you misspelled.
Yep. Fixing that right now.
So… So, Andrew, you had five minutes and you fixed that script yet?
I wish I knew what I actually broke other than running apt update.
Is it time that we turned off the recording? Question it is.
I suppose we've been having a good time. We probably probably uh Actually, we have had some good comments here on the side.
Or to some specific point but yes i guess we should close it up.
Let's see. Any other final comments before we turn off the recorder?
Okay, this was Wednesday, February the 12th, excuse me 2025. This was the general monthly meeting of slug And tonight's presenter, main topic presenter was Andrew Denner.
Who was talking to us about logging in System D. And our basic tutorial earlier this evening.
Was done by Grant Taylor and it was on What was it on?
Ssh keys.
Ssh keys. There we go. Obviously, Gary's brain is somewhat fried. But both presentations were very, very good. We really appreciate it. Thank you very much.
And with that, thank you all for coming and look on the slug.org slash calendar.
For our next meeting, which should be next Thursday night, it'll be the next the monthly st louis log meeting And don't forget to vote.
And don't forget to vote. And don't forget to send your bribes into the appropriate voting authority.
Okay, thank you all and turn off the recorder. Good night.