There it is, there it is. Okay, I got it.
Yep. More record…
Yeah, Steve, we're starting recording.
Steve.
Can you hear me, Lee? Is it under more? This meeting is being…
Okay, welcome everybody! This is, uh… Wednesday, November the 12th, 2025.
Uh, this is the monthly general meeting of SLU, the St. Louis Linux Unix Users Group.
Uh, we have our two presentations tonight, uh… To begin with, we have the basic tutorial, which is being done by Grant Taylor.
He's doing a talk on, uh, uh, doing some troubleshooting using.
round robin database, uh, that should be interesting. And then after that, we'll have some, uh.
Uh, announcements, brief Q&A. And then we'll have the main presentation, but… Uh, Lee Lambert, and that will be on, uh, Evolving IPconfig and Route.
to IP route, uh, the newer. Linux network configuration commands.
So, with that, I will turn it over to Grant.
I suppose I should share my screen, but I have to heckle somebody for a moment.
What Linux has IP config?
Mary can't read.
Okay.
Here's mine stuck in the wind blows world.
All right. Thank you for covering my window.
Can you see my screen?
I see your screen.
Round Robin Database. Complex name…
A complex name?
Steve?
Got it.
Starting again. Round robin database, a complex name. For, uh, simple… tool with a simple concept.
that has lots of minutiae to it. Um… technical jumbo, it is called a time series database.
Somebody have a question? Okay. Um… Designed for storing data points.
in a loop over time. And it manages… storing information and removing the old information, and aggregating.
does a lot of housekeeping for you. Um, I… use RRD to assist with diagnostics.
figure out trends and patterns and things. Basically, any time you get.
a number, either… Output from a command, like the amount of memory you have free.
Or you run word count. lines on a file.
what have you, if you get a number. you can store an RRD and track it over time.
Um, you might find it beneficial. for things like… cacti or MRTG, or.
a whole bunch of things use RRD under the hood.
Um, but you can use it directly yourself. and get those features.
Relatively easily, and I'll kind of tease with some examples.
So, um, any place. You can benefit from saving numbers over time and looking at them.
It's a good place to use RRD. I say RRD… is basically.
three main phases. You collect data, ongoing, or you get data out of something, is the collection.
You then record what you collected. And ultimately, over time.
You graph what you have recorded to try to make sense of.
So, anyone who can benefit from collecting, recording, and graphing numbers.
As in, if you were listening to this, you could probably benefit from it.
can, uh, probably find RRD useful at some point.
Questions, comments, concerns?
Um, suppose I have a command that lets call it super fancy, collect command.
gives me a whole bunch of numbers as output, one line per second, three numbers.
kinda looks like some weird things. don't really know what they are.
Um… being the SOB that created them.
I know exactly what the pattern is. And it's multiple hundred lines before the pattern repeats.
But you might not be able to see it, even if you saw all.
hundreds of the lines, even if you happen to know.
That I have done some manipulation, happened to do it by prime numbers.
I personally am not good. at looking at numbers as they come out on the screen.
say, once a second or. So, and seen a pattern, especially if it is bigger than one screen.
So, I find RRD collecting it. and then subsequently graphing it, quite helpful.
Um, and… these numbers, I actually used command.
use the same command to get the data. and send it in to AUK to have it.
synthetically create a command to store it. Once I have it stored, I graph it.
So, those numbers, does anybody want to take a guess at what those might.
B? Lee?
Gary, you want to take a guess at what those numbers are?
Lee's not available at the moment.
Hmm… No.
kind of hard to tell from the numbers. Can you tell what that is, or do you recognize it?
Sine wave?
Yes. And quite literally. I kind of go into the details here.
I put together a loop that did 0 to 360.
And the red line is the angle. 0 through 360.
The green line didn't get the graph quite aligned right.
is the sign of the angle. The blue line is the cosine of the angle.
I just happened to, because I'm a. BLFH at times. I multiplied the numbers by something else so they didn't exactly match what somebody that.
spends a lot of time. with trig output would recognize.
But I think this is a good example of how.
What might not be obvious. showing up in terminal output.
all of the sudden becomes pretty obvious when you look at it visually.
comments or questions?
Yep, I love… I love graphs, exactly what you said.
A good graph can tell you so much. More, so much more quickly.
Yeah, looking at numbers.
Yep. And one of the things, and I'll go into it in a moment, is… it can collect the data points.
That's it.
And I can look at them from while I was sleeping.
I hear. Go for it.
This is Stan of… What graphing tool did you use? I must have missed something.
round robin database.
Oh, so it actually includes the graphing tool.
Yes, sir, it does. And I'll kind of go into that in a few tabs.
Thank you.
Mm-hmm. So, when you start with RRD.
you have to create. a file for it to store things in.
It looks like there's a fair bit to it, but it's actually not bad, and it kind of repeats.
the file. We're gonna… put it in.
We give some details. About how often we're gonna feed it data.
In my case, at least once every 2 seconds.
Then we say there's something I want to graph.
Um, it's a data source. It has a name of alpha, or Bravo, or Charlie.
It's a type gauge, and that's… There's a few different things if something's always incrementing.
Like, if you're pulling cable out of a box, the number of feet.
Left… those numbers grow with each other type thing.
that'd be a different data type. Man page has more details on.
Um, again, some more minutiae about… The, uh, data point, but it's kind of that same thing.
Repeated. For the three different data points that I wanted to graph, you know?
The angle, the sine, and the cosine. And then there's a concept of round-robin archive.
And the idea behind that is you specify, I want to keep.
X number, and I think I said 720. Primary data points.
And then I can specify… Uh, like if I was graphing bandwidth.
I would want to know what it was every single second.
for the last day. Subsequent days, I want the largest value over 5-minute intervals.
That's where the round robin archive comes in, is it takes.
The big detailed data for the full day, and then it keeps more of it.
less granular. Any comments, questions, or concerns?
So… that creates the file. Now, we need to put some data in it.
And, uh, my super secret collect command. was a script that I created for this presentation.
it just took the second of the day, did some math on it.
and printed some things to the screen. But it really could have been.
my custom command. You could be checking the amount of disk space, the number of connections you have.
Or the number of GPS satellites. Or the number of satellites your GPS receiver is connected to. Really, it's any command.
that gives you numerical data. That is how you collect.
So, there we show the same command. Collecting… And I like AUK.
Um, a whole bunch of tools can do it.
But AUK makes it real simple to take standard N.
do some interesting things with it. And in this case, I'm just generating.
this command… from input data, and running it through a system call.
say the data I give it is going to match this pattern, which comes from when you created it.
the data you want to put in. that is the sum total of the command to run.
once a second, once a minute, once however often.
to store data. in the file.
And I make a comment that it's probably good that the RRD update is one of the simplest commands.
Because it's what runs most frequently. Like, I've got some things I'll show on subsequent slides.
where I'm doing that once a second. For a bunch of different things.
First diagnostics. Comments, questions?
Then we get into graph. This is probably the most complex thing.
But it's not bad. Um, it starts… Starts off housekeeping.
Um, you'll kind of see a very common pattern.
if it's RRD tool, you give it a subcommand.
graph in this case. Um, and we tell it… the file that I want to work on.
Apparently, I missed that line. There should be a line in here that says, tell it which file we're going to work on.
We specify the start and subsequently the end is assumed to be current.
Of what we're gonna graph. And I can kind of show some other examples where that.
means a little bit more. Like, I have some things for work that show the last.
30 minutes, another thing that shows the last 90 minutes, last 3 hours, last 6, last 12, last 24.
Um… we define some variables to use. They're relatively easy.
We're saying, um… I want to define.
a variable named in. this case. The variable is named alpha.
And it lives in that file. under that name in the file.
this pattern repeats. exact same thing for the other data points.
Um… this.
So when I multiplied by the arbitrary prime numbers to store it.
here's how I'm undoing that. Then we actually draw the line on the graph.
And put some text on it. So it looks like there's a lot.
But when you break it apart. It's relatively simple.
And it repeats with minor changes.
comments or questions?
What's the, uh, max? MAX.
Um, that is consolidation function. And that tells it… how to store, like I was saying a few minutes ago, if I have at one-second intervals for the first day.
It's related to, um… how it relates to the archive to say.
consolidate 5 minutes down to one data point. store the largest value within that 5-minute window.
Oh, okay, okay.
Okay, anything else?
On the next line, uh… The parameter after alpha is 7.
And then the next thing, paragraph, it's 13 and then 19, what's that?
Incrementing?
That, um, the CDEF, which is… computed or calculated definition.
I'm taking, um… So the DEF defines a graphing variable named alpha.
the alpha parameter. in that file.
So on the calculated definition function. I'm defining a new variable.
Alpha prime… That is calculated by taking alpha.
dividing it by 7. And then when I graph, I'm graphing using.
calculated variable.
You see the… difference there.
Yeah, yeah.
So, um, RRD has… almost a domain-specific language.
That's not quite a programming language, but it's certainly not Turing complete.
But I can manipulate data with it. And I'll show some other uses.
Where it calculates the maximum value and the minimum and average.
within what is displayed, and I create some lines with it.
uh, definition is just… define this variable as that.
calculated definition is, do some math or something else on it.
the first variable and store it in the second one, so you can use the second one.
Later. Make sense?
Gary? Okay. Crunch, crunch, crunch.
Yes, yes, thank you.
Um, why might somebody want to do this instead of relying on another package to do it for them?
Well, my recent use case is I was having some problems with a Nagios server that I have inherited.
that is monitoring about 20,000 services. Like, every 5 minutes, it's checking 20,000 things.
And we were having a problem where… The number of services coming back in an unknown state.
Or 3, if you're used to Nangio's parlance. would fluctuate, and… I, as a human that has a brain that gets distracted by things can't sit and watch that.
All the time. my brain won't let me, I need to sleep, I have other responsibilities.
My wife wants me to help with things around the house. I can't sit and watch it.
So I figured out how to get that number.
Out of Nagios. And I used RRD.
store that number. for me, and then subsequently graph it.
And all of a sudden, I had that type of instant, oh wow.
There's a daily cycle to this, and something we thought was problems in the morning is, no, it's just.
natural process. And it's recovering, and it's happy, and it's doing what it needs to.
Um… wrong tab. And as I continued in my diagnosis.
I needed to keep track of. The latency for connections, so that's a numeric.
Value. I created something to collect that. and store it, and subsequently graph it.
Um, I didn't see any pattern. in that… Other than… when we had problems.
Latency would spike. But it didn't really give any… good insight, it just kind of corroborated what we already knew.
But that's nice. confirmation. Um… digging deeper.
I have long wanted the ability. to track the number of open TCP connections.
on a server. because I routinely have cases where something will happen to a VPN.
And clients… one of multiple clients on a server can no longer get to it.
and the number of connections goes down. What I don't know what normal is, what current is, etc.
So I wanted a way to track that. I got off my lazy posterior and actually created that. I'll show you a graph of that in a couple of minutes.
Um, that's that line. Another kind of niche.
use case is this presentation. showing off what RRD can do.
That may or may not be a compelling reason to use it to you.
Comments, questions?
Mm-hmm.
All right, talking points. Um, there's a number of them, I'll kind of… Go over it. Um, most of the graphs that I have created are very basic. Read the variable.
create the line, put the text below the line.
the computation, I hadn't really done before this graph.
Um… but it… I have used it since then, and I'll show an example of how I used it.
Um… I try to keep my variable names. in the file, and when generating the graph, the same.
So my brain doesn't have to go, what variable name do I use here?
you can use whatever you want within. Reason. I keep them the same.
Um, I've used ROD multiple times over the last 20 years.
I've used it a lot in the last 2 weeks. I'll show you some other examples.
Um… many programming languages have interfaces to it.
what I wrote was Pearl. I cut my teeth in the 90s.
that's what I hark back to. I tend to put my collections in whatever thing.
dash collect, and then another thing, thing. dash graph, because they run at different times.
collect one second, graph once a minute. Um, I'm leveraging a few HTML tricks.
to… I can open the window and look at it.
And it will automatically refresh. Once a minute.
Uh, graphing can seem tedious. But it's really… once you get one working.
you can kind of copy and paste it and tweak a few things.
It's… you've probably seen it with a lot of things once you get one working copy.
Others are really easy to do. Um, there I talk about what… The script I broke down to read netstat output.
And record the number of connections. And Lee, I hear you screaming to yourself, why net stat, why not SS?
I want something portable. Um, I deal with Solaris AX.
free BSD SCO open server. SS is Linux-specific.
So… Um… I think that's… main things on here.
Now, some of… I'm gonna make the screen a little bit bigger.
the other examples.
Here… It's a big graph.
Um, you… I hope it's coming through on Zoom. There's a dashed green line at the top.
solid green line, dash green line. The graft value, a large green line.
Then I dashed green line. Is that coming through Zoom?
Okay.
Uh, just barely. I mean, the green shows up, but the others is just very… Minimal.
That's better.
Okay. So… I talked about the computation definition.
I can actually ask RRD. What is the maximum value.
Of the things that have been craft. And we can see that below the graph.
and put a line there. So, as I'm looking at things… Did you do?
Oh, I went back. Sorry. Like, if I'm looking at the 1006… I know… That it is below the maximum.
1006 was above the average, which is another computation.
and graph. And it's well above. The minimum.
Um, and the solid green line, when I scroll to the right, you'll see that was the most recent value.
So if I'm trying to compare current value against a previous time, like if I want to compare current against.
straight up 10 o'clock. Well, I know it's a little bit below what 10 o'clock was.
It is an above-the-average. And I can see max and min.
Does that make sense?
It does to me.
Okay. So… That's one graph.
Here is the actual output.
For the, uh… NetStat.
It's a whole bunch of things on one graph.
And due to natures of graphing, these smaller values kind of get lost in the wash.
But I can see multiple things. I have the maximum current average.
Minimum repeated. But below the graph, when I talked about, uh, creating something and printing it to it as numbers.
I can actually see… Uh, like, when this graph was.
created, there were 1,133 established. TCP connections.
inbound to the system. and 34.
outbound. Minimum, average. And maximum.
Makes sense?
Yes, can you go back to the top, and… Okay, that green line squiggle represents.
What?
Uh, that green… Um… will be… TCP established in.
I am not an art person. So, my color choices were kind of, like, do some math.
Okay.
And, like, FFAA. 5'5", and I just played with that pattern.
Right. Okay.
So that is me, not the tool.
I really wanted you to point out that there's a.
Again, the green at the top and the red at the bottom, and the.
Purple in between, and… How they're relative that you can… Grab multiple things.
Mm-hmm. This one, I think, has 27 different things graphed on it.
Right.
The third example I have… is a bigum.
Wow.
Yeah. Uh, this… and this is actually taken from some of the diagnostics I've been doing at work.
Um, we use Gearman and Mod Gearman in our Nagios system, so I've redacted the system names.
It's the only change I've made to this graph.
And I can see… Current minimum, average, and maximum.
Which… and it's doing it for… Like, 20-some… Variables… three different data points, like… Available, waiting, and running.
And we can very clearly see something fluctuates all over the place.
There's a trend across the top. Um… and it's like, you're talking about you can do more than one. Yeah, here's 60-some.
M1 graph. So…
That was that. You can specify a title on the graph.
This screenshot was taken very early in the process.
And I have the title. Needs to be fixed.
So…
I think that is it. That is… An introduction, teaser to… round robin database, how you can collect things.
And subsequently graph them and learn things.
Lee or Gary, do I hear a question there?
Yeah, um… Over a broader time period, uh… like, uh, let's see, on the graphs you were just showing.
Uh, what's the entire time period, left or right?
Um, these are 30-minute… graphs.
Well.
I can… get and show longer, so I have set up, because when you generate the graph.
First, when you create the data, the RRD file.
you specify how many data points and time on them.
So you can make it whatever you want. I have been keeping full granules, one second granularity.
for a day. And then I do… 5-minute, what is the largest value in that 5-minute.
intervals before that. going back a full week.
Um, and I have graphs at work created at.
30-minute, 90-minute. 3-hour, 6-hour.
12-hour, one day, one week. And, um, if anyone is familiar with.
Um… Zymon. you… it actually has some more web.
UI tricks where you can zoom in and have RRD. I want to see a graph.
Between timestamp that is Unix epoch number of seconds through Unix epoch number of seconds.
and how to show it. RRD supports that.
Um, there's a whole bunch of other options. But that's getting into more details.
Did that answer your question?
Yeah, yeah. That's kind of where I was going, was how… Uh, you can, uh, scale up.
To still be able to see that close in, in your case.
Yes.
30 minutes. But, uh… Obviously, you've done exactly what I was… looking for, and how to, uh… How to get a feel for broader perspectives for that.
Losing that track. Good, nice.
This.
Oh… So… Um, here… Don't know why… oh, okay.
Um, Zymon is a monitoring tool I use. The graph should look surprisingly similar.
And I can go to it, and here it's showing a 48-hour, a 12-hour, a 12-day, a 48-day.
And it goes back more than a year. And this is RRD.
You're the hood. Zymon has some options. where I can say, I want to see.
That time frame. And I can zoom in. Rrd supports that. It's what parameters you specify.
generate the graph. So, the data is there.
you don't have to worry about storing new data, you don't have to worry about removing old data.
You don't have to worry about the. gooey details of creating the graph.
create it, or you create the database file. You feed it data.
And then, as you desire or see fit. You say, give me a graph.
bound by these parameters, and it goes burp, here you go.
This is Stan. I have some. resource questions.
Okay.
Ah, um… Is… are there… is there a man page for this, or… Other documentation?
There are… there are many. Um…
And I'm just sharing just the web browser is the reason I'm bringing this up.
Ah, I mentioned… The math you can do?
Right.
is actually done as. RPN. There's a man page that talks specifically about that.
There's a man page. It talks about… the various definitions, um… So, when I create…
Where is it? Um… The green lines, and I did… the… I create a… VDEF, which is create something over the visual field.
Calculate that value. and create an H rule.
Which is…
There are many main pages, um, there are many tutorials.
I am happy to share examples I have. I was mainly wanting to wake people's whistle.
Would you say that this tool is in most of the.
Uh, repositories of the… Distributions.
has been for about 25 years.
Is it project status still? Reliable?
I've seen updates, uh, multiple times this year.
Great. Oh…
You have a lot. Of… enterprise… Monitoring tools, or… various things using RRD under the hood.
Now, as to a specific question, you may not know the answer to, do you know how large the variable names can be?
Offhand.
No, but I think we can find out. Um… Where is create?
I assume it's in here… Data source name, 19 characters.
Cool. It's always been a sticker for me, is… In my, uh, crawl, because… way back when, I remember basic.
It… you can have, like, a 15 or 20 character name.
But it only used, like, the first. 7 or 8.
characters are named, so you could… I wrote a whole bunch of stuff trying to make it work that.
had different… specific… Variable names, and it… just glommed them all together, so it was no good.
Yep. Um… I created this graph.
I actually tried. using the actual… host name, which some of them had.
Just the unqualified name, some had a fully qualified name, some had a weird.
Something else, um, and I actually had to shorten some names to come in under 19, or… 19 characters or less.
But it is still very easy to see. of the things that I'm graphing.
On the non-redacted version. what each thing gets. Uh, 19 characters you can probably be pretty descriptive.
If 19 characters is not enough. You are probably trying to put.
too much in one. Like, here, TCP established out.
that is the variable name. I believe, from memory.
I'd have to count letters. But it's something like that.
Excellent, thank you.
Mm-hmm.
And just about every graph I have created within the last two weeks has provided.
insight. Like, this particular system. I thought it was reaching out.
to other systems. Yet, I can clearly see in the graph.
That most of the established connections. are coming INTO it.
surprised me, but recording and graphing. provides those types of insights.
Or, like, yeah, the count of… something does go up of a morning.
But I also… hmm?
Have you for… Have you provided the output, the graphs, to other people to see what kind of.
Yes, um… I am actually.
Information they glean from what you presented. I…
I have collecting data and graphing on the same system at work, and it is the Nagio server.
And I am just outputting the… PNG files under docroot.
And I have created a webpage around it. And I have shared that webpage with colleagues, and multiple of us now have it open.
just updating in the background, and we're… periodically taking a look at it.
Excellent.
I have also… You know, while dealing… looking at the webpage, right-click.
I don't know if the right-click menu shows up in the screen share.
save image as… Um, and the way things work at work.
I connect to a Citrix, and then connect into something else.
And it's a convoluted… path, but I can… do… save an image, and it actually ends up on my work notebook's desktop.
and then I can then drag that from my work notebook desktop.
Yeah.
onto Microsoft Teams, the instant messaging work uses. And I can go, crap, this is a really weird thing. Hey, Chris, take a look at this, and drop it on a chat to him.
And it's 5-second old picture.
Build your own tools.
Isn't it a wonderful Kinker toy system? You can just put stuff.
Yep. It is…
All together in different ways.
I love it.
And, uh, if anybody wants to know where this pre… presentation is to look at it, and see the commands, or see examples.
It's already on the Slug archive, that's actually what we're looking at right now.
And the whole presentation will be. Video and sound will be.
It's… yeah.
up on our archive page later.
Gary, what question is floating around in your head?
And Lee is muted, so I can't hear you.
Any other questions from the… Group.
The floor is open.
Yeah, it looks like an interesting tool. It's called RRD.
There we go.
Round robin. round robin database.
round robin debate.
Yep. That's exactly what this presentation is for.
You have to remember that. That's exactly what this presentation is for.
And you say this thing has been around for a long time?
Yep.
Yep. Good.
Are they letting children change it weekly? Um, I don't think so. Good.
Um, I don't think so.
Maybe not even monthly. And maybe not quarterly.
Maybe not even monthly. You cannot quarterly.
I… I… I… You know, badly.
I feel badly. I'm not… well aware of this tool, and I'm very happy you presented it. It looks like a great tool to use, and I didn't.
I had heard the name before, but I really didn't have any knowledge of.
How good it was. That sounds real good.
Yeah, um, it is the foundation. behind the recording and graphing.
Um, the things I'm aware that use it… MRTG… cacti, noneos itself.
Um, and some other names… Libre NMS.
uses it. You see it a lot. in network interface statistics.
But you can tell that it can be used for many different things.
If you can get a number from a current point in time, and you want to track what the number is at different points in time.
RRD tool should be one of the early tools you reach for.
Uh…
Good deal. I like it.
If there are no other questions at this time, people can ask.
Cool.
Questions on our discussed mailing list. Correct?
Yes. And if you say, hey. I saw that neat thing. Can you help me?
figure out how to do this. Ask or undiscuss, or uh.
Find my email there, and let me know, and I will happily help people get started with it.
Hey, you're not going to get a better deal than that offer.
All right, well, thank you very, very, very much, Grant. Like I said.
All right.
Uh. I think we all learned a lot. That's great.
Look forward to you.
Great, you're… you're welcome. That was the intention. Uh, looks like there might have been some… maybe something else in chat.
Um… Carrie is saying graph is worth a thousand data points.
Yes. Yes, it is. Essentially, every graph I looked at over the last 2 weeks.
I learned something and asked. more intelligent questions about the problem.
That's a good rule of thumb. I… yeah, that's…
What's this gooey middle thing I keep hearing talked about?
Excellently stated.
Ah, funny you should ask. Here we are in the gooey middle of the meeting. Yes, it's time for.
announcements, general questions and answers, and things like that, so… Here we are in the gooey middle.
Uh, by the way, that tutorial was by Grant Taylor, and it was on, uh.
troubleshooting with round robin database RRD tool. Uh, and now to a GUI middle, um… Uh… as it said this is our normal setup for our general meetings on Wednesday nights.
Uh, and then we'll have a little call for help after announcements here.
And next slide, Stan. There we go. Okay, um… The, uh, next, uh… Uh, general meeting will be on the second Wednesday of next month. It'll be December the 2nd, uh, December the 10th, excuse me.
Uh, 8 days from now, uh, next week will be on Thursday, it'll be the STL Linux meeting.
Uh, and then 5 days after that, every 4th Tuesday.
is the new Linux meeting. Uh, the Slack meetings are usually on the first Thursday of each month.
Uh, interesting… dates coming up here, uh, are already past, uh, October the 14th, Microsoft Windows at its end of life.
Uh, Microsoft Windows 10. Excuse me, had his end of life.
And then, uh, next year, on March 20th. Uh, Project Hail Mary, uh, link to, uh, official trailer.
Uh, and uh… Did you go somewhere with that, Stan?
Well, that's good. Okay. Okay.
Uh, not really, uh, at this point, just, uh… call for other announcements would be appropriate.
Okay. Uh, alright. With that, uh… Let's see, uh, is there… are there any other announcements?
one in the back of my head, which I didn't, uh… make much effort for so far. As I had mentioned a couple months ago.
This time of year is frequently when people start looking for holiday parties.
And, uh, especially with our group here no longer holding physical gatherings.
Um, every now and then somebody says they'd like to try and get together.
Uh, as I mentioned previously. Uh, anybody who knows of one for techie-type people.
Uh, it used to be that, you know, we were one of the contributing techie groups that tried to organize some of these, and there were some other groups in town that did good jobs of organizing them.
I've seen a couple of them come up, I haven't put them on the calendar yet.
But, uh, if anybody knows of a good techie holiday gathering, by all means.
Send a note about it to editor at slug.org, and we'll try and put it on the calendar.
And, uh, let's see… other issues… Any other announcements or things coming up that we should know about?
Well, uh, just going along, I'd like to point out to anybody that's been living under a rock, there's a bye-bye Windows.
You can book that's available. in PDF form for free.
Um, there… if you can just go to Amazon.com, it's got a paperback copy for.
$11.95, and uh… there's a Kindle version, it's also available, and it's on… And if you've got a Kindle Unlimited membership is free.
Uh, it… it, uh, half the book talks about.
How to install Linux Mint. And the other half talks about how to make Windows.
10 work going to Windows 11. Uh, so it's pretty well-balanced book.
And there's another thing, there's a project, endof10.org, and.
came out of Europe. I've posted a few things there.
There's a couple other sites that posted things. I'd like to point out that there's a good way for.
other Linux groups to get the word out of.
that they exist, as… Over the last couple of years, there's no centralized list of all the Linux user groups, and this is about the closest thing to it.
that I've seen and they said. The URL is endoften.org, and it's well worth going to.
Um, be… And that's all I would like to add to.
To what's going on, I'd like to point out that.
this meeting and the St. Louis Linux users group meeting are recorded and posted on our archive page.
Uh, anyone might be recording. Now, and I also gave an example here on the… The general meeting, uh, slideshow on, on… the video that I did back… A couple of years ago in October 2022.
Um… Our archives are available, but the resources, presentations. It's also on every.
One of our supported groups' web pages down in the lower right-hand corner.
Um, we also have a calendar. And then on the calendar, we have everything that's… our sponsored groups are all.
In… in red, and I'd like to show an example real quickly of… This is the calendar, what it looks like. All the items in red are the sponsored things that we have.
Going on. So, and… This, uh… So too many things listed here, but the, the, the.
Next… next week's, uh… Linux user group is, uh… do we have a topic yet on that, Gary?
No, we don't. So, if anybody has. Any good, hot-breaking topics? And somebody who might be speaking on it, or be able to.
We're all ears, let us know.
Come on… yeah. Okay, okay… So… I forgot where I was.
Sweet. End of 10? Yeah, you're just finishing up end of 10, that…
Um, how did that go?
Here we go. Yeah, Andy 10. Privacy and security, arch, uh, archive page, calendar.
Oh, we have mailing lists, don't we? People can mail stuff to us.
And discuss things. So forth.
I'd like to say anything about this list, Gary?
The couple of them are really low traffic, for instance, uh, steering committee.
Uh, that's, as I referred to, that's where the sausage gets made. It's how we plan meetings and try and keep the organization running smoothly.
Uh, so that's where the sausage is being made. The disgust is really where.
Technical type issues come up. And, uh, again, that's a great place as Stan was saying, just.
Drop little tidbits of information out there. If you know something happening or going wrong.
Uh, in the technical world, uh, you know, throw it out there and get a little traffic going, a little conversation.
announce also is very limited traffic. It's usually just.
very specific messages that, uh… Uh, our meetings, for instance, this general meeting.
is being held on Wednesday night at 6.30, and here's how you get in.
Uh, so it's a moderated list, so yeah, if you don't want to see very much.
The announce list basically just gets you, you know, the needed highlights kind of thing.
Sysadmin is basically just, uh. Bro, system administration of our computers, of the user group's computers.
Uh, it's how we, uh… The people who are working on our computers to keep.
The mailing lists, the archives, those things running. So, uh, um… Yeah, that's why you would be on the admin list, is if you're helping do that work.
Do we have enough volunteers?
Funny, you should mention that. Yeah, uh, we can always use more volunteers. Obviously, administrating systems.
Something that seems near and dear to the heart of most of us in this list.
Um, so, yeah, you can help doing that. Uh, other things that we kind of need… Which may not be obvious, is, for instance, publicity.
I kind of wish that we had gotten. More information out sooner on tonight's meeting, and that could just be somebody.
Who does the postings to the meetup group, or who… Post to discuss, uh, so, uh… Um, you know, just other things.
Wasn't your clone able to do that, Gary?
My clone was… I don't know what my clone was doing.
But, uh… But yeah, so there's just all kinds of little things, uh.
And even if it's something, you know, rather than, you know, waiting for us to suggest.
You know, a particular role or whatever. You know, if you have an idea of what you'd like to help contribute to the group, it's probably a good idea, maybe, uh.
Just let us know you want to, you know.
try doing something, and it might work out really well.
Our loosely affiliated special interest groups. The St. Louis Linux user group, i.e. The lug.
Um, there used to be a number of them. St. Charles, and… Old Monroe, and uh… we're now kind of down to.
STL lug, and uh… Southern Illinois, uh… log that's still active. I haven't been to it in quite some time, but… I believe they're still meeting.
The new Linux, uh… Difference in that between the lug and the new Linux group.
Uh, first off, there's no… set topic. There's no… Uh, uh, pre-made.
Uh, uh, presentation. For the new lug. It's basically… Hello.
No agenda. Beginner workshop. The new guy in the room asks all the questions.
Yep. Well said, Stan. So, uh, and… but at the same time, it's good for the old guys to come in. The gray hair's like me.
Uh, because it's amazing. You think, oh, well, that's an easy question to answer.
And somebody else will, you know, with as much experience as you have or more, will say.
Oh, you can do it this way, and it's like, I've been doing that for, you know.
15, 20 years, and I didn't know you could do it that way, or I were trying doing it that way, and so.
Uh, it's amazing that even experienced people find some new information in there.
At the same time, it provides a lot of help to.
the new users. Uh, Slack, St. Louis Area Computer Club.
Uh, it's more hardware-oriented. They'll even talk about. Microsoft Windows.
Oh… For example.
And Apple. It's a hardware agnostic.
But that depends on who shows up. We don't have anybody that shows up that knows anything about the Apple, or we… Aren't able to answer the questions.
However, we can sometimes provide guidance and. And, uh, method of resolving problems.
Call for help. Okay, before we go on, uh, after the main presentation.
Uh, there'll be, uh… More time for a call for help.
Um, but, uh… Here, this middle point of the meeting, uh… Uh, for those who don't think they can, uh, stay around late or afraid that other.
People may disappear who might have the answers. Uh, this is another chance where you can just.
Interject some quick questions if you want, see if there are any.
Uh, quick answers here in the middle before we, uh… Get to the end. Anybody have any questions they'd like to float, or… Here in the middle.
What'd you break recently? How'd you fix it? What's the Mega Million?
What's your mega Million?
Can we graph that?
Can we graph that round robin database?
Does anybody here, uh… Familiar, real familiar with net plan?
And…
Boy, what a… I meant to avoid it.
I know to avoid it. Uh, Gary, yes, if you have the data.
Good graph. the Mega Millions.
Uh, here, yes, if we have the data. Craft Mega Million…
mute. Mute.
Mute. Well, I guess that answered my question.
By the way, way back…
way back… 25 years ago, maybe.
more. Uh, one of the physics professors down at SLU.
Way back 25 years ago, maybe before. Uh, one of the physicists down in Fluid-up.
Got it.
Uh, good lineup. coming up with enough mathematics to give you a, uh… Um.
coming up with mathematics into a, uh…
Bill?
There we go. Uh, came up with some equations that he actually was coming up with a way.
Not to do an absolute prediction, but… a nice narrowing down of the Missouri lottery.
And, uh, yeah, it actually was working, and then the Missouri Lotter.
People found out about it. But, you know, they changed the way they did it. But it was a… It showed that you could possibly.
If you do the math right, make a winning on some of these things.
Alrighty, um… Let's see… main presentation this evening.
Lee Lambert is going to do the main presentation.
Uh, it's going to be on, uh… Using… I… Uh, I… I screwed it up at the beginning here. I simplified the towel when I built the slide deck.
Upgrading your network tools. Upgrading your network tools. Good! That way I don't have to get my IPs and IFs configured.
With that, I'm going to turn it over to Lee, and thank you very much.
All right, folks. Uh, for… probably… 2 or 3 years nowadays, I've been duncing around with.
Lee, do you want people to hold their comments and questions to the end, or what?
Well, what happened on that? I don't care.
Thank you.
Do whatever you feel like.
I've been fooling around with various, uh… If you went here…
We see the title, Upgrade Your Network Tools. Thank you.
Cool. Uh, you know, every time I sat down in a system, I'd have to install network… net tools deprecated to get IF config, and… Kinda tired of doing it, so I had to start learning the new versions.
And we're going to talk about those tonight. There is only one minor problem here, which… Uh, Grant alluded to.
This is the Linux world. And the Linux world is… You know, migrating forward, moving forward in terms of technology and upgrades.
I'm really amazed at some of the stuff coming in the new version, sushi version of Leap, SUSY 16.
You know, so a lot of the tools are migrating into.
another… type of tool where… They try to simplify the logic, make it make sense.
In the meantime, they really screw it up. For people that are used to the old tools.
So, I thought it'd be a good idea to… Go through some new tools like IP and SS and things like that.
And take a look at it. And also, I will share a workshop sheet.
to editor, and Stan can post it on a website when he saves the recordings.
And if anybody wants a copy of it up front, just let me know, and I can send it… email it directly.
Start off with, we have… Some old tools, a lot of them have been around for 20, 25, 30 years.
I have config, route, ARP, netstat. IP tunnel, I'm not even sure what that is. Grant, if you wanna… Chime in here, just, you know, feel free.
Uh, that I believe has to deal with multicast.
Okay. Uh, each one of those commands had a different syntax.
A factor getting more important over the last few years, and it will be.
More important yet, moving forward. Is it that a lot of them don't talk IPv6 very well?
In addition, they were fairly inconsistent. We'll talk here in a few minutes about IP Route 2.
And that is one of the modern replacement that unifies all those tasks under one coherent command. It's called the IP2 suite.
And the command is… IP.
Back up a second, the suite IP Route 2 is the current standard networking toolkit.
It replaces if. with IP address, IP link.
It replaces route. with IP route.
It replaces ARP with IP Neighbor. replaces net stat with SS socket status.
Uh, similarly. Say again?
I have to… I have to heckle you a moment and say IP.
Doesn't do anything for some… Esoteric things that if config did.
Good point. Good point.
Like, IPX or Apple Talk.
Uh, anyway, uh, design goals… IPv6 and IPv4 natively.
Easily handle multiple addresses per interface. Oh, wow. That brings back memories. 30-some-odd years ago, the first Linux I actually put together was called Linux Pro.
And I sat here for… an hour trying to figure out how to add.
a second IP on interface. And I actually had to call him out in Colorado.
And they explain the syntax to me. You know, luckily, we've been doing it.
for those 30 years now, and IP simplifies it yet again.
The IP route tool set. I also handles policy multi-path, and a lot of the other.
Technologies and protocols that I don't deal with. Uh, is, uh, Apple talking some of those in there in some fashion, Grant, or are they just non-existent at this point?
Um, they are not inside IP, uh, or the IP route to suite.
The LegacyNet tools still are used to configure them.
Okay.
But that's, like…
Yeah. Got it, and luckily, you don't have to install it, you don't have to install it every time, just because it's the only thing you know.
Some of the core concepts. IP, object, command, and options.
Standard… Command syntax.
For example, IP address show. Although, I will mention that.
IPA… is my default.
See if I can get some… Morty's on the screen.
And that does it all you can do… also do.
IP address show to get the same thing. You know, is there a show?
Okay, so A is short for address?
Right. And… Grant kind of, uh… Beat me up a little bit on this.
And especially with IP Neighbor. He was saying that.
for our learning experience, we need to use the.
syntax at its most memorable, even if it takes a few more characters.
But being a true Unix head. If I can do anything in 3 characters, I'm certainly not going to use 12.
And I support that. It's just while teaching people.
I go the long way around the mountain to use the long command to make it easier for people.
Right.
to learn. By all means, when you're in the thick of it, troubleshooting, use as short a command as you want.
Okay, so again, IPA is a shortcut for IP address show.
If you want to turn up an interface. I have config interface up, I have config interface down, or.
IP link set Ethernet, whatever, or whatever the interface name is up.
Wait a minute, I don't understand what that's doing.
Far down. Okay? Uh, you used to be you had an IFUP command.
Oh, you turn it off, turn it… turn it on.
IFDN command, take an interface up or down.
Right, enable, disable. Any IP… yeah, in the IP world, it's IP Linkset E-T-H zero up.
Thank you.
Which is, you know, quite a few more characters, but.
IP is always there. Link is always there for a link.
Set is the command. ETH0 is the interface.
And up is the parameter for command. The argument is, and… I'm actually getting used to it myself.
The command is…
And in doing so, if you can remember. The order…
And that it is to remember that. Oh, uh, this is Rout. Route add…
It's in the older… IP libraries.
Instead of route, IP route? And… And… subnet via an interface?
This actually makes sense. To me more than doing it with a route command.
Although I have memorized route. Add default gateway.
And the IP address on the interface, because I use it all the time around here.
Because I've got… we've got one firewall box. That interfaces the office to… Our main link, and also we have our.
server room on that box. So, if the main link goes down for some reason, like Schroeder was prone to doing every couple years.
I'd have to change the route around to route.
The office traffic out our fiber. The dedicated fiber.
are what we used before. Which actually started on… 128K ISD hadn't 40 years ago.
As Grant was mentioning, here's IP Neighbor, which is the ARP commands.
IP Neighbor Show. And we'll be hitting them again here.
As we walk through. Think of IP as the single command.
Which, logically it is. And then a number of sub-commands.
For our specific network object.
Yes.
That makes sense to everybody? So what if I wanted to determine…
what my IP neighbors are. There's ETH0.
Which is my default route for the office fiber.
Here's 10.01 ETH1. Which is, uh… Another subnet, I think that's the VPN.
We have a 172 for Docker. And we have a specific route of 150.
I don't know what it is, but it's stale.
And 172 is also sales since Docker isn't running right now.
And those numbers change. From time to time, as your day goes on, right?
Uh, these don't here. Because… they're fixed.
Activity.
Okay.
Sometimes they will, sometimes they won't stand. Grant, do we have any logical rules we can… Talk about, uh, on what occasions there, or what times are going to change.
Um, in… your environment, they probably won't change very often at all.
In… like a VPS environment.
Where you have your… virtual server and a separate company manages the infrastructure and the default gateway.
they may swap out the gateway using some various options.
So a MAC address may change. Um, you could stop communicating with something, say you need to send email once every few hours.
And your system may… time expire out.
the MAC address for the mail server, if it actually hasn't communicated with it recently, if it's in the local network.
Um… Layer 2's usually, largely… boring and almost static.
But change is just… frequently enough that it's not static.
Um, or if you change wireless networks. You might see some things change.
Okay. Well, this is actually VPS up in Chicago, I think.
And notice the difference between… ARP neighbor, and route.
Obviously, it's two different functions. So we're talking about two different logical parts of IP Route 2.
But… You know, we have…
a Docker route here. is down because Docker's not running.
But it doesn't even show as a neighbor. ARP-wise, whereas it did… on my workstation here in the office.
I note that my local land changes based on what I do with the different machines in my local.
Um…
setup. Throughout the day.
In, uh… Let's talk about a coffee shop type environment.
Where you have people… coming and going with their notebooks on the wireless.
network, um, you would see. Presumably, if they send a DHCP broadcast, hey, can I have an IP?
Your computer would receive that broadcast, so you may have a neighbor or ARP cache entry.
for that computer for a few minutes. And there may be some churn as people come and go, or they've got their IP and they no longer need to broadcast.
Um, or if somebody is being malicious and trying to use something like FireSheep.
Go back a decade or more now. And mess with you, you might see them… with that, or a Wi-Fi pineapple?
Um… you might see a little bit more churn.
Thank you, that's a good way to explain what's happening.
in a network that way.
Cool, thanks. Grant, technical term there, an IP pineapple.
Mm-hmm.
Wi-Fi pineapple.
Wi-Fi pineapple. Um, it is… a tool.
that allows you to do… some various things on the network and wireless layers.
That normally are not done. And you're either a security researcher.
A student… Or you're up to no good.
Are the NSA van, right?
They are one of the three in that. list.
Hello.
Essentially, man-in-the-middle stuff, right?
Uh, yeah, that's the… Realm, I believe, Wi-Fi Pineapple plays in, and there's some other things that do that as well.
Um, there are some leg… legitimate reasons to do things like that, usually in a corporate world.
Um… I would expect… to not see that type of behavior.
At your local coffee shop.
Okay. Anybody have any questions about.
You know, some of the basics here. What is IP Neighbor showing again?
Oh, wipe my tongue.
Yeah, please.
That shows who on your land is active. Right? Or has been recently.
close, doctor. that your computer has seen.
Yes. Okay.
So, that is germane. Because in a switched network.
Like most offices are. The… your coworker's computer.
talking to the server. They have already… cashed each other's MAC addresses, and they don't need to send a broadcast.
They can talk no problem. your ARP cache or neighbor show.
Should not show them. Because Layer 2 switching.
between your coworker's computer and the print server, or the file server, whatever.
your computer doesn't see it. So it's things your computer has seen.
Not necessarily all activity on the network. Does that difference make sense to you, Stam?
Yes. Lee, just to throw a monkey wrench into everything, just by itself, would you type IP to show what it does by itself?
Boom. Thank you.
Sure. You'll notice we… it is… The same syntax we were talking about a few minutes ago.
IP command, whatever options you have. The object for the options, and then the command to perform the operation.
Okay, what's this guy do? IP route.
Lots of interesting things. I have fun with that one.
Well, what is it showing here?
Um, do you want me to spill the beans?
No, not yet, but let's see if anybody learned anything yet.
Okay.
I did, but I'd forgotten it already.
Should I give them a hint, Grant?
Sure.
Put that together.
Default routing. Shows you the pathways.
Right, right, you got it, Stan. This shows you the configured routes on the machine, our pathways, if you will.
Default is your default. Outbound link.
via this interface. On device, ETH0.
And… I don't remember what Onlink is. Grant?
Um, that is… You might have heard the phrases directly attached.
Um, that is, it's… part of the network. You're part of the same.
pool in the same room, in the same subnet.
Routing can actually do some recursive things. Um, so… Scope link, and… I'm not sure why that's on Link versus others.
Do a man space. IP dash route.
Search for OnLink. ip-route.
Uh, IP dash route, no spacing. Search for OnLink, one word.
Ooh.
Yes, IP Route 2 has ex… extensive man pages.
Oh.
Keep going. You should get to a description that is…
Like, a one- or two sentence thing.
Pretend that the next hop is directly attached to this link, even if it does not match any interface.
Okay. So that is… and that was on your default.
Ah.
that is a way of… Routing is really dealing with destination network addresses.
Um, and you end up dealing with trying to match things.
It is saying… For traffic that is going to.
All not otherwise specified. destinations, and we can talk about what otherwise specified in a moment.
Um… goes to the default.
on the link, even though the destination address is not.
part of the link you're sending it to. So if you'll go back and show your IP route.
I've got an extreme question I want to pop in here at some point.
Please.
Thank you.
Momentarily. So… When I said otherwise specified the next 4 lines.
You have 4 subnets. explicitly defined.
Uh, the… Network configuration has… to find them, and they are scope-link. So, 10-10-10-0 slash 24.
is on the link, WGTP Hub. And you've got some others as well.
152, 370 slash 24 on ETH 0. But we want to send to a destination, like 8.8.8.8, or 1.1.1.
1. send it via 107-152-37.1.
on the ETH Zero link, even though the address we're sending to.
doesn't match 107 anything. Does that make sense?
Kinda.
Okay. there's a lot of minutiae. You don't necessarily need to know this to successfully use it.
So, this is one it is probably okay to gloss over.
What's your question, Stan?
My extreme question is, can I use this to make one subnet talk to another subnet?
Yep. It's… that's what's happening under the hood.
And here's your evidence, Dan. If you remember.
Oh my goodness.
Uh, last month, we talked about a VPN. So, this is… the subnet for the VPN.
It's WG-TPHub. Remember?
No, but I'll click…
Okay. The, the, uh… connecting a Windows 11 remote machine to a Linux server over VPN.
This is WireGuard TP Hub. This… this VPS happens to be that hub.
Okay.
That we were talking about in that presentation. So this is the name, it's WireGuard TP Hub.
Okay.
The remote machine connects to this. BPS, and this VPS has a static connection to.
the Linux server in the office. And… These are… or this is the network in the office.
Where we want to talk to two servers. So, basically, I… asked, or I calculated the smallest subnet.
That could… that include the two servers in the office.
And this is the subnet for those servers.
I'm hoping I can… Use this to have 192.168.1.
Whatever, talk to… 192.168.2.whatever.
Is that possible, you think? Oh my goodness.
Yes, it is. Um, you notice how… The 107, 152, 370 slash 24.
is part of Eat Zero.
Oh my goodness.
So, Lee, just highlight that entire line, please. stand hypothetically, let's presume instead of 107-something, it's the.
1921681. And there's an ETH1.
Yes. Yes, because what hap… what happens is my phone shows up as a dot.
that has 1921682. Something slash 24. That is what you would…
There's a .2, dot something. And I want to be able to see it in my 1.1. Something.
Roll up.
Um, you will actually have to do routes. But this is how you configure the routing for it.
Yes. Yes.
Understood, understood.
And I would be happy to… Yeah, that sounds like a great discussion on, um… discuss.
Hypothetical, here's what Stan might do type thing.
And if he takes good notes, it can be a base next month.
Yeah.
That's scary. Stan takes one step backwards.
Uh, how about, uh…
That's okay, you took two steps forward.
Okay, but…
A heckle equally, Stan.
Okay. Well… At this point, I'll give a National Guard command, middle round men, middle round!
All right, anybody else want to talk about anything here on.
IP neighbor, or IP route?
Alright. So we talked about the basic command concept.
Legacy Command equivalents. ifconfig.
IP address show… I think that's supposed to be… yeah, R… Hyperlink show.
I have configured ETH0 up. IP-Link said HTH zero.
Assign an IP. I have configured ETH0, IP, and subnet.
IP address add. To device ETH zero.
Route minus N. IP route show.
Add a default. Route add default gateway.
IP route add default via. 192168.1.1.
R minus N. IP neighbor show.
Our delete from the cache. IP neighbor delete.
Device E T Hero.
Any questions on these?
Working with neighbors. What's ND, Grant?
Dear Discover, okay.
Neighbor Discovery. Uh, ARP is… nominally IPv4.
Some other things, technically, this BEC allows it. Um, IPv6 renamed that Neighbor Discovery.
Okay, so, and ND is the IPv6, okay.
Yep.
Where are you? Oh, okay, I see. Yeah, and ND is Neighbor Discovery IPv6, which we don't use, so… Uh, I'm dreading the day we have to get there, but… Playa will be… it will be long past.
By shutting the business down.
IP neighbor, and go ahead. Go ahead, go ahead, Grant.
Oh, isn't. And… IPv6 isn't that bad.
I didn't say it was that bad, I just said we don't have a use for it.
That you know.
Well, since we're not routing any IP… any V6.
I see.
Uh, anywhere. I would have to say that.
It's a non-issue.
Fair enough.
Okay, so if you want to add an entry?
IP neighbor ad.
IP address, link level address. MAC address, device E T H zero.
I'm going to delete it, IP neighbor delete.
IP and device.
Does this stuff, uh, take effect immediately when you do this?
Yes, it does.
It does… yeah, it does on this machine, but… You're not purging entries anywhere else on the network.
Network at this point.
The other thing to keep in mind is IP command is real-time.
Um, it is not. persistent.
So it's not, or uh… Startups.
Yeah, it's only changing what's currently saved in the running kernel.
Good point, thank you.
Okay. L Louder.
Uh, link level, link level address. Okay. And… in the Mac, okay.
MAC address, yeah.
Uh, yeah, link level address, as opposed to IPv4 address, as opposed to IPv6 address.
Gotcha. Okay. Yeah.
Write the settings…
Now, the wonky one. Or, or… SS. Socket status. Again, Linux only, the comment grant made.
Linux only.
Or the reminder.
NetStat is deprecated. And slow, primarily because it parses slash proc directly.
In addition, it doesn't support V6. are a bunch of other protocols that have been.
Implemented and… Propagated… Long after NetStat was designed.
I question the capacity of the doesn't support V6.
SS.
I don't know.
I'll confirm momentarily.
Okay. part of IP Route 2, fast, filterable, and script-friendly.
Basic equivalences. Netstat. SS, the main… the root command.
That's that, tango uniform, Lima November. SS, tangle uniform Lima November.
Tango Alpha November Papa? Shame.
RN route show. IP route show?
That's that minus S. SS minus S.
Contemporary net stat very much so does support IPv6.
See what we got here.
Say that again, please?
contemporary net stat. Does support IPv6.
Thank you.
But it is still subject to the inconsistent syntax.
that Lee was talking about earlier.
Alright, here's, uh, SS-S on the VPS we were looking at a few minutes ago.
229 active, establish 1, closed 0, orphan zero, time weight 0.
Here's my workstation, 80. Established 26 plus 40, or from 0, time weight 0.
Notice that my local machine has… Has a lot more traffic right now.
That's interesting. Or the actual IPv6 coming from, Grant?
Um… do IP and hit enter.
Um… do an IDP space dash… Six space A space S.
6. address, or a… Yes.
Again, there's… Oh, and… And then…
Okay, um… If you… Yeah.
Okay, that's fine. Because these guys have V6 addresses signed, even though.
They don't have a route.
Yeah. There's a… well, they do. It is just on… Um… your ether interfaces, those are link local addresses.
So, you have two IPv6 networks right now.
You're just not utilizing them for anything. Um, if you run the proper SS command, and I'd have to look it up unless you know it.
You should be able to see. What… Um… IPv6.
connections are in place. And something to keep in mind is some programs.
that support IPv6. actually use a form of IPv6.
for their IPv4. connections.
docket status, IPv6. Unconnected.
No packet traffic. Local address, ICMP.
Yeah, I wasn't even thinking about that. Uh, the kernel assigns an IP, or a V6 address by default, even if you've got it turned off.
Our MAC address, rather.
Yeah, I… to truly disable IPv6, you need a, uh… PROC or SYS Control setting.
Yeah.
Any questions at this point?
SS Manus Tango Uniform November, early November.
Bunch of stuff going on here.
Can you add a P to that? That's the… List of things. Papa.
That doesn't fit.
Okay.
It shows the program that's running that particular thing.
What's the PD, Stan?
Yeah. process, and NetStat, it showed the.
on some…
Hid, which sometimes would be a command here, it's a process and.
Yeah.
you're getting more details.
Right.
You know, a lot of them are defined, but not connected, though.
It's just line wrapping on his terminal.
Firefox.
Yeah.
Python 3.
Yeah, this is the, uh… Slide deck.
Uh-huh.
Did you… Put sudo in front of the command.
Uh, I run a…
Yeah, there's more… process information that you couldn't get without, uh, root privileges.
That's good to know.
Uh, you're running Bomba, aren't you? Avaki, that's bonjour.
Zoom is listing on a bunch of things.
Yeah, Zoom… Zoom's causing a bunch of it. I do have SAML running, because.
This machine is a server. bar my Windows VMs when I'm working on a client project.
Pesky clients.
Yep. And then…
Yeah, but they tend to do nice things, like pay the bills.
Cups, obviously…
billable hours, billable hours…
Could be…
You say Microsoft Windows does have some purposes. Yeah.
Available hours, billable hours.
If nothing else. Windows pays the bills with stuff I don't have to worry about.
And I can ignore as much as possible.
All right, so remember, that's one of your good… Uh, basic commands to ULNP.
Like, you know, grab for SMB connections.
Uh… come on, now. Uh, I know SS has… syntax for filtering ports and IPs and various things. I don't know if it could do.
a name… Like you're doing.
That, to me, is one of the advantages of SS.
is it includes… some filter capabilities that normally, with NetStat, you would just go straight into a grip.
Yeah.
Now, so I… I haven't unlearned those habits yet.
Mm-hmm.
Fair enough. I'm guilty as charged as well.
But, you know, if we're embracing the new. It's embrace the new.
Amen.
And worry if I didn't heckle you.
Oh, wrong state name?
Well, it looks like we got a faux pas here.
Or is that, uh, that require privilege?
No, it shouldn't.
Nope.
Check the man page for SS.
Search for state.
Missouri.
Misery.
It may be established fully spelled out.
Thank you, Stan. I was hoping somebody would bite on that.
Isn't it misery?
That's what I said. Misery.
What is this showing me?
Uh, SS information. filtering on established connections.
Spelled… spelled properly, established TCP sessions.
Anything else we should demo, Grant?
I don't know. Um, I will toss out there.
Yeah, go ahead.
If you're done, I have some comments. Regarding, like, um… IP… Is the method to do some.
newer, fancier things that… have come out in the last 10 to 20 years, like.
MPLS. or network namespaces.
Manage… some tunnels.
Uh, you can manage transformations, and that. comes into play with things like IPsec.
encryption, compression. those are managed with the IP command.
Or they can be. Go ahead.
What do you… What do you mean by namespaces?
Um… Trying to think the… You know how if you have virtual… 4 machines, each machine can have its own independent network stack.
Right?
Imagine having those independent network stacks. all under one… running kernel on one system and different identifiers.
for those network stacks.
Okay. Would we have.
That is Network Namespaces adjacent to that, there's… Of slightly different thing.
virtual routing and forwarding. associated as well. It's a more advanced.
network thing that… Most individuals.
will not use. But it is something that the Colonel is capable of.
containers make extensive use of it. I, as a weirdo that I am, will.
create multiple virtual routers. as network namespaces.
And I can play with routing and do… a whole bunch of nifty things, like if I want to.
create a chain of 5 or 6. routers to emulate, like.
router, I've got a virtual internet cloud router. and I've got another virtual ISP router.
and another virtual home router. and I want to play with… doing tunnels between.
If I want to simulate, like, my office versus Lee's office, and.
Minimum. Internet simulation.
between those two emulated routers, and I want to play with VPNs.
in point-to-end point, I can do that. all with the IP command.
on one machine. Most people don't. I do really crazy things.
Yeah, but you have so much fun.
I do. I call it edutainment, educational entertainment.
Yeah…
You can also use SS. To filter.
Here's some of the filter you were talking about, Grant.
Yes, yes.
Yep. And your state, your source, your destination. you can put those all on one line, so I could say… Um, and listening won't have.
a remote IP. But I can filter.
with SS, my mail server. IP slugs mail server IP.
Port 25. as just on the SS command, and I can see… connections from my mail server to Slug to deliver a message to discuss.
Well, SS… does SS have a show pit option?
Yes, that was the "-P. that Stan asked for.
Ah, right, right, that's what I stand. Thank you.
Yeah, I can remember many times over the years trying to show a PID for.
Mm-hmm.
A socket that was open. And I think LSOF use on BSD, I couldn't figure out a way to do it for some reason.
Yeah.
It really depends on the version and what commands are installed.
Uh… Yeah.
Okay, so filter on listening.
Hello.
State listening.
Mm-hmm.
It's MTP… Microsoft… Anybody ever seen this on Linux system?
Yes, unfortunately, not one I've logged into, but… friends, yes.
And also that… is just a translation of the, uh… Port number to name.
And probably comes out of the Etsy services file.
Yeah, but it's valid.
Yeah, this machine actually has…
SQL Server running.
I'm sorry.
That's a bunch of green bone stuff.
I need to kill that.
Here's a bonus question. How do I… how do I shut down something?
45 caliber, usually effective. Uh, 9 mils more portable.
Yanking the power cord's more fun.
Yeah, but then I kill everything else.
Good way to test your backup system, right?
Dr. Kapoztown.
Slight distraction.
You know, see, filtering for… use UDP, right?
Grant Manager. And T for TCP.
Um, I believe so.
On source or destination. IP address.
Or you can look for 443 established connections.
Yeah, but I would not do the grappy stab.
Good examples.
I would take the… from the pipe. to the end off and do… I think it's AND.
state space established.
Oh, so in other words, replace this with AND, AND for establish?
Yeah.
And spell out established, yeah. SS can do all of that on… One command.
Okay.
Anybody want to see anything else?
Um… Question, uh, can you… can you… feed in lists of IP numbers, or host names?
Into this? These commands?
Uh, I think you can do… oars, like… host IP address or host.
IP address 2, or host IP address 3. I'd have to look up the exact syntax.
But you should be able to.
Again, type in IP at the command line by itself.
Yeah.
IP. batch file name usage.
Oh, wow.
That's something to explore. On my own.
Um, I think…
Interesting. Why, why is the syntax different, though? So this…
Um… probably IP domain-specific language.
Like, the subcommands? And only IP.
Not other… things that might be germane if you're trying to do multiple things.
all at one time in one IP invocation?
This is going, yeah. So, well, my first thought is Stan wanted to dump in a list of 100 IPs.
You would have to include… The command and the list.
You know, the command and the list in the file.
I thought he was wanting… To do with, um… socket stamp.
Okay.
Yeah, that's what I was thinking about, yes.
I don't see any evidence of a batch.
Filter, state filter, GCP states. Connected.
Hmm.
That comes in version 4, right?
Dan, uh, clarify what you're wanting to do with multiple IPs?
Their version up here?
I was just thinking of some of the examples we had here to just, you know.
selectively do. You know, 2 or 3 IPs.
in the stuff we've already gone through.
Like, adding them to an interface, or looking with them… Or for them in socket stat.
Perfect. They're adding the IP thing.
Oh, um…
Yeah.
I mean, I was… I was thinking about, you know.
It would…
Dealing with the series. 192, 168.2… whatever, into 192.68.1.whatever.
Mm-hmm. Um…
0 slash 24, I guess.
So, that would be… Like, IP address add 192.168.
1.1 slash 24 device. ETH Zero IP address add 192.1682.
2.1 slash 24 device ETH0. two separate command.
Or some subset, yeah.
Yeah, Stan, this might be something that. Play with.
Ooh.
Filter equals file. Each line is interpreted like a single command line option.
Wow.
You know, every time you read a man page, you don't necessarily understand every… thing is there until the seventh time through.
Amen to that.
And a presentation like this really, really helps. expand the brain cells.
Amen for that. Yeah. And besides, you know you're gonna sleep well tonight, you've got so much brain exercise.
Alright, uh, let's see… Bonus statistics! Yeah! Displays TCP UDP totals, retransmits queues, great for diagnostics.
dry watch for live stats. Everybody knows what watch is, don't they?
Yeah, kind of.
Watch it change.
Now, if we could just put this into a round-robin database.
Uh, yes, you very easily… Could, well, relatively easily.
The tough part, which I comment on one of my slides, is.
getting the data out of something, and I wouldn't use watch.
I would actually just do the SS-S. Um, could very easily use AUK.
to look at… Field 1 is equals TCP.
to know what the next 3 fields are, and you could then generate.
a synthetic RRD update. in for now, colon… field 2, feel 3, field 4.
Uh, and I actually show an example of that in the slide deck.
So instead of watch, you use a ground job?
how I populated for the example data.
No, for the example I've been doing, um, I put.
a while true. socket stat-s, pipe it into AUK.
And the AUX script would run a system which.
call which calls already update with the parameters. And then, like, a sleep one.
Okay. And then you just wait for… for… 500 milliseconds or whatever.
One second. Yeah.
Okay.
And that's what I have been doing. For multiple things for the last week, two weeks.
Okay.
Okay. Cool.
So yet another reason. to learn something from what we're looking at arbitrarily.
gain insight experience.
All right, almost there. More examples.
IP minus brief, I haven't looked at that.
Okay.
Um, do IP-based-1 space adder.
Uh, I'm sorry. Double dash… the Word 1… dash line.
I think. It's kind of ugly, but it puts it all on one line.
Which, if you're trying to parse and process.
Well, this is… this is one line. is wrapped.
Yep. you notice the backslash and extra space.
Yeah, if I shrink the screen, you actually get one of mine.
Okay. Okay, yeah.
IP route show default.
Um… One thing to be mindful of.
is you have what is called a default route.
based routing and… Rules for what routing table to use when.
terms reused. Particularly default.
Okay.
I'm not gonna fool with my interfaces. We talked about IP-Link set.
Device up by peeling set device down. We did neighbor show.
We did IP route, and we did… Shocker status to U L M P.
So, a side benefit? of the redesign.
It's easier… to pump them into scripts.
Structure predictable output. And… you actually have a JSON option.
JSON option, if that… Works better for your script.
Typically, JSON is easier to parse because. The data has good delimiters, basically.
Although it's a lot harder to read. It can be a lot easier to programmatically pull out what you want.
It's fast, it doesn't block. Runs IPv6.
It's great, great script use.
Um, I actually question structured, predictable output. Um, I think… Some people have said that.
But I have seen IP output. change across versions.
of IP Route 2. Um, so I consider scraping.
Stock IP output. And ifconfig and net stat.
to be comparable. That being said, the JSON output.
is going to be much… More machine parsable.
How… hard would it have been to, uh, use JSON in your application?
Um, considering it's Linux-specific, and mine, I specifically want to be.
cross-platform. JSON would have been a non-starter on.
Most of my target platforms. Because there is no IP.
Well, in addition to which some of your target platforms don't have IP and SS.
You were saying it. In addition to which, some of your platforms don't even have IP and SS.
I'm sorry, say again?
Um, only one of the platforms I work on does.
Oh.
Okay.
Quick reference cheat sheet, I think we've covered all these.
There's one thing I have not figured out how to do.
with IP route 2. in using it for 20 years.
And that is how to assign an IP address and bring an interface up.
in one command. You know, the venerable… if config ETH0192.168.1.1.
slash 24 space up.
Huh.
I'm typing. if config…
ACH.
Um… space, device, space, IP address. slash netmask space.
Up.
That's 24 up.
Let's see if ChatGPT can come up with a quick answer.
Okay.
Yeah… Nope, two-liner.
Mm-hmm.
Got the address… And set it up.
ampers and ampersand.
Do what? Oh. shell trick to do it on one.
Never mind.
10%, think about it.
command line, but… it's still too convenient.
But it's not one command. Everybody follow that?
Right. Yeah.
So, set up… Uh, see where was the other one.
Nice.
It's almost like I know what I said.
I like it, and I do what… But you said that for other things frequently.
Yeah.
90 check comp dash Z. ampersand, ampersand, R and D.
See, reload.
And somebody else tells me what… bank or whatever ampersand, amp percentage.
You're… you're a script junkie.
That's old school. Shell.
Oh, yeah. Well, that's also why I ask, because he's older school than I am.
Oh, wow.
Ah.
So, Gary would not be running the second command.
I don't recall. Well…
Because he failed to remember.
Yep. Yep.
You like that leak?
I don't know. He's sitting here, he might whack me on the head.
Well, the ampersand, ampersand is… shell nomenclature for run the subsequent command.
if the previous command. completed successfully, as in return code 0.
So… Go ahead, Brian.
I was gonna say, it's a logical land.
I was gonna say, it's a logical lens. Not to be confused with two vertical bars.
That's an aura.
That's the North.
Anybody else?
I'd like to point out that. James Shaw put something in the chat.
Specifying that there's a… Um… Verge of Bethesda saying rrlug.org.
Tomorrow night, apparently a meeting.
Sure.
Care to comment on that, James Shaw? Turn your mic on.
Yeah, uh, sure. I, uh…
I spent 11 years in Rockford, so… and when I first arrived,
I met up with our lug,
And they had already been around
Wow, good. 10 years before that. So, I mean, they've been around a while.
a while, so…
And, uh, I…
When I moved to, uh…
Chicago, I never lost.
touch with them, so…
Okay. Did I do direct traffic?
Do they…
No, but, uh, there's one guy who's a… he owns his own, uh, security company.
So, uh…
there's a lot of security talk.
He always talks about the latest thing that comes up.
Hopefully, he'll be there.
And it'll be…
Should be interesting. It always is.
Jitsi, the…
or whatever.
The leader is very…
open source. So, he only uses, uh,
Jitsi. So…
F.
Uh, well, just for information, I've been using big blue button.
Because that's what Odo uses. And I kind of like it better than Jitsi or Zoom.
Really? Well, let's…
Discuss it tomorrow.
Okay.
How frequently do they meet? Monthly? Or what?
Uh, we're pretty religious. Every…
Second Thursday.
For the past…
30 years, so, yeah.
Uh, that is…
And at what time?
6 o'clock.
Yep. We used to meet at the… a local college.
Just like everyone else, and then…
And then virtual happened.
never looked back.
Cool, have an updated web page yet.
Sam, he's the… he's the main guy. He's…
So, uh, I heard you talk earlier about your frustration about Mozilla.
He was so avid on daily…
Firefox updates, and he just…
He's so lost.
his…
I don't know,
So, yeah, I don't know, when Mozilla started changing their…
Composure…
culture and uh… yeah, their culture, he just…
He was so hurt about that, so…
Hmm. Oh, good, he's got a link to end of 10.
I love it. I love that tombstone, that's great.
Yeah, no kidding.
Does he… does he post his meetings on end of 10? I hadn't noticed.
Um, I… you know, I… I looked that up, and I didn't…
It didn't seem familiar.
But, um…
Rockford, let's… let's tell Illinois. We're north.
Where is Rockford, and what's… what state?
very north center of…
the TIFA, Illinois.
Okay.
Uh, Rock River,
is… there's a river that runs right through Rockford,
Hence its name, a rock.
Yeah. Right.
forward passage, you know, so…
So, I heard about you guys…
I remember Rock… Rock River, yeah, Rockford. Okay…
There's a guy that shows up every once in a while,
And, uh…
He says, alright, you know, I, uh…
I belong to the…
you know, the St. Louis.
And then I… I looked you up, and my…
My, uh… my brother and one of my best friends live in St. Louis, so…
I was like, wow. Okay.
So, small world.
Yeah.
Hey, um… Unless there's more… you want to talk about Rock River, I've got a… Jump back, I want to do into the presentation.
Please go, yeah?
talk, Robert?
Uh, back on the topic here. Okay, so I… I see this… Written in different ways, and I'm wondering… What the differences are of what they're trying to say.
So, I see… things that are written IP, space, route.
And then I see things that are written. route, all one word.
And then I see things written IP route 2.
Okay, so… What's the distinction?
IP space route… Would be a command.
I suspect… Yeah, I suspect IP route.
Like here.
No space is a typo. Of the former.
And IP route 2? is the name of… the sweet slash.
package that contains IP. It contains TC… And I'm not sure what else is in it, libraries and various things in support.
So there wasn't a package called IP route, versus a package called IP Route 2.
Of it. Is that…
Not that I'm aware of.
And… You did a nice job.
Showing what's deprecated, I like that chart. Yeah. Which kind of leads to my next question, so… With, uh… Uh… with what release… Do we need to have gotten our stuff?
migrated. I mean, is there…
to have IP route to?
Or rather, not to depend on ifconfig, or that stat.
Uh, anything within the last two decades.
Right. That's, I believe net stat has been deprecated for about 10 years now.
Right? And it's… Longer?
Yeah, the… Oh, no, it was deprecated in… 2000 through 2005.
Okay.
So… IP route 2 is… Uh, well, it can drink, and it can vote.
Okay. Can it be drafted?
Probably.
Um, no, it prefers to be scripted.
There you go.
Those are inside jokes.
I love it.
So you mean all these decades, and so now… Wow, tonight, for the first time, we're doing a presentation on it. Okay, we've… Alright, let's wrap it up, Gary.
Um, I… well, I feel like it would be remiss to say.
All right.
that a lot of. user land applications that get installed.
depend on the older. often called net tools or net dash tools.
Um, like a lot of monitoring things, or some other scripting things.
that have been written years ago to the older.
route and NetStat. you still see those tools.
Mm-hmm.
very, very, very, very often installed.
Even if the OS uses… newer things, and the admins use newer things.
monitoring or some other… package they use, gotta install the deprecated stuff.
Well, there might be a… argument with that, because if it's deprecated, you have to install the package to deprecate it.
NetTools deprecated.
Still… yeah. Definitely.
It might still be available, but… I wouldn't… I wouldn't expect it to be installed.
Yep. Correct.
Unless someone stabs it with a fork.
But as part of the base, it's all.
It's not deprecated.
Correct. Alright.
Alright, so this is what we've been talking about tonight.
Updating your network tools. ifconfig… is replaced by IP address or IP link.
Route, IP route, ARP, IP neighbor. That's that SS socket status.
And some of the… More specialized IP commands.
Now have I… now I have IP versions, IP multicast address.
and IP tunnel. That makes sense to everybody?
goes to me.
Oh, alright. In which case… Old habits, again, I have configured route, ARP, netstat.
Modern tools, IP address, IP route, IP neighbor, SS.
Unified, scriptable, and future-proof.
Fortunately, the slight downside there is they're only in the Linux world.
And a lot of us have to deal with.
Other worlds… Other Unixes?
So there's… Yeah, X, whatever.
That's it? Ip help SS minus minus help. Excellent, excellent.
So, does…
Hey. If I can make an observation.
Excellent, excellent. Well, it does. Hey. If I can make an observation.
Uh, who, who's. No, no, who's got the echo here?
Uh, who, who…
Rich Seibel. Comes from Rich.
I don't know how I got it. No, I don't have the Echo, do I?
Yes.
In any case, an observation. Unix was created.
one tool to do one job. It seems like we're going just absolutely the opposite direction with this IP.
It's a catch-all thing. Throw everything in one pot.
You're fading out, Rich.
There you go. Could you repeat that?
Oh, sorry.
Um…
Uh, I was saying… But it looks like IP goes the opposite direction of Unix.
Unix was created one command to one job. putting together yourself.
revolted the hodgepodge of foreign air. Everything in one big application.
Seems like we're going in the wrong direction. Permix.
So, let me… can I answer that question with a question?
Um…
Sure.
So, does… does the IP command do anything above Layer 3?
of the IP stack.
Uh, that is a complex question, and I'd like to come back to it in a moment, and I want to speak.
Go… go for it.
To Rich's question first. Um… I don't know that it is actually.
One monolithic. Command. I believe… at least from multiple documentation and source code.
It is actually… a command with subcommands.
And they… basically, the subcommands are separate. separated out in some capacity.
So, it appears… as one command.
to a human. But I believe under the hood, it is actually multiple things.
So, and we're kind of seeing an evolution of that model.
Where commands are more… for humans in some aspect of.
you're thinking about this task, you're doing it, there are these various aspects.
it is logical for a human. how it's implemented under the hood.
What?
whatever is best for the situation.
Um, you will also see that in something like OpenSSL.
where you have OpenSSL in lieu of IP. you have X509, or you have, uh, hash, or you have various other things.
They're not all one monolithic binary. there are multiple things in a common command namespace.
PowerShell does the same thing.
Um, and it also makes it… quite easy to.
extend. and add new features.
And just join the existing thing. Um, and I believe Git.
also does similar things. Like, you can add. subcommands that are outside of it that appear in the get command namespace.
As for doing anything above layer 3. Um… That gets real tricky.
Well, SS… SS is…
is above Layer 3.
You're… you're kind of looking at the…
what application the port is mapped to, so it's kind of like…
Yeah, I was thinking… configuration, not reporting.
you can do…
traffic control, or the TC command. is part of the IP route 2 suite.
I would argue that. quality of service is under Layer 3, but I know there's also some things you can do.
With netting? But that's our gooply… Layer 3?
Um, tunnels. you could argue that is… happens above layer 3, but creates a new virtual or logical.
It's a gray area statement, I agree.
Layer 2 or layer 3. Um…
Yeah. I would say, nominally.
IP functions. on layer 3.
And the… Layer 2 that is required to support it, like, or layer 1, if you prefer, to bring a link up.
Um, there's… bonding and… Ether channel LACP.
can't… you can't get an ARP table from… from IP?
those happen under…
Exactly. Yeah, okay. So, that's…
IP neighbor.
take it.
Right, right.
But those are under Layer 3.
No? Yes. Very much so. Thank you.
So did that answer your question?
Mm-hmm.
I'd like to point out that Phil Bunch. Posted some links in the chat file.
Pointing at the… Route 2 through the, uh, Linux Foundation organization and the Debian organization, some references.
People might want to look at that before they leave.
Before we shut down, etc.
I suggested we, uh…
People are thinking for a moment. I'm gonna ask a slightly non-technical question.
Uh, anyone who has. if this is your first time, or if you just haven't been to one of our meetings in a long time.
uh… Uh, I'm sorry, James.
Uh, obviously told us he was from Rockford. Somebody up there tipped him off to come to our meeting.
If anybody else would like to say. Why did you come tonight, even if it's not your first time?
What motivated you to come this evening, if you haven't been here in a while?
We are curious.
I just need to see it here.
Gary twisted my arm.
Hmm.
He's good again. Okay. There we go.
The older I get, the more I have to make those wheels keep turning.
There we go. I'm sorry, who said that just now?
Good, okay. Thank you, James.
James, sure.
I wanted to.
Okay, uh… And I guess if there's nothing else.
Bookend?
Uh, this may be the end of our meeting this evening.
Uh, thank you all very much for coming and participating. I think we covered a lot of.
Good, interesting stuff. Uh, this was the monthly general meeting of SLUG, the St. Louis.
Linux, Unix user group. Uh, the date is Wednesday, November the 12th.
2025, uh… And, uh, main topic tonight was, uh.
Uh, the, uh, the new… Uh, IP route the commands.
And uh… basically, upgrading your network tools, upgrading your network tools, there we go.
And it was done by Lee Lambert. And our basic tutorial tonight was done by Grant Taylor, and he was, uh.
Showing us how to do troubleshooting using round robin database.
Once again