Wed @ 6:30 & 7"15 is the monthly general "Stl Linux Unix Users Group"
mtng.
https://www.meetup.com/saint-louis-unix-users-group/events/311956042/
Learn how to replace deprecated "ifconfig, route, & netstat" commands with
the modern "ip & ss" commands. Clearer output, better troubleshooting, full
IPv6 support, and future-proof networking.
*"ip: Because Real Admins Don't Use ifconfig Anymore"*
*MAIN Topic:* Evolving "ifconfig & route" to "ip route"
Speaker: Lee_Lammert
<https://www.google.com/url?q=https://www.sluug.org/bio/Lee_Lammert&sa=D&source=calendar&usd=2&usg=AOvVaw1zJdyS9F6qtGRX3OETuxUW>
.
The above MAIN is at ~7:15pm. It is immediately PRECEDED at
~7:00pm by Announcements & short Q&A which is immediately preceeded at
6:30pm by
*BASIC Tutorial:* Troubleshooting with a Round Robin DB by Grant Taylo
<https://www.google.com/url?q=https://www.sluug.org/bio/Grant_Taylor&sa=D&source=calendar&usd=2&usg=AOvVaw3U_U07yFRWkEcq5yYvuhbX>
r
- For the last week or so I've been spending a lot of time recording
statistics in Round Robin Databases (RRD files) and generating graphs of
things as I try to troubleshoot some problems with the Nagios server at
work.
-
- This morning I spent some time writing some Perl to parse the
output of `netstat --all --numeric --tcp --udp` to calculate the number
of connections in various states, both inbound and outbound.
- I have something like 27 different data points that I'm graphing.
-
- As has been the case with three other (sets of) graphs that I've
created for this, I've asked questions within a few minutes of looking
at -> consuming -> using the graphs.
-
- I first started collecting, recording, and graphing the number of
Nagios tests that were in the `Unknown` (3) state.
- That turned out to be quite enlightening.
- We were seeing periodic rapid swings from a few hundred to nearly
two thousand tests in the `Unknown` (3) state.
- The visual nature of the graph showed that the number of tests in
the `Unknown` (3) state was ebbing and flowing.
- It also showed some atypical / jittery / not-smooth spikes as
opposed to the ramp-up / ramp-down that one might expect.
- As in nearly triple in a matter of a few minutes and then taper
back down a few minutes later.
-
- It turns out that Gearman / gearmand -- which we're using behind
Nagios for ${QUESTIONABLE_REASONS} -- was running out of file
descriptors.
- Once I raised the max file handles in the systemd unit file, things
were much smoother.
- So I started collecting, recording, and graphing the number of open
file descriptors.
- That too was quite telling.
-
- Gearman / gearmand was averaging about 1200 file descriptors
ranging from a few hundred to upwards of 1800.
- While working on the file descriptor problem I noticed that
connections to Gearman / gearmand would block until there was a free
file descriptor to accept the connection.
- Remember, TCP connections use a file descriptor.
-
- Knowing that slow connections to Gearman / gearmand were a symptom, I
started collecting, recording, and graphing connection latency. That graph
didn't show much of a symptom to chase. But it did provide a LOT of data in
that I now know that the vast majority of the time, connections to Gearman
/ gearmand take < 20 ms.
-
- So I have statistics / data to back monitoring -> alerting thresholds.
- I noticed that the number of open file descriptors drops from
nominally 1200 to below 500 during times that we're having problems.
- So, this morning I wrote about 210 lines of Perl to parse the
output of `netstat --all --numeric --tcp --udp` to collect and record
connection data.
-
- Graphing of said connection data yielded questions in a matter of
minutes. Not the least of which is that the vast majority of the
connections are inbound to Gearman / gearmand, not outbound like I thought
they were.
-
- I determined inbound vs outbound by looking at the listening
sockets in the `netstat` output. I defined listening as the remote IP
being 0.0.0.0 (IPv4) or ::(IPv6) and the ports thereon to be `*`.
-
- Knowing what is listening allowed me to compare the local IP & port
connections use with the list of listeners.
-
- If the local IP & port is a listener, then the connection is an
inbound connection.
- If the local IP & port isn't a listener, then the connection is an
outbound connection.
- So now I have more data and am effectively waiting for a problem
event to analyze the data that I'm collecting during the event for a
problematic time-frame after-the-fact.
The above are both Wed 12 Nov 2025 =≠=≠=≠=≠≠≠====≠EVERY MONTH:[
http://www.sluug.org/](
https://www.google.com/url?q=http://www.sluug.org/&sa=D&source=cale…
)The URL link to Zoom or Jitsi connection instructions for this meetng is
posted earlier on the day of the meetng, at the above home page. It is the
link called "linked here".
