[DISCUSS] St. Louis Linux Users Group - 19 Oct 2006

Craig Buchek booch at slashmail.org
Fri Oct 20 15:32:18 CDT 2006

> Craig Buchek will do a talk on "Troubleshooting Linux Problems"

I demoed quite a few tools in the talk. Unfortunately for those unable to
attend, there's no written form of the presentation. (I've put pertinent
parts of my shell history at the bottom of this post, but that'll be less
useful without the verbal context.) I figured I'd briefly mention on the
list some of the things covered though.

The sar tool provides good baselines of metrics over a period of time.
It's available in Debian in the sysstat package. That package also
includes iostat and mpstat. Sar requires a crontab entry (enabled when you
install the Debian package) to gather the metrics periodically.

There's lots of good stuff in /proc. Most of the tools like ps and top
gather their info from what the kernel provides in /proc, and format it
nicely. There's a /proc directory for each PID, and the non-numbered
files/directories are for system-wide info. The files within /proc/sys
allow you to view/change various system parameters. The procinfo package
displays a lot of the /proc info in an easier to read format. Other OSes
have /proc filesystems (I think BSD and Sun had it first) but each kernel
has a layout, files, and info.

The strace command lists all the system calls that a process makes. It can
attach to a running process, and can also attach to subprocesses. There's
a similar program called ltrace that does the same thing, but lists all
library calls. One caveat I forgot to mention -- I'm not sure it applies
anymore, but there was a bug that caused strace to make threaded apps
using pthread hang. Also, be sure to turn off your traces as soon as
you're done with them, as they can easily fill up your file system

Some of the typical things to look for -- CPU utilization, memory
utilization, disk space used, file handles in use, network I/O used, and
disk I/O used. Many performance problems will end up falling into one of
those categories.

We covered quite a bit more than this, but not a whole lot of detail on
any given tool or practice. If anyone has any follow-up questions, feel
free to ask.

Special thanks to Mike Knight and Stanford Baldwin for their input
regarding similar tools on AIX, HP-UX, etc.


apt-get install htop
mkdir /tmp/analysis
cd /tmp/analysis
hostname > hostname
uname -a > uname
cat /etc/redhat-release > redhat-release
cat /etc/debian_version > debian_version
ps auxw > ps-auxw
df -h > df
mount > mount
free > free
free -m
export > environment
netstat -nap | grep LISTEN | grep tcp > ports-open
cat /proc/cpuinfo > cpuinfo
ulimit -a > ulimit
cat /proc/sys/vm/laptop_mode
echo 1 > /proc/sys/vm/laptop_mode
cat /proc/sys/vm/laptop_mode
echo 0 > /proc/sys/vm/laptop_mode
apt-get install sysstat
iostat -x
/usr/lib/sysstat/sa1 2 3
sar -A
dmesg | less
less /var/log/dmesg

ps auxw | grep apache
ls /proc/11272/fd | wc -w
cat /proc/11272/maps
cat /proc/11272/stat
cat /proc/11272/statm
cat /proc/11272/status
strace -f -p 11272 -o apache.strace
grep -E '(read|write|open)' apache.strace | less
apt-get install ltrace libelfg0
ltrace -f -p 11272 -o apache.ltrace
ldd /usr/sbin/apache2
du -sk /var/* | sort -nr

St. Louis Unix Users Group - http://www.sluug.org/
To unsubscribe from the SLUUG discussion mailing list, send a message to
discuss-request at sluug.org with the word 'unsubscribe' as the body

More information about the Discuss mailing list