As moon-howling, tree-hugging, packet-sniffing pagans, we like to celebrate the Summer Solstice by doing a security audit of the systems on our network. Spending the longest day of the year auditing systems will help us avoid spending the longest night of the year restoring a hacked one.
In our audit, we will be disabling unused accounts, shutting down unnecessary services, and generally cleaning up and hardening. We will also be checking for rootkits and other signs that a system’s been hacked already. Preparing a toolkit to help us determine if a system has been compromised is the focus of this article.
We want to use the chkrootkit tool to check our systems. It can detect trojaned binaries, LKM trojans, hidden directories, sniffer logs, and rootkit configuration files. Be sure to download the latest version, as the rootkit signatures can be updated fairly frequently as new ones come out.
The problem with running chkrootkit on a live system is its use of external binaries — the same external binaries which are usually trojaned by rootkits, such as ps, ls, and netstat. So if the system’s been owned already, we can’t completely trust what chkrootkit says.
One solution would be to copy the binaries from a pristine system to media which we will then mount temporarily on the system to be examined. But binaries on modern systems use shared libraries.
To run them, we would be relying on the integrity of the shared libraries on the suspect system. We don’t want to trust anything on the system. So we are going to build static binaries from source on a trusted system, and a statically linked chkrootkit tool set too. These can then be stored in a safe place and mounted on the systems to be examined.
Building the Binaries
The chkrootkit tools use awk, cut, echo, egrep, find, head, id, ls, netstat, ps, strings, sed, and uname. For a starting point, since we’re running Red Hat, we’ll do an rpm query to find out what package each of these binaries belongs to:
[root@hazel rescue]# rpm -qf /bin/ps procps-2.0.7-12 |
A google search leads us to freshmeat.net, and ultimately we can download the source for procps utilities from http://procps.sourceforge.net. We download, untar the tarball, and read the README, which reveals that if we want statically linked binaries, we should run the following:
[usr-4@hazel procps-3.1.9]$ make SHARED=0 |
The output of gcc spews forth, with no serious errors, and all is well. We have a number of binaries in our build directory. Let’s make sure they’re really static:
[usr-4@hazel procps-3.1.9]$ ls -F AUTHORS Makefile pmap.1 skill.1 t* top.o vmstat.o BUGS minimal.c pmap.c skill.c tload* uptime* w* COPYING NEWS pmap.o skill.o tload.1 uptime.1 w.1 COPYING.LIB pgrep* proc/ snice* tload.c uptime.c watch* free* pgrep.1 procps.lsm snice.1 tload.o uptime.o watch.1 free.1 pgrep.c procps.spec sysctl* TODO v* watch.c free.c pgrep.o ps/ sysctl.8 top* vmstat* watch.o free.o pkill* README sysctl.c top.1 vmstat.8 w.c kill* pkill.1 README.top sysctl.conf.5 top.c vmstat.c w.o kill.1 pmap* skill* sysctl.o top.h vmstat.h [usr-4@hazel procps-3.1.9]$ ldd w libc.so.6 => /lib/libc.so.6 (0x4002b000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000) [usr-4@hazel procps-3.1.9]$ ls -l w -rwxr-xr-x 1 usr-4 usr-4 174555 Jun 17 13:27 w |
It looks like the w binary is dynamically linked! What’s up with that? Compare it to the w binary from procps built without the SHARED=0 option:
[usr-4@hazel procps-3.1.9]$ ldd w libproc.so.3.1.9 => not found libc.so.6 => /lib/libc.so.6 (0x4002c000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000) [usr-4@hazel procps-3.1.9]$ ls -l w -rwxr-xr-x 1 usr-4 usr-4 76138 Jun 17 13:31 w |
It appears that the SHARED=0 option only keeps procps from dynamically linking its own library, the libproc.so. So the binary is statically linked against that library, but not the system libraries.
A further perusal of the README tells us that we can pass gcc options to the compiler. This may be what we need:
[usr-4@hazel procps-3.1.9]$ make clean rm -f uptime tload free w top vmstat watch skill snice kill sysctl pmap pgrep pk ill proc/.depend proc/lib*.so* proc/lib*.a proc/alloc.o proc/devname.o proc/esca pe.o proc/ksym.o proc/prof.o proc/pwcache.o proc/readproc.o proc/sig.o proc/stat us.o proc/sysinfo.o proc/version.o proc/whattime.o ps/ps ps/debug DEADJOE *~ *.o core gmon.out proc/DEADJOE proc/*~ proc/*.o proc/core proc/gmon.out ps/DEADJOE ps/*~ ps/*.o ps/core ps/gmon.out [usr-4@hazel procps-3.1.9]$ make SHARED=0 CC='gcc -static' --output from make snipped-- [usr-4@hazel procps-3.1.9]$ ldd w not a dynamic executable [usr-4@hazel procps-3.1.9]$ ls -l w -rwxr-xr-x 1 usr-4 usr-4 627172 Jun 17 13:35 w |
There’s the joy. This package has given us some nice tools in addition to the required ps. We will also take w, free, vmstat, top, and uptime. We’ve made a directory called audit, created a bin directory beneath it, and are copying all of our statically linked binaries there. Only 9 more packages to go! Don’t worry, we’ll give you the executive summary from this point forward.
awk
download: http://ftp.gnu.org/gnu/gawk/gawk-3.1.1.tar.gz
build:
[usr-4@hazel gawk-3.1.1]$ ./configure [usr-4@hazel gawk-3.1.1]$ make CC="gcc -static" |
postinstall: in your audit bin, make a symbolic link to gawk from awk, because chkrootkit will look for awk.
cut,echo,head,id,ls,uname
We are in luck! The three packages textutils, shellutils, and fileutils have been combined into one coreutils package. Six of our needed utilities in one package! download: http://ftp.gnu.org/pub/gnu/coreutils
build:
[usr-4@hazel coreutils-5.0]$ ./configure [usr-4@hazel coreutils-5.0]$ make CC="gcc -static" |
Copy the desired binaries from the src directory to your audit bin. I’m thinking md5sum might also be nice to have.
grep
download: http://www.gnu.org/directory/GNU/grep.html
build:
[usr-4@hazel grep-2.5]$ ./configure [usr-4@hazel grep-2.5]$ make CC="gcc -static" |
Copy the binary from src directory, make a symbolic link to it from “egrep”.
netstat
download: http://www.tazenda.demon.co.uk/phil/net-tools/net-tools-1.60.tar.bz2
build:
[usr-4@hazel net-tools-1.60]$ make config (interactive - defaults worked for us) [usr-4@hazel net-tools-1.60]$ make CC="gcc -static" |
This one has some other goodies that might be handy in an audit too, like route, arp, and even ifconfig.
strings
download: http://ftp.gnu.org/gnu/binutils/binutils-2.13.1.tar.gz
build:
[usr-4@hazel binutils]$ ./configure [usr-4@hazel binutils]$ make -e LDFLAGS=-all-static |
Copy the utilities from the binutils subdirectory. For some reason, running gcc with the static flag did not result in statically linked binaries. So I set the LDFLAGS variable instead and got the joy.
sed
download: http://ftp.gnu.org/pub/gnu/sed/sed-4.0.7.tar.gz
build:
[usr-4@hazel sed-4.0.7]$ ./configure [usr-4@hazel sed-4.0.7]$ make CC="gcc -static" |
find
download:
ftp://ftp.linuxfromscratch.org/lfs-packages/4.0/findutils-4.1.tar.bz2
ftp://ftp.linuxfromscratch.org/lfs-packages/4.0/findutils-4.1.patch.bz2
Note: though the official location for findutils is gnu.org, findutils will not build on our system (Red Hat 7.3, all current updates). Since we need a patch which is not available on the gnu server, we’ll download both from the LFS server. After expanding the findutils tarball, we unzipped the patch and copied it into the build directory.
build:
[usr-4@hazel findutils-4.1]$ patch -Np1 -i findutils-4.1.patch [usr-4@hazel findutils-4.1]$ ./configure [usr-4@hazel findutils-4.1]$ make CC="gcc -static" |
Building chkrootkit
Now let’s build chkrootkit statically:
[usr-4@hazel chkrootkit-0.40]$ make sense CC="gcc -static" |
After building, we copied the entire build dir to our audit directory.
[usr-4@hazel sec]$ cp -a chkrootkit-0.40/ audit/ |
Using the Toolkit
We also built a statically compiled bash shell, downloading it from gnu.org and building it in the same way as the other gnu packages. Now we’ve got a good little toolkit for auditing systems. Because the executables are statically linked, they are MUCH bigger than dynamically linked programs. These tools are not going to fit on a floppy. Either burn them onto a cdrom (the best) and mount them from this read-only media, or mount them read-only from a trusted NFS server.
If you must, you can copy them to the machine being audited, but once you’re working on the local file system, you are through the looking glass. I’ve exported my audit tree using NFS from a trusted system. Now, I will use it to check for rootkits on another system. I’ve exported it read-only, so I don’t need to make sure it’s mounted read-only. To use our statically compiled binaries, we will supply the -p (path) option to chkrootkit.
[root@sketchy root]# mount hazel:/u01/audit /mnt/tmp [root@sketchy root]# cd /mnt/tmp [root@sketchy chkrootkit-0.40]# ./chkrootkit -p ../bin [root@sketchy chkrootkit-0.40]# ./chkrootkit -p ../bin ROOTDIR is `/' Checking `amd'... not found Checking `basename'... not infected Checking `biff'... not found Checking `chfn'... not infected Checking `chsh'... not infected Checking `cron'... not infected Checking `date'... not infected Checking `du'... not infected Checking `dirname'... not infected Checking `echo'... not infected Checking `egrep'... not infected Checking `env'... not infected Checking `find'... not infected Checking `fingerd'... not infected Checking `gpm'... not infected Checking `grep'... not infected Checking `hdparm'... not infected Checking `su'... not infected Checking `ifconfig'... not infected Checking `inetd'... not tested Checking `inetdconf'... not found Checking `identd'... not infected Checking `init'... not infected Checking `killall'... not infected Checking `ldsopreload'... not infected Checking `login'... not infected Checking `ls'... not infected Checking `lsof'... not infected Checking `mail'... not infected Checking `mingetty'... not infected Checking `netstat'... not infected Checking `named'... not found Checking `passwd'... not infected Checking `pidof'... not infected Checking `pop2'... not found Checking `pop3'... not found Checking `ps'... not infected Checking `pstree'... not infected Checking `rpcinfo'... not infected Checking `rlogind'... not infected Checking `rshd'... not infected Checking `slogin'... not infected Checking `sendmail'... not infected Checking `sshd'... not infected Checking `syslogd'... not infected Checking `tar'... not infected Checking `tcpd'... not infected Checking `tcpdump'... not infected Checking `top'... not infected Checking `telnetd'... not infected Checking `timed'... not found Checking `traceroute'... not infected Checking `w'... not infected Checking `write'... not infected Checking `aliens'... no suspect files Searching for sniffer's logs, it may take a while... nothing found Searching for HiDrootkit's default dir... nothing found Searching for t0rn's default files and dirs... nothing found Searching for t0rn's v8 defaults... nothing found Searching for Lion Worm default files and dirs... nothing found Searching for RSHA's default files and dir... nothing found Searching for RH-Sharpe's default files... nothing found Searching for Ambient's rootkit (ark) default files and dirs... nothing found Searching for suspicious files and dirs, it may take a while... /usr/lib/perl5/site_perl/5.6.1/i386-linux/auto/DBD/Oracle/.packlist Searching for LPD Worm files and dirs... nothing found Searching for Ramen Worm files and dirs... nothing found Searching for Maniac files and dirs... nothing found Searching for RK17 files and dirs... nothing found Searching for Ducoci rootkit... nothing found Searching for Adore Worm... nothing found Searching for ShitC Worm... nothing found Searching for Omega Worm... nothing found Searching for Sadmind/IIS Worm... nothing found Searching for MonKit... nothing found Searching for Showtee... nothing found Searching for OpticKit... nothing found Searching for T.R.K... nothing found Searching for Mithra... nothing found Searching for OBSD rk v1... nothing found Searching for LOC rootkit ... nothing found Searching for Romanian rootkit ... nothing found Searching for HKRK rootkit ... nothing found Searching for anomalies in shell history files... nothing found Checking `asp'... not infected Checking `bindshell'... not infected Checking `lkm'... nothing detected Checking `rexedcs'... not found Checking `sniffer'... eth0 is not promisc Checking `wted'... nothing deleted Checking `scalper'... not infected Checking `slapper'... not infected Checking `z2'... nothing deleted |
Under suspicious files and directories, we’ve got one entry. All sorts of hidden files and directories can show up here. Let’s check this one.
[root@sketchy chkrootkit-0.40]# cd /usr/lib/perl5/site_perl/5.6.1/i386-linux/auto/DBD/Oracle [root@sketchy Oracle]#/mnt/tmp/bin/head .packlist DBD::Oracle.3pm DBD::Oraperl.3pm Oracle.bs Oracle.h Oracle.pm Oracle.so Oraperl.pm dbdimp.h mk.pm ocitrace.h |
This system looks good. We can continue our system audit with a high degree of confidence that there are no rootkits installed because we used our clean tools. If further investigation is desired, we can use our clean tools to poke around.
Who’s logged on? Use your statically linked w command.
What connections are open on this machine? Use your statically linked netstat.
You could also put a statically linked version of tcpdump and other networking utilities in your audit toolkit.
Now we’re going to finish our audit so we can go hug some trees.
