License
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.
Written by Christian Stigen Larsen
I've had servers been compromised a few times. It's practically bound to happen if you're running servers with any open ports these days.
Here is my list of things to remember to check when investigating break-ins. It's just a small list of stuff I need to remember.
This is the single most important thing you do. Not only will you have recorded what you found, you will also be able to present proof in a subsequent filing of the incident.
The script command records all output to a file:
script -f ~/logfile.txt
You should also include a timestamp. If something happens while you investigate, you can cross-check later on with other logfiles.
Also understand that if your computers don't have synchronized clocks, you won't be able to effectively compare logged events!
$ export PS1='\t \u@\h:\W$ ' 10:17:59 luke@darkside:~$
As Cliff Stoll wrote in his book Cuckoo's Egg:
Copy logfiles off site. Ideally, you are already forwarding syslog messages across the network to another, more secure computer. Many attackers either delete or modify log files.
Read through all your logs. Anything unusual, especially garbage, have led me to discover attacks.
I always check /var/log/messages, /var/log/secure as well as web server logs and whatever the server is running.
Check last and .bash_history in user directories. Missing history files or files symlinked to /dev/null is a pretty sure sign you've had an uninvited guest.
Don't rely too much on history files. It's pretty easy to avoid dumping command history, but many attackers (especially clueless script kiddies) often forget covering their tracks. A common technique is to run an editor like vi and type :shell. This spawns a non-login shell that doesn't write to command history.
See if any important files have been altered, like ps, ls, grep, netstat, /bin/sh and so on:
# files modified today find /bin -mtime 0 find /sbin -mtime 0 find /usr -mtime 0 # files created within the last 24 hours find / -ctime -1
Using ps, pstree, top, lsof will quickly give an indication if anything suspicious is running.
Some rootkits effectively hide their processes. This is typical for kernel module injections.
ps auxwww # shows wide listing of command-lines pstree # process inheritance tree top # realtime view lsof # list open files
If you find a suspicious process running in memory, it might have been deleted from disk. You can still copy the executable image back to disk with the symbolic link in /proc/???/exe.
Quick overview of /proc:
cd /proc/1234 # check out the pid 1234
cat cmdline # shows commandline used to run the process
file exe # shows file info, especially if the symboltable still
# exists. if the file has been deleted, then exe links
# to the image in-memory, so you can cp exe /tmp/somefile
strings -a exe # see all ascii strings in the file, *extremely* useful
# for googling
cat maps # currently mapped memory regions
You can also use the listps program to explicitly investigate each process in the /proc filesystem. This can lead to discovery of programs hidden by normal ps output.
Use strings -a, hexdump -C, gdb, objdump (or otool on darwin) to examine suspicious executables. In many cases you'll find strings like r00tkit by zum azzHat — these are extremely helpful to identify and find exploit code. Google for anything you find. Don't forget to search google groups as well.
Examine executables more closely with debuggers and programming tools.
Use file somefile to see if it's a stripped executable. Hopefully it's not, and you will be able to view symbols with nm somefile. Also run ldd somefile to see which dynamic libraries are used.
Use gdb to debug the executable, but be careful when executing exploit code. Advanced code will check if it's being debugged, so you might have to circumvent that. (In worst case, get yourself a ring-0 debugger.) I've seen programs call ptrace and exit if they're being debugged.
If you have no qualms running the executable you can use strace somefile (or ktrace on darwin) to see all system calls made while running. That should give you a pretty good idea of the nature of the exploit code and lead to identify security holes in other programs.
Below is an investigation of stkill, a rootkit one of my computers was infected with.
$ file stkill
stkill: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
dynamically linked (uses shared libs), not stripped
$ ldd stkill
libc.so.6 => /lib/tls/libc.so.6 (0x42000000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
$ nm stkill
U atoi@@GLIBC_2.0
U bcopy@@GLIBC_2.0
08049a78 A __bss_start
w bzero@@GLIBC_2.0
0804997c d completed.3
w connect@@GLIBC_2.0
08049988 d __CTOR_END__
08049984 d __CTOR_LIST__
08049974 D __data_start
08049974 W data_start
w __deregister_frame_info@@GLIBC_2.0
08048770 t __do_global_ctors_aux
08048530 t __do_global_dtors_aux
... etc.
$ strace ./stkill # might be dangerous
execve("./stkill", ["./stkill"], [/* 22 vars */]) = 0
uname({sys="Linux", node="eris", ...}) = 0
brk(0) = 0x8049a90
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x40016000
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or
directory)
... etc.
Check for open backdoors in your system, either by using netstat, lsof or a full nmap-scan from another computer.
Running ifconfig -a and looking for the PROMISC flag tells you if any ethernet cards are running in promiscuous mode. This means it is picking up all network packets, usually the signature of a sniffer being installed.
You can run ifconfig eth0 -promisc to turn off promiscuous mode.
Example:
$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:DA:4D:84:C5
inet addr:81.191.35.198 Bcast:81.191.35.199 Mask:255.255.255.248
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:63608975 errors:0 dropped:0 overruns:1 frame:0
TX packets:87694104 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:3711876829 (3539.9 Mb) TX bytes:2017229474 (1923.7 Mb)
Interrupt:10 Base address:0xb800
Put your analysis on the web so other people can benefit from your work. All the times I've been hit with computer breaches, I've found helpful advice on the web.
You can just as well prepare for the next break-in right away, as it will happen again.
Set up syslog forwarding, routinely copy log files to other computers. Uninstall all unneeded inet servers (telnet, ftp, rpc, etc.), consider making some binaries or filesystems readonly (but don't believe this is enough to prevent modifications) and so on.
Since I wrote this, network scanning has exploded over the last few years. My typical server runs sshd on port 22 and httpd on port 80. That's the minimum setup I can live with.
Port 22 is scanned daily — hourly, actually. So I just changed sshd to listen to another port. This will rule out most large-scale network sweeps, since they don't have the time to scan each and every port on thousands of computers.
Regarding the web, running dynamic stuff like php is basically asking for someone to come crashing in. This is even more true if you've installed a lot of applications like forum engines and forget to keep them updated.
As for scripted attacks, many try executing things like wget to download exploit binaries. Simply renaming wget is usually enough to plug scripted holes (but again, don't count on it!). Do the same with curl if you've got it.
I know the last few tips sound pretty naive, but trust me, it works against botnet agents trying to find a vulnerable host on the net. Though if you've been singled out as a target, these measures of course aren't enough.
In closing, remember that security is not a matter of adding stuff to your system. Security is an attitude, an approach and thought-set.
Whenever you code something, you should know in the back of your head that you can't ever trust input. Stay away from setuid programs. Read up on security on the net, check out djb's security course if you're into secure programming, get a book on the subject and sharpen your skills.
Not caring about security because you're too lazy is simply not acceptable, not even on proprietary systems.
You'll find a lot of useful security related stuff over at insecure.org.