Quoting Gary (saclug@garymcglinn.com):
I guess I still don't understand how they got in.
Well, that's after all a hard problem, isn't it?
Here's a rhetorical question, which I'm going to ask not to be a jerk but rather as lead-in to some practical suggestions going forward: Have you ever portscanned your system from a nearby IP, in order to better understand and document your system's attack surface? Did you ever then go through each service thus proven to be exposed to public network traffic, studying each and its configuration details, to reduce the attack surface and use only very solid code to handle public traffic?
Long ago, when my mother-in-law Cheryl moved down from British Columbia to join me and my wife, and introduced to our household an MS-Windows bow (first Win98 and then later WinXP), one of the favours I did for her was nmap'ing her IP, like this:
# nmap -vv -sT -sR -O -I - oN nmap.log -n 10.0.1.3 #TCP scan
# nmap -vv -sU -P0 -sR -O -oN nmap2.log 10.0.1.3 #UDP scan
# nmap -vv -sA -sR -O -n -oN nmap3.log -n 10.0.1.3 #version detect/traceroute
Reading those logfiles with Cheryl clarified what was potentially attackable on her Winbox, and thus where to spend time locking things down or switching them off or otherwise mitigate risk.
On my own server system, I learned the hard way that the default CGI mode of Perl-based Web server logfile analysing ("Web analytics") program AWstats suffered repeated and catastrophic security holes that were fatal to system security. (See: http://www.awstats.org/awstats_security_news.php .) However, a few seconds of pondering revealed that it was not _necessary_ to have AWstats continuously updating as dynamic Web code -- that it would be more than sufficient to have it run via a local-process cron job and write out a static HTML report every minute.
Likewise, I made the same overall judgement about public-facing PHP, that it suffered too many and too severe, recurring security meltdowns for me to justify exposing mod_php to the Internet. Therefore, I re-engineered all PHP-driven features on my site to be _static_ HTML pages created by running /usr/lib/php _locally_ via Makefiles or cron jobs.
Maybe you'll never be able to tell for certain how the bad guys got in. Maybe you'll only be able to make an educated guess.
Another rhetorical question, Gary: Does your system run logcheck? Is it well-tuned, i.e., have you spent some time fine-tuning it by adding tailored lines to /etc/logcheck/ignore.d.server/local-rules, reducing logcheck's irrelevant noise in its e-mails?
I remember the day I realised that PHP's security menace was intolerable, and that I needed to do something instantly. It was an almost worst-case scenario: I was partway across the Pacific Ocean on a trans-Pacific cruise to Sydney, which meant that I got basically about an hour per day of maddeningly slow, unreliable, and high-latency satellite Internet connectivity. _But_, I had developed a well-honed /etc/logcheck/ignore.d.server/local-rules , such that normally logcheck said literally or almost nothing in its e-mailed reports. Therefore, the sudden automated attack on my system via lib_php and Apache really stood out, and I instantly knew I had a big problem -- made worse by the fact that my ssh session was only barely functional.
I knew that the automated bot had almost certainly handed remote access via Apache to remote criminals with user www-data. I'd already made sure nothing in Apache's tree was writeable by that UID, but also correctly guessed that the next stage would be to attempt a series of local privilege-escalation attacks to try to crack root-user authority. If I were lucky, this would be relatively slow and human-guided. So, I quickly took several steps, including adding a new job to /etc/crontab that killed all Apache httpd processes and restarted the httpd literally every minute. (I also took some other steps that I'll not detail here.)
When I reached Sydney and had enough bandwidth to do some more examination, I found to my amusement that this inspired hack of mine had probably given some criminal severe frustration: I found signs that a system-cracking toolkit had repeatedly gotten downloaded and started with www-data authority, but then the process was killed less than a minute later, each and every time.
Take that, asshats.
My longer-term remedy was: No public-facing PHP, no public-facing PHP security problem. Simple!
[...] and the logs don't show that.
It's possible that the (_various_) logs do show other things, though. If you really want to make an educated guess for how they got in, and how they escalated to system privilege, that should be part of where you (and preferably your analysis scripts; don't just read raw logs) look.
Beware of most security advice. (I don't exempt myself. Be skeptical of mine, too.) _Way_ too many people having "security ideas" are really just gadget freaks. Before adding "a firewall rule that simply walls out any IP that tries to log in with a user that isn't [your] account", for example, stop to ponder hard the most important question in any troubleshooting scenario: What problem am I solving? Is it the right problem?
I once upon a time co-wrote a modestly successful essay called "How to Ask Questions the Smart Way", that bears on troubleshooting, but, well, modest forfends. ;->
I will close with:
http://linuxmafia.com/~rick/lexicon.html#zebra-hunting
Zebra Hunting
Diagnostic failure mode, in which investigation goes wrong through failure to consider obvious, simple explanations first (as suggested by Occam's Razor). As celebrated medical researcher Theodore E. Woodward, Chair of Medicine at University of Maryland's School of Medicine, used to advise interns: "When you hear hoofbeats, think of horses, not zebras." When helping computerists diagnose problems, I must sometimes intervene to halt energetic, futile hunts for imaginary zebras.
Tip of the hat to Katherine Ottaway, MD, for her light verse poem Catching Zebras[1], which also bears, if mischievously, on this matter.