Quoting Gary (saclug(a)garymcglinn.com):
I guess I still don't understand how they got in.
Well, that's after all a hard problem, isn't it?
Here's a rhetorical question, which I'm going to ask not to be a jerk
but rather as lead-in to some practical suggestions going forward:
Have you ever portscanned your system from a nearby IP, in order to
better understand and document your system's attack surface? Did you
ever then go through each service thus proven to be exposed to public
network traffic, studying each and its configuration details, to reduce
the attack surface and use only very solid code to handle public
traffic?
Long ago, when my mother-in-law Cheryl moved down from British Columbia
to join me and my wife, and introduced to our household an MS-Windows
bow (first Win98 and then later WinXP), one of the favours I did for her
was nmap'ing her IP, like this:
# nmap -vv -sT -sR -O -I - oN nmap.log -n 10.0.1.3 #TCP scan
# nmap -vv -sU -P0 -sR -O -oN nmap2.log 10.0.1.3 #UDP scan
# nmap -vv -sA -sR -O -n -oN nmap3.log -n 10.0.1.3 #version detect/traceroute
Reading those logfiles with Cheryl clarified what was potentially
attackable on her Winbox, and thus where to spend time locking things
down or switching them off or otherwise mitigate risk.
On my own server system, I learned the hard way that the default CGI
mode of Perl-based Web server logfile analysing ("Web analytics")
program AWstats suffered repeated and catastrophic security holes
that were fatal to system security. (See:
http://www.awstats.org/awstats_security_news.php .) However, a few
seconds of pondering revealed that it was not _necessary_ to have
AWstats continuously updating as dynamic Web code -- that it would be
more than sufficient to have it run via a local-process cron job
and write out a static HTML report every minute.
Likewise, I made the same overall judgement about public-facing PHP,
that it suffered too many and too severe, recurring security meltdowns
for me to justify exposing mod_php to the Internet. Therefore, I
re-engineered all PHP-driven features on my site to be _static_ HTML
pages created by running /usr/lib/php _locally_ via Makefiles or cron
jobs.
Maybe you'll never be able to tell for certain how the bad guys
got in. Maybe you'll only be able to make an educated guess.
Another rhetorical question, Gary: Does your system run logcheck?
Is it well-tuned, i.e., have you spent some time fine-tuning it
by adding tailored lines to /etc/logcheck/ignore.d.server/local-rules,
reducing logcheck's irrelevant noise in its e-mails?
I remember the day I realised that PHP's security menace was
intolerable, and that I needed to do something instantly. It was an
almost worst-case scenario: I was partway across the Pacific Ocean on
a trans-Pacific cruise to Sydney, which meant that I got basically about
an hour per day of maddeningly slow, unreliable, and high-latency
satellite Internet connectivity. _But_, I had developed a well-honed
/etc/logcheck/ignore.d.server/local-rules , such that normally logcheck
said literally or almost nothing in its e-mailed reports. Therefore,
the sudden automated attack on my system via lib_php and Apache really
stood out, and I instantly knew I had a big problem -- made worse by
the fact that my ssh session was only barely functional.
I knew that the automated bot had almost certainly handed remote access
via Apache to remote criminals with user www-data. I'd already made
sure nothing in Apache's tree was writeable by that UID, but also
correctly guessed that the next stage would be to attempt a series of
local privilege-escalation attacks to try to crack root-user authority.
If I were lucky, this would be relatively slow and human-guided. So, I
quickly took several steps, including adding a new job to /etc/crontab
that killed all Apache httpd processes and restarted the httpd literally
every minute. (I also took some other steps that I'll not detail here.)
When I reached Sydney and had enough bandwidth to do some more
examination, I found to my amusement that this inspired hack of mine had
probably given some criminal severe frustration: I found signs that
a system-cracking toolkit had repeatedly gotten downloaded and started
with www-data authority, but then the process was killed less than a
minute later, each and every time.
Take that, asshats.
My longer-term remedy was: No public-facing PHP, no public-facing PHP
security problem. Simple!
[...] and the logs don't show that.
It's possible that the (_various_) logs do show other things, though.
If you really want to make an educated guess for how they got in, and
how they escalated to system privilege, that should be part of where you
(and preferably your analysis scripts; don't just read raw logs) look.
Beware of most security advice. (I don't exempt myself. Be skeptical
of mine, too.) _Way_ too many people having "security ideas" are really
just gadget freaks. Before adding "a firewall rule that simply walls
out any IP that tries to log in with a user that isn't [your] account",
for example, stop to ponder hard the most important question in any
troubleshooting scenario: What problem am I solving? Is it the right
problem?
I once upon a time co-wrote a modestly successful essay called "How to
Ask Questions the Smart Way", that bears on troubleshooting, but, well,
modest forfends. ;->
I will close with:
http://linuxmafia.com/~rick/lexicon.html#zebra-hunting
Zebra Hunting
Diagnostic failure mode, in which investigation goes wrong through
failure to consider obvious, simple explanations first (as suggested by
Occam's Razor). As celebrated medical researcher Theodore E. Woodward,
Chair of Medicine at University of Maryland's School of Medicine, used
to advise interns: "When you hear hoofbeats, think of horses, not
zebras." When helping computerists diagnose problems, I must sometimes
intervene to halt energetic, futile hunts for imaginary zebras.
Tip of the hat to Katherine Ottaway, MD, for her light verse poem
Catching Zebras[1], which also bears, if mischievously, on this
matter.
[1]
https://everything2.com/title/Catching+Zebras