Interesting. Looks like keys are a risk. But, I still don't understand, and I don't think the article was clear about, how a brute force SSH password attack is possible over my very limited network. AT&T's cheapest plan isn't a very big pipe and I use good passwords. The attacker would have to have a great deal of luck.
Since I use ssh for personal access, I'm considering looking into a firewall rule that simply walls out any IP that tries to log in with a user that isn't my account. Another thing that makes me wonder about this vector is the successful login wasn't from an account on my system.
It seems to me that the keys/certifcates are being attacked directly. I've read other articles indicating that there are attackers in the wild doing this.
So, it seems to me that eliminating key/certificate logins from outward facing systems may buy a lot of extra security. There were keys in my .ssh directory, but whether I installed them or the attacker did, is not clear. I wouldn't need them, but it is possible I could have created them at some point over the last 15 years I've been using this system :)
-Gary
On Fri, Aug 05, 2022 at 03:54:33AM -0600, Linus Sphinx wrote:
Join the club. https://www.bleepingcomputer.com/news/security/new-linux-malware-brute-force...
On Tue, Aug 2, 2022 at 12:43 PM Gary saclug@garymcglinn.com wrote:
Hi Brian,
Open letter.
I remember years ago you did a presentation on snort.
Do you still like it?
-Gary
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
In thinking a little more, even if they are attacking the certificate directly, it is using the user they are logging in as .ssh certificate. If that user doesn't exist, it seems that the attack should never work.
I guess I still don't understand how they got in. But, my system was rebooted. There must be more going on. They couldn't log in as root directly since I, like most of the world, have that disabled and the logs don't show that.
-Gary
On Fri, Aug 05, 2022 at 09:42:20AM -0700, Gary wrote:
Interesting. Looks like keys are a risk. But, I still don't understand, and I don't think the article was clear about, how a brute force SSH password attack is possible over my very limited network. AT&T's cheapest plan isn't a very big pipe and I use good passwords. The attacker would have to have a great deal of luck.
Since I use ssh for personal access, I'm considering looking into a firewall rule that simply walls out any IP that tries to log in with a user that isn't my account. Another thing that makes me wonder about this vector is the successful login wasn't from an account on my system.
It seems to me that the keys/certifcates are being attacked directly. I've read other articles indicating that there are attackers in the wild doing this.
So, it seems to me that eliminating key/certificate logins from outward facing systems may buy a lot of extra security. There were keys in my .ssh directory, but whether I installed them or the attacker did, is not clear. I wouldn't need them, but it is possible I could have created them at some point over the last 15 years I've been using this system :)
-Gary
On Fri, Aug 05, 2022 at 03:54:33AM -0600, Linus Sphinx wrote:
Join the club. https://www.bleepingcomputer.com/news/security/new-linux-malware-brute-force...
On Tue, Aug 2, 2022 at 12:43 PM Gary saclug@garymcglinn.com wrote:
Hi Brian,
Open letter.
I remember years ago you did a presentation on snort.
Do you still like it?
-Gary
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
Gary,
You were running Fedora 13?
Brian
On Fri, Aug 05, 2022 at 09:55:57AM -0700, Gary wrote:
In thinking a little more, even if they are attacking the certificate directly, it is using the user they are logging in as .ssh certificate. If that user doesn't exist, it seems that the attack should never work.
I guess I still don't understand how they got in. But, my system was rebooted. There must be more going on. They couldn't log in as root directly since I, like most of the world, have that disabled and the logs don't show that.
-Gary
On Fri, Aug 05, 2022 at 09:42:20AM -0700, Gary wrote:
Interesting. Looks like keys are a risk. But, I still don't understand, and I don't think the article was clear about, how a brute force SSH password attack is possible over my very limited network. AT&T's cheapest plan isn't a very big pipe and I use good passwords. The attacker would have to have a great deal of luck.
Since I use ssh for personal access, I'm considering looking into a firewall rule that simply walls out any IP that tries to log in with a user that isn't my account. Another thing that makes me wonder about this vector is the successful login wasn't from an account on my system.
It seems to me that the keys/certifcates are being attacked directly. I've read other articles indicating that there are attackers in the wild doing this.
So, it seems to me that eliminating key/certificate logins from outward facing systems may buy a lot of extra security. There were keys in my .ssh directory, but whether I installed them or the attacker did, is not clear. I wouldn't need them, but it is possible I could have created them at some point over the last 15 years I've been using this system :)
-Gary
On Fri, Aug 05, 2022 at 03:54:33AM -0600, Linus Sphinx wrote:
Join the club. https://www.bleepingcomputer.com/news/security/new-linux-malware-brute-force...
On Tue, Aug 2, 2022 at 12:43 PM Gary saclug@garymcglinn.com wrote:
Hi Brian,
Open letter.
I remember years ago you did a presentation on snort.
Do you still like it?
-Gary
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
Quoting Brian E. Lavender (brian@brie.com):
Gary,
You were running Fedora 13?
If so, _that_ is likely a big problem. Fedora 13's initial release was May 25, 2010, and it was EOLed on June 24, 2011.
Because Fedora. If you don't want to keep moving to newer versions, it's about the worst possible distro. (But it's possible Gary meant that he did _original_ installation 15 years ago, but has been following the recommended upgrade treadmill^W path.
Linus Sphinx wrote:
https://www.bleepingcomputer.com/news/security/new-linux-malware-brute-force...
You know, I have a _lot_ of things to be grateful for, and somewhere on the list is the glad tidings that I don't need to rely on Bleepingcomputer.com for IT information.
Over the past 1.5 months since its discovery, the new botnet used over 3,500 unique IPs worldwide to scan and attempt brute-forcing Linux SSH servers. [...] The SSH brute-forcing relies on a list of credentials downloaded from the [command and control server]. [...]
*snore*
So, doorknob-twisting for "joe accounts", like user=service password=manager and like that.
Guestimate the math, and measure the lengthly setup and teardown times for remote connections to an sshd, and you'll find that dictionary-attacking an sshd with any reasonable rules set about password quality and length is going to take an appreciable fraction of the time to the heat death of the universe, to succeed.
I mentioned upthread that a lot of IT device comes from gadget freaks. The _other_ common problem is that most security _articles_ are copied-pasted press releases from security/antimalware firms. So, they're big on shockhorror, and small on conveying understanding.
I've only quick-glanced at this article about enforcing password policy via PAM, so won't swear to it being a good one: https://www.techrepublic.com/article/controlling-passwords-with-pam/ Of course, if you're the -only- user, you ought to stick to decent passwords without PAM forcing you to. (Also, a user who can su to root has the power to overrule PAM. But if you do that, you have only yourself to blame for consequences.)
A lot of issues and suggestions have been made and raised. My brief response is that yes, according to nmap and my intention I had port 23, for ssh (I moved it) and port 5900 open and the rpc port, I think. I'm going by memory.
Theoretically the only pinhole in the ISP router firewall was port 23. To use port 5900, you had to use an ssh tunnel.
My web server is on another machine. Since I only use it for development, I don't leave it up.
My actual web pages are hosted externally on a vendor's box.
My experience with the upgrade treadmill is that it is a waste of time. By its own admission (if a concept can have that) the new versions will have issues and you need to upgrade. If the issues with an older version don't affect you, then it is perfectly fine to use it. Why upgrade and risk the fact that the new issues will affect your use case. The developers for Fedora 13 were no smarter or dumber than the developers who are writing Fedora 36. Or pick your distro of choice.
The issue with an older release is that nifty new things come out and you can't really use them. But, if you run a VM with a recent version of the distro, you can use them just fine.
The other issues is that if you are getting vendor support, they can only reasonably commit to supporting a limited number of versions.
This is AMD hardware from '06.
-Gary
On Fri, Aug 05, 2022 at 06:19:54PM -0700, Rick Moen wrote:
Quoting Brian E. Lavender (brian@brie.com):
Gary,
You were running Fedora 13?
If so, _that_ is likely a big problem. Fedora 13's initial release was May 25, 2010, and it was EOLed on June 24, 2011.
Because Fedora. If you don't want to keep moving to newer versions, it's about the worst possible distro. (But it's possible Gary meant that he did _original_ installation 15 years ago, but has been following the recommended upgrade treadmill^W path.
Linus Sphinx wrote:
https://www.bleepingcomputer.com/news/security/new-linux-malware-brute-force...
You know, I have a _lot_ of things to be grateful for, and somewhere on the list is the glad tidings that I don't need to rely on Bleepingcomputer.com for IT information.
Over the past 1.5 months since its discovery, the new botnet used over 3,500 unique IPs worldwide to scan and attempt brute-forcing Linux SSH servers. [...] The SSH brute-forcing relies on a list of credentials downloaded from the [command and control server]. [...]
*snore*
So, doorknob-twisting for "joe accounts", like user=service password=manager and like that.
Guestimate the math, and measure the lengthly setup and teardown times for remote connections to an sshd, and you'll find that dictionary-attacking an sshd with any reasonable rules set about password quality and length is going to take an appreciable fraction of the time to the heat death of the universe, to succeed.
I mentioned upthread that a lot of IT device comes from gadget freaks. The _other_ common problem is that most security _articles_ are copied-pasted press releases from security/antimalware firms. So, they're big on shockhorror, and small on conveying understanding.
I've only quick-glanced at this article about enforcing password policy via PAM, so won't swear to it being a good one: https://www.techrepublic.com/article/controlling-passwords-with-pam/ Of course, if you're the -only- user, you ought to stick to decent passwords without PAM forcing you to. (Also, a user who can su to root has the power to overrule PAM. But if you do that, you have only yourself to blame for consequences.)
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
It was only 6pm when I wrote this, but I must have been tired anyway:
I mentioned upthread that a lot of IT device comes from gadget freaks.
^^^^^^ "advice"
The _other_ common problem is that most security _articles_ are copied-pasted press releases from security/antimalware firms. So, they're big on shockhorror, and small on conveying understanding.
Sorry about the mistype.
Yes, the hardware is old and nothing more recent than that would install and run. I use VM's that had more recent versions, but I was stuck at Fedora 13 for the host hardware. I have a new system on order from Dell. When I revive the hacked system, it will still probably be running Fedora 13, but I'll use it for something else.
-Gary
On Fri, Aug 05, 2022 at 12:53:42PM -0700, Brian E. Lavender wrote:
Gary,
You were running Fedora 13?
Brian
On Fri, Aug 05, 2022 at 09:55:57AM -0700, Gary wrote:
In thinking a little more, even if they are attacking the certificate directly, it is using the user they are logging in as .ssh certificate. If that user doesn't exist, it seems that the attack should never work.
I guess I still don't understand how they got in. But, my system was rebooted. There must be more going on. They couldn't log in as root directly since I, like most of the world, have that disabled and the logs don't show that.
-Gary
On Fri, Aug 05, 2022 at 09:42:20AM -0700, Gary wrote:
Interesting. Looks like keys are a risk. But, I still don't understand, and I don't think the article was clear about, how a brute force SSH password attack is possible over my very limited network. AT&T's cheapest plan isn't a very big pipe and I use good passwords. The attacker would have to have a great deal of luck.
Since I use ssh for personal access, I'm considering looking into a firewall rule that simply walls out any IP that tries to log in with a user that isn't my account. Another thing that makes me wonder about this vector is the successful login wasn't from an account on my system.
It seems to me that the keys/certifcates are being attacked directly. I've read other articles indicating that there are attackers in the wild doing this.
So, it seems to me that eliminating key/certificate logins from outward facing systems may buy a lot of extra security. There were keys in my .ssh directory, but whether I installed them or the attacker did, is not clear. I wouldn't need them, but it is possible I could have created them at some point over the last 15 years I've been using this system :)
-Gary
On Fri, Aug 05, 2022 at 03:54:33AM -0600, Linus Sphinx wrote:
Join the club. https://www.bleepingcomputer.com/news/security/new-linux-malware-brute-force...
On Tue, Aug 2, 2022 at 12:43 PM Gary saclug@garymcglinn.com wrote:
Hi Brian,
Open letter.
I remember years ago you did a presentation on snort.
Do you still like it?
-Gary
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
-- Brian Lavender http://www.brie.com/brian/
"There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies."
Professor C. A. R. Hoare The 1980 Turing award lecture _______________________________________________ Lug-nuts mailing list -- lug-nuts@bigbrie.com To unsubscribe send an email to lug-nuts-leave@bigbrie.com
Quoting Gary (saclug@garymcglinn.com):
I guess I still don't understand how they got in.
Well, that's after all a hard problem, isn't it?
Here's a rhetorical question, which I'm going to ask not to be a jerk but rather as lead-in to some practical suggestions going forward: Have you ever portscanned your system from a nearby IP, in order to better understand and document your system's attack surface? Did you ever then go through each service thus proven to be exposed to public network traffic, studying each and its configuration details, to reduce the attack surface and use only very solid code to handle public traffic?
Long ago, when my mother-in-law Cheryl moved down from British Columbia to join me and my wife, and introduced to our household an MS-Windows bow (first Win98 and then later WinXP), one of the favours I did for her was nmap'ing her IP, like this:
# nmap -vv -sT -sR -O -I - oN nmap.log -n 10.0.1.3 #TCP scan
# nmap -vv -sU -P0 -sR -O -oN nmap2.log 10.0.1.3 #UDP scan
# nmap -vv -sA -sR -O -n -oN nmap3.log -n 10.0.1.3 #version detect/traceroute
Reading those logfiles with Cheryl clarified what was potentially attackable on her Winbox, and thus where to spend time locking things down or switching them off or otherwise mitigate risk.
On my own server system, I learned the hard way that the default CGI mode of Perl-based Web server logfile analysing ("Web analytics") program AWstats suffered repeated and catastrophic security holes that were fatal to system security. (See: http://www.awstats.org/awstats_security_news.php .) However, a few seconds of pondering revealed that it was not _necessary_ to have AWstats continuously updating as dynamic Web code -- that it would be more than sufficient to have it run via a local-process cron job and write out a static HTML report every minute.
Likewise, I made the same overall judgement about public-facing PHP, that it suffered too many and too severe, recurring security meltdowns for me to justify exposing mod_php to the Internet. Therefore, I re-engineered all PHP-driven features on my site to be _static_ HTML pages created by running /usr/lib/php _locally_ via Makefiles or cron jobs.
Maybe you'll never be able to tell for certain how the bad guys got in. Maybe you'll only be able to make an educated guess.
Another rhetorical question, Gary: Does your system run logcheck? Is it well-tuned, i.e., have you spent some time fine-tuning it by adding tailored lines to /etc/logcheck/ignore.d.server/local-rules, reducing logcheck's irrelevant noise in its e-mails?
I remember the day I realised that PHP's security menace was intolerable, and that I needed to do something instantly. It was an almost worst-case scenario: I was partway across the Pacific Ocean on a trans-Pacific cruise to Sydney, which meant that I got basically about an hour per day of maddeningly slow, unreliable, and high-latency satellite Internet connectivity. _But_, I had developed a well-honed /etc/logcheck/ignore.d.server/local-rules , such that normally logcheck said literally or almost nothing in its e-mailed reports. Therefore, the sudden automated attack on my system via lib_php and Apache really stood out, and I instantly knew I had a big problem -- made worse by the fact that my ssh session was only barely functional.
I knew that the automated bot had almost certainly handed remote access via Apache to remote criminals with user www-data. I'd already made sure nothing in Apache's tree was writeable by that UID, but also correctly guessed that the next stage would be to attempt a series of local privilege-escalation attacks to try to crack root-user authority. If I were lucky, this would be relatively slow and human-guided. So, I quickly took several steps, including adding a new job to /etc/crontab that killed all Apache httpd processes and restarted the httpd literally every minute. (I also took some other steps that I'll not detail here.)
When I reached Sydney and had enough bandwidth to do some more examination, I found to my amusement that this inspired hack of mine had probably given some criminal severe frustration: I found signs that a system-cracking toolkit had repeatedly gotten downloaded and started with www-data authority, but then the process was killed less than a minute later, each and every time.
Take that, asshats.
My longer-term remedy was: No public-facing PHP, no public-facing PHP security problem. Simple!
[...] and the logs don't show that.
It's possible that the (_various_) logs do show other things, though. If you really want to make an educated guess for how they got in, and how they escalated to system privilege, that should be part of where you (and preferably your analysis scripts; don't just read raw logs) look.
Beware of most security advice. (I don't exempt myself. Be skeptical of mine, too.) _Way_ too many people having "security ideas" are really just gadget freaks. Before adding "a firewall rule that simply walls out any IP that tries to log in with a user that isn't [your] account", for example, stop to ponder hard the most important question in any troubleshooting scenario: What problem am I solving? Is it the right problem?
I once upon a time co-wrote a modestly successful essay called "How to Ask Questions the Smart Way", that bears on troubleshooting, but, well, modest forfends. ;->
I will close with:
http://linuxmafia.com/~rick/lexicon.html#zebra-hunting
Zebra Hunting
Diagnostic failure mode, in which investigation goes wrong through failure to consider obvious, simple explanations first (as suggested by Occam's Razor). As celebrated medical researcher Theodore E. Woodward, Chair of Medicine at University of Maryland's School of Medicine, used to advise interns: "When you hear hoofbeats, think of horses, not zebras." When helping computerists diagnose problems, I must sometimes intervene to halt energetic, futile hunts for imaginary zebras.
Tip of the hat to Katherine Ottaway, MD, for her light verse poem Catching Zebras[1], which also bears, if mischievously, on this matter.
So, afterthought to one bit I wrote, the other day:
Here's a rhetorical question, which I'm going to ask not to be a jerk but rather as lead-in to some practical suggestions going forward: Have you ever portscanned your system from a nearby IP, in order to better understand and document your system's attack surface? Did you ever then go through each service thus proven to be exposed to public network traffic, studying each and its configuration details, to reduce the attack surface and use only very solid code to handle public traffic?
Some years ago, I was senior sysadmin at a firm operating a "merchant bank", which is a term of art for a data farm that processes credit card transaction data. As you might imagine, we took software and network security very seriously.
One of the recurring tasks I had to shoulder every six months was guiding us through PCI DSS -- the Payment Card Industry Data Security Standard testing and certification. We paid one of many specialised firms to probe our infrastructure using their own bot software, which fingerprinted our site's software versions and issued a report about what they believed we were running, and then listed known vulnerabilities in some configurations of those versions of those pieces of software.
I would login to the testing company's site, and read a number of such claims about parts of our site, and those candidate vulnerabilities -- and my job was to prove to them that our site did _not_ suffer those vulnerabilities in fact, or alter/upgrade our software so as to eliminate the possible problem, and, again, prove that no problem (any longer) existed.
Along the way, I read and analysed a lot of CVEs, which IIRC stands for Common Vulnerabilities and Exposures (record). E.g., I'd read the report that says "We detected that you're running Apache 2.2.15, so prove to us you are not vulnerable to the flaws described in this list of CVEs, in order to pass DSS certification." So, for many of those CVEs, I'd query CentOS package metadata, thus:
$ rpm -q --changelog {package-name} | grep CVENUMBER
...and find that CentOS/Red Hat had backported a fix to that package of Apache httpd v. 2.2.15, quote that return data to the PCI DSS tester, and Robert's your father's brother.
In other cases, even though CentOS/RHAT hadn't backported a patch, reading the CVE details revealed that the vulnerability existed only in configurations of the software we had carefully avoided by switching off unwanted features or not having enabled non-default ones.
And that is a theme I wished to stress: Code you are not running is code that cannot be attacked. If you are not sure you need a software feature (especially in code exposed to public data), maybe you should disable it in your running configuration, on general principle.
In rare cases, to sidestep/eliminate the risk scenario described in a CVE, I would need to upgrade/replace software or otherwise make substantive site changes. But, point is, that was _rare_. A quite large fraction of CVEs could be finessed by pared-down site configuration, and using maintained packages that backport security fixes.
By the way, it's worth subscribing to, and skim-reading, your distro's security announce mailing list.
(And beware, especially in Internet-facing roles, of distro releases that are past their security-coverage EOL dates.)
-
Brian E. Lavender -
Gary -
Gary -
Rick Moen