Quoting saclug@garymcglinn.com (saclug@garymcglinn.com):
Again sent earlier. Thanks for the intervening comments about vulnerabilities and versions.
Glad to. Hope they're useful.
A lot of issues and suggestions have been made and raised. My brief response is that yes, according to nmap and my intention I had port 23, for ssh (I moved it) and port 5900 open and the rpc port, I think. I'm going by memory.
My instant reaction would be to think that having an RPC portmapper process running and exposed (TCP/UDP port 111) to public networks is... curious.
RPC portmappers have a troubling security history, although that's a complex story that includes but is not limited to them making firewalling complex to the point of near-impossibility, because by definition they hand out service port assignments to other services, and that alone is a challenge to control. Possibly, you're already fully aware of that.
The other thing that is curious about that is that, when I think of RPC portmapper daemons, I think of them as the foundation of NIS and NFS -- both of which are likewise security-challenging. Thus the old joke dating back to SunOS 2.0 days that NFS stands for "No Friggin' Security".
Theoretically the only pinhole in the ISP router firewall was port 23. To use port 5900, you had to use an ssh tunnel.
I'm curious what you use the RPC portmapper for, then, and how you manage the security of services handed ports by it.
My experience with the upgrade treadmill is that it is a waste of time. By its own admission (if a concept can have that) the new versions will have issues and you need to upgrade. If the issues with an older version don't affect you, then it is perfectly fine to use it. Why upgrade and risk the fact that the new issues will affect your use case. The developers for Fedora 13 were no smarter or dumber than the developers who are writing Fedora 36. Or pick your distro of choice.
Well, let's talk about that.
Any host is going, over time, to require security maintenance, as to (at minimum, all software exposed to public data, which includes the kernel's network stack and all network services that use the network stack (and can be reached from public).
It is _absolutely_ feasible to adopt a set of software issued 15 years ago as part of a distro release, and for it to retain reasonable security going forward indefinitely, long past when distro security updates cease for that release -- _provided_ that you diligently assume manual security maintence duties from that time forward.
I may be doing a similar thing with my next server rebuild: I plan to have a very sparse, very hardened base system, that is just enough of a system to run KVM/qemu as a hypervisor. The system would then run two guest VM sessions, one of which would be my production server, the other the in-progress beta for its replacement.
In all likelihood (pending testing), I may compile my own host-system kernel in order to exercise maximum control on completely omitting code I don't absolutely need, so that it isn't present even as unloaded modules. And maybe I'll do the same for related code, maybe switching the libc to one of the minimal ones, and so on.
If I do that, then any such portion of the host-OS system will be outside the distro package regime, though I might bother to construct local packages so that the distro regime is aware of them, maybe with package pinning. The entire host-OS system would probably be under Ansible or Puppet configuration management, anyway.
But, the main point is, _you_ become the security maintainer of such a system. _You_ have to spend time reading CVEs and deciding what's system-relevant and what to do about it.
Backing up a bit, I'll confess I wasn't really paying close attention to your original "Hacked" posting eight days ago. If I understand correctly:
1. After noting an unexplained reboot,
2. you looked in "the audit log" and found successful SSH login (not by you) using a keypair, and some stuff about a service on a high-numbered port.
[Ugh, Mailman3's Hyperkitty archiver is _still_ dreadful.]
I'm not at all sure that initial entry was via ssh. That sounds quite improbable, for lots of reasons, including "how did the user's public key get into authorized_keys"?. OTOH, I have insufficient data to say what the avenue of entry was.
Going by your detailed description of the many things that the intruders changed and broke, it may be way too late to figure out for sure what the avenue of entry was. However, if you really _were_ running a 15-year-old Linux kernel, then don't forget that the public attack service includes the network stack -- and CVEs for the kernel get ignored for 15 years at your peril.