Q I have a standard Enterprise Linux 3 server with no control panels and no hardware firewall. I only run an HTTP web server that requires MySQL and Sendmail for outgoing mail via PHP. There is only one user on the server me. I have not touched the default install apart from disabling VSFTP. All data is backed up daily. What I'd like to know is where the system is vulnerable, where any attack is likely to target and which area of my system should I be monitoring closely. This is how I see things from a security point of view:
Portsentry stops port scans.
I don't have a firewall and see no reason for one, or iptables, although I could be very wrong on this one.
A You've asked a host of really great questions - I'll try to answer them all briefly. To see what else on your system besides just HTTPd and Sendmail is 'exposed', we need to look at what makes up what I call your network profile. To see this for yourself, run netstat -antp to see which TCP services/ports are bound (and in your case exposed) as well as which binaries are associated with each:
# netstat -antp Active Internet connections (servers and established) Proto R-Q S-Q Loc.Addr. For.Add State PID/Program name tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 1018/perl tcp 0 0 0.0.0.0:1 10 0.0.0.0:* LISTEN 16577/xinetd tcp 0 0 0.0.0.0:143 0.0.0.0:* LISTEN 16577/xinetd tcp 0 0 0.0.0.0:1 1 0.0.0.0:* 1 LISTEN 1809/portmap tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 918/httpd tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 875/vsftpd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16351/sshd tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 18632/sendmail: acc tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 918/httpd tcp 0 48 69.20.9.105:22 64.39.0.38:32910 ESTABLISHED 19647/0
As you can see, there are around eight or nine different daemons binding to ports on a stock system. Now compare this with a remote portscan of your server using a tool like nmap (eg nmap -sS <IP>) to see what the world sees as your network profile. Remember to turn off Portsentry on your server or it will block you if you try a portscan! Your first layer of security is the network and/or iptables, so yes, I would look again at your decision not to have it. Iptables will block anything bad getting to the kernel. If a vulnerability is discovered in the Linux networking stack you could be vulnerable without iptables. It is excellent at preventing malformed or invalid packets from reaching your server.
One big problem in this day of web forums, blogs and other cool web apps is back-door bugs in non-vendor-supplied application layer packages such as phpBB and VBulletin. These cool web apps offer themselves out via your daemons and expose you to all kinds of bugs that are not fixed through anything you have set up on your system. In fact, this is probably the most successful 'flank attack' that we see these days with web hosting customers. Attackers exploit some weak code in phpBB or VBulletin, which gets them local user access as the user Apache, and then they're free to upload and launch local exploits or strong-arm attack tools to try for escalated user privileges (ie root access). All I can say is that if you choose a package like phpBB or VBulletin, you should track the bugs and patches very closely. The sftp/SSHroot restriction is actually set in the /etc/ssh/sshd.conf file, but you're right, the root user access control can also be controlled at the PAM layer.
Portsentry is a good outer layer warning and lockdown system that has saved many an insecure box. Try to combine manually going over your logwatch emails with tools such as chkrootkit, regular netstat and MD5SUM baseline comparisons. Red Hat's up2date is an essential tool in an enterprise server environment. If you're away for a few days the server can patch itself against most big vulnerabilities until you can get to manual patching.