Advance and Be Mechanized

There’s a security problem in Webmin that affects all versions previous to the latest one (1.290) : the Artbitrary remote file access.

In short, anyone can grab any file from your server and in particular the passwords file /etc/shadow . Once a cracker has this file, he can run a brute force or rainbow attack to get the original Linux user passwords, so he could have login privileges including root and therefore fully compromising the server.

Do I run Webmin?

This is the easy part. Webmin is a web server that runs by default in port 10000, so you can see if it’s running by looking at https://yourip:10000 or http://yourip:10000 (Usermin from Webmin runs at port 20000)

Inside the server you can test if Webmin is running for instance with: ps aux|grep webmin or /etc/init.d/webmin status

To see if you have Webmin installed at all you can do a locate webmin (if locate complaints you can do a updatedb first) or just:
find / -name webmin

To see the version of webmin you have: cat /etc/webmin/version
If the version is earlier than 1.290 and webmin is running then your server is vulnerable.

Have I been compromised?

To check if somebody has taken a file from your server using this exploit, you can see the webmin log for unauthenticated access with:
grep unauthen /var/webmin/miniserv.log | grep -v jpg

(the –v jpg is to disregard the display of an icon). Credit to Peter at Rimuhosting.

if you get any result, the end of each line will tell you which file has been taken away:

w.x.y.z - - [23/Jul/2006:02:51:36 -0500] "GET /unauthenticated/..%01/..%01/
%01/..%01/..%01/..%01/..%01/..%01//etc/shadow HTTP/1.1" 200 32

In this case we see that the cracker has copied the /etc/shadow password file (if you get a 404 instead of a 200 "OK" at the end that's http for "not found" and you're lucky), and we also get a timestamp and the ip of the intruder (w.x.y.x here to protect the guilty, in this day and age you never know who sues who).

What do I do if I’ve been compromised?

This is just the short answer but basically:

1. Stop webmin if it’s running: service webmin stop or /etc/init.d/webmin stop
Confirm that it’s not running with ps aux|grep webmin

2. Change all your Linux user passwords.
Look at the users with passwords (for instance with less /etc/shadow); the file has fields separated by colons :, the second field is the encrypted password and you will only see root and the users you created with a long string after their name, the rest have an asterisk * in that field, for instance:

root:$1$Jn3xi7Tp7$7jc0T725mD4eXLWh0wA581:13371:0:99999:7:::
bin:*:12821:0:99999:7:::
daemon:*:12821:0:99999:7:::

So change the password of all the Linux users with: passwd username

Now, if you plan on using again Webmin, you have to upgrade to the latest version, do so by downloading from http://webmin.com/ and installing it.

It's also a good idea to harden Webmin by changing the port it listens to and using IP Access Control, both options are in the Webmin configuration module.

If you are using only a few modules of webmin you can check if there are safer alternatives, for instance phpmyadmin for administration of MySQL databases or Squirrelmail as a web mail client.

You also have to check if the cracker already logged in with one of the usernames and passwords stolen, you can see this with the ‘last -a’ command and looking for unrecognized login locations since the date the passwords were stolen.

This vulnerability is yet another reason to use ssh only with key pairs only, in which case the cracker wouldn’t be able to log in (but other problems like compromised email accounts would persist).

So I'm not one for criticising something I haven't tried or don't know much about, but that's precisely what I'm going to do here; giving some thoughts about the security in Microsoft's Windows Vista. After all, I'm familiar the security of previous Windows versions and for the improvements I'll just take a look at what Microsoft itself is saying.

So let's start with Microsoft's own marketing literature by examining the Windows Vista Features - Security page.

What are the new improvements in this new version of Windows?. Let's see: there's a thing called User Account Control (UAC) that takes away the default administrative privileges and minimizes its use. This is a great idea and it was about time, since it's so 1980's (or 1970's) - finally Windows is catching up with UNIX -

What are the other great new security features?. There are none described in this page. They have four bullets, but three of them (automatic updates, security center and firewall) are already available in Windows XP, and the fourth one is the Malicious Software Removal Tool (MSRT) that is not part of Vista, but a tool to download.

More interesting is this interview with Windows chief Jim Allchin: Buy Vista for the security.

There are two highlights for me. One is having the responsible of a product that has been over 12 years in the market say with a straight face: "A standard Windows XP computer can get hacked the moment it is connected to the Internet." It's like the ad on the paper they had when they released Windows 2000 (or something) and there was a picture of the "blue screen of death" and then some copy of how the new Windows would have none of that. Imagine an ad by Honda cars showcasing the new Accord with a picture of last year's model totally wrecked and a caption that read: "New Accord: this time the breaks will work".

The second interesting point is a new "double-checking" Windows security feature that it wasn't mention in Microsoft's page: "As an example of double-checking, Allchin said Microsoft has marked the OS services to know what network ports they should open and what OS functions they should call. Then, another part of the OS verifies the process. "If we ever find something trying to open a port that the developer said it should not be opening, it is immediately shut down," he said.".

And apparently Vista is going to take on spyware and make a lot of users and tech support centre people very happy, and as a side effect it will make anti-spyware software companies miserable. That's what happens when you are a good parasite (living off others and making them better) and your business depends on the whims of somebody else.

The purpose of this new project I started is to have a security level management tool similar to Mandriva's msec.

Ubuntu page: Ubuntu Security Levels. Everybody is welcome to join.

The idea is to harden (and monitor/log) the security of Ubuntu by having well-known states or levels that are easy to understand and manage by users and sysadmins.

• Easy, like in Mandriva, by typing just "msec 3" we go to a level deemed appropriate for desktops connected to the Internet. No need to go through screens answering difficult questions like with Bastille Linux.


• System administrators will be aware that the systems are in a particular well-known configuration regarding basic aspects of security ("this web server is level 4, that critical server is level 5"). The caveat of course is to have a false sense of security.


• The proposed difference in philosophy with Mandriva's msec is that the users won't be able to customize (at least easily) the directives for the levels; the simpler the better

There are several Linux Live CDs that are specialized in network security tools like:

L.A.S.
Trinux
PHLAK
Knoppix STD

I like Knoppix STD; it's a very complete infosec toolkit. (First time users: remember to right-click with the mouse once you're on the desktop).

I've been getting involved in Ubuntu and a project I just started is the Ubuntu Security LiveCD, an Ubuntu LiveCD remastered with many security tools.

The security live CD I put together works pretty well and I intend to publish the iso file once I polish it a little more.

The Problem

The problem is malicious people trying dictionary attacks against an SSH server.

We can detect the attack in the /var/log/secure file, for instance:

Apr 5 18:09:54 testguest sshd[1220]: Invalid user andres from ::ffff:218.54.172.190
Apr 5 18:09:57 testguest sshd[1220]: Failed password for invalid user andres from ::ffff:218.54.172.190 port 2446 ssh2
Apr 5 18:10:00 testguest sshd[1222]: Invalid user barbara from ::ffff:218.54.172.190
Apr 5 18:10:02 testguest sshd[1222]: Failed password for invalid user barbara from ::ffff:218.54.172.190 port 2583 ssh2
Apr 5 18:10:06 testguest sshd[1224]: Invalid user adine from ::ffff:218.54.172.190
Apr 5 18:10:08 testguest sshd[1224]: Failed password for invalid user adine from ::ffff:218.54.172.190 port 2720 ssh2
Apr 5 18:10:11 testguest sshd[1226]: Invalid user test from ::ffff:218.54.172.190
Apr 5 18:10:13 testguest sshd[1226]: Failed password for invalid user test from ::ffff:218.54.172.190 port 2860 ssh2
Apr 5 18:10:16 testguest sshd[1228]: Invalid user guest from ::ffff:218.54.172.190
Apr 5 18:10:19 testguest sshd[1228]: Failed password for invalid user guest from ::ffff:218.54.172.190 port 3001 ssh2

Summary / Best Bang for your Buck

• Change the port sshd is listening to. Edit the sshd config file (/etc/ssh/sshd_config for instance) and change the value of Port to something other than 22 and restart the sshd server (/etc/init.d/sshd restart). This will reduce dramatically the number of attacks you'll get.


• Use strong passwords for all your Linux users (long, mixing lower and upper case, symbols and numbers) or use ssh keys that will make the whole ssh attack pointless (but you have the keys to manage now)

Let's take a look at several approaches to protect ourselves from these attacks.

Somebody must have written a script about this

Yes, there are several scripts out there, like Tattle - Automatic Reporting Of SSH Brute-Force Attacks (Perl), sshd_sentry (Perl) and DenyHosts (Phyton)

Banning guilty IP addresses

Another option is to ban the ip address from the attackers that are recorded in the previous log by using iptables.

So here's a little shell script I wrote as an exercise that will parse the /var/log/secure file for the guilty IP addresses and then will reject any incoming packets from them by inserting a drop rule in the INPUT chain before its two last rules of our current iptables (the one to accept incoming SSH connections and the last default deny all). To check the before and after effect: # iptables -L -n

# Fernando Duran, August 2004
# Ban ssh brute-forcing IP addresses with iptables

#!/bin/bash
# delete old ip bans, restart iptables
# /etc/init.d/iptables restart
/sbin/service iptables restart

# bad login lines
ssh_line=/tmp/ssh_line
touch $ssh_line

# all ip addresses, duplicated after each script execution
all_ip=/tmp/blacklist
touch $all_ip

# intermediary merge (old + new ips) file
merge_ip=/tmp/blackmerge
touch $merge_ip

# list of unique banned ip addresses
black_ip=/var/log/blacklist
touch $black_ip

# extract guilty ip addresses (Red Hat, other distros may vary the format)
grep sshd /var/log/secure | grep Failed | grep invalid > $ssh_line
cut -d : -f 7 $ssh_line | cut -d \ -f 1 | sort | uniq > $all_ip

# get rid of duplicates
echo -n "" > $merge_ip
cat $all_ip | sort | uniq > $merge_ip
cat $black_ip | sort | uniq >> $merge_ip
cat $merge_ip | sort | uniq > $black_ip

# insert all unique ips into iptables
lines=`/sbin/iptables -L INPUT -n | wc -l`
rules=`expr $lines - 3`
file=`cat $black_ip`
for ip in $file; do
/sbin/iptables -I INPUT $rules -s $ip -j DROP
#echo " $ip banned"
done

The way to restart iptables and the format of /var/log/secure will have to be adapted to your Linux distro.
The script is meant to be ran periodically in order to add new bad IPs to our black list, for example we can run it as a cron job every hour for instance.

The problem is that the script doesn't solve much. The attacker can try hundreds of user/passwords until the script is fired, and we'd be placing more burden on the system if we'd run it frequently, like every minute or so.

Limiting the rate of connections

Netfilter's iptables is very powerful, and one of its features allows us to mitigate brute force attacks and DoS attacks by limiting the rate of packets matching a rule.

So for instance we can append these rules:

/sbin/iptables -A INPUT -p tcp --dport 22 --syn -m limit --limit 1/m --limit-burst 2 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 22 --syn -j DROP

The effect is that the firewall will only admit two (for example) incoming new SSH connections per minute (it will reset the "burst" counter every minute).

So this will be enough to discorage those idiots right?
No, actually it may make things worse because we can lock ourselves out if we want to log in with SSH and at the same time there's an attack going on; then iptables will be preventing any incoming SSH connection (including ours).

If the previous shell script to ban ip addresses is used then this problem is less severe since we'll "only" have to wait for the cron job to run the script and ban the bad ip address so that later iptables can let us in. Another thing we can do is to place before those rules one that will grant SSH access to our (known) ip.

Using SSH keys

A way to get rid once and for all of the SSH brute force attacks is to allow SSH logins only with encryption keys. The price to pay is that we have to create the keys first (not a problem unless we have many users) and we have to take care of the private keys (have secure copies of them).

Basically we have to create a pair of public and private key in the client computer and copy the public key in the server.

- Linux:

# ssh-keygen -t rsa

(We can also use dsa encryption instead of dsa)
If we enter a passphrase then we'll have to enter it every time that we wan to connect to the server using SSH.

The private key is generated into $HOME/.ssh/id_rsa, and the public key into $HOME/.ssh/id_rsa.pub
It's the public key that we have to append to the server's .ssh/authorized_keys file, located in the home directory of the user we want to authenticate.

- In Windows, if we are using Putty we can use puttygen to create the pair, then save the private key to the local computer (we can protect the file with a passphrase) and similarly copy the public key to the same file in the server. The formats of the keys in Putty are different that the ones generated with OpenSSH, but if we already have an OpenSSH one puttygen can convert it to its format.

At the server, if there's no .ssh or authorized_keys file for that user, we can create them when running as that user:

$ cd ~
$ mkdir .ssh
$ touch .ssh/authorized_keys
$ chmod 600 .ssh/authorized_keys

After testing that we can log in with the keys, we can change the SSH server configuration file so that sshd will only accept keys as form of authentication.

Edit /etc/ssh/sshd_config and set:

PasswordAuthentication no

It's also a good idea to restrict the accepted version of SSH:

Protocol 2

If we use the "no password" setting it's very important to keep one or more copies of our private keys in removable media like a flash USB thumb drive or similar, so we can access the server in case we have not access the hard drive of the client computer.

After the changes the SSH server has to be restarted:

# service sshd restart
(or # /etc/init.d/sshd restart in Debian System-V style)

Let's take a look at a
Screenshot of beginning of SSH session

Taken with Ethereal and using Ubuntu Linux both in the client and the server, as can be seen in the first packet, along with the SSH version.

Note the declaration of supported algorithms in the client-server "Key Exchange Init" packet.

Ubuntu Linux comes with the SSH client by default (ssh-client). The server package openssh-server is easily installed with Synaptic or using the "apt-get install" command. The SSH server can be run with no further configuration; to get it started just do: /etc/init.d/ssh start.

The keys and configuration are under the /etc/ssh directory (both in client and server):

# ls -l /etc/ssh/
total 140
-rw-r--r-- 1 root root 111892 2004-10-07 14:29 moduli
-rw-r--r-- 1 root root 1185 2004-10-07 14:29 ssh_config
-rw-r--r-- 1 root root 1743 2005-06-15 18:54 sshd_config
-rw------- 1 root root 668 2005-06-15 18:54 ssh_host_dsa_key
-rw-r--r-- 1 root root 602 2005-06-15 18:54 ssh_host_dsa_key.pub
-rw------- 1 root root 887 2005-06-15 18:54 ssh_host_rsa_key
-rw-r--r-- 1 root root 222 2005-06-15 18:54 ssh_host_rsa_key.pub

SSH2 tries to authenticate in one of three ways (and in this order): 1) hostbased method (the server has a file of which hosts to trust) 2) public key (the client user generates a public/private key pair and the server knows the public key) and 3) sending the encrypted password of an existing user in the server, like in our case.

SSH also supports challenge-response authentication.

On the client side, the SSH server is added as a known host:

ls ~/.ssh/
known_hosts
# cat ~/.ssh/known_hosts
192.168.0.100 ssh-rsa AAAAB3NzaC...(rest of the server's public key)

If we make a second SSH connection, the server is authenticated if there's a mathing entry in this file.

Links:

Drink from Da source
Man pages and Internet Drafts
Wikipedia entry
OpenSSH Notes / Unix
Presentation

nmap is an excellent open source network scanner tool that is used for gathering network information in the form of hosts and services enumeration.

Here's an snippet of nmap discovering the open ports of one of my servers:

# nmap 127.0.0.1

Starting nmap 3.50 ( http://www.insecure.org/nmap/ ) at 2005-05-30 23:23 EDT
Interesting ports on localhost.localdomain (127.0.0.1):
(The 1654 ports scanned but not shown below are in state: closed)
PORT STATE SERVICE
25/tcp open smtp
80/tcp open http

But when nmap sees a port open it just names the standard service that usually runs in that (well-known) port, as listed for example in /etc/services .

Let's check this by changing the port that the target web server is listening to. In my case I'm running Apache2, so I edit the file /etc/apache2/ports.conf, and I change the "80" with "8888" for example. Then I restart the server: /etc/init.d/apache2 restart

Let's run nmap again:

# nmap 127.0.0.1

Starting nmap 3.50 ( http://www.insecure.org/nmap/ ) at 2005-05-30 23:29 EDT
Interesting ports on localhost.localdomain (127.0.0.1):
(The 1654 ports scanned but not shown below are in state: closed)
PORT STATE SERVICE
25/tcp open smtp
8888/tcp open sun-answerbook

So as we can see, nmap finds port 8888 open but declares the wrong server running in that port.

Amap to the rescue. (We'll forget about more complex vulnerability assessment tools like Nessus etc).

So I downloaded Amap,and installing is the standard "untar, configure and make" routine. Let's see what Amap has to say now:

./amap 127.0.0.1 8888
amap v5.0 (www.thc.org/thc-amap) started at 2005-05-30 23:34:14 - MAPPING mode

Protocol on 127.0.0.1:8888/tcp matches http
Protocol on 127.0.0.1:8888/tcp matches http-apache-2

Unidentified ports: none.

amap v5.0 finished at 2005-05-30 23:34:20

Amap is able to correctly identify the right service (and version!) running on the unconventional port. It uses signatures of the server responses to identify the right services. It's a fast scanning tool and complements nmap by using nmap's output as its input: Amap's README.

Since I have this server running (conveniently) http and smtp, let's see how we can gain more information by telneting into these services.

For the mail server:

# telnet 127.0.0.1 25
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 localhost.localdomain ESMTP Postfix (Ubuntu)

So right away the mail server tells me what application is and even the Linux distribution.

We can now pass SMTP commands to the server. For example if we want to know if there's an user called "fernando":

vrfy fernando
252 fernando

It said "yes". Finally we disconnect:

quit
221 Bye
Connection closed by foreign host.

For the web server:

We telnet into its port (80 or to the one we changed) :

# telnet 127.0.0.1 8888
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

and pass some garbage:

blah
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found< </h1>
<p>The document has moved <a href="http://localhost.localdomain/apache2-default/">here</a>.</p>
<hr />
<address>Apache/2.0.50 (Ubuntu) PHP/4.3.8 Server at localhost.localdomain Port 80</address>
</body></html>
Connection closed by foreign host.

The web server returns an error page with lots of information about itself.

Another service that we can apply this technique of telneting plus passing commands is an FTP server; in this case we can try the SYST command to get information about the system.