Anth's Computer Cave

AAIMI Server Dog 0.2

12th Sep 2017

AAIMI Server Dog is a Python server-monitoring script for htaccess-protected Apache2 web servers running private web sites and applications.

It produces logs of suspicious access attempts showing usernames attempted, IP addresses and DNS details of the visitors, and any open TCP ports on the server. It then embeds the logs for viewing in an admin section of your website or web application.

We've embedded it into the latest versions of AAIMI Home Automation and it works great in that, and now it's time to release a stand-alone version.

Have a look at the image below, which shows an actual readout from AAIMI Server Dog running on a Raspberry Pi test server. These are the creatures that come knocking on your router at night if you run a web-facing server.

The readout from AAIMI Server Dog showing unauthorized access attempts on an Apache web server. Picture: Anthony Hartup.
The readout from AAIMI Server Dog showing unauthorized access attempts on an Apache web server. Picture: Anthony Hartup.

This screenshot actually shows a quiet day, and this is a non-domain server the gets a new public IP address almost weekly. With a static IP and a domain you would see many more entries every day.

Before you start worrying, these bots aren't getting through into the server. It has htaccess enabled and only genuine users can access anything. All the bad attempts above simply bounce off. It's still good to know who's knocking, though, which is why I made the Dog.


How it works

AAIMI Server Dog reads two pairs of Apache2 log files, the access.log files and the error.log files. These logs are anything-but human-friendly, but AAIMI can read and sort them in seconds.

First it determines the details for your server then it reads the access.log and access.log.1 files, and catalogs general access issues. By default it filters with username and IP whitelists, but you can also choose to filter by just one or the other.

AAIMI performs a two-way DNS lookup on each suspicious visitor and open TCP port, and provides any known domain details.

Next AAIMI checks the error.log and error.log.1 files, and gathers more details on username/password errors.

The readout from AAIMI Server Dog showing unauthorized access attempts on an Apache web server. Picture: Anthony Hartup.

You can see the time and duration of the attempts, whether they were flagged because of name or password, and all the names used. The names generally seem to be default usernames for common routers, camera, web applications and content-management systems like Wordpress. There's a good lesson from this: Change the default username and passwords on your devices!

It also displays the actual target directories the bots tried to access. You can see above the attempts to get into an sqlite database, most likely one using default configuration.


Limitations

# This is BETA software and should not be the sole tool used to monitor a production server.

# AAIMI Server Dog is tuned to work on a Raspberry Pi server running the standard Raspian operating system and Apache2 configurations. It should work on most other Debian-based servers but may require slight path changes.

# In this version the program only monitors access attempts. This can help you to spot a concerted attacks and allow you to take action. The next version will be able to actively block these bots on the fly. We already have it running in the dev-version, we're just streamlining a few things and putting some safeguards in-place to avoid false-positives.

# AAIMI stores long-term visitor details in JSON format, but this feature is not fully operational yet. Depending on the time you run the program it may overwrite partial days. We are fixing this now.


Download and configure the Dog

You can download the AAIMI Server Dog setup folder here. Extract the folder outside the webroot on your server and open the aaimi_server_dog.py file for editing.

Go to line 82 and note the mode variable is "standard". This is the default and checks access attempts by all criteria. That means bad IPs, bad usernames, bad passwords, and CONNECT attempts.

If you want to refine the mode to filter just by usernames and passwords, change mode to "user". This may be necessary if you have many remote devices connecting to your server and wish to disable filtering results by IP. If you wish to filter soley by IP, change mode to "ip_range". This could help if you have many users, but most of them connect over the same LAN.

Next we'll add our IP details. Ignore the public_ip variable(AAIMI will detect that) and go to the my_ips list on line 18.

The python list for excluding IP addresses from AAIMI monitoring. Picture: Anthony Hartup.

The localhost partial IP address is already in the list to exclude internal connections from monitoring, as is the IPV6 locahost IP. You need to add the first two sections of your LAN IP address, or the network IP the server is on, to the end of the list. For instance, if your LAN IP address for your server is 192.168.0.12, enter "192,168". You'll also need to add the first two sections of the IP your phones and other devices will connect with remotely. This excludes these IP ranges from the main monitoring functions.

We're using partial IP addresses instead of full addresses because your WAN devices like phones, etc, will receive a new IP regularly. In most cases the first two sections will remain the same and only the last two will change, meaning you shouldn't need to update the IP list each time. This means others with that IP range are filtered from results as well, but IP addresses are just one of the factors in the calculations and the other factors should still trigger. You can enter full IPs if you are really locking things down, but using partial IPs is a compromise between ease of use and the level of detection.

Next we go to line 31 and add the server's authorized users to the allowed_names list in quotes, separated by commas

The python list for excluding usernames from AAIMI monitoring. Picture: Anthony Hartup.

Finally, on lines 63 and 67 you can change the location to save the access_file.html and error_file.html files, ideally an admin-only section of your website you can access over the Internet. You'll need to move these access and error files from the main folder into your chosen locations.


Release the Hound

That's all you need to do to take the Dog for a quick run from the terminal. SSH into your server and navigate to the aaimi_server_dog folder. Type:

sudo aaimi_server_dog.py and press Enter

The access list from AAIMI monitoring. Picture: Anthony Hartup.

Now open the access_file.html file with your browser and you should see the same list.

The access list from AAIMI monitoring. Picture: Anthony Hartup.

Open the error_file.html file with your browser and you should see the more details for any bots that have deliberately tried to guess usernames.

The access list from AAIMI monitoring. Picture: Anthony Hartup.

It looks pretty dull like this, but when you load the list into your web html via PHP or Javascript it will use your website's CSS styles to smarten it up.

Embedd the program

In AAIMI Home Automation we use the 24/7 room-control program to call aaimi_server_dog.py daily.

In the home automation web GUI we've embedded the two log files in the Logs tab in the left-column with three control buttons.

The access list from AAIMI monitoring. Picture: Anthony Hartup.

The Access button displays the main access log, the AuthErrors displays the error log. AAIMI Home Automation runs the Dog every day automatically, but you can also update the logs any time with the Refresh button.

We use PHP to call the refresh, and to display the two AAIMI log files in the Logs column.

I'll go into more details about embedding AAIMI Server Dog in you applications in the next article, coming soon.

That's it for this version. Version 0.3 should be ready in a month or so.

Cheers

Anth

_____________________________________________


Comments

Leave a comment on this article