Logs logs logs everyone loves logs

• logging, security, improvements
• by Seth

Audio for this article

I'm going to start by telling you a story of hacking, confusion and ultimately losing my stuff (for want of a politer term). 

As an application security engineer, I dip my toes in lots of domains, from systems administration to analytic to software engineering, even blue team work. When  I first started out I had a small server called Glados (named after the Portal2 computer). Glados was a public facing web server where I served up some useful stuff and it ran some of my programs, and also kept lots of my files some of them important files. until one day something strange happened. 

Now lets rewind. First of all this isn't a LAMP stack, nor is it windows server, it’s a bog standard off the shelf windows 7 machine, and poorly patched at that! 

I started noticing my files go missing, or renamed and some even edited to include random text, some English some not, I also noticed my disk space disappearing at an alarming rate. Before long, things I ran on it like a small minecraft server were struggling pretty bad, and I would come down at night and notice the tower was getting a little hot to say the least. I started looking into it, first of all pulling the logs from windows, and examining them, then I started looking at CPU usage and processes running. It didn’t take long to find something unusual, two files the first called rbaccess.exe and the second btcoin.exe. You guessed it, someone had hacked my machine, installed a backdoor for persistent access and started using it as a bitcoin miner. So I took the PC offline, wiped it (actually changed the hdd incase there were bitcoin I could nab off it) and started over, new system but this time I made sure it was patched properly and had some decent AV on it.

Lesson learned. So I don’t often use windows systems any more, except for client side stuff, but I do still program in c# so there is always a dip back into the operating system structure etc. I talk about how I hate active directory hacking, and how windows systems sometimes baffle me, but at the heart of it, I’ve only been involved in LAMP systems for about 10 years now, while windows has been my upbringing. That said LAMP is what I know and study lots and lots. 

This brings us back to my point, one of the first things I did when I started examining that infected machine was pull up the logs, they were the off the shelf windows event logs and it took a while to comb through them and see things but it makes a very important point that logging is very important. I’ve actually seen companies reduce there logging because they say that its not ‘required’ or ‘wastes space on the sever’, to me that’s like saying CCTV shouldn’t be recorded because ‘someone is watching the monitor anyway’. Yes they are indeed watching the monitor but when an incident goes down, all you have as proof is that persons account, while a tape of the incident would speak a million words!

So like most things I do I like to first define rules, then define a process. Logging comes in two forms, the first is systems logs, as in the logs built into the system already. You will find most stuff has logging out the box, for example apache2 does a pretty good job of logging access attempts at sites and also any errors both in separate log files and automatically archived into manageable files. Then there is custom logging, this comes in the form of something created or implemented that is additional to the systems logging. Examples of this is perhaps a database of all ip addresses accessing a website (good for top level analysis without combing through apache2 logs), or a Marco interceptor which emails you when a hacking attempt is detected in a URL, there are a few I will mention below in processes, but there are thousands out there. So lets start with rules. Here are mine, they may not suite every ones needs.

1. When starting out, have logging set high, you can fine tune as you go, but it will quickly help trouble shoot problems as they crop up.
2. Logging is a constantly changing and adapting process, be prepared to add, modify or remove logging
3. More is better than less.
4. That said, don’t log personal information such as emails, names, and addresses, as it’s a pain to ‘strip out’ later if someone does a GDPR request
5. Archive and store logging information off server, preferably in a log vault. This is very important!
6. Logging and Backing up are not the same process, and should never be treated as such.
7. Avoid using a block chain in logging. I'll explain this later.

Processes

So there are several processes to logging, I’m only going to cover the basics for now. Lets start with out the box processes. Sorry I’m only covering LAMP at the moment.

Apache2 - Apache2 logs two things, the first is access attempts and the second is errors. here is a great guide. It's a vast subject.

Mysql - This doesn't come with advanced logging out the box, and I should caution you about turning this on if you have small hard drives as it can eat up space fast, but like rule 1 says start big and scale down. Mysql does come with service logging which logs what's happening within the mysql service (errors, events etc)  here is a starting point, but additionally if you want to watch out for all the sql commands being run including, dangerous commands such as DROP or TRUNCATE,  you could write out logs using the following sql command to enable (you only need to run this once!) 

SET global general_log_file='/var/log/mysql_general_logging.log';
SET global log_output = 'file';
SET global general_log = on; ## make sure you do this last!

This logs ALL the sql ran on the server, from there having a crons script to check it regularly like so would highlight any issues 

files=( T*.CDT )
if grep -q 'DROP' "${files[-1]}"; then
    mail -s 'Alert' [email protected] <<END_MESSAGE
We've found "FCE-Error" in ${files[-1]}, do something!
END_MESSAGE
fi

This needs some tidying up and you would also then need to move the log file to a new location for analysis otherwise you would keep getting alerts! Further more you would need to ensure management of these log files carefully, as they grow and are very likely to contain customer information (raw sql statements remember)

Authlog - This is an active log of all activity requiring authorization from the system. It lists crons activity, when someone SSH's into the system etc. Its on by default (dont turn it off!) and is usually located in the /var/log/ directory called auth.log 

Bash log - Next lets look at some custom idea that I use. The first is a simple bash commands log, this is very useful if you want to see what a specific user is up to and also great for detecting or analyzing a hack or infiltrator. Its simple to implement as well. Here is how

Create a folder in /var/log called 'bash'
Add the following line to /etc/profile this will save local / non root users bash history

export PROMPT_COMMAND='echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(history 1)" >> /var/log/bash/bash-history-$(date "+%Y-%m-%d").log'

add the following to /root/.bashrc - This saves root commands only. (again useful to know whos been logging in as root

export PROMPT_COMMAND='echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(history 1)" >> /var/log/bash/bash-history-root$(date "+%Y-%m-%d").log'

Fail to ban - Fail to ban or F2B is a great product, but doesn't come installed automatically (boo!). Because there is so much you can use it for, such as monitoring web traffic, SSH logins, ftp, the list is quite long. Here is a great tutorial by linode (and no they are not sponsoring me) as a starting point. Remember when you code to return 403's when someone fails to login else F2B wont know! 

Interceptor rules - I'm not going to dig too deep into this here as I plan on writing a post about how I created my own interceptor (a non Joomla one!). This is a little like F2B only its specifically looking for attempts to hack in the URL. I get emails from my interceptor on a weekly basis people scanning my site trying to figure out what I am running etc. Here are some examples: 

You get the idea (by the way don't visit any of those links unless you wish to get your ip banned.) 

The interceptor is considered an active measure, and this should be noted, it doesn't just record it blocks as well, as does F2B. Passive measures simply record information and move on. When designing a logging architecture, try and get a good mix of both. 

A final word on logging 

Well I hope this has helped clarify just how important logging is. One final point, I've heard of people using block chain technology to prevent log entries being removed, although a great idea in principal, be very careful when doing so, as if you accidentally start logging PII  and someone asks you to remove it under legal means, then you break your logging or break the law, depending on what you choice to do.