Linux Hardening

Summary

A look at Linux hardening in the AWS or cloud environment. This article refers to Redhat Enterprise Linux (RHEL) versions 5-7, CentOS versions 5-7 and Amazon Linux all versions.

Operating system hardening is a long, detailed and constantly evolving topic. As such we will not discuss on this article database, general or web application security practises. Here we will focus on the Linux, its operating and network software and its related settings.

Because this is a cloud environment we won't look at physical security or hardware issues.

Further references

https://www.sans.org/media/score/checklists/linuxchecklist.pdf

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/pdf/Security_Guide/Red_Hat_Enterprise_Linux-6-Security_Guide-en-US.pdf

General practise

Compartmentalization

To protect against attacks and to reduce the attack vector footprint for compromised systems multi-tier architecture or "n-tier" deployment approach is still best practise. For example in a web application a front end should consist of a proxy on one environment of instances, the web or application server in another and the back-end database in yet another with the appropriate level of network separation between each component. In general co-location of software on the same instance is a bad practise.

Minimalization

Software installed

A general practice in any type of server hardening is to install only what is actually required particularly in production environments. This aso usually means to not install compilers and other such development tools on production environments.

This should be proved in staging first before implementing in production by reducing suspected un-needed software and ensuring by QA process the application continues to work as expected.

Services running

As well as reducing the software installed sometimes you cannot remove everything in particular some services like Apache web server. Make sure that services that are not needed but have dependencies requiring them to be installed are not running. Use commands like:

To see what services are enable at boot:

$ sudo chkconfig
abrt-ccpp       0:off   1:off   2:off   3:on    4:off   5:on    6:off
abrt-oops       0:off   1:off   2:off   3:on    4:off   5:on    6:off
abrtd           0:off   1:off   2:off   3:on    4:off   5:on    6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
...

Disable and stop un-needed services:

$ sudo chkconfig  ypbind off
$ sudo service ypbind off

Xinetd and inetd.conf

If running the older /etc/inetd.conf file, be sure to disable unnecessary services by removing them (or commenting them out) from the inetd.conf file. For example, to remove telnet access, remove the following line:

telnet  stream  tcp    nowait  root  /usr/sbin/telnetd  telnetd

On systems running scripts from the xinetd.d directory, disable the services by changing the script from ‘disable = no’ to ‘disable = yes’. On redhat type systems the xinetd directory is in in /etc/xinetd.d.

service rsync
{
        disable = yes
        flags           = IPv6
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/bin/rsync
        server_args     = --daemon
        log_on_failure  += USERID
}

You will need to send a HUP signal to the inetd process after modifying the configuration files (kill -HUP processID)

Removed poorly authenticated and un-encrypted network commands

Under most network configurations, user names, passwords, FTP / telnet / rsh commands and transferred files can be captured by anyone on the same network using a packet sniffer. The common solution to this problem is to use either OpenSSH , SFTP, or FTPS (FTP over SSL), which adds SSL or TLS encryption to FTP.

To remove outdated and in-secure services: NIS:

sudo yum erase inetd xinetd ypserv tftp-server telnet-server rsh-serve

Note that sometimes the telnet client is used for troubleshooting and it is appropriate for it to remain installed for that purpose. The package name is simply "telnet".

Patching

Keep patches up to date with

$ sudo yum update

Accounts

Individual accounts are preferred over service accounts.

Expire accounts when individual no longer requires access or leaves the company or project.

Use chage command to list or change account details related to password changes and expiration.

sudo chage --list kinscoe
Last password change                                    : Jan 13, 2014
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7

You can set expiration date when the account is created --expiredate option to useradd.

sudo useradd -e 2017-07-30 someuser

Separate partitions for file systems

Separate Disk Partitions or EBS volumes

Separation of the operating system files from user files may result into a better and secure system.

One such reason is to protect against hard links attempting to access alternate directory paths. This is mostly defended now by SELinux in RHEL 6 and later. See article http://danwalsh.livejournal.com/64493.html

Make sure the following filesystems are mounted on separate partitions:

/usr
/home
/var
/var/tmp
/tmp

In AWS with prebuilt AMI's this is not always practical (although it can be done) so at the very least it is recommended to mount /home and the rest of the operating system separately.

The second reason to use a separate partition for running services and data is to protect against execution and SUID attacks particularly for web server and FTP server roots.

Edit /etc/fstab file and make sure you add the following configuration options:

• noexec – Do not set execution of any binaries on this partition (prevents execution of binaries but allows scripts).
• nodev – Do not allow character or special devices on this partition (prevents use of device files such as zero, sda etc).
• nosuid – Do not set SUID/SGID access on this partition (prevent the setuid bit).

Sample /etc/fstab entry to to limit user access on /dev/sda5 (ftp server root directory):

/dev/xvdeg1  /ftpdata          ext4    defaults,nosuid,nodev,noexec 1 2

Disable Unwanted SUID and SGID Binaries

See also http://tldp.org/HOWTO/html_single/Security-HOWTO/#AEN432

All SUID/SGID bits enabled on a file can be misused when the SUID/SGID executable has a security problem or bug. All local or remote user can use such file. It is a good practise to find all such files.

Use the find command as follows:

#See all set user id files:
find / -perm +4000

# See all group id files
find / -perm +2000

# Or combine both in a single command
find / \( -perm -4000 -o -perm -2000 \) -print
find / -path -prune -o -type f -perm +6000 -ls

World-writable files

Ensure you do not have world-writable files on the system. If world-writable files are truly required (for example sharing between accounts) consider revising the design of the application such that the service account is in the same unix group as the individuals who are accessing application files. The make the files group-writable. You can also use ACL's fir finer grain access control. In the majority if not in all cases world-writable files are the result of a poor design and are not truly required. In many cases world-writable files are left behind from troubleshooting access issues and they were never cleaned up.

Use the following command to find all world writable and sticky bits set files:

find /dir -xdev -type d \( -perm -0002 -a ! -perm -1000 \) -print

You need to investigate each reported file and either set correct user and group permission or remove it.

Files owned by non-existent accounts

Files not owned by any existent account or group can pose a security risk or more importantly may be a sign of an already impending attack.

Some recommend commenting out or disabling revoked users in the local passwd file so that you can track file ownership to revoked accounts.

sudo passwd -l accountName

If user-id's (UID) appear in files that have never been issued an account locally that may be sign of an attack that has begun.

Keep in mind if NFS is used the UID may belong to a remote account. This is a good reason for using organized, separate and pre-defined UID and group-id's (GID) such as can be used in NIS.

Locate files on the local system which do not belong to a valid user and a valid group. You will need to run this on every local non-NFS filesystem mount:

sudo find /dir -xdev \( -nouser -o -nogroup \) -ls

Kerberos

One caveat for Kerberos is that time is required to be in sync between clients and the kservers. ktickets in particular will expire in a short amount of time. See the next paragraph on time synchronization.

Kerberos performs authentication as a trusted third party authentication service by using cryptographic shared secret under the assumption that packets traveling along the insecure network can be read, modified, and inserted. Kerberos builds on symmetric-key cryptography and requires a key distribution center. You can make remote login, remote copy, secure inter-system file copying and other high-risk tasks safer and more controllable using Kerberos. So, when users authenticate to network services using Kerberos, unauthorized users attempting to gather passwords by monitoring network traffic are effectively thwarted. For more in Kerberos see https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Using_Kerberos.html

Kerberos is provided through AWS Directory Services. https://aws.amazon.com/directoryservice/faqs/

Time correction and synchronization

The system time should always be correct and ideally all in the same time zone for event detection, correlation and management.

The recommended time synchronization method on Linux systems in AWS is to use NTP. More on NTP in the AWS environment see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

Software firewalls

Note that the firewall running in Linux may have to be disabled to allow certains kinds of network communication. In RHEL 5 and 6 this is user-mode process called iptables (service name is iptables for IPv4 and ip6tables for IPv6). In RHEL 7 it is re-worked and referred to as firewalld. In addition there may exists policies enforced on RHEL called selinux. selinux is not a firewall per se but rather a set of policies controlling access to system resources including but not limited to network communication.

iptables

Otherwise known as Netfilter. Introduced in Linux kernel 2.4 and still used. See also https://en.wikipedia.org/wiki/Iptables, iptables to allow Apache and Tomcat ports and Disable iptables and firewalld.

Work in progress

Firewalld

See also Disable iptables and firewalld.

SELinux

Security Enhanced Linux (SELinux) which is built into RHEL and CentOS provides a flexible Mandatory Access Control (MAC). Under standard Linux Discretionary Access Control (DAC), an application or process running as a user (UID or SUID) has the user’s permissions to objects such as files, sockets, and other processes. Running a MAC kernel protects the system from malicious or flawed applications that can damage or destroy the system.

The default settings for RHEL are usually quite restrictive however RHEL 5 by default SELinux is set to Permissive only which means it only reports violations of policy but does not enforce them. In RHEL 6 and above it is enforced and restrictive. Note that in Amazon Linux it appears SELinux is not installed by default: http://www.chrisumbel.com/article/selinux_amazon_aws_ec2_ami_linux.

To install SELinux:

TBD

To tell if SELinux is running and enforcing:

$ getenforce
Enforcing

Enable SELinux in RHEL 6 & 7:

For Redhat 6 and 7 see - https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Security-Enhanced_Linux/sect-Security-Enhanced_Linux-Enabling_and_Disabling_SELinux-Disabling_SELinux.html

Edit file /etc/selinux/config and change line "SELINUX=" to "enforcing" and SELINUXTYPE to "targeted".

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

This won't take effect until the next reboot so until then to enable it now:

$ sudo setenforce 1
$ sudo getenforce   
Enforcing          

For more on SELinux see http://selinuxproject.org/.

See also Disable SELinux.

SSH access and practise

OpenSSH while secure is prone to many automated attacks and as such should not be available externally without some kind of IP address limiting or control.

AWS has Security Groups to prevent such access however this can also can be controlled within the instance as well by updating /etc/sysconfig/iptables to accept connection only from 192.168.1.0/24 and 202.54.1.5/29, enter:

-A RH-Firewall-1-INPUT -s 192.168.1.0/24 -m state --state NEW -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -s 202.54.1.5/29 -m state --state NEW -p tcp --dport 22 -j ACCEPT

We have a defined set of internal HMH subnets available at HMH internal office subnets.

Brute force is a method of defeating a cryptographic scheme by trying a large number of possibilities using a single or distributed computer network. There are a number of brute force attack scripts that can be run if external unfettered access to SSH is required. We won't go into detail only list them for further research.

fail2ban is probably the most popular of them.

• DenyHosts is a Python based security tool for SSH servers. It is intended to prevent brute force attacks on SSH servers by monitoring invalid login attempts in the authentication log and blocking the originating IP addresses. http://denyhosts.sourceforge.net/
• Fail2ban is a similar program that prevents brute force attacks against SSH. http://www.fail2ban.org/
• BlockHosts Automatic blocking of abusive IP hosts. (This software is now over 10 years old). http://www.aczoom.com/faq/blockhosts

Note also that is a devops trend to not even turn on OpenSSH except to perform troubleshooting. Code can be deployed by a variety of means without using terminal access in repeatable patterns. If software fails and a post-mortem is required it is possible to mount (or take snapshot and redeploy) the root and deployment EBS volumes from the failing system on another instance for evaluation.

SSH settings

/etc/ssh/sshd_config on all servers

• Turn on privilege separation (UsePrivilegeSeparation yes and VerifyReverseMapping yes)
• Prevent the use of insecure home directory and key file permissions (StrictModes yes)
• Only Use SSH Protocol 2. SSH protocol version 1 (SSH-1) has man-in-the-middle attacks problems and security vulnerabilities. SSH-1 is obsolete and should be avoided. (Protocol 2)
• Disable .rhosts Files. Don’t read the user’s ~/.rhosts and ~/.shosts files. (IgnoreRhosts yes)
• Disable root logins (PermitRootLogin no)
• Disable password logins (PasswordAuthentication no)
• Set the idle-timeout (TCPKeepAlive, ClientAliveInterval and ClientAliveCountMax)
• Disable Host-Based Authentication (HostbasedAuthentication no)
• Enable a Warning Banner (Banner /etc/issue)
• Chroot SSHD (Lock Down Users To Their Home Directories). By default users are allowed to browse the server directories such as /etc/, /bin and so on. With the release of OpenSSH 4.8p1 or 4.9p1, you no longer have to rely on third-party tools such as rssh or complicated chroot(1) setups to lock users to their home directories. See more at https://debian-administration.org/article/590/OpenSSH_SFTP_chroot_with_ChrootDirectory. (ChrootDirectory)

These are of lesser risk but included for completeness.

• Disable port forwarding on servers except during software installs (AllowTcpForwarding no and X11Forwarding no)

Passwords and SSH keys

In general best practise is to not use password logins. In RHEL 5 by defaut password logins to SSH are allowed and used. In RHEL 6 and beyond SSH keys are required. Amazon Linux keys are also required for SSH access.

Key storage

TBD

Key rotation

TBD

Root logins

In general best practise is to not login to root directly. In AWS instances by default root logins are usually not permitted. Instead one logins to a non-root account and performs a sudo command to elevate privileges. In addition this non-root logins requires a key supplied by AWS only one time so it must be stored.

Single sign-on and federated logins

TBD

Running privileged commands and preventing privilege escalation

Privilege escalation is the act of exploiting a bug, design flaw or configuration oversight in an operating system or software application to gain elevated access to resources that are normally protected from an application or user.

Shell command should always be run using the sudo command as opposed to changing to the root user and running the commands directly. The root use should only be used for system rescues.

Services should be run in a chroot jail when possible.

Stack exploits

Stack smashing, stack canaries and Stack-Smashing Protection (SSP).

This is mostly the C compiler's job but the kernel can detect and prevent some forms of stack pollution.

See also http://7h3ram.github.io/2012/exploit-mitigation-techniques-on-linux.html, https://en.wikibooks.org/wiki/Grsecurity and https://en.wikipedia.org/wiki/PaX

Enable Address Space Layout Randomization (ASLR):

Set the /proc/sys/kernel/randomize_va_space to 2. This should be the default in RHEL/CentOS 5 and above. See also https://securityetalii.es/2013/02/03/how-effective-is-aslr-on-linux-systems/

sudo echo "2" > /proc/sys/kernel/randomize_va_space

You can also use the sysctl command:

sudo sysctl -w kernel.randomize_va_space = 2

Reload the sysctl config into the kernel using command:

sudo sysctl -p

To survive reboots by adding following line to /etc/sysctl.conf file:

kernel.randomize_va_space = 2

Buffer overflow

Enable ExecShield buffer overflows protection.

ExecShield is security Linux kernel patch to avoid worms and other problems and has been present and enabled in the Linux kernel for a 15 years now.

Exec Shield is a project that got started at Red Hat, Inc in late 2002 with the aim of reducing the risk of worm or other automated remote attacks on Linux systems. The first result of the project was a security patch for the Linux kernel that adds an NX bit to x86 CPUs. While the Exec Shield project has had many other components, some people refer to this first patch as Exec Shield.

To enable ExecShield protection:

sudo sysctl -w kernel.exec-shield = 1

Reload the sysctl config into the kernel using command:

sudo sysctl -p

To survive reboots by adding following line to /etc/sysctl.conf file:

kernel.exec-shield = 1

Network attacks

Review running and open ports

sudo netstat -atulpn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 127.0.0.1:199               0.0.0.0:*                   LISTEN      1379/snmpd          
tcp        0      0 0.0.0.0:873                 0.0.0.0:*                   LISTEN      1873/rsync          
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      1612/mysqld         
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1189/rpcbind        
tcp        0      0 0.0.0.0:36146               0.0.0.0:*                   LISTEN      1211/rpc.statd      
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1462/sshd           
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      12251/master        
tcp        0     52 10.33.14.81:22              155.44.96.39:62761          ESTABLISHED 27519/sshd  
...

Remove any ports and services that are not required. Some services are controlled by xinetd rather than service command.

More on xinetd - TBD

TCP SYN Cookie Protection

Turn On TCP SYN Cookie Protection:

sudo sysctl -w net.ipv4.tcp_syncookies = 1

Reload the sysctl config into the kernel using command:

sudo sysctl -p

To survive reboots by adding following line to /etc/sysctl.conf file:

net.ipv4.tcp_syncookies = 1

IP Spoofing

Red Hat Enterprise Linux 6 and above invalidate and discard packets when the route for outbound traffic differs from the route of incoming traffic to help prevent spoofing. RHEL6's (and RHEL7's) default setting is more strict than RHEL5's, as RHEL6 follows the Strict Reverse Path Forwarding filtering recommended in RFC 3704 - Ingress Filtering for Multihomed Networks. See notes at https://access.redhat.com/solutions/53031.

Check with Network Engineering before changing the default.

For RHEL 6 and 7 use:

sudo sysctl -w net.ipv4.conf.default.rp_filter = 2
sudo sysctl -w net.ipv4.conf.all.rp_filter = 2

Reload the sysctl config into the kernel using command:

sudo sysctl -p

To survive reboots by adding following line to /etc/sysctl.conf file:

net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2

For RHEL 5:

The sysctl net.ipv4.conf.default.rp_filter selects the default RPF filtering setting for IPv4 networking. (It can be overridden per network interface through net.ipv4.conf.INTERFACE.rp_filter).

RHEL7, RHEL6, and RHEL5 ship with a default /etc/sysctl.conf that sets this sysctl to 1, but the meaning of this value is different between the RHEL6 and the RHEL5 kernel.

In RHEL5, the sysctl is documented as follows:

/usr/share/doc/kernel-doc-2.6.18/Documentation/networking/ip-sysctl.txt

rp_filter - BOOLEAN

1 - do source validation by reversed path, as specified in RFC1812
       Recommended option for single homed hosts and stub network
       routers. Could cause troubles for complicated (not loop free)
       networks running a slow unreliable protocol (sort of RIP),
       or using static routes.

0 - No source validation.

       conf/all/rp_filter must also be set to TRUE to do source validation on the interface

       Default value is 0. Note that some distributions enable it in startup scripts.

In RHEL6 and RHEL7 there are three possible values for this setting:

/usr/share/doc/kernel-doc-2.6.32/Documentation/networking/ip-sysctl.txt

rp_filter - INTEGER
        0 - No source validation.
        1 - Strict mode as defined in RFC3704 Strict Reverse Path 
            Each incoming packet is tested against the FIB and if the interface
            is not the best reverse path the packet check will fail.
            By default failed packets are discarded.
        2 - Loose mode as defined in RFC3704 Loose Reverse Path 
            Each incoming packet's source address is also tested against the FIB
            and if the source address is not reachable via any interface
            the packet check will fail.

        Current recommended practice in RFC3704 is to enable strict mode 
        to prevent IP spoofing from DDos attacks. If using asymmetric routing
        or other complicated routing, then loose mode is recommended.

        The max value from conf/{all,interface}/rp_filter is used 
        when doing source validation on the {interface}.

        Default value is 0. Note that some distributions enable it
        in startup scripts.

with the value 2 corresponding to the behaviour that value 1 provided in RHEL5.

Misc

Add following line in “/etc/sysctl.conf” file to ignore ping or broadcast request.

# Ignore ICMP request:
net.ipv4.icmp_echo_ignore_all = 1

# Ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts=1

net.ipv4.icmp_ignore_bogus_error_messages=1

# Make sure spoofed packets get logged
net.ipv4.conf.all.log_martians = 1

Load new settings or changes, by running following command

sudo sysctl -p

Detection

Intrusion Detection Systems

WIP / TBD

Logwatch and logcheck

Monitor Suspicious Log Messages With Logwatch / Logcheck

Read your logs using logwatch or logcheck. These tools make your log reading life easier. You get detailed reporting on unusual items in syslog via email. A sample syslog report:

 ################### Logwatch 7.3 (03/24/06) ####################
        Processing Initiated: Fri Oct 30 04:02:03 2009
        Date Range Processed: yesterday
                              ( 2009-Oct-29 )
                              Period is day.
      Detail Level of Output: 0
              Type of Output: unformatted
           Logfiles for Host: www-52.nixcraft.net.in
  ##################################################################
 --------------------- Named Begin ------------------------
 **Unmatched Entries**
    general: info: zone XXXXXX.com/IN: Transfer started.: 3 Time(s)
    general: info: zone XXXXXX.com/IN: refresh: retry limit for master ttttttttttttttttttt#53 exceeded (source ::#0): 3 Time(s)
    general: info: zone XXXXXX.com/IN: Transfer started.: 4 Time(s)
    general: info: zone XXXXXX.com/IN: refresh: retry limit for master ttttttttttttttttttt#53 exceeded (source ::#0): 4 Time(s)
 ---------------------- Named End -------------------------
  --------------------- iptables firewall Begin ------------------------
 Logged 87 packets on interface eth0
   From 58.y.xxx.ww - 1 packet to tcp(8080)
   From 59.www.zzz.yyy - 1 packet to tcp(22)
   From 60.32.nnn.yyy - 2 packets to tcp(45633)
   From 222.xxx.ttt.zz - 5 packets to tcp(8000,8080,8800)
 ---------------------- iptables firewall End -------------------------
 --------------------- SSHD Begin ------------------------
 Users logging in through sshd:
    root:
       123.xxx.ttt.zzz: 6 times
 ---------------------- SSHD End -------------------------
 --------------------- Disk Space Begin ------------------------
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/sda3             450G  185G  241G  44% /
 /dev/sda1              99M   35M   60M  37% /boot
 ---------------------- Disk Space End -------------------------
 ###################### Logwatch End #########################

Rootkits

A rootkit is a program (or combination of several programs) designed to take fundamental control (“root” access) of an operating system, without authorization by the system’s owners Most rootkits use the power of the kernel to hide themselves, they are only visible from within the kernel.

Chkrootkit

chkrootkit is a tool to locally check for signs of a rootkit.

Type the following command to install chkrootkit

sudo yum install chkrootkit

Start looking for rootkits, enter:

$ sudo chkrootkit

Look for suspicious strings, enter:

$ sudo chkrootkit -x | less

You need to specify the path for the external commands used by chkrootkit such as awk, grep and others. Mount /mnt/safe using nfs in read-only mode and set /mnt/safe binaries PATH as trusted one, enter:

$ sudo chkrootkit -p /mnt/safe

rkhunter

rkhunter (Rootkit Hunter) is a Unix-based tool that scans for rootkits, backdoors and possible local exploits. rkhunter is a shell script which carries out various checks on the local system to try and detect known rootkits and malware. It also performs checks to see if commands have been modified, if the system startup files have been modified, and various checks on the network interfaces, including checks for listening applications.

Type the following command to install rkhunter:

sudo yum install rkhunter
</pre

The following command option tells rkhunter to perform various checks on the local system:

<pre>
sudo rkhunter --check

The following command option causes rkhunter to check if there is a later version of any of its text data files:

$ sudo rkhunter --update

The following option tells rkhunter which directories to look in to find the various commands it requires:

$ sudo rkhunter --check --bindir /mnt/safe

Zeppoo

Zeppoo allows you to detect rootkits on i386 and x86_64 architecture under Linux, by using /dev/kmem and /dev/mem. Moreover it can also detect hidden tasks, connections, corrupted symbols, system calls and so many other things. eppo is not included in the redhat base repos but can be downloaded source at https://sourceforge.net/projects/zeppoo/

Linux Host-based IDS applications

Host-based intrusion detection (HIDS) applications

Tripwire

WIP

Once a free and open source project it is now a commercial product.

AIDE

http://aide.sourceforge.net/

samhain

http://www.la-samhna.de/samhain/

Osiris now SNARE

Once an open and free project now a commercial product.

https://www.intersectalliance.com/our-product/

Logging

Best practise is to configure logging and auditing to collect all attack vector attempts. This is also useful to find out software misconfiguration which may open your system to various attacks.

By default syslog stores data in /var/log/ directory.

rsyslog vs syslog-ng

Linux kernel by default uses syslog. This is a very minimal implementation and a better process bases system logger should be used. There are two options: rsyslog and syslog-ng.

rsyslog is a very basic but capable system logger.

syslog-ng is very robust and is recommended however it requires a more complex configuration.

WIP TBD

Linux log files and locations

TBD / WIP

• /var/log/messages - The default location for most log messages out of the box.
• /var/log/syslog -
• /var/log/boot.log - A log of boot up messages
• /var/log/cloud-init.log or /var/log/cloud-init-output.log - Output produced by cloud-init process in Redhat from AWS.
• /var/log/auth.log – Authentication logs.
• /var/log/kern.log – Kernel logs.
• /var/log/cron.log – Crond logs (cron job).
• /var/log/maillog – Mail server logs.
• /var/log/mysqld.log – MySQL database server log file.
• /var/log/secure – Authentication log (login failures go here).
• /var/log/utmp or /var/log/wtmp : Login records file. This is a binary file and not readable in a text editor. Use commands who, last and lastb to access them.
• /var/log/yum.log: Yum log files.

How to send logs to a remote loghost

Central syslog host for analysis and archival.

TBD

Log rotation

TBD

dmesg

TBD

Auditing

System Accounting with auditd

The auditd is provided for system auditing.

It is responsible for writing audit records to the disk.

During startup, the rules in /etc/audit.rules are read by this daemon. You can open /etc/audit.rules file and make changes such as setup audit file log location and other option. With auditd you can answers the following questions:

• System startup and shutdown events (reboot / halt).
• Date and time of the event.
• User responsible for the event (such as trying to access /path/to/topsecret.dat file).
• Type of event (edit, access, delete, write, update file & commands).
• Success or failure of the event.
• Records events that Modify date and time.
• Find out who made changes to modify the system’s network settings.
• Record events that modify user/group information.
• See who made changes to a file etc.

Monitor user activities

GNU acct tools

There are two useful tools that come as a part of the GNU acct tools-set called ‘psacct‘ and ‘acct‘. They are used for monitoring user activities and processes on a system.

For documentation the GNU acct tools see http://www.gnu.org/software/acct/manual/accounting.html

See also http://binpipe.blogspot.com/2013/04/psacct-in-linux-user-process-history.html

sudo yum install psacct

accton - turns process accounting on or off

sudo /sbin/accton

ac

ac - print statistics about users’ connect time

Some examples of using ac

sudo ac --individual-totals 
        thorner                             12.11
        ec2-user                            27.70
        jobs                                96.08
        kinscoe                            378.79
        nveeder                            183.54
        jdeprin                            390.98
        sysoper                            176.06
        total     1265.25

Print usage per day.

sudo ac -d -p
...
Jul  6  total        5.22
        kinscoe                              3.47
        jdeprin                              9.37
Jul  7  total       12.84
        jdeprin                              0.55
Jul  8  total        0.55
        kinscoe                              8.21
Jul 10  total        8.21
        kinscoe                              2.74
        jdeprin                              2.78
Today   total        5.52

Display per day login time of a particular user. Can be used to spot unusual trends.

ac -d kinscoe
...
Mar 29  total        0.35
Apr 19  total        1.19
May  2  total        0.63
May  9  total        0.11
May 19  total        5.89
May 20  total       18.43
May 23  total        3.15
Jul  7  total        3.47
Jul 10  total        8.21
Today   total        2.87

Last used commands

sudo lastcomm
locate                  root     pts/1      0.30 secs Mon Jul 11 12:24
httpd             SF    apache   __         0.02 secs Mon Jul 11 12:23
httpd             SF    apache   __         1.58 secs Mon Jul 11 11:27
httpd             SF    apache   __         0.26 secs Mon Jul 11 12:23
httpd             SF    apache   __         0.36 secs Mon Jul 11 12:23
bash               F    root     pts/1      0.00 secs Mon Jul 11 12:24
sudo              S     root     pts/1      0.00 secs Mon Jul 11 12:23
chkconfig         S     root     pts/1      0.08 secs Mon Jul 11 12:23
sudo              S     root     pts/1      0.00 secs Mon Jul 11 12:22
service           S     root     pts/1      0.00 secs Mon Jul 11 12:22
psacct                  root     pts/1      0.00 secs Mon Jul 11 12:22
touch                   root     pts/1      0.00 secs Mon Jul 11 12:22
accton            S     root     pts/1      0.00 secs Mon Jul 11 12:22

sa

The "sa" command is used to print the summary of commands that were executed by users.

To enable this collection:

sudo chkconfig psacct on
sudo service psacct start
Starting process accounting:                               [  OK  ]

 sudo sa  
      13       3.73re       0.04cp    37971k
       4       3.71re       0.04cp    64064k   httpd*
       7       0.01re       0.01cp    23038k   ***other*
       2       0.00re       0.00cp    38048k   sudo

utmp, wtmp and btmp files

Login and login failure binary log files are stored /var/log/utmp, /var/log/wtmp and /var/log/btmp. These are binary files and not readable in a text editor.

Use commands who, last and lastb to access them.

last - Show login history.

sudo last | more
kinscoe  pts/1        155.44.96.39     Mon Jul 11 09:45   still logged in   
jdeprin  pts/0        10.82.66.237     Mon Jul 11 09:43   still logged in   
kinscoe  pts/0        155.44.96.39     Sun Jul 10 12:50 - 21:02  (08:12)    
jdeprin  pts/0        10.82.66.67      Fri Jul  8 11:59 - 12:33  (00:33)    
jdeprin  pts/0        10.82.66.73      Thu Jul  7 18:50 - 22:01  (03:11)    
kinscoe  pts/1        155.44.97.17     Thu Jul  7 11:24 - 14:52  (03:28)    
jdeprin  pts/0        10.82.66.73      Thu Jul  7 11:16 - 17:27  (06:11)    
nveeder  pts/1        10.92.17.171     Wed Jul  6 12:51 - 12:52  (00:01)    
jdeprin  pts/0        10.82.67.11      Wed Jul  6 11:03 - 16:14  (05:11)    
jdeprin  pts/0        10.82.66.35      Sun Jul  3 12:57 - 16:08  (03:11)    
jdeprin  pts/1        10.82.66.35      Sat Jul  2 13:14 - 16:25  (03:11)    
jdeprin  pts/0        10.82.66.35      Sat Jul  2 13:12 - 17:23  (04:11)    
...
jdeprin  pts/0        155.44.96.241    Tue Oct  6 14:20 - 14:21  (00:00)    
jdeprin  pts/2        155.44.96.241    Tue Oct  6 13:27 - 16:38  (03:11)    
sysoper  pts/1        155.44.96.241    Tue Oct  6 13:05 - 18:16  (05:11)    
jdeprin  pts/1        155.44.96.241    Tue Oct  6 12:48 - 12:48  (00:00)    
sysoper  pts/0        155.44.96.241    Tue Oct  6 10:41 - 13:52  (03:11)    
kinscoe  pts/2        10.82.66.232     Mon Oct  5 13:20 - 13:21  (00:01)    
kinscoe  pts/1        10.82.66.232     Mon Oct  5 13:18 - 13:21  (00:03)    
kinscoe  pts/0        10.82.66.232     Mon Oct  5 13:16 - 13:22  (00:05)    
kinscoe  pts/0        10.82.66.169     Fri Oct  2 12:51 - 15:47  (02:55)    
sysoper  pts/0        155.44.96.214    Thu Oct  1 15:10 - 18:21  (03:11)    

wtmp begins Thu Oct  1 15:10:15 2015

who - who is logged in?

 who
kinscoe  pts/1        2016-07-11 09:45 (155.44.96.39)

Encryption

AWS does provide encrypted EBS volumes. Linux can encrypt volumes and files using commands like OpenSSL, GnuPG, dm-crypt (and the now non-maintained TrueCrypt) however using encryption at the Amazon level would be preferred for performance and also secure (key) reasons.

Network traffic on Linux can be encrypted by

• IPSec (https://www.strongswan.org/)
• VPN (https://openvpn.net/)
• Point 2 Point tunneling (http://www.linuxfoundation.org/collaborate/workgroups/networking/tunneling)
• SSH tunneling (openssh.org)

OpenVPN and IPSec are the preferred method of encrypting IP traffic.

Some examples:

IPsec - https://mescanef.net/blog/2014/05/encrypted-network-traffic-between-two-linux-hosts-the-ipsec-way/