Troubleshooting Ideas of Linux Intrusion Issues

This document will guide you on how to troubleshoot Linux intrusion issues and provide security optimization suggestions after the intrusion.

Notes:

If you have identified the intrusion as a mining script or Trojan, see Mining Trojan Self-Service Cleanup Guide.

Analyzing and Identifying the Cause of Intrusion

1. Checking for Hidden Accounts and Weak Passwords

1. Check for weak passwords in the system and application accounts.
   • Description: Check whether administrator accounts, database accounts, MySQL accounts, Tomcat accounts, or website administrator accounts are using weak passwords. Weak passwords are easy targets for brute force cracking.
   • Solution: Log in to the system or application backend as an administrator and change the password to a strong password.
   • Risk level: High.
2. Execute the last command to check recent login records for suspicious accounts or IP addresses.
   • Description: Attackers or malicious software often inject hidden system accounts into the system to escalate privileges or carry out other malicious activities.
   • Solution: If suspicious users are found, execute the usermod -L username command to disable the user or the userdel -r username command to delete the user.
   • Risk level: High.
3. Execute the command less /var/log/secure|grep 'Accepted' to check for suspicious IP addresses that logged in to the server.
   • Description: Attackers or malicious software often inject hidden system accounts into the system to escalate privileges or carry out other malicious activities.
   • Solution: If suspicious users are found, execute the usermod -L <username> command to disable the user or the userdel -r <username> command to delete the user.
   • Risk level: High.
4. Check whether the system uses the default management port.

   • Check whether the management ports used by the system, such as SSH, FTP, MySQL, and Redis®, are default ports. These default ports are often easily exploited by automated tools through brute force cracking.

   • Solution:
     a. You can edit port 22 in the /etc/ssh/sshd_config file, change it to a non-default port, and restart the ssh service.

     Notes:

     After modifying the port, update the inbound rules of the CVM’s security group in the CVM console to allow traffic on the new port.

     b. You can execute the /etc/init.d/sshd restart (CentOS) or /etc/init.d/ssh restart (Debian / Ubuntu) command to restart the system for the configuration to take effect.
     c. Modify the default listening ports (such as 21 for FTP, 3306 for MySQL, and 6379 for Redis®) in the respective configuration files.
     d. You can restrict the IP logged in remotely by editing /etc/hosts.deny and /etc/hosts.allow files.

   • Risk level: High.

5. Check the /etc/passwd file for unauthorized accounts.
   • Description: Attackers or malicious software often inject hidden system accounts into the system to escalate privileges or carry out other malicious activities.
   • Solution: Execute the usermod -L username command to disable the user or the userdel -r username to delete the user.
   • Risk level: Medium.

2. Checking for Malicious Processes and Unauthorized Ports

1. Execute the netstat -antp command to check whether unauthorized ports are being listened on, and identify the corresponding PID.
   • Description: Check whether the server has any unauthorized processes listening on ports. Malicious processes often open listening ports to communicate with external control servers.
   • Solution: If unauthorized processes are found, execute ls -l /proc/$PID/exe or file /proc/$PID/exe (where $PID is the corresponding PID) to check the file path of the process.
     If it is a malicious process, you can delete the corresponding file.
   • Risk level: High.
2. Execute the ps -ef and top commands to check for abnormal processes.
   • Description: Execute the above commands. If you detect unauthorized processes that change their name continuously and occupy excessive system CPU or memory resources, they may be malicious programs.
   • Solution: After confirming the process is malicious, terminate the process with kill -9 PID or block the external connections of the process through a firewall.
   • Risk level: High.

3. Checking for Malicious Programs and Suspicious Startup Items

1. Execute the chkconfig --list and cat /etc/rc.local commands to check for abnormal startup services.
   • Description: Attackers often configure malicious programs in the startup configuration for automatic startup.
   • Solution: If malicious processes are found, execute the chkconfig service_name off command to disable them or check for abnormal entries in /etc/rc.local and comment out any suspicious lines.
   • Risk level: High.
2. Check the cron directory for illegal scheduled tasks or scripts.
   • Description: Check the content of /etc/crontab, /etc/cron.d, /etc/cron.daily, cron.hourly/, cron.monthly, and cron.weekly/ for suspicious scripts or programs.
   • Solution: If you identify an unknown scheduled task, check whether it is part of the normal workload operations. If not, you may comment out the task or remove the corresponding script.
   • Risk level: High.

4. Checking for Vulnerabilities in Third-Party Software

1. If your server runs applications such as web services or databases, restrict application accounts’ write permissions to the file system, and run applications as a non-root user.
   • Description: Running applications as a non-root user ensures that even if the application is compromised, attackers cannot immediately gain remote control of the server, thus minimizing damage.
   • Solution: Go to the root directory of the web service or the database configuration directory.
     You can run the chown -R apache:apache /var/www/xxxx and chmod -R 750 file1.txt command to configure website access permissions.
   • Risk level: Medium.
   • For more information, see the example of directory file permissions for a website.
2. Upgrade and patch application vulnerabilities
   • One common cause of server intrusions is the use of outdated application software versions that contain unpatched vulnerabilities, which attackers can exploit.
   • Solution: Common vulnerabilities include ImageMagick, OpenSSL, and glibc. You can follow security advisories or execute apt-get/yum to upgrade and patch the software.
   • Risk level: High.

**Example of directory file permissions for a website:**Scenario:
Assume the HTTP server runs under the user and user group www, the website user is centos, and the website root directory is /home/centos/web.
Methods/Steps:

1. Set the website directory and file owner and group to centos and www, and execute the following command:

 chown -R centos:www /home/centos/web

2. Set the website directory permissions to 750, which indicates that the centos user has read, write, and execute permissions, the group has read and execute permissions, and other users have no permissions.

find -type d -exec chmod 750 {} \;

3. Set the website file permissions to 640, which indicates that only the centos user can modify the files, the HTTP server can read the files but cannot modify them, and other users have no permissions.

find -not -type d -exec chmod 640 {} \;

4. Grant write permissions to specific directories as needed. For example, cache directories or the /data/ directory of Discuz X2 require write access for the HTTP server.

find data -type d -exec chmod 770 {} \;

Security Optimization Recommendations After an Intrusion

• It is recommended to use SSH keys for login to reduce the risk of brute force cracking.
• Change the SSH port by editing /etc/ssh/sshd_config and setting Port 22 to another non-default port. After modification, execute the following command to restart the SSH service:

restart /etc/init.d/sshd (CentOS) or /etc/init.d/ssh (Debian/Ubuntu)

Notes:

After modifying the port, update the inbound rules of the CVM’s security group in the CVM console to allow traffic on the new port.

• If SSH passwords are required for management, set a strong password.
• Set complex, unique passwords for all application management backends (such as websites, middleware, and Tomcat), remote SSH, remote desktop, and databases.
• Example of strong passwords (spaces are allowed):
  1qtwo-threeMiles3c45jia and
  caser, lanqiu streets.
• Examples of weak passwords, which are often based on common phrases or personal information:
  The company name and date, such as coca-cola2016xxxx, and
  common phrases, such as Iamagoodboy.
• Execute the following command to check open ports and close non-workload ports on your host:

netstat -antp

• Use security group firewall rules or edit /etc/hosts.deny and /etc/hosts.allow to allow access only from specified IP addresses.
• Applications should avoid using root permissions whenever possible.
• Programs such as Apache, Redis®, MySQL, and Nginx should avoid running with root permissions whenever possible.
• Fix privilege escalation vulnerabilities in the system and programs running as a root user to prevent malware from gaining root access and installing backdoors.
• Update systems and applications regularly, including programs such as Struts2, Nginx, ImageMagick, and Java.
• Disable remote management features of applications such as Redis® and NTP. If remote management is not required, disable external listening ports or related configurations.
• Regularly back up workload data from CVM.
• Perform cross-region or cloud backups for important workload data to ensure recovery after an intrusion.
• In addition to backing up /home and /root directories, you should also back up /etc and /var/log, which are crucial for forensic analysis.
• Install a CWP agent to identify risks in the event of an attack.

  Notes:

  If the preceding steps do not resolve the issue, it is recommended to contact the Ops personnel for support.