Deep diving into F5 Secure Vault

Published on Tue 04 June 2024 by myst404 (@myst404_)

In our article Post-Exploiting an F5 BIG-IP: root, and now what?, there was a brief about the Secure Vault and how we can decrypt encrypted values.
This article gives more details about the internal mechanisms of the F5 Secure Vault.

NOTE: All tests depicted below were performed on F5 BIGIP-17.1.1.2-0.0.10 LTM installed from the official VMWare image for this version (sha256: aff4610bdb4f4067c7305a63a7a27f24c47dbf99e55fbed79eed3de3796b5fd6).

Table of contents

  1. Introduction
  2. Unit key
  3. Master key
  4. Security considerations
  5. F5 response

1. Introduction

In F5 BIG-IP systems, sensitive data like secrets are stored in the Secure Vault. According to the documentation and whitepaper:

The Secure Vault feature provides an additional layer of security to the BIG-IP system by encrypting passwords and passphrases, which the system stores in its configuration files.
Secure Vault, a super-secure SSL-encrypted storage system introduced in BIG-IP version 9.4.5, allows passphrases to be stored in an encrypted form on the file system.

Passwords and passphrases that the Secure Vault encrypts include RADIUS secrets or imported SSL key passphrases for example.

How are these secrets encrypted?

2. Unit key

Unit key generation

According to the documentation the unit key is:

Used to encrypt and decrypt the master key. Because a unit key protects the master key, unit keys must be safely stored, usually in electrically erasable programmable read-only memory (EEPROM), on hardware platforms that include EEPROM.

The unit key is a 256-bit key used to encrypt the master key using AES-256.
For non-compatible systems (i.e. BIG-IP Virtual Edition (VE) systems or any system without EEPROM), the unit key is stored in the file /config/bigip/kstore/.unitkey. The remainder of this article assumes we work on a BIG-IP Virtual Edition. The unit key file looks like this:

[root@f5-bigip:Active:Standalone] ~ $ hexdump -C /config/bigip/kstore/.unitkey
00000000  da d2 00 20 00 20 00 00  66 4f 48 73 f2 78 00 51  |��. . ..fOHs�x.Q|
00000010  e1 59 16 2c c0 16 7f 55  63 d7 74 a5 18 3d 5a 1c  |�Y.,�..Uc�t�.=Z.|
00000020  24 e9 4e 8a 5e 32 c8 03  66 ea f7 a5 98 2b 3a ea  |$�N.^2�.f���.+:�|
00000030  44 12 38 ff 81 f1 f9 2e  c7 9e 2b 9f 40 34 b2 10  |D.8�.��.�.+.@4�.|
00000040

Strangely, the file is 512-bit long and the beginning does not really look random: 00 20 00 20 00 00.

How is this unit key generated? Reversing the mcpd daemon (used to handle the proprietary MCP protocol) reveals that the unit key is generated at boot using the library libmcpdcommon.so located in the folder /usr/lib/.
The key is generated in the function generate_aes_unit_key(). Here is a summary of its work:

  1. A random source is set up in a subroutine via OpenSSL's function RAND_load_file(), which reads 128 bytes from /dev/urandom.

  2. Another subfunction generates the encryption key by calling OpenSSL's function EVP_BytesToKey() with the following parameters:

    • type: EVP_aes_256_ecb()
    • md: EVP_sha1()
    • salt: NULL
    • data: 32 bytes generated via the cryptographically secure pseudo random generator RAND_bytes()
    • data1: the number of bytes in data, i.e. 32
    • count: the number of iterations, 3

  3. The unit key is then written to the file /config/bigip/kstore/.unitkey via the halwrite_unitkey() function. This function write 512 bits, divided as follow:

    • 64 bits: a static value: dad2002000200000
    • 32 bits: the generation timestamp: 664f4873 (Thursday, May 23rd, 2024, 13:45:23)
    • 160 bits: the SHA-1 hash of the AES-256 unit key. $ echo -n 24e94e8a5e32c80366eaf7a5982b3aea441238ff81f1f92ec79e2b9f4034b210 | xxd -r -p | shasum -b outputs f2780051e159162cc0167f5563d774a5183d5a1c
    • 256 bits: the AES-256 unit key: 24e94e8a5e32c80366eaf7a5982b3aea441238ff81f1f92ec79e2b9f4034b210

To sum up, here is how the /config/bigip/kstore/.unitkey file is built:

Unit key file details

Getting the unit key

Knowing the construction of the unit key file, we can easily extract the AES-256 unit key:

[root@f5-bigip:Active:Standalone] config $ xxd -c 32 -s 32 -p /config/bigip/kstore/.unitkey
24e94e8a5e32c80366eaf7a5982b3aea441238ff81f1f92ec79e2b9f4034b210

We now have the unit key, what can we do with it?

3. Master key

Getting the master key

According to the documentation:

When the BIG-IP system starts, the Master Control Program (MCP) daemon mcpd process reads and validates configuration files prior to loading them into running memory and performs the following decryption tasks:

  • Retrieves the unit key from its storage location.

  • Decrypts the master key using the unit key.

The encrypted 128-bit master key is stored in the /config/bigip/kstore/master file:

[root@f5-bigip:Active:Standalone] ~ $ cat /config/bigip/kstore/master 
$S$Qz$99rwWH0+Fwa+/D5EWCYKGB5K9dsaQdmiC5kuXk2vb2s=

The output looks like a hash format, but the last part is actually the encrypted master key. We can decrypt it using the previously obtained unit key:

user@attackervm:~ $ echo '99rwWH0+Fwa+/D5EWCYKGB5K9dsaQdmiC5kuXk2vb2s=' | openssl enc -aes-256-ecb -d -base64 -K '24e94e8a5e32c80366eaf7a5982b3aea441238ff81f1f92ec79e2b9f4034b210'
Qz�&µ?A:z�ҏԭ''�%

Notice the Qz prefix which was part of the string in the master key file: $S$Qz$99rwWH0+Fwa+/D5EWCYKGB5K9dsaQdmiC5kuXk2vb2s=.
This prefix is probably used as a control value. It is randomly generated by taking 2 characters in the following static string: 1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ. The same control check mechanism is used when secrets are encrypted.

So, the only remaining step to obtain the master key is to remove this 2-character prefix:

user@attackervm:~ $ echo '99rwWH0+Fwa+/D5EWCYKGB5K9dsaQdmiC5kuXk2vb2s=' | openssl enc -aes-256-ecb -d -base64 -K '24e94e8a5e32c80366eaf7a5982b3aea441238ff81f1f92ec79e2b9f4034b210' | xxd -p -s 2
c726c2b53f413a7aacd28fd4ad2727b9

Conveniently, F5 provides a utility to directly obtain the cleartext AES-128 master key. We can confirm that we have properly decrypted the master key:

[root@f5-bigip:Active:Standalone] $ f5mku -K | base64 -d | xxd -p
c726c2b53f413a7aacd28fd4ad2727b9

We know have the cleartext master key! We described in our previous article Post-Exploiting an F5 BIG-IP: root, and now what? how to use the master key to decrypt the secrets.

Master key generation

Initial master key generation

We skipped on purpose an important cryptographic step: how is this master key generated?

When the F5 BIG-IP is installed, a master key is generated via the function generate_master_key() in the library libmcpdcommon.so.
This function calls OpenSSL's EVP_BytesToKey() function used with the following parameters:

  • type: EVP_aes_128_ecb()
  • md: EVP_sha1()
  • salt: NULL
  • data: 16 bytes generated via the cryptographically secure pseudo random generator RAND_bytes()
  • data1: the number of bytes in data, i.e. 16
  • count: the number of iterations, 3

The generation is nearly the same as for the unit key, the only difference is that here a 128-bit key (instead of 256) is generated.

Resetting the master key

According to F5:

F5 recommends that for any new BIG-IP system, you reset the master key as part of the initial system setup. You can perform the following task to reset the master key on a vCMP guest.

Indeed, knowing your master key is useful for migrations, backups or BIG-IP systems in clusters for example.
The tmsh utility is recommended to change the master key, we will use Almond_OffSec as password for this example:

[root@f5-bigip:Active:Standalone] ~ $ f5mku -K
xybCtT9BOnqs0o/UrScnuQ==
[root@f5-bigip:Active:Standalone] ~ $ tmsh modify /sys crypto master-key prompt-for-password
enter password: 
password again: 
[root@f5-bigip:Active:Standalone] ~ $ f5mku -K
byT70hBShpil6V1T9o8+Ow==

We have validated that after the tmsh command, the master key has changed. In this case, the function transform_key() is called.
This function calls OpenSSL's EVP_BytesToKey() function with the following parameters:

  • type: EVP_aes_128_ecb()
  • md: EVP_sha1()
  • salt: NULL
  • data: the password set by the user
  • data1: the number of bytes in data
  • count: the number of iterations, 3

Basically, this applies the SHA-1 function 3 times to the password set. We can confirm this key generation algorithm:

user@attackervm:~ $ echo -n 'Almond_OffSec' | openssl dgst -binary -sha1 | openssl dgst -binary -sha1 | openssl dgst -binary -sha1 | base64
byT70hBShpil6V1T9o8+O6WQnC0=

4. Security considerations

While no major issues were discovered in this analysis, some security weaknesses have been found in the "super-secure SSL-encrypted storage system".

ECB

The AES cipher mode ECB is used:

  • In AES-256 to encrypt the master key with the unit key
  • In AES-128 to encrypt the secrets with the master key

In practice however, encrypted secrets are mostly short, so this issue is unlikely to induce a big risk.

Quick example with a long secret with identical characters encrypted:

[root@f5-bigip:Active:Standalone] ~ $ tmsh modify /auth radius-server myServer secret AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
[root@f5-bigip:Active:Standalone] ~ $ tmsh list /auth radius-server
auth radius-server myServer {
    secret $M$tW$yX2LdKtsyv3kNlnI3IavyY5BuocmMYNcSOSBhgFoGHSOQbqHJjGDXEjkgYYBaBh0jkG6hyYxg1xI5IGGAWgYdI5BuocmMYNcSOSBhgFoGHR2kxKG5S8/Njdo+29MjFzo
    server 10.0.13.37
}

Notice the repetition of the pattern 5BuocmMYNcSOSBhgFoGH. The first block is always different because of the 2-character randomly generated prefix.

Key generation

The caveat here is that the master key derivation function is really weak. Using only 3 rounds of SHA-1 to derive a key is a bad practice for 2 reasons:

  • SHA-1 is a deprecated algorithm.
  • 3 rounds are not enough, as SHA-1 is very fast to compute (~ 50GH/s on a Nvidia RTX 4090). For example, OWASP recommends at least 600.00 rounds of HMAC-SHA-256 to store passwords.

F5 recommends to use a "strong" password:

  • 10 or more characters
  • One or more capital letters
  • One or more lowercase letters
  • One or more numbers
  • One or more special, non-null characters

However, no password policy can be configured, you can enter a weak (or even empty) password.

We can imagine two attack scenarios:

  • An attacker steals the hard disk of a virtual appliance. The attacker can decrypt the master key thanks to the unit key stored on the disk. It is then possible to decrypt the secrets. This behavior is by design and there is no real countermeasure because F5 products do not have a whole disk encryption option.
  • An attacker steals the hard disk of a physical appliance. The unit key is stored in the EEPROM, it may be difficult to obtain it. However, the encrypted master key is stored on the disk. It is possible to try decrypting the master key by bruteforce, knowing the key generation algorithm.

F5 response

Below the response as is from the F5 Security Incident Response Team (SIRT):

F5 considers ECB and SHA-1 to be secure given their implementation in BIG-IP.

Since crypted content in ECB mode is generally short and is not human-language text, repeating blocks of ciphertext are not expected to be present, thereby mitigating risk arising from the use of AES ECB.

Brute-forcing the hash to determine the master key and subsequently decrypting SecureVault content requires an attacker to 1) be a highly privileged user or 2) obtain an off-line copy of the disk image or physical disk.

F5 appreciates the opportunity to work with 3rd party researchers and is committed to delivering ongoing security improvements to our customers. BIG-IP Next, F5’s next-generation products, does not use SecureVault and are therefore not impacted in any way.