Top security measures for ESXi environments

Businesses globally utilize the VMware ESXi hypervisor for virtualization. ESXi is labeled as a type-1 (or “bare metal”) hypervisor, residing directly on the hardware instead of atop an operating system like Windows. Many enterprises host critical servers on one or more ESXi hosts, all under the management of vCenter Server (VMware’s platform for overseeing such environments and their components).

Regrettably for defenders, ESXi hosts do not inherently support EDR (endpoint detection and response). While certain events on these hosts can be sent to a SIEM if logging is enabled, this workaround is far from ideal for various reasons. Numerous small and mid-size businesses lack both a SIEM and the necessary personnel to effectively monitor and respond to SIEM logs and alerts. Attackers have exploited this security gap, particularly in countless ransomware incidents over the years.

The Sophos Managed Risk team frequently addresses concerns about insecure host setups and offers guidance on how to rectify them. While nothing beats detailed discussions with real people, we have compiled a list of the top ten recommended practices for this article. Where appropriate, we detail and provide links to the latest instructions typically maintained by VMware (Broadcom) itself. In some instances, we share insights or tips acquired through our own experience with such remediations.

What’s the allure of ESXi?

Why are ESXi hosts so attractive to attackers? Speed, efficiency, and ESXi’s substantial market share play key roles.

Typically, with insecure host setups, attackers can avoid dealing with the exploits that EDR would typically flag. In other words, by targeting the host, attackers can bypass several layers of protection and swiftly cause significant damage to an organization — encrypting an entire ESXi host and the VMs it hosts with just one click. For certain organizations, even solely encrypting the ESXi infrastructure could result in havoc and ransom demands. While Sophos X-Ops’ Incident Response team has explored ways to extract data from encrypted virtual disks, it’s best to never reach such a predicament.

Fortunately, defenders can take steps to thwart an attack on ESXi. At the very least, these measures can impede attackers (offering defenders more time to detect and respond) and potentially halt attacks on ESXi altogether. This article outlines ten strategies, accompanied by relevant source materials and additional information where necessary. These tactics are not arranged in any specific order:

1. Ensure that vCenter and ESXi hosts run supported versions and are fully patched
2. Contemplate refraining from joining vCenter and ESXi hosts to the domain
3. Activate standard lockdown mode
4. Turn off SSH when not in use
5. Impose password complexity for vCenter and ESXi hosts
6. Mandate account lockout after failed login attempts
7. Enable UEFI Secure Boot
8. Set up the host to exclusively run binaries distributed through signed VIB
9. Disable Managed Object Browser (MOB), CIM, SLP, and SNMP services if unused
10. Establish persistent logging

This guide primarily focuses on ESXi (as opposed to vSphere) to designate the target host-plus-management-center configuration.

Where feasible, this guide outlines how to implement the recommendations for environments managed by vCenter and those that aren’t.

Ensure vCenter Server and ESXi hosts run supported versions and are fully patched

How this enhances security

Ensuring that all vCenter Servers and ESXi hosts run supported versions of their software and receive regular patches minimizes the attack surface related to known vulnerabilities addressed by patches.

Steps to follow

It’s advisable to start by updating vCenter Server (if an update is available) before moving on to update the ESXi hosts. Ensuring that the management layer updates are complete before commencing host updates is crucial.

As of early August 2024, only vCenter Server / ESXi versions 7.0 and 8.0 are officially supported. The support for 7.0 will end on April 2, 2025, with no further updates scheduled. If you haven’t transitioned to 8.0 yet, plan and execute your upgrades before April 2025. VMware strongly advises having the vCenter Server on the same or a higher version of the ESXi Host build number to avoid complications. Managing upgrades for unsupported versions becomes urgent and complex. Upgrading vCenter Server appliances prior to versions 6.7 requires an intermediary upgrade to version 6.7 or 7.0 before moving to version 8.0.

The process for patching is relatively straightforward compared to version upgrades. The patching process is conducted through the vCenter Server Management portal; detailed instructions are available on the VMware Docs site. It’s recommended to back up vCenter Server before applying any updates or patches.

To upgrade and patch ESXi hosts connected to vCenter, leverage the vSphere Lifecycle Manager. VMware has a comprehensive video guide outlining this multi-step process, making it simpler to follow visually than step-by-step instructions.

To patch a standalone ESXi host through the web client, access the host via SSH (Secure Shell protocol). We will delve into proper SSH practices in a later section. Perform the following steps:

1. Choose Host > Actions > Enter maintenance mode
2. Expand Actions, select Services > Enable Secure Shell (SSH)
3. Access the host through SSH
4. Execute the command to check the current updates and VIBs installed:
   esxcli
5. Obtain the software profile information
6. To permit web traffic through the firewall, execute the subsequent directive:
   esxcli network firewall ruleset set -e true -r httpClient
7. Discover the available online update packages and filter by your version at the end for a quicker response:
   esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep -i ESXi-7
8. Select the desired package for installation (preferably the latest one) and include the package name in this command:
   esxcli software profile update -p PACKAGE-NAME -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
9. Upon completion of the update, restart the host
10. Double-check the installation success by executing this command again:
    esxcli software profile get
11. If the installation was successful, deactivate web traffic through the firewall with this command:
    esxcli network firewall ruleset set -e false -r httpClient

Temporary Solutions to Mitigate Risk

Ensuring that your software is up-to-date and fully patched is always the primary objective. However, circumstances may arise where upgrading to newer software versions necessitates hardware upgrades. In cases where immediate upgrading is not feasible due to timing or budget constraints, consider segregating the management functions of ESXi hosts onto a separate network, specifically dedicated to ESXi management. This can be implemented using VLANs, tagging, or a combination of physical networking equipment. The primary aim here is to minimize the network exposure of the host until an upgrade is possible. Remember, this should not serve as a permanent alternative to upgrading.

Evaluate the Option of Avoiding Domain Joining for vCenter and ESXi Hosts

Benefits of this Approach

Similar to the advice of keeping systems updated, maintaining secure passwords is crucial for ESXi and vCenter security. If unauthorized access leads to the compromise of privileged domain credentials, attackers may exploit this to target vCenter or ESXi hosts for lateral movements or data encryption. By isolating vCenter and ESXi hosts from the organizational domain, this attack surface is reduced significantly, especially coupled with unique, complex passwords.

Recent advisories from Microsoft highlighted vulnerabilities that granted full administrative access to the ESXi hypervisor by default to accounts in the ESX Admins AD group. These vulnerabilities further emphasize the benefits of refraining from domain joining vCenter and ESXi hosts.

Implementation Guidelines

Implementing strong password management necessitates using separate credentials for vCenter/ESXi administration. These credentials should be distinct from domain passwords and should be managed efficiently, preferably using a corporate password manager like 1Password or Keeper. Corporate password managers offer advantages such as security log auditing, password complexity enforcement, and the potential for multifactor authentication (MFA). It is advisable to assign individual named accounts to ESXi administrators rather than sharing generic accounts like “root” or “administrator”. Within vCenter, roles like Operator, Administrator, and Super Administrator provide varying levels of permissions, with Super Administrator being the default for the root user in ESXi. Root accounts should only be used for specific tasks, like adding hosts to vCenter or managing local accounts.

To view a list of all local user accounts in vCenter, access the vCenter appliance shell with Super Administrator privileges and execute:

• localaccounts.user.list

To add an admin account, use the following command, followed by entering the password:

• localaccounts.user.add –role admin –username test –password

For adding an admin account with full name and email:

• localaccounts.user.add –role admin –username test –password –fullname TestName –email test@mail.com

To update a user’s password:

• localaccounts.user.password.update –username test –password

Additional Recommendations

While native MFA support for vCenter access by local accounts is lacking, enterprises can implement strategies to enhance security. Utilizing robust passwords, setting up isolated networks for ESXi management, and employing MFA-enabled remote access solutions are viable approaches to bolster security. By restricting access to management portals, ensuring only specific users have access, and implementing MFA, security risks can be mitigated effectively.

Enable Standard Lockdown Mode

Advantages of this Action

Enabling standard lockdown mode enforces management of ESXi hosts via vCenter Server, thereby controlling access and roles to mitigate unauthorized activities. In this mode, access to hosts is restricted, and users are required to have the Administrator role on the host to access it from the ESXi Shell or SSH (this control is not available for standalone ESXi hosts).

While some operations like backups and troubleshooting may be impacted by standard lockdown mode, temporary deactivation is an option, provided reactivation occurs post-operation completion.

specified task is a standard procedure.

Steps to perform

For an ESXi host, using the vSphere Web Client:

1. Opt for the host
2. Click on Configure, then expand System and choose Security Profile > Lockdown Mode> Edit
3. Click on the Normal radio button

To connect to the ESXi host, and from a PowerCLI command prompt, execute the commands provided below. (These commands are displayed in the list format below but can all be inputted simultaneously. If you intend to copy and paste from this article, ensure you exclude the bullets.)

• $level = “lockdownNormal”
• $vmhost = Get-VMHost -Name | Get-View
• $lockdown = Get-View $vmhost.ConfigManager.HostAccessManager
• $lockdown.ChangeLockdownMode($level)

Deactivate SSH when not actively using it

Importance of this action

Occasionally, there is a need to utilize SSH while working with vCenter Servers and ESXi hosts – such as during patching, as mentioned earlier. Nonetheless, disabling SSH when not in use reduces vulnerability by eliminating a potential target for brute force attacks or the use of compromised credentials.

Steps to execute

For vCenter, refer to the instructions on the linked page and ensure the Enable SSH login radio button remains unselected.

For an ESXi host, via the vSphere Web Client:

1. Choose the host
2. Select Configure > System > Services
3. Proceed with > SSH > Edit Startup Policy
4. Ensure the Startup Policy is configured to Start and Stop Manually
5. Click OK
6. While ESXi Shell is still selected, click Stop

Alternatively, utilize the subsequent PowerCLI command (taking note of the bullet):

• Get-VMHost | Get-VMHostService | Where { $_.key -eq “TSM-SSH” } | Set-VMHostService -Policy Off

For a standalone ESXi host through the web client:

1. Access Manage > Services > TSM-SSH > Actions
2. Click on “Stop”
3. Choose Actions again, then Policy > Start and stop manually

Impose password complexity requirements for vCenter alongside ESXi hosts

Importance of this security measure

Utilizing complex passwords helps in thwarting brute force attacks. Attackers frequently employ publicly available password lists or create custom lists based on information about your organization obtained before or during an attack. Enforcing the rejection of non-complex passwords on vCenter and ESXi hosts supports password policy enforcement. Additionally, as mentioned earlier, a password manager can significantly aid in this endeavor, thereby enhancing security and auditability.

Steps to carry it out

The enforcement of password complexity is overseen through the Security.PasswordQualityControl parameter. This control allows for the regulation of permissible password length, character set requirements, and failed logon attempt restrictions.

The recommended setting according to CIS benchmarks is

retry=3 min=disabled,15,15,15,15 max=64 similar=deny passphrase=3

ESXi utilizes the pam_passwdqc module for password management, which is outlined elsewhere. By referring to that manual, we can swiftly decipher the functionality of each component in the aforementioned CIS recommendation:

• Using “retry=3,” users are prompted up to three times if the new password is deemed insufficiently strong or if the password is entered incorrectly twice
• Regarding the “min” component:
•      The “disabled” setting prohibits passwords comprising characters from a single character class only (the four character classes being digits, lowercase letters, uppercase letters, and other characters)
•      The initial 15 signifies the minimum length for a password consisting of two character classes
•      The subsequent 15 denotes the minimum length required for a passphrase
•      The final two 15s indicate the mandatory lengths for passwords with three and four character classes, respectively
• The “max=64” component specifies the maximum password length
• The “similar=deny” component disallows a password that closely resembles the previous one. (Passwords are deemed similar when there exists a lengthy common substring between them, and removing this substring would result in a weakened password; e.g., password123 and password124)
• The “passphrase” switch dictates the minimum number of words (in this case, three) needed to establish a passphrase; this requirement is in addition to the character length obligation specified earlier

For an ESXi host, using the vSphere Web Client:

1. Opt for the host > Configure > System > Advanced System Settings
2. Select the Security.PasswordQualityControl value and set it, as presented above, to “retry=3 min=disabled,15,15,15,15 max=64 similar=deny passphrase=3” (adjust the values to align with your organization’s standards if necessary)

For a standalone ESXi host via the web client:

1. Choose Manage > System > Advanced settings
2. Locate Security.PasswordQualityControl through scrolling or searching
3. Select the Edit option
4. Set the value to “retry=3 min=disabled,15,15,15,15 max=64 similar=deny passphrase=3” (adjust the values to adhere to your organization’s policies)
5. Click on Save

Regarding vCenter setup, the CIS benchmark does not explicitly delve into vCenter password regulations. Nevertheless, based on our interpretation of the CIS benchmark recommendation components, certain sections can be partially applied to fortify vCenter password configurations.

1. In the vSphere Client, select Administration from the hamburger menu
2. Under Single Sign On, access Configuration
3. Choose Local Accounts > Password Policy > Edit
4. Configure the Maximum lifetime number according to your organization’s password lifetime policy
5. Adjust Restrict reuse based on your organization’s policy regarding password reuse
6. Set Maximum length to 64, as per the aforementioned settings
7. Set Minimum length to 15, in line with the previously mentioned values
8. For Character requirements, establish the “At least” value to comply with your organization’s directives; with the minimum being 1
9. Configure “Identical adjacent characters” in adherence to your organization’s policy on password-adjacent characters

Mandatory account lockout after unsuccessful login attempts

Importance of this measure

Implementing account lockouts also acts as a deterrent against brute force attacks. While, theoretically, an attacker can attempt a brute force attack, they would require remarkable luck to succeed within just five attempts before being locked out. This control applies to vCenter, SSH, and vSphere Web Services SDK access but not for the Direct Console Interface (DCUI) and the ESXi Shell.

Steps to perform

The recommended CIS configuration is to set hosts to have the Security.AccountLockFailures parameter configured at 5. This control can also be instituted on vCenter.

For vCenter specifically:

1. Log in with root
2. Choose Administration > Single Sign-on > Configuration > Local Accounts > Lockout Policy
3. Establish the maximum number of failed attempts to 5

For an ESXi host, using the vSphere Web Client:

1. Select the host
2. Select Configure > System > Advanced System Settings
3. Set the Security.AccountLockFailures value to 5

While connected to the ESXi host from a PowerCLI command prompt, run the subsequent command (exercise caution when copying and pasting):

• Get-VMHost | Get-AdvancedSetting -Name Security.AccountLockFailures | Set-AdvancedSetting -Value 5

For

Adjust settings of a single ESXi host using the web interface:

1. Access Manage > System > Advanced settings
2. Locate Security.AccountLockFailures by scrolling or searching
3. Choose the Edit option
4. Set the value to 5
5. Click on the “Save” button

Activation of UEFI Secure Boot

Advantages of this Action

UEFI Secure Boot primarily ensures that only approved and validated boot loaders and operating system kernels can run during system startup. By validating the digital signatures of bootable applications and drivers, Secure Boot effectively prevents potentially malicious code from infiltrating the boot process, thereby minimizing the vulnerability of ESXi hosts. This setting is a mandatory prerequisite for the guideline mentioned in the next section, which is “Restrict host to executing only binaries provided through signed VIB.”

How to Implement

To set up UEFI Secure Boot, the target ESXi host must possess a Trusted Platform Module (TPM); older hardware versions might lack TPM. Assuming the presence of TPM in your hardware, follow these steps:

• Access the relevant ESXi host through the ESXi shell
• Verify the current status of secure boot with the given command (note: the “a.” mentioned is part of the list formatting and should be considered when copying)
•      a. esxcli system settings encryption get
•           i. If the value of Require Secure Boot is “true,” no modifications are needed
•           ii. If the value is “false,” proceed with enabling
•           iii. If the value is “none,” activate a TPM in the host’s firmware first and then execute the following command (taking into account the formatting with “1.”):
•                1. esxcli system settings encryption set –mode=TPM
• Enable the secure boot environment
•      a. Conduct a graceful shutdown of the host
•           i. Right-click on the ESXi host within the vSphere Client and opt for Power > Shut Down
•      b. Turn on secure boot in the host’s firmware
•           i. The process varies depending on the hardware used for ESXi hosts; consult the hardware documentation from your specific vendor
• Restart the host
• Execute the following ESXCLI command (considering the formatting with “a.”):
•      a. esxcli system settings encryption set –require-secure-boot=T
• Validate the change (considering the formatting with “a.”):
•      a. esxcli system settings encryption get
•           i. If Require Secure Boot reads “true,” you have completed the task
• To save the configured setting, run the following command (considering the formatting with “a.”):
•      a. /bin/backup.sh 0

Restrict host to executing only binaries provided through signed VIB

Benefits of this Action

To reinforce the integrity of the system, administrators can set an ESXi host to exclusively execute binaries originating from a legitimate, signed vSphere Installable Bundle (VIB). This measure thwarts potential intrusions by attackers utilizing prebuilt toolkits on the host. Activation of this setting necessitates UEFI Secure Boot to be enabled.

How to Proceed 

The controlling parameter for this behavior is VMkernel.Boot.execInstalledOnly, which needs to be set to True.

For an ESXi host, through the vSphere Web Client:

1. Select the host
2. Opt for Configure > System > Advanced System Settings
3. Identify the value “VMkernel.Boot.execInstalledOnly” and adjust it to True

For a stand-alone ESXi host via the web client

1. Proceed to Manage > System > Advanced settings
2. Search or scroll for VMkernel.Boot.execInstalledOnly
3. Select the Edit choice
4. Set the value to True
5. Save the changes

Disable Managed Object Browser (MOB), CIM, SLP, and SNMP services if unutilized

Significance of this Action

Disabling all externally accessible services not in use by your organization is crucial for shrinking the potential attack area; specifically, effective management of these four services is essential.

• The Managed Object Browser (MOB) is a web-based server tool for inspecting and altering system objects and settings
• The Common Information Model (CIM) setup offers an interface for hardware-level management through standardized API access from remote applications
• The Service Location Protocol (SLP) assists in discovering and choosing network services in local networks; facilitating administration by enabling automatic service discovery for computers. The associated service is named SLPD (Service Level Protocol Daemon), detailed in the steps below
• The traditional Simple Network Management Protocol (SNMP) supports the management of networked devices

Execution Instructions

For an ESXi host, via the vSphere web client:

1. Choose the host
2. Navigate to Configure > System > Advanced System Settings
3. Select the option for “Config.HostAgent.plugins.solo.enableMob” and set it to False
4. Access Configure > System > Services > CIM Server > Modify Startup Policy
5. Adjust the Startup Policy to Manual Start and Stop
6. If the CIM Server service is active, halt it
7. Click on SLPD > Edit Startup Policy
8. Set the Startup Policy to Manual Start and Stop
9. If the SLPD service is running, stop it
10. Opt for SNMP Server > Edit Startup Policy
11. Set the Startup Policy to Manual Start and Stop
12. If the SNMP Server service is operational, cease it

For a stand-alone ESXi host via the web client:

1. Proceed to Manage > System > Advanced settings
2. Lookup or scroll for Config.HostAgent.plugins.solo.enableMob
3. Select Edit and alter the value to False
4. Save the changes
5. Access Services > SLPD > Perform Actions
6. Click “Stop”
7. Perform Actions once more, choose Policy
8. Choose Manual start and stop from the menu
9. Select sfcbd-watchdog (CIM server) and Execute Actions
10. Click “Stop”
11. Carry out Actions > Policy once again
12. Opt for Manual start and stop from the menu
13. Select snmpd and click Actions
14. Click “Stop”
15. Choose Actions > Policy again
16. Select Manual start and stop from the options

Establishing Persistent Logging

Purpose of this Action

Configuration of persistent logging stands as the sole recommendation on this list that doesn’t directly shrink the attack surface. Nevertheless, it becomes invaluable in case of a security breach impacting ESXi hosts. By default, ESXi logs are stored in an in-memory filesystem that retains logs for only a day. As these logs are stored in memory, they are lost upon a restart. While having a persistent local log is a notable upgrade, forwarding the logs to a remote syslog aggregator is even better. In unfortunate scenarios where ESXi hosts and attached drives are encrypted, with a syslog collector in place, chances are higher for retaining access to logs, at least partially. Another benefit of transferring logs out of the host is the potential integration into a SIEM, allowing for ingestion of ESXi logs there as well.

Implementation Steps 

Begin by setting up a persistent location. After this step:

For an ESXi host, via vSphere Web Client:

1. Select the host
2. Go to Configure > System > Advanced System Settings
3. Locate the value Syslog.global.logDir and change it to the designated storage location for logs

For a stand-alone ESXi host via the web client:

1. Proceed to Manage > System > Advanced settings
2. Search or scroll for Syslog.global.logDir
3. Click Edit and modify the value to the designated log storage location
4. Save the changes

Next, specify a target syslog collector for ESXi logs and enable outbound syslog traffic through the ESXi host firewall.

For an ESXi host, via vSphere Web Client:

1. Select the host
2. Access the Configure tab
3. Navigate to Logging > Actions > Modify Settings
4. Within Host Syslog Configuration, choose the option to Send log data to a remote syslog server
5. Set the value to the address associated with your syslog server
6. Click OK
7. Still on the Configure tab for the host, expand System and select Firewall
8. Locate the syslog outbound rule and activate it

For a stand-alone ESXi host via the web client:

1. Proceed to Manage > System > Advanced settings
2. Search or scroll for Syslog.global.logHost
3. Click Edit
4. Adjust the value to the address linked with your syslog server
5. Save the changes
6. In the left sidebar navigator, select Networking > Firewall rules
7. Opt for the syslog rule and choose Actions
8. Activate it

Conclusion

While implementing the recommendations outlined in this article doesn’t ensure absolute safety for your ESXi hosts, it significantly raises the barrier for attackers seeking to cause swift and severe damage. Additionally, employing multiple controls adds complexity for potential attackers, increasing their time and effort investment – a deterrent that possibly thwarts their intentions of exploiting ESXi vulnerabilities. Simultaneously, it provides defenders a larger detection window and more response alternatives for enhanced security.

About Author