Aug 15, 2018 Warning “esx.problem.hyperthreading.unmitigated” after installing ESXi patches. This warning may appear after installing patches contained in release ESXi601 (14 Aug 2018) and you have not updated your vCenter Server. L1TF related “esx.problem.hyperthreading.unmitigated” vCenter Server Updates: CVE-2018-3646 (57374) Symptoms After applying the ESXi patches in VMSA-2018-0020 or patches from later releases, vCenter Server displays one of these L1 Terminal Fault related notifications.
- Vmware Warning Esx.problem.hyperthreading.unmitigated
- Vmware Vsphere Esx.problem.hyperthreading.unmitigated
- Vmware Hyperthreading Unmitigated Command
- Vmware 6.5 Esxi.problem.hyperthreading.unmitigated
- Vmware Hyperthreading Mitigation
This is part 1 of 2 blogs that will cover how hyper-threading impacts virtual SAP sizing and performance. Many virtual SAP deployments leverage INTEL’s hyper-threading (HT) technology. For each processor core that is physically present, the hypervisor sees two logical processors and shares the workload between them when possible. A vCPU can be scheduled on a logical processor on a core while the other logical processor of the core is idle. In this blog this is referred to as one vCPU scheduled per core. Two vCPUs can be scheduled on the two logical processors of the same core. This is referred to as two vCPUs scheduled per core. For more background on vSphere scheduling functionality, please see the whitepaper The CPU Scheduler in VMware vSphere .
I will show three different sizing scenarios.
Scenario 1
The first scenario above shows
- 14 physical cores with HT enabled (28 logical CPUs).
- A virtual machine (VM) with 14 vCPUs.
- vSphere will schedule each vCPU on a logical CPU on a separate dedicated physical core (default behavior). The scheduler prefers a whole idle core, where both logical CPUs of the core are idle, over a partial idle core, where one logical CPU is idle while the other is busy.
- There is spare capacity for more performance as not all the logical CPUs are utilized.
Scenario 2
The scenario above shows:
- A virtual machine with 28 vCPUs.
- vSphere schedules the vCPUs across all the logical CPUs – the two logical CPUs of each physical core are both utilized. This can be achieved a number of ways:
- Setting manual CPU affinity in the VM to force the vCPUs to be scheduled on specific logical CPUs.
- Provisioning number of vCPUs greater than number of cores on the host.
- Deploying a VM with twice the number of vCPUs as cores in a socket and setting the VM level parameter “Numa.PreferHT” to true . All the vCPUs will be scheduled across all the logical CPUs within the socket/NUMA node.
- Utilization of all the logical CPUs in Scenario 2 provides on average 15% boost in SAP performance/transaction throughput compared to scenario 1. In SAP sizing transaction throughput and performance are measured in the metric “SAPS”. So scenario 2 provides about 15% more SAPS than scenario 1.
Scenario 3
This scenario shows:
- 16 physical cores with HT enabled (32 logical CPUs)
- A virtual machine with 16 vCPUs. vSphere will schedule each vCPU on a logical CPU on a separate dedicated physical core (default behavior) – same as Scenario 1.
- The performance/SAPS throughput is approximately the same as Scenario 2 (based on 15% HT benefit).
- As we linearly scale up vCPUs and cores in Scenario 1, adding an extra 15% vCPU (and cores) will provide us equivalent performance to Scenario 2.
- Scaling up vCPUs in Scenario 1 by 15% = 1.15 x 14 ≈ 16 vCPUs (on 16 cores) – this is Scenario 3.
Comparing Scenarios
SAP sizing involves calculations in SAPS. You can see an example at https://blogs.vmware.com/apps/2017/06/awg_s4hana_part1.html#more-2217 . The methodology and example shown here enables you to calculate the number of vCPUs required for business requirements provided in SAPS. You then have the option to design the VMs like Scenario 2 or 3:
- If we need 16 vCPUs on 16 cores (Scenario 3) an alternative configuration with less cores and equivalent SAPS performance is Scenario 2 (28 vCPUs on 14 cores). The calculation is: 16 / 1.15 ≈ 14 i.e.
M = # of cores utilized (either 2 vCPUs or 1 vCPU scheduled per core)
SAPS of [M cores with 1 vCPU per core] = SAPS of [ M/1.15 cores with 2 vCPUs per core]
Vmware Warning Esx.problem.hyperthreading.unmitigated
- If we need 28 vCPUs on 14 cores (Scenario 2) an alternative configuration with equivalent SAPS with less vCPUs but more cores is Scenario 3 (16 vCPUs on 16 cores). The calculation is: 14 x 1.15 ~ 16 i.e.
SAPS of [ M cores with 2 vCPUs per core] = SAPS of [M x 1.15 cores with 1 vCPU per core]
The above equations are estimates as we assume linear scalability of SAPS with vCPUs in all the scenarios and an average HT benefit of 15%.
Conclusion
I have shown above when sizing VMs we have the option to configure the VMs with 1 vCPU scheduled per core or 2 x vCPUs scheduled per core. An equation shows how these options are numerically related. The following table summarizes the difference between the options.
1https://blogs.vmware.com/performance/2017/03/virtual-machine-vcpu-and-vnuma-rightsizing-rules-of-thumb.html
2https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/vmw-vsphere-virtual-saphana-application-workload-guidance-design.pdf
3 https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/sap_hana_on_vmware_vsphere_best_practices_guide-white-paper.pdf
![Vmware hyperthreading unmitigated failed Vmware hyperthreading unmitigated failed](/uploads/1/3/3/9/133919741/814291135.png)
Part 2 of this blog will seek to demonstrate some of the concepts discussed here with an actual SAP workload.
Vmware Vsphere Esx.problem.hyperthreading.unmitigated
18 August 2018Andrea Mauro
Reading Time: 7minutesThe L1 Terminal Fault (aka Foreshadow) bug is another speculative execution side channel attack that affects Intel Core processors and Intel Xeon processors only.
For VMware vSphere, there are some patches available as described in this document: VMSA-2018-0020. All patches have been released on August, 14th 2018.
VMware Product | Product Version | Running On | Severity | Replace_with/Apply_Patch | Build |
---|---|---|---|---|---|
VC | 6.7 | Any | Important | 6.7.0d | Build 9451876 |
VC | 6.5 | Any | Important | 6.5u2c | Build 9451637 |
VC | 6.0 | Any | Important | 6.0u3h | Build 9451619 |
VC | 5.5 | Any | Important | 5.5u3j | Build 9313450 |
ESXi | 6.7 | Any | Important | ESXi670-201808401-BG* ESXi670-201808402-BG** ESXi670-201808403-BG* | Build 9484548 |
ESXi | 6.5 | Any | Important | ESXi650-201808401-BG* ESXi650-201808402-BG** ESXi650-201808403-BG* | Build 9298722 |
ESXi | 6.0 | Any | Important | ESXi600-201808401-BG* ESXi600-201808402-BG** ESXi600-201808403-BG* | Build 9313334 |
ESXi | 5.5 | Any | Important | ESXi550-201808401-BG* ESXi550-201808402-BG** ESXi550-201808403-BG* | Build 9313066 |
*These patches DO NOT mitigate the Concurrent-context attack vector previously described by default. For details on the three-phase vSphere mitigation process please see KB55806and for the mitigation process for Workstation and Fusion please see KB57138.
**These patches include microcode updates required for mitigation of the Sequential-context attack vector. This microcode may also be obtained from your hardware OEM in the form of a BIOS or firmware update. Details on microcode that has been provided by Intel and packaged by VMware is enumerated in the patch KBs found in the Solution section of this document.
Of course, vSphere previous v5.5 are no more supported and there is no solution for them. And also vSphere 5.5 will go out of support soon!
But patching is just a step in the entire flow process:
As described in VMware KB 55806, the mitigation process for CVE-2018-3646 is divided into three phases:
- Update Phase: Apply vSphere Updates and Patches
- Planning Phase: Assess Your Environment
- Scheduler-Enablement Phase: Enable the ESXi Side-Channel-Aware Scheduler
But of course, there is also the phase 0 needed to identify if your system is affected. In this RuneCast can help you (as happened with Spectre and Meltdown).
Then you have to apply the patches and in this case is very important to upgrade the vCenter FIRST and then the ESXi hosts. If you patching ESXi before their vCenter Server there can be a generic error such as
xxx esx.problem.hyperthreading.unmitigated.formatOnHost not found xxx
or esx.problem.hyperthreading.unmitigated
.For vCenter updating, with VCSA this could be managed simply with VAMI. For ESXi, VUM can help you.
Note that the L1TF related patches are (at least in vSphere 6.7) in the Non-Critical Host Patches baseline (sigh).
Note for VSAN users: seems that the VSAN baselines are not updated yet to include also this patch (in my case the VSAN 6.7 baseline remains on build 8169922). Maybe it’s just my problem, but I suggest to temporally add the non-critical patches baseline to proper patch your hosts.
After the host remediation each affected host will have a warning alarm with the proper reference at the issue:
A now? Now you have to fix your host by enabling the ESXi Side-Channel-Aware Scheduler:
Using the vSphere Web Client or vSphere Client:
- Connect to the vCenter Server using either the vSphere Web or vSphere Client.
- Select an ESXi host in the inventory.
- Click the Manage (5.5/6.0) or Configure (6.5/6.7) tab.
- Click the Settings sub-tab.
- Under the System heading, click Advanced System Settings.
- Click in the Filter box and search VMkernel.Boot.hyperthreadingMitigation
- Select the setting by name and click the Edit pencil icon.
- Change the configuration option to true (default: false).
- Click OK.
- Reboot the ESXi host for the configuration change to go into effect.
- Connect to the ESXi host by opening a web browser to https://HOSTNAME.
- Click the Manage tab
- Click the Advanced settings sub-tab
- Click in the Filter box and search VMkernel.Boot.hyperthreadingMitigation
- Select the setting by name and click the Edit pencil icon
- Change the configuration option to true (default: false)
- Click Save.
- Reboot the ESXi host for the configuration change to go into effect.
Using the CLI with the esxcli command:
- SSH to an ESXi host or open a console where the remote ESXCLI is installed. For more information, see the http://www.vmware.com/support/developer/vcli/.
- Check the current runtime value of the HTAware Mitigation Setting by running esxcli system settings kernel list -o hyperthreadingMitigation
- To enable HT Aware Mitigation, run this command: esxcli system settings kernel set -s hyperthreadingMitigation -v TRUE
- Reboot the ESXi host for the configuration change to go into effect.
Why is this remediation not active by default? Good question… the main answer seems to be the possible performance/capacity impact. How huge is performance degradation? Could be around 30% but a correct answer requires to read carefully the VMware KB 55767. The reality is that the degradation depends on the cluster usage level and is directly correlated to it.
But CPU usually is not a bottleneck and security by default must be the first approach… So, in my opinion, was better enable by default and then let the administrator choose if disable it.
Considering also that enable later means that you need a second host reboot, this becomes very time to consume, especially in a hyper-converged cluster where you need to reboot (usually) hosts sequentially one by one.
VMware has provided a tool (basically a PowerCLI script) to assist in performing both the Planning Phase and the Scheduler-Enablement Phase at scale. The tool is called HTAware Mitigation Tool and can be found in KB56931 along with detailed instructions on its usage, capabilities, and limitations.
Basically, you can extract the ZIP file and import the module. But the module is not signed, so the first step is to change the execution policy:
Then, you can import the module:
Connect to the vCenter server:
Start the analysis:
And remediate all the hosts (but first check for a possible performance impact):
The remediation will only apply the advanced setting value, but you need to reboot your host after the process:
If a host has not been patched properly, the script will return a message like this: “ESXi host may not have been patched”.
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Unrestricted
Then, you can import the module:
Import-Module .HTAwareMitigation.psd1
Connect to the vCenter server:
Connect-VIServer -Server vcenter.lab.local
Start the analysis:
Get-HTAwareMitigationAnalysis
And remediate all the hosts (but first check for a possible performance impact):
Set-HTAwareMitigationConfig .output.csv -Enable
The remediation will only apply the advanced setting value, but you need to reboot your host after the process:
Processing ESXi host esxi1.lab.local ...
Unable to find HT-aware mitigation setting, ESXi host may not have been patched ...
Processing ESXi host esxi2.lab.local ...
Successfully updated 'VMkernel.Boot.hyperthreadingMitigation' to True
If a host has not been patched properly, the script will return a message like this: “ESXi host may not have been patched”.
Note that the script does not check if the setting is already present and also does not provide (yet) the reboot orchestration flow.
Vmware Hyperthreading Unmitigated Command
Important: Some vendors or other OSes suggest to disable Intel Hyperthreading in firmware/BIOS (you can also disable via ESXi using VMkernel.Boot.Hyperthreading). But VMware does not recommend this operation because it precludes potential vSphere scheduler enhancements and mitigations that will allow the use of both logical processors. As such, disablement of hyperthreading to mitigate the Concurrent-context attack vector will introduce unnecessary operational overhead as hyperthreading may need to be re-enabled in the future.
See also:
Related Posts
- New patches for VMware vCloud SuiteVMware has recently released several patches for its vCloud Suite: VMware vCenter Server 5.1.0a and modules (Release Notes) VMware vSphere Data Protection 5.1.1 (Release Notes) vSphere Storage Appliance 5.1.1 (Release Notes) VMware vCloud Networking and Security 5.1.1 vCloud Director 5.1.1 Note also that View 5.1.x…
- vSphere 5 downloadFinally the vSphere 5 download is now available on VMware site: http://downloads.vmware.com/d/info/datacenter_cloud_infrastructure/vmware_vsphere/5_0 VMware ESXi 5.0 (Build 469512) VMware vCenter 5.0 (Build 456005) Also the official documents is now available: http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html
- vSphere 5 downloadFinalmente, dopo più di un mese dal Lancio ufficiale di VMware vSphere 5, è disponibile la versione per il download, direttamente dal sito di VMware: http://downloads.vmware.com/d/info/datacenter_cloud_infrastructure/vmware_vsphere/5_0 VMware ESXi 5.0 (Build 469512) VMware vCenter 5.0 (Build 456005) Oltre al download dei binari, sono disponibili anche i…
Andrea Mauro
Vmware 6.5 Esxi.problem.hyperthreading.unmitigated
Virtualization, Cloud and Storage Architect. Tech Field delegate.VMUG IT Co-Founder and board member. VMware VMTN Moderator and vExpert 2010-20 and vExpert Pro. Dell TechCenter Rockstar 2014-15. Microsoft MVP 2014-16. Veeam Vanguard 2015-19. Nutanix NTC 2014-20.Several certifications including: VCDX-DCV, VCP-DCV/DT/Cloud, VCAP-DCA/DCD/CIA/CID/DTA/DTD, MCSA, MCSE, MCITP, CCA, NPP.
Vmware Hyperthreading Mitigation
VMware, vSecurity, vSpherenone