Operational Defect Database
BugZero updated this defect 38 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 94343
vSphere UI reports intermittent crash with OutOfMemoryError due to PatchManager.ScanV2 tasks getting queued up
Last update date:
4/11/2024
Affected products:
vSphere
Affected releases:
7.0
Fixed releases:
No fixed releases provided.
Description:
Symptoms
vSphere Client becomes inaccessible intermittently and requires service restart to recover Accessing vCenter service in ELM (linked mode) might return the error "Could not connect to one or more vCenter Server systems:https://<vcenter FQDN>:443/sdk" /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log [2023-09-18T13:08:40.010Z] [ERROR] VLSI-I/O dispatcher-1 com.vmware.vise.util.concurrent.WorkerThread VLSI-I/O dispatcher-1 terminated with exception: java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.http.nio.util.HeapByteBufferAllocator.allocate(HeapByteBufferAllocator.java:52) at org.apache.http.nio.util.ExpandableBuffer.expandCapacity(ExpandableBuffer.java:108) at org.apache.http.nio.util.ExpandableBuffer.expand(ExpandableBuffer.java:121) at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:69) at org.apache.http.nio.protocol.BasicAsyncResponseConsumer.onContentReceived(BasicAsyncResponseConsumer.java:82) at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.consumeContent(AbstractAsyncResponseConsumer.java:147) at com.vmware.vim.vmomi.client.http.impl.InetAddressSniffingResponseConsumer.consumeContent(InetAddressSniffingResponseConsumer.java:54) at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)[2023-09-18T14:02:19.789Z] [ERROR] linkedVcGroup-pool-2053 70004081 100125 200019 com.vmware.vise.vim.commons.vcservice.impl.LinkedVcGroupImpl Failed to connect to VC https://<vcenter FQDN>:443/sdk (ID) java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED at org.apache.http.util.Asserts.check(Asserts.java:46) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123) at com.vmware.vim.vmomi.client.http.impl.HttpAsyncExchange.invokeWithinScope(HttpAsyncExchange.java:83) at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:31) Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Cause
This issue is caused due to "host.PatchManager.ScanV2" task being spawned for each separate ESXi host when user scan for patches for a Cluster, Datacenter or vCenter. The task type "host.PatchManager.ScanV2" generates a XML as result (>2 Mb size)
Resolution
VMware is aware of this issue and working to resolve this in a future release
Workaround
As a workaround, we need to manually set the results of the tasks to null to avoid OutOfMemory error of vSphere UI. To do this, please follow the steps below: Download the attached script prune_PatchManager_Task_Results.sh and upload to the impacted vCenter Server to the /tmp folder. In case of connection failure, refer to Error when uploading files to vCenter Server Appliance using WinSCP Use SSH to connect to the VCSA and enter the Bash shell Change the file permission as executable chmod +x /tmp/prune_PatchManager_Task_Results.sh Run the following command in the terminal ./tmp/prune_PatchManager_Task_Results.sh Restart the vsphere-ui service using the below command service-control --restart vsphere-ui To schedule the same script as Cron Job Save the script to a persistent location on the vCenter Server Run the below command to open the crontab configuration file : crontab -e Append the crontab file with this following command : <Minute> <hour> <day_of_the_month> <month> <day_of_week> /tmp/prune_PatchManager_Task_Results.sh Below example will run the script after every 24 hours of interval : 02 17 * * * /tmp/prune_PatchManager_Task_Results.sh Which will run the script everyday at 17:02 UTC Save the file and exit The changes take effect immediately and the script run at the next scheduled time. Note: The above script performs a copy of the vpx_task table from vCenter server database (VCDB) and post that executes the cleanup of the specific column having the xml data. During this process, the copy of the table will be saved under "/exports/vcdb/" directory on the VCSA.The above file can be cleaned up manually via logging in to the VCSA using ssh or can add the below action as Cron Job. 02 19 * * * find /exports/vcdb/vpx_task*.csv -type f -mtime +1 -exec rm {} \;which will run the cleanup everyday at 19:02 UTC and cleanup file older than 1 day
