Operational Defect Database
BugZero found this defect 4336 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
Veeam | kb1653
Unable to collect files information from datastore in a timely manner due to high vCenter server load
Last update date:
3/29/2021
Affected products:
Veeam ONE
Affected releases:
8.0
Fixed releases:
No fixed releases provided.
Description:
Challenge
Veeam ONE Reporter collects data from datastores using “SearchDatastoreSubFolders_Task” tasks that are pre-defined by VMware. Some of the datastores might become unavailable or experience performance issues. Subsequently, Veeam ONE Reporter may be unable to complete data collection tasks against these datastores.The following error messages appear in the SchedulingService.log file:XX/XX/XXXX XX:XX:XX [info] : xxxxx.xxxxx.xxxxx: Exporting virtual machine files XX/XX/XXXX XX:XX:XX [warning] : xxxxx.xxxxx.xxxxx: Failed to collect VM files: Datastore [**datastore_name**] timeout is exceededUnable to collect files information from datastore [**datastore_name**] in a timely manner due to high vCenter server load....Unable to collect files information from datastore [XXXXXX] in a timely manner due to high vCenter server load.The collection job in Veeam ONE Reporter will finish with a "Warning" status. Additionally, the session log may contain the following entry: XX/XX/XXXX XX:XX:XX [warning]: xxxxx.xxxxx.xxxxx: There were errors while collecting VM files: Failed in retrieve some VM files because of the timeout is exceeded.
Cause
Veeam ONE Reporter uses a 900 secondtimeout in order to wait for 10 datastore search tasks (default number) to complete.Any performance or access issues affect the entire collection process, and the collection job finishes with a "Warning" status. The error messages above mean that some issues have occurred on datastores. Due to lack of response, we’ve failed datastore scan tasks and are unable to collect data from some of the datastores. Basically, the problem is either on the datastore’s or on VMware’s side, but we may try to find a workaround for this issue and apply changes to the collection process on Veeam ONE Reporter’s side in order to increase the time to wait until all datastore tasks are completed. NOTE: This workaround just lets you run the “SearchDatastoreSubFolders_Task” task with the new timeout specified to make sure the datastore collection process does complete.
Solution
Here is how to tune the product in order to make your datastore search tasks complete successfuly: 1) Find a suitable timeout value for completing all the “SearchDatastoreSubFolders_Task” tasks. 2) Apply registry changes and set a proper timeout value.STEP 1:Find a suitable timeout value for completing all the “SearchDatastoreSubFolders_Task” tasks in each particular scenario by running the PowerShell script (available below in "More Information" section) using VMware vSphere PowerCLI. This shows how much time it takes to complete all mentioned datastore tasks. You can download VMware vSphere PowerCLI package for free by the following link:http://www.vmware.com/support/developer/PowerCLI/index.html Instructions on how to run the script: 1) Open PowerCLI and connect to your VMware vCenter server using the "connect-viserver" command. 2) Specify the credentials to connect to VMware vCenter server. 3) Change the current directory to the directory where the script is located (use "cd" command). 4) Run the script and wait until it's finished. The following is an example of the output in the same VMware vSphere PowerCLI window. The script will run the task against every datastore that is registered under the VMware vCenter server. Error messages such as the following might appear: Trying to collect *.vmdk and *.flat files from **datastore_name** ( **datastore_id** ) Exception calling "SearchDatastoreSubFolders" with "2" argument(s): "Datastore' [**datastore_name**]' is not accessible." At {path_to_the_script}:\Veeam_BrowseDatastores.ps1:45 char:41 These errors mean that there are some problems with the datastores. But from the output, you should see how much time it takes to complete scanning all datastores. STEP 2:Increase the timeout Veeam ONE Reporter uses to complete the entire datastore search process: 1) Log on to the server with Veeam ONE Reporter installed. Open Registry Editor and navigate to the Veeam ONE Reporter registry folder: For both x86bit and x64bit systems - HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Reporter Enterprise 2) Create the following registry key: Name – GetVmSnapshotTimeout Type – DWORD Value – XXXXX *. * Equal to or more than time it takes to complete scanning all datastores according to script's output. The value is calculated in seconds and should be specified in DECIMAL format.
More Information
Here is the script mentioned above: write-host (Get-Date).TimeOfDay Getting datastores... $dss = get-datastore write-host (Get-Date).TimeOfDay $dss.Count datastores found $fqf = New-Object VMware.Vim.FileQueryFlags $fqf.FileSize = $true $fqf.FileType = $true $fqf.Modification = $true $fqf.FileOwner = $true $fq1 = New-Object VMware.Vim.FileQuery $ss1 = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec $ss1.details = $fqf $ss1.sortFoldersFirst = $true $ss1.MatchPattern += "*.*" $ss2 = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec $ss2.details = $fqf $ss2.sortFoldersFirst = $true $ss2.MatchPattern += "*delta.vmdk" $ss2.MatchPattern += "*flat.vmdk" $ss1.Query = $fq1 foreach($ds in $dss) { $dsv = $ds | get-view if($ds.Accessible -ne $true) { write-host (Get-Date).TimeOfDay Skip files collecting because $dsv.Summary.Name"("$dsv.Summary.Datastore.Value")" is not accessible. } else { $dsb = get-view $dsv.browser $rp = "["+$dsv.summary.Name+"]" write-host (Get-Date).TimeOfDay Trying to collect *.vmdk and *.flat files from $dsv.Summary.Name"("$dsv.Summary.Datastore.Value")" $sr2 = $dsb.SearchDatastoreSubFolders($rp, $ss2) write-host (Get-Date).TimeOfDay Completed. $sr2.Count files found on $dsv.Summary.Name"("$dsv.Summary.Datastore.Value")" write-host (Get-Date).TimeOfDay Trying to collect other files from $dsv.Summary.Name"("$dsv.Summary.Datastore.Value")" $sr1 = $dsb.SearchDatastoreSubFolders($rp, $ss1) write-host (Get-Date).TimeOfDay Completed. $sr1.Count files found on $dsv.Summary.Name"("$dsv.Summary.Datastore.Value")" } }
