Operational Defect Database
BugZero found this defect 4553 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
Veeam | kb1377
VSS wait timeout
Last update date:
12/29/2021
Affected products:
Veeam Backup & Replication
Affected releases:
11
Fixed releases:
No fixed releases provided.
Description:
Challenge
Either of the following errors occurs in the job statistics: "VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec." "VSSControl: Failed to freeze guest, wait timeout"
Cause
These errors look similar but their causes are unrelated. "VSSControl: Failed to prepare guest for freeze, wait timeout 900 sec" Refers to VSS preparation, which consists largely of enumerating VSS writers, their components, and associated files and volumes. The timeout may be exceeded because of performance limitations of the VM guest, or because a specific writer is enumerating a very large number of files. This timeout is configurable. "VSSControl: Failed to freeze guest, wait timeout" Refers to the limit imposed by Microsoft VSS writers on the duration of a freeze. This timeout is not configurable. Veeam uses VSS to freeze applications immediately prior to creating the VMware snapshot, and then sends the thaw command as soon as snapshot creation is complete. VSS will only hold a freeze on the writers for up to 60 seconds (20 for Exchange), so several steps must fit within this timeframe: Verification of freeze state1 Snapshot creation request via VIM API2 Snapshot creation on the ESXi host Return of snapshot information via VIM API2 Thaw request to Microsoft VSS1 Thawing of VSS writers’ I/O 1 If a network connection to the guest OS is not available, VIX API will be used, which introduces additional latency.2 These steps should usually be near-instantaneous, but if the vCenter is heavily loaded or has a high latency to the ESXi hosts, the delay may be significant.
Solution
Review the relevant section below: Note: Improving in-guest performance will help in resolving either error. Anti-Virus software may also impede VSS operations. As an isolation step, consider temporarily uninstalling the Anti-Virus, rebooting, and rerunning the Veeam job. A complete uninstall is necessary as simply disabling some anti-virus solutions may still leave components active. (AV Exclusions) Failed to prepare guest for freeze, wait timeout 900 sec. You can change this timeout by modifying the value in the registry of the server where Veeam Backup & Replication is installed as follows: Go to registry key HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication Add a DWORD (32-bit) value named VssPreparationTimeout. The value is in milliseconds (decimal), the default timeout is 900000, which equals 15 minutes. Set the value to 1800000. This will change the timeout to 30 minutes. Stop any running jobs or wait for them to complete, then restart the Veeam Backup Service. Notes: Certain configurations (such as MS SQL Server with FILESTREAM) or third-party VSS writers may require significant time for preparation. The timeout can be extended to several hours. On Exchange servers, this error typically indicates a large number of transaction logs. You can increase the timeout or truncate the logs by another method. Compare the dates on the transaction logs to the date of the last successful Veeam backup to verify transaction logs are being truncated correctly. On SharePoint servers, you may need to run PSCONFIG. For more info see Potential issues after installing SharePoint Foundation 2010 SP1. Unusual search scopes have also sometimes been implicated. Identification of Problem Writer: Collect the logs in KB1789 and provide them when opening a technical support case. Alternatively, use the Diskshadow utility from an administrator command prompt within the VM guest OS. The output file will contain a list of files associated with each writer. If a particular writer has a very large number of associated files, it is probably the culprit. diskshadow /l <path to output file>.txt list writers detailed Failed to freeze guest, wait timeout On Exchange servers, this is a variation on “Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].” See KB1680. If the VM is a Windows vCenter server (non-VCSA) with a local database (local instance of SQL), or a SQL server which hosts a vCenter database, you may be encountering a problem with the automatic vCenter database exclusion. See KB1051. Otherwise, this usually indicates excessive VMware snapshot creation time, an in-guest performance issue, delays related to VIX API, or a combination of factors. The list of possible solutions is identical to KB1680, but the timeout is 60 seconds instead of 20. This often occurs in combination with VSS-related event log errors containing the word “abort”. This error message also sometimes appears when there are other problems with VSS that should generate additional logging. Collect the logs in KB1789 and provide them when opening a technical support case.
More Information
Both errors are related to Application Aware Image Processing. As a temporary workaround, you can disable AAIP in order to obtain crash-consistent backups. For more information, consult the user guide overview of Transaction Consistency.Overview of Processing a Backup Under VSS has a visual explanation of the freeze timeout (the green box).
