Veeam Backup Fails: VSS Writer Errror 0x800423f4 (Azure AD Connect)

An updated post is here 31/10/2017

Hi Guys.

An interesting issue over the last few days. Our backup logs have had the following failures in Veeam.

Unable to release guest. Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Digging through the event logs, you will see errors like the following:

  1. Inital backup
    SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=19908. Thread=18488. Client. Instance=LOCALDB#SH7A2278. VD=Global\{C7140958-2759-4979-BA55-0E3F258064ED}1_SQLVDIMemoryName_0.

    vpxclient_2016-09-08_13-09-55

  2. Followed by
    A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,
    the error is likely to reoccur.
    . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. 
    
    Operation:
     PrepareForSnapshot Event
    
    Context:
     Execution Context: Writer
     Writer Class Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
     Writer Name: SqlServerWriter
     Writer Instance Name: SQL Server Code-Named 'Denali' CTP2:SQLWriter
     Writer Instance ID: {2f97c809-8eb4-431c-93ac-b0f81e610013}
     Command Line: "C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe"
     Process ID: 19908

    mstsc_2016-09-08_13-17-47

    mstsc_2016-09-08_13-18-02

We have found this issue is related to an update of the Microsoft Azure AD Connect client.

On all servers that were affected by this, we had just upgraded to the latest Azure AD Connect client. This seems to corrupt the SQL writer somehow.

We haven’t seen many reports of this. There are various fixes for the 0x800423f4 error, but the easiest fix is to repair the LocalDB SQL instance.

To do this:

  1. Go to Add/Remove Programs
  2. Select Microsoft SQL Server 2012 Express LocalDB
  3. Select Repair
    vpxclient_2016-09-08_13-11-17

Once this is finished, you will need to reboot.

We had this issue with many servers. This process fixed it on everyone.

Disable Windows Update Services (WSUS) – Small Business Server and Doman Controllers

Windows Update Services, or WSUS can be a real pain at the best of times. On Small Business Servers, which are often neglected, some strange issues can arise.

The most common issue is that WSUS goes crazy and downloads everything. Yes, this does happen.

You may not know, but you shouldn’t uninstall WSUS on a domain controller, IE Small Business Server.

Here is how you can disable it and clean up the mess.

  1. Under Options, select Products and Classifications 
  2. Unselect everything, and then select Bing Bar. You need something selected.
    2015-11-13_12-22-02
  3. Under Classifications, select Critical Updates
    2015-11-13_12-21-47
  4. Remove Automatic Approvals
    2015-11-13_12-23-29
  5. Run the Server Cleanup Wizard with everything selected
  6. Once this is done, you can leave WSUS as is, or you can disable the service
    2015-11-13_12-25-46
  7. With the service stopped and disabled, you can also delete the content database. Only do this if you are never going to use WSUS again.
    2015-11-13_12-27-03

Hopefully this has helped someone.

Installing Disk Cleanup (cleanmgr.exe) Server 2008 and Server 2012

It’s annoying that Disk Cleanup (Cleanmgr.exe) is no longer installed by default. You can install Disk Cleanup by installing the Windows Desktop Experience – but, there is another, less annoying way…

Server 2012

copy C:WindowsWinSxSamd64_microsoft-windows-cleanmgr_31bf3856ad364e35_6.2.9200.16384_none_c60dddc5e750072acleanmgr.exe %systemroot%System32

copy C:WindowsWinSxSamd64_microsoft-windows-cleanmgr.resources_31bf3856ad364e35_6.2.9200.16384_en-us_b6a01752226afbb3cleanmgr.exe.mui %systemroot%System32en-US

Server 2008 R2 (x64)

copy C:Windowswinsxsamd64_microsoft-windows-cleanmgr.resources_31bf3856ad364e35_6.1.7600.16385_en-us_b9cb6194b257cc63cleanmgr.exe.mui %systemroot%System32En-US

copy C:Windowswinsxsamd64_microsoft-windows-cleanmgr_31bf3856ad364e35_6.1.7600.16385_none_c9392808773cd7dacleanmgr.exe %systemroot%System32

Server 2008 (x64)

copy C:Windowswinsxsamd64_microsoft-windows-cleanmgr.resources_31bf3856ad364e35_6.0.6001.18000_en-us_b9f50b71510436f2cleanmgr.exe.mui %systemroot%system32en-us

copy C:Windowswinsxsamd64_microsoft-windows-cleanmgr_31bf3856ad364e35_6.0.6001.18000_none_c962d1e515e94269cleanmgr.exe %systemroot%System32

Server 2008 (x32)

copy C:Windowswinsxsx86_microsoft-windows-cleanmgr.resources_31bf3856ad364e35_6.0.6001.18000_en-us_5dd66fed98a6c5bccleanmgr.exe.mui %systemroot%System32En-US

copy C:Windowswinsxsx86_microsoft-windows-cleanmgr_31bf3856ad364e35_6.0.6001.18000_none_6d4436615d8bd133cleanmgr.exe %systemroot%System32

Now go to Run and type cleanmgr.exe

cleanmgr.exe

You will now see Disk Cleanup run.

Disk Cleanup

Unfortunately on Windows 2012 R2 you must install the Windows Desktop Experience.