Veeam Backup Fails: VSS Writer Errror 0x800423f4 (Azure AD Connect)

Hi Guys.

An interesting issue over the last few days. Our backup logs have had the following failures in Veeam.

Unable to release guest. Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Digging through the event logs, you will see errors like the following:

  1. Inital backup
    SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=19908. Thread=18488. Client. Instance=LOCALDB#SH7A2278. VD=Global\{C7140958-2759-4979-BA55-0E3F258064ED}1_SQLVDIMemoryName_0.

    vpxclient_2016-09-08_13-09-55

  2. Followed by
    A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,
    the error is likely to reoccur.
    . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. 
    
    Operation:
     PrepareForSnapshot Event
    
    Context:
     Execution Context: Writer
     Writer Class Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
     Writer Name: SqlServerWriter
     Writer Instance Name: SQL Server Code-Named 'Denali' CTP2:SQLWriter
     Writer Instance ID: {2f97c809-8eb4-431c-93ac-b0f81e610013}
     Command Line: "C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe"
     Process ID: 19908

    mstsc_2016-09-08_13-17-47

    mstsc_2016-09-08_13-18-02

We have found this issue is related to an update of the Microsoft Azure AD Connect client.

On all servers that were affected by this, we had just upgraded to the latest Azure AD Connect client. This seems to corrupt the SQL writer somehow.

We haven’t seen many reports of this. There are various fixes for the 0x800423f4 error, but the easiest fix is to repair the LocalDB SQL instance.

To do this:

  1. Go to Add/Remove Programs
  2. Select Microsoft SQL Server 2012 Express LocalDB
  3. Select Repair
    vpxclient_2016-09-08_13-11-17

Once this is finished, you will need to reboot.

We had this issue with many servers. This process fixed it on everyone.

Exchange 2013 Veeam backup logs are not truncated

We have an Exchange 2013 shared environment split over multiple servers. We have our 2xDAG and 2xCAS.

When running a Veeam backup (Full and Incremental), we noticed that the backup logs were not being truncated.

khdn3kju.nkd1What we found is that the DAG’s must be backed up before the CAS servers. If not, the DAG servers, for whatever reason do not truncate the logs.

veeam

Once we changed the order of VMs, we found the logs truncated.

CaptureThe reason I am posting this is that many forum users are saying to install CU1. CU1 has many issues itself, so we didn’t want to do this. This proves CU1 does not need to be installed.

Before anyone installs CU1, please read the blog posts here.

When offline files go bad… [Offline files recovery]

Interesting situation. A few days ago I had a customers machine that had been pulled off a domain without syncing the offline files. These files had no been synced for a good 1.5 years. Ouch. Anyway, to cut a long story short, we had to recover a few GB of offline files to a new machine. Is this possible? Well yes it is. CSCCMD.exe is a command line executeable which comes on the Windows 2003 resource kit.  Extended commands for csccmd.exe are here.

Here is how to do it:

Instructions:
1. Copy c:windowscsc to folder to the new computer
2. On the new computer select the start button and click on run. Type cmd.
3. Type csccmd.exe /enable
4. Type csccmd.exe /EXTRACT /RECURSE /TARGET:c:recovered_files

CSCCMD.exe Download:
Download Link 1