Veeam Backup Fails: VSS Writer Errror 0x800423f4 (Azure AD Connect)

An updated post is here 31/10/2017

Hi Guys.

An interesting issue over the last few days. Our backup logs have had the following failures in Veeam.

Unable to release guest. Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Digging through the event logs, you will see errors like the following:

  1. Inital backup
    SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=19908. Thread=18488. Client. Instance=LOCALDB#SH7A2278. VD=Global\{C7140958-2759-4979-BA55-0E3F258064ED}1_SQLVDIMemoryName_0.

    vpxclient_2016-09-08_13-09-55

  2. Followed by
    A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,
    the error is likely to reoccur.
    . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. 
    
    Operation:
     PrepareForSnapshot Event
    
    Context:
     Execution Context: Writer
     Writer Class Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
     Writer Name: SqlServerWriter
     Writer Instance Name: SQL Server Code-Named 'Denali' CTP2:SQLWriter
     Writer Instance ID: {2f97c809-8eb4-431c-93ac-b0f81e610013}
     Command Line: "C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe"
     Process ID: 19908

    mstsc_2016-09-08_13-17-47

    mstsc_2016-09-08_13-18-02

We have found this issue is related to an update of the Microsoft Azure AD Connect client.

On all servers that were affected by this, we had just upgraded to the latest Azure AD Connect client. This seems to corrupt the SQL writer somehow.

We haven’t seen many reports of this. There are various fixes for the 0x800423f4 error, but the easiest fix is to repair the LocalDB SQL instance.

To do this:

  1. Go to Add/Remove Programs
  2. Select Microsoft SQL Server 2012 Express LocalDB
  3. Select Repair
    vpxclient_2016-09-08_13-11-17

Once this is finished, you will need to reboot.

We had this issue with many servers. This process fixed it on everyone.

42 thoughts on “Veeam Backup Fails: VSS Writer Errror 0x800423f4 (Azure AD Connect)”

  1. Interesting, have this issue on a 2012 r2 server where we recently installed Azure AD connect… Unfortunately I have not worked out how to repair it yet, it prompts for install files out of a temp folder. Now just have to work out how to get the correct files for the repair!

  2. It kept happening each time we installed Windows updates… so I made a simple Powershell to automate the repair; used it on >25 servers in different domains so far without fail:

    stop-service -displayname “Microsoft Azure AD Sync”
    $FQlogonaccount = Get-WmiObject -Class Win32_Service | ? { $_.displayname -match “Microsoft Azure AD Sync”} | select Startname
    $split = $FQlogonaccount.startname.Split(“\”[0])
    $username = $split[1]
    $sqlprocess = Get-WmiObject -Query “Select * from Win32_Process where name = ‘sqlservr.exe'” |
    Select Name, Handle, @{Label=’Owner’;Expression={$_.GetOwner().User}} | ? { $_.owner -match $username} | select handle | Out-String
    $sqlpid= $sqlprocess.Split(“`n”)[3]
    Stop-Process -id $sqlpid -force
    start-process -filepath “MsiExec.exe” -argumentlist “/f {6C026A91-640F-4A23-8B68-05D589CC6F18}” -wait
    Start-Service -displayname “Microsoft Azure AD Sync”

  3. Any hope for a more permanent solution? This has happened a few time’s recently. The fix is still vaild but it’s a PITA to have to deal with.

  4. Found this post. Amazed this is such a widespread issue. Running the script seems to be working for us right now (backup 30% done).

    I see mentioned running as a pre-backup script. Can that be done? When I ran the script, I was prompted that some services needed to be stopped first. That type of prompt, would it break a pre-backup script?

  5. Having the same issue. Veeam support wasn’t able to figure it out. It is happening with a 2012 R2 server running the latest 1.1.647.0 of ADSync at one client and on 2008R2 at another. Another similarity is they are both DC’s. I’ve tried repair trick mentioned above, but I am still having the failed backups. The only thing that seems to work is to reboot the VM but then it only works once and then fails again. I would LOVE a permanent solution.

    Anyone else running ADSync on a DC? I have one client that has the latest version on a member VM, and the Veeam backups are running w/o issue….so far 🙂

    1. I opened a case with Microsoft. This is not a Veaam issue, but VSS issue within the VM. I will update this thread if any solution is found.

  6. Let me add this, I did run the SQL repair, reboot and then the sync would work. But it also works when I would reboot w/o running the repair. The server I did the repair on then failed at the next backup. So I don’t think it was the repair as much as it was the reboot.

  7. The “fix” of Repairing the LocalDB and rebooting no longer works for me on a 2012R2 DC, anyone else seeing the same? This is causing errors with our NetApp VSC Backups, just like the Veeam Folks.

    1. That’s what I said above. This is a different issue now. I suspect permissions if you look at the user sql is now running under

  8. Hi,

    I have the excact same issue and have had it for a while now.
    The only way I have been able to fix it (for a day) is a reboot of the DC that fails.

    M$ really needs to address this problem and fast !!

  9. Same for me. It stop with this error.
    Running pre-freeze script in the guest OS
    Error: Script finished execution with unexpected exit code: 1

    I give up 🙂

  10. Hi, can you explain a little more detailed what you think could be the problem ?

    I am in contact with M$ and will post their answer here as soon as I get it. Hope they finally have a working solution.

  11. Hi, I got an answer from M$, the short version is that M$ will not support because Veeam is a 3rd part application and they belive that if we don´t use Veeam, there is no problem.
    Thanks M$

  12. Upgrading MSSQL Server 2012 Express LocalDB to SP4 also fixed the issue for me. Successfully completed multiple back up jobs after that.

  13. This works,

    If you can accept that the AADconnect, is dead in the wather for a short period of time, a simple solution is this.

    Use task scheduler:

    1. Stop the AADconnect service shortly before the Veeambackup is set to run. (Powershell Stop-Service -DisplayName ‘Microsoft Azure AD Sync’)

    —- The backup is running

    2. Start the AADconnect again after a the backup has finished. (mine takes about 15min, so I start the service after 30min.) (Powershell Start-Service -DisplayName ‘Microsoft Azure AD Sync’)

    Thats it.
    My backup has been running 3 times this weekend without any further problems 🙂

  14. In case you’re wondering why this happens:
    – ADSync launches an SQL Server Local DB under it’s own user account
    – The User Profile Service thinks ADSync is no longer logged on, and unloads the registry
    – SQL Server though still has handles to the registry, but they’re invalid now

    Detailed explanation:
    https://support.microsoft.com/en-us/help/2287297/a-com-application-may-stop-working-on-windows-server-2008-when-a-user

    In short: Computer Configuration->Administrative Templates->System->User Profiles->Do not forcefully unload the user registry at user logoff

  15. Hi Ingo,
    You are the man, what a “simple” solution to this big problem. My backup runs perfectly now with no other “tips and tricks”

Leave a Reply