Veeam Backup Fails: VSS Writer Errror 0x800423f4 (Azure AD Connect) – New 31/10/2017

This is an updated post about the issue with Veeam backup failures. The original post is here.

This can be fixed by using the following script and using it as a pre-script before backup.

Script

stop-service -displayname "Microsoft Azure AD Sync"
$FQlogonaccount = Get-WmiObject -Class Win32_Service | ? { $_.displayname -match "Microsoft Azure AD Sync"} | select Startname
$split = $FQlogonaccount.startname.Split("\"[0])
$username = $split[1]
$sqlprocess = Get-WmiObject -Query "Select * from Win32_Process where name = 'sqlservr.exe'" | Select Name, Handle, @{Label='Owner';Expression={$_.GetOwner().User}} | ? { $_.owner -match $username} | select handle | Out-String
$sqlpid= $sqlprocess.Split("`n")[3]
Stop-Process -id $sqlpid -force
start-process -filepath "MsiExec.exe" -argumentlist "/f {6C026A91-640F-4A23-8B68-05D589CC6F18}" -wait
Start-Service -displayname "Microsoft Azure AD Sync"

Steps

  1. Save the script as ps1 file. Put somewhere in the Veeam server.
  2. Right-click the backup task and choose “Edit”
  3. Choose “Guest Processing” -> “Enable application-aware processing” -> “Applications”
  4. Choose the one VM that have AAD tool installed and click “Edit”
  5. Go to “scripts” -> “Require successful script execution”, then locate the script we just created.
  6. Now you’re done.

Veeam to AWS VTL (Virtual Tape Library) – Guide

Introduction

You may or may not know that there is quite a cheap way to archive your backup jobs to tape, without actually having a tape drive.

While I was looking at options to store my backup jobs off-site, I researched a number of things:

  • Buying a tape library
  • Buying another enterprise NAS (Network Attached Storage)
  • Using a JBOD (Just a bunch of disks) system like Microsoft Storage Spaces

All of these are expensive. All of these require some sort of hardware plus additional network and configuration. All of these are a pain in the ass.

Continue reading Veeam to AWS VTL (Virtual Tape Library) – Guide

Veeam cannot see ReFS 3.1 volumes in backup (Server 2012 R2)

We had a problem recent where Veeam 9.5 which was installed on Windows 2012 R2 trying to restore from Exchange 2016 on Server 2016 with ReFS datastores.

It turns out, Server 2012 R2 cannot recognise ReFS 3.1 which is on Server 2016. From the screenshot below, this backup is missing two drives which are ReFS:

When logging in to the Exchange server, you will see the correct drives

To fix this you really only have one option: Install Veeam B&R on Server 2016, or at the bare minimum, the mount points on Server 2016.

Once you have a Server 2016 mount point, you will be able to see the ReFS drives.

Veeam Backup Fails: VSS Writer Errror 0x800423f4 (Azure AD Connect)

An updated post is here 31/10/2017

Hi Guys.

An interesting issue over the last few days. Our backup logs have had the following failures in Veeam.

Unable to release guest. Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Digging through the event logs, you will see errors like the following:

  1. Inital backup
    SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=19908. Thread=18488. Client. Instance=LOCALDB#SH7A2278. VD=Global\{C7140958-2759-4979-BA55-0E3F258064ED}1_SQLVDIMemoryName_0.

    vpxclient_2016-09-08_13-09-55

  2. Followed by
    A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,
    the error is likely to reoccur.
    . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. 
    
    Operation:
     PrepareForSnapshot Event
    
    Context:
     Execution Context: Writer
     Writer Class Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
     Writer Name: SqlServerWriter
     Writer Instance Name: SQL Server Code-Named 'Denali' CTP2:SQLWriter
     Writer Instance ID: {2f97c809-8eb4-431c-93ac-b0f81e610013}
     Command Line: "C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe"
     Process ID: 19908

    mstsc_2016-09-08_13-17-47

    mstsc_2016-09-08_13-18-02

We have found this issue is related to an update of the Microsoft Azure AD Connect client.

On all servers that were affected by this, we had just upgraded to the latest Azure AD Connect client. This seems to corrupt the SQL writer somehow.

We haven’t seen many reports of this. There are various fixes for the 0x800423f4 error, but the easiest fix is to repair the LocalDB SQL instance.

To do this:

  1. Go to Add/Remove Programs
  2. Select Microsoft SQL Server 2012 Express LocalDB
  3. Select Repair
    vpxclient_2016-09-08_13-11-17

Once this is finished, you will need to reboot.

We had this issue with many servers. This process fixed it on everyone.

Veeam Backup & Replication 6.5 Patch3 Released

Veeam Backup & Replication 6.5 Patch3 has been released and includes a number of fixes.

URL: http://www.veeam.com/kb_articles.html/kb1751

Resolved Issues

General
• Application-aware image processing may cause Windows Server 2012 Domain Controller to stop booting if virtual machine is configured to use EFI.
• File level recovery process hangs on dynamic disks with partitions size being multiple of 4GB.
• Re-IP fails for replicas if host where replica VM was originally created is deleted from the cluster.
• Disabled ability to move folders in the Files tree with Shift + drag and drop operation because this functionality was not implemented and may result in data loss if the move process is cancelled.

VMware
• Adding virtual disks that were originally excluded back to the job results in wrong change ID used during the first incremental backup.
• Upgrading vCenter or ESX(i) hosts may results in duplicate hosts appearing under Managed Servers, causing jobs to fail with object not found errors.
• VM Copy job always logs the following warning when the target is another VMFS datastore: "Could not perform threshold check for backup location."
• Deleting temporary VM snapshot manually instead of letting the job delete it results in vCenter connection duplication. As the result, vCenter Server may stop responding due to too many connections already opened with the following error: 503 Service Unavailable
• Under certain circumstances, additional registry processing required for SureBackup jobs and re-IP addressing may cause registry corruption with VM failing to boot with the following error: "System hive error"  or "Windows could not start because the following file is missing or corrupt: WINDOWSSYSTEM32CONFIGSYSTEM"
• If vCenter Server is registered with Backup Infrastructure twice (as vCenter Server, and as a Windows server), replica seeding and backup mapping fails with the following error: "Cannot find VM in the backup file specified for seeding."
• Improved performance of enumerating infrastructure objects in large vSphere deployments.

Hyper-V
• Changed block tracking (CBT) driver does not monitor newly appearing virtual disks on volumes that were in redirected access mode at the time when CBT driver starts. This results in full scan incremental runs for the affected virtual disks with the following warning: "Failed to flush change tracking data before snapshot."
• Adding virtual disk files located on volumes mounted into the folder under changed block tracking fails with the following error: "The device object parameter is either not a valid device object or is not attached to the volume specified by the file name."
• Instant VM Recovery fails if virtual disk files are located on a mount point.
• Copying very large files from Windows Server 2012 CSV volume may consume lots of host memory.
• Under rare circumstances, backup file update may fail with the following error: "Failed to store all blob data at the metastore."

Veeam Explorer for Exchange
• Exporting a very large amount of individual items may use up all available system memory.
• Opening certain mailbox databases may fail with the "Jet error -1206" error when Veeam Backup & Replication is installed on Windows 8 or Windows Server 2012.
• Restoring emails that were sent using Outlook Web Access (OWA) fails with the following error: "Error code: ErrorItemSavePropertyError".

PowerShell
• Start-VBRInstantVMRecovery cmdlet fails with the following error: "Cannot complete login due to an incorrect user name or password."

Exchange 2013 Veeam backup logs are not truncated

We have an Exchange 2013 shared environment split over multiple servers. We have our 2xDAG and 2xCAS.

When running a Veeam backup (Full and Incremental), we noticed that the backup logs were not being truncated.

khdn3kju.nkd1What we found is that the DAG’s must be backed up before the CAS servers. If not, the DAG servers, for whatever reason do not truncate the logs.

veeam

Once we changed the order of VMs, we found the logs truncated.

CaptureThe reason I am posting this is that many forum users are saying to install CU1. CU1 has many issues itself, so we didn’t want to do this. This proves CU1 does not need to be installed.

Before anyone installs CU1, please read the blog posts here.

Cannot use CBT: Soap fault. Error caused by file …

I’ve recently had a crash on one of my VMWare 5.1 hosts. When the next scheduled backup executed, I got the following error:

21/02/2013 2:02:28 a.m. :: Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/50bb131b-59817aa0-f7e5-6cae8b1b67ca/test/test01.vmdkDetail: '', endpoint: '

There is an article on the Veeam site KB1113.

The CBT file tracks changes when doing block level backups and is enabled when you do your first backup.

Changed Block Tracking (CBT) is a VMware feature that helps perform incremental backups. VMware Data Recovery uses this technology, and so can developers of backup and recovery software.

You can find out more information here.

You can safely ignore this error. However, if the issue continues, you will need to follow the instructions in KB1113.

Veeam: Failed to prepare guest for hot backup. ‘Cannot connect to the host’s administrative share.’

I recently configured Veeam to backup two VMs that were not on a domain. The servers are running Windows 2008 R2.

The behaviour of Windows Administrative shares has changed in Windows 2008 R2 and Windows 2012. UAC stops remote accessibility of these shares.

The following registry key will have to be added:

HKLMSOFTWAREMicrosoftWindowsCurrentVersionPoliciessystemLocalAccountTokenFilterPolicy

0 – build filtered token (Remote UAC enabled)
1 – build elevated token (Remote UAC disabled)

By setting the DWORD entry to 1, you will be able to access the administrative shares since the remote logon token will not be filtered.

See KB947232 for more information.