Veeam to AWS VTL (Virtual Tape Library) – Guide

Introduction

You may or may not know that there is quite a cheap way to archive your backup jobs to tape, without actually having a tape drive.

While I was looking at options to store my backup jobs off-site, I researched a number of things:

  • Buying a tape library
  • Buying another enterprise NAS (Network Attached Storage)
  • Using a JBOD (Just a bunch of disks) system like Microsoft Storage Spaces

All of these are expensive. All of these require some sort of hardware plus additional network and configuration. All of these are a pain in the ass.

Continue reading Veeam to AWS VTL (Virtual Tape Library) – Guide

The provided manifest file is invalid VMWare OVA

Recently while trying to import the AWS Virtual Tape gateway, I found the OVF template for VMWare was failing to install.

After getting pretty frustrated, I decided to open the OVA file in 7zip and extract the MF file.

What I found was the file contained a lot of whitespaces.

If you remove this whitespace and add this file back to the OVA archive, the OVA will deploy.

Happy days.

Accessing VMWare CLI via Lenovo IMM2 (KVM)

I’ve just recently run in to a problem trying to access ESXi’s CLI from the console using IMM2 (Integrated Management Module). The KVM seems to be intercepting commands incorrectly. This seems to be the case for a number of KVM tools.

The normal command for accessing the CLI once enabled, is to press ALT+F1. This won’t work. You will need to hold down ALT+Space, release Space and press F1 while keeping ALT held down.

ALT+Space release space ALT+F1

VMWare: Set IP Hash policy via CLI

I recently lost a host at one of my sites which have a failover policy set of IP Hash. No problem, connect a new host remotely and add it back to the network. Problem is, I cannot access this host from vCenter because of this policy. After posting on Reddit, someone gave me the following command to enter via CLI

You will need to enable the CLI from the console -> troubleshooting options.

esxcli network vswitch standard portgroup policy failover set -p "Management Network" -l iphash

Once this is set, you should now be able to see your host from vCenter.

Veeam Backup Fails: VSS Writer Errror 0x800423f4 (Azure AD Connect)

Hi Guys.

An interesting issue over the last few days. Our backup logs have had the following failures in Veeam.

Unable to release guest. Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Digging through the event logs, you will see errors like the following:

  1. Inital backup
    SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=19908. Thread=18488. Client. Instance=LOCALDB#SH7A2278. VD=Global\{C7140958-2759-4979-BA55-0E3F258064ED}1_SQLVDIMemoryName_0.

    vpxclient_2016-09-08_13-09-55

  2. Followed by
    A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,
    the error is likely to reoccur.
    . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. 
    
    Operation:
     PrepareForSnapshot Event
    
    Context:
     Execution Context: Writer
     Writer Class Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
     Writer Name: SqlServerWriter
     Writer Instance Name: SQL Server Code-Named 'Denali' CTP2:SQLWriter
     Writer Instance ID: {2f97c809-8eb4-431c-93ac-b0f81e610013}
     Command Line: "C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe"
     Process ID: 19908

    mstsc_2016-09-08_13-17-47

    mstsc_2016-09-08_13-18-02

We have found this issue is related to an update of the Microsoft Azure AD Connect client.

On all servers that were affected by this, we had just upgraded to the latest Azure AD Connect client. This seems to corrupt the SQL writer somehow.

We haven’t seen many reports of this. There are various fixes for the 0x800423f4 error, but the easiest fix is to repair the LocalDB SQL instance.

To do this:

  1. Go to Add/Remove Programs
  2. Select Microsoft SQL Server 2012 Express LocalDB
  3. Select Repair
    vpxclient_2016-09-08_13-11-17

Once this is finished, you will need to reboot.

We had this issue with many servers. This process fixed it on everyone.

VMWare Workstation: The VMware Authorization Service is not running

This error can occur after the Windows 10 update, either 1511 or 1607.

This issue occurs as part of the upgrade removes the VMWare authorization service. If you click Start->Run and type service.msc you will see the service missing.

Go to Control Panel->Add/Remove Programs and repair your VMWare Workstation installation.

The other option is to update your version. Ensure you have the correct licensing before you do this.

IBM v3700 + Fusion MT HBA + Lenovo x3650 M5 – Multipath issue on VMWare 6

I’ve been working on an issue for the past week with the following hardware/software:

3x Lenovo x3650 M5 Type 5462
6x Fusion-MPT 12GSAS SAS3008 (two each host)
1x IBM v3700 SAN
VMWare 6.0 U2 (Lenovo image)

The HBA’s and SAN were configured in the following manner:

FC-attach+(1)

What I didn’t realise early on was that multipathing from the SAN to VMWare was not working. As I was in a rush, I saw the SAS connections were live. The SAN said everything was ok, so I didn’t think twice.

However, on closer inspection on the SAN, I found that only one SAS HBA on each host was active. Hmm, what was going on?

Capture (1)

VMWare was also reporting the same issue:

cap2

Initially, I thought this was a SAN issue. I contacted support who checked out the SAN and couldn’t find any issue.

I then contacted VMWare who initially said the configuration was not supported (driver wise). Actually, what I found is VMWare were referring to the wrong driver.

After about a week of going back and forward, I noticed the drivers that were shipped with the Lenovo VMWare image were not the latest. I proceed to update the drivers which in turn, enabled multi-pathing in VMWare.

VMWare:

chrome_2016-08-09_21-50-40

SAN:

chrome_2016-08-09_21-57-30

This was quite a simple issue but made a bit more complicated as all the hardware seemed supported and at the right driver level.

The correct driver was the lsi-msgpt3 driver found here. lsi-msgpt3 version lsi-msgpt3 version 13.00.00.00-1OEM. The installed version was lsi-msgpt3 version 12.00.00.00-1OEM.

Sometimes it pays to check the basics.

Specified argument was out of the range of valid values. Parameter name: utcDate (RDWeb)

Recently had this issue with RDWeb (Web gateway for Remote Desktop Services). After a lot of searching, we found this was being caused by VMWare. When you have VMWare Tools installed, by default it updates hosts to the VMWare hosts time. If you have NTP enabled on your operating system, this will then update the time again, correcting it. This becomes a loop. This causes the above error. Here is the full output

Specified argument was out of the range of valid values.
Parameter name: utcDate

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: utcDate

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: utcDate]
System.Web.HttpCachePolicy.UtcSetLastModified(DateTime utcDate) +4974654
System.Web.HttpCachePolicy.SetLastModified(DateTime date) +285
Microsoft.TerminalServices.Publishing.Portal.ResourceFileHandler.ProcessRequest(HttpContext context) +1398
System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +913
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +165

Configure your VMWare host to get time from a NTP server, or disable VMWare Tools from updating the hosts time.

Cannot use CBT: Soap fault. Error caused by file …

I’ve recently had a crash on one of my VMWare 5.1 hosts. When the next scheduled backup executed, I got the following error:

21/02/2013 2:02:28 a.m. :: Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/50bb131b-59817aa0-f7e5-6cae8b1b67ca/test/test01.vmdkDetail: '', endpoint: '

There is an article on the Veeam site KB1113.

The CBT file tracks changes when doing block level backups and is enabled when you do your first backup.

Changed Block Tracking (CBT) is a VMware feature that helps perform incremental backups. VMware Data Recovery uses this technology, and so can developers of backup and recovery software.

You can find out more information here.

You can safely ignore this error. However, if the issue continues, you will need to follow the instructions in KB1113.

Veeam: Failed to prepare guest for hot backup. ‘Cannot connect to the host’s administrative share.’

I recently configured Veeam to backup two VMs that were not on a domain. The servers are running Windows 2008 R2.

The behaviour of Windows Administrative shares has changed in Windows 2008 R2 and Windows 2012. UAC stops remote accessibility of these shares.

The following registry key will have to be added:

HKLMSOFTWAREMicrosoftWindowsCurrentVersionPoliciessystemLocalAccountTokenFilterPolicy

0 – build filtered token (Remote UAC enabled)
1 – build elevated token (Remote UAC disabled)

By setting the DWORD entry to 1, you will be able to access the administrative shares since the remote logon token will not be filtered.

See KB947232 for more information.