Veeam Backup Fails: VSS Writer Errror 0x800423f4 (Azure AD Connect)

Hi Guys.

An interesting issue over the last few days. Our backup logs have had the following failures in Veeam.

Unable to release guest. Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Digging through the event logs, you will see errors like the following:

  1. Inital backup
    SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=19908. Thread=18488. Client. Instance=LOCALDB#SH7A2278. VD=Global\{C7140958-2759-4979-BA55-0E3F258064ED}1_SQLVDIMemoryName_0.

    vpxclient_2016-09-08_13-09-55

  2. Followed by
    A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,
    the error is likely to reoccur.
    . Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer. 
    
    Operation:
     PrepareForSnapshot Event
    
    Context:
     Execution Context: Writer
     Writer Class Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
     Writer Name: SqlServerWriter
     Writer Instance Name: SQL Server Code-Named 'Denali' CTP2:SQLWriter
     Writer Instance ID: {2f97c809-8eb4-431c-93ac-b0f81e610013}
     Command Line: "C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe"
     Process ID: 19908

    mstsc_2016-09-08_13-17-47

    mstsc_2016-09-08_13-18-02

We have found this issue is related to an update of the Microsoft Azure AD Connect client.

On all servers that were affected by this, we had just upgraded to the latest Azure AD Connect client. This seems to corrupt the SQL writer somehow.

We haven’t seen many reports of this. There are various fixes for the 0x800423f4 error, but the easiest fix is to repair the LocalDB SQL instance.

To do this:

  1. Go to Add/Remove Programs
  2. Select Microsoft SQL Server 2012 Express LocalDB
  3. Select Repair
    vpxclient_2016-09-08_13-11-17

Once this is finished, you will need to reboot.

We had this issue with many servers. This process fixed it on everyone.

VMWare Workstation: The VMware Authorization Service is not running

This error can occur after the Windows 10 update, either 1511 or 1607.

This issue occurs as part of the upgrade removes the VMWare authorization service. If you click Start->Run and type service.msc you will see the service missing.

Go to Control Panel->Add/Remove Programs and repair your VMWare Workstation installation.

The other option is to update your version. Ensure you have the correct licensing before you do this.

Block xmlrpc.php attacks with fail2ban + iptables wordpress

One of the issues I’ve faced on this server is xmlrpc.php attacks. These are normally bots trying to exploit old bugs in xmlrpc.php within WordPress. Many legit plugins use calls to this file such as Jetpack. So blocking it isn’t really an option.

In my case, I wanted to block these attacks with iptables. So I went about creating a rule using fail2ban.

To get started, get iptables and fail2ban installed:

apt-get install fail2ban iptables

Once installed, edit the default jail file. This won’t exist on a new install

nano /etc/fail2ban/jail.local

Add the following lines (make sure the path matches your own)

[xmlrpc]
enabled = true
filter = xmlrpc
action = iptables[name=xmlrpc, port=http, protocol=tcp]
logpath = /var/log/apache2/access.log
bantime = 43600
maxretry = 2

Now edit the following file (this won’t exist)

nano /etc/fail2ban/filter.d/xmlrpc.conf

Add the following lines

[Definition]
failregex = ^<HOST> .*POST .*xmlrpc\.php.*
ignoreregex =

Restart the fail2ban service

service fail2ban restart

You can watch the log in real-time to ensure it’s blocking correctly:

tail -f /var/log/fail2ban.log

putty_2016-08-12_13-25-33

IBM v3700 + Fusion MT HBA + Lenovo x3650 M5 – Multipath issue on VMWare 6

I’ve been working on an issue for the past week with the following hardware/software:

3x Lenovo x3650 M5 Type 5462
6x Fusion-MPT 12GSAS SAS3008 (two each host)
1x IBM v3700 SAN
VMWare 6.0 U2 (Lenovo image)

The HBA’s and SAN were configured in the following manner:

FC-attach+(1)

What I didn’t realise early on was that multipathing from the SAN to VMWare was not working. As I was in a rush, I saw the SAS connections were live. The SAN said everything was ok, so I didn’t think twice.

However, on closer inspection on the SAN, I found that only one SAS HBA on each host was active. Hmm, what was going on?

Capture (1)

VMWare was also reporting the same issue:

cap2

Initially, I thought this was a SAN issue. I contacted support who checked out the SAN and couldn’t find any issue.

I then contacted VMWare who initially said the configuration was not supported (driver wise). Actually, what I found is VMWare were referring to the wrong driver.

After about a week of going back and forward, I noticed the drivers that were shipped with the Lenovo VMWare image were not the latest. I proceed to update the drivers which in turn, enabled multi-pathing in VMWare.

VMWare:

chrome_2016-08-09_21-50-40

SAN:

chrome_2016-08-09_21-57-30

This was quite a simple issue but made a bit more complicated as all the hardware seemed supported and at the right driver level.

The correct driver was the lsi-msgpt3 driver found here. lsi-msgpt3 version lsi-msgpt3 version 13.00.00.00-1OEM. The installed version was lsi-msgpt3 version 12.00.00.00-1OEM.

Sometimes it pays to check the basics.

Fosshub.com compromised – Malware on popular downloads

As discovered tonight, popular download website, Fosshub, has been compromised.

Hackers have targeted popular downloads such as Audacity, WinDirStat, qBittorrent, MKVToolNix, Spybot Search&Destroy, Calibre, SMPlayer, HWiNFO, MyPhoneExplorer, IrfanView and others.

https://twitter.com/CultOfRazer/status/760668803097296897

When installing these programs from Fosshub, you will be infected with malware. This malware will re-write your MBR and you will not longer be able to boot into your operating system.

https://twitter.com/CultOfRazer/status/760752941066313728

CultOfTheRazor has claimed responsibility.

Microsoft removes features from Windows 10 Professional

Earlier in the month Microsoft announced their new subscription service for Windows 10 Enterprise. Starting at just $7 USD per month, end users are able to subscribe to Windows 10 enterprise on a monthly basis.

This, I  think, is a good move by Microsoft.

However, what is disturbing is what Microsoft is changing on lower tier versions, such as Windows 10 Professional. See the following changes:

Windows 10 changes

I’d say this is to target the small business users who now cannot disable the shop within Windows. This will push slightly larger businesses to go to Enterprise.

While a small change, I hope Microsoft doesn’t fall into the trap many other monthly subscription providers have fallen into.

See the following for more information.

The attempted operation failed. An Object could not be found Outlook

When selecting Rules and Alerts from the home page of Outlook, you will receive this error.

This can actually be caused by a number of issues, such as:

  • Corrupt rules
  • Broken search folders
  • Corrupt Outlook profile

To fix this issue, try the following (in order of less destructive)

  1. Run Outlook.exe /cleanrules from the command line. This will remove all rules from Outlook and Exchange.
  2. Delete all search folders and create a default search folder with Unread mail as the default
  3. Re-create the Outlook profile. This is a last resort, but in most cases will work. This indicates something wrong with the local configuration of the Outlook profile

 

Performance Tuning Forefront TMG 2010

Microsoft Forefront TMG 2010 has been a solid product. It has not had an update in some time, and that’s a shame.

I was reviewing my TMG server a few weeks ago in order get a bit more performance from it. The server is virtualized, and currently only gives me around 50-60mb/s throughput with all its rules. I wanted to increase this and make it more responsive in the process.

So let us proceed:

#1 Sort Rule Priority

Like most other firewalls, TMG processes rules from top to bottom. If you have a frequently accessed rule, like web browsing, for example, put this at the top. For me, this was a spam filter listening rule. TMG had connections coming in for the spam filter at the rate of 5-10 per second. I moved these to the top

tmg1

 

The next rule you want to sort out is traffic from TMG server. I would generally add this as my second/third rule.

tmg1.1

After this came my office outbound rulestmg2

 

After this came the rest of my rules which consisted of various servers/voip etc.

#2 Disabled Legacy Services

Since TMG is now discontinued, many of the services in TMG are no longer updated. It’s up to you, but you might as well disable them to recover some performance. These services are:

  • IPS
  • Spam Filtering
  • Virus and Content Filtering

Ensure each one is disabled. You might be a bit hesitant to disable these. If you check your definitions, you will find they have not been updated in some time.

tmg3

tmg4

#3 Remove Old Rules

Lastly, remove any old rules. We don’t tend to look at firewalls often. So this simple task often gets overlooked.

As well as removing the old rules, ensure that you make your listener rules only listen to what’s needed. This will cut down on processing time.

Conclusion

TMG is a great product, but now beyond its used by date. TMG will always hold a special place in my software archive as a product that could have become a great firewall appliance <3

I hope this helps.

Making WordPress Faster with Google’s mod_pagespeed – Part #1

WordPress is a website framework used by millions around the world. Getting WordPress to work reliably and quickly, however, can be a difficult task.
Over the last few days, I’ve been trying to get my WordPress site operate as efficiently and swiftly as possible. I run my current website on Linux, Ubuntu to be exact. On top of this, I run the Web server Apache.

Google PageSpeed Insights listed my website with a score of 74/100. This is not a good score, and it will push your Google ranking down. My goal was to get this ranking up.

My first step was to look at mods for Apache. The one model used in the past which is very good as a Google’s mod pagespeed. This mod does many things such as compressing images, resizing images, combining JavaScript, combining CSS, removing blocking JavaScript, and many other tasks. So the first part of this post is going to look at installing mod page speed and configuring it for your website.

The first thing to do is to log into your Web server via SSH or telnet and install mod page speed.

Now to install mod page speed you need to go to the developers Google website. The easiest way to install mod page speed is to copy the link from the website and use wget to download it and to install it with apt-get. I recommend you only use the latest stable version.wget

wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_amd64.deb
dpkg -i mod-pagespeed-*.deb
apt-get –f install

The great thing is once the mods install on your server, it adds to a repository to your servers repository list. In future, you only have to type apt-get upgrade to install new versions.

Once pagespeed is installed, you need to enable it. To do this type the following:

a2enmod pagespeed

Another mod you also want to make sure it is enabled, is expires. To enable this mod, type the following:

a2enmod expires

Once these mods have been enabled, you need to restart Apache. The easiest way to do this is to type the following:

services apache2 restart

Now we need to set some configuration options. I don’t usually set configuration options on pagespeed’s global configuration. Instead, I do the page speed configuration at that the vhost level.

To set you configuration options you need to edit your sites availability configuration file which is typically hosted on the Apache folder. You can find this location here:

cd /etc/apache2/sites-available/

In this directory, you should have the configuration file for your current site. I am going to too much detail around these configuration file. This is something you probably need to look up on your own and figure out how this is configured yourself.

The first thing we need to do an vhost file, is make sure that pagespeed is turned on. To do this open up you’re vhost file, and add the following commands:

ModPagespeed on

As you may have gathered, this turns pagespeed on. As well is this, though, you need one other command to make this work. The commander is the following:

ModPagespeedRewriteLevel CoreFilters

The core filters command tells pagespeed to apply the core filters. The core filters are a set of configurations which are applied to pagespeed. You can find out more about these filters here.

While and the vhost configuration file, I also suggest we turn on the expires module. This is done by typing the following:

ExpiresActive on

However as with pagespeed you need one more command to make this function properly. That is the following:

ExpiresDefault "access plus 1 week"

there are additional settings we can apply to this. Some examples are listed below:

ExpiresByType image/jpg "access plus 1 week"

ExpiresByType image/jpeg "access plus 1 week"

ExpiresByType image/gif "access plus 1 week"

ExpiresByType image/png "access plus 1 week"

This is the basics of our configuration file for the vhost. We now need to restart Apache again, by doing the following:

service apache2 restart

If you now try the Google PageSpeed Insight Test again, you should find that your score increases. Our score increased up to about 85/100 just by using mod pagespeed. There are additional filters that you can apply to mod pagespeed to do different things. These additional filters, can be viewed here. If you follow the column in court filters, and it is selected is no, this filter is not applied currently. To apply one of these filters, said it in your vhost file as follows:

ModPagespeedImageRecompressionQuality 70

ModPagespeedEnableFilters defer_javascript,inline_preview_images,resize_mobile_images,remove_comments,sprite_images

ModPagespeedEnableFilters inline_google_font_css,insert_image_dimensions,combine_css

If you reload Apache again, you will find that your Google page rank score increases even further.

There are further tricks that we can apply to pagespeed. But they are out of the scope of this post. I will follow this up with an additional post on the most advanced options of pagespeed.

My full configuration file was as follows:

        ServerName www.website.com
        ServerAdmin info@websome.com
 
        ServerAlias website.com
 
        DocumentRoot /var/www/website.com/public_html/
        
        AllowOverride All     
 
        AccessFileName .htaccess
 
        #LogLevel info ssl:warn
 
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
 
        ExpiresActive on
        ExpiresDefault "access plus 1 week"
        ExpiresByType image/jpg "access plus 1 week"
        ExpiresByType image/jpeg "access plus 1 week"
        ExpiresByType image/gif "access plus 1 week"
        ExpiresByType image/png "access plus 1 week"
 
        ModPagespeed on
        ModPagespeedStatistics on
        ModPagespeedStatisticsLogging on
        ModPagespeedLogDir /var/log/pagespeed     
        ModPagespeedRewriteLevel CoreFilters
        ModPagespeedStatistics on
        ModPagespeedFileCachePath            "/var/cache/pagespeed/"
        ModPagespeedFileCacheSizeKb          400000
        ModPagespeedFileCacheCleanIntervalMs 3600000
        ModPagespeedFileCacheInodeLimit      500000
        ModPagespeedImageRecompressionQuality 70
        ModPagespeedEnableFilters defer_javascript,inline_preview_images,resize_mobile_images,remove_comments,sprite_images
        ModPagespeedEnableFilters inline_google_font_css,insert_image_dimensions,combine_css

If you have any questions, please list them below.