Windows Server Technical Preview – File-Based Backup

In Microsoft endeavors to finally close the book on backup issues, the Hyper-V team is switching to file-based backup, and moving from the non-scalable VSS backup. Let’s face it – most hardware VSS Providers have been like a curse.

When you backup a VM in vNext, a “backup checkpoint” is created. This forks the VM’s configuration is forked and the virtual hard disk(s) is forked too using an AVHD. This is done for a short period of time. This allows changes to continue while the backup is being done. The virtual machine can be live exported as a backup.

image

After this operation a dateless Reference Point is created. The AHVD(s) is merged back into the parent VHD(s). This reference point notes the Resilient Change Tracking ID (per VHD), so we know what changes are made after the AVHD was created, and now we know what blocks must be backed up in a following incremental backup.

image

Some notes:

  • Incremental and “synthetic full” backups can now follow the full backup and this is done using a Differential Export.
  • A restore is basically a process of copying the VM files from backup media and importing the VM.

SAN-based backup is different. A LUN snapshot will retain the parent VHD and AVHD, and only the VM configuration is exported by Hyper-V. CDS, SMI-S or network providers be used to create the LUN backup. The LUN snapshot is removed and job done.

Hyper-V PM Taylor Brown talks about file-based backup in his session at TechEd Europe 2014.

image

Windows Server Technical Preview – Resilient Change Tracking

Windows Server Hyper-V has had an … interesting … history when it comes to backup. It has been a take-it-personally mission of the Hyper-V team to stop backup being an issue for Hyper-V customers. Backup of CSV in Windows Server 2008 R2 was not fun. Things got better in WS2012, and again in Windows Server 2012 R2. And we might finally be getting there with the next release of Windows Server.

An important change to Hyper-V backup is to enable partners to keep up with the pace of change of Windows Server – we’ve seen some backup vendors take years to catch up with a new version, and this prevents mutual customers from keeping their hosts in step with Microsoft.

In order for a backup product to do incremental backups, it needs to do block based change tracking. Each vendor has to create one of these filter drivers that sits in the storage stack. This stuff is hard to do right, and it can cause stability and performance issues if not done correctly. And it also slows down the development/re-test/re-certify of BackupProduct2016 to keep up with the release of Windows Server 2016.

Some bad change tracking implementations, that you may know of, lived in memory as bitmaps. If the host had an un-planned outage then the next backup had to be a full backup. Or maybe if the VM live migrated to another host, that VM would have to do a full backup because the change tracking was no longer in the memory of the host.

Resilient Change Tracking is built-in backup change tracking of changed blocks within virtual hard disks. It is used for incremental backup, and it is the underlying engine for differential export. The change tracking bitmap lives in memory and on-disk. The on-disk bitmap is not as granular because it is the fallback from the much more detailed in-memory bitmap.

The goal now is that backup vendors should stop writing their own filter driver to implement change tracking. If they use the built-in resilient change tracking then they can focus more time on feature development testing/certification, and keep up with Microsoft’s frequent releases of Windows Server. And hopefully, Microsoft’s change tracking will undergo suitable levels of testing that will give all customers a universally stable and well-performing subsystem.

Hyper-V PM Taylor Brown talks about Change Tracking in his session at TechEd Europe 2014.

Windows Server Technical Preview – Delivery of Integration Components

Those of you who have run more than one generation of Hyper-V will understand the pain of updating integration components in a VM’s guest OS. If you run Windows (this is not applicable to Linux because the process is different) the you need to run the latest ICs in a guest OS for that VM to have:

  • The latest virtualization features
  • Stability
  • Performance

We typically have seen new versions of the ICs in three occasions:

  • A hotfix or Windows Update
  • A service pack – no longer relevant but an update rollup might bring new ICs
  • A new version of Hyper-V

The process was that VMGuest.ISO was updated on the host, and we would mount that ISO from the VM to install the latest integration components. This assumed that:

· We had admin rights to the guest OS – not applicable usually in a cloud

  • Network access
  • Time
  • Patience

We had workarounds such as using PowerShell or System Center, but again, this assumed we had rights to the guest OS or network access.

Microsoft was keen to solve this issue … and they went to a method that I think many of us will approve of: updates to the Windows integration components for Hyper-V will be delivered by Windows Update (and hence WSUS). This has started with delivery to any of the following guest OSs running on the Tech Preview of vNext:

  • Windows Server 2012
  • Windows Server 2008 R2
  • Windows 8
  • Windows 7

Microsoft uses KVP (enabled by default) to determine that the VM is running on vNext.

This new process will give cloud admins control over the IC release (via WSUS) and will automate the delivery of the ICs to all guests that run Windows Update, ensuring that clients are up to date and can avail of the best that Hyper-V can offer. No more McGuyvering required.

Microsoft News – 17 December 2014

Things have settled down a little after last week’s surprise Azure announcements.

Hyper-V

Azure

Office 365

Microsoft Partners

Licensing

  • Price increases in 2015!! SPLA/hosting licensing costs are going up. Hosters will have no choice but to pass that on to their customers.

Miscellaneous

Windows Server Technical Preview – Cluster Functional Level

This feature is tied into Cluster Rolling Upgrade. When we are doing a rolling upgrade of a cluster (from WS2012 R2 to vNext), we will temporarily have a mixture of WS2012 R2 hosts and Windows Server vNext hosts. This is referred to as mixed mode.

To avoid issues, the cluster will either:

  • Run only features of the down-level hosts
  • Run features only on down-level hosts

For example, on a storage cluster (SOFS) running disk deduplication (for VDI personal VMs), the cluster will only run deduplication optimization jobs on the down-level hosts because vNext optimization will make changes that are not backwards compatible.

This mixed mode is TEMPORARY – it should not be run for months or years!!!!

EDIT: Today (17/Dec/2014) I learned that Microsoft will have a support limited of 4 weeks for running a cluster in mixed mode. They want you to finish the migration ASAP, but 4 weeks will be the support limit. There is not hard coded limit, but now you know what they’ll support.

EDIT: I also learned on 17/Dec/2014 that Microsoft will not put a limit on how long v5.0 VMs (VMs with WS2012 R2 host compatibility – more on this later) can run on a higher level cluster.

The goal is that you upgrade each node in the cluster as quickly as possible. You might pause half way through to observe the stability of your hosts (drivers, firmware and external storage/networking hardware). If you’re happy you continue the upgrade. If not, you can rebuild the upgraded hosts to WS2012 R2 and live migrated your VMs back.

Note: VMs will be manually upgraded to vNext and this is why you can live migrate VMs between WS2012 R2 and vNext hosts within the cluster without any issues.

When you have finished the upgrade you will complete the project by raising the cluster functional level. This opens up the cluster’s vNext features and then you should manually upgrade the VMs to v6.0 (vNext).

Windows Server Technical Preview – Cluster Operating System Rolling Upgrade

A big pain point with Microsoft’s increased cadence of releases of Windows Server is that upgrading a cluster has been a royal pain in the arse.

Previous to the release of WS2012 R2, there was only one process: build a new cluster (either by buying new hardware or draining a host from the old cluster) and do an out-of-band migration of VMs from the old cluster to the new one. WS2012 R2 decreased the pain slightly by allowing Cross-Version Live Migration – but we let Microsoft know that this wasn’t enough.

Windows Server vNext will allow you to perform a cluster rolling upgrade without creating a new cluster. You will be able to do this from WS2012 R2 to vNext (note I did not say WS2012 or W2008 R2 or W2008).

The process is that you will rebuild each node in the cluster, one at a time with Windows Server vNext. You will, for a short time only, have hosts running WS2012 R2 Hyper-V and vNext Hyper-V in the cluster at the same time. This is called Mixed Mode, and VMs can live migrate between the nodes. Eventually you will complete the project by raising the cluster functional level after all hosts in the cluster are on vNext.

image

The flow chart of the process is shown here:

image

This feature will be a major time and money saver, and will allow Hyper-V customers to keep up with the fast release of new versions of Windows Server Hyper-V.

Windows Server Technical Preview – Cluster Cloud Witness

Imagine a multi-site or stretch cluster that spans sites A and B. Hosts are running in Site A and Site B and are nodes in that cluster. A critical function of a cluster is to calculate quorum in the event of a site outage or a link failure – one site must have more than half of the available votes to have quorum and continue operating as a cluster (and failover the VMs from the other site).

The recommended solution for this in the past was to create a file share witness … operating in a third site. So first you needed a third site. That’s not going to be cheap! And then you needed, ideally, a file server cluster running in that third site to ensure that the file share witness was highly available.

image

Windows Server vNext offers a new advanced quorum option of using a Cloud Witness. This is where we use Azure as a witness. Folks – this is going to be one seriously affordable option!

You create a blob storage account in Azure. This will store just an incremental sequence number; it’s just a vote so the cluster will remain operational if Azure has issues. This will cost just a few cents per month to operate.

You then configure the cluster with the storage account name and storage account key for secure access. The cluster will then use Azure as a witness. This is a lot easier and cheaper than configuring a file share witness on a cluster in a third site.

Microsoft News – 12 December 2014

It’s December, the month when Microsoft employees normally head away for a long vacation and nothing much happens. Or so we thought. Azure went wild last night, releasing loads of new features either into preview or GA. Oh yeah, loads of December updates from Microsoft have problems.

Windows Client

System Center Virtual Machine Manager

Azure

Miscellaneous

You Do Not Need To Run SCVMM To Replicate Hyper-V To Azure

If you follow Microsoft then you are used to December being a dead month. So I checked my Twitter feed last night and was stunned by some big Azure announcements.

The most important of the announcements to me was the change that is being made to Azure Site Recovery (ASR), AKA DR in the cloud. Previous to last night, you need to run SCVMM on premises to replicate Hyper-V to Azure. This baffled me:

  • You had to install the protection agent on each host/cluster node anyway
  • SMBs, the companies that are most likely to use ASR, cannot afford System Center
  • There is a low adoption rate of SCVMM with System Center/Hyper-V customers

The feedback on this was given – and Microsoft made a change. Last night they announced the general availability of Disaster Recovery (DR) to Azure for Branch offices and SMB feature in our Azure Site Recovery (ASR) service … AKA ASR without SCVMM. This will allow you to replicate Hyper-V VMs into Azure without using System Center on premises.

The hosts must be running WS2012 R2 Hyper-V. Replication is done using Hyper-V Replica. You get centralized replication monitoring and orchestration as a part of the service. And you get the one-click test, planned and unplanned failover types.

THIS IS FRAKKING GREAT NEWS!

Why am I so excited? The original releases of ASR were targeted at customers with System Center licensing. Those are mid-large customers and are likely the ones that already have DR sites. Adoption rates were going to be low. The customer base that needs ASR are the SMBs that run Hyper-V hosts on-premises. That is a huge breadth market. Microsoft partners can enable those customers via Azure in Open licensing – buy some credits ($100 value each), try out ASR with no long term CAPEX or contractual commitments, and see what it can do for your business. And then give your insurance company a call to see what having a remote DR site will do for the company’s insurance premium.

Feedback Matters Once Again In Microsoft

Microsoft has changed – and I’m not just talking about the obvious stuff that’s in the headlines. Two years ago I saw a pre-beta product and gave feedback to the PMs that this product needed serious changes. I was ignored (FYI, I was not alone in my opinions and in giving this feedback). 18 months ago, I saw a beta release and I gave the same feedback to PMs. I was ignored. 12 months ago the product was released. I blogged about the issues, was quoted by some big tech press names, and suddenly I was Mr. Popular with the PMs and marketing folks. I gave my feedback and I was ignored. 6 months ago the product went through some big changes, but the issue was still there. I commented on it, and once again the marketing folks and PMs were all over me. I gave my feedback … and I was ignored.

But …

A few months ago I found out that someone listened. Enough people like me spoke up. And maybe my market predictions had come true – not that anyone would ever admit that! The product was going to change. And guess what – that change did arrive.

Thank you to the team involved.

The Microsoft that I’ve been working with has changed a lot in the last 24 months. The secrecy of the Sinofsky era is over. Stuff that me and other MVPs have fed back to product groups is being listened to – I wish I could share those stories!!! I’m certainly not saying we design anything. And I’m not saying that everything we feedback is done. But I do see a correlation between feedback and product changes.

If you have feedback then give it to Microsoft. Please don’t make the mistake of telling some local rep – they probably will forget it straight away. Your feedback need to go to the product groups. Microsoft announces forums for feedback every once in a while – I blogged about one yesterday. Take advantage of those. MVPs like myself and Didier Van Hoye actively request feedback before we go to the MVP Summit in Redmond to meet those PMs – take advantage of that too. I might be a jerk in dealing with some comments on this site, but I note the constructive ones and take those ideas to Microsoft.

Microsoft has changed a lot. It was extremely noticeable when we MVPs were last there. It’s a fantastic change – it was by far the most engaging experience I’ve had in Redmond. They really want our feedback to avoid the issues that we saw in the past. So take advantage of this and get the product that you want.