Azure Files With Sync

Speakers:

• Klaas Langhout, Principal Program Manager, Azure Storage
• Mine Tanrinian Demir, Principal Program Manager, Azure Storage

This is the one feature that is announced this week that I know for certain will turn into business for my customers so I’ve been looking forward to it finally going public.

Today

• Simplify share management using the cloud.
• Leverage snapshots to backup your data
• Use files to sync between offices
• Tier cold storage to the cloud.

Azure is a bunch of lego blogs that can be assembled to produce services. A keystone is Azure Storage. Hyperscale at >30 trillion transactions per second at the moment across trillions of objects. It’s durable, secure, highly available, and OpenSource friendly.

One distributed storage system system offers, blob, files, disks, tables and queues, across more regions than any other cloud.

Azure Files (Preview)

Originally launched for lift-and-shift. If you had a legacy LOB app that needed a file share, you deployed Files instead of a VM file server. It was not intended for end user access. Offers SMB 2.1 and SMB 3.0. And if offers encryption at rest.

Why File Servers?

People still do not store things in the cloud. OneDrive and SharePoint online aren’t for everyone. Reasons:

• App compat: file path lengths, etc.
• Performance: latency to the cloud is an issue for things like AutoCAD.

Customer Pain

They still want to use file servers, but they’re struggling:

• Cold data that must be kept
• Capacity management
• DR
• Backup/restore

Companies with branch offices have a multiplier effect of the above.

Value Prop

• Centralize file services in a managed cloud service
• Reduce complexity associated with server sprawl
• Preserve the end user experience – keep the file servers and performance

What it Does

A customer with a file server and the disk storage is a problem. Join the file server to a sync group in Azure Files. Older (actually all) files are moved to the cloud (transparent tiering with “stubs” on prem). If you lose the file server, you build a new one, add it into the existing Files namespace, and the meta data is downloaded. That means users see the shares/data very quickly. Over time, hot data is downloaded as files are used.

You can add another file server and join it to the same sync group, or create more. This synchronizes the files between the file servers via Azure Files (the master now).

Coming soon, not in the current preview), you can synchronize Azure Files from one Azure region to another for DR/performance reasons. You can than hang servers close to that region off of that copy, with inter-region sync if you need it. If one region dies, the file servers associate with it fail over to the other region.

Existing file server access doesn’t change.

If you are using Work Folders (HTTPS access to file shares from Windows, iOS or Android) then this continues to work with the file server.

Users can access file shares ove3r SMB/REST directly via Azure Files.

There is Azure Backup integration so you can backup your file shares in Azure without doing any backup at all on-prem. Killer!

Demo – Setup

He’s in the Azure Portal and searches for Azure File Sync. He clicks Create. Simple creation of entering name and resource group. Supports West US, Souteast Asia, East Australia, and West Europe today, but more will be added.

He’s already downloaded the MSI for the agent. Installs this on a file server. Today, you must installed Azure RM PowerShell but this will be folded into the agent install later. The file server is registered via an Azure sign-in. Then picks a subscription, picks a resource group, selects the Storage Sync Service. This requires another sign-in and a trust is created between the file server and Azure Files.

Back in the portal, he opens the sync service resource, and the file server is shown as Online, with OS version and agent version info.

He creates a sync group and associates it with a pre-created Azure File Share. There are no server endpoints – things we sync to the cloud from a file server, e.g. a path. You can synchronize multiples sets of folders, using sync endpoints as policy objects. You cannot sync the system root.

In the Azure File Share – Storage Account > Files – we can see the contents of the file share are now in Azure. He renames a file on the file server, and 2 seconds later it’s renamed in Azure.

Scenarios

• Multi-site sync
• Cloud tiering
• Direct cloud access
• Integrated cloud backup
• Rapid file server DR

Demo – Tiering & Rapid Restore

There are 2 sync groups. One of them has two file servers sycnrhonizing to it. One of them has a policy to keep 95% free space (not realistic but engineered for demo reasons). This means that you can control tiering, to ensure that there’s always at least a certain amount of free space on a file server. Server 2 has a policy to keep 10% free space.

Tiering takes time to quiesce. Attributes show if a file is offline (O) or in Azure. The icon also shows the file as being offline by being transparent.

Questions from the audience:

• About synchronized locking. Today, there is no lock sync. It operates like OneDrive. If there are two clashing writes, both will succeed. But, one will be written as a copy. MS knows that lock sync is a hot request.
• This has nothing to do with DFS-R. It uses something called the Microsoft Sync Framework that is around for over 5 years and is used by SQL Server.
• How is StorSimple affected? StorSimple is intended as on-prem storage in a single site. It uses blob storage which isn’t user accessible. Azure File Sync
• Is this in CSP? He’s not sure, but if it’s not, it will be soon.
• Are there file size limits, etc? There are file size limits but there are things being done. They’re published in the release notes. 5 file servers per sync group in the preview. 1 TB per file. They’ve tested up to approx 30 million files. The maximums will grow as they test during the preview.

Back to demo. He added a blank server to the sync group with contents. Meta data of the share/files appears almost instantly. That’s “rapid restore” in action:

• Add file share to a new file server
• DR scenario.

Talon Storage – Charles Foley

Customer: TSK that designs & fits out workplaces. They want as little on-prem IT as possible. Not a huge company. They had people in multiple locations with file servers, collaborating. They used Talon FAST in front of Azure Files, enabling sites to see a single share across sites. And this supports file locks in Azure Files, preventing the overwrite scenario.

Azure Files Use Cases – What’s New

Mine from Microsoft takes over.

Top Use Cases:

• Highly available FTP Server. Creating load balanced stateless FTP servers that use Azure Files to store shared content. Results in scalable and highly available FTP server.
• Store scripts in Azure Files instead of on a file server VM. SMB 3.0 encryption should be used in hybrid scenarios. Output sent to Azure Files and can be processed later on-prem.

New in 2017

• Security: Encryption At Rest using your own key (Key Vault), SMB encryption for Linux.
• End-to-end integration: Data import, a new tamper proof 100 TB disk device announced yesterday. Getting start tools for Windows and Linux. Export is coming.

Announcing Today

• Azure File Sync Preview
• Network ACLs Preview – secure your storage account with layer 4 firewall rules.
• Azure Monitor Preview to troubleshoot or manage performance

Coming soon:

• Share Snapshots Preview – a data consistent share snapshot
• Azure Backup Integration Preview – create policies to backup a storage account.
• LRS price reduction of 25%

Demo – Storage Accounts

She opens Files in a storage account. There are some shares. She shows that you can use Net Use or Sudo to connect to a file share over the network. She creates a snapshot. Then she views snapshots. Loads of them there already because Azure Backup is enabled. In the recovery services vault, she opens Backup Items. We can see shares in there. She adds another in the same Backup wizard as usual. A backup policy is selected.  We see that we can manually restore a share or a file. On a VM file server, she shows a mounted file share with files in it. She has also mounted a snapshot. Because of this method, Previous Versions in the file share can be used to view/mount snapshots.

Azure Backup is Azure Files Sync aware.

Retention up to 120 days. Storage costs are incremental. You pay per storage account being backed up.

EDIT:

I met with some of the Azure Backup team later in the week to discuss backup of Azure File Sync because the above system worried me. Here’s what I learned. The above system is just for the preview. The system will change when Azure File Sync goes GA:

• Backups will be to the recovery services vault
• Longer retention will be possible

Roadmap

• AD integration and ACLs
• Larger shares (~100 TB instead of 5 TB)
• Azure file sync GA
• Cross region sync of storage
• ZRS – sync writes across three availability zones

Questions

• Supported OS for File Sync: WS2012 R2 and WS2016. PCs are not affected because they connect to file servers.
• Expansion of file share max capacity will roll out to all existing shares.
• Any road map on compliance and legal hold? Bit of a woolly answer.
• Any character file path limits? Published publicly. Some characters are not supported, but they’re using telemetry to monitor that for future support. Non-compliant files are skipped, and an error is created on the server. Same happens with files that are too large.
• You can do around 10-20 sync groups per file server … that can be lots of shares.
• Deduplicated volumes are not support at this time, but they plan on adding support. They are investigating using dedupe to reduce transmission and storage costs.
• Egress charges: The Talon guy talks up. Their customer’s egress charges are under 1% of their total bill, in the 10s or 100s of dollars range.
• The file sync protocol is REST-based.

AzureStorageFeedback@microsoft.com for any feedback/questions.