Designing the Building Blocks for a Windows Server 8 Cloud

Speakers: Yigal Edery and Ross Ortega from Microsoft.

Windows Server 8 apparently is cloud optimized.  That rings a bell … I expect some repetition so I’ll blog the unique stuff.

There is no one right cloud architecture.  The architecture depends on the environment and the requirements.  Don’t take from this that there are no wrong cloud architectures   “Building an optimized could requires difficult decisions and trade-offs among an alphabet soup of options”.  This session will try provide some best practices.

Requirements

• Cost
• Scalability
• Reliability
• Security
• Performance
• High availability

Balance these and you get your architecture: workloads, networking, storage and service levels.

Which workloads will run in my cloud?

You need to understand your mission.

• Cloud aware apps or legacy/stateful apps? Are you IaaS or PaaS or SaaS?
• Are workloads trusted?  This is an important one for public clouds or multi-tenant clouds.  You cannot trust the tenants and they cannot trust each other.   This leads to some network security design decisions.
• Compute-bound or Storage-bound?  This will dictate server and storage design … e.g. big hosts or smaller hosts, big FC SAN or lower end storage solution.
• Workloads size?  And how many per server?  Are you running small apps or big, heavy apps?  This influences server sizing too.  Huge servers are a big investment, and will cost a lot of money to operate while they are waiting to be filled with workloads.

Networking

• Are you isolating hoster traffic from guest traffic?  Do you want them on the same cable/switches?  Think about north/south (in/out datacenter) traffic and east/west (between servers in datacenter) traffic.  In MSFT datacenters, 70% is east/west traffic.
• Will you leverage existing infrastructure?  Are you doing green field or not?  Green field gives you more opportunity to get new h/w that can use all Windows Server 8 features.  But trade-off is throwing out existing investment if there is one.
• Will you have traffic management?

Infiniband VS 10 GBE vs 1 GbE

10 GbE:

• Great performance
• RDMA optional for SMB 2.2
• Offers QoS (DCB) and flexible bandwidth allocation
• New offloads
• But physical switch ports are more expensive
• New tech appears on 10 GbE NICs rather than on 1 BgE

InfiniBand (32 Gb and 56 Gb):

• Very high performance and low latency
• RDMA includes for SMB 2.2 file access
• But network management different than Ethernet.  Can be expensive and requires a different skillset.  Can be hard to find staff, requires specific training.  Not many installations out there.

1 GbE:

• Adequate for many workloads
• If investing in new equipment for long life, then invest in 10 GbE to safeguard your investment

Price of WAN traffic is not reducing.  It is stable/stuck.  Datacenter north/south WAN links can be a fraction of the bandwidth of east/west LAN links.

How many NICs should be in the server? 

We are shown a few examples:

Physical Isolation with 4 NICs:

• Live Migration –1
• Cluster/Storage – 1
• Management – 1
• Hyper-V Extensible Switch – 2 bound together by Windows 8 NIC teaming, use Port ACLs for the VMs

Many people chose 10 GbE to avoid managing many NICs.  Windows Server 8 resloves this with NIC teaming so now you can use the b/w for throughput.

2 NICs with Management and guest isolation:

• Live Migration, Cluster/Storage, Management (all on different subnets) – 1
• Hyper-V Extensible Switch – 1 NIC, use Port ACLs for the VMs

1 * 10 GbE NIC:

• Live Migration, Cluster/Storage, Management all plug into the Hyper-V Extensible Switch.
• VMs plug into the Hyper-V Extensible Switch
• 1 * 10 GbE NIC for the Hyper-V Extensible Switch
• Use QoS to management bandwidth
• Use Port ACLs for all ports on the Hyper-V Extensible Switch to isolate traffic
• This is all done with PowerShell

Windows Server 8 NIC Scalability and Performance Features

• Data Center Bridging (DCB)
• Receive Segement Coalescing (RSC)
• Receive Side Scaling (RSS)
• Remote Direct Memory Access (RDMA)
• Single Root I/O Virtualisation (SR-IOV)
• Virtual Machine Queue (VMQ)
• IPsec Offload (IPsecTO)

Note: no mention of failover or Hyper-V cluster support of the features.  E.g. We don’t recommend TOE in W2008 R2 … not supported.

Using Network Offloads for Increase Scale

• NIC with RSS for native (parent) traffic: Live Migration, Cluster/Storage, Management
• NIC with VMQ for virtualisation traffic: Hyper-V Extensible Switch

Note: RSS and VMQ cannot be enabled on the same NIC.  RSS not supported on the Hyper-V switch.

• Raw performance: RDMA and SR-IOV:
• Flexibility and scalability: Hyper-V extensible switch, network virtualisation, NIC teaming, RSS, VMQ, IPsecTO

Notes:

• SR-IOV and RSS work together.
• Offloads require driver and possibly BIOS support.
• When you are working with 1 or restricted number of NICs, you need to pick and choose which features you use because of support statements.

Storage

HBAs VS NICs.  HBA (FC, iSCSI, or SAS) bypasses networking stack and has less CPU utilisation.

Storage Architectures

2 possible basic solutions:

• Internal/DAS disk: cheap with disk bound VMs
• External disk: expensive but mobile VMs, can grow compute and storage capacity on 2 different axis, compute bound VMs, storage offloading

The Great Big Hyper-V Survey of 2011 findings are that the breakdown in the market is 33% use A, 33% use B, and 33% use both.

Service Levels

• What performance guarantees do you give to the customers?  More guarantees = more spending
• How important is performance isolation?
• What are the resiliency promises?  This is the challenging one: in-datacenter or inter-datacenter. 

More on the latter:

• Some failure is acceptable.  You can offer cheaper services with storage/compute bound VMs.  Often done by hosters.  Windows Server 8 trying to offer mobility with non HA Live Migration.
• Failure is not acceptable: Failover clustering: make everything as HA as possible.  Dual power, dual network path, N fault tolerant hosts, etc.  Maybe extend this to another data center.  Often done in private cloud and legacy apps, rarely done by hosters because of the additional cost.  Windows Server 8 trying to reduce this cost with lower cost storage options.

Representative Configurations by Microsoft

Tested in MS Engineering Excellence Center (EEC).  Optimized for different cloud types.  Guidance and PowerShell script samples.  These will be released between now and beta.

Start with:

The traditional design with 4 NICs (switch, live migration, cluster, and parent) + HBA: physically isoated netwowkrs, HBA, and W2008 R2 guidance.

Enable Support for Dmeanding Workloads:

• Put Hyper-V switch on 10 GbE. 
• Enable SR-IOV for better scale and lower latency

Enable 10 GbE for Storage:

• Enable RSS
• Fast storage
• Ethernet so you have single skill set and management solution

Converge 10 GbE if you have that network type:

• Use the NIC for Live Migration, Clsuter/Storage/Management.  Enable QoS with DCB and RSS.  MSFT saying they rarely see 10 GbE being fully used.
• Switches must support DCB
• QoS and DCB traffic classes ensure traffic bandwidth allocations

Use File Servers:

• Share your VM storage using a file server instead of a SAN controller.  Use JBOD instead of expensive SAN.
• Enable RDMA on file server NIC and converged 10 GbE NIC on host
• RDMA is high speed, low latency, reduced CPU overhead solution.
• “Better VM mobility”: don’t know how yet

High Availability and Performance with 3 * 10 GbE NICs

• 2 teamed NICs for parent, cluster/storage, parent with DCB and RSS (no RDMA)
• File server has 10 GbE
• Hyper-V Switch and 10 GbE

Sample Documented Configuration:

• 10 GbE NIC * 2 teamed for Live Migration, Cluster/Storage, and parent with DCB, RSS, and QoS.
• 1 * 1 GbE with teaming for Hyper-V switch.
• File server with 2 by 10 GbE teamed NICs with RSS, DCB, and QoS.
• File server has FC HBA connected to back end SAN – still have SAN benefits but with fewer FC ports required and simpler configuration (handy if doing auto host deployment)

Damn, this subject could make for a nice 2 day topic.