Day 4: Deploying VM’s Using VMM 2008

Michael Nystrom is the speaker.

I arrived in 5 minutes to late (work interfering!).  This guy needs to realise he has an audience.  He was muttering away to himself while using WSIM and MDT on a huge resolution screen.  These tools have tiny writing to begin with and are already confusing even for someone familiar with them.  I was half way down the room and couldn’t make out what he was doing.  A smaller resolution, some step-by-step bullet points in a powerpoint and a zoom utility would help – and stop muttering to yourself.  This is unfortunate because he appears to know this stuff inside/out but he’s not good at getting it across to a large room.

Hyper-V point: Make sure the time zone is set correctly on the VM.  Even if you disable all integration services, the VM will synch the clock with the host when it boots up.

It appears the concept he’s trying to get across is that you can have many machine profiles that specify virtual machine specifications.  You can use answer files created in WSIM to configure Vista or Windows Server 2008 installation and configuration.  Deployment of the VM’s and configuration of the OS would be almost completely automated.  These can be made available via the self-service portal in VMM 2008.  Non-host administrators can select the "templates" to deploy new VM’s.  The resources available to them are controlled by a quota.  VMM 2008 Intelligent Placement will pick a VM to locate the VM on a host. 

There’s no discussion here of the storage  side of things – Windows 2008/Hyper-V really relies on individual LUN’s per VM that must be sized appropriately for the individual VM.  W2008/VMM 2008 cannot deploy that storage for you.  This is a nice idea but for me, it won’t be a viable solution until Windows Server 2008 R2.  R2 will have a storage solution similar to VMware VMFS: a single LUN is accessible to multiple hosts and therefore can host many virtual machines.  LUN assignment is no longer an issue and doesn’t require SAN/server admin for each VM to be deployed.

Also, this solution appears to be a fresh install every time.  I think I’d prefer sysprepped VHD/template deployment because it would be slightly quicker.  This would be followed by "RunOnce" to run a script(s), e.g. unattended servermanagercmd.exe scripts.  However, Core does not really work well with RunOnce.

I think I’ll be waiting until the video is available to download and try this out myself  before I even attempt to discuss it further.  This guy’s presentation skills are pretty poor and he needs to get some training.

Day 3: Operations Manager 2007 Cross Platform Extensions

Installation

  • Integrated into 2007 R2
  • New pre-req checker for WS-Management 1.1
  • Unix/Linux agents are copied to the OpsMgr server

Configuration

  • Import management packs for the OS’s you use.
  • Limitation in the beta: the profile user account can only support 1 user in OpsMgr 2007 – Fixed in R2.  This account uses SSH for discovery and 1 diagnostic.  Monitoring is done through WS-Management.  You can limit this account to certain hosts to prevent crack attempts with unauthenticated health service installations.  This can even be filtered to objects or groups or classes in the OpsMgr database.
  • Create Run As Accounts
  • Create profiles

Discovery Wizard

  • Built on OpsMgr discovery framework
  • Fully integrated – choose between windows, Unix/Linux and network devices.
  • The discovery is moved inR2 into the Administration space of the console.  This was done to hide functionality from the operators.
  • For Unix we can search by IP, DNS name or IP range.  SSH is required on the box for this discovery otherwise it fails.  SSH not used if an agent already exists. 
  • There is functionality to allow for SSH via low privilege user and SU to root – tick box and an extra password.
  • Now we import the management packs – downloaded from the catalogue (when published).

What Can We Do Now?

  • We can monitor Application/Service, hardware, operating system (including daemons) and heartbeat.
  • Note only 6 daemons are monitored by default because every Unix/Linux box is different.  Bespoke discovery available.
  • Heartbeat: alert if machine down, DNS name changes or agent cert expires.
  • Discovers logical disks, NIC’s, processors, etc.
  • Monitors health, performance, utilisation, availability, etc.
  • Seems identical to Windows  monitoring on the face of it.   Completely integrated.
  • Knowledge is integrated in the MP.  The CPE team has Unix/Linux background and they hired real *nix administrators.
  • Log file (text search) monitoring: SU usage, root logon failure, critical authentication failures, break-in attempts, SSH authentication failure, successful login to root.  Completely extensible beyond this, e.g. you do this or third party.
  • New MP templates: log file and service (daemon).  You use these to create custom *nix monitoring.
  • Rails proved to be a PITA to monitor via text log.  A tool MS used to test is included in the rule wizard.
  • Using a template you can monitor process or daemons not included in the default set of 6.  You have to apply this to a server and maybe a group of servers.
  • Daemon checking happens every 5 minutes.
  • The fault resolution includes a hyperlink for applying the fix via Tasks; just like Windows.
  • If the WS-Man daemon (the "agent") goes offline then it can be restarted via SSH

Reporting

  • There are reports.
  • If you import reports but they’re not visible then refresh the site.
  • Reporting works just like with Windows, i.e. seamless.

How It Works

  • This works via a polling mechanism.
  • You can customise the polling times for different systems.
  • The providers on the monitored box filter events and then share with the polling management server.
  • The unofficial scalability: 200 Unix per management server was the aim.  They hit 600 before optimising the code.  No application management packs in that test.

Beta

  • Out now.
  • New one in a couple of weeks.

Application Monitoring

  • MS are not writing Oracle management packs, MySQL, etc.
  • MS leaving this to partners or vendors.
  • Authoring of management packs is the same as with Windows.
  • Discovery, scripting and data extraction is dependant on the application/OS.
  • Partners: Novell for SUSE – Samba, DNS, DHCP, etc.  Xandros – (lots of stuff including TomCat, MySQL, DB2, Sendmail, Oracle, Websphere and Apache.  Also building some Windows ones too). 

Notes

  • MS Serious about this.  2 people 18months ago.  37 now.
  • Note: F5 has management packs.
  • OpsMgr 2007 R2 RTM in Q2 2009 with CPE.

Day 3: Vista SP1 By Mark Minasi

There are 5 sessions that I’d like to go to at the same time during this time slot!!! Luckily the attendees get to download videos of the session.

I won’t be blogging this session because Mark makes a living from his copyrighted sessions – and he’s a friend.  Sorry!

Anyway – you should install SP1 because MS stats show crashes reduced since it was deployed.

Day 3: Microsoft System Center Virtual Machine Manager Advanced Features

The speaker is Edwin Yuen, Senior Technical Product Manager from MS.  This is a level 300 session compared to yesterday’s level 200.  See that post.

The vast majority of people in the room are running VMware.  Maybe 20% of those are looking at using VMM 2008 to manage ESX – hence get a better management experience, not necessarily as complete as with Hyper-V.

Here we go again on Powershell 😉  See previous posts.

The focus here will be on the management of Virtual Center using VMM 2008.  High Availability for Hyper-V using VMM – you need to know what to do in storage/Windows and what to do in VMM.  VMM can’t do everything, e.g. provision LUN’s.

VMware

Support for VMware covers:

  • VC 2.5
  • VC 2.0.1
  • ESX 3.5
  • ESX 3.0.2
  • ESX 3i *new to RTM*

VMM will be the manager of managers.  You can have many VC instances managed by a single VMM 2008.  Uses:

  • VI SDK API’s.
  • SFTP: File operations on ESX 3.5 and 3.0.2.
  • HTTPS: File operations on 3i

More features:

  • Secure mode is on my default.  This uses SSL for management using the VI SSL cert.
  • Host credential operations requires root SSH to be enabled. to move certain operations from "OK (Limited)" to "OK" (status of ESX host in console): power state, VM configuration, VMotion, Checkpoint, save state and migration.  Add credentials into the properties (security tab)  of the host in VMM to complete the configuration of the host.
  • Enabling PRO on ESX is possible – that surprises me to be honest and is impressive. You should not turn on PRO and DRS.  They will definitely conflict with each causing constant VMotion of VM’s.
  • There is a new network diagram view in the RTM release for 2008.
  • Do your host/VM installs in VC and then do day-day operations in VMM 2008.  Resource Pools are manageable within VMM.
  • You can do P2v and V2V of a VMware VM to Hyper-V.
  • Powershell can be used to manage VMware.
  • Do your VMware trouble shooting from within VC.
  • VMotion is referred to as Live Migration in VMM 2008.  Usable from within the console.  VC is still a requirement for VMotion.

Clustering Hyper V Step by Step

In server:

  • Configure Node (BIOS, Ent/DC per node), add failover clustering.
  • Storage: (iSCSI or Fibre Channel, Storage must suppoort persistent reservations, recommended 1 GUID LUN per VM).
  • Networking: 2 NIC’s recommended.
  • Add/remove nodes to/from cluster.

In VMM:

  • Add host cluster
  • VMM handles all future node additions/removals
  • Surface available disk

Clustering the VM i now a tick box in the properties of the VM.  Use intelligent placement strategy to place it on a suitable host.  A VM can be moved to a library but it retains the HA property for when you return it to the cluster, e.g. a template of a highly available VM.  If you tick the box on a VM that’s not on a cluster then you’re prompted by intelligent placement (IP) to move the VM to a suitable host.

Refresh the cluster in VMM after adding storage.

Day 3: Invite Only Power Consumption Meeting

This is a research meeting for MS to drive strategy.

  • PC numbers vary between 10 and 55,000.
  • Laptop penetration: the biggest company here is 40% laptop, 60% desktop.  All of us above 25%.  Everyone allows roaming computing but some with corporate policies.
  • Everyone but me is on XP.  They’re waiting until Windows 7.  Good idea based on the features being offered: minimise project time to get those benefits.
  • The big company is green: carbon neutral factory using wind turbine power …. except when it isn’t windy of course :).  New data centres will reuse heat.  They own the 3 turbines and got the local power company to invest: excess power is exported to the power company.
  • There is agreement that there isn’t "green it".  Call it greener IT.
  • Data centres are at 19 degrees not for servers but for people.  Servers will run at 30 degrees Celsius with no problems.
  • Finland data centres are regulated into consuming more power, etc, e.g. underground which makes cooling harder.
  • The big company is a car company.  One of their 3 aims as a corporation is to be greener.  They take it very seriously.
  • Some companies using centralised power management, some not.
  • Measurement of power (watt meters per machine) 50% usage for tests.
  • Company politics are often an obstacle.
  • On the data centre we stressed the need to be able to optimise beyond the server: storage, network, all OS’s, etc.
  • Measurement: we suggested OpsMgr for monitoring H/W generated metrics and ConfigMgr.
  • Asked if we had interest in application virtualisation for server applications (not terminal services).  Interesting but there’s a trust and an OEM support issue.  Would have to be out a while and trusted.  Probably only for lightweight stuff that can exist on one server.  Removes compatibility issues.
  • Machine recycling?  Somewhat.  Ireland not much unfortunately.  Lack of trust on the disk data.  75% of the attendees donate to charity or given to dedicated recycle firms (paid service).  There is a story about recycled machines disappearing on a journey to recycle.  No idea if the disks were wiped or not.

Day 2: Windows Management and Fundamentals: Windows 7

This was by MS Jason Leznek and should have been called "I really liked what we did with Vista".  There was very little on Windows 7.  When he started demonstrating Powershell, half the room walked out … at once!  It was funny.

Here’s the Win7 promises:

  • Better device compatibility *
  • Better application compatibility *
  • Better performance: startup/shutdown/hibernate, UI responsiveness, Outlook performance.
  • Better power utilisation
  • Reliability
  • Security

* Notes: This is because it’s really Windows Vista R2 so the underlying security model will not change.  If it works on Vista, it should work on Win7.  Anyone skipping Vista hoping for an XP-like OS will be disappointed.  MS will not undo Vista.

The above are being referred to as the "6 dimensions of quality".

After that we got 45 minutes of Vista.  He then started of on Powershell and I joined the half the room that went for coffee.  This was the last session slot of the day.

Worst session I’ve been at since 2005.

Day 2: Virtual Machine Manager 2008 Technical Overview

Edwin Yuen is the speaker.

Most people here are running VMware ESX.  About 50% are either running or have tried Hyper-V.  Relevant?  Yes: VMM 2008 can manage ESX.  It will import your layout from Virtual Center for the VMM host layout/navigation groups.  You can aggregate multiple Virtual Centers – VMware still working on that.  This was allegedly the #1 request from customers – I’m guessing it was the usual Fortune 1000 that determine MS strategy.  Most SME’s won’t mix hypervisor platforms.  MS are not adding VMotion without Virtual Center.  You can manage it, resource pools, etc as if using Virtual Center.

Aim of virtualisation: it’ll be a skill not a speciality.  Integrated management.  Not only reduce, but have the best TCO/ROI.

VMM 2008 was available for download as of Monday.  Hosts support 24 cores now (6 Core Intel CPU with Hyper-V patch).

The library is just a file share.

About 15 people (50% full auditorium) are using VMM 2007.  New features: Hyper-V and ESX support.  Failover clustering in W2008.  Delegated admin.  PRO (integration with OpsMgr via a connector).

Deploying Hosts

Adding a new W2003 host installs Virtual Center automatically.  Adding a W208 host enables Hyper-V.  Adding a new W2008 host to a cluster also configures clustering automatically. 

Conversions

We can do P2V and reconfigure the "H/W" specifications as required.  It only copies the necessary files.  This is a pure clone.  The machine is not automatically turned on => avoid name/IP clash.

The host can be done live via VSS: WXP, W2003, Vista and W2008.

Self Service

Delegate to non-admins via a web page.  They are assigned access to VM’s and what can they do to them.  Even allow them to build machines.  You can assign a resource quota to the user.  There is delegation to the normal admin MMC but with restricted control.

Library

Stores VHD’s in a file share, ISO, sysprep answer files, offline VM’s, Powershell scripts and templates.

Intelligent Placement (of VM’s)

We can use load balancing or resource maximisation strategies out of the box.  We can tweak these.  The latter seems like a "green" solution…make the most of a host before adding VM’s to another host.

During deployment of a VM, you can save the entire job as a Powershell script in notepad.  The network copy of the VHD is done using BITS 2.0; very nice.

IP is run every time you move a VM.  Expected load for the VM is compared against all potential hosts.  If the server has space then it’s further analysed to score the hosts’ potential.  CPU, RAM, Network I/O and Disk I/O are compared.  Host reserve (configurable) is taken into this – e.g. assign 2GB RAM to a host (see my RAM calculation spreadsheet and postings). 

Powershell

  • 170+ command line functions.
  • Powershell is the foundation of the MMC and website.
  • All Powershell operations are logged and audited.
  • Citrix Xen Desktop integrates using Powershell.

PRO

Performance and Resource Optimisation.  Use OpsMgr to manage health and performance, e.g. SLA.  PRO is a OpsMgr connector.  VMM 2008 is added as a series of new management packs.  VMM now is instructed what to do in the case of an error or warning.  Human intervention required by default but we can automate using filters for severity or host. End-end management of all aspects of the server platform.

HP, Dell, Brocade, Quest and Tripwire are writing vendor specific PRO packs, e.g. Brocade monitors their SAN: a fibre channel link to a host gets overloaded so VM’s are migrated to another host.  Very, very sweet.  I didn’t expect this and I’m impressed.  Imagine if HP do this for their blades with Virtual Connect?  If a physical network mapped to a Hyper-V virtual switch dies then you can do get the virtual connect to re-map the NIC to another VC virtual network.

ConfigMgr 2007

We can offline patching/servicing of offline VM’s using WSUS.  Available now for VM 2007.  Will be available soon for VMM 2008. You ca also service sysprepped templates.

Purchasing

You can purchase this 3 ways:

  • As part of the Server Management Suite Enterprise CAL.  This also includes DPM, OpsMgr and ConfigMgr CAL’s.  This is a per host CAL – all VM’s get free CAL’s, hence the price.  It’s half the price of all for CAL’s alone.
  • Standalone
  • Workgroup (manage up to 5 hosts).

The VMM CAL is per host, no VMM.

Summary

"Virtualisation without good management is more dangerous than not using virtualisation in the first place" Tom Bittner, Gartner.

There will be a VMM vNext to coincide with Hyper-V V2.0 in Windows Server 208 R2 (2010).

Day 2: Windows Server 2008 Failover Cluster Troubleshooting & Tips

The speaker is David Dion from MS.

Windows Server 2008 is the last x86 release.  All nodes do not need to be exactly identical in W2008 Clustering.

Cluster Validation

Lots of problems in deployments of previous editions of Windows clustering (MSCS) were caused by configuration issues.  Cluster Validation tool resolves this.  Built into W2008.  Tests servers, OS and storage to check if the configuration is valid.  Should be run before cluster build or after adding node, adding drivers, patches, update firmware or BIOS (server or device), etc.  You can also run the validate tool as a troubleshooting tool – primary course of action.

Very easy to use; it’s just a wizard.  Best to run all of the tests.  However, doing all of the storage tests can take hours with hundreds of disks, e.g. a 16 node Hyper-V cluster.  A report is generated as an MHT file in IE.  You get pass, pass with warning or fail.  This is stored in the WindowsClusterReports folder. 

Do not assume the hardware configuration will be fine; run the validation utility to test it.

Concerns:

  • Validation of storage requires that the storage be offline. Beware for Hyper-V.  Schedule a full cluster maintenance window.
  • Running validate with a single node is pointless.

W2003 clustering required the H/W was on a clustering HCL.  Niche H/W, therefore expensive.  Everyone hated it.  Not used in W208.  The validation tool is your cluster certification.  Purchase gear with W2008 logo.  Run the tool and if you get a pass then you’re certified.  Keep a copy of the report for PSS.

MS recommends you purchase "Failover Cluster Configuration Program" solutions from vendors, i.e. the pricey niche solutions, e.g. a cluster kit.  Interestingly, HP is not one of the 9 partners in the program.  Dell and IBM are.

Event Viewer

Check MicrosoftWindowsFailover Clustering log.  Event logs are no longer replicated across all nodes in the cluster.  You should use the MMC to view events from all nodes.  You can also build event queries there.  You can filter events for applications and resources.  Because of this pooling of events, beware using the MMC remotely from the cluster and killing the WAN.  Normally we only see critical and warning events.  By enabling the operational "log" you can see information events.

Start with events if looking at non-configuration issues on the cluster.

Cluster Debug Logging

Lots of information and not user friendly.  The legacy cluster log file no longer exists.  Logging to to an event trace session: "Microsoft-Windows-FailoverClustering".  Log enabled by default.  You can produce a human readable log using "Cluster.exe log" command.

Tracrpt.exe can be used to dump the trace session.  .EVTX and view the file in event viewer.  .XML for you scripting freaks or to open in IE.  Cluster.exe can raise or reduce the level of logging 3 is default.  1 is low, 5 is high.  Running this command on one node configures all the nodes.  Changing the size of the file causes historical logs to be lost.  Copy them safely before doing this.  It’s quite verbose at level 5.  Running at level 3 (default) is recommended. 

This is the last logging solution you should pick.  Retaining 72 hours of data as a minimum is recommended.  What size of log is 72 hours?  How long is a piece of string.  File shares are quiet.  Exchange is noisy.  Hyper-V probably could be as well if VM’s are moving about.  Change the log size first, then set the required verbosity.  Cluster logs are always GMT time zone.  You’ll have to mentally map this when comparing with Windows Event Viewer if in different time zone to GMT.

Windows Server 2008 R2

  • Validation Tool includes best practices tests.  Quorum configuration, status of cluster resources, network name settings in multi-site cluster.
  • Performance Counters are added into perfmon for clustering.
  • There will be Powershell support.
  • There is a read only mode for the console.

Best Practices For Now

  • Try to use identical hardware on all nodes. Especially storage: HBA, firmware, driver, cables, etc.
  • Run the validation tool.
  • Don’t add resources to the Cluster Group or the Available Storage Group.
  • Keep regular system state backups.  This includes the cluster database automatically.
  • Use "preferred owners" and "possible owners" to balance the cluster.
  • Multi-site clusters are more complex so check out the MS site for a whitepaper.

Quorum:

  • Node and disk majority where there is shared storage.  Small disk – 512MB at least.  Only use it for the quorum. Use it as a GUI drive to discourage alternate usage.  No need to backup on the quorum.
  • Node and File Share Majority: use one file serve for many clusters but dedicate 1 share to each cluster.  OK to use a clustered file server but keep it in a different cluster (chicken and egg).  File server should be in the same forest as the cluster.  Avoid DFS namespaces.
  • More information available.

Old 2003 best practices that are gone:

  • You can add nodes as you want – nodes do not need to be powered off.
  • No NIC teaming restrictions any more.
  • No need to stagger boot times, e.g. w2003 required 30-60 second gaps.
  • Clustering runs as local system now.  No password to change for the service.
  • Keep an eye on the hotfixes page for clustering.