Day 3: Invite Only Power Consumption Meeting

This is a research meeting for MS to drive strategy.

  • PC numbers vary between 10 and 55,000.
  • Laptop penetration: the biggest company here is 40% laptop, 60% desktop.  All of us above 25%.  Everyone allows roaming computing but some with corporate policies.
  • Everyone but me is on XP.  They’re waiting until Windows 7.  Good idea based on the features being offered: minimise project time to get those benefits.
  • The big company is green: carbon neutral factory using wind turbine power …. except when it isn’t windy of course :).  New data centres will reuse heat.  They own the 3 turbines and got the local power company to invest: excess power is exported to the power company.
  • There is agreement that there isn’t "green it".  Call it greener IT.
  • Data centres are at 19 degrees not for servers but for people.  Servers will run at 30 degrees Celsius with no problems.
  • Finland data centres are regulated into consuming more power, etc, e.g. underground which makes cooling harder.
  • The big company is a car company.  One of their 3 aims as a corporation is to be greener.  They take it very seriously.
  • Some companies using centralised power management, some not.
  • Measurement of power (watt meters per machine) 50% usage for tests.
  • Company politics are often an obstacle.
  • On the data centre we stressed the need to be able to optimise beyond the server: storage, network, all OS’s, etc.
  • Measurement: we suggested OpsMgr for monitoring H/W generated metrics and ConfigMgr.
  • Asked if we had interest in application virtualisation for server applications (not terminal services).  Interesting but there’s a trust and an OEM support issue.  Would have to be out a while and trusted.  Probably only for lightweight stuff that can exist on one server.  Removes compatibility issues.
  • Machine recycling?  Somewhat.  Ireland not much unfortunately.  Lack of trust on the disk data.  75% of the attendees donate to charity or given to dedicated recycle firms (paid service).  There is a story about recycled machines disappearing on a journey to recycle.  No idea if the disks were wiped or not.

Day 2: Windows Management and Fundamentals: Windows 7

This was by MS Jason Leznek and should have been called "I really liked what we did with Vista".  There was very little on Windows 7.  When he started demonstrating Powershell, half the room walked out … at once!  It was funny.

Here’s the Win7 promises:

  • Better device compatibility *
  • Better application compatibility *
  • Better performance: startup/shutdown/hibernate, UI responsiveness, Outlook performance.
  • Better power utilisation
  • Reliability
  • Security

* Notes: This is because it’s really Windows Vista R2 so the underlying security model will not change.  If it works on Vista, it should work on Win7.  Anyone skipping Vista hoping for an XP-like OS will be disappointed.  MS will not undo Vista.

The above are being referred to as the "6 dimensions of quality".

After that we got 45 minutes of Vista.  He then started of on Powershell and I joined the half the room that went for coffee.  This was the last session slot of the day.

Worst session I’ve been at since 2005.

Day 2: Virtual Machine Manager 2008 Technical Overview

Edwin Yuen is the speaker.

Most people here are running VMware ESX.  About 50% are either running or have tried Hyper-V.  Relevant?  Yes: VMM 2008 can manage ESX.  It will import your layout from Virtual Center for the VMM host layout/navigation groups.  You can aggregate multiple Virtual Centers – VMware still working on that.  This was allegedly the #1 request from customers – I’m guessing it was the usual Fortune 1000 that determine MS strategy.  Most SME’s won’t mix hypervisor platforms.  MS are not adding VMotion without Virtual Center.  You can manage it, resource pools, etc as if using Virtual Center.

Aim of virtualisation: it’ll be a skill not a speciality.  Integrated management.  Not only reduce, but have the best TCO/ROI.

VMM 2008 was available for download as of Monday.  Hosts support 24 cores now (6 Core Intel CPU with Hyper-V patch).

The library is just a file share.

About 15 people (50% full auditorium) are using VMM 2007.  New features: Hyper-V and ESX support.  Failover clustering in W2008.  Delegated admin.  PRO (integration with OpsMgr via a connector).

Deploying Hosts

Adding a new W2003 host installs Virtual Center automatically.  Adding a W208 host enables Hyper-V.  Adding a new W2008 host to a cluster also configures clustering automatically. 

Conversions

We can do P2V and reconfigure the "H/W" specifications as required.  It only copies the necessary files.  This is a pure clone.  The machine is not automatically turned on => avoid name/IP clash.

The host can be done live via VSS: WXP, W2003, Vista and W2008.

Self Service

Delegate to non-admins via a web page.  They are assigned access to VM’s and what can they do to them.  Even allow them to build machines.  You can assign a resource quota to the user.  There is delegation to the normal admin MMC but with restricted control.

Library

Stores VHD’s in a file share, ISO, sysprep answer files, offline VM’s, Powershell scripts and templates.

Intelligent Placement (of VM’s)

We can use load balancing or resource maximisation strategies out of the box.  We can tweak these.  The latter seems like a "green" solution…make the most of a host before adding VM’s to another host.

During deployment of a VM, you can save the entire job as a Powershell script in notepad.  The network copy of the VHD is done using BITS 2.0; very nice.

IP is run every time you move a VM.  Expected load for the VM is compared against all potential hosts.  If the server has space then it’s further analysed to score the hosts’ potential.  CPU, RAM, Network I/O and Disk I/O are compared.  Host reserve (configurable) is taken into this – e.g. assign 2GB RAM to a host (see my RAM calculation spreadsheet and postings). 

Powershell

  • 170+ command line functions.
  • Powershell is the foundation of the MMC and website.
  • All Powershell operations are logged and audited.
  • Citrix Xen Desktop integrates using Powershell.

PRO

Performance and Resource Optimisation.  Use OpsMgr to manage health and performance, e.g. SLA.  PRO is a OpsMgr connector.  VMM 2008 is added as a series of new management packs.  VMM now is instructed what to do in the case of an error or warning.  Human intervention required by default but we can automate using filters for severity or host. End-end management of all aspects of the server platform.

HP, Dell, Brocade, Quest and Tripwire are writing vendor specific PRO packs, e.g. Brocade monitors their SAN: a fibre channel link to a host gets overloaded so VM’s are migrated to another host.  Very, very sweet.  I didn’t expect this and I’m impressed.  Imagine if HP do this for their blades with Virtual Connect?  If a physical network mapped to a Hyper-V virtual switch dies then you can do get the virtual connect to re-map the NIC to another VC virtual network.

ConfigMgr 2007

We can offline patching/servicing of offline VM’s using WSUS.  Available now for VM 2007.  Will be available soon for VMM 2008. You ca also service sysprepped templates.

Purchasing

You can purchase this 3 ways:

  • As part of the Server Management Suite Enterprise CAL.  This also includes DPM, OpsMgr and ConfigMgr CAL’s.  This is a per host CAL – all VM’s get free CAL’s, hence the price.  It’s half the price of all for CAL’s alone.
  • Standalone
  • Workgroup (manage up to 5 hosts).

The VMM CAL is per host, no VMM.

Summary

"Virtualisation without good management is more dangerous than not using virtualisation in the first place" Tom Bittner, Gartner.

There will be a VMM vNext to coincide with Hyper-V V2.0 in Windows Server 208 R2 (2010).

Day 2: Windows Server 2008 Failover Cluster Troubleshooting & Tips

The speaker is David Dion from MS.

Windows Server 2008 is the last x86 release.  All nodes do not need to be exactly identical in W2008 Clustering.

Cluster Validation

Lots of problems in deployments of previous editions of Windows clustering (MSCS) were caused by configuration issues.  Cluster Validation tool resolves this.  Built into W2008.  Tests servers, OS and storage to check if the configuration is valid.  Should be run before cluster build or after adding node, adding drivers, patches, update firmware or BIOS (server or device), etc.  You can also run the validate tool as a troubleshooting tool – primary course of action.

Very easy to use; it’s just a wizard.  Best to run all of the tests.  However, doing all of the storage tests can take hours with hundreds of disks, e.g. a 16 node Hyper-V cluster.  A report is generated as an MHT file in IE.  You get pass, pass with warning or fail.  This is stored in the WindowsClusterReports folder. 

Do not assume the hardware configuration will be fine; run the validation utility to test it.

Concerns:

  • Validation of storage requires that the storage be offline. Beware for Hyper-V.  Schedule a full cluster maintenance window.
  • Running validate with a single node is pointless.

W2003 clustering required the H/W was on a clustering HCL.  Niche H/W, therefore expensive.  Everyone hated it.  Not used in W208.  The validation tool is your cluster certification.  Purchase gear with W2008 logo.  Run the tool and if you get a pass then you’re certified.  Keep a copy of the report for PSS.

MS recommends you purchase "Failover Cluster Configuration Program" solutions from vendors, i.e. the pricey niche solutions, e.g. a cluster kit.  Interestingly, HP is not one of the 9 partners in the program.  Dell and IBM are.

Event Viewer

Check MicrosoftWindowsFailover Clustering log.  Event logs are no longer replicated across all nodes in the cluster.  You should use the MMC to view events from all nodes.  You can also build event queries there.  You can filter events for applications and resources.  Because of this pooling of events, beware using the MMC remotely from the cluster and killing the WAN.  Normally we only see critical and warning events.  By enabling the operational "log" you can see information events.

Start with events if looking at non-configuration issues on the cluster.

Cluster Debug Logging

Lots of information and not user friendly.  The legacy cluster log file no longer exists.  Logging to to an event trace session: "Microsoft-Windows-FailoverClustering".  Log enabled by default.  You can produce a human readable log using "Cluster.exe log" command.

Tracrpt.exe can be used to dump the trace session.  .EVTX and view the file in event viewer.  .XML for you scripting freaks or to open in IE.  Cluster.exe can raise or reduce the level of logging 3 is default.  1 is low, 5 is high.  Running this command on one node configures all the nodes.  Changing the size of the file causes historical logs to be lost.  Copy them safely before doing this.  It’s quite verbose at level 5.  Running at level 3 (default) is recommended. 

This is the last logging solution you should pick.  Retaining 72 hours of data as a minimum is recommended.  What size of log is 72 hours?  How long is a piece of string.  File shares are quiet.  Exchange is noisy.  Hyper-V probably could be as well if VM’s are moving about.  Change the log size first, then set the required verbosity.  Cluster logs are always GMT time zone.  You’ll have to mentally map this when comparing with Windows Event Viewer if in different time zone to GMT.

Windows Server 2008 R2

  • Validation Tool includes best practices tests.  Quorum configuration, status of cluster resources, network name settings in multi-site cluster.
  • Performance Counters are added into perfmon for clustering.
  • There will be Powershell support.
  • There is a read only mode for the console.

Best Practices For Now

  • Try to use identical hardware on all nodes. Especially storage: HBA, firmware, driver, cables, etc.
  • Run the validation tool.
  • Don’t add resources to the Cluster Group or the Available Storage Group.
  • Keep regular system state backups.  This includes the cluster database automatically.
  • Use "preferred owners" and "possible owners" to balance the cluster.
  • Multi-site clusters are more complex so check out the MS site for a whitepaper.

Quorum:

  • Node and disk majority where there is shared storage.  Small disk – 512MB at least.  Only use it for the quorum. Use it as a GUI drive to discourage alternate usage.  No need to backup on the quorum.
  • Node and File Share Majority: use one file serve for many clusters but dedicate 1 share to each cluster.  OK to use a clustered file server but keep it in a different cluster (chicken and egg).  File server should be in the same forest as the cluster.  Avoid DFS namespaces.
  • More information available.

Old 2003 best practices that are gone:

  • You can add nodes as you want – nodes do not need to be powered off.
  • No NIC teaming restrictions any more.
  • No need to stagger boot times, e.g. w2003 required 30-60 second gaps.
  • Clustering runs as local system now.  No password to change for the service.
  • Keep an eye on the hotfixes page for clustering.

Day 2: A Tour Of The Exhibitors Hall

I started of talking to Jalasoft.  They produce extensions for OpsMgr 2007 to monitor things like network devices and other OS’s and applications.  My interest was in Cisco network devices.  License is around $100 per device.  There is a Jalasoft console purely for configuring connectivity to the monitored devices and identifying what ports to monitor and what rules to enable.  All monitoring activity and reporting is done in the OpsMgr console.  Looks nice.

I moved on to Ask The Experts: Hyper-V.  I raised my AV issue which raised some eyebrows – they want to swap some emails on the issue.  I also raised NIC teaming.  MS is working on fixing that.  No timetables.

Then on to HP.  I brought up HP’s Network Configuration Utility NIC teaming for Hyper-V.  It’s the #1 issue for their internal S/W development right now.  They are working with MS on resolving the issue.  They admitted poor communications with the public.  There will be announcements in a few weeks time.  It will be an industry solution rather than a HP solution.

I then talked to Beyond Trust (t-shirt!).  They provide a solution where no one needs to be an admin.  You use GPO to define tasks that require admin rights.  Anytime their agent sees this running, the task is elevated as an admin but still runs as the user.  All logging shows the user doing the work.  GPO is per user and per machine.  Using loopback you can get into some nice scenarios for controlling admin usage.

Day 2: System Center And The Data Centre

The exhibition hall isn’t open so here I am.  Much of what’s discussed is old technology but it’s good to get it all as one message – a nice refresher.  I’ve either deployed already or am doing it in the next couple of months.

Jeff Wettlaufer and Sacha Dawes from MS speaking.  The two big complaints they hear about is space and power management.  True enough – I can relate to that.

  • 2/3 of enterprises increasing data centre budget.
  • 82% of data centres track SLA’s
  • Data centres to consume 2% of USA power by 2009
  • 42% of data centres to exceed power capacity in 12 to 24 monts
  • 66% of data centres have already deployed virtualisation.
  • 81% of customers doing server consolidation in net 2 years.
  • 50% of European and USA enterprises adopted ITIL.
  • 10% of IT budgets for compliance.

System Center can assist with some of this:

  • Configuration management
  • End to end monitoring
  • Server compliance
  • Data protection and recovery

There’s too much time being spent in the presentation telling us what we already know – the pain points in the data centre.  It’s very repetitive.

Planning

The first step in deployment is modelling, i.e. System Center Capacity Planning.  It’s limited to SCOM and Exchange still.  Has been for years unfortunately.  That’s a powerful tool that should be more widely adopted by MS, e.g. Hyper-V.  We get a demo of Capacity Planner for OpsMgr 2007.  SCCP is a free download and having used it for Exchange 2007 I’d say it’s a bit clunky to use but well worth getting used to because it’s a powerful estimation tool for hardware specification and Exchange deployment.  There’s not too many of us who really need to use it for OpsMgr 2007 to be honest.

Deployment

Use System Center to deploy operating systems.  Same for Vista as for W2008.  Same for XP as for W2003.  You’ve got MDT, WDS or ConfigMgr.  Can also use these deployment tools for VM’s – better to use System Center VMM for template deployment.

Multicast was added in WDS – that’s the built into Windows Server imaging solution.  BTW, it’s very disk efficient on the WD server.  It’s creates file based images and uses single instance storage across images… hence a file is stored once on the server.  Ghost is sector based so you can’t do this.  But, WDS is limited to NTFS file systems.

Leverages "panther" the new installation routive introduced in Vista.  Steep learning curve but well worth doing.  If you’re here, talk to Rhonda Layfield in "Ask The Experts": she’s the queen of this stuff.

W2008 configuration is very easy to script and hence can be automated 100% using SCCM task sequences or by using an unattend script which runs after install using WDS.  Check my Server Manager document.  SCCM task sequences was the big coding project in the last release.  You know what that means: it’s going to be leveraged more in the future.  It lives on top of WDS so you’re using WIM images just like WDS, ImageX, MDT, BDD, etc.  Task sequences also allow you to deploy BitLocker and configure it via the wizard.  Useful for branch office W2008 servers in insecure locations.

Update Management

Deploy via WSUS.  ConfigMgr allows you to set up scheduled and recurring maintenance windows for reboots and integrates with the OpsMgr 2007 agent to avoid unnecessary alerts.  Check my blog for a scripted solution for WSUS and OpsMgr maintenance mode.  ConfigMgr allows integration with W2008 Network Access Protection to ensure compliance.  Advanced reporting on compliance and updates.  Desired Configuration Management can highlight absolutely critical updates not being installed with a tiny bit of customisation.

Consolidation = Virtualisation

Use OpsMgr 2007 and Virtual Machine Manager 2008 via PRO connector to audit potential hosts and VM’s.  ConfigMgr can audit hardware specs for potential hosts.  VMM is recommended if you run a few Hyper-V hosts and can also manage ESX … obviously so you can migrate to Hyper-V 😉  PRO connector also allows rating and recommendations of which hosts to best deploy a VM on using a simple star rating mechanism. ESX admins using Virtual Center will find it familiar to use.

End-End Monitoring

OpsMgr 2007 can monitor everything it can discover and offer best practice guidance.  It monitors everything from hardware to application and all points between, including the Hyper-V hypervisor.  Health, performance including in this section.  I can confirm from experience that it really works.  It can ID failing hardware (HP written free Proliant management pack) before the hardware fails.  I’ve replaced RAM proactively before it totally failed in the past.  MS bringing in Linux/UNIX support natively using cross platform extensions.  You can model "services" (ITIL point of view) using components: network devices, servers, disks, functions, synthetic session monitoring, etc.  If anything in the model fails – the owner of the "service" knows and can easily drill down to ID the issue.  SLA monitoring can sit on top of this.  Lots of reports on granular or SLA detail.  You can model your security to control access to the service (admin or operator or read only operator).  You can have granular control, e.g. monitored object classes, e.g. SQL, or a set of servers so: a DBA can monitor all SQL servers or an application owner can monitor all equipment that runs their "service".  This all uses management packs written by the vendors of the monitored products, e.g. HP, Dell, Citrix, MS development teams, etc.  There are 3rd party management packs and you can author your own (a true black art).

Compliance

Two products.  ConfigMgr Desired Configuration management allows you to define templates of what compliance is for your servers/applications.  You can author your own or download ones, e.g. SOX, HIPPA, EU data protection, for MS products.

OpsMgr Audit Collection Services sucks security log entries up in "near realtime" into a central database.  You can secure this so administrators have no access and only security officers/auditors have access.  You can run reports on all this stuff, e.g. who logged on using privileged accounts, where, when and what did they do.

Service Manager (probably end of 2009 or early 2010 – it keeps slipping) will tie the tools and their databases together.  It’s a work allocation tool – more than just a helpdesk.  Work allocation doesn’t mean more work for engineers – it can be allocation to automation solutions.

A new term: "configuration drift".  This is a measure of how far your services move away from desired configuration over time.

Release Dates: Config Mgr V3: 2010.  DPM V3 2010.  OpsMgr R2 2009.  Service Manager V1 2010.  Ops Mgr V4 2010.

Day 2: Ugh, My Head!

Hour 1 doesn’t have anything new that is grabbing my attention.  Just as well.  MS Ireland took the Irish delegates out to a local restaurant and the wine (when it eventually arrived) was flowing.  I’ll head to the exhibition hall to collect some t-shirts for an hour.

Have I mentioned this venue sucks?  They ran out of coffee at 8:30 this morning.  The guys staffing the desks just shrugged their shoulders.  And not a single taxi driver knows this city.  I can navigate it better than them and I’ve been here since Sunday.  You tell them "internacional convencions centre" and they pull a Manuel on you.  It’s the same with hotels.  Tip: Have a map with you and just point to where you want to go.