The Worst IT Project I Was Ever A Part Off

This post will discuss a failed project that I was brought into and what I observed and learned from that project. It’s a real scenario that happened years ago, involving an on-premises deployment.

Too Many Chefs Spoil the Broth

Back in 2010, I joined a Dublin-based services company. The stated intention from the MD was that I was to lead a new Microsoft infrastructure consulting team. As it turned out, not a single manager or salesperson in the company believed that there was any work out there in Microsoft infrastructure technology – really! – and it never really got off the ground. But I was brought into one customer, and this post is the story of that engagement.

It was a sunny, cold day when I drove out to the customer’s campus. They are a large state-owned … hmm … transport company. I had enough experience in IT to know that I was going to be dealing with strong personalities and opinions that were not necessarily based on fact. My brief was that I would be attending a meeting with all the participants of a failing Windows Server 2008 R2 Hyper-V and System Center 2008 R2 project. I came in and met a customer representative – the technical lead of the project. He immediately told me that I was to sit in the corner, observe, and not talk to any of the participants from the other services providers. Note that the last word is plural, very plural.

I sat at the far corner of a long board room table and in came everyone. There was the customer IT managers and tech staff, the storage manufacturer (HP – now HPE) and their partner, the networking manufacturer (Cisco) and their partner, a Microsoft Premier Field Engineer, the consultants that implemented Hyper-V, the consultants that implemented the System Center management of Hyper-V, and probably more. Before I continue, I think that the Hyper-V cluster was something like 6 nodes and maybe 50-100 VMs.

Quickly it became evident that this was the first time that any of the participants in the meeting had talked to each other. I should re-phrase that: this was the first time any of the participants in deploying the next-generation IT infrastructure for running the business had been allowed to talk to each other.

  • A new W2008 R2 Hyper-V cluster was built. Although the customer was adamant that this was not true (it was), a 2-site cluster was built as a single-site cluster. There was a latent link between the two sites, with no control of VM placement and no third-site witness.
  • HP P4000 “Lefthand” module-based iSCSI storage was used without any consideration of persistent iSCSI reservations – a common problem in the W2008 R2 era, where the volumes in the SAN would “disappear” from the cluster because the scale-out of NICs/CSVs/SAN nodes went beyond the limits of W2008 R2 – a result of poor understanding of storage performance and Hyper-V architecture.
  • I remember awful problems with backup. DPM was deployed by a consulting firm but configured by a local staff member. He had a nightmare with VSS providers (HP were awful at this) and backup job sizing. It was not helped by the fact that backup was an afterthought in Hyper-V back then, not resolved really until WS2012 when it became software-defined. This combined with how the P4000 worked, the multi-site cluster that wasn’t, and Redirected IO caused all sorts of fun.
  • VMs would disappear – yup the security officer insisted that AV was installed on each host and it scanned every folder, including the CSVs. They even resisted change when presented with the MS documentation on scan exceptions that must be configured on Windows Server roles/features, including Hyper-V.

These were just a few of the technical issues; there were many more – inconsistent or missing patching, NIC teaming issues, and so on. I even created a 2-hour presentation based on this project that I (unofficially) called “How to screw up a Hyper-V project”.

My role was to “observe” but I wanted this thing fixed, so I contributed. I remember I spent a lot of time with the MS PFE on the customer site. He was gathering logs on behalf of support and we shared notes. Together we identified many issues/solutions. I remember one day, the customer lead shouted at me and ordered me back to my desk. I was not there “to talk to people but to observe”. The fact that I was one of two people on site that could solve the issues was lost on him.

The customer’s idea of running a project was to divide it up into little boxes and keep everyone from talking to each other. Part of this was how they funded the project – once it went over a certain monetary level it had to be publicly tendered. They had their preferred vendors and they went with them, even if they were not the best people. This created islands of knowledge/expertise and a lack of a vision. The customer thought they could manage this, and they were wrong. Instead, each supplier/vendor did their own thing based on assumptions of what others were doing and based on incorrect information shared by the customer’s technical team. And it all blew up in the customer’s face.

In the end, I heard that the customer blamed the software, the implementors, and everyone else involved in the project but themselves. They scrapped the lot and went with VMware, allegedly.

Lessons Learned

I think that there were three major lessons to be learned from this project. I know that these lessons apply equally today, no matter what sort of IT project you are doing, including on-premises, hybrid, or pure cloud.

The Business

IT enables or breaks the business. That’s something that most boards/owners do not understand. They think of IT as the nerds playing Doom in a basement, with their flashing lights and whirring toys. Obviously, that’s a wrong opinion.

When IT works, it can make the IT faster, more agile, and more competitive. New practices, be they operational or planning, can change IT, but I’ve even read how SCRUM/Agile concepts can even be brought to business planning.

Any significant IT project that will impact the business must start with the business. Someone at the C-Level must own it, be invested in it, and provide the rails or mission statement that directs it. That oversight will force those involved in the project to operate correctly and give them guidance on how to best serve the business.

Architecture

Taking some large-impact IT project and treating it as a point solution will not work. For example, building an entirely new IT infrastructure without considering the impact of or the dependencies on networking is stupid! You cannot just hand-off systems to different vendors and wish them bon voyage. There must be a unified vision. This technical vision starts with the previously mentioned business vision that guide-rails the technical design. All components that interconnect and have direct/indirect involvements must be designed as a whole.

Unified Delivery

The worst thing one can do is divvy up IT infrastructure to 5 or 6 vendors and say, you do that, and I will participate in a monthly meeting. That’s not IT! That’s bailing out on your responsibility! IT vendors can play a role, when chosen well. But they need a complete vision to do their job. And if they cannot get that from you, they must be allowed to help you build it. If your IT department’s role is to manage outsourcing contracts and nothing more, you have already failed the business and should just step aside.

A unified delivery must start with internal guidance, sharing the complete vision with all included parties, internal and external, as early as possible. Revealing significant change that you are working on with Vendor A 6 months into a project with Vendor B is a fail. Isolating each of the vendors is a fail. Not giving each vendor clear rules of engagement with orchestrated interaction is a fail. The delivery must be unified under the guidance of the architect who has a complete vision.

Bad IT Starts at The Top

In my years, I’ve done plenty of projects, reviewed many customer’s IT systems, and worked as a part of IT departments. Some of them were completely shocking. A common theme was the CIO/CTO: typically, an accountant or finance officer who was handed the role of supervising IT because … well … it’s just IT and they have a budget to manage. Someone who doesn’t understand IT, hires/keeps bad IT managers, and bad IT managers hire bad IT staff, make bad IT decisions, and run bad IT projects. As the saying goes, sh&t rolls downhill. When these bad projects are happening to you, and you run IT, then you must look at the mirror and stop pointing the finger elsewhere.

And before you say it, yes, there are crap consultants too 😊

 

 

Working From Home – The Half Year Update

It’s July now, and it’s just over half a year (just over 7 months to be more accurate) since I started a new job where I work from home. Here are some thoughts and lessons from my experience.

The Change

In my previous job, I was mostly in the office. I had a certain amount of flexibility where I could work from home to get things done in peace and quiet, away from the noise of an open-plan office, but mostly I commuted (by car, because there is no public transport option from here to the outskirts of Dublin where most private sector employees work) with my wife every morning and evening. We worked together – having met at the office – we were “podmates” for years before she moved desks and we started going out. I was happy in that job, and then along came Innofactor Norway who offered me a job. A big thing for me was not to travel – we have a young family and I don’t want to miss life. It was agreed and I changed jobs, now working as a Principal Consultant, working mostly with Azure networking and security with large clients and some internal stuff too.

Work Environment

This is not my first time having a work-from-home job. I worked with a small company before where they eliminated the office from expenses to keep costs down. It suited everyone from the MD down that we could work from home or on the road – one salesman used to camp out in a hotel in west Dublin, order coffee all day, and use their WiFi while having meetings close to potential customers’ offices. I worked from my home in the midlands. I was single then and worked long hours. I remember one evening when I’d ordered a pizza. The poor delivery guy was stuck talking to me, who hadn’t spoke to anyone in a week, and was trying to get away from this overly chatty weirdo. I had a sort-of-office; it was a mess. I ended up getting into the habit of working from the sitting room – which was not good. It was uncomfortable and there were entertainment distractions.

And important part of this job was going to be the work environment. I have an office in the house, but had rarely used it. A lot of my writing work for Petri or even writing courses for Cloud Mechanix was actually done on the road, sometimes literally while in the car and waiting for my daughter outside ballet or gymnastics classes. The office needed work so I cleared it out completely, painted and redecorated, and then set up all the compute from scratch. A nice fast PC with dual monitors was isntalled. I added a smart TV from the new year’s sales to the wall for YouTube/conference streams, as well as an extra screen (the MS adapter is much better than built-in screen casting). For work I bought a new Surface Laptop 2 (black, 16 GB RAM, i7) with the Surface Dock. And the folks in my old job gave me a present of a StarTech 2 port KVM switch, to allow me to share the keyboard and monitors between my NUC PC and Surface Laptop. The Laptop is my primary machine with Microsoft Whiteboard and the pen being hugely important presenting tools. The desk in the above picture is gone – it was purchased 3 years ago as a temporary desk when we moved into our house. I replaced it with a white IKEA L-shaped desk because the old one was squeaky and annoying me during Teams calls. Next to be replaced will be the chair.

Teams & Browser

The two tools I use the most are Microsoft Teams and the browser.

Teams is the primary communications tool at work. Very little internal comms use email. Email tends to be for external comms or some sort of broadcast type of message. I work with another Irish MVP, Damian Flynn. We’ve worked on a couple of big projects together, which has been awesome, and it’s not unusual for us to have a window open to each other for most of the day, even if we’re working on different things. It’s like we’re “podmates” but with half the width of Ireland between us. I’ve really noticed the absence this week – Damian is on vacation and most of Norway is on vacation too. I’m starting my second week with barely any conversation during the workday.

Working in cloud, obviously the browser is important. I started off in Chrome. Then I discovered identity containers in Firefox and I switched. However, I had to use Edge too – our time keeping is (how we bill each hour) is done in CRM and it won’t work in a non-MS browser. I was having problems with Teams randomly dropping calls. I even was considering switching to Slack/Skype (consumer) for my work with Damian. Then I realised that I was running out of RAM and that was killing Teams calls. The culprit? Firefox was eating many GBs of RAM. I had started to play with the Chromium Edge preview (aka ChrEdge) and was impressed by the smoothness of it (particularly scrolling for reading). When I realised how it had implemented identities (which is not perfect) I made the switch. Now ChrEdge is the only browser running all of the time and Teams is not losing calls.

Of course, I do a lot of JSON, so VS Code is nearly always open all of the time too. With that, Git client, and Azure DevOps, I have a way of doing things at scale, repeatedly, and collaboratively.

Work Habits

I am quite strict with myself, as if I am in an office, even though “my office” is half way up the west coast of Norway. I’ve heard presenters on home working talk about getting on a bike or in a car to drive around the block and “come to the office” with a work mentality. I’ll be honest, I might not even put on pants in the morning (kidding!) but I do clock in at 8:00 every morning. By the way, Norwegians start work before most of Europe is even awake. I’m the “late riser” in the company. Sure it’s 9am Norwegian time when I start, but they’ve already been at work for hours when I start. I have a routine in the morning, getting the kids ready, having breakfast, and bringing a coffee into my office. I have a morning start routine when there isn’t an immediate call, and then I’m straight into it. My lunch routine involves a bike ride or walk, weather permitting, and a quick lunch, and then I work the rest of my hours. I try to work longer hours at the start of the week so I can finish early on Friday – it’s amazing how a 2 day weekend feels like 3 days when you can finish just a little bit earlier.

More Than Just Work

Being at home means I’m also here for my kids. I can pick up our youngest earlier from day care. Our eldest can be brought to training or football games – and for checkups after breaking her arm – without taking time off from work. All are nice perks from working from home and having flexible hours/employer. This week, my wife is away at Microsoft Inspire. Without the time waste of a commute, I can use that time to do work around the house and keep on top of things. Soon enough, our youngest will start pre-school and I will be able to bring her there and back again. And when school starts, she’ll be walked to the end of our road, where the school is, and back again. Even when things are normal, there are times when I’ll have dinner started before my wife gets home – my culinary knowledge and skills are limited so I won’t subject my poor family to me cooking every day!

I’ve mentioned that I am using my lunch break for exercise. I’m overweight – like many in the business – a byproduct of sitting 8-10 hours a day. I started cycling 2-3 times a week spring of last year and put the bike away for the winter. It came out again this spring, and I was on it 2 times per week. II needed more time on it, so I’ve started going out every day. Sometimes I’ll do a long/fast walk to break it up, giving me a chance to catch up on podcasts and audio books.

More Productive & Less Stress

With less time wasted in traffic jams and more time focused on doing productive work, I am sure that I am more productive at home. Study after study have documented how bad open plan offices are. In the quiet of my office, I can focus and get stuff done. In fact, this time lead to me doing some business changing work with Damian. I’m less stressed than ever too.

If you can find a way to work, have reliable broadband, and have an employer that has realized how a happier employee is a more productive one, then I couldn’t recommend this style of working enough.

 

I’m Speaking At IP Expo in Manchester This Week

I will be in Manchester, UK, this week. I will be presenting an updated version of my “Solving the Azure Storage Maze” talk during the Altaro-sponsored slot at 12:45 on Wednesday in the Infrastructure Modernisation Theatre of IP Expo – registration is free.

If you have ever struggled with understanding all the storage options (not including databases – because I have 30 minutes, not 1 day) in Azure then I will help you navigate through all of the options. Note that I have updated this talk with Premium Blob and Premium Files content.

I Have Joined Innofactor

As I posted before, I finished my previous job with MicroWarehouse before Christmas. On January 3rd, I joined Innofactor.

Who Are Innofactor?

Innofactor is an IT consulting and services company that operates in the Nordic countries: Finland (HQ), Denmark, Sweden, and Norway. If you attend conferences or user groups, then there’s a good chance you’ve seen some of their employees speaking; Innofactor has a number of MVPs on their books including (but not limited to) Damian Flynn (also Irish), Olav Tvedt, Alexander Rodland, and Stefan Shorling. Quite simply, Innofactor is the A-Game in the Nordic countries, if not Europe.

I’ve known of Innofactor for quite a while. Lumagate was the company I knew first, but they were acquired by the Finnish company in 2016. My first contact with Lumagate was through Kristian Nese, now a part of AzureCAT in Microsoft. Kristian was the technical reviewer of Private Cloud Computing which I co-wrote (I did a tiny part) with other MVPs, including Damian Flynn. Damian joined Innofactor a few years ago – I’ve known Damian since he first became a Hyper-V MVP many moons ago. More often than not, we are roommates at the MVP summit. I got to know Olav through bringing him to Ireland to run EMS training for MicroWarehouse’s customers. It’s through these connections, and eventually meeting some of the other MVPs that I got to know what Innofactor is and wants to do.

My Role

My title is Principal Consultant, but I’m doing more than just project work. I don’t want to discuss that publicly yet … it’s a strategic thing 🙂 I’ll say this: the project work I have done so far has been quite cool, and the other thing I’m doing gives me an opportunity to work with great people across the Nordic countries. I’ve had a lot of meetings in my first 7 working days and we’ve figured out a lot of stuff that I thought might have taken weeks or months.

I am not moving to Norway. In fact, I spent a few days over the Christmas holidays working on my home office. It’s not finished yet – I want to put in a new desk and do a little bit more painting, but it’s getting there. Yes; I am working from home – the commute is wicked tough 😉 Almost all of my communications and collaboration is done through Microsoft Teams, something one of the directors pushed out before I joined. I was skeptical at first but it works well. And because the work we do is in The Cloud, we can do it from anywhere.

I said Innofactor was the A-Game. If you’ve seen any of the consultants present, then you know what I’m talking about. The conversations that I’m having on a daily basis are all state-of-the-art. What lies ahead of me will be both challenging and amazing.

Culture Difference

I have worked for American, German and Irish companies, including corporates, finance, and startup/small/medium businesses. My last employed, MicroWarehouse, was very flexible – the MD is very supportive of staff and deals with family issues in a thoughtful manner. But I have never experienced anything like working for a Norwegian company – maybe it’s an Innofactor thing, maybe it’s a Nordic or Scandinavian thing, I don’t know. But they take life-work (notice the order) balance very seriously. There is work, but most of your time is life. Don’t get me wrong, work has to be done and it has to be done well, but there is no doubt in your communications that no one is expecting 18 hour days.

Working from home has huge advantages, especially when you combine it with flexitime. I start work at 8 am most days, earlier on Fridays, and I’m able to finish earlier. I have more time at home with my family, I am here in case something needs to be delivered or done at the house (very useful last week when a new heating furnace was required in an emergency), and I can use my flexitime to deal with things for the kids like dental appointments, sports, or events. It really is life changing.

A hard part of working from home is being disciplined. It is so easy to say “I’ll work in the sitting room and stick on the TV for background noise”. Next thing you know, you’re binge-watching Netflix! I’ve heard home workers offer all kinds of solutions. Some get dressed for work, get in their car, drive around the block and then start work in their home! I think the most important thing to do is to create a workplace.

The small (box) room in our house was set up as an office when we moved in. It required some work over the holidays, but I tidied it up and got it ship-shape for the new job. This is where I work – nowhere else. When I come in here, the door shuts behind me and I am in work mode. I even use a “work laptop”, not my personal PC. That virtual line puts me in work mode. And it works – after 7 working days, today was actually the first time I’ve ever logged into Facebook on this machine.

Future

The scale of work and types of customers that I am working with are very different than I did over the previous 7 years. Customers are bigger, and they are what we in MicroWarehouse called “end customers”, not partners/resellers. This means I’ll have new things/scenarios to talk about. Already, I’ve got some new talks ideas floating around my head 🙂

Leaving MicroWarehouse

It’s with great sadness that I am announcing that this is my last day at MicroWarehouse, a company that I have enjoyed working with for over 7 years.

I joined MWH in 2011. My role was to work with Irish Microsoft partners, mostly in the small/medium business space, to grow the System Center business. My work with Hyper-V was a gateway to System Center and, at first, this worked … until Microsoft changed the licensing of System Center & enterprise agreements which killed that business for us.

Nearly 5 years ago my role changed from on-premises to The Cloud, when Microsoft asked us to take the lead on growing Azure business in the breadth market. I logged into Azure for the first time and started learning– for 9 months before I spoke to my first customer. And I’ve been learning every day since because that is the nature of Azure.

The great thing about working for MicroWarehouse was the people. The company has a family feel about it. We are literally thrown out of the office if we’re still in the building at 17:45! And the MD gives out to us for answering email at night! The company had my back when I went through some bad times, giving me time off to deal with things. I can even say that MWH changed my life, because one day this new hire sat at the desk beside me, and eventually married me 😊

The staff in MWH are the very best at what they do. The sales team totally know their stuff, and no one can match them. Any little tricks I know about licensing come from Rob, Angela, and Nicole. Our core sales team knows every little nook and cranny – you want to know about Surface or it’s accessories then they know the lot. The accounts and logistics team are all over everything – they sort out so many problems in Microsoft that customers could never comprehend – they’re the ninjas behind the scenes. And, of course, there’s the Marketing team; every time I’ve been speaking at an MWH event, training course or webinar, they’re the ones running things.

In short, if you are a Microsoft partner operating in Ireland, Northern or Republic of, then there’s no better choice than MicroWarehouse as your distributor or CSP Indirect. We’ve gone up against the best the UK has to offer and they are no match – there’s a reason we crushed the competition! Even though my time here is ending, I hope that MWH continues to thrive.

I’ve also made lots of friends in the Microsoft and MS partner world. My job was to engage with partners and that’s what I did … a lot. I’ve met some impressive people over the years and I’ve learned a lot from our customers too. I’m going to miss that.

I don’t view my departure as me leaving MWH. I didn’t look to leave, I never entered any job hunting process. I was really happy here. Instead, I’m going to something … I’ll share that soon.

Thank you MicroWarehouse. I’ve had an amazing time.

Not A Hyper-V MVP Anymore

It’s with some sadness that I have to report that I am no longer a Hyper-V MVP.

11 years ago, I got and email to say that I had been awarded MVP status … in System Center Configuration Manager. Yes, I used to do a lot of stuff on ConfigMgr. But by the time I’d been awarded, that had all stopped and I had refocused on server stuff, particularly virtualization and especially Hyper-V. A year later, my expertise was changed to that of Hyper-V, which later merged into a larger grouping of Cloud & Datacenter Management.

Being a Hyper-V MVP changed my career. I had early access to information and I was able to pose questions about things to my fellow MVPs and the program managers of Hyper-V, Failover Clustering, networking, and Windows Server storage. I learned an incredible amount, and the many posts on this site and my books all had input from my time as an MVP. Job openings appeared because of the knowledge I obtained, and I got to write for Petri.com. And being an MVP opened up speaking opportunities at many events around the world, including TechEd Europe and the very first Ignite.

There’s so many people to thank from over the years. I won’t name names because I’ll offend someone  because I’ll surely forget someone. My (ex-)fellow Hyper-V MVPs are an awesome bunch. We all found are niche areas and I can remember many times we’d meet at a user group event and pool our knowledge to make each other better. In particular, I remember speaking at an event in Barcelona during the build-up to WS2012 and spending hours in a meeting room, going over things that we’d learned in that dizzyingly huge release.

I want to thank the Program Managers in Windows Server, Hyper-V, Failover Clustering & Storage, and Networking for the many hours of deep dive sessions, the answers they’ve given, the time they’ve taken to explain, the tips given, and the opportunity to contribute. Yes, I got a lot out of being a Hyper-V MVP, and I love looking at the feature list and thinking to myself, “me and <person X> were the ones that asked for that”. The PMs are a patient bunch … they have to be to deal with the likes of me … but they’re the ones that make the MVP program work. I’d love to tell stories, but you know … NDAs Smile

I knew that this day when I’d stop being a Hyper-V MVP was coming. Actually, that suspicion started back in the WS2012 era when I saw where MS was going with Hyper-V. The product was evolving for a market that is very small in Ireland. I knew I had to change, and that was triggered when Microsoft Ireland came to our office at work, and asked us to help develop the Azure business with Microsoft Partners. 4.5 years ago, I made the change, and I started to work with the largest Hyper-V clusters around.

Last year I was made a dual-expertise MVP with Azure being added. I work nearly 100% on Azure, and I have always written about what I work with. Anytime I find a solution, or learn something cool (that I can talk about) I write about it. I was re-awarded yesterday as an Azure MVP, but my Cloud & Datacenter Management expertise was dropped. I expected it because I simply had not earned the privilege over the last year to be re-awarded. I have a full and happy family life and I don’t have enough time to give a dual-expertise status what I think it deserves from me. I was not surprised, but I was a bit sad because being a Hyper-V MVP was a career changer for me and I made lots of great friends.

For those of you who are new to the program or who want to get involved in being an MVP, I have some advice: Make the most of it. The opportunity is awesome but you only get from it what you put in. Take part, learn, contribute, and share. It’s a virtuous cycle, and the more you do, the more you get out from it.

Being a part of the community hasn’t ended for me. I’ll still be writing and speaking about Azure. In fact, my employers are running a big community event on October 17th in Dublin (details to come soon) on Azure, Windows Server 2019, and more. And who knows … maybe I’ll still write some about Hyper-V every now and then Smile

Left to right: Tudor Damian, me, Carsten Rachfahl, Ben Armstrong (Hyper-V), Didier Van Hoye – Hyper-V MVPs with Ben at Cloud & Datacenter Conference Germany 2017.

Thank You Mark Minasi @mminasi

A friend of mine, and a man who has influenced many IT careers around the world, is retiring. This week, Mark Minasi was at the Microsoft MVP Summit in Bellevue/Redmond to say farewell to many of his friends from across the globe. I’m taking this opportunity to say “thank you”.

I’m in a position to be able to write this post partly because of Mark. I’ve written before that 5 years into my IT career I found myself at a point where had to reinvent myself. I realised that relying on my employer to educate me was hopeless. I had to educate myself and put in the extra effort. I learned that continuous learning was the key to continued success. A few years later, in 2003, I landed a dream job, with the opportunity to design a Microsoft infrastructure that would span sites across North America, Europe, and Asia. I had a good idea of what I wanted, but I wanted more information, especially because I had decided to base everything on the just-released Windows Server 2003.

I believe in being prepared, so I went to a local book store and picked up two Windows Server 2003 books, including the infamously huge (1800 pages) Mastering Windows Server 2003 by Mark Minasi. I poured over his book for a weekend, pulling the pieces that were relevant and building out a design. A few months later, we went live, and it was gooooood.

There’s always Q&A about new things or problems, so I went online to find a place to ask questions. Mark ran a forum at the time and I joined. At first, I asked questions, but over time, that moved from asking questions to answering questions.

In 2004, my boss asked how I wanted to spend my training budget. I didn’t want to sit in Microsoft training classes because I knew that stuff. Instead, I wanted deeper information. I found a conference called Windows IT Connections that was running in Lake Las Vegas, and I signed up. There I saw great speakers like Jeremy Moskowitz (I bought his GPO book from him there, and it was amazingly detailed), and Mark Minasi. Mark’s sessions were interesting because he delivered information and entertainment all at once. His consulting work lead to more interesting topics, like understanding how AD replication worked in multi-site environments, the process of password changing, account lockouts, etc. These were all things I had to engineer around, and the information was more than one could find anywhere else.

I arrived early for one session, and sat in the fourth row. I was early and minding my own business. Mark came in, got himself set up, and then started to visit with the early birds. He came over to me and said hi. We talked about his one previous visit to Ireland, when he did some training in Galway. I didn’t tell him that I was “joe elway” on his forum – I thought that was being a bit weird.

A few years later, the membership on the forum became more and more friendly. The regulars were getting to know each other and chat about non-IT things. Some even knew each other from attending conferences. I don’t remember exactly how it started, but it was decided to arrange a meetup, which evolved into a mini-conference ran in Mark’s home town of Virginia Beach. That mini-conference saw some great minds attend, and share their knowledge. It was here that I got to know Mark personally, and he became a friend and great influence.

Myself, Mark Minasi, and Nathan Winters, on my “stag” photography trip to the Saltee Islands [Credit: Jimi Vigotty]

Mark had a genuine desire to see the attendees excel in all ways, whether it was at their techie job, or in the community as writers, speakers, etc. He freely shared his knowledge and experience.

One year, Mark taught a class on how to deliver a technical presentation. He talked about the preparation, the delivery, how to speak, etc. In technical audiences, there can sometimes be a difficult person: an ego that must prove itself as the best, a desire to be disruptive, or a hater of what you are saying. Mark shared his technique for dealing with that person – identify them while you are setting up before the presentation, and then go down to visit with them. Disarm the person by being friendly before the presentation … hmm 😊

When I think back to that mini-conference and the forum, the collection of people was impressive. One is leading huge Office 365 migration projects around the world. Another is at the tip of the spear on Microsoft’s global push of Azure Stack. A few of them were the best minds in Active Directory, inside or outside of Microsoft.  Another was a PowerShell guru, another a Small Business Server genius, and many of them became conference/user group speakers, and eventually became Microsoft MVPs.

A few of us discovered that we shared the hobby of photography and we start to go on trips together. We started by doing these trips before/after the conference in Virginia Beach and that became dedicated trips. I started to blog and do presentations. One year, Mark commented on my writing and encourage me to do more. And that became an offer to contribute to his latest Mastering Windows Server book – I was going from reading this series to writing this series! I wrote a number of chapters and my name was in the cover. That was a wow moment in my career. From that came other offers, and then I went to being the lead author on a few books, including the two Hyper-V books.

Mark’s advice on writing was to assume nothing about the reader. Explain the subject at a high level, introduce it, add some advanced content, and then cover the monitoring & troubleshooting. That’s the approach that was used in the Master series of books, and it’s something I adopted for my writing style.

My writing eventually lead to an offer to write for Petri.com. The editor back then came from Windows IT Connections (it’s a small world) and he wanted me to use my style or writing to explain Microsoft’s often confusing messaging in the Hyper-V and Windows Server context, and that migrated into Azure subject matter.

The offers for speaking kept coming, and Mark was always offering advice. He thought I should speak at, what was then, TechEd. There’s a longer story, but I eventually did get into TechEd Europe and the first Microsoft Ignite and, thanks to Mark’s advice over the years, I got great scores. I’ve spoken at many events and webinars. At work, I found myself writing and delivering larger and more complex training, eventually becoming 1-, 2-, and 3-day courses.

For years, Mark has been telling me to start doing my own thing. He reckoned that I should have been travelling Europe and the US, teaching Hyper-V classes. I didn’t really have the confidence that I could make that work but he always told me that I was silly to lack the required confidence. He was certain that people would attend. Last year, my wife said the same thing. Last year, I started Cloud Mechanix, and last month I taught my first Azure training course to a full room in London. Next month, I teach my second class in Amsterdam.

Mark’s influence on me changed my career. He’s always been a supporter, and encouraged me to do more. I know I’m not alone in that either. What he’s done for me lead to me asking him to be one of my groomsmen when I got married and I was honoured when he said yes.

That’s just my story. I’m certain that probably dozens of other people have similar stories of how Mark Minasi has impacted them. Mark has attended the Microsoft MVP Summit this week to say farewell to many friends because he has retired. I’ve seen some of that – people are coming up to say “hi” and “thank you”. I don’t think he can walk down a hallway without someone stopping him – it’s a nod to his impact on so many of us working with Microsoft products.

Mark, if you ever read this, thank you on behalf of me and the many that you have helped or influenced at your classes, the events you have spoken at, or via the books and articles you have written over the last 4 decades.

Aidan 2, Vodafone 0

If you follow me on social media, you might have noticed my escalation (phase 2) against Vodafone Ireland to get my issues resolved or my contract cancelled. I wrote a post, which was a diary of my costs and difficulties in trying to get a normal phone service from the telecoms company; I’d been without mobile data or text services for 10 days (11 including today). I tweeted Vodafone Ireland, to let them know what was to come. This morning, I escalated:

  • I posted to the Vodafone Ireland forum on Boards.ie (a huge forum in Ireland), and responded to every relevant article with my story.
  • I repeated that process on Vodafone Ireland’s own support forum.
  • I replied to every tweet by Vodafone Ireland, including their support tweets – I was about to configure Microsoft Flow to do that for me … that sort of makes this a tech story, right?

By 10:45, things started to happen.

I got the complaint reference number that I’d first asked for on the 19th of Feb (today is the 26th).

By the way, Vodafone blocked me on Twitter straight after that message. That was pointless – I can easily see every message using a second account (I have a few for various things). That was effectively an admission that I had an impact, and would only cause me to escalate.

Armed with the complaint reference number, I opened an official case on the Vodafone site, and then contacted Comreg. They normally have to wait 10 days after a complaint case starts, but after explaining the delay by Vodafone (I have all the screenshots, as you can see in these posts), the regulator offered to backdate the 10 days.

While I was talking to Comreg, a DM arrived in from Vodafone, at 10:54. Apparently some engineers had finished the break they’d started 11 days ago, and were able to look at my case. I rebooted my phone and …. tah dah! … I have data and text services once again.

So what’s the lesson here? Good customer service is cheaper than what a motivated and pissed off customer will do to you. I’d say at least 20,000 people saw what I did, and that stuff will stay on the Internet forever. And that was only phase 2 of my 3-phase plan. I was willing to purchase Google Adwords and advertise on Facebook to do more damage! I’ve bent Vodafone to my will twice, Eircom (now Eir) wrote an apology letter to me once, a debt collector (sent illegally by Eircom for a non-existing contract) apologised to me on the phone, saying I’d never hear from him again, and I made a camera store in the UK apologise and refund me after some dodgy payment practices. Be willing to use the voice that you have, and punish bad customer service. Poor customer service works for bad companies while they save money and you are willing to live with it. When you cost them money by damaging their reputation with the truth, they have to capitulate.

How Bad is Vodafone Ireland’s Mobile Phone Service?

Let me tell you a tale … a tale of a contracted-for and paid-for service mobile phone Internet & SMS/Text data services that Vodafone Ireland is failing to provide me with.

The Outage

This tale of woe starts on Thursday 15th February. I was on the road to visit a client in Limerick (I live in northern Kildare). I’d pulled into a service station and wanted to check to see what podcasts or audio books I had to listen to on the long drive. And I found that I had no Internet access. It’s not unusual to have blackspots so I ignored the inconvenience. Near the client site, I pulled in and opened my phone to navigate my way there because they had recently moved. And now, in Limerick City, I had no data services, even though I had a full 4G or 3G signal. Hmm! I made my way there anyway, and texted my wife … and the text failed to send. So I had lost all data services: Internet and SMS. Maybe it was a network glitch.

I left 5 hours later and I still had no service. I drove home, verified that I could browse on my wi-fi. I still had no data services when just on the Vodafone Ireland network. I made my first call.

Vodafone Interaction #1 (Day 0)

I got home and called “Customer Care”. I explained the issue clearly and patiently. I rebooted my fairly new iPhone 8 several times as requested. After about half an hour, I was told that either my phone was broken or the SIM needed to be replaced. There was “nothing else” that could be done.

My wife, also with an iPhone and also on Vodafone, came home. We swapped SIMs. My SIM (my Vodafone Ireland account) in her phone wouldn’t work. Her SIM (Vodafone Ireland account) in my phone did work. That meant:

  • My phone was fine.
  • Either my SIM or my Vodafone Ireland account was at fault.

Replace The SIM (Day 3)

On Sunday 18th February, I finally had time to visit a Vodafone Ireland store – a 1 hour round trip. I walked in and quickly got a replacement SIM. I got home, swapped the SIM, rebooted and …

The problem was still there. The services that Vodafone Ireland are contracted to offer and paid to do were still not available:

  • Data
  • Mobile.

Costs that will be invoiced to Vodafone Ireland:

  • Mileage: 42 miles * €0.75 = €31.50
  • Hourly rate: €350

Vodafone Interaction #2 (Day 3)

Immediately, after that, still on Sunday 18th February, I called “Customer Care”. I must have spent 45-60 minutes explaining the problem and rebooting the phone as requested. Eventually the agent said there was nothing more he could do for me.

Finally, I thought, my case would be escalated to an engineer. Remember now, this is Sunday. This is when the agent informed me that, yes, my case would be escalated, but an engineer wouldn’t be looking at my problem until Wednesday at the earliest. 3 days!

My problem here is that I needed my phone on Wednesday. My business, Cloud Mechanix, was running a training class in London on Thursday/Friday and I would be travelling on Wednesday. This just was not going to be good enough.

This is when I told the “care” agent that at this time, Vodafone Ireland was no longer offering me the service I had contracted for and paid for. I demanded a release from my contract. I needed a service, and if Vodafone Ireland would not offer me my mobile broadband and texting services, then I would have to get these services that my business needs from elsewhere.

The agent flipped. I repeated myself several times. Eventually I hung up. He called me back. I demanded to speak to a manager, and I was told that was not possible. If you read my previous posts about Vodafone Ireland, then you’ll know the truth about manager callbacks.

By the way, those posts remained on the first page of search results for “Vodafone Ireland Home Broadband” for several years – fixing my technical problem efficiently would have been much cheaper than the SEO damage that I caused this company.

I ended up hanging up again. And he called again, and again, and probably more. I stopped answering.

Workaround (Day 3)

I ended up having to drive 30 minutes, each way, to an Eir store to buy a pay-as-you-go SIM. After unlocking the SIM and paying credit, I had instant text and data services on the Eir network. At least I would have a phone service for my work trip to London.

Costs that will be invoiced to Vodafone Ireland:

  • Mileage: 42 miles * €0.75 = €31.50
  • Hourly rate: €350
  • SIM cost: €20
  • PAYG data: €20

Vodafone Interaction #3 (Day 4)

I tweeted my displeasure on Monday 19th February, making sure that @VodafoneIreland was mentioned. Eventually, they (“Aoife”) asked me to DM my details. I did.

Vodafone Ireland Interaction #4 (Day 4-5)

I waited several hours, and nothing had happened. I warned them that I would be blogging about this, and doing some SEO damage. I also demanded a complaint ID so I could open a case with COMREG, the Irish telecoms regulator. Here was the response, on Tuesday 20th February:

Sunday 25th February (Day 10)

The promised contact from the “tech team” didn’t happen. I’ve been to London and back, using my Eir SIM, even though I’ve paid my contracted €60 for this month’s Vodafone Ireland services including:

  • Voice
  • Data
  • Text

Two thirds of those essential services have been unavailable for 10 days now.

So up to now Vodafone Ireland also owes me: €13.01 for services paid that have not been provided. That amount owed back to me increases by ~€1.30 per day for each day that the contracted for and paid for services are not supplied by Vodafone Ireland.

What I Expect

I expect the following:

  • An official complaint ID to be provided to me by Vodafone Ireland so I can file a complaint with COMREG.
  • The paid-for and contracted-for services to be provided to me immediately, or my contract to be cancelled, at no fault or costs to me, by Vodafone Ireland – an admission of their fault.
  • The above costs to me, including my costs to Eir, myhourly rates, and the refunds owed for unprovided services, to be paid in full.

If you work for Vodafone Ireland social media/marketing, then be aware that this will escalate. I’ve done it to you before. I’ve done it to Eircom. I’ll do it to you. It will be cheaper to make me happy than to undo the damage I will incur.

You have until 17:30 on Monday 26th February to agree to my terms.

Update

I won. I also got a refund via credit to my account.

How To Use Docker To Stop And Remove All Windows Server Containers

I’ve been playing around with Containers on Windows Server 2016 GA. I can’t say I’m enthralled with Docker being the default interface for Containers now, but I understand Microsoft’s motivation.

I needed a way to quickly:

  • Stop all running containers on a host
  • Remove all containers from the host

If this was PowerShell, it would have been easy. But dragging open source onto Windows causes issues …. they do things inconsistently and all the docs are for Linux. Grep! Really!?!?

Eventually I found 1 variation of a solution that worked. The first line stops all running containers:

docker stop (docker ps –qa)

The second line removes all running containers:

docker rm (docker ps –qa)