A Factual Analysis of Cloud Computing VS The USA Patriot Act

Note: This article applies to public cloud computing.  Private clouds where you own the equipment and software in your computer room/data centre are not affected.

Regular readers will know that I used to work in the hosting business and that something I warn people to be aware of is the USA Patriot Act – a legacy of George W. Bush’s war on terror (some might argue it was a war on freedom) and that lives on under the “moderate” Democratic government a decade later.

The ZDnet article, “Case study: How the USA PATRIOT Act can be used to access EU data”, by Zack Whittaker is an excellent analysis of the problems that the Patriot Act causes for non-American organisations with cloud services provided by USA owned companies, no matter where their subsidiaries or data centres are located.

I’ve been able to attend a number of cloud computing events since the trend kicked off.  Those who have invested themselves in the likes of Amazon, Azure, or Google, will vehemently deny that the Patriot Act applies.  Some of them will toss their toys out.  They kind of remind me when Irish PM Bertie Ahern told us critics to go commit suicide when we questioned the health of the economy (he resigned a few months later when he finally saw the financial tsunami that was coming).  Their lack of willingness to discuss or listen should make you wonder.

Last year I asked an Amazon evangelist about the Patriot Act and how it would apply to data stored in Amazon’s European data centres.  The rather cocky answer was that it wouldn’t because the Amazon company in Ireland was an Irish registered company.  Indeed it is, but it is also owned by a USA owned corporation that must comply with the Patriot Act.

A few years ago at the Microsoft BPOS launch, I asked a MSFT speaker about the Patriot Act (I am a bold boy!).  He had to admit that there was an issue even if the data stayed in the Dublin data centre.  But straight away, MSFT sales and marketing were out talking about geo-location and how that was the solution to data protection issues.  Some of us knew that to be BS, and others went and developed their HR SaaS, or whatever, applications on Azure (I did have a few giggles, I have to admit, thinking of the impending ex-employee versus employer lawsuits that could follow).  Finally Steve Ballmer admitted to the issue at a CEO conference … but try stop sales and marketing!

For you nay-sayers, here’s a couple of bits from this excellent article:

“The bottom line is that both Microsoft and Google — and therefore any other cloud service provider operating in Europe — cannot provide satisfactory guarantees that data supplied by EU customers and housed in datacenters on European soil will not leave the European Economic Area under any circumstances”.

“These subsidiary companies and their U.S.-parent corporations cannot provide the assurances that data is safe in the UK or the EEA, because the USA PATRIOT Act not only affects the U.S.-based corporations but also their worldwide wholly-owned subsidiary companies based within and outside the European Union”.

I’ve met loads of people who love EC2.  I know an Azure MVP and he’s fallen for it as a developer.  All quite understandable.  To me, things like Office365 do offer amazing opportunities in the right circumstances.  Will anything change regarding the Patriot Act?

Rumour is that Amazon and MSFT lobby strongly over this issue.  Some believe they had a lot to do with some of the contentious pieces of the Cyber Security Act being stripped out.  I’d believe it – the USA might be the big player in cloud computing right now, but if data laws continue to cause concerns then what’s to stop a Chinese operator dominating there, or a French/UK/German operator dominating in Europe, or a South American provider dominating down there?  That would put a seriously big pinch on Amazon’s plans to be online content kings of the world, and Microsoft’s plans to dominate PaaS/SaaS just like they’ve dominated Office software.  Maybe there will come a time when the USA government will cop on and relinquish these communist-like demands over hosters.  That would be of benefit to us all.  But we have learned from history that both USA political parties are willing and able to undo freedoms at a moment’s notice; we only have to look at the original drafts of the Cyber Security Act to see that.

So are people listening to the warnings?  As I’ve already alluded to: no they’re not.  The louder voices of those who are already invested are drowning out those urging caution.  And there are those who see those oh-so-tempting low sticker prices of an Azure or an EC2 and then don’t want to listen to anything else.  I’ve had those conversations in the past.  To be quite honest, most people don’t want to listen.  It’s like telling a gambling/spending addict they they shouldn’t get that sixth credit card.  They either berate you for questioning “progress”, try to change the topic of conversation to that of technical features, say that the Patriot Act will never be used against them (it’s known to have been used over 80 times – and the fact is that data that is susceptible to the PA is at risk to not being protected by the European Data Protection Act), or they engage the lah-lah-lah arguement.  I gave up; that’s why I’m not in the hosting business any more.

So give the ZDnet article a read.  It’s well constructed, telling the story of the author’s investigation.  He uses a case study and approaches some big service providers directly to get their official responses on the issue.

Now, let me get back to folding that tin (aluminium) foil hat to keep those pesky NSA satellites out of …

Technorati Tags: ,

BitLocker & My Personal Laptop

My personal laptop contains some stuff that I don’t want to lose control of, including the original Word documents for a few books.  As such,I take precautions to protect that content.

My laptop runs Windows 7 Ultimate Edition.  That includes a feature called BitLocker which can encrypt an entire disk.  With a TPM 1.2 chip enabled in the BIOS, I enabled that, saving the recovery key to a USB stick.  I want to keep that key safe – just in case.  So I moved it from the USB stick to a folder on my laptop.  That folder is replicated to my other machines using Live Mesh.  That means I can access the recovery key for the laptop from anywhere using my Live ID.

My data is secure, and I can recover the laptop if something should go awry.

Technorati Tags: ,

74% Of Workers Plug Personal Devices Into Work Network

I’ve just read a story on techcentral.ie that discusses a Virgin Media (UK-based ISP) report.  It says that 74% of company employees are bringing personal devices into work and plugging them into the company network.  This is the sort of thing I was talking about in my previous millenials post.  It’s also the sort of thing that has impacted decision making by corporates: personal preferences for a better appliance or utility can improve the working experience, and the corporate decision making process.  We have to decide how we respond?

Do we try to block everything?  We can try.  Group Policy and utilities like DeviceLock can lock down what is plugged into PCs.  Network Access Protection (Windows)/Network Access Control (Cisco) can control what is allowed to connect to the network.  I’ve taken the device lock approach before.  But a valid business case always overrules global policy, and you might be surprised how many people come up with “valid” business cases.  Soon the policy resembles swiss cheese, only affecting the minority of users.  The result is that IT is disliked – it’s a blocking force once again.

The user-centric approach that we’re seeing with private cloud, App-V, and System Configuration Manager 2012 is an example of how we need to think.  My millenials post also suggests a way forward.  Maybe we need to allow personal appliances, but use those policy tools like Network Access Control to place the appliances into networks that are not central, kind of like the guest network that is often used.  Or maybe we need to change how we think about the PC altogether and treat the entire PC network as a guest network. 

The latter approach might work very well with the user-centric approach.  If end users are using their own PCs, tablets, and phones, then we cannot apply corporate policy to them.  Maybe we just provide user-centric self-service mechanisms and let them help themselves.  Or maybe things like VDI and/or RemoteApp are the way forward for LOB client delivery.  If everythign was cloud (public/provate) and web-client based then application delivery would be irrelevant.  Maybe it’s a little bit from column A and a little from column B?

It’s a big topic and would require a complete shift in thinking … and a complete re-deployment of the client network, including LOB application interfaces.

KB2345316: Prevent a DDOS Attack From A Hyper-V VM

Microsoft has released the second ever (since the release of Windows Server 2008!) security fix for Hyper-V. 

“This security update resolves a privately reported vulnerability in Windows Server 2008 Hyper-V and Windows Server 2008 R2 Hyper-V. The vulnerability could allow denial of service if a specially crafted packet is sent to the VMBus by an authenticated user in one of the guest virtual machines hosted by the Hyper-V server. An attacker must have valid logon credentials and be able to send specially crafted content from a guest virtual machine to exploit this vulnerability. The vulnerability could not be exploited remotely or by anonymous users”.

In other words, you have to be logged into a VM running on the host (be a legit internal user) and have sufficient rights in the VM’s operating system to craft this packet.

The issue affects Windows Server 2008 and Windows Server 2008 R2.

Mastering Hyper-V Deployment Book is Available Now

Amazon has started shipping the book that I wrote, with the help of Patrick Lownds MVP, Mastering Hyper-V Deployment.

Contrary to belief, an author of a technical book is not given a truckload of copies of the book when it is done.  The contract actually says we get one copy.  And here is my copy of Mastering Hyper-V Deployment which UPS just delivered to me from Sybex:

BookDelivered

Amazon are now shipping the book.  I have been told by a few of you that deliveries in the USA should start happening on Tuesday.  It’s been a long road to get to here.  Thanks to all who were involved.

Network Security in the Hypervisor

I just read an interesting article that follows up some presentations at VMWorld.  It discusses the topic of security in the Hypervisor (ESX in this case) – the author is actually focusing solely on network security.  Other aspects such as policy, updating, etc, are not discussed. 

The author asks 4 questions:

Q) Security is too complicated, and takes too many separate devices to configure/control.
A) Yes – and I agree, sort of.

Security should be simple.  It isn’t.  It requires too many disparate point solutions.  Let me step back a moment.  Why do I like Windows, AD, System Center, Hyper-V, etc?  It’s because they are all integrated.  I can have one tidy solution with AD being the beating heart of it all.  And that even includes security systems like WSUS/ConfigMgr (update management), NAP (policy enforcement), BitLocker/BitLocker To Go, device lock downs on personal computers, remote access (DirectAccess or VPN via RADIUS/IAS) etc.

Things start to fall apart for network security.  Sure you can use whatever ISA Server is called these days (Sorry ForeFront; you are the red headed stepchild in Redmond, locked away where no one knows you exist).  Network security means firewall appliances, IDS systems, VPN appliances, VPN clients that make every living moment (for users and admins) a painful existence, etc.  None of these systems integrate.

To VMware’s credit, they have added vShield into their hypervisor to bring firewall functionality.  That would be find for a 100% virtual or cloud environment.  That’s the sort of role I had for 3 years (on ESX and Hyper-V).  I relied on Cisco admins to do all the firewall work in ASA clusters.  That’s way out of my scope and it meant deployments took longer and cost more.  It slowed down changes.  It added more systems and more cost.  A hypervisor based firewall would have been most welcome.  But I was in the “cloud” business.

In the real world, we virtualization experts know that not everything can be virtualized.  Sometimes there are performance, scalability, licensing, and/or support issues that prevent the installation of an application in a virtual machine.  Having only a hypervisor based firewall is pretty pointless then.  You’d need a firewall in the physical and the virtual world.

Ugh!  More complications and more systems!  Here’s what I would love to see (I’m having a brainfart) …

  • A physical firewall that has integration in some way to a hypervisor based firewall.  That will allow a centralized point of management, possibly by using a central policy server.
  • The hypervisor firewall should be a module that can be installed or enabled.  This would allow third parties to develop a solution.  So, if I run Hyper-V, I’d like to have the option of a Checkpoint hypervisor module, a Microsoft one, a Cisco one, etc, to match and integrate with my physical systems.  That simplifies network administration and engineering.
  • There should be a way to do some form of delegation for management of the hypervisor firewall.  In the real world, network admins are reluctant to share access to their appliances.  They also might not want to manage a virtual environment which is rapidly changing.  This means that they’ll need to delegate some form of administrative rights and limit those rights.
  • Speaking of a rapidly changing virtual environment: A policy mechanism would be needed to allow limited access to critical VLANs, ports, etc.  VMs should also default to some secure VLAN with security system access.
  • All of this should integrate with AD to reuse users and groups.

I reckon that, with much more time, this could be expanded.  But that’s my brain emptied after thinking about it for a couple of minutes, early in the morning, without a good cup of coffee to wake me up.

Q) Security now belongs in the hypervisor layer.
A) Undecided – I would say it should reside there but not solely there.

As I said above, I think it needs to exist in the hypervisor (for public cloud, and for scenarios where complicated secure networks must be engineered, and to simplify admin) and in the physical world because there is a need to secure physical machines.

Q) Workloads in VMs are more secure than workloads on physical systems.
A) Undecided – I agree with the author.

I just don’t know that VM’s are more secure.  From a network point of view, I don’t see any difference at all.  How is a hypervisor based firewall more secure than a physical firewall?  I don’t see the winning point for that argument.

Q) Customers using vShield can cut security costs by 5x compared to today’s current state-of-the-art, while improving overall security.
A) Undecided – I disagree with VMware on this one.

The need for a physical environment is still required to protect physical infrastructure.  That cost is going nowhere.

This is all well and good, but this all forgets about security being a 3D thing, not just the signle dimension of firewall security.  All those other systems need to be run, ideally in an integrated management, authentication/authorisation environment such as AD.

CAO Calls in the Cops Over DDOS Attack

The Irish Independent is reporting that the CAO has called in the Gardaí (Irish police force) to investigate the repeat DDOS attacks.  Logs have been handed over.  The Gardaí actually don’t do any investigation; it’s done by one of the universities (UCD I think).  Maybe they should run Windows Server 2008 R2 for their web servers and add the beta of Dynamic IP Restrictions Extension for IIS.

Dynamic IP Restrictions Extension for IIS Beta

DDOS was the topic of the week with the CAO office in Ireland being repeatedly attacked.  Microsoft released a beta of a new IIS module, called Dynamic IP Restrictions Extension for IIS.  The idea is that the web server will deny connection requests from detected DDOS and brute force password attackers.  I don’t know how automated this is: remember that DDOS attackers tend to be botnets of infected PC’s that will have DHCP addresses on the net.  I really like the brute force attack defence.  I can tell you that this is a huge problem for web hosting companies; I’ve seen it myself on a pretty large shared web hosting farm.  I’d like to see this followed up with similar feature for SQL: those farms present TCP 1433 naked to the net … I can hear the shrieks from enterprise DBA’s now. 

This module is a very cool development from the impressive IIS group.

Reduce the chances of a Denial of Service attack by dynamically blocking requests from malicious IP addresses

Dynamic IP Restrictions for IIS allows you to reduce the probabilities of your Web Server being subject to a Denial of Service attack by inspecting the source IP of the requests and identifying patterns that could signal an attack. When an attack pattern is detected, the module will place the offending IP in a temporary deny list and will avoid responding to the requests for a predetermined amount of time.

Minimize the possibilities of Brute-force-cracking of the passwords of your Web Server

Dynamic IP Restrictions for IIS is able to detect requests patterns that indicate the passwords of the Web Server are attempted to be decoded. The module will place the offending IP on a list of servers that are denied access for a predetermined amount of time. In situations where the authentication is done against an Active Directory Services (ADS) the module is able to maintain the availability of the Web Server by avoiding having to issue authentication challenges to ADS.

Features

  • Seamless integration into IIS 7.0 Manager.
  • Dynamically blocking of requests from IP address based on either of the following criteria:
    • The number of concurrent requests.
    • The number of requests over a period of time.
  • Support for list of IPs that are allowed to bypass Dynamic IP Restriction filtering.
  • Blocking of requests can be configurable at the Web Site or Web Server level.
  • Configurable deny actions allows IT Administrators to specify what response would be returned to the client. The module support return status codes 403, 404 or closing the connection.
  • Support for IPv6 addresses.
  • Support for web servers behind a proxy or firewall that may modify the client IP address.
Technorati Tags: ,,

Hyper-V: Blue Screen & Unable to Access Data Folder

I am doing some work on my Hyper-V lab machine at home that requires a lot of VM’s and a lot of disk space.  My eSATA disk just does not have the space so I had to do the unthinkable: use a 1TB USB 2.0 drive that I had sitting spare to store some VM’s (please do not ever do this in production).  It will be slow but I can live with that for some lab stuff.

I attached the drive, cleared off a few bits and pieces, and used VMM Quick Storage Migration to move a bunch of VM’s over.  I deployed a new VM and started working on it.  It blue screened soon after boot up.  Strange!  I haven’t seen that before.  I worked on it again and *bang* it was gone again.

My first suspect was W 2008 R2 SP1 beta, but I soon had a clue that it wasn’t at fault.  In event viewer, under Hyper-V-VMMSAdmin I saw a bunch of errors telling me that Hyper-V could access various folders, including snapshots (I know I tell you not to use them in production but I use them in a lab) and data folders.  The alerts associated with my new VM cooincided with the crash.

I appeared to have a permissions issue.  I didn’t have time to figure out exactly what was at fault.  Instead I moved the VMs, formatted the volume, and moved the VMs back again.  Everything is working perfectly.

I reckon the info on this post has the answer.

CAO Website Hit by DDOS Attack Yesterday

Yesterday I talked briefly about the college course application process.  This is managed by a government organization called the CAO.  Students can find out about their coolege course offers via a website, or later via the post.

The website in question was a victim of a DDOS attack yesterday, the day the announcements were posted online. 

A DDOS (distributed denial of service) attack is a ccordinated attack that makes use of comprimised PCs from around the world.  These PCs are infected with trojan downloaders.  A DDOS client is downloaded and installed.  The DDOS client receives instructions from an IRC channel or a website on a regular basis.  The entire architecture is referred to as a botnet.  There are many such botnets in the world, some containing a few hundred machines, some a few thousand, some with hundreds of thousands of DDOS clients, and it’s rumoured that there are some with millions of machines under their control.

The owner of these botnets will sell their services or even access to parts of the botnet.  The botnets can be easy to use; there are even online videos to train you in the use of a simple GUI command console.

Together, even a few hundred bots (or DDOS clients) can fire an amazing amount of traffic at a web server or online presence.  These requests can be valid, or they can be simple TCP connect handshakes that aren’t completed by the client (SYN attack).  The recipient server or intermediary network appliances can be overwhelmed.  A TCP conenct table can be filled, a CPU can be driven to 100% utilization, or a network connection can be filled.

The motive for an attack can be varied.  Sometimes it is a practice run: an attacker will go after a small target to verifiy the system works before hitting a bigger target.  It can be a case of blackmail.  An email will be received by the victim soon after the attack starts to demand payment to cease the attack.  Sometimes it is a case of someone getting their jollies for bragging rights, e.g. “I took down XYZ!!!” on some blackhat forum.  It can even be a case of corporate espionage (this does happen!).  And it can be political: Al Jazeera was allegedly hit not long after the George W. Bush & Tony Blair Iraq war.  There is talk of Georgia being hit during their troubles with Russia a few years ago.

A past customer of mine was once hit.  They were a small business.  It started on a Sunday with a SYN attack.  The web servers couldn’t deal with it.  We configured the network appliances to deal with it by reducing the TCP handshake timeout.  All was well for a few hours.  Then the attacker simply increased the size of the attack.  The network appliances were overwhelmed and we had to implement filters to block all attempts to reach the web servers.

This attack went after the URL of the website in question.  Changing the IP address of the server would make no difference (and it didn’t – the customer demanded it was done).  Changing the location of the server would make no differnce.  Distributing the website across servers in many locations might have worked for a while … until the DDOS attack grew in size once again.  The customer thought about buying a dedicated DDOS prevention appliance.  Nice idea but:

  1. They are not perfect.  They have false positives (blocking legitamte connections and losing online customers) and they also allow a certain amount of attack traffic through.
  2. The appliance will start out by handling the attack.  This requires network, memory, and CPU resources.  The attacker can simply grow the attack with a few mouse clicks and the spend of a few Euros or Rubels.  This will cause one of those resources to become a bottleneck and the website is offline once again.

These _very_ expensive appliances cannot grow to match the capabilities of a DDOS attack at the same pace or even the same price.

What hope is there?  Only the most serious of attacks will last more than 3 days.  I know, 3 days is an eternity in the online world.  There are certain *ahem* professionals out there who can trace the botnet coordinator and DDOS it.  That will terminate an attack.  You can pay the ransom … but that means the attacker knows you are desperate enough to pay.  Pay once and you might pay again and again.  You can call the authorities but that might do little for you.  If the botnet is rented or it’s a relatively small attack then it will prbably end after 3 days because that appears to be the normal period to rent a botnet.  That’s what I was told by a security expert when my old customer was hit.  Sure enough, the attack ended after 3 days.

The only real defence I can see is an IDS (intrusion detection system) that is hosted and maintained by your ISP.  This has to be a massive system.  The bad news is that gaining access to these systems is very expensive.  The configuration is a pain for the admins.  Some schemes will initiate the IDS for your IP addresses when you inform the ISP of an attack, taking a short while for the defence to kick in.  Some are online all of the time but you risk false positives with legitimate traffic being filtered.

What about the CAO?  A consultant that was quoted in the article said:

“This is something every website is vulnerable to. There is not really anything they can do short of spending huge sums of money on extra servers in differing places around Ireland,”

The computer says “no!”.  Sorry, but if an attack is hitting a URL then it doesn’t matter where you move the site to or how you load balance it.  Eventually the DNS record TTL will expire and the attack will commence on the new location.  Load balancing just scales out your system and a DDOS will scale out much quicker and more economically than you can.  The attackers aren’t idiots.  Even if you do succesfully come up with alternative URLs, they can update their attack instructions in seconds.

He said hackers usually go after more high-profile sites such as Amazon or eBay.

The computer says “no!”.  The Irish media reported that there were a spate of attacks on small Irish businesses earlier this year.  They were ransom attacks, i.e. “we’ll stop the attack if you pay us”.  The irish police and an associated research unit confirmed the story.  We don’t hear about these attacks because companies are embarressed.  They see them as a breach of security (they aren’t).  We only hear about these attacks when they are visible, i.e. big attacks that might take down a Twitter, an Amazon, or the CAO.

Unfortunately, DDOS is a result of the fairly trusting nature of the basics of Internet technology.  Firewalls, IDS appliances, and all that stuff can only do so much.  You can do your bit to reduce the risk by ensuring that your computers are up to date with patches every month.  This vastly reduces the risk of being infected with a trojan downloader.