Tag: Linux

E: Could Not Open File /var/lib/apt/lists

In this post, I’ll show how I solved a failure, that occurred during an Azure Image Builder (Packer) build with a Ubuntu 20.04 image, which resulted in a bunch of errors that contained E: Could not open file /var/lib/apt/lists/ with a bunch of different file names.

Disclaimer

I am Linux-disabled. I started my career programming on UNIX but switched to being a Microsoft infrastructure person a year later – and that was a long time ago. I am not a frequent Linux user but I do acknowledge its existence and usefulness. In other words, I figured out a fix for me, but it might not be a fix for you.

The Problem

I was using Azure Image Builder, which is based on Packer, to allow the regular creation of a Ubuntu 20.04 image with the latest updates and bits for acting as the foundation of a self-hosted DevOps agent VM Scale Set in a secure Azure network.

I had simple needs:

  1. Install Unzip
  2. Install Terraform

What makes it different is that I need the installations to be non-interactive. Windows has a great community with that kind of challenge. After a lot of searching, I realise that Linux does not.

I set up the tasks in the image template and for a month, everything was fine. Images built and rebuilt. A few days ago, a weird issue started where the first version of a template build was fine, but subsequent builds failed. When I looked at the build log, I saw a series of errors when apt (the package installed) ran that started with:

E: Could not open file /var/lib/apt/lists/…

The Solution

I tried a lot of things, including:

apt-get update
apt-get upgrade -y

But guess, what – the errors just moved.

I was at the end of my tether when I decided to try something else. The apt package installation for WinZip worked some of the time. What was wrong the rest of the time? Time – that was the key word.

Something needed more time before I ran any apt commands. I decided to embed a bunch of sleep commands to let things in Ubuntu catchup with my build process.

I have two tasks that run before I install Terraform. The first prepares Linux:

            {
                "type": "Shell",
                "name": "Prepare APT",
                "inline": [
                    "echo ABCDEFG",
                    "echo sleep for 90 seconds",
                    "sleep 1m 30s",
                    "echo apt-get update",
                    "apt-get update",
                    "echo apt-get upgrade",
                    "apt-get upgrade -y",
                    "echo sleep for 90 seconds",
                    "sleep 1m 30s"
                ]
            },

The second task installs WinZip and some other tools that assist with downloading the latest Terraform zip file:

            {
                "type": "Shell",
                "name": "InstallPrereqs",
                "inline": [
                    "echo ABCDEFG",
                    "echo sleep for 90 seconds",
                    "sleep 1m 30s",
                    "echo installing unzip",
                    "sudo apt install --yes unzip",
                    "echo installing jq",
                    "sudo snap install jq"
                ]
            },

I’ve ran this code countless times yesterday and it worked perfectly. Sure, the sleeps slow things down, but this is a batch task that (outside of testing) I won’t be waiting on so I am not worried.

Azure Bastion For Secure SSH/RDP in Preview

Microsoft has announced a new preview of a platform-based jumpbox called Azure Bastion for providing secure RDP or SSH connections to virtual machines running or hosted in Azure.

Secure Remote Connections

Most people that are using The Cloud are using virtual machines, and one of the great challenges for them is secure remote access. You need RDP or SSH to be able to run these machines in the real world.

Remember: for 99.9% of customers, servers are not cattle, they are sacred cows.

Just opening up RDP or SSH straight through a public IP address is bad – hopefully you have an NSG in place, but even that’s bad. If you enable Standard Tier Security Center, the alerts will let you know how bad pretty quickly. And if the recent scare about the RDP vulnerability didn’t wake you up to this, then maybe you deserve to have someone else’s bot farm or a bitcoin mine running in your network.

There are ways that you can secure things, but they all have the pluses and minuses.

VPN

The real reason that we have point-to-site VPN in Azure virtual network gateway was as an admin entry point to the virtual network.

The clue is in the maximum number of simultaneous connections which is 128, way too low to consider as an end user solution for a Fortune 1000, who Microsoft really do their planning for.

If you have supported end user VPN then you know that it’s right up there with password resets for helpdesk ticket numbers, even with IT people like developers. Don’t go here – it won’t end well.

Just-in-Time VM Access

JIT VM Access is a feature of Security Center Standard Tier. It modifies your NSG rules to deny managed protocols such as RDP/SSH (the deny rules are stupidly made as low priority so they don’t override any allow rules!).

When you need to remote onto a VM, an NSG rule is added for a managed amount of time to allow remote access via the selected protocol from a specific source IP address.

So, if it’s all set up right, you deny remote access to virtual machines most of the time. But you will open direct access. And the way JIT VM Access manages the rules now is wonky, so I would not trust it.

An RDP Jumpbox

This is an old method – a single virtual machine, or maybe a few of them, are made available for direct access. They are isolated into a dedicated subnet. You remote into a jumpbox, and from there, you remote into one of your application/data virtual machines.

Unfortunately, it’s still straight RDP/SSH into a machine that is directly accessible on the Internet. So in the remoting protocol vulnerability scenario, you are still vulnerable at the application layer. You could combine JIT VM Access, but now normal daily operations are going to be a drag and I guarantee you that people will invest time to undermine network security. Also, you are limited to 2 RDS connections per jumpbox without investing in a larger RDS (machines + licensing) solution.

Guacamole

This one is relatively new to me. At first it looked awesome. It’s a HTTPS-based service that allows you to proxy into Linux or Windows virtual machines via RDP or SSH.

All looked good until you started running Windows Server 2016 or later in your virtual machines and you needed NLA for secure connections via RDP. Then it all fell apart. The solution requires you to either disable NLA in the guest OS (boo!) or to hard code a username/password with local logon rights for your guest OS’s into the Guacamole server (double-boo!).

Azure Bastion

In case you don’t know this, a bastion host is another name for a jumpbox – an isolated machine that you bounce through. In this case, Bastion is a service that is accessible via the Azure Portal. You sign into the portal, click Connect and use the Bastion service to connect to a Linux or Windows virtual machine via SSH/RDP in the Portal. The virtual machine does not require a public IP address or a “NAT rule”, but it’s still SSH/RDP.

Azure Bastion

On the downside:

  • There’s no multi-factor authentication (MFA)
  • It requires that you sign into the Azure Portal – many people running in the guest OS might not even have those rights!
  • VNet peering is not supported – so larger enterprises are ruled out here … no one in their right mind will deploy 500 bastion hosts (one per VNet) in a large enterprise.

Microsoft did say that these things will be worked on, but when? After GA, which based on the time of year I guess will be just before/after Ignite in early November?

In my opinion, Bastion is the right idea, but more of the backlog should have been included in the minimal viable product.

A Gateway to a Better Solution

If you are a Citrix or a RDS person then you’ve been screaming for the last 5 minutes. Because you’ve been using something for years that most people still don’t know is possible. Both Citrix and RDS have the concept of an SSL gateway.

In the case of RDS, we can deploy one or more (load balanced) Windows Server virtual machines with the RDS Gateway role. If we combine that with NPS and Azure AD, we can also add MFA. With a simple tweak to the Remote Desktop Connection client (MSTSC.EXE), we can RDP to a Windows machine behind the RDS Gateway. The connection from the client to the gateway is pre-authenticated, x.509 certificate protected, HTTPS traffic encapsulating the RDP stream. That connection terminates at the RDS Gateway and then forwards as RDS to the desired Windows Server virtual machine behind it.

Unlike the previous jumpbox solution:

  • This can be a low-end machine, such as a B-Series.
  • It can scale out using a load balancer
  • Many people can relay through a single jumpbox machine.
  • You won’t need RDS licensing at all, not even to scale out to more than 2 users per gateway machine.

So – there’s no SSH here. So Linux is a problem.

Opinion

We don’t really have a complete solution right now. Azure Bastion probably will be the best one in the long-run, but it has so many missing features that I couldn’t consider it now. For Windows, an RDS Gateway is probably best, and for Linux, a Guacamole server might be best.

What do you think?

Linux Integration Services 4.1 for Hyper-V

Microsoft has released a new version of the integration components for Linux guest operating systems running on Hyper-V (2008, 2008 R2, 2012, 2012 R2, and 2016 Technical Preview, Windows 8, Windows 81, and Azure).

What’s new?

  • Expanded Releases: now applicable to Red Hat Enterprise Linux, CentOS, and Oracle
  • Linux with Red Hat Compatible Kernel versions 5.2, 5.3, 5.4, and 7.2.
  • Hyper-V Sockets.
  • Manual Memory Hot Add.
  • SCSI WNN.
  • lsvmbus.
  • Uninstallation scripts.
Technorati Tags: ,

The Genuine Need for Disaster Recovery In Ireland/EU

How many times have you watched or read the news, saw some story about an earthquake, hurricane, typhoon, or some other disaster and think “that will never happen here”? Stop kidding yourself; disasters can happen almost everywhere.

I’ve always considered Ireland to be relatively safe. We don’t have (anything you’d notice) earthquakes, typhoons, or tornadoes; our cattle and sheep don’t need flying licenses. Our weather is dominated by the gulf stream, keep Ireland temperate. It doesn’t get hot here (we are quite northerly) and our winters consist of cloud, rain, and normally about half a day of snow. We get the tail end of some of those hurricanes that hit the east coast US, but there’s not much left by the time they reach us – some trees get knocked over, some tiles knocked on our roofs, but it’s not too bad. Even when we look at our neighbours in England, we see how their more extreme climate causes them disasters that we don’t get. Natural disasters just don’t happen here. Or do they?

The last month or so has revealed that to be a lie. Ireland has been battered by 6 storms in the past month. The latest, Storm Frank, was preceded with warnings that the country was saturated. That means that the ground has absorbed all of the water that it can; any further rainfall will not be absorbed, and it will pool, flow, and flood.

This morning, I woke to these scenes:

image

Enniscorthy, Co. Wexford [Image source: Paddy Banville]

Embedded image permalink

Graignamanagh, Co. Kilkenny [Image source: Graignamanagh G.A.A]

image

Middleton, Co. Cork [Image source: Fiona Donnelly]

Frank isn’t finished. It’s still blowing outside my office and more rain is sure to fall. There are stories of communities being evacuated to hotels, and the above photos are just the easy ones for the media to access.

This isn’t just a case of cows trapped in fields, stick a sandbag on it and you’re sorted, or somewhere far away. This is local. And Ireland is a relatively safe place – we’re not Oklahoma, a place that some deity has decided should be subject to cat 5 tornadoes every time you’re not looking. Dorothy, the point is, that disasters happen everywhere, including in the EU where we think it safe.

Let’s bring this back to business. Businesses have been put out of action by these floods. Odds are any computers or servers were either on the ground floor or in the basement. Those machines are dead. That means those businesses are dead. They might be lucky enough to have tapes (let’s leave that for another time) stored offsite but how reliable are they and will bare-metal restore work, or will it take forever? How much money will those businesses lose, or more critically, will those businesses survive loss of customers?

This is exactly why these businesses need a disaster recovery (DR) solution. There are several reasons why they don’t have one now:

  • Fires and other unnatural disasters happen everywhere
  • They couldn’t afford one
  • The business owners didn’t think there was a need for one
  • Some resellers didn’t think there was demand for one so they never brought it up with their customers

The need is there, as we can clearly see above. And thanks to Microsoft Azure, DR has never been so affordable. FYI, it comes in at a price that is a small fraction of the cost of solutions from the likes of Irish companies such as KeepITSafe – I’ve done the competitive pricing – and it opens that customer up to more technical opportunities with hybrid cloud solutions.

Microsoft Azure Site Recovery Services (ASR) is a disaster recovery-as-a-service (DRaaS) or cloud DR site offering from Microsoft. The beauty of it is that it’s there for everyone from the small business to the large enterprise. It works with Hyper-V, vSphere or physical machines, and it works with Windows or Linux as long as the OS is supported by Azure (W2008 R2 or later on the Windows side).

Note: There is a cost overhead for vSphere or physical machines to allow for on-premises conversion and forward and in-cloud management and storage, so you need a certain scale to absorb that cost. This is why I describe ASR as being perfect for SMEs with Hyper-V and mid-large companies with Hyper-V, vSphere or physical machines.

If I had ASR in place, and I has a business on the quayside in Cork, near the Slaney in Enniscorthy, or anywhere else where the rivers were close to bursting the banks then I would perform a planned failover, requiring about 2 minutes of my time to started a pre-engineered and tested one-click failover. My machines would shut down in the desired order, flush the last bit of replication to Azure, and start up the VMs in the desired order in Azure, and my machines and data would be safe. I can failback to new equipment or stay in Azure if the disaster wipes out my servers. And if that disaster doesn’t happen, I can easily failback to new equipment, or choose to stay in Azure and not worry about local floods again.

Technorati Tags: ,,,,,

Microsoft News – 19 October 2015

It turns out that Microsoft has been doing some things that are not Surface-related. Here’s a summary of what’s been happening in the last while …

Hyper-V

image

Windows Server

Windows Client

Azure

Office 356

Miscellaneous

Microsoft News – 28 September 2015

Wow, the year is flying by fast. There’s a bunch of stuff to read here. Microsoft has stepped up the amount of information being released on WS2016 Hyper-V (and related) features. EMS is growing in terms of features and functionality. And Azure IaaS continues to release lots of new features.

Hyper-V

Windows Client

Azure

System Center

Office 365

EMS

Security

Miscellaneous

Microsoft News – 7 September 2015

Here’s the recent news from the last few weeks in the Microsoft IT Pro world:

Hyper-V

Windows Server

Windows

System Center

Azure

Office 365

Intune

Events

  • Meet AzureCon: A virtual event on Azure on September 29th, starting at 9am Pacific time, 5pm UK/Irish time.

Microsoft News – 16 July 2015

It’s been a busy week with WPC driving announcements that affect partners.

Hyper-V

Windows Server

Windows Client

Azure

clip_image001

System Center

  • Datazen Enterprise Server: Datazen Enterprise Server is a collection of web applications and Windows services. Acts as a repository for storing and sharing dashboards and KPIs.

Office 365

Licensing

Miscellaneous

Microsoft News–13 July 2015

I don’t have all that much for you, but the big news is the Azure Site Recovery (ASR, Microsoft’s DR site in the cloud) now supports VMware virtual machines and physical servers, without using System Center. You do need to run some stuff on-prem and in the cloud to make it work though, so there will be a tipping point where the solution becomes affordable.

Azure

clip_image001

System Center

Office 365

Microsoft News – 29 June 2015

As you might expect, there’s lots of Azure news. Surprisingly, there is still not much substantial content on Windows 10.

Hyper-V

Windows Server

Windows Client

clip_image001_thumb.png

Azure

Office 365

EMS

Misc