Defending Against Supply Chain Attacks

In this post, I will discuss the concepts of supply chain attacks and some thoughts around defending against them.

What Is A Supply Channel Attack?

The recent ransomware attack through Kaseya made the news but the concept of a supply chain attack isn’t new at all. Without doing any research I can think of two other examples:

  • SolarWinds: In December 2020, attackers used compromised code in SolarWinds monitoring solutions to compromise customers of SolarWinds.
  • RSA: In 2011, the Chinese PLA (or hackers sponsored by them) compromised RSA and used that access to attack customers of RSA.

What is a supply chain attack? It’s pretty hard to break into a network, especially one that has hardened itself. Users can be educated – ok, some will never be educated! Networks can be hardened and micro-segmented. Identity protections such as MFA and threat detection can be put in place.

But there remains a weakness – or several of them. There’s always a way into a network – the third party! Even the most secure network deployments require some kind of monitoring system – something where a piece of software is deployed onto “every VM”.  Or there’s some software vendor that’s deep into your network that has openings all over the place. Those are your threats. If an attacker compromises the software from one of those vendors then they will get into your network during your next update and they will use the existing firewall holes & permissions that are required by the software to probe, spread, and attack.

Protection

You still need to have your first lines of defense, ideally using tools that are designed for protection against advanced persistent threats – not your regular AV package, dumby:

  1. Identity
  2. Email
  3. Firewall
  4. Backup with isolated offline storage protected by MFA

That’s a start, but, but a supply chain attack bypasses all that by using existing channels to enter your network as if it is from a trusted source – because the attack is embedded in the code from a trusted source.

Micro-Segmentation

The first step should be micro-segmentation (AKA multi-segementation). No two nodes on your network should be able to communicate unless:

  1. They have to
  2. They are restricted to the required directions, protocols, and ports.
  3. That traffic passes through a firewall – and ideally several firewalls.

In Microsoft Azure, that means using:

  • A central firewall, in the form of a network firewall and/or web application firewall (Azure or NVA). This firewall controls connections between the outside world and your workloads, between your workloads, and importantly from your workloads to the outside world (prevents malware from talking to its human controller).
  • Network Security Groups at the subnet level that protect the subnet and even isolate nodes inside the subnet (use a custom Deny All rule because the default Deny All rule is useless when you understand the logic of how it works).
  • Resource firewalls – that’s the guest OS firewall and Azure resource firewalls.

If you have a Windows ADDS domain, use Group Policy to force the use of Windows Firewall – lazy admins and those very same vendors that will be the channel of attack will be the first to attempt to disable the firewall on machines that they are working on.

For Azure resources, consider the use of Azure Policy to force/audit the use of the firewalls in your resources and a default route to 0.0.0.0/0 via your central firewall.

An infrastructure-as-code approach to the central firewall (Azure Firewall) and NSGs brings documentation, change control, and rollback to network security.

Security Monitoring

This is where most organisations fail, and even where IT security officers really don’t get it.

Syslog is not security monitoring. Your AV is not security monitoring. You need something bigger, that is automated, and can filter through the noise – I regularly use the term “be your Neo to read the Matrix”. That’s because even in a small network, there is a lot of noise. Something needs to filter through that noise and identity the threats.

For example, there’s a lot of TCP 445 connection attempts coming from one IP address. Or there are lots of failed attempts to sign in as a user from one IP address. Or there are lots of failed connections logged by NSG rules. Or even better – all of the above. These are the sorts of things that malware that is attempting to spread will do. This is the sort of work that Azure Sentinel is perfect for – Sentinel connects to many data sources, pulls that data to a central place where complex queries can be run to look for threats that a human won’t be able to do. Threats can create incidents, incidents can trigger automated flows to eliminate the noise, and the remaining incidents can create alerts that humans will act upon.

But some malware is clever and not so noisy. The malware that hit the HSE (the Irish national health service) uses a lot of manual control to quietly spread over a very long time. Restricting outbound access to the Internet to just the required connections for business needs will cripple this control mechanism. But there’s still an automated element to this malware.

Other things to implement in Azure will include:

  • IDPS: An intrusion detection & prevention in the firewall, for example Azure Firewall Premium. When known malware/attack flows pass through the firewall, the firewall can log an alert or alert/deny the flows.
  • Security Center: Enabling Security Center “Azure Defender” (the tier previously known as the Azure Security Center Standard) provides you with oodles of new features, including some endpoint protections that are very confusingly packaged and licensed by Microsoft.

Managed Services Providers

MSPs are a part of the supply chain for their customers. MSP staff typically have credentials that allow them into many customer networks/services. That makes the identities of those staff very valuable.

A managed service provider should be a leader in identity security process, tooling, and governance. In the Microsoft world, that means using Azure AD Premium with MFA enabled for all staff. In the Azure world, Lighthouse should be used to gain access to customers’ cloud implementations. And that access should be zero-trust, powered by Privileged Identity Management (PIM).

Oh Cr@p!

These attackers are not script kiddies. They are professional organisations with big budgets, very skilled programmers and operators, and a lot of time and will. They know that with some persistent effort targeting a vendor, they can enter a lot of networks with ease. Hitting a systems management company, or more scarily, a security vendor, reaps BIG rewards because we invest in these products to secure our entire networks. The other big worry is those vendors that are deeply embedded with certain verticals such as finance or government. Imagine a vendor that is in every branch of a national government – one successful attack could bring down that entire government after  a wave of upgrades! Or hitting a well known payment vendor could open up every bank in the EU.

Monitoring & Alerting for Windows Defender in Azure VMs

In this post, I will explain how one can monitor Windows Defender and create incidents for it with Azure VMs.

Background

Windows Defender is built into Windows Server 2016 and Windows Server 2019. It’s free and pretty decent. But it surprises me how many of my customers (all) choose Defender over third-parties for their Azure VMs … with no coaching/encouragement from me or my colleagues. There is an integration with the control plane using the antimalwareagent extension. But the level of management is poor-none. There is a Log Analytics solution, but solutions are deprecated and, last time I checked, it required the workspace to be in per-node pricing mode. So I needed something different to operationalise Windows Defender with Azure VMs.

Data

At work, we always deploy the Log Analytics extension with all VMs – along with the antimalware extension and a bunch of others. We also enable data collection in Azure Security Center. We use a single Log Analytics workspace to enable the correlation of data and easy reporting/management.

I recently found out that a table in Log Analytics called ProtectionStatus contains a “heartbeat” record for Windows Defender. Approximately every hour, a record is stored in this table for every VM running Windows Defender. In there, you’ll find some columns such as:

  • DeviceName: The computer name
  • ThreatStatusRank: A code indicating the health of the device according to defender:
    • 150: Health
    • 470: Unknown (no extension/Defender)
    • 350: Quarantined malware
    • 550: Active malware
  • ThreatStatus: A description for the above code
  • ThreatStatusDetails: A longer description
  • And more …

So you can see that you can search this table for malware infection records. First thing, though, is to filter out the machines/records reporting that there is no Defender (Linux machines, for example):

let all_windows_vms =
Heartbeat
| where TimeGenerated > now(-7d)
| where OSType == 'Windows'
| summarize makeset(Resource);
ProtectionStatus
| where Resource in (all_windows_vms)
| sort by TimeGenerated desc

The above will find all active Windows VMs that have been reporting to Log Analytics via the extension heartbeat. Then we’ll store that data in a set, and search that set. Now we can extend that search, for example finding all machines with a non-healthy state (150):

let all_windows_vms = Heartbeat
| where TimeGenerated > now(-7d)
| where OSType == 'Windows'
| summarize makeset(Resource);
ProtectionStatus
| where Resource in (all_windows_vms)
| where ThreatStatusRank <> 150
| sort by TimeGenerated desc

Testing

All the tech content here will be useless without data. So you’ll need some data! Search for the Eicar test string/file and start “infecting” machines – be sure to let people know if there are people monitoring the environment first.

Security Center

Security Center will record incidents for you:

You will get email alerts if you have configured notifications in the subscription’s Security Center settings. Make sure the threshold is set to LOW.

If you want an alternative form of alert then you can use a Log Analytics alert (Scheduled Query Alert resource type) based on the below basic query:

SecurityAlert 
| where TimeGenerated > now(-5m)
| where VendorName == 'Microsoft Antimalware'

The above query will search for Windows Defender alerts stored in Log Analytics (by Security Center) in the last 5 minutes. If the threshold is freater than 0 then you can trigger an Azure Monitor Action Group to tell whomever or start whatever task you want.

Workbooks

Armed with the ability to query the ProtectionStatus table, you can create your own visualisations for easy reporting on Windows Defender across many machines.

 

The pie chart is made using this query:

let all_windows_vms =
Heartbeat
| where TimeGenerated > now(-7d)
| where OSType == 'Windows'
| summarize makeset(Resource);
ProtectionStatus
| where TimeGenerated > now(-7d)
| where Resource in (all_windows_vms)
| where ThreatStatusRank <> '150'
| summarize count(Threat) by Threat

With some reading and practice, you can make a really fancy workbook.

Azure Sentinel

I have enabled the Entity Behavior preview.

Azure Sentinel is supposed to be the central place to monitor all security events, hunt for issues, and where to start investigations – that latter thanks to the new Entity Behavior feature. Azure Sentinel is powered by Log Analytics – if you have data in there then you can query that data, correlate it, and do some clever things.

We have a query that can search for malware incidents reported by Windows Defender. What we will do is create a new Analytic Rule that will run every 5 minutes using 5 minutes of data. If the results exceed 0 (threshold greater than 0) then we will create an incident.

let all_windows_vms =
Heartbeat
| where TimeGenerated > now(-7d)
| where OSType == 'Windows'
| summarize makeset(Resource);
ProtectionStatus
| where TimeGenerated > now(-5m)
| where Resource in (all_windows_vms)
| where ThreatStatus <> 'No threats detected' or ThreatStatusRank <> '150' or Threat <> ''
| sort by Resource asc
| extend HostCustomEntity = Computer

The last line is used to identity an entity. Optionally, we can associate a logic app for an automated response. Once that first malware detection is found:

You can do the usual operational stuff with these incidents. Note that this data is recorded and your effectiveness as a security organisation is visible in the Security Efficiency Workbook in Azure Sentinel – even the watchers are watched! If you open an incident you can click investigate which opens a new Investigation screen that leverages the Entity Behavior data. In my case, the computer is the entity.

The break-out dialogs allow me to query Log Analytics to learn more about the machine and its state at the time and the state of Windows Defender. For example, I can see who was logged into the machine at that time and what processes were running. Pretty nice, eh?

 

Microsoft Ignite 2019 – End-to-End Security for All Your XaaS Resources

Speaker: Yinon Costica

Intelligent Security

  • Identity and access management
  • Threat protection
  • Information protection
  • Cloud security

Threat Actors

Exposure -> Access -> Lateral Movements -> Actions

How Your Teams and Users Work With The Cloud

  • Users use SaaS (sanctioned), apps you build.
  • Developers code apps you build, deploy to IaaS/PaaS (sanctioned).
  • DevOps operate apps you build and IaaS/PaaS (sanctioned).

Plus there is un-sanctioned SaaS/IaaS/PaaS

Where Do Problems Occur?

DevOps:

  • Misconfigured resources
  • Infrastructure vulnerabilities
  • Open network ports

Developers

  • Secret leakage in code
  • App vulnerabilities
  • Open source vulnerabilities

Users:

  • Passwords
  • More

Protect the Infrastructure

Not just VMs. Visibility and protection across all resources and cloud with Azure Security Center.

  • Visibility with Secure Score
  • Avoid misconfigurations with control plane recommendations
  • Patch infrastructure vulnerabilities
  • Close open endpoints using AI powered attack surface reduction controls

Driving Secure Score Through the Organization

AF: I don’t use Secure Score because too many recommendations are wrong and Secure Score changes without infrastructure changes, so a hammer is swung without mistakes.

ASC uses Azure Policy to run an assessment. Driving secure score using governance.

More workloads added to ASC

Didn’t have a chance to note them, but I saw AKS and Key Vault in there.

AKS

  • Protecting the IaaS hosts
  • Protecting the containers

DevOps Good Practices

  1. Good hygiene
  2. Turn on threat protection
  3. Reduce your attack surface
  4. Integrate alerts into your SIEM.
  5. Identify root cause

Shipping Secure Applications

  1. Build secure applications – security is in the pipeline
  2. Protect every layer of the application
  3. Use guidance – best practices, Secure DevOps toolkit.

Securing Your Codebase with GitHub

Understand and secure your software supply chain – very important with opensource. See dependency insights and dependabot. Get automated security alerts and version patches.

And more.

Protect the Usage

Average app uses 1,000 apps.

Cloud App Security. I lost interest here – sorry!

Do Not Enable Azure Storage Account Firewall – IaaS

If you read through the security recommendations in Azure Security Center, you do get given out to a lot. A lot of it makes no sense if you understand Azure and the recommendations. One that appeared to make sense was to enable the relatively new firewall in Azure Storage:

  • Only allow trusted subnets – nice idea to limit the attack surface on the storage account in conjunction with service endpoints.
  • Allow “trusted Microsoft services” to access the storage account (on by default).

Note: A storage account can only be connected if you know one of the really long random access keys.

But if you do enable this firewall in an Azure deployment, things will break:

  • Boot Diagnostics: Does not know how to write to a secured storage account, even with firewall rules and service endpoints enabled.
  • Serial Console Access: Requires Boot Diagnostics to be working so that’s dead too.
  • NSG Flow Logs/Traffic Analytics: Another feature that doesn’t understand a secured storage account, even with “trusted Microsoft services” marked as enabled (default).

And there might be more!

So you have to aks yourself – do you want maximum security or a usable & manageable system? Storage account firewalls are pretty new – we didn’t need them a few months ago. So we can drop that feature, and maybe use the new Advanced Threat Protection for storage accounts feature instead?

It’s a pit that some joined-up thinking and integration testing weren’t done here.