Overcoming CVE Shock with Effective Kubernetes Vulnerability Scanning

ARMO’s new feature revolutionizes Kubernetes vulnerability scanning based on eBPF technology to help Kubernetes and DevSecOps practitioners focus on fixing the vulnerabilities that impact their security posture the most

Revolutionizing Kubernetes vulnerability scanning: introducing relevancy and prioritization

“CVE shock” is a term that describes the overwhelming feeling that DevOps and security teams experience when faced with a large number of vulnerabilities. The volume of vulnerabilities can make it challenging to prioritize and mitigate those that are the most critical. The resulting overwhelming feeling, and absence of clear indications of priority, can paralyze teams. This can result in delays, or even indefinite deferral of the problems at hand, leading to an increased risk of cyber-attacks and data breaches. Which will hurt a business’s reputation and bottom line.

When scanning containers for vulnerabilities, scanners consider the entire image, with all the libraries and software artifacts that are packaged as part of it. The scanner will cover everything, including packages that are not in memory at runtime. This can result in dozens or even hundreds of vulnerabilities identified in a typical environment. Patching and managing vulnerabilities that are not relevant to the environment in question results in a huge waste of resources.

We’re excited to introduce a new feature that takes Kubernetes vulnerability scanning to the next level. Our relevancy and prioritization feature allows ARMO Platform and Kubescape users to focus on the most critical vulnerabilities, transforming the way we approach Kubernetes vulnerability scanning. By first deprioritizing vulnerabilities that are less critical, users can focus on addressing the ones that pose a greater threat to their running cluster.

This approach helps the vulnerability assessment and remediation process become more efficient and effective, ultimately, strengthening security posture and saving both time and money. We have seen an average reduction of between 60% – 80% of vulnerabilities to manage, allowing teams to focus on the vulnerabilities most relevant to their specific environments.

ARMO Platform’s relevancy and prioritization feature is based on eBPF. An open-source agent scans your running environment, mapping out all the artifacts and libraries that are loaded into memory, and that actually get executed. The relevancy and prioritization feature uses this information to identify and pinpoint the most relevant vulnerabilities for each customer-specific environment, based on the environment variables used, and the specific use case of every customer.

Where do Kubernetes vulnerabilities come from?

We all know that in the context of Kubernetes a container image is a self-contained package of code, runtime, libraries, and system tools required to run an application or service. 

Vulnerabilities can make their way into container images in several ways:

• Base image vulnerabilities: Images are often built using a base image, which is a pre-built image that contains a minimal runtime environment and set of tools required to build more container image layers and run an application or service. If the base image has known vulnerabilities, those vulnerabilities will also exist in the final image.
• Vulnerabilities in application dependencies: Images are built with the application code and its dependencies, including third-party libraries. If any of these dependencies have vulnerabilities, they can be exploited by attackers to compromise the image or the container. 

In a perfect world, you would use minimal base images that include fewer vulnerabilities. There are many practical reasons why this isn’t immediately applicable for everyone. Some software requires certified base images. Another example is software that is used elsewhere in the company on VMs and thus the development team wants to use the same Linux distribution in the Kubernetes environment. As a result, we find many customers using images without employing the full extent of their content.

Best practices for prioritizing CVEs in Kubernetes vulnerabilities

Vulnerabilities and exposures are often referred to as “CVEs”, from the Common Vulnerabilities and Exposures database maintained by MITRE. It is generally considered a best practice to prioritize and treat CVEs according to their severity, with NIST adding a recommendation for ordering based on the number of machines a vulnerability impacts.

The severity of a CVE is expressed by a rating, ranging from negligible to critical. Critical vulnerabilities are perceived as posing the highest risk to an organization. Factors that contribute to a vulnerability’s severity include its potential impact on the confidentiality, integrity, or availability of a system, as well as the ease with which it can be exploited.

By prioritizing CVEs based on their severity, organizations focus their resources and efforts on addressing the most critical vulnerabilities first. This is perceived as helping reduce the risk of cyber attacks and data breaches. This approach also helps organizations avoid getting overwhelmed by the volume of CVEs and ensure that their security efforts are focused on the risks perceived as most significant.

Relevancy – the new Kubernetes vulnerability scanning best practice

The astute reader will have noticed how often I used the word perceived in the section above. The reasoning is that companies often use images without employing the full extent of their content.

At ARMO, we have found that patching CVEs solely according to severity is a lot of work that may have only a marginal impact on your security posture, even after the critical vulnerabilities are addressed. Relevancy, or a contextual visibility into the vulnerabilities that actually open you up to attack, is a much better starting point. This is achieved by finding out which packages the pod actually uses. This, in effect, drives whether the CVE puts you at obvious risk, or not. 

The next step is to formulate your plan.

We recommend you first fix vulnerabilities that are both fixable (i.e. a known fix exists) and have a network attack or remote code execution (RCE) vector (depicted as RCE in the diagram).

Next, mitigate the relevant CVEs with a network attack vector (depicted as mitigate). This mitigation may be out of scope for DevOps or Platform teams. Nevertheless, the security team need to know about them, so they can provide a stopgap that is not available within the Kubernetes infrastructure.

Next, patch those that are fixable but do not have RCE capabilities.

After these steps, you can deal with the rest.

It’s important to note that while this feature helps prioritize vulnerabilities, it’s not an excuse to ignore any CVEs in your system. For the best security posture all vulnerabilities should be patched, mitigated, or dealt with as false positives.

How does ARMO Platform calculate relevancy?

Today, vulnerability scanners work by cross-referencing the packages you have in an image against an industry-recognized database. As we have shown, this will typically result in a long list of vulnerabilities that you will be expected to prioritize and fix.

While we all strive to create and use the leanest containers possible, this is not always a top priority for a busy DevOps team. The reality is that most customers have containers full of code that is not used at runtime. 

We propose you think about vulnerabilities in another way.

What matters most for prioritization is whether the software component is used at runtime. If it isn’t, it can be removed from the Software Bill of Materials (SBOM) that is fed into the vulnerability scanner. This two-pass approach means we have two lists: the full list, as you have had up until now, and a shorter (lower noise) and more relevant (higher signal) list for priority attention. 

You shouldn’t leave insecure software around, because someone who can gain access to it may be able to use it to gain extra capabilities. But, you will want to prioritize fixing other vulnerabilities first. 

Relevancy – from theory to practice

ARMO has built relevancy detection into Kubescape, the CNCF-hosted engine for scanning containers and clusters for misconfigurations and vulnerabilities. 

In a representative example, the Kubescape container scanner found 134 vulnerabilities in a particular release of the nginx container, and the relevancy engine was able to focus down on 8, with none considered Critical or High.

apiVersion: kubescape.io/v1
kind: RuntimeVulnSummary
metadata:
  creationTimestamp: "2023-04-15T05:24:30Z"
  generation: 1
  name: namespace-default.deployment-nginx.name-nginx-6799fc88d8-fmnlz
  resourceVersion: "69738865"
  uid: 7890b55e-d083-4db8-916b-c2a6065817cb
spec:
  imageName: docker.io/library/nginx@sha256:970fab39f49cfac758a2c9a73414a483fc56bded6f8c578c651408567dceb356
  summary:
    description: 8 relevant vulnerabilities detected from 134 images
    imageVulns:
      all: 134
      critical: 2
      high: 17
      low: 6
      medium: 21
      negligible: 86
    runtimeVulns:
      all: 8
      critical: 0
      high: 0
      low: 2
      medium: 2
      negligible: 4

This vulnerability information is uploaded to ARMO Platform, where our vulnerability management console helps focus you on the issues that require your attention based on actual usage patterns from your own environment.

How the Taster agent improves Kubernetes vulnerability scanning with eBPF and SBOM

We added a new agent to the Kubescape in-cluster components, which we call the Taster. The Taster uses eBPF probes to look at the file activity of a running container. When a pod starts on a node, the taster will watch its containers for a configurable learning period — our default is 10 minutes — and store an activity log.

During the process of scanning a container, an SBOM is generated. This contains the vulnerability scanner’s understanding of which components are installed in the container. When we check for vulnerabilities we also provide the engine with a filtered SBOM, including only the packages that relate to files that were accessed during the learning period. This provides us a filtered list of vulnerabilities that are more likely to be relevant.

In order to increase the signal-to-noise ratio of the results of vulnerability scanning, you need to see what’s happening inside the container. Once you have an SBOM, you want to find out what components you actually use and then scan them for vulnerabilities. The results will be a shorter and more manageable list of vulnerabilities. Patching these vulnerabilities first can have a direct impact on your security posture, your team’s capacity and your bottom line.

Take action on relevant Kubernetes vulnerabilities with ARMO Platform

So, next time you do a vulnerability scan, make sure your results are relevant and actionable. Try ARMO Platform and get to the vulnerabilities that really matter.

If you’d like to dive deeper into this topic, you can watch this talk by ARMO CTO, Ben Hirschberg.

Check out the Kubescape blog soon for an upcoming post about how we implemented the eBPF analysis in Kubescape, and how you can use in-cluster objects to get access to relevancy data.