Unraveling the State of Kubernetes Security in 2023
In this post we’re going to unpack some of Red Hat State of Kubernetes security...
Apr 18, 2023
ARMO is launching a new Kubernetes vulnerability relevancy and prioritization feature based on eBPF technology to help Kubernetes and DevSecOps practitioners focus on fixing the vulnerabilities that impact their security posture most.
“CVE shock” is a term that describes the overwhelming feeling that devops and security teams experience when faced with a large number of vulnerabilities. The volume of vulnerabilities can make it challenging to prioritize and mitigate those that are the most critical. The resulting overwhelming feeling, and absence of clear indications of priority, can paralyze teams. This can result in delays, or even indefinite deferral of the problems at hand, leading to an increased risk of cyber attacks and data breaches. Which will hurt a business’ reputation and bottomline.
When scanning containers for vulnerabilities, scanners consider the entire image, with all the libraries and software artifacts that are packaged as part of it. The scanner will cover everything, including packages that are not in memory at runtime. This can result in dozens or even hundreds of vulnerabilities identified in a typical environment. Patching and managing vulnerabilities that are not relevant to the environment in question results in a huge waste of resources.
Today, we are pleased to announce a new capability: Relevancy and prioritization. Relevancy and prioritization allows ARMO Platform and Kubescape users to deprioritize vulnerabilities that belong to unused software packages and components. By first deprioritizing vulnerabilities that are less critical, users can focus on addressing the ones that pose a greater threat to their running cluster.
This approach helps the vulnerability assessment and remediation process become more efficient and effective, ultimately, strengthening security posture and saving both time and money. We have seen an average reduction of between 60% – 80% of vulnerabilities to manage, allowing teams to focus on the vulnerabilities most relevant to their specific environments.
ARMO Platform’s relevancy and prioritization feature is based on eBPF. An open-source agent scans your running environment, mapping out all the artifacts and libraries that are loaded into memory, and that actually get executed. The relevancy and prioritization feature uses this information to identify and pinpoint the most relevant vulnerabilities for each customer-specific environment, based on the environment variables used, and the specific use case of every customer.
We all know that in the context of Kubernetes a container image is a self-contained package of code, runtime, libraries, and system tools required to run an application or service.
Vulnerabilities can make their way into container images in several ways:
In a perfect world, you would use minimal base images that include fewer vulnerabilities. There are many practical reasons why this isn’t immediately applicable for everyone. Some software requires certified base images. Another example is software that is used elsewhere in the company on VMs and thus the development team wants to use the same Linux distribution in the Kubernetes environment. As a result, we find many customers using images without employing the full extent of their content.
Vulnerabilities and exposures are often referred to as “CVEs”, from the Common Vulnerabilities and Exposures database maintained by MITRE. It is generally considered a best practice to prioritize and treat CVEs according to their severity, with NIST adding a recommendation for ordering based on the number of machines a vulnerability impacts.
The severity of a CVE is expressed by a rating, ranging from negligible to critical. Critical vulnerabilities are perceived as posing the highest risk to an organization. Factors that contribute to a vulnerability’s severity include its potential impact on the confidentiality, integrity, or availability of a system, as well as the ease with which it can be exploited.
By prioritizing CVEs based on their severity, organizations focus their resources and efforts on addressing the most critical vulnerabilities first. This is perceived as helping reduce the risk of cyber attacks and data breaches. This approach also helps organizations avoid getting overwhelmed by the volume of CVEs and ensure that their security efforts are focused on the risks perceived as most significant.
The astute reader will have noticed how often I used the word perceived in the section above. The reasoning is that companies often use images without employing the full extent of their content.
At ARMO, we have found that patching CVEs solely according to severity is a lot of work that may have only a marginal impact on your security posture, even after the critical vulnerabilities are addressed. Relevancy, or a contextual visibility into the vulnerabilities that actually open you up to attack, is a much better starting point. This is achieved by finding out which packages the pod actually uses. This, in effect, drives whether the CVE puts you at obvious risk, or not.
The next step is to formulate your plan. We recommend you first fix vulnerabilities that are both fixable (i.e. a known fix exists) and have a network attack or remote code execution (RCE) vector (depicted as RCE in the diagram). Next, mitigate the relevant CVEs with a network attack vector (depicted as mitigate) and patch those that are fixable but do not have RCE capabilities. After these steps, you can deal with the rest. However, it’s important to note that while this feature helps prioritize vulnerabilities, it’s not an excuse to ignore any CVEs in your system. For the best security posture all vulnerabilities should be patched, mitigated, or marked as a false positive.
Today, vulnerability scanners work by cross-referencing the packages you have in an image against an industry-recognized database. As we have shown, this will typically result in a long list of vulnerabilities that you will be expected to prioritize and fix.
While we all strive to create and use the leanest containers possible, this is not always a top priority for a busy DevOps team. The reality is that most customers have containers full of code that is not used at runtime.
We propose you think about vulnerabilities in another way.
What matters most for prioritization is whether the software component is used at runtime. If it isn’t, it can be removed from the Software Bill of Materials (SBOM) that is fed into the vulnerability scanner. This two-pass approach means we have two lists: the full list, as you have had up until now, and a shorter (lower noise) and more relevant (higher signal) list for priority attention.
You shouldn’t leave insecure software around, because someone who can gain access to it may be able to use it to gain extra capabilities. But, you will want to prioritize fixing other vulnerabilities first.
ARMO has built relevancy detection into Kubescape, the CNCF-hosted engine for scanning containers and clusters for misconfigurations and vulnerabilities.
In a representative example, the Kubescape container scanner found 134 vulnerabilities in a particular release of the nginx container, and the relevancy engine was able to focus down on 8, with none considered Critical or High.
apiVersion: kubescape.io/v1 kind: RuntimeVulnSummary metadata: creationTimestamp: "2023-04-15T05:24:30Z" generation: 1 name: namespace-default.deployment-nginx.name-nginx-6799fc88d8-fmnlz resourceVersion: "69738865" uid: 7890b55e-d083-4db8-916b-c2a6065817cb spec: imageName: docker.io/library/nginx@sha256:970fab39f49cfac758a2c9a73414a483fc56bded6f8c578c651408567dceb356 summary: description: 8 relevant vulnerabilities detected from 134 images imageVulns: all: 134 critical: 2 high: 17 low: 6 medium: 21 negligible: 86 runtimeVulns: all: 8 critical: 0 high: 0 low: 2 medium: 2 negligible: 4
This vulnerability information is uploaded to ARMO Platform, where our vulnerability management console helps focus you on the issues that require your attention based on actual usage patterns from your own environment.
We added a new agent to the Kubescape in-cluster components, which we call the Taster. The Taster uses eBPF probes to look at the file activity of a running container. When a pod starts on a node, the taster will watch its containers for a configurable learning period — our default is 10 minutes — and store an activity log.
During the process of scanning a container, an SBOM is generated. This contains the vulnerability scanner’s understanding of which components are installed in the container. When we check for vulnerabilities we also provide the engine with a filtered SBOM, including only the packages that relate to files that were accessed during the learning period. This provides us a filtered list of vulnerabilities that are more likely to be relevant.
In order to increase the signal to noise ratio of the results of vulnerability scanning, you need to see what’s happening inside the container. Once you have an SBOM, you want to find out what components you actually use and then scan them for vulnerabilities. The results will be a shorter and more manageable list of vulnerabilities. Patching these vulnerabilities first can have a direct impact on your security posture, your team’s capacity and your bottom line.
So, next time you do a vulnerability scan, make sure your results are relevant and actionable. Try ARMO Platform and get to the vulnerabilities that really matter.
If you’d like to dive deeper into this topic, you can watch this talk by ARMO CTO, Ben Hirschberg.
Check out the Kubescape blog soon for an upcoming post about how we implemented the eBPF analysis in Kubescape, and how you can use in-cluster objects to get access to relevancy data.
Experience effective, end-to-end, from dev to production, Kubernetes protection:
Manage Kubernetes role-based-access control (RBAC) visually
Eliminate misconfigurations and vulnerabilities from your CICD pipeline – from YAML to cluster
Full Kubernetes security compliance in a single dashboard
In this post we’re going to unpack some of Red Hat State of Kubernetes security...
Guest post originally published on Jit's blog by Aviram Shmueli, Co-Founder, Chief Research and Innovation...
Kubernetes misconfigurations can create security risks and other issues. This article covers common types, their...