Unraveling the State of Kubernetes Security in 2023
In this post we’re going to unpack some of Red Hat State of Kubernetes security...
Apr 13, 2023
The Kubernetes 1.27 release brings multiple enhancements and security updates, providing users with an improved, more flexible, and secure platform for building and managing containerized applications. This post will provide an overview of the security-related updates and most significant enhancements in v1.27.
The Kubernetes release team has once again delivered a new version. The release title of “Chill Vibes” suggests a smooth release, and while there were 60 enhancements — as well as countless bug fixes — at this stage in the project’s maturity, thankfully there are few major changes or breakages. This post will provide an overview of the security-related updates and most significant enhancements in v1.27. We’ll dive into its key features; explore their impact on the Kubernetes ecosystem; and demonstrate how these updates can help users optimize their Kubernetes workloads and build more resilient, secure, and efficient applications.
We must, of course, lead with a deprecation which was first announced over a year ago: Kubernetes images are no longer going to be published to
k8s.gcr.io, the home of the project since its launch. To continue operating a secure environment you need to keep up to date with the latest versions. If you build and run your own Kubernetes environment (or use tooling which relies on downloading Kubernetes container images) then you must update to use
registry.k8s.io. in v1.27, as well as patch releases for the supported versions from v1.24 onwards, which will only be posted there.
The backend for
registry.k8s.io is dynamic, and will change as providers add mirrors. If you need to allow-list domains or IPs in your environment, the Kubernetes project highly recommends mirroring images to a location under your control.
Want to check you’re not using this registry for any of your containers? We have implemented a control in Kubescape to validate this.
Kubernetes 1.27 improves security by allowing the kubelet to use seccomp by default. This feature—graduating to stable in this release—reduces the attack surface of containers by limiting the system calls they can access. Thus providing a more secure environment for running applications. This feature must be enabled on each node: make sure you look at the upgrade strategy in the enhancement proposal first.
Pod scheduling readiness ensures that a pod is not considered ready until it has met certain conditions. This improves the efficiency of pod scheduling and prevents resources from being wasted on pods that are not yet ready to serve traffic.
To use the feature, you must add
schedulingGates to pods as a list of strings. The pod’s status will show as
SchedulingGated and be blocked from scheduling. When you want to remove the gates, you update the pod spec until all gates are removed. The scheduler will then continue working and assign a node for the pod.
This feature was introduced as Alpha in 1.26 and will now be available to most Kubernetes users having now reached Beta.
The Kubescape team ported many of our Rego controls for use with the new Validating Admission Policies feature released in v1.26.
Kubernetes 1.27 adds support for match conditions, providing a more expressive and efficient way to define policy webhook triggers. This enhancement simplifies configuration and allows users to optimize the execution of their admission control policies.
For example, with the new release, you can exclude resources from a wildcard rule such as:
rules: # Match CREATE & UPDATE on all resources: - operations: - CREATE - UPDATE apiGroups: '*' apiVersions: '*' resources: '*' matchConditions: - name: 'exclude-leases' expression: '!(request.resource.group == "coordination.k8s.io" && resource.resource == "leases")'
We are working with SIG API Machinery on how to group policies to be able to take advantage of these new features while still enabling policies atomically.
Other enhancements to validating admission policies in this release include runtime cost calculation, custom failure messages, and type checking.
Kubernetes has no resources to identify and manage users as part of its API. Instead, it uses authenticators to get user attributes from tokens, certificates, OIDC providers, or webhooks. v1.26 introduced a new API endpoint to see what attributes the current users have, and it has moved to Beta in 1.27.
SelfSubjectReview API powers the
kubectl auth who-am-i command and the feature will reduce the obscurity of complex authentication. This will help users debug the authentication stack.
Kubernetes publishes a feed of its vulnerabilities, launched in Alpha in v1.25. Moving to Beta in v1.27, improvements include publishing as valid JSON, Atom format for reading in RSS readers, and ordering from most recent to least recently announced.
Kubectl now supports fetching logs from nodes, making it easier for users to diagnose and troubleshoot kubelet issues. With the new API endpoint in
kubelet and the corresponding implementation in
kubectl, you can run a command like this:
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"
See the documentation for more information.
Kubernetes allows a configurable grace period for probes. Now users can adjust the time between the termination of a container and the deletion of its associated resources. This update offers more flexibility in managing the lifecycle of containers, enabling users to fine-tune their Kubernetes workloads. By setting
terminationGracePeriodSeconds in the probe as follows, you can define how long the kubelet should wait before forcibly terminating when the probe fails:
livenessProbe: httpGet: path: /health port: liveness failureThreshold: 1 periodSeconds: 60 # New field # terminationGracePeriodSeconds: 60
This feature is Stable as of v1.27.
The v1.27 release brings support for gRPC in liveness, readiness, and startup probes to stable. This feature allows users to query gRPC services for their state, saving the need to also run a HTTP server or connect with TCP..
For example, it is now possible to use gRPC in readiness probes:
readinessProbe: grpc: port: 9090 service: my-service initialDelaySeconds: 5 periodSeconds: 10
Timezone support is now Stable in CronJob, enabling users to schedule jobs based on specific time zones. Before this feature, you could set the schedule for a CronJob, but the time zone used during the job creation depended on where kube-controller-manager was running.
This feature makes managing jobs across multiple regions easier and provides more flexibility for global organizations. The new
timeZone field in
CronJobSpec allows you to specify the time zone from the tz database.
Kubernetes now supports OpenAPI v3, which brings a more modern and feature-rich API specification format to the Kubernetes ecosystem. This update enables better documentation, validation, and tooling support for Kubernetes API users.
Here are a few smaller or newer enhancements that nonetheless caught the eye of our team:
The latest release introduces a new access mode for PersistentVolumes called
ReadWriteOncePod. This allows a volume to be mounted as read-write by a single pod, improving data consistency and security for stateful applications.
Access mode is already part of the API and supports the
ReadWriteOncePod field as follows:
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: single-writer-only spec: accessModes: - ReadWriteOncePod # Allow only a single pod to access single-writer-only. resources: requests: storage: 1Gi
Kubernetes 1.27 adds support for
VolumeGroupSnapshot, a new feature that allows users to create consistent snapshots of multiple volumes simultaneously. This enhancement simplifies data backup and recovery for stateful applications running on Kubernetes, with new CRDs:
v1.27 introduces a new mechanism to classify pod failures as either retriable or non-retriable. Currently, Kubernetes restarts the pods of jobs when
backoffLimit is larger than zero. However, this approach overlooks the root cause of the failure, and restarts all failed pods.
The enhancement lets users better manage the execution of jobs and ensure that only necessary retries are performed; this, in turn, conserves resources and improves reliability.
kubectl explain command has been improved with better support for custom resource definitions (CRDs) and OpenAPI v3. This update makes it easier for users to understand and navigate the structure of their custom resources.
The Kubernetes 1.27 release brings multiple enhancements and security updates, providing users with an improved, more flexible, and secure platform for building and managing containerized applications.
For additional information on each feature and its implications, refer to GitHub and the Kubernetes project’s official blog, which will have deep dives into many of these features in the upcoming weeks. Don’t hesitate to explore these new features and updates, as they are sure to have a positive impact on your Kubernetes security.
Experience effective, end-to-end, from dev to production, Kubernetes protection:
Manage Kubernetes role-based-access control (RBAC) visually
Eliminate misconfigurations and vulnerabilities from your CICD pipeline – from YAML to cluster
Full Kubernetes security compliance in a single dashboard
In this post we’re going to unpack some of Red Hat State of Kubernetes security...
Guest post originally published on Jit's blog by Aviram Shmueli, Co-Founder, Chief Research and Innovation...
Kubernetes misconfigurations can create security risks and other issues. This article covers common types, their...