Optimized Kubernetes cluster architecture: considerations and best practices
Kubernetes is a powerful platform for managing containerized applications at scale, but configuring a Kubernetes...
Jul 27, 2021
Kubernetes helps with scaling, deploying, and managing containerized workloads, facilitating a faster deployment cycle and configuration management—all while providing improved access control. Kubernetes is also a CNCF project, meaning it’s cloud-native and can be easily deployed through any cloud provider.
This blog will compare on-premises, or self-hosted, Kubernetes clusters to managed ones, as well as outline your options for Kubernetes in the cloud. To do this, we’ll look at ease of use and set-up, custom node support, cost, release cycles, version support, and more.
Building and maintaining infrastructure requires both experienced engineers and domain experts. But not every organization can assemble such a dream team. Domain experts are rare, and almost all of them are already working with large software companies, tech giants, etc.
So, when choosing between managed and self-hosted Kubernetes, here are the main points you’ll need to take into consideration.
Self-managed Kubernetes means you’re running the Kubernetes installations, either in your data center or on virtual machines in the cloud. This entails a separate cost attached to the machines used to run your control plane, meaning you’ll have to plan for high availability and disaster recovery on your own. You’ll also have to set up automation to scale the nodes along with dependencies and provision your network for increased load.
As noted above, self-managed Kubernetes requires a big team that understands the deployments of different Kubernetes components. Your team will need to be able to handle etcd, the control plane, nodes, the Container Network Interface (CNI), service mesh, and small components like RBAC.
With managed Kubernetes, you don’t have to manage etcd or control planes, and many managed Kubernetes oversee their CNI and service mesh for you. This makes it a lot easier for a small team of three to five people to handle a Kubernetes cluster than if you went with self-managed clusters.
Upgrading clusters is a big undertaking and can take a lot of time if you’re handling it yourself—including planning and researching changes in the next Kubernetes version, as well as components or APIs that are deprecated.
Managed Kubernetes, on the other hand, can be easily upgraded in one or two steps. You don’t need to take care of an etcd backup or make sure you individually upgrade control plane nodes to maintain high availability. All of this is taken care of for you by your cloud provider.
That said, relying on your provider to release a patched version of Kubernetes can cause a delay in fixing vulnerabilities, like in the case of the runC vulnerability in 2018. Meanwhile, the release cycle for a new Kubernetes version takes six to nine months.
Moving between cloud providers costs more than money. You might also incur performance and reliability costs. It’s important to consider which provider will best meet your needs so you can avoid these peripheral costs. If you’re managing your infrastructure as code, there will be changes required for this migration as well.
If you’re thinking about going with cloud-agnostic Kubernetes clusters, you may consider managing them yourself, as this will give you the flexibility to move clusters across clouds since you aren’t depending on any cloud resources except the underlying machine. You have to choose your CNI very carefully though, as not all CNIs work directly on all clouds.
Managing your own Kubernetes will require in-house Kubernetes experts who can dig deeper into issues and find are solution.
Given the above considerations, a managed Kubernetes cluster is preferable to a self-managed one if:
You’ll find managed Kubernetes are easier to upgrade and highly available and—most importantly—you’ll have support from your cloud provider. There are many managed Kubernetes options to choose from; we’ll take a look at the main ones below.
Below are the key factors you should consider when looking into managed Kubernetes clusters.
In most cases, you’ll be using kubectl from the command line instead of the GUI. But for the most intuitive UI, Amazon EKS is very easy to use, giving you only the options you need. AKS and GKE, on the other hand, have a lot of options that you may not use. OpenShift and Tanzu come with a custom UI for a better developer experience.
Generally speaking, deploying from the UI is discouraged. You typically wrap your deployments with pipelines that are supposed to deploy on Kubernetes, so having a UI doesn’t add a lot of value.
On GKE, it’s easier to manage Day-2operations, and it gives the unique option of config sync, allowing you to sync your Kubernetes cluster from a git repository.
All of the providers noted above are easy to set up.
Generating a kubeconfig file in GKE and getting started is especially easy. With AKS and EKS, you have to make changes in the IAM, then add the user to the cluster. Google provides a few options for deploying different CNI plugins and a service mesh, while AWS and AKS are beginners in this area. However, you can always deploy CNI and service mesh solutions on top of your clusters—if you’re willing to manage them yourself. Finally, OpenShift comes packed with its own components and has Istio as its service mesh. VMware Tanzu also has its own CNI and service mesh.
Each of the providers mentioned has great integration with their other cloud offerings, like RDS or storage, through an internal network of various resources. AWS, for example, provides the most reliable platforms, while Azure offers a lot of configuration options.
It’s also good to note that AWS use slayer 2 networking, while AKS and GKE use layer 3 networking. So, if you’re planning to use CNI, which uses BGP for routing, you may face issues in GKE and AKS.
Due to the extra features it provides, OpenShift is the most expensive among the options listed here. AKS and EKS have almost the same cost, while GKE is cheaper than the other two. This is thanks to GKE’s sustained use discount feature, which means their machines become cheaper as you use the same VM more in a billing cycle. You can see a more detailed comparison of costs in Figure 1 below.
AKS does not support custom worker nodes and has no plans to include it. GKE and OpenShift don’t offer it either. In fact, such nodes are restricted in GKE, meaning you cannot create a node that has basic tooling or software, like installing antivirus and then using it as a Kubernetes node. On the other hand, EKS has very good support for custom worker nodes and VMware Tanzu supports them as well.
Node upgrades are automated in GKE and AKS, while you have to manually update the worker nodes in EKS. AKS also provides a manual option for node replacement. When you upgrade on GKE, you don’t have control over how, or in what order, nodes are upgraded, thus your workload will be affected.
If the workload is stateful, this can cause issues. Manual node upgrades give you control over your own node replacement, so you can move your workload before taking the VM down for an upgrade.
The table below provides a convenient reference for these many aspects of the primary managed Kubernetes solutions.
Kubernetes clusters take time and manpower to set up. That might mean a few minutes in the cloud or hours in a self-hosted version of Kubernetes. Ultimately, your ability to handle mammoth open-source projects is the biggest factor to consider when choosing managed over on-premises Kubernetes. If you choose to go it alone, you’ll need experts on your team who can handle such a large undertaking. Separately managing etcd, upgrades, high availability, and reliability requires far more expertise than running a managed Kubernetes cluster.
For your managed solutions, there are a number of options. If your budget allows and you want many features already deployed, OpenShift is the best option. VMware Tanzu is also a great choice if you’re considering CI/CD, are worried about security, and are already using VMware products. You should, however, note that you cannot run more than 150 nodes in VMware Tanzu Kubernetes clusters. If you’re looking to play with networking, KS’ layer 2 networking makes it a good choice, and EKS also offers more pods per node and a lot of nodes. On the other hand, AWS has world-class support, and GKE may offer substantial savings.
Handling self-managed Kubernetes is a huge task. When you combine it with demands like security and compliance, it grows exponentially harder to manage. These cloud providers implement security with pre-defined constructs like security groups, firewalls, and subnet and VNet segregation. For security and auditing, you can deploy tools like Gatekeeper, Falco, and Wazuh.
You can also check out ARMO, which provides more visibility and control while also taking care of security and compliance for you. Watch a demo to get started today.
Kubernetes is a powerful platform for managing containerized applications at scale, but configuring a Kubernetes...
Learn about Kubernetes compliance challenges, consequences of non-compliance, and get guidance on maintaining a secure...
A detailed overview of the Remote Code Execution (RCE) attacks, how it affects the Kubernetes...