So… do you use Kubernetes?

A question i been asked often or even find my self asking is:

“So… do you use Kubernetes?”

This has been asked by all entities customers, developers, sys-arch, DevOps and others. This is also a question i ask myself after been heavily engage in designing cloud architecture (in an AWS guy, no one is perfect) when planning a new service road map. Why do i use Kubernetes as my primary deployment platform ?

Why Kubernetes?

K8S is the most widely known system for large scale orchestration of containerized software. Other container orchestration as Mesos is rapidly fading in popularity, the less said about Docker Swarm is following it. Kubernetes was developed by engineers with serious large scale production workload experience, applying real-world proven techniques from Google’s Borg orchestration system. It has endless vendor support, and you don’t launch a software product these days without official Docker images and a Kubernetes deployment guide.

Most Kubernetes deployments today seem to happen on virtual machines, most often on public cloud providers. The original project Borg was mainly focus on baremetal deployments. Public cloud is where a major motivation comes in for many organizations deploying Kubernetes today.cDevelopers are already comfortable with Docker, and Kubernetes makes it easy to get the same container running in production. With namespaces, built-in resource management, as well as built-in virtual networking features, it also allows deploying multiple interacting services maintained by different teams without stepping on each other’s feet.

A different journey

Lets assume your company uses containers and run on AWS, lets assume they company use only EC2 that host the containers. Lets also assume that there is no orchestration layer for those containers. Lets see how we can get the same solution without it. When created, each instance already knows which containers to run based on which autoscaling group it is in. For automation you can use a boot service on any instance as part of the boot image that pull the right container image and start it on the instance. There is no scheduler that decide of the instance is back or front end, instead each container is assigned to the instance that is part of the overall VPC design. As for monitoring a services on each instance could respawn a required container if it dies, and self-terminate the instance if it is running a version of any software that is no longer the preferred version for that cluster.

For traffic intake, an NLB could do the work. As one autoscaling group equals one production service, we can use the normal AWS method of pointing one NLB at one autoscaling group as the target group, with no additional abstraction layer required. For internal service-to-service communication, just simply use AWS network. Because services are not arbitrarily mixed on machines, the machine’s automatically assigned IP address is and could be sufficient.

Why containers if you do not plan to use K8S?

Docker is still a rather convenient format for deploying software, the deployable unit is thousands of interdependent files in complex directory trees, plus an execution runtime that needs to be the exact right version for that snapshot of the source tree. You can distribute your software builds as simple tarballs with the management service on each instance downloading and unpacking them. As Docker images are just a collection of tarballs bundled with a JSON , crul and tar have been replaced by Docker pull.

Resource management

When it comes to resource management, a decision which EC2 type should be done based on the needs. Most software services can only use a certain amount of resources effectively. A process with two threads doesn’t need 16 CPUs; a process that only writes to disk once a minute doesn’t need SSD storage with capacity for 90,000 writes per second, alternative CPU architecture can be better value for money. Picking the right components out of the many AWS offerings allows to minimize overhead and control costs.

Autoscaling

EC2 autoscaling groups know how to automatically increase or decrease the number of instances in the group to meet demand. The tooling available is similar to Kubernetes. The desired number of instances in each autoscaling group can be manually set at any time, and the autoscaling policies will take over again from there.

Traffic Ingress

Traffic ingress is a solved problem on all major cloud providers — that is, as long as you can consistently map a service that accepts external traffic to the set of machines it is running on. Customer traffic can either reach NLB or first meet CloudFront. From the perspective of services running within each region, it doesn’t make a difference. Using a load balancer in each region (assuming you deploy your app in multi region) is another option to enable horizontal scalability, in this case to handle significant changes in the number of connections, such as when the super-ball starts.

DevOps

DevOpa engineers can participate meaningfully in the management of teh production systems, developers can set a single configuration value — “This cluster should be running this version of this component now, please” — and all instances of that service will be replaced by instances running the new version over time.

And with Kubernetes?

How would environments and processes change if you migrated production to Kubernetes?

Resource management

When managing resources via Kubernetes, instead of using EC2 instances sized appropriately for each service, we would use large instances and pack them with containers. Packing servers has low advantage of using spare resources on existing machines instead of additional machines for small-footprint services. It also has the major disadvantage of running heterogeneous services on the same machine. Cloud providers have the same problem — known as “noisy neighbors” — with virtual machines. However, cloud providers have a decade’s of experience in their systems to mitigate this issue for their customers as much as possible. On Kubernetes, you get to solve it yourself.

One possible approach is to attempt to preserve the one VM for one service model while using Kubernetes. The Kubernetes worker node don’t have to be identical, they can be virtual machines of different sizes, and Kubernetes scheduling constraints can be used to run exactly one logical service on each worker mode. So if you are running fixed sets of containers on specific groups of EC2 instances, why do you have a Kubernetes layer in there instead ?

Autoscaling

The Kubernetes cluster would be able to launch additional service pods only as long as capacity is available on a cluster node. Scaling the cluster up is relatively simple for the cluster autoscaler. Scaling down, however, gets complicate, you will likely end up with nodes that are mostly idle, but not empty. Remaining pods need to be migrated to other nodes to create an empty node before terminating that node to shrink the cluster. On autoscaling it should work similarly to how it does now, but instead of one autoscaling problem we would be solving two autoscaling problems, and they are both more complicated than the one we have now. On-demand or scheduled capacity management (manual intervention) would be a bit more complicated, as we would first have to ensure that there are enough Kubernetes nodes, and only then that there are enough pods for the required services.

Traffic Ingress

Traffic Ingress would be easy with Kubernetes. The AWS EKS team has made some commendable design choices there if the cluster is configured that way, each pod receive an AWS managed IP address fully integrated with the VPC. These IPs are directly reachable for things running inside the cluster, from things running on AWS in the same VPC but outside the cluster, and through both kinds of AWS virtual load balancers. A controller will automatically create AWS load balancers and point them directly at the right set of pods when an Ingress or Service section is added to the Kubernetes specification for the service. Downside , is that this level of integration is completely AWS-specific. For anyone trying to use Kubernetes as a way to go multi-cloud, it is very difficult.

DevOps

In Kubernetes, software version management would be functionally very similar to what you have now. Instead of specifying a new target version in your custom configuration system and all EC2 instances automatically going through rolling replacements, developers would specify a new target version in a K8S deployment and the pods would go through rolling replacements.