How to run data on Kubernetes: 6 starting principles
Kubernetes is fast becoming an industry standard, with up to 94% of organizations deploying their services and applications on the container orchestration platform, per a survey. One of the key reasons companies deploy on Kubernetes is standardization, which lets advanced users see productivity gains of up to two times.
Standardizing on Kubernetes gives organizations the ability to deploy any workload, anywhere. But there was a missing piece: the technology assumed that workloads were ephemeral, meaning that only stateless workloads could be safely deployed on Kubernetes. However, the community recently changed the paradigm and brought features such as StatefulSets and Storage Classes, which make using data on Kubernetes possible.
While running stateful workloads on Kubernetes is possible, it is still challenging. In this article, I provide ways to make it happen and why it is worth it.
Do it progressively
Kubernetes is on its way to being as popular as Linux and the de facto way of running any application, anywhere, in a distributed fashion. Using Kubernetes involves learning a lot of technical concepts and vocabulary. For instance, newcomers might struggle with the many Kubernetes logical units such as containers, pods, nodes, and clusters.
If you are not running Kubernetes in production yet, don’t jump directly into data workloads. Instead, start with moving stateless applications to avoid losing data when things go sideways.
If you can’t find an operator that matches your needs, don’t worry, because most of them are open-source.
Understand the limitations and specificities
Once you are familiar with general Kubernetes concepts, dive into the specifics for stateful concepts. For example, because applications may have different storage needs, such as performance or capacity requirements, you must provide the correct underlying storage system.
What the industry generally calls storage “profiles” is termed Storage Classes in Kubernetes. They provide a way to describe the different types of classes a Kubernetes cluster can access. Storage classes can have different quality-of-service levels, such as I/O operations per second per GiB, backup policies, or arbitrary policies, such as binding modes and allowed topologies.
Another critical component to understand is StatefulSet. It is the Kubernetes API object used to manage stateful applications, and offers key features such as:
- Stable, unique network identifiers that let you keep track of volume, and detach and reattach them as you please;
- Stable, persistent storage so that your data is safe;
- Ordered, graceful deployment and scaling, which is required for many Day 2 operations.
While StatefulSet has been a successful replacement for the infamous PetSet (now deprecated), it is still imperfect and has limitations. For example, the StatefulSet controller has no built-in support for volume (PVC) resizing — which is a major challenge if the size of your application data set is about to grow above the current allocated storage capacity. There are workarounds, but such limitations must be understood well ahead of time so that the engineering team knows how to handle them.
