Managing and Securing Kubernetes Resources: A Step-by-Step Guide

Providing developers with a container orchestration platform that allows them to scale applications easily and deploy them in a simple way has made Kubernetes the holy grail of container orchestration and microservices.

Kubernetes is popular because it improves the DevOps workflow impressively by automating deployments and managing containerized applications. Adopting Kubernetes is surely better than not using it, but getting the best out of it requires more skill and time even though the rewards are great as you will be able to reduce cluster costs efficiently.

Implementing Kubernetes security and managing the health of your Kubernetes cluster is vital for your company’s prosperity — and that’s why factors such as cost monitoring, resource management, and security should be taken into consideration when using Kubernetes.

This article will show you how to manage and secure Kubernetes resources effectively.

1. Set resource requests limits

Excessive or low resource consumption can be detrimental to your containers. When containers consume resources excessively, it leads to the noisy neighbours issue (where applications compete for the same resources and the performance is degraded). On the other hand, inadequate resource consumption leads to nodes running out of CPU resources.

Kubernetes enables you to control resource consumption by using resource request limits — a mechanism that restricts containers from passing a certain resource consumption set.

The following YAML file shows a Pod which has a container that has set its cpu limits to 128Mi and the memory set to 500m. These limits will prevent the containers from using excess resources.

---
apiVersion: v1
kind: Pod
metadata:
  name: newpod
spec:
  containers:
  - name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

After you have set resource limits it is highly recommended that you consistently monitor them to ensure that the request limit is not underestimated because that will degrade performance. On the other side of the spectrum, overestimated limits result in pods excessively consuming resources.

Understanding how your application consumes resources on a daily basis will help you get and set accurate resource limits.

2. Use an automated cost solution system

Managing Kubernetes cost by manually calculating cost per container is a cumbersome and painful procedure. Wherever human power fails, computer-automated features excel.

Automated cost management systems eliminate the need to calculate costs for each container — It generates cost reports, sets thresholds, and provides financial visualization. Some examples of such automated cost management systems include Kubecost and Cast.

• Kubecost is an automated cost management solution that monitors resource allocation and usage in order to calculate your Kubernetes spending and generate reports. It will notify you when you have reached financial thresholds and limitations you have set. In addition to this, Kubecost will give you suggestions on how you can reduce and optimize your costs without reducing the cluster’s performance. Automated cost solution systems like Kubecost are a great asset as you can set a threshold that alerts you when a certain amount of cost has been reached. This will help you take action and stop the containers from consuming resources excessively on time.
• Cast uses autoscaling and rightsizing concepts to automatically optimize your cluster’s performance in return reducing costs. It also provides cluster metrics and debugging which are crucial for decision making and rectifying mistakes. Cost metrics and visualizations help you keep track of how much money you’re spending and if we are being honest this data is very essential for making financial and resource allocation plans and decisions.

3. Analyze and inspect logs constantly

Logging gives you an insight into how applications are performing and the entire cluster’s health. Logging makes fixing and identifying problems in Kubernetes easy. It also helps you identify noisy neighbours that are draining resources excessively and failures such as invalid images and application errors in the container or node level.

To appropriately audit logs you have to log the following levels:

• Host-level
• Application-level
• Cloud infrastructure

How to log Kubernetes

You can start logging Kubernetes by writing text to the standard output stream every second. You can do this by setting the args field under the container map when creating a Pod as:

[/bin/sh, -c,
            'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']

Here is the full YAML file that creates a pod and sends text to the standard output stream.

apiVersion: v1
kind: Pod
metadata:
  name: logger
spec:
  containers:
  - name: count
    image: busybox:1.28
    args: [/bin/sh, -c,
            'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']

Use the following command to create the pod with the above content:

You will get the following output:

After your pod has been created successfully use the following command to fetch logs:

You will get the following output that shows that text is being written to the standard output stream:

0: Wed Apr  6 20:00:05 UTC 2022
1: Wed Apr  6 20:00:06 UTC 2022
2: Wed Apr  6 20:00:07 UTC 2022
3: Wed Apr  6 20:00:08 UTC 2022
4: Wed Apr  6 20:00:09 UTC 2022

4. Enhance cluster organizations by using Kubernetes namespaces and labels

Kubernetes is a dynamic and complex platform as applications are distributed across containers and are also administered by different people from various departments.

Namespaces are a mechanism that helps you categorize and divide a single cluster into multiple virtual clusters. While labels are key pairs that are attached to Kubernetes objects.

Namespaces and labels boost your ability to organize and structure your Kubernetes services when your Kubernetes environment has many developers and DevOps engineers working on the same Kubernetes cluster. Labels on the other hand help you filter kubectl outputs efficiently and visualize hierarchies of your API objects.

The following command is used to create a namespace:

After creating a namespace use the following command to get more details about the namespace:

You will get the following output:

Name:         devops-team
Labels:       kubernetes.io/metadata.name=devops-team
Annotations:  <none>
Status:       Active

No resource quota.
No LimitRange resource.

5. Use RBAC to strengthen Kubernetes security

Getting to control who accesses a specific Kubernetes resource is the beginning of Kubernetes security as it can help you and your team avoid various security vulnerabilities. Since Kubernetes is used by different departments in a company it is important to implement Role Based Access Control (RBAC).

RBAC enables you to set rules and permissions that define which users access what resource at namespace (aka role level) or at a cluster level. For example, you can limit who has access to etcd, etc.

For context, ClusterRoles set permissions that apply to the cluster level while Roles set and define permissions that apply to a specific namespace.

When implementing RBAC use the rbac.authorization.k8s.io/v1 apiVersion. The following YAML file is an example of RBAC. In this YAML file, a Role object is created which sets permissions for the namespace called organisation. This Role sets the rule that the user will access the services resource and the verbs field specifies which actions the user will be able to take. In this case, the user will be able to get, watch and list services in that namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: organization
  name: service-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["services"]
  verbs: ["get", "watch", "list"]

Use the following command to create the role:

You will get the following output:

A RoleBinding gives permission to the user stated by a Role. The following command creates a RoleBinding called test and binds it to the Role created earlier called service-reader. This RoleBinding gives the serviceaccount the permissions given by the role in the organization namespace:

You will get the following output:

rolebinding.rbac.authorization.k8s.io/test created

Use the following command to view the RoleBinding you just created:

You will get the following output:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  creationTimestamp: "2022-04-05T07:04:02Z"
  name: test
  namespace: organization
  resourceVersion: "47073"
  uid: f169775d-ffd0-4b2b-9e7d-a4125d09644a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: service-reader
subjects:
- kind: ServiceAccount
  name: default
  namespace: foo

6. Update your Kubernetes version

It is critical to upgrade to Kubernetes 1.24 (the latest version at the time of this writing) in order to start using the Pod security admission controller which eliminates limitations that are presented by the PodSecurityPolicy admission controller.

PodSecurityPolicy admission controller has a model and an inconsistent unbounded API. The Kubernetes 1.24 release enhances security and simplifies the Kubernetes code base which in turn improves performance.

Conclusion

The ever-changing needs dynamic solutions as applications are highly distributed making Kubernetes management hard. Therefore, automated cost management systems and Kubernetes developer platforms should be used to eliminate manual tasks that make Kubernetes harder to be analyzed when calculating cluster costs and inspecting logs.

Securing Kubernetes components is also imperative. Kubernetes security starts by making sure that all components are configured well and RBAC is implemented. Misconfigurations such as allowing containers to run with highly privileged escalation rights are detrimental when cyber attackers seize containers. Consistently eliminating misconfigurations and learning the Kubernetes security best practices are some of the best ways to keep up with the ever-changing world of Kubernetes and containers.