Kube-green: An operator to reduce CO2 footprint of your kubernetes clusters

Overview

As we all know that kubernetes is a popular container orchestration platform. Many Organisations have already adopted kubernetes platform to run their applications & products.

Typically in any organisation, we usually see people creating kubernetes clusters to do either development work or for testing or for running production workload etc. This is fine to run workloads but keep that in mind cost is directly proportional to the number of clusters that you create.

So question is -> is there a way to reduce the amount that we spend on clusters?

Ans is -> There are multiple best practices that need to be followed to achieve that . One of the best practice is to shutdown the workloads when you dont need them. We can achieve this using kube-green.

What is kube-green

kube-green is a simple k8s addon that automatically shuts down (some of) your resources when you don’t need them. If we shuts down the resources(for ex: pods) underlying computing resources will become free (or available) and then we can leverage the cluster autoscaler to reduce the number of nodes. So this is how we can save lot of money.

Use Case

• Do we need to run pods during weekends?
• Do we need to run pods at night?

If you want to implement any of the above use case then go for kube-green.

How It works?

kube-green stops the following resources.

• deployments
• cronjobs

Note: By default only deployment pods will be stopped. If you want cronjob pods need to stopped you need to explicitly enable it. We will discuss about this scenario later.

Creating Kubernetes cluster

We need kubernetes cluster to test kube-green functionality. You can use any of the kubernetes cluster for example AWS EKS or Google GKE or Azure AKS or minikube or kind etc etc..

I am going to use kind kubernetes cluster.

kind create cluster --name kube-green

Note: if you dont know what is kind & how to create kubernetes cluster using kind then refer to my other blog.

Now lets switch to new context using below command.

kubectl cluster-info --context kind-kube-green

You can verify cluster using any of the below commands.

kubectl get nodes

kubectl get componentstatus

Installing certmanager

We need to install cert manager because kube-green uses certmanager to issue certificates. Execute the below command to install the cert-manager.

kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml

wait until all the pods goes into running state.

kubectl get pods -n cert-manager

Installing kube-green

kube-green can be installed in multiple ways i.e either kubectl apply or kustomize or via operator.

In this demo I am going to use the simplest method i.e kubectl apply. Execute the below command.

kubectl apply -f https://github.com/kube-green/kube-green/releases/latest/download/kube-green.yaml

The above command creates bunch of resources which you can see from the above screenshot. New namespace “kube-green” gets created and controller pod will be deployed in it.

kubectl get pods -n kube-green

The above command also creates a custom resource definition named “sleepinfos.kube-green.com” which can be seen by executing the below command.

kubectl get crds | grep sleep

This CRD will be used to define when the resources will be shutdown and wakeup.

Testing

Before we tell kube-green to shutdown some of the resources. Lets deploy some sample applications.

kubectl create deployment frontend --image purushothamkdr453/frontend --replicas 3

kubectl create deployment backend  --image purushothamkdr453/frontend --replicas 3

kubectl get pods

There are currently 6 pods running which are from 2 different deployments.

Scenario-1: Shutting down all the resources

Lets create sleepinfo crd with the below content. create a file named “sleep-info.yaml” with below content.

apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: sleep-test
spec:
  weekdays: "*"
  sleepAt: "15:00"
  wakeUpAt: "15:05"
  timeZone: "Asia/Kolkata"

kubectl apply -f sleep-info.yaml

Following are the attributes specified in CRD.

weekdays — * indicates all days, for specific days you can mention the range 1–5(1 is Monday, 2 is Tuesday etc).

SleepAt -at what time resources should be shutdown

wakeUpAt — at what time resource should be wakeup

timeZone — timezone in which you want to run the sleepinfo logic. you can get the other timezone details here.

Adjust these details according to you.

at 15.00PM IST, pods are shutdown. Screenshot below for reference.

at 15.05PM IST, pods are wakeup. Screenshot below for reference.

lets delete the sleepinfo crd.

kubectl delete -f sleep-info.yaml

Scenario-2: Exclude specific resources from shutting down

Lets try to exclude specific resource(in this case backend deployment resource) from shutting down.

Create a file named “sleepinfo-exclude.yaml” with below content.

apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: working-hours
spec:
  weekdays: "*"
  sleepAt: "16:17"
  wakeUpAt: "16:22"
  timeZone: "Asia/Kolkata"
  excludeRef:
    - apiVersion: "apps/v1"
      kind: Deployment
      name: backend

kubectl apply -f sleepinfo-exclude.yaml

kubectl get sleepinfo -A

at 16.17 IST, excluding backend all other resources(pods) are shutdown.

at 16.22 resources which are shutdown are back.

kubectl delete -f sleepinfo-exclude.yaml

Scenario-3: shutting down cron resources

In the previous scenarios we have seen how to shutdown deployment resources. Now lets look at how to suspend cronjobs.

Lets deploy a cronjob by executing below commands.

wget https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/application/job/cronjob.yaml

kubectl apply -f cronjob.yaml
kubectl get cronjobs

Cron job creates pods under the good.

kubectl get pods | grep hello

create a file named “sleep-cron.yaml” with below content.

apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: sleep-cronjobs
spec:
  weekdays: "*"
  sleepAt: "17:05"
  wakeUpAt: "17:10"
  timeZone: "Asia/Kolkata"
  suspendCronJobs: true
  suspendDeployments: false

kubectl apply -f sleep-cron.yaml

kubectl get sleepinfo sleep-cronjobs

we have included 2 new attributes in the above crd. let us take a look at them.

suspendCronJobs -> By default cron jobs will not be suspended i.e default value is false. we need to enable it explicitly by setting .

suspendDeployments -> By default deployments will be suspended i.e default value is true. Just incase if you want to disable it then set it to false.

at 17.05PM IST, cron job will be suspended. Screenshot below for reference.

kubectl get cronjobs

at 17.10PM IST cronjob will be restored. Screenshot below.

Now lets delete the sleepInfo resource.

kubectl delete sleepinfo sleep-cronjobs

Where is the State Stored

how does the kube-green controller knows what to restore & when to restore. Complete state information is stored in a secret where the “SleepInfo” crd has been created. For example

kubectl get secrets

References:

kube-green

Summary:

In this blog post we have seen what is kube-green and how we can use kube-green to suspend deployments or cronjobs. Feel free to comment if you have any queries.