How to Monitor health and Resources Usages of Pods and Containers in Kubernetes

There are several ways to monitor the health and resource usage of pods and containers in Kubernetes, including:

Using Kubernetes built-in monitoring tools:

The Kubernetes kubectl command-line tool can be used to view the status and resource usage of pods and containers.Here are a few common commands for checking the health of pods in Kubernetes:

kubectl get pods: This command will show you the current state of all pods in the cluster, including their names, current state, and the number of restarts.

kubectl describe pod <pod-name>: This command will provide detailed information about a specific pod, including its current state, resource usage, and any events or errors that have occurred.

kubectl logs <pod-name>: This command will show the logs for a specific pod, which can be useful for troubleshooting issues or understanding the current state of the pod.

kubectl exec -it <pod-name> <command>: This command allows you to run a command inside a specific pod, which can be useful for debugging or troubleshooting.

kubectl top pod <pod-name>: This command will show you the resource usage of a specific pod, including the CPU and memory usage.

kubectl get pod --watch: This command will show you the current state of all pods in the cluster and will continue to update the output every 2 seconds, allowing you to monitor the state of pods in real-time.

kubectl get pod -o json <pod-name> : This command will get the json representation of the pod, which is useful when we need to see the detailed and specific information of the pod.

kubectl rollout status deployment <deployment-name> : This command gives the status of the rollout of the deployment, also allows to know if any rollback happened

The Kubernetes Dashboard, a web-based UI, can also be used to view the status and resource usage of pods and containers.

Using Prometheus and Grafana:

Prometheus is a monitoring tool that can collect and store metrics from pods and containers.

Grafana is a visualization tool that can be used to display the metrics collected by Prometheus.

Using ELK stack:

Elasticsearch can be used to store logs generated by pods and containers.

Logstash can be used to process and filter the logs before they are stored in Elasticsearch.

Kibana can be used to display the logs in a user-friendly way.

Using third-party monitoring solutions:

Datadog, New Relic, Sysdig are some examples of commercial solutions that can be used to monitor the health and resource usage of pods and containers.

Using Kubernetes API : You can use Kubernetes API to get the information of your pods, nodes and any other Kubernetes objects.

Kubernetes API server is exposed via REST API, this will allow you to use any programming language to get the health and resource usage of your pods and containers.

It's important to keep in mind that, the monitoring tool or solution you choose will depend on your specific use case, and the requirements and constraints of your organization. Also, it's important to monitor the health and resource usage of the pods and containers regularly and take action if necessary, such as scaling the application, replacing failed pods, or performing a rolling update.

example of a simple shell script that monitors the status of all pods and containers in a Kubernetes cluster:

Save as k8s_monitor.sh

Run by command : sh k8s_monitor.sh (inside your worker node / Run First in Development Server for code validation )

#!/bin/bash

while true; do

echo "----- Pods -----"

kubectl get pods --all-namespaces

echo "----- Containers -----"

kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].name}{"\t"}{.status.containerStatuses[*].ready}{"\n"}{end}'

echo "----- End -----"

sleep 5

done

-----------------------------END-----------------------------

This script will run an infinite loop that, at each iteration, prints out the current state of all pods in the cluster (using kubectl get pods --all-namespaces), and the status of all containers in all pods (using kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].name}{"\t"}{.status.containerStatuses[*].ready}{"\n"}{end}').

It will print the information every 5 seconds, but you can adjust the interval based on your needs.

You may want to consider sending the output to a log file, and sending notifications when certain events occur (e.g., a pod or container goes into a failed state).

This is an example and the base of a monitoring script, you may want to add specific checks on the pod or container state and based on that you can perform actions on your custom script.