This past week I studied…

  • Docker
  • Kubernetes

Docker

Though I’ve installed plenty of Docker containers in my Homelab, I don’t fully understand what is going on ‘under the hood.’ How are containers built? What’s the difference between an image and a container? What is the method to the madness of port mapping and other configuration options? I no longer wish to simply run Docker containers, I wish to fully understand them.

As such, I am taking a two-pronged approach to learning about Docker. The first is that I’m reading the book, ‘Docker Deep Dive’ by Nigel Poulton. The second is that I am making my way through the Docker Training Course for the Absolute Beginner on KodeKloud.

I’ll often tackle learning a new subject via multiple avenues. I find it keeps me from getting bored as well as it gives me a different perspective as each instructor/author has a unique way to approaching a topic. These multiple inputs on the same topic work well to keep me energized and learning.

I’ve learned some new commands and reinforced others. I’ve also dug in deep on some commands such as docker exec nameofimage cat /etc/hosts to reveal information about the underlying OS, and docker run -it kodekloud/simple-prompt-docker to enter interactive mode and attach to the terminal in the container.

These latter two commands demonstrate my commitment to understanding Docker far beyond simply running containers on my homelab. Knowing these types of commands and understanding their value are critical to interacting with and troubleshooting Docker containers.

Kubernetes

As with learning Docker, my approach to learning Kubernetes has been multi-dimensional. I’m about 95% complete with the KodeKloud Certified Kubernetes Administrator (CKA) with Practice Tests course on Udemy. As well, I’m making my way through The Kubernetes Book by Nigel Poulton. And I’ve got a four node K3s cluster I built from the ground up in my homelab to learn and apply everything I can about Kubernetes in a real world environment.

This past week, as I’ve traversed the (Troubleshooting Section) labs on KodeKloud, I’ve struggled, and learned from my struggles, some valuable skills. We are tasked with prompts like this…

The cluster is broken. We tried deploying an application but it’s not working. Troubleshoot and fix the issue.

That’s all they give you. However, they do give you a hint… ‘Start looking at the deployments.’

Here’s a peak at the troubleshooting approach I learned and implemented…

First, let’s take a look at the nodes

kubectl get nodes

Nodes are in a ready state. That’s a good thing. Next, let’s take a look at the Deployments.

kubectl get deploy

OK, I see the app that is not deploying successfully. The Ready state is 0/1 so this requires further investigation. This is where the describe command comes in handy to show us the deployment manifest.

kubectl describe deploy app

Nothing jumps out as problematic here so next let’s take a look at the ReplicaSet.

kubectl get rs # Note I am using rs as shorthand instead of replicaSet

We have one replicaSet so let’s take a look at it

kubectl describe rs app-4872bddbc87

The Pods Status is 1 Waiting. So, let’s take a look at the pod

kubectl get pod

The pod is in a pending state so let’s take a closer look

kubectl describe pod app-4872bddbc87

We can see that the Events, listed at the bottom, have not started. The pod is in a pending state and it is not assigned a node. This leads us to the Scheduler since it is the Scheduler job to assign a Pod to a Node.

The Scheduler is in the kube-system namespace so let’s check it out

kubectl get pods -n kube-system

I can see that the kube-scheduler-controlplane status is CrashLoopBackOff so further investigation of this pod is warranted.

kubectl describe pod -n kube-system kube-scheduler-controlplane

In the Events section, we can see errors. One item in particular stands out

exec: "kube-schedulerrr": executable file not found in $PATH: unknown

Aha! Someone made a typo in the command. Let’s fix it.

We know that the kube-scheduler is a static pod whose manifest resides in /etc/kubernetes/manifests/ so let’s edit the file (using Vim, of course) and fix the typo.

vi /etc/kubernetes/manifests/kube-scheduler.yaml

Let’s see if that did the trick

kubectl get pods -n kube-system --watch

I add the --watch flag to see the pod status update in real time. After a bit, the pod is in a running state.

Finally, let’s check out the pod on the default namespace to see if the problem has been fixed

kubectl get pods

Indeed, the pod is running. For good measure, let’s check out the Deployment as well

kubectl get deploy

The Deployment shows 1/1 as ready.

Problem solved. This is the kind of sleuthing that I find fascinating. There is a method to the troubleshooting madness and I find it enjoyable and rewarding.

In the DevOps Skool community that I belong to, one of the members shared with me A visual guide on troubleshooting Kubernetes deployments, a valuable resource created by learnk8s which is quite handy.

In addition to focusing on troubleshooting, I’ve been writing atomic notes in my note taking system, breaking down core Kubernetes topic and explaining them in my own words. As my mentor says…

If you can’t write about it, then you don’t understand the topic. - ‘Everything Starts with a Note-taking System’ Mischa van den Burg

I take this to heart. I have found that when I break a topic down into it’s most basic component and write about it in my own words, my comprehension increases ten fold.

I’m not interested in simply passing the CKA and other exams. My intention is to become a valuable, incredibly knowledgable and enthusiastic Kubernetes Administrator.