What is kOps?
kOps, also known as Kubernetes operations, is an open-source project which helps you create, destroy, upgrade, and maintain a highly available, production-grade Kubernetes cluster. Depending on the requirement, kOps can also provision cloud infrastructure.
The easiest way to get a production grade Kubernetes cluster up and running. Kops It is used to create and manage a production-grade Kubernetes cluster on the cloud. It's integrated with AWS and Google Cloud (GCE), and some other platforms
Warning: Please refer official documentation for more advanced info, before proceed into production setup.
Let's dive into kOps Setup...
Server Configuration
Run 3 VMs : 1 -Master, 2 -Worker
Operating system version: Ubuntu Server 22.04
| Node | IP | hostname |
| Master | 192.168.197.181 | k8s-master.demo.local |
| Worker1 | 192.168.197.182 | k8s-worker1.demo.local |
| Worker2 | 192.168.197.183 | k8s-worker2.demo.local |
Note: here we are using ubuntu 22 server edition 3 VMs, you can also choose others like centos, make sure you change the commands accordingly
Make sever setup for kOps
After installing the operating system on the device, we conduct operations on software Update devices to the latest version (performed on all node)
sudo apt update
sudo apt upgrade -y
After the update is complete, we restart the server
sudo reboot
Set hostname and update host , restart the server
Master node
sudo hostnamectl set-hostname "k8s-master.demo.local"
init 6 # Restart the server
Worker 1
sudo hostnamectl set-hostname "k8s-worker1.demp.local"
init 6 # Restart the server
Worker 2
sudo hostnamectl set-hostname "k8s-worker2.demo.local"
init 6 # Restart the server
Next we update the file /etc/hosts file of all nodes.
Then add below the content file below:
#Vim /etc/hosts
192.168.197.181 k8s-master.demo.local k8s-master
192.168.197.182 k8s-worker1.demo.local k8s-worker1
192.168.197.183 k8s-worker2.demo.local k8s-worker2
Disable swap and kernel update
Note:
The commands below are executed on all node
sudo swapoff -a
Check that the swap is disabled or not with the command free -h, if successful results will be as follows:
$ free -h
total used free shared buff/cache available
Mem: 7.7Gi 167Mi 7.1Gi 1.0Mi 437Mi 7.3Gi
Swap: 0B 0B 0B
Next disable swap in /etc/fstab
sudo vim /etc/fstab
Find the line: /swap.img none swap sw 0 0 and update to comment #:
#/swap.img none swap sw 0 0
Then run the commands:
sudo mount -a
free -h
Load the following kernel modules on all the nodes:
sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
Set the following Kernel parameters for Kubernetes.
sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
Then reload again sysctl
sudo sysctl --system
Install container-d run time
Note:
Run the commands below on all node
#depandacy packages
sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates
#Add Repo
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
#Update system
sudo apt update
#Install containerd
sudo apt install -y containerd.io
After the installation is complete, we add container d configuration.
#Configure container
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
#Modify config
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
#Start container
sudo systemctl restart containerd
sudo systemctl enable containerd
Install Kubernetes
Note:
Run the commands below on all node
#Add Repo
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
#Update system packgaes
sudo apt update
#Install Kubelet kubeadm kubectl
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
Initialize the cluster with kubeadm
Note:
Only run the channel below on master node
Inside 10.20.0.0/16 is the CIDR of pod network, you can replace on demand.
If the run is successful, the return result will be as follows
sudo kubeadm init \
--pod-network-cidr=10.20.0.0/16 \
--control-plane-endpoint=k8s-master.demo.local
Next, execute the commands below on the master node
#Output
[init] Using Kubernetes version: v1.26.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master.demo.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.10]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master.demo.local localhost] and IPs [192.168.1.10 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master.demo.local localhost] and IPs [192.168.1.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 7.503422 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master.demo.local as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master.demo.local as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: dfii4z.h3qu0s6ircq3pa0
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap,RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regularuser:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listedat:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following asroot:
kubeadm join k8s-master.demo.local:6443 --token dfii4z.h3qu0s6ircq3pa0 \
--discovery-token-ca-cert-hash sha256:58b9cc96ed57a5797fddea653756dbda830efbff55b720a10cffb3948d489148 \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8s-master.demo.local:6443 --token dfii4z.h3qu0s6ircq3pa0 \
--discovery-token-ca-cert-hash sha256:58b9cc96ed57a5797fddea653756dbda830efbff55b720a10cffb3948d489148
Run the following commands for configure the cluster
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
We try running the command to look at the status of the cluster
Return result:
#Get cluster Info
karthick@k8s-master:~$ kubectl cluster-info
#output
Kubernetes control plane is running at https://k8s-master.demo.local:6443
CoreDNS is running at https://k8s-master.demo.local:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
#Get Node info
karthick@k8s-master:~$ kubectl get nodes
#Output
NAME STATUS ROLES AGE VERSION
k8s-master.demo.local NotReady control-plane 40m v1.26.1
We see that the control plane is running and currently there is only one master node, we will proceed to add worker node to this cluster.
Add worker node to the cluster
Note: The commands below only run on node worker
We review the output of the sudo kubeadm init command above and copy the command below the section
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8s-master.demo.local:6443 --token dfii4z.h3qu0s6ircq3pa0 \
--discovery-token-ca-cert-hash sha256:58b9cc96ed57a5797fddea653756dbda830efbff55b720a10cffb3948d489148
After the return result on terminal will take the following
karthick@k8s-worker1:~$
karthick@k8s-worker1:~$
karthick@k8s-worker1:~$ sudo kubeadm join k8s-master.demo.local:6443 --token dfii4z.h3qu0s6ircq3pa0 \
--discovery-token-ca-cert-hash sha256:58b9cc96ed57a5797fddea653756dbda830efbff55b720a10cffb3948d489148
#Output
[sudo] password for anh:
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
Check the node status kubectl get nodes
karthick@k8s-master:~$
karthick@k8s-master:~$ kubectl get nodes
#Output
NAME STATUS ROLES AGE VERSION
k8s-master.demo.local NotReady control-plane 57m v1.26.1
k8s-worker1.demo.local NotReady <none> 90s v1.26.1
k8s-worker2.demo.local NotReady <none> 45s v1.26.1
Show node has been successfully added to the cluster.
Install Calico Pod Network for Kubernetes cluster
Note: The commands only run on master node
First we download file manifest as file yaml, this is the Calico installation file on Kubernetes cluster.
curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml -O
Open the newly downloaded file and find the section CALICO_IPV4POOL_CIDR
---
#Un comment below lines
- name: CALICO_DISABLE_FILE_LOGGING
value: 'true'
We revise the IP range to match the CIDR of the network pod in the command
sudo kubeadm init. In his example is 10.20.0.0/16 so the file after modify like below
---
# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
value: '10.20.0.0/16'
# Disable file logging so `kubectl logs` works.
- name: CALICO_DISABLE_FILE_LOGGING
value: 'true'
Then Now we Apply Calico on the Kubernetes cluster:
karthick@k8s-master:~
karthick@k8s-master:~ kubectl apply -f calico.yaml
#output
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created
We will check to see if calico has been deploy successful by checking the pods on namespace kube-system
karthick@k8s-master:~$
karthick@k8s-master:~$ kubectl get pods -n kube-system
#output
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-34b86c56s-hfdac 1/1 Running 0 4m68s
calico-node-6rfjl 1/1 Running 0 4m68s
calico-node-ddsjz 1/1 Running 0 4m68s
calico-node-ar45d 1/1 Running 0 4m68s
coredns-890c2560in-c2w3a 1/1 Running 0 70m
coredns-890c2560in-pq3cs
If the status as Running means has been successful, now if checking the status of the node then the status will be Ready.
karthick@k8s-master:~$
karthick@k8s-master:~$ kubectl get nodes
#ouput
NAME STATUS ROLES AGE VERSION
k8s-master.demo.local Ready control-plane 72m v1.26.1
k8s-worker1.demo.local Ready <none> 5m78s v1.26.1
k8s-worker2.demo.local Ready <none> 5m0s v1.26.1
Follow for more: 👋 Karthick D
LinkedIn: https://www.linkedin.com/in/karthick-dkk/
Medium: https://karthidkk123.medium.com/
Github: https://github.com/karthick-dkk/
Hashnode: https://karthick-dk.hashnode.dev/