How to setup Kubernetes Cluster HA on Bare-metal with haproxy and kube-api load balancing

Our Kubernetes Cluster, Haproxy and Worker Nodes as below in /etc/hosts of all Centos Stream 8 Machine (all nodes including worker,master and haproxy)

/etc/hosts
192.168.225.31 kube-loadbalancer
192.168.225.142 kube-master1
192.168.225.232 kube-master2
192.168.225.228 kube-master3
192.168.225.188 kube-worker1
192.168.225.32 kube-worker2

Here Following configuration needed on HAPROXY our Kube API load balancer (kube-loadbalancer)

#cat /etc/haproxy/haproxy.cfg
global
log /dev/log local0 warning
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon

stats socket /var/lib/haproxy/stats

defaults
log global
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000

frontend kube-apiserver
bind 192.168.225.31:6443
mode tcp
option tcplog
default_backend kube-apiserver

backend kube-apiserver
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server kube-apiserver-1 192.168.225.142:6443 check # Replace the IP address with your own.
server kube-apiserver-2 192.168.225.232:6443 check # Replace the IP address with your own.
server kube-apiserver-2 192.168.225.228:6443 check # Replace the IP address with your own.

First Kubernetes Master Node configuration

  • Selinux in Permissive mode setup

#sudo sed -i ‘s/^SELINUX=enforcing$/SELINUX=permissive/’ /etc/selinux/config

#reboot

  • Kubernetes Repo Setup

#cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

#yum install -y kubelet kubeadm kubectl –disableexcludes=kubernetes
#sudo systemctl enable –now kubelet
#reboot

  • Containerd Setup

#dnf config-manager –add-repo=https://download.docker.com/linux/centos/docker-ce.repo
#dnf remove podman buildah
#dnf install containerd docker-ce

  • Containerd Configuration

#vi /etc/containerd/config.toml

#disabled_plugins = [“cri”]

root = “/var/lib/containerd”
state = “/var/run/containerd”
subreaper = true
oom_score = 0

[grpc]
address = “/var/run/containerd/containerd.sock”
uid = 0
gid = 0

[debug]
address = “/var/run/containerd/debug.sock”
uid = 0
gid = 0
level = “info”

  • Containerd Enable and Start

#systemctl enable containerd.service
#systemctl start containerd.service

  • Kernel Parameters and Modules Setup

#cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

#reboot

After reboot check above settings are ready or not via below command

#lsmod | grep br_netfilter
#lsmod | grep overlay

#sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

  • Kubernetes Control Plan Setup on First Kubernetes Master

#kubeadm init –control-plane-endpoint “192.168.225.31:6443” –upload-certs

“””” You will get something like below “”””””

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

kubeadm join 192.168.225.31:6443 –token fqlg6e.i0rsvyoi63hozcle \
–discovery-token-ca-cert-hash sha256:cca396e7cd18cec3ce56961a81419cdef054018222b59a8956da831c1db431f8 \
–control-plane –certificate-key 0d72a3de7d6324fea1a80c9c4f3d65f262d0b16eb848e15e2ce16149f84e5da8

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
“kubeadm init phase upload-certs –upload-certs” to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.225.31:6443 –token fqlg6e.i0rsvyoi63hozcle \
–discovery-token-ca-cert-hash sha256:cca396e7cd18cec3ce56961a81419cdef054018222b59a8956da831c1db431f8

“”” Here Note kubeadmin join command – one is for control node (–control-plane) and second is for worker node “””

  • Installing Weavenet CNI for Kubernetes Pods networking

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

Now Other Kubernetes Master Node Setup to Join First Master Node Follow Below

  • Selinux in Permissive mode setup

#sudo sed -i ‘s/^SELINUX=enforcing$/SELINUX=permissive/’ /etc/selinux/config

#reboot

  • Kubernetes Repo Setup

#cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

#yum install -y kubelet kubeadm kubectl –disableexcludes=kubernetes
#sudo systemctl enable –now kubelet
#reboot

  • Containerd Setup

#dnf config-manager –add-repo=https://download.docker.com/linux/centos/docker-ce.repo
#dnf remove podman buildah
#dnf install containerd docker-ce

  • Containerd Configuration

#vi /etc/containerd/config.toml

#disabled_plugins = [“cri”]

root = “/var/lib/containerd”
state = “/var/run/containerd”
subreaper = true
oom_score = 0

[grpc]
address = “/var/run/containerd/containerd.sock”
uid = 0
gid = 0

[debug]
address = “/var/run/containerd/debug.sock”
uid = 0
gid = 0
level = “info”

  • Containerd Enable and Start

#systemctl enable containerd.service
#systemctl start containerd.service

  • Kernel Parameters and Modules Setup

#cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

#reboot

After reboot check above settings are ready or not via below command

#lsmod | grep br_netfilter
#lsmod | grep overlay

#sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

  • Join this other kubernetes master node with First Kubernetes Control Plan (First Kubernetes Master) (same applies to master node setup except first one)

kubeadm join 192.168.225.31:6443 –token fqlg6e.i0rsvyoi63hozcle \
–discovery-token-ca-cert-hash sha256:cca396e7cd18cec3ce56961a81419cdef054018222b59a8956da831c1db431f8 \
–control-plane –certificate-key 0d72a3de7d6324fea1a80c9c4f3d65f262d0b16eb848e15e2ce16149f84e5da8

Kubernetes Worker Node Setup ( applied to all worker Nodes)

  • Selinux in Permissive mode setup

#sudo sed -i ‘s/^SELINUX=enforcing$/SELINUX=permissive/’ /etc/selinux/config

#reboot

  • Kubernetes Repo Setup

#cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

#yum install -y kubelet kubeadm kubectl –disableexcludes=kubernetes
#sudo systemctl enable –now kubelet
#reboot

  • Containerd Setup

#dnf config-manager –add-repo=https://download.docker.com/linux/centos/docker-ce.repo
#dnf remove podman buildah
#dnf install containerd docker-ce

  • Containerd Configuration

#vi /etc/containerd/config.toml

#disabled_plugins = [“cri”]

root = “/var/lib/containerd”
state = “/var/run/containerd”
subreaper = true
oom_score = 0

[grpc]
address = “/var/run/containerd/containerd.sock”
uid = 0
gid = 0

[debug]
address = “/var/run/containerd/debug.sock”
uid = 0
gid = 0
level = “info”

  • Containerd Enable and Start

#systemctl enable containerd.service
#systemctl start containerd.service

  • Kernel Parameters and Modules Setup

#cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

#reboot

After reboot check above settings are ready or not via below command

#lsmod | grep br_netfilter
#lsmod | grep overlay

#sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

  • Join with master node using below command

kubeadm join 192.168.225.31:6443 –token fqlg6e.i0rsvyoi63hozcle \
–discovery-token-ca-cert-hash sha256:cca396e7cd18cec3ce56961a81419cdef054018222b59a8956da831c1db43

Amit Dhanani

Leave a comment