The Hotel Hero

Notes by a Sysadmin

Cluster | Philosophy | Stack

Vanilla k8s on pi4 HA

February 28, 2023 | Cluster

The following is a HA setup of stacked etcd/cp on vanilla k8s, on rpi4 with 3 cp and 3 workers.

Storage is NFS nvme m2 hosted by one of the workers thru USB3.

OS: Ubuntu 22.04 lts server (arm).

Deploy tool: Kubeadm

Prepare the PI

Use the Image burner "Raspberry Pi Image burner", choose you headless/server image from Ubuntu (under Custom image, after download) and choose "Advanced" for setting up user and other configurations. (SSH, hostname, user and password)

I'll post an Ansible script here at some point. For prepping the OS for K8S.

But, basically there are some steps to follow:

Kube-vip as software LB for Control Plane

Important here: It is difficult to setup this afterward, so start by setting this one up.

So, you might remove you existing installation of kubernetes, by:

sudo kubeadm reset -f
# It will delete all content in the /etc/kubernetes/manifests. So, you might backup any special files here.

Before deploying our new CP loadbalanced cluster, we need to generate a file for the above mentioned directory:

export VIP= # (choosen LB address on the same network segment as the cluster)
export INTERFACE=eth0 # (Net interface)
# Get the latest release of kube-vip
KVVERSION=$(curl -sL | jq -r ".[0].name")
alias kube-vip="ctr run --rm --net-host$KVVERSION vip /kube-vip"
# Make the file for the manifest folder
kube-vip manifest pod --interface $INTERFACE --vip $VIP --controlplane --arp --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

kubeadm init --control-plane-endpoint # (or via DNS is best practis)
# BUT typically you would make sure --pod-network-cidr= --service-cidr= is not the same as the local segment
kubeadm init --control-plane-endpoint --pod-network-cidr= --service-cidr= --v=5 | sudo tee /var/log/kubeinit.log

and basically what you might want to do for the long run is to have you kubeadm config as a file and not as --flags. So, if you have a running cluster export the configuration, for later use:

kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml

Then if you had that file (before hand):

# Something in the line of
sudo kubeadm init --control-plane-endpoint= --upload-certs --config ~/kubeadm.yaml

and afterwards, if you had included the lb address in the "kubeadm.yaml" file:

kind: ClusterConfiguration
kubernetesVersion: v1.26.1
controlPlaneEndpoint: loadbalanceraddress.local:6443 <----- here
  dnsDomain: cluster.local

you would just initialize the cluster with:

sudo kubeadm init --config ~/kubeadm.yaml

Kubectl setup

To be able to make kubectl work with your normal user.

mkdir $HOME/.kube
sudo cp /etc/kubernetes/admin.conf .kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config


Deploy a CNI:

# There are several options here and you might spend some time, figuring out the right solution for your needs.
kubectl apply -f

Adding additional master nodes (Control Planes)

We need a new temporary access token and certificate uploads (I think it expires within 10 minuttes or so).

sudo kubeadm init phase upload-certs --upload-certs # You don't overwrite any configuration with the phase flag
# Notice the certificate key at the bottom of the output
kubeadm token create --print-join-command

The last this is to join your other Control planes:

sudo kubeadm join <DNS CNAME of load balancer>:6443 \
--token <bootstrap-token> \
--discovery-token-ca-cert-hash sha256:<CA certificate hash> \
--control-plane --certificate-key <certificate-key>

# And remember 
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

and your minions you just use your join command from the "token create" output.

Load balancer for Services

Now that we have load balancing for the Control Plan we also would like to have decoupled load balancer for the services. We can use Kube-VIP for that too.

Deploy the "Kube-VIP cloud provider":

$ kubectl apply -f

the ressources can be seen with:

kubectl describe deployment/kube-vip-cloud-provider -n kube-system
POD_NAME=$(kubectl get po -n kube-system | grep kube-vip-cloud-provider | cut -d' ' -f1)
kubectl describe pod/$POD_NAME -n kube-system

Now, there are a lot of option of limiting and segmenting for which namespaces can get what IP's etc. But the most basic install is global access:

kubectl create configmap --namespace kube-system kubevip --from-literal cidr-global= # Ofcause you choose a range that fits your network segment 

Now, you are able to retrieve an ip, when you specify type of loadBalancer.


By following the above instructions the service load balancer is not enabled in /etc/kubernetes/manifests/kube-vip.yaml. You would need to enable it in your daemonset/pod by adding this env. var. to the manifest:

        - name: svc_enable
          value: "true"


Check the post about NFS:


If you get an error like:

[ERROR SystemVerification]: missing required cgroups: memory

You need to add cgroup to memory. Depending on your OS, the file may be at one of the following locations:


adding the following line to the file and rebooting the kernel should do the job:

cgroup_enable=memory cgroup_memory=1
sudo update-grub
sudo reboot

Issues with containerd:

I had some issues following the official docs, the following error occured on several nodes:

[ERROR CRI]: container runtime is not running: output: time="2023-03-02T22:59:57Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1

First of:

sudo containerd config default | sudo tee /etc/containerd/config.toml

and then:

sudo containerd config default | sed 's/SystemdCgroup = false/SystemdCgroup = true/' | sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd.service

you may also need to restart the kubelet. Make sure the above is prevailent on all systems, before continuing.


I'm a Sysadmin, network manager and cyber security entusiast. The main purpose of this public "notebook" is for referencing repetitive tasks, but it might as well come in handy to others. Windows can not be supported! But all other OS compliant with the POSIX-standard can (with minor adjustments) apply the configs on the site. It is Mac OSX, RHEL and all the Fedora based distros and Debian based (several 100's of OS's), all the BSD distros, Solaris, AIX and HP-UX.