Vanilla k8s on pi4 HA

February 28, 2023 | Cluster

The following is a HA setup of stacked etcd/cp on vanilla k8s, on rpi4 with 3 cp and 3 workers.

Storage is NFS nvme m2 hosted by one of the workers thru USB3.

OS: Ubuntu 22.04 lts server (arm).

Deploy tool: Kubeadm

Prepare the PI

Use the Image burner "Raspberry Pi Image burner", choose you headless/server image from Ubuntu (under Custom image, after download) and choose "Advanced" for setting up user and other configurations. (SSH, hostname, user and password)

I'll post an Ansible script here at some point. For prepping the OS for K8S.

But, basically there are some steps to follow:

Find you new Pi's the have probably been given a random ip address:
```
nmap 192.168.1.1/24 (or whatever network it resides on)
```

We need to give the Pi a fixed IP, if not netplan is already installed by default. But first there probably is some updating to do, as the image might not be the latest:

sudo apt update && sudo apt upgrade -y

After that is done, figure out if netplan is installed "which netplan" if not, "sudo apt install netplan.io". Backup the old netplan config:

sudo cp /etc/netplan/*.yaml /etc/netplan/backup.yaml

Edit the default netplan config:

sudo vi /etc/netplan/50-cloud-init.yaml

and change the following:

network:
  ethernets:
    eth0: # Replace with your network interface name
      dhcp4: true
      addresses: 
        - 192.168.1.100/24 # Replace with your desired IP and netmask
      routes:
        - to: 0.0.0.0/0
          via: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]
  version: 2

See server preping script: https://thehotelhero.com/use-ansible-to-prep-your-server
Prepping for Vanilla k8s with Kubeadm on pi, ansible script will follow later.. But basically instaling Kubeadm, Kubectl, Kubelet, containerd.

Kube-vip as software LB for Control Plane

Important here: It is difficult to setup this afterward, so start by setting this one up.

So, you might remove you existing installation of kubernetes, by:

sudo kubeadm reset -f
# It will delete all content in the /etc/kubernetes/manifests. So, you might backup any special files here.

Before deploying our new CP loadbalanced cluster, we need to generate a file for the above mentioned directory:

export VIP=192.168.0.40 # (choosen LB address on the same network segment as the cluster)
export INTERFACE=eth0 # (Net interface)
# Get the latest release of kube-vip
KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
alias kube-vip="ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip"
# Make the file for the manifest folder
kube-vip manifest pod --interface $INTERFACE --vip $VIP --controlplane --arp --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

kubeadm init --control-plane-endpoint 192.168.0.40:8443 # (or via DNS is best practis)
# BUT typically you would make sure --pod-network-cidr=10.101.0.0/16 --service-cidr=10.80.0.0/16 is not the same as the local segment
kubeadm init --control-plane-endpoint 192.168.0.40:8443 --pod-network-cidr=10.101.0.0/16 --service-cidr=10.80.0.0/16 --v=5 | sudo tee /var/log/kubeinit.log

and basically what you might want to do for the long run is to have you kubeadm config as a file and not as --flags. So, if you have a running cluster export the configuration, for later use:

kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml

Then if you had that file (before hand):

# Something in the line of
sudo kubeadm init --control-plane-endpoint=192.168.0.40:6443 --upload-certs --config ~/kubeadm.yaml

and afterwards, if you had included the lb address in the "kubeadm.yaml" file:

kind: ClusterConfiguration
kubernetesVersion: v1.26.1
controlPlaneEndpoint: loadbalanceraddress.local:6443 <----- here
networking:
  dnsDomain: cluster.local
  podSubnet: 10.101.0.0/16
  serviceSubnet: 10.96.0.0/12

you would just initialize the cluster with:

sudo kubeadm init --config ~/kubeadm.yaml

Kubectl setup

To be able to make kubectl work with your normal user.

mkdir $HOME/.kube
sudo cp /etc/kubernetes/admin.conf .kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

CNI

Deploy a CNI:

# There are several options here and you might spend some time, figuring out the right solution for your needs.
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml

Adding additional master nodes (Control Planes)

We need a new temporary access token and certificate uploads (I think it expires within 10 minuttes or so).

sudo kubeadm init phase upload-certs --upload-certs # You don't overwrite any configuration with the phase flag
# Notice the certificate key at the bottom of the output
kubeadm token create --print-join-command

The last this is to join your other Control planes:

sudo kubeadm join <DNS CNAME of load balancer>:6443 \
--token <bootstrap-token> \
--discovery-token-ca-cert-hash sha256:<CA certificate hash> \
--control-plane --certificate-key <certificate-key>

# And remember 
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

and your minions you just use your join command from the "token create" output.

Load balancer for Services

Now that we have load balancing for the Control Plan we also would like to have decoupled load balancer for the services. We can use Kube-VIP for that too.

Deploy the "Kube-VIP cloud provider":

$ kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml

the ressources can be seen with:

kubectl describe deployment/kube-vip-cloud-provider -n kube-system
POD_NAME=$(kubectl get po -n kube-system | grep kube-vip-cloud-provider | cut -d' ' -f1)
kubectl describe pod/$POD_NAME -n kube-system

Now, there are a lot of option of limiting and segmenting for which namespaces can get what IP's etc. But the most basic install is global access:

kubectl create configmap --namespace kube-system kubevip --from-literal cidr-global=192.168.0.220/29 # Ofcause you choose a range that fits your network segment

Now, you are able to retrieve an ip, when you specify type of loadBalancer.

Important:

By following the above instructions the service load balancer is not enabled in /etc/kubernetes/manifests/kube-vip.yaml. You would need to enable it in your daemonset/pod by adding this env. var. to the manifest:

        - name: svc_enable
          value: "true"

NFS

Check the post about NFS: https://thehotelhero.com/nfs-share-shared-storage

Troubleshooting

If you get an error like:

[ERROR SystemVerification]: missing required cgroups: memory

You need to add cgroup to memory. Depending on your OS, the file may be at one of the following locations:

/boot/firmware/cmdline.txt
/boot/firmware/nobtcmd.txt 
/etc/default/grub

adding the following line to the file and rebooting the kernel should do the job:

cgroup_enable=memory cgroup_memory=1

sudo update-grub
sudo reboot

Issues with containerd:

I had some issues following the official docs, the following error occured on several nodes:

[ERROR CRI]: container runtime is not running: output: time="2023-03-02T22:59:57Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1

First of:

sudo containerd config default | sudo tee /etc/containerd/config.toml

and then:

sudo containerd config default | sed 's/SystemdCgroup = false/SystemdCgroup = true/' | sudo tee /etc/containerd/config.toml
and
sudo systemctl restart containerd.service

you may also need to restart the kubelet. Make sure the above is prevailent on all systems, before continuing.

The Hotel Hero