K3s comes with Traefik pre-installed which in many ways are a good thing, mainly for developers and other who just want to spin up a cluster that works out of the box. But, it can also be a pain to find documentation and understand Ranchers "way" of doing additional configuration on these addons. So, in the following I'll remove the pre-installed Traefik ingress controller and install it again with a helm chart or manifest files (what ever seems more suitable).

Removing the Traefik Ingress Controller from K3S

K3s deploy helm charts via their own repository. So, to start with we will use helm delete to se if we can delete the deployment. 

kubectl -n kube-system delete helmcharts.helm.cattle.io traefik

This may not work and the attempt will timeout...

Then the second attempt will be to remove the /var/lib/rancher/k3s/server/manifests/traefik.yaml or atleast move them out of the folder to another place. In the "manifests" dir:

mv traefik.yaml ../

Then lets try again with the --force flag:

kubectl -n kube-system delete helmcharts.helm.cattle.io traefik --force

It may take some time to delete this "gracefully" and if this does'nt succede one might try with:

kubectl -n kube-system delete helmcharts.helm.cattle.io traefik --grace-period=0 --force 

Failed too!

Now we have tried the conventional way to get rid of Traefik

Now, we will stop the systemd service and reconfigure those in each CP node:

sudo service k3s stop

and edit the service:

sudo vi /etc/systemd/system/k3s.service

and add or change the following line in the bottom of the file, to this:

ExecStart=/usr/local/bin/k3s \
    server --disable servicelb,traefik \ 
(do not add servicelb, that is just another service I have added)

and reload the systemd daemon:

sudo systemctl daemon-reload

Before you do the last step, you might want to make sure that the traefik.yaml has not been recreated in the "manifests" directory.

and restart k3s service:

sudo service k3s start

Do this step on all the Control Planes.

The Traefik deployment still exists on our cluster, in the "kube-system" namespace and vi want to get rid of it.

Get an overview of all Traefik related ressources.

After deleting each ressource I found the issue. The service traefik was stuck in a Finalizer issue with the old loadbalancer. After removing the line:

finalizers:
  - service.kubernetes.io/load-balancer-cleanup

in the service, it was automatically removed.

So, the above unforeseen issue is a typical scenario in k8s. And the approach would probably be very individual. But, basically it would be a good idéa to start with the most gracefull and dig more into troubleshooting the actual resource that is blocking the process.

Install Traefik via helm

Following the official docs: https://doc.traefik.io/traefik/getting-started/install-traefik/#use-the-helm-chart

Add the repo:

helm repo add traefik https://traefik.github.io/charts

And for now we don't need any special configuration, just the default. But, get the values out anyway:

helm show values traefik/traefik > values.yaml

And deploy the chart in your choosen namespace:

helm install traefik traefik/traefik -n traefik

and that should be it, for now. Make sure everything looks fine.