Notes by a Sysadmin
The first idéa I had with this part IV, was to add Restic to the Rclone Backup server. But, due to higher complexity (It should be easy to backup and recover) and not least that it seems like Restic would need to be implemented on the host with Rclone (multiple instances). I skipped the idéa.
So, for now.... Read more
Now, here two years after is set up the backup server. I need to do it all again, the other Pi was erased and used as part of a cluster and I kind of lost data since and was confirmed in the reason to have backup of your data.
Part I:
The following is a HA setup of stacked etcd/cp on vanilla k8s, on rpi4 with 3 cp and 3 workers.
Storage is NFS nvme m2 hosted by one of the workers thru USB3.
OS: Ubuntu 22.04 lts server (arm).
Deploy tool: Kubeadm
Use the Image burner "Raspberry Pi Image burner",... Read more
K3s comes with Traefik pre-installed which in many ways are a good thing, mainly for developers and other who just want to spin up a cluster that works out of the box. But, it can also be a pain to find documentation and understand Ranchers "way" of doing additional configuration on these addons. So, in the following I'll... Read more
Currently I had a problem with Longhorn not compatible with k3s version v1.25.4+k3s1. So, after a lot of debugging I came to the conclusion that I had to do a downgrade of the cluster. I guess almost the same procedure would apply for an upgrade.
As you probably have some more fine grained... Read more
As time goes by, you may forget small features and settings you apply to your servers to make them fit you need. There is a tendency not to document every thing you do when you try out different solutions and suddenly everything works and you forget about the documentation. That's where Ansible comes into play.
Small... Read more
By default Traefik is deployed in K3s. So, in production we would like automating valid wildcard certificate creation. In this case there are two main approaches to generate and store certificates; cert-manager and traefik acme. On it's own Traefik acme can be used to create and store the certificate, but to get HA this way is... Read more
The following will assume that a K3s cluster is already up and running. We want to make it a starting point of our cluster management with the Rancher server distribution.
Note: If the installation fail or you somehow decide to make a reinstall of rancher. You will need to make a complete cleanup of... Read more
K3s is Ranchers (SUSE) small kubernetes solution. In the following I'll try to create a small HA kubernetes cluster, made up of cheap VPS.
BTW: K3s can actually be run... Read more
Though this package manager is also available for Linux, the chances are that you've mostly heard of it in relation to Apple's macOS. Why install such a tool on your macOS, does'nt it increase voulnerbillity of a generally secure operating system with all of Apple's validation... Read more
Basically you would usually don't have to concern your self to much about this subject, when using the established cloud providers, as they usually have their own Load Blancer on the 2/3 OSI level. And networking in a K8s cluster is allready difficult for most people to get compfortably with. But, not to get confused with the... Read more
If you have a desent firewall, having some dedicated interfaces with VPN from the firewall out may be a great idea because it can help with.
There is a lot of different hypervisors and virtual machine managers out there. After using a commercial Type-2 hypervisor for many years, I have started to use KVM whenever possible.
KVM is native to all linux systems today, it works directly on the kernel level and is considered as a Type-1... Read more
Backup is often a messy business, we are allways told to backup. But, what is a backup really and what is the purpose of the backup?
There are many aproaches to these questions, and how could you decide what the right aproach is for your circumstances?
In th "old days" there used to be a... Read more
As private user and a Open Source advocate. There is only a couple of SIEM systems to choose from, one of them is an offspring of the probably most known of them all OSSEC, this system is called Wazuh. One of the great things about Wazuh is the abillity to deploy a scalable container setup. In the following i will try to set... Read more
Often times it comes in handy to encrypt data, that are used for automation. Sensitive data like passwords and other credidentials for infrastructure should be handled with a certain amount of precaution.
Ansible vault is an encrypted storage to Ansible, even though it is also possible to... Read more
There may be occations where some tasks, that usually requires root privileges, would be more handy to run without the need to provide a password.
In this example it is an automation for update, upgrade and reboot. Ofcause it is not advisable to run all sudo commands without providing passwords, in particular not on a... Read more
There are two popular tools for working with screens/sessions in the terminal; Screen and Tmux. Screen is probably to consider the most native and simple of the two. But, this post will be about the Tmux application.
It does not come as native in contrary to Screen, so you have to install it:
Make a traefik.yml in you project directory:
entryPoints:
# http redirect to https
web:
address: ":80"
http:
redirections:
entryPoint:
to: websecure
scheme: https
# https
websecure:
address: ":443"
http:
tls:
... Read more
Start by setting up an overlay network:
docker network create --driver=overlay traefik-publicThen we will export some variables befor deploying out docker-compose.yml:
export EMAIL=admin@example.com
export... Read more
Using jwilders configuration. Go to you project directory and create one directory (atleast), "conf.d" and maybe "html". Create a docker-compose.yml in the directory to (remember to change to the right path). Of cause you also need to figure out if you are running nginx on one or several nodes, in regard to storage and... Read more
Create an image on Docker Hub, there are other ways of creating images. But, the following is based on a "Dockerfile".
"cd" into the directory of your Dockerfile, which is the formula for your image, and execute the following line:
docker build -t... Read more
Portainer is a GUI admin tool for Docker (and K8s). And it is almost a "must have", not that you can't do without. But, this tool is really help full in getting and understanding of your whole cluster, it is stable and great for debugging.
The following "Stack" deployment will setup two services that run... Read more
Now, I came to Swarm after have been in the K8s universe for a year or so. And I just fell in love with Docker Swarm from first try. So, I would encurage anyone working with Pi's to choose Docker Swarm over any K8s solution out there, any day under any circumstances!
The first think and most complicated is to install... Read more
By default one could use the Pi's internal SD-card for storage. But, for many reasons - as stabillity and speed - external storage like an USB 3.x stick on a Pi 4 would be preferable.
Find the node you would like to hold your storage device, and SSH in to the node. And locate the... Read more
Install the Pi Imager tool from Raspberry Pi's homepage: https://www.raspberrypi.org/software/
Insert your SD-card in your PC, and select OS (choose an 64-bit version if you are running Rpi 3 < ) and SD-card and continue..
If you have never generated a key on your system, or if you want a separate key. Do the following:
ssh-keygenAnd following the instructions. Just press enter all way thru, if you do this for the first... Read more
Setting a MQTT broker up on Swarm.
I have setup a nfs share that resides on each node and mounting is done on this share. In preparation I have have made the directories in the nfs-share that the bind is going to use.
The "stack" deployment file would look like this (usually you would probably... Read more
Follow instructions on Docker for setting up Swarm and Portainer (easy)...
In Docker Swarm containers run in "services". Create a service:
docker service create --name redis --replicas=5 redis:3.0.6Docker "ls" to list services:
In my experience Swarm is the ideal orchestration tool for ARM based clusters. Wait! Isn't Kubernetes the future for orchestration, and will Docker not drop support for Swarm in 2022 (you might say). And yes, very sadly! Most datacenters and companies working with container orchestration is moving towards the more complex... Read more
Resently I experienced that my pi's with an ubuntu install did'nt resolve in my routers DNS server. The following mDNS install on the pi will solve that problem.
sudo hostnamectl set-hostname [your-pi-name]followed by:
sudo apt... Read more
There is a lot of different types of volume mounting, mainly because of the fact that Kubernetes (K8s) is often used for the purpose of high availabillity. So, the whole concept of storage on one node, is kind of opposed to that therminology. But, to try and make things simple here, we will use... Read more
Installation for MicroK8s can be found here: https://ubuntu.com/tutorials/how-to-kubernetes-cluster-on-raspberry-pi#1-overview
Enable the dashboard:
microk8s enable... Read more
There is a lot of tutorials out there about how to turn your Raspberry Pi's into a cluster. Before you do this yourself here are a couple of advices, depending of your goal with the project.
The reason you might want mail sending (smtp) capabillity on a Pi, is notifications. This is a nice feature, if some thing goes wrong as Pi's are often setup headless for a specific task.
Start by setting up mailutils and postfix:
sudo apt install... Read more
This is a continuation of "rclone setup..", so the configuration and setting up a of rclone and "Remotes" can be found there..
Operating with rclone you have two options, a GUI called
By default Raspbian currently only take advantage of 32 bit in the core and user land event thou it has an 64 bit CPU in the latest PPi 4. Depending of the purpose of your Raspberry Pi, you may want to take advantage of the 64 bit ARM architecture ( you may be developing some applications or maybe want to use it for container... Read more
There are a lot of backup tools available out there and depending on what the goal is, one should consider what meets the needs. Basically there are different kinds of backup setups:
OS: Ubuntu 20.04
Firewall: UFW
Unfortunately there is no easy solution or "out of the box" features, implemented in most UNIX-like system, for blocking connections by country. But, in a lot of real life cases it would make a lot of sense to implement. Say; you have a website written in Swedish and only expected people... Read more
Providing infrastructure, including IaC with high availabillity and single point of failure in consideration. Help developers containerize and deploy their code. Make sure the organisation is compliant with any legislation (as GDPR). Make use of, and implement encryption, where ever nessesary. Diversify, Segregate, Hash, Backup... Read more
Setting up a new server can be a repetitve task. The following is a LAMP stack setup (Linux, Apache(2), MySQL (could be MariaDB), PHP).
First spin up an instance/VPS/PS of your OS. If you are not very familiar with the UNIX-like world, choose a Debian based OS (such as Debian or Ubuntu) with long term support (LTS). By... Read more
Basically there are stuff that is "out of your control" (to some extend), and stuff that is easier to control.