How to run high availability Mosquitto on Kubernetes
If you use mosquitto as your MQTT for home automation you may be benefit from running it in a “high availability” mode such that all your HA don’t go down as easily. A major reason for poeple skipping zigbee2mqtt and mosquitto in favour of ZHA is that it adds more moving parts.
It actually isn’t very hard to instances of mosquitto which share all message bi-directionally. I use Kubernetes my home and so it makes sense to leverage services to achieve a HA broker. While other solution such as HiveMQTT exists and provide full on Kubernetes operators, I chose to stick with mosquitto due to arm64 builds and simplicity.
Single Deployment
A single deployment of mosquitto follows. This is what I storted off with.
apiVersion: apps/v1
kind: Deployment
metadata:
name: mosquitto
namespace: automation
spec:
selector:
matchLabels:
app: mosquitto
template:
metadata:
labels:
app: mosquitto
type: primary
spec:
containers:
- image: eclipse-mosquitto:2.0.12
name: mosquitto
ports:
- containerPort: 1883
- containerPort: 9001
command:
- mosquitto
args:
- -c
- /mosquitto-no-auth.conf
securityContext:
runAsUser: 1883
runAsGroup: 1883
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: [mosquitto]
topologyKey: kubernetes.io/hostname
The service
You’ll then what to add a service so you can reach your mosquitto pod with a predictable DNS record.
apiVersion: v1
kind: Service
metadata:
name: mosquitto
namespace: automation
spec:
ports:
- name: mqtt
port: 1883
targetPort: 1883
- name: wss
port: 9001
targetPort: 9001
selector:
app: mosquitto
type: LoadBalancer
externalTrafficPolicy: Local
# Use an loadBalancerIP (e.g. MetalLB) or externalIP depending on your setup
# externalIPs:
# - ${EI_MQTT}
Making it High Availability
So how do we make it HA you ask? Simply for a second mosquitto broker which will bridge to the first one using the service we just added.
apiVersion: v1
kind: ConfigMap
metadata:
name: bridge-conf
namespace: automation
data:
mosquitto.conf: |
listener 1883
allow_anonymous true
connection broker0
address mosquitto-p.automation
topic # both 0
Add a service which only selects the primary pod:
apiVersion: v1
kind: Service
metadata:
name: mosquitto-p
namespace: automation
spec:
ports:
- name: mqtt
port: 1883
targetPort: 1883
selector:
app: mosquitto
type: primary
And add a second deployment which uses this config:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mosquitto-b
namespace: automation
spec:
selector:
matchLabels:
app: mosquitto
template:
metadata:
labels:
app: mosquitto
type: bridge
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mosquitto
topologyKey: kubernetes.io/hostname
containers:
- args:
- -c
- /mosquitto/config/mosquitto.conf
command:
- mosquitto
image: eclipse-mosquitto:2.0.12
name: mosquitto
ports:
- containerPort: 1883
- containerPort: 9001
volumeMounts:
- mountPath: /mosquitto/config
name: config
securityContext:
runAsGroup: 1883
runAsUser: 1883
volumes:
- configMap:
name: bridge-conf
name: config
That’s it! Now you have a super robust mosquitto setup. The two pods will run different nodes and sync all messages.
You can see get the manifests directly off of my Github repo, where I use a kustomization to generate the config..
You can test your automations working through rollout restarts and enjoy increased robustness of your home automations.