How to run high availability Mosquitto on Kubernetes
Table of Contents
If you use mosquitto as your MQTT broker for home automation you may be benefit from running it in a “high availability” mode such that all of your home automations don’t go down as easily. A major reason for people preferring home assistant’s built in ZHA over zigbee2mqtt and mosquitto is that it adds more moving parts.
It actually isn’t very hard to get two instances of mosquitto which share all messages bi-directionally. I use Kubernetes at home, so it makes sense to leverage services to achieve a high availability broker. While other solution such as HiveMQTT exists and provide full on Kubernetes operators, I chose to stick with mosquitto and plain manifests.
Single Deployment #
A single (non HA) deployment of mosquitto follows. This is what I started off with.
apiVersion: apps/v1
kind: Deployment
metadata:
name: mosquitto
namespace: automation
spec:
selector:
matchLabels:
app: mosquitto
template:
metadata:
labels:
app: mosquitto
type: primary
spec:
containers:
- image: eclipse-mosquitto:2.0.12
name: mosquitto
ports:
- containerPort: 1883
- containerPort: 9001
command:
- mosquitto
args:
- -c
- /mosquitto-no-auth.conf
securityContext:
runAsUser: 1883
runAsGroup: 1883
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: [mosquitto]
topologyKey: kubernetes.io/hostname
The service #
You’ll then what to add a service so you can reach your mosquitto pod with a predictable DNS record.
apiVersion: v1
kind: Service
metadata:
name: mosquitto
namespace: automation
spec:
ports:
- name: mqtt
port: 1883
targetPort: 1883
- name: wss
port: 9001
targetPort: 9001
selector:
app: mosquitto
type: LoadBalancer
externalTrafficPolicy: Local
# Use an loadBalancerIP (e.g. MetalLB) or externalIP depending on your setup
# externalIPs:
# - ${EI_MQTT}
Making it High Availability #
So how do we make it high availability you ask? All we need is a little config “to bridge” to the two instances of Mosquitto.
apiVersion: v1
kind: ConfigMap
metadata:
name: bridge-conf
namespace: automation
data:
mosquitto.conf: |
listener 1883
allow_anonymous true
connection broker0
address mosquitto-p.automation
topic # both 0
Add a service which only selects the primary pod:
apiVersion: v1
kind: Service
metadata:
name: mosquitto-p
namespace: automation
spec:
ports:
- name: mqtt
port: 1883
targetPort: 1883
selector:
app: mosquitto
type: primary
And add a second deployment which uses this config. We use podAntiAffinity to schedule it on another node:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mosquitto-b
namespace: automation
spec:
selector:
matchLabels:
app: mosquitto
template:
metadata:
labels:
app: mosquitto
type: bridge
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mosquitto
topologyKey: kubernetes.io/hostname
containers:
- args:
- -c
- /mosquitto/config/mosquitto.conf
command:
- mosquitto
image: eclipse-mosquitto:2.0.12
name: mosquitto
ports:
- containerPort: 1883
- containerPort: 9001
volumeMounts:
- mountPath: /mosquitto/config
name: config
securityContext:
runAsGroup: 1883
runAsUser: 1883
volumes:
- configMap:
name: bridge-conf
name: config
That’s it! Now you have a much more robust mosquitto setup. The two pods will run different nodes and sync all messages.
You can grab all the manifests directly off of my Github repo, where I use a kustomization to generate the config..
You can test your are automations working through rollout restarts and enjoy increased robustness of your home automations.