How to run high availability Mosquitto on Kubernetes

October 31, 2021 3 minutes

If you use mosquitto as your MQTT for home automation you may be benefit from running it in a “high availability” mode such that all your HA don’t go down as easily. A major reason for poeple skipping zigbee2mqtt and mosquitto in favour of ZHA is that it adds more moving parts.

It actually isn’t very hard to instances of mosquitto which share all message bi-directionally. I use Kubernetes my home and so it makes sense to leverage services to achieve a HA broker. While other solution such as HiveMQTT exists and provide full on Kubernetes operators, I chose to stick with mosquitto due to arm64 builds and simplicity.

Single Deployment

A single deployment of mosquitto follows. This is what I storted off with.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mosquitto
  namespace: automation
spec:
  selector:
    matchLabels:
      app: mosquitto
  template:
    metadata:
      labels:
        app: mosquitto
        type: primary
    spec:
      containers:
        - image: eclipse-mosquitto:2.0.12
          name: mosquitto
          ports:
            - containerPort: 1883
            - containerPort: 9001
          command:
            - mosquitto
          args:
            - -c 
            - /mosquitto-no-auth.conf
      securityContext:
        runAsUser: 1883
        runAsGroup: 1883
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values: [mosquitto]
            topologyKey: kubernetes.io/hostname

The service

You’ll then what to add a service so you can reach your mosquitto pod with a predictable DNS record.

apiVersion: v1
kind: Service
metadata:
  name: mosquitto
  namespace: automation
spec:
  ports:
    - name: mqtt
      port: 1883
      targetPort: 1883
    - name: wss
      port: 9001
      targetPort: 9001
  selector:
    app: mosquitto
  type: LoadBalancer
  externalTrafficPolicy: Local
# Use an loadBalancerIP (e.g. MetalLB) or externalIP depending on your setup
#  externalIPs:
#    - ${EI_MQTT}

Making it High Availability

So how do we make it HA you ask? Simply for a second mosquitto broker which will bridge to the first one using the service we just added.

apiVersion: v1
kind: ConfigMap
metadata:
  name: bridge-conf
  namespace: automation
data:
  mosquitto.conf: |
    listener 1883
    allow_anonymous true

    connection broker0
    address mosquitto-p.automation
    topic # both 0    

Add a service which only selects the primary pod:

apiVersion: v1
kind: Service
metadata:
  name: mosquitto-p
  namespace: automation
spec:
  ports:
    - name: mqtt
      port: 1883
      targetPort: 1883
  selector:
    app: mosquitto
    type: primary

And add a second deployment which uses this config:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mosquitto-b
  namespace: automation
spec:
  selector:
    matchLabels:
      app: mosquitto
  template:
    metadata:
      labels:
        app: mosquitto
        type: bridge
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - mosquitto
            topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - -c
        - /mosquitto/config/mosquitto.conf
        command:
        - mosquitto
        image: eclipse-mosquitto:2.0.12
        name: mosquitto
        ports:
        - containerPort: 1883
        - containerPort: 9001
        volumeMounts:
        - mountPath: /mosquitto/config
          name: config
      securityContext:
        runAsGroup: 1883
        runAsUser: 1883
      volumes:
      - configMap:
          name: bridge-conf
        name: config

That’s it! Now you have a super robust mosquitto setup. The two pods will run different nodes and sync all messages.

You can see get the manifests directly off of my Github repo, where I use a kustomization to generate the config..

You can test your automations working through rollout restarts and enjoy increased robustness of your home automations.