↓ Skip to main content

How to run high availability Mosquitto on Kubernetes

·3 mins
Table of Contents

If you use mosquitto as your MQTT broker for home automation, you may benefit from running it in a “high availability” mode such that all of your home automations don’t go down as easily. A major reason for people preferring Home Assistant’s built-in ZHA over zigbee2mqtt and mosquitto is that it adds more moving parts.

It actually isn’t very hard to get two instances of mosquitto which share all messages bi-directionally. I use Kubernetes at home, so it makes sense to leverage services to achieve a high availability broker. While other solutions, such as HiveMQTT, exist and provide fully-featured Kubernetes operators, I chose to stick with mosquitto and plain manifests.

Single Deployment
#

A single (non HA) deployment of mosquitto follows. This is what I started off with.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mosquitto
  namespace: automation
spec:
  selector:
    matchLabels:
      app: mosquitto
  template:
    metadata:
      labels:
        app: mosquitto
        type: primary
    spec:
      containers:
        - image: eclipse-mosquitto:2.0.12
          name: mosquitto
          ports:
            - containerPort: 1883
            - containerPort: 9001
          command:
            - mosquitto
          args:
            - -c 
            - /mosquitto-no-auth.conf
      securityContext:
        runAsUser: 1883
        runAsGroup: 1883
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values: [mosquitto]
            topologyKey: kubernetes.io/hostname

The service
#

You’ll then want to add a service so you can reach your mosquitto pod with a predictable DNS record.

apiVersion: v1
kind: Service
metadata:
  name: mosquitto
  namespace: automation
spec:
  ports:
    - name: mqtt
      port: 1883
      targetPort: 1883
    - name: wss
      port: 9001
      targetPort: 9001
  selector:
    app: mosquitto
  type: LoadBalancer
  externalTrafficPolicy: Local
# Use an loadBalancerIP (e.g., MetalLB) or externalIP depending on your setup
#  externalIPs:
#    - ${EI_MQTT}

Making it High Availability
#

So how do we make it high availability you ask? All we need is a little config “to bridge” to the two instances of Mosquitto.

apiVersion: v1
kind: ConfigMap
metadata:
  name: bridge-conf
  namespace: automation
data:
  mosquitto.conf: |
    listener 1883
    allow_anonymous true

    connection broker0
    address mosquitto-p.automation
    topic # both 0

Add a service which only selects the primary pod:

apiVersion: v1
kind: Service
metadata:
  name: mosquitto-p
  namespace: automation
spec:
  ports:
    - name: mqtt
      port: 1883
      targetPort: 1883
  selector:
    app: mosquitto
    type: primary

And add a second deployment which uses this config. We use podAntiAffinity to schedule it on another node:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mosquitto-b
  namespace: automation
spec:
  selector:
    matchLabels:
      app: mosquitto
  template:
    metadata:
      labels:
        app: mosquitto
        type: bridge
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - mosquitto
            topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - -c
        - /mosquitto/config/mosquitto.conf
        command:
        - mosquitto
        image: eclipse-mosquitto:2.0.12
        name: mosquitto
        ports:
        - containerPort: 1883
        - containerPort: 9001
        volumeMounts:
        - mountPath: /mosquitto/config
          name: config
      securityContext:
        runAsGroup: 1883
        runAsUser: 1883
      volumes:
      - configMap:
          name: bridge-conf
        name: config

That’s it! Now you have a much more robust mosquitto setup. The two pods will run different nodes and sync all messages.

You can grab all the manifests directly off of my Github repo, where I use a kustomization to generate the config..

You can test your automations working through rollout restarts and enjoy increased robustness of your home automations.

Vasil Zlatanov
Author
Vasil Zlatanov
Principal Engineer at Netcraft