How machines communicate - MQTT protocol

How machines communicate - MQTT protocol



In the previous article, we discussed Modbus protocol , which is the de facto standard in the industry for M2M interaction. Developed back in 1979, it has a number of significant flaws that MQTT solves.

The MQTT protocol is quite young (standardized only in 2016), but it has already gained wide acceptance in industry and IoT. It was specifically designed to be as compact as possible for unstable Internet channels and low-power devices, and allows you to deliver messages guaranteed in the event of packet loss and communication failure.

Key features of the MQTT protocol:

  • Compact and lightweight - minimal overhead for sending data to save traffic.
  • Loss Resistance - guaranteed delivery in unstable network connections.
  • Asynchronous - allows you to serve a large number of devices, and does not depend on network delays.
  • QoS support - the ability to control the priority of messages and ensure the delivery of the message to the addressee.
  • Dynamic configuration - does not require pre-coordination of fields and data formats, can be configured on the fly.
  • Works for NAT - clients can be behind NAT, only the server (broker) must have a real IP. Allows you to do without VPN and port forwarding.
  • Easy Addressing - data fields have textual names that are understandable to humans. No need to memorize numeric addresses and bit offsets.

In the article, we will compare MQTT and Modbus, analyze the structure of the protocol, the basic concepts, try using the example of a cloud MQTT broker in the context of an unstable Internet connection.

MQTT protocol history


MQTT was developed by IBM in 1999, and was initially used internally for its solutions.

In November 2011, IBM, together with Eurotech, announced their participation in the Eclipse M2M working group and the transfer of the MQTT code to the Eclipse Paho project.

In 2013, the OASIS (Organization for the Advancement of Structured Information Standards) consortium begins the process of standardizing the MQTT protocol. Up to this point, the protocol specification was published under a free license, and companies such as Eurotech (previously known as Arcom) are already using the protocol in their products.

In October 2014, OASIS publishes the first official standard of the MQTT protocol.

In 2016, the protocol was standardized by the International Organization for Standardization ISO and received ISO/IEC 20922.

Since 2014, interest in the protocol begins to grow rapidly and, judging by the Google Trends schedule, today exceeds interest in Modbus.


Google Trends Comparative Chart

Basic Concepts


MQTT has a client-server architecture. Messaging occurs through a central server called a broker. Under normal conditions, clients cannot communicate directly with each other, and all data exchange takes place through a broker.

Clients can act as data providers (Publisher) and as data recipients (Subscriber). In Russian translation, these terms are often translated as publisher and subscriber, but to avoid confusion, we will use only original terminology.


In the MQTT protocol, clients communicate with each other via a central site

At the application level, the protocol runs on top of TCP/IP and can easily connect remote objects directly via the Internet, without the need for using VPN tunnels. It is enough that the broker has a real IP address and all clients can connect to it. At the same time, clients can be behind NAT. Since clients initiate a connection in the MQTT protocol, it is not necessary to forward ports to establish a connection, while in Modbus/TCP the server initiates a connection (master), which requires direct network accessibility.

The standard port of the MQTT broker for incoming TCP connections is 1883 . When using a secure SSL connection, use 8883 .

Broker


Broker - is the central node of MQTT, providing customer interaction. Data exchange between clients occurs only through a broker. Server software or a controller can act as a broker. Its tasks include receiving data from clients, processing and storing data, delivering data to clients, and monitoring the delivery of messages.

Publisher/Subscriber


To understand the difference between Publisher and Subscriber, let's take a simple example: a humidity sensor measures the humidity in a room, and if it drops below a certain level, the humidifier turns on.

In this case, the humidity sensor acts as Publisher : its task is only to publish data in the direction of the broker. The humidifier plays the role of Subscriber : it subscribes to updates of humidity data and receives relevant data from the broker, while the humidifier can decide at what point to turn on humidification.

In this scheme, MQTT clients, that is, the sensor and humidifier, are not aware of each other's existence, and do not interact directly. The broker can receive data from different sources, manipulate them, for example, calculate the average value from several sensors, and return the already processed data to the subscriber.


Publisher sends data to the broker, Subscriber subscribes to updates of this data

At the same time, the asynchronous MQTT protocol provides that the sensor and humidifier can be online at different times, lose packets, and be unavailable. The broker will take care of storing the latest data received from the sensor and ensuring their delivery to the humidifier.

Topic


Topics are used to identify entities in MQTT, in the Russian translation they are also called channels. Topics consist of UTF8 characters, and have a tree structure similar to a UNIX file system. This is a convenient mechanism for allowing entities to be named in a human-readable form.

Sample Topics in MQTT

  # Temperature sensor in the kitchen
 home/kitchen/temperature

 # Temperature sensor in the bedroom
 home/sleeping-room/temperature

 # Light sensor on the street
 home/outdoor/light
  

This approach allows you to visually see what data is being transmitted, and it is convenient to develop and debug the code without having to memorize the numeric address of the data placement, as is done in Modbus.

Topics also provide for wildcard syntax familiar to those working with the UNIX file system. Wildcard can be single-level and multi-level.

A sibling wildcard is indicated by the symbol " + ".

For example, to get data from temperature sensors in all rooms in the house, the subscriber needs to subscribe to this topic:

  home/+/temperature
  

As a result, he will sign up to receive data from such sensors:

  home/kitchen/temperature
 home/sleeping-room/temperature
 home/living room/temperature
 home/outdoor/temperature
  

A layered wildcard is indicated by the symbol " # ".
An example of obtaining data from all sensors in all rooms in the house:

  home/#
  

Subscribing to such a topic will allow receiving data from such sensors:

  home/kitchen/temperature
 home/kitchen/humidity
 home/kitchen/light
 home/sleeping-room/temperature
 home/sleeping-room/humidity
 home/sleeping-room/light
 ....
  

Customer Identification


To control access, MQTT provides client authentication, as opposed to the Modbus protocol, which does not have this function. The following fields are used for access control:

ClientId - (mandatory) a unique identifier of the client. Must be unique to each client. The current version of the MQTT 3.1.1 standard allows you to use an empty ClientId field, if you do not want to save the connection state.

Username - (optional field) login for authentication, in UTF-8 format. May not be unique. For example, a group of clients can log in with the same login/password.

Password - (optional field) can only be sent with the Username field, and the Username can be transmitted without the Password field. Maximum 65535 bytes. It is important to know that the name and password are transmitted in the clear, so if the data is transferred over public networks, you must use SSL to encrypt the connection.

Package Structure


As mentioned above, clients always initiate the connection in the MQTT protocol, regardless of whether they are recipients (Subscriber) or suppliers (Publisher) of data. Disassemble the connection setup packet captured using Wireshark.


MQTT option packet transmitted over an unencrypted channel

The TCP header shows that the packet was transmitted on port 1883, that is, encryption is not used, which means that all data is available in clear form, including the login and password.

Title


Message Type - Connect (command 0x0001), establishing a connection with a broker. Basic commands: Connect, Disconnect, Publish, Subscribe, Unsubscribe. There are also commands to confirm receipt, keep alive, etc.

DUP flag - means that the message is re-transmitted, used only in the message types PUBLISH, SUBSCRIBE, UNSUBSCRIBE, PUBREL, in cases where the broker has not received confirmation of receipt of the previous message.
QoS Level - Quality of Service flag. We will analyze this topic further further.
Retain - data published with the retain flag is stored on the broker. Upon subsequent subscription to this topic, the broker will immediately send a message with this flag. Used only in messages with type Publish.

Use in practice




Now, having familiarized with the theory, we will try to work with MQTT in practice. To do this, we will use the open-source program Mosqutto , which can work in both client mode and server (broker) mode. Works on Windows, macOS, Linux. The program is very convenient for debugging and studying the MQTT protocol, while it is also widely used in industrial operation.We will use it as a client to send and receive data from a remote cloud broker.

Many cloud providers provide MQTT broker services, such as Microsoft Azure IoT Hub , Amazon AWS IoT , and others. In this example, we will use the Cloudmqtt.com service, since it has the simplest registration, and the free tariff is enough for training.

After registration, in your personal account are available details for connecting to the broker. Since we connect to the server through public Internet networks, it is wise to use the SSL port to encrypt the traffic.


Details of access to the MQTT broker in the personal account of the cloud provider

The flexibility of the MQTT protocol allows the client to transfer data not previously defined on the broker. That is, there is no need to pre-create the necessary topics in which Publisher data can be written. Using the data obtained from your personal account, we will try to manually create a request to publish data in the topic habr/test/random and read from it.

mosquitto_sub - subscriber utility client
mosquitto_pub - publisher utility client

First, let's connect to the broker as a subscriber, and subscribe to receive data from the topic
habr/test/random .

  mosquitto_sub -d --capath/etc/ssl/certs/--url mqtts://hwjspxxt: 7oYugN7Fa5Aa@postman.cloudmqtt.com: 27529/habr/test/random
 Client mosq/zEPZz0glUiR4aEipZA sending CONNECT
 Client mosq/zEPZz0glUiR4aEipZA received CONNACK (0)
 Client mosq/zEPZz0glUiR4aEipZA sending SUBSCRIBE (Mid: 1, Topic: habr/test/random, QoS: 0, Options: 0x00)
 Client mosq/zEPZz0glUiR4aEipZA received SUBACK
  

It can be seen that the connection was successful, and we subscribed to the topic habr/test/random , and now we are waiting for the data in this topic from the broker.
Since the SSL connection is used, to verify the certificate, you must specify the path by which the program will look for root encryption certificates. Since the service in our example uses a certificate issued by a trusted certificate authority, we specify the path to the system store of root certificates: - capath/etc/ssl/certs/ < br/>
In the case of a self-signed certificate, you need to specify the path to the desired CA. It is also important to take into account the difference in the URI format for an SSL connection - mqtt s ://, and a connection without encryption - mqtt://. In case of a certificate verification error, the program terminates without an error message. For a more detailed output, you can use the key - debug
Now let's try to publish the data in the topic, without interrupting the first program.

  mosquitto_pub -d --capath/etc/ssl/certs/--url mqtt://hwjspxxt: 7oYugN7Fa5Aa@postman.cloudmqtt.com: 27529/habr/test/random -m  "Hi habr!"
 Client mosq/sWjh9gf8DRASrRZjk6 sending CONNECT
 Client mosq/sWjh9gf8DRASrRZjk6 received CONNACK (0)
 Client mosq/sWjh9gf8DRASrRZjk6 sending PUBLISH (d0, q0, r0, m1, 'habr/test/random', ... (22 bytes))
 Client mosq/sWjh9gf8DRASrRZjk6 sending DISCONNECT
  

It can be seen that the data was successfully accepted by the server and published in the desired topic. At the same time, in the first window in which the mosquitto_sub program is running, we see how the message was received, with even Unicode working, you can see the message in the Russian language.

  Client mosq/zEPZz0glUiR4aEipZA received PUBLISH (d0, q0, r0, m0, 'habr/test/random', ... (22 bytes))
 Hi habr!
  

QoS and delivery guarantee


However, it’s not surprising to send a message in real time, because the same can be done even with the banal utility nc .Therefore, we will try to simulate an unstable connection between the subscriber and the sender. Imagine that both clients are working via GPRS, with huge packet loss, and even a successful TCP connection setup is rare, and you need to ensure that the subscriber receives the sender's message. In this case, QoS options come to the rescue.

By default, messages have the QoS flag set to 0 , which means “Fire and forget”: Publisher publishes a message to the broker, but does not require that the message be guaranteed to be delivered to the subscriber. This is suitable for data whose loss is not critical, for example, for regular measurements of humidity or temperature.

QoS 1: at least once - at least one . This flag means that until Publisher receives confirmation of delivery to the subscriber, this publication will be sent to the broker, and then to the subscriber. Thus, the subscriber must receive this message at least once.

QoS 2: Exactly once - guaranteed one . The QoS flag, which provides the highest guarantee of message delivery through the use of additional procedures for the confirmation and completion of publication (PUBREC, PUBREL, PUBCOMP). Let's apply for situations when it is necessary to exclude any losses and duplication of data from sensors. For example, when an alarm is triggered from a received message, an emergency call.

To simulate a bad connection, disable both clients and try to send a message with the highest QoS priority, and add the Retain option so that the sent message is saved on the broker.

  mosquitto_pub --retain --qos 2 -d --capath/etc/ssl/certs/--url mqtt://hwjspxxt: 7oYugN7Fa5Aa@postman.cloudmqtt.com: 27529/ habr/test/random -m "Very important hello!"
 Client mosq/Xwhua3GAyyY9mMd05V sending CONNECT
 Client mosq/Xwhua3GAyyY9mMd05V received CONNACK (0)
 Client mosq/Xwhua3GAyyY9mMd05V sending pUBLISH (d0, q2, r1, m1, 'habr/test/random', ... (37 bytes))
 Client mosq/Xwhua3GAyyY9mMd05V received PUBREC (Mid: 1)
 Client mosq/Xwhua3GAyyY9mMd05V sending PUBREL (m1)
 Client mosq/Xwhua3GAyyY9mMd05V received PUBCOMP (Mid: 1, RC: 0)
 Client mosq/Xwhua3GAyyY9mMd05V sending DISCONNECT
  

Now, after a while, our recipient was finally able to establish an Internet connection and connected to a broker:

  mosquitto_sub -d --capath/etc/ssl/certs/-d --url mqtts://hwjspxxt: 7oYugN7Fa5Aa@postman.cloudmqtt.com: 27529/habr/test/random
 Client mosq/VAzcLVMB1MiWhYxoJS sending CONNECT
 Client mosq/VAzcLVMB1MiWhYxoJS received CONNACK (0)
 Client mosq/VAzcLVMB1MiWhYxoJS sending SUBSCRIBE (Mid: 1, Topic: habr/test/random, QoS: 0, Options: 0x00)
 Client mosq/VAzcLVMB1MiWhYxoJS received SUBACK
 Subscribed (mid: 1): 0
 Client mosq/r6UwPnDvx8aNInpPF6 received PUBLISH (d0, q0, r1, m0, 'habr/test/random', ... (37 bytes))
 Very important hello!
  

Conclusion


MQTT is a modern advanced protocol, devoid of many flaws in predecessors. Its flexibility allows you to add client devices without settings on the broker, which saves time. The threshold of entry for understanding and setting up the protocol is quite low, and the presence of libraries for many programming languages ​​allows you to select any technology stack for development. The message delivery guarantee significantly distinguishes MQTT from its predecessors, and allows you not to waste time on the extra development of your own integrity control mechanisms at the network level.

Source text: How machines communicate - MQTT protocol