Eclipse Hono is an open-source project for connecting large numbers of heterogeneous devices to a (cloud) back end. Hono defines service interfaces that enable uniform communication between devices, which may use widely differing protocols, as well as business applications. The project has been around for some time now and has been successfully used in Bosch IoT Device Management for several years. What is new is that it now supports Apache Kafka for messaging. The resulting benefits and changes in the use of Hono are introduced in this article.
©Bosch.IO
Data coming from devices is transformed in Hono to a uniform message format by protocol adapters (e.g. there are some for MQTT, HTTP, and CoAp) and passed to a messaging system. Business applications read these messages, which are either telemetry messages or events, directly from the messaging system (they can also use Command and Control to send messages to devices and receive responses). For messaging, which is not part of Hono but must be provided externally, the AMQP 1.0 protocol has been used so far.
Since version 1.7, Hono supports Apache Kafka for messaging as a technical preview. A key feature of Hono is the definition and implementation of APIs for forwarding messages to and from a messaging system. Since the API definitions are based on messaging and depend directly on its properties, supporting a new messaging system is a fundamental innovation for Hono. For all messaging APIs in Hono (telemetry, event, and command and control), new specifications based on Kafka were created, both facing the protocol adapter as well as the business application. Care was taken to ensure that they behave as similarly as possible to the AMQP-based APIs.
Benefits
Apache Kafka is a distributed event-streaming platform designed for very high throughput while providing certain guarantees in the order of messages. It is a great fit for Hono’s requirements regarding messaging for several reasons. Key benefits are:
- Scaling: Multiple Kafka servers are operated simultaneously as a cluster that can be expanded as needed. Kafka is optimized to allow many processes to write and read data simultaneously. With AMQP 1.0, multiple processes can also consume messages from the same address in parallel. However, that means they lose their order.
- Message ordering: One of Kafka’s core features is the partitioning of data by means of a partition key, which can be used to select data for which the order must be maintained and data which can be processed in parallel. In Hono, this allows us to easily ensure that all events from a device are provided to the business application in the correct order. The telemetry data of the device can be processed in parallel, and its order is also guaranteed. Messages from all other devices can be consumed in parallel, completely independently, even in multiple instances of the business application being used concurrently.
- Wide-scale adoption: Kafka is used by a large number of companies and is supported by many other technologies. Several companies offer Kafka as a service that can be booked on various cloud platforms. This makes it easy to provide Hono with a professionally operated messaging system. Vendor lock-in is also avoided.
Changes
AMQP 1.0 is a communication protocol where peers connect with each other to exchange messages. In contrast, a Kafka cluster consists of brokers that coordinate the writing (and reading) of data to permanent storage. This difference has two consequences: First, a message is available for retrieval for a while before it is eventually deleted. To achieve this effect in Hono with AMQP, you typically provide a separate message broker where events are stored to avoid message loss should the business application be temporarily unavailable. With Kafka, every message is stored. This includes telemetry messages (the storage time, however, depends on the configuration of the telemetry topics in the Kafka cluster). Secondly, communicating via permanent storage decouples the send and receive operations from each other: The protocol adapter can no longer report back to the device if there is currently no consumer connected. Instead, it simply stores the messages without “knowing” if or when a business application will read them.
Trying it out
When installing Hono, you need to deploy either an AMQP messaging network or a Kafka cluster. There is also the possibility of configuring Hono to use both messaging systems. Then, each tenant (a subset of devices and configuration) can be configured to use either AMQP or Kafka. The Hono project provides a publicly accessible sandbox environment on the Internet where you can experiment with Hono using AMQP or Kafka-based messaging.
Practical example
In the following, we walk you through a small example of how to use Apache Kafka-based messaging in the Hono sandbox. You will send a telemetry message to the HTTP protocol adapter and then read it from the Kafka cluster.
We simulate an Internet-enabled device by sending HTTP commands from the command line. For this, we use the popular open-source tool curl. If currently not installed on your computer, please refer to the installation instructions on the curl homepage. To simulate the business application, we use Hono’s command-line client, which can be downloaded from the Hono download page. Please note that Java must be installed in at least version 11 in order to run it.
If any of the commands shown below do not work (which can always happen with such instructions), please refer to the Kafka Messaging Guide where the explanations cover more details.
Registering a device
First, you need to create a new tenant in the sandbox to avoid interfering with other users. The following command creates a tenant with a random ID and configures it to use Kafka for messaging.
$ curl -i -H "content-type: application/json" --data-binary '{
"ext": {
"messaging-type": "kafka"
}
}' http://hono.eclipseprojects.io:28080/v1/tenants
The output should contain something along the lines of: {"id":"85f63e23-1b78-4156-8500-debcbd1a8d35"}.
Now copy this ID and set it as an environment variable as follows:
$ export MY_TENANT=85f63e23-1b78-4156-8500-debcbd1a8d35Register a device for the tenant:
$ curl -i -X POST http://hono.eclipseprojects.io:28080/v1/devices/$MY_TENANTand once again provide the returned ID as an environment variable as follows:
$ export MY_DEVICE=4412abe2-f219-4099-ae14-b446604ae9c6Finally set a password for the device:
$ export MY_PWD=my-pwd
$ curl -i -X PUT -H "content-type: application/json" --data-binary '[{
"type": "hashed-password",
"auth-id": "'$MY_DEVICE'",
"secrets": [{
"pwd-plain": "'$MY_PWD'"
}]
}]' http://hono.eclipseprojects.io:28080/v1/credentials/$MY_TENANT/$MY_DEVICE
Sending a message
Now that you have registered a device, you can send and receive messages.
Open a second terminal window to receive the messages. Go to the directory where you downloaded the Hono command-line client. Now start the client with the following command, making sure to replace my-tenant with the ID of the tenant you just created:
# in directory where the hono-cli-*-exec.jar file has been downloaded to
$ java -jar hono-cli-*-exec.jar --spring.profiles.active=receiver,sandbox,kafka --tenant.id="my-tenant"
The client is ready to receive messages if the output contains a line similar to the following:
10:23:01.807 [vert.x-eventloop-thread-0] INFO org.eclipse.hono.cli.app.Receiver - Receiver [tenant: 85f63e23-1b78-4156-8500-debcbd1a8d35, mode: all] created successfully, hit ctrl-c to exit
Now you can send a message from your simulated device to Hono. To do this, switch back to the first terminal window. The following command sends a telemetry message to the HTTP protocol adapter:
$ curl -i -u $MY_DEVICE@$MY_TENANT:$MY_PWD -H "content-type: application/json" --data-binary '{"temp": 42}' http://hono.eclipseprojects.io:8080/telemetryThe message should now be printed in the second terminal.
You can also send an event:
$ curl -i -u $MY_DEVICE@$MY_TENANT:$MY_PWD -H "content-type: application/json" --data-binary '{"hello": "kafka"}' http://hono.eclipseprojects.io:8080/eventNow you should have, hopefully, been successful in sending messages from a device to Hono and receiving them from the Kafka cluster. As a next step, you could take a look at the more detailed examples in Hono’s Kafka Messaging Guide. It also shows you how to send a command from the business application to the device and respond to it. If you want to dig deeper, you might want to take a look at the specifications of the Kafka-based APIs, which can be found in Hono’s API Specifications.
Conclusion
The messaging system is the foundation for Eclipse Hono’s core functionality because its interfaces are provided on top of it. Supporting Kafka alongside AMQP 1.0 is a very profound change for Hono, where great emphasis has been put on ensuring that interfaces can be easily mapped to each other. The key benefits of Kafka are its scalability, its ordering guarantees, its wide-scale adoption, and wealth of commercial service offerings. The main differences to messaging with AMQP are that all message types are brokered. This means that messages can be delivered even if the recipient was disconnected for a moment. The communication is also decoupled in terms of time so that direct feedback from the recipient to the sender of a message is no longer possible. In a brief practical example, we showed you how to send messages from a simulated device to the Hono sandbox and consume them from the Kafka cluster.