EMQX Cloud

This tutorial shows how to integrate Upstash Kafka with EMQX Cloud

EMQX, a robust open-source MQTT message broker, is engineered for scalable, distributed environments, prioritizing high availability, throughput, and minimal latency. As a preferred protocol in the IoT landscape, MQTT (Message Queuing Telemetry Transport) excels in enabling devices to effectively publish and subscribe to messages.

Offered by EMQ, EMQX Cloud is a comprehensively managed MQTT service in the cloud, inherently scalable and secure. Its design is particularly advantageous for IoT applications, providing dependable MQTT messaging services.

This guide elaborates on streaming MQTT data to Upstash by establishing data integration. This process allows clients to route temperature and humidity metrics to EMQX Cloud using MQTT protocol, and subsequently channel these data streams into a Kafka topic within Upstash.

Initiating Kafka Clusters on Upstash

Begin your journey with Upstash by visiting Upstash and registering for an account.

Kafka Cluster Creation

1. After logging in, initiate the creation of a Kafka cluster by selecting the Create Cluster button.

2. Input an appropriate name and select your desired deployment region, ideally close to your EMQX Cloud deployment for optimized performance.

3. Choose your cluster type: opt for a single replica for development/testing or a multi-replica setup for production scenarios.

4. Click Create Cluster to establish your serverless Kafka cluster.

Topic Configuration

1. Inside the Cluster console, navigate to Topics and proceed with Create Topic.
2. Enter emqx in the Topic name field, maintaining default settings, then finalize with Create.

Setting Up Credentials

1. Go to Credentials in the navigation menu and choose New Credentials.

2. Here, you can customize the topic and permissions for the credential. Default settings will be used in this tutorial.

With these steps, we have laid the groundwork for Upstash.

Establishing Data Integration with Upstash

Enabling EMQX Cloud’s NAT Gateway

1. Sign in to the EMQX Cloud console and visit the deployment overview page.

2. Click on the NAT Gateway section at the bottom of the page and opt for Subscribe Now.

Data Integration Setup

1. In the EMQX Cloud console, under your deployment, go to Data Integrations and select Upstash for Kafka.

2. Fill in the Endpoints details from the Upstash Cluster details into the Kafka Server fields. Insert the username and password created in Create Credentials into the respective fields and click Test to confirm the connection.

3. Opt for New to add a Kafka resource. You’ll see your newly created Upstash for Kafka listed under Configured Resources.

4. Formulate a new SQL rule. Input the following SQL command in the SQL field. This rule will process messages from the temp_hum/emqx topic and append details like client_id, topic, and timestamp.

SELECT
timestamp as up_timestamp,
clientid as client_id,
payload.temp as temp,
payload.hum as hum
FROM
"temp_hum/emqx"

5. Conduct an SQL test by inputting the test payload, topic, and client data. Success is indicated by results similar to the example below.

6. Advance to Next to append an action to the rule. Specify the Kafka topic and message format, then confirm.

# kafka topic
emqx

# kafka message template
{"up_timestamp": ${up_timestamp}, "client_id": ${client_id}, "temp": ${temp}, "hum": ${hum}}

7. View the rule SQL statement and bound actions by clicking View Details after successfully adding the action.

8. To review created rules, click View Created Rules on the Data Integrations