Get Started with AWS Lambda and Kafka

This tutorial shows how to produce Kafka messages in AWS Lambda. If you want to consume Kafka messages in AWS Lambda then check this one

TODO: andreas

Create Kafka

First, create an Upstash Kafka cluster and topic following those steps. You will need the endpoint, username and password in the following steps.

Create Project

We will use Serverless Framework to create the application.

kafka-examples git:(master) serverless
 What do you want to make? AWS - Node.js - HTTP API
 What do you want to call this project? produce-in-lambda
Downloading "aws-node-http-api" template...
Project successfully created in produce-in-lambda folder

Then we will initialize a node project and install axios dependency.

npm init
npm install axios

Implement the Lambda Function

Open the handler.js and update as below:

const fetch = require("axios").default;

module.exports.hello = async (event) => {
  const msg = "Hello";
  const address = "https://REPLACE_YOUR_ENDPOINT";
  const user = "REPLACE YOUR USERNAME";
  const pass = "REPLACE YOUR PASSWORD";
  const auth = Buffer.from(`${user}:${pass}`).toString("base64");
  const response = await fetch(`${address}/produce/newtopic/${msg}}`, {
    headers: {
      Authorization: `Basic ${auth}`,
    },
  });
  const res = response.data;

  return {
    statusCode: 200,
    body: JSON.stringify(
      {
        header: "Pushed this message to Upstash Kafka with REST API!",
        message: msg,
        response: res,
      },
      null,
      2
    ),
  };
};

You need to replace the endpoint, username and password above with the values that you copy from the Upstash Console.

The above code simply creates a producer and sends the message to Kafka.

Deploy the Lambda Function

You can deploy your function to AWS by running:

serverless deploy

This command will output your URL. The output should be something like this:

{
  "header": "Pushed this message to Upstash Kafka!",
  "message": {
    "value": "Hello message"
  }
}

Test the Function

Now let’s validate that the messages are pushed to Kafka. We can consume the Kafka topic using the REST API. You can copy the curl code to consume from the Upstash Console.

produce-in-lambda git:(master) ✗ curl https://full-mantis-14289-us1-rest-kafka.upstash.io/consume/GROUP_NAME/GROUP_INSTANCE_NAME/newtopic -u REPLACE_USER_NAME:REPLACE_PASSWORD

[ {
  "topic" : "newtopic",
  "partition" : 0,
  "offset" : 98,
  "timestamp" : 1639610767445,
  "key" : "",
  "value" : "Hello message",
  "headers" : [ ]
} ]%

REST vs Kafka Client

We can also use a native Kafka client (e.g. KafkaJS) to access our Kafka cluster. See the repo for both examples. But there is a latency overhead if connecting (and disconnecting) to the Kafka with each function invocation. In our tests, the latency of the function with REST is about 10ms where it is about 50ms when KafkaJS is used. Kafka client's performance could be improved by caching the client outside the function but it can cause other problems as explained here.

Troubleshooting: If Lambda function outputs internal error, check the cloudwatch log (Lambda > Monitor > View logs in CloudWatch).