oslabs-beta/kafkasocks

Kafka Socks

An easy-to-use, lightweight KafkaJS-to-Socket.io library for connecting and visualizing data in realtime.

Table of Contents

• About
• Features
• Getting Started
• Example
• Contributors
• License

About

Kafka Socks is an easy-to-use and lightweight framework that combines Kafka consumer functionality with WebSockets to pipe the Kafka messages directly to the frontend client, in realtime. Kafka Socks abstracts away much of the boilerplate and setup of this oft-used Kafka-Websocket architecture, providing developers with a simple and intuitive set of classes to achieve a powerful result on the client-side.

The typical use case for the Kafka Socks library is rendering realtime data on a frontend client, but Kafka Socks framework is unopinionated and flexible enough to process realtime data in whatever way the developer may see fit.

Without a WebSocket, the only way a web client could access data consumed by the Kafka consumer on the server side would be fetch requests. Not only are fetch requests notoriously slow, browsers also limit the number of responded fetch requests a client may have at any given time (most browers set this limit below 10). In short, fetch requests could get the data to the frontend, but doing so would mean that the frontend would lose the ability to access this data in realtime; in applications where frontend rendering in realtime is necessary, fetch requests simply won't work.

Using the observer design pattern, WebSockets permit the server to pipe data in time because there is always an established and open link between the server and client. Kafka Socks did not invent this system design. In fact, it is a relatively common pattern to achieve realtime data processing on the frontend. Instead, Kafka Socks abstracts away the details of implementing this kafka-websocket design pattern, providing developers with an easy way to implement this pattern in a few lines of code.

Features

• Confluent : A singleton class used to instantiate a Kafka Cluster object using cluster hosted by Confluent.io
  
  
• Consumer: A wrapper around a kafkaJS Consumer object, instantiate as many (or as few) consumers as needed
  
  
• Subject: Used to create a new Kafka Socks Subject, which pipes the messages consumed by the Kafka consumers to the specified websocket namespace

Getting Started

Install Kafka Socks as an npm module and save it to your package.json as a dependency.

npm install kafka-socks

Once installed, you can now require the modules necessary to implement Kafka Socks:

import { Confluent, Consumer, Subject } from 'kafka-socks';

How to Use

1. Import the library classes needed:

import { Confluent, Consumer, Subject } from 'kafka-socks';

2. Instantiate a websocket server. (Done here using socket.io to wrap around an express server):

const express = require('express');
const http = require('http');
const { Server } = require('socket.io');

const app = express();
const server = http.createServer(app);
const io = new Server(server);

3. Instantiate the Kafka Cluster object using the Kafka Socks Confluent class:

const kafka = new Confluent(
    API_KEY,
    API_SECRET,
    KAFKA_BOOTSTRAP_SERVER
  )
  .create("client-id");

4. Instantiate Kafka Socks Consumer object - you can create as many as you need:

const kafkaConsumer = kafka.consumer({ groupId: 'your-groupId-here' });
const kafkaSocksConsumer = new Consumer(kafkaConsumer, 'kafka-topic', 'websocket-event-ID')

5. Link the Kafkasocks Consumers with websocket namespaces for the front end:

const kafkaSocksSubject = new Subject(io, 'websocket-namespace-ID')

6. Then simply set up your WebSocket listener on the front end using your favorite WebSockets framework!

Contributors

Kafka Socks is an open-source community project on Github. While the project is maintained by a small group of dedicated engineers (below), we are grateful to the community for bug fixes, feature development and other contributions.

Allison Jacobs @allisonIsCoding

Jason Fricano @jfricano

Jenessa Chapalamadugu @jenessachap

Vinit Patel @v-za

We welcome contributions to Kafka Socks, but we also would love to see a thriving third-party ecosystem. If you are interest in creating an open-source project that builds on top of Kafka Socks, please don't hesitate to reach out, and we'd be happy to provide feedback and support.

License

This product is licensed under the MIT License - see the LICENSE.md file for details.

This is an open source product. We are not affiliated nor endorsed by either the Apache Software Foundation or KafkaJS.

This product is accelerated by OS Labs.

kafkasocks's Issues

Can't install module

Hello.

I am currently trying out your project.

Currently when I do:

package.json

{
  "name": "kafka-to-socket",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "tsc": "./node_modules/typescript/bin/tsc",
    "build": "tsc",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@types/node": "^12.20.37",
    "typescript": "^4.5.3"
  },
  "dependencies": {
    "kafka-socks": "0.1.10",
    "ts-node": "^10.4.0"
  }
}

And when I add:

import { Confluent, Consumer, Subject } from 'kafka-socks';

const express = require('express');
const http = require('http');
const { Server } = require('socket.io');

const app = express();
const server = http.createServer(app);
const io = new Server(server);

const KAFKA_BOOTSTRAP_SERVER = ['localhost:31659']
const kafka = new Confluent(KAFKA_BOOTSTRAP_SERVER).create("client-id");

const kafkaConsumer = kafka.consumer({ groupId:  "test-group" });
const kafkaSocksConsumer = new Consumer(kafkaConsumer, "topic", 'websocket-event-ID')
const kafkaSocksSubject = new Subject(io, 'websocket-namespace-ID')

I get this error:

I am day to day, not working with JS or typescript so I am a bit confused here?

I tried to install the type but don't think this exists.

npm i --save-dev @types/kafka-socks 

npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@types%2fkafka-socks - Not found
npm ERR! 404 
npm ERR! 404  '@types/kafka-socks@*' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/simon/.npm/_logs/2021-12-13T00_43_05_792Z-debug.log

Is this something you have seen before?