Giter Site home page Giter Site logo

clj-kinesis-to-firehose's Introduction

kinesis-to-firehose

Clojars Project

Push events from a Kinesis stream to a Redshift/S3 Firehose stream

Build status

Build Status

Installation

Add the following to your project.clj :dependencies:

[kinesis-to-firehose "0.1.4"]

Usage

Lambda function example with uswitch

Simple lambda function that reads events in JSON format from a Kinesis stream and writes them to different Firehoses based on it's type.

(ns my-lambda.core
  (:require [uswitch.lambada.core :refer [deflambdafn]]
            [clojure.java.io :as io]
            [cheshire.core :refer [generate-string parse-string generate-stream]]
            [kinesis-to-firehose.core :refer [kinesis->firehose!]]))

; Create a config map
; { :dispatch (fn which picks values for dispatch-value check)
;   :rules [
;     {:name :name-of-rule
;      :dispatch-value #{"matches results for dispatch fn"} or :default
;      :transform #(fn which prepares record for S3 or Redshift write)
;      :streams ["vec of firehose streams"]}]

(def my-kinesis-stream-mappings-and-transformations
  {:dispatch (fn [event] (:type (cheshire.core/parse-string event true)))
   :rules
   [{:name :lol-cats
     :dispatch-value #{"cat"}
     :transform (fn [event] (cheshire.core/generate-string event))
     :streams ["lol-cats-redshift-firehose-1" "lol-cats-redshift-firehose-2"]}
    {:name :lol-dogs
     :dispatch-value #{"dogs"}
     :transform (fn [event] (cheshire.core/generate-string event))
     :streams ["lol-dogs-s3-firehose"]}
    {:name :all-the-rest-lol-things
     :dispatch-value :default
     :transform (fn [event] (cheshire.core/generate-string event))
     :streams ["all-the-rest-lol-things-s3-firehose1"]}]})

(deflambdafn my-lambda.LambdaFn [is os context]
  (let [stream-writer (io/writer os)
  	results (kinesis->firehose! is my-kinesis-stream-mappings-and-transformations)]
    (generate-strean results stream-writer)
    (.flush stream-writer)))

clj-kinesis-to-firehose's People

Contributors

jaakkos avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.