Giter Site home page Giter Site logo

example-applications's Introduction

Example Applications

This repository contains a number of example applications that can be built and run on PNDA. Each application directory contains more detailed information.

Spark Streaming

Examples of consuming data from Kafka and populating both HBase and OpenTSDB with simple Scala based Spark Streaming applications.

Spark

Example of consuming data ingested by Gobblin on a batch basis and producing Parquet datasets, optimized for consumption by Impala.

Jupyter

Example of a notebook for manipulating network data.

H2O

Application that runs the H2O data science platform as an application on PNDA.

Flink Streaming

  • Count Words (scala) Count the words from Socket.
  • Count Words (python) Count the words from input file.
  • Flink Windows (java) host-network-data-usage illustrating Flink windows, triggers and event processing.
  • Count Hashtags (java) specific word count from input file illustrating metrics, counters and accumulators.

Compound Packages

An example of a package containing multiple application component types, in this case a Spark app and related Jupyter notebook.

example-applications's People

Contributors

dharaneeshvrd avatar donaldh avatar ganeshmanal avatar hitesh-maplelabs avatar jeclarke avatar jegarnie avatar jgarnier avatar jubarbot-cisco avatar manasa-sriram-ml avatar pndacicd avatar stephanesan avatar trsmith2 avatar willis27 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

example-applications's Issues

Spark Magic Usage

User guide is required for usage of Spark Magic.

Steps regarding usage of sparkmagic in spark kernel will be helpful in order to use various predefined magics. Details regarding livy connectivity to interact with spark can be added in steps which will help in creating and managing sessions.

Maven Build Issue

Ran into this issue trying to build spark-streaming application.

Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.6:single (make-assembly) on project spark-streaming-to-hbase-example-app: Execution make-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:2.6:single failed: group id '1385651864' is too big ( > 2097151 ). Use STAR or POSIX extensions to overcome this limit -> [Help 1]

Fix seems to adding tarLongFileMode in appropriate pom.xml file - app-package/pom.xml in this case.

    ```
<configuration>
           <tarLongFileMode>posix</tarLongFileMode>
           <descriptors>
               <descriptor>src/main/assembly/assembly.xml</descriptor>
           </descriptors>
           <appendAssemblyId>false</appendAssemblyId>
        </configuration>

Should Kafka Version in examples be 0.10.0.1

In examples that are using kafka (eg: spark-streaming) should pom.xml be updated to have 0.10.0.1 version of kafka clients? I believe that is what is being used in platform.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.