ibm-watson-data-lab/spark.samples

tutorials and samples that show you how get the most out of IBM Analytics for Apache Spark

#Spark Tutorials

This repository contains tutorials and samples that show you how get the most out of IBM Analytics for Apache Spark.

Watch this repo for new content. Meanwhile, try these tutorials:

• Start Developing with Spark

• Sentiment Analysis of Twitter Hashtags

• Real-time Sentiment Analysis of Twitter Hashtags with Spark

• Getting started with GraphFrames in Apache Spark

• Predict Flight Delays with Apache Spark MLLib, FlightStats, and Weather Data

• Analyze Market Trends in Twitter Using Apache Spark, Python, and dashDB

• PixieDust: Magic for Your Python Notebook

spark.samples's Issues

Feature request: bare minimum sample project for spark streaming + message hub

Hi @DTAIEB, the streaming-twitter example comes up a lot when searching online for spark streaming + messagehub. However, as a scala noob I find it difficult to extract just the bits relating to spark streaming + messagehub.

It would be great if you had a bare minimum sample project (helloSparkStreamingMessageHub?) that could be used as a template for those who want to get started with spark streaming + messagehub without needing to understand all the twitter code or using an 'unoficial' jar file like:

%AddJar https://github.com/ibm-messaging/message-hub-samples/raw/master/java/message-hub-login-library/messagehub.login-1.0.0.jar
%Addjar https://github.com/ibm-messaging/iot-messgehub-spark-samples/releases/download/v0.1/streaming-kafka.jar

Using the new 1.5 jar no tweets come in

Using the new 1.5 jar I never succeed in collecting any tweets. I am quite sure about the credentials as when I use the same ones with the 1.2 jar I get tweets being collected. Off course then the tone analysis doesn't work as it is still trying to call the api in the old way.

I have been looking into the code but don't seem to find the actual issue.

Even after adapting the code to new version of DSX on spark getting errors

See my pull request.

%AddJar https://github.com/tjhgit/jarhub/raw/master/streaming-twitter-assembly-1.6.jar -f
val demo = com.ibm.cds.spark.samples.StreamingTwitter
// val config = demo.getConfig()
//Watson Tone Analyzer service
demo.setConfig("watson.tone.url","https://gateway.watsonplatform.net/tone-analyzer/api")
demo.setConfig("watson.api.version","2017-09-21")
demo.setConfig("watson.tone.password","xxx")
demo.setConfig("watson.tone.username","xxxx")

demo.setConfig("twitter4j.oauth.consumerKey","xxxx")
demo.setConfig("twitter4j.oauth.consumerSecret","xxxx")
demo.setConfig("twitter4j.oauth.accessToken","xxxx")
demo.setConfig("twitter4j.oauth.accessTokenSecret","xxxx")

import org.apache.spark.streaming._
demo.startTwitterStreaming(sc, Seconds(30))

Leads to the following error:

Name: Compile Error
Message: :42: error: type mismatch;
found : org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext
required: org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext
demo.startTwitterStreaming(sc, Seconds(30))
^
StackTrace:

Spark-2.0 Kernel startup is failing due to "data/libs/streaming-twitter-assembly-1.6.jar" has jopt-simple version 3.2

Spark-2.0 Kernel startup is failing with below error when user has pushed library "streaming-twitter-assembly-1.6.jar" to his "data/libs" directory.

"data/libs/streaming-twitter-assembly-1.6.jar" has jopt-simple version 3.2 which not compatible with "toree" with spark02.0, can you please update your "streaming-twitter-assembly-1.6.jar" with "jopt-simple-4.6.jar" or above level version.

TMP=/gpfs/fs01/user/sc01-4b6feb1d1106ba-5fc93277e5ba/kgateway/tmp
USER=sc01-4b6feb1d1106ba-5fc93277e5ba
Starting Spark Kernel with SPARK_HOME=/usr/local/src/spark20master/spark
Exception in thread "main" java.lang.NoSuchMethodError: joptsimple/OptionParser.allowsUnrecognizedOptions()V
at org.apache.toree.boot.CommandLineOptions.(CommandLineOptions.scala:31)
at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:25)