azure-samples/hdinsight-python-storm-wordcount

How to use Python components in an Apache Storm topology on HDInsight

services	platforms	author
hdinsight	java,python	blackmist

hdinsight-python-storm-wordcount

How to use Python components in an Apache Storm topology on HDInsight

This topology uses the Flux framework to define a Storm topology using YAML. The components (spout and bolts) that process the data are written in Python.

This example has been tested with HDInsight 3.6 (Storm 1.1.0).

Prerequisites

• Python 2.7 or higher

• Java JDK 1.8 or higher

• Maven

• (Optional) A local Storm development environment. This is only needed if you want to run the topology locally. For more information, see Setting up a development environment.

How it works

• /resources/topology.yaml - defines what components are in the topology and how data flows between them.

• /multilang/resources - contains the Python components.

• /pom.xml - dependencies and how to build the project.

Build the project

From the root of the project, use the following command:

mvn clean compile package

This command creates a target/WordCount-1.0-SNAPSHOT.jar file.

Run the topology locally

To run the topology locally, use the following command:

storm jar WordCount-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux -l -R /topology.yaml

Once the topology starts, it emits information to the local console similar to the following text:

24302 [Thread-25-sentence-spout-executor[4 4]] INFO  o.a.s.s.ShellSpout - ShellLog pid:2436, name:sentence-spout Emiting the cow jumped over the moon
24302 [Thread-30] INFO  o.a.s.t.ShellBolt - ShellLog pid:2438, name:splitter-bolt Emitting the
24302 [Thread-28] INFO  o.a.s.t.ShellBolt - ShellLog pid:2437, name:counter-bolt Emitting years:160
24302 [Thread-17-log-executor[3 3]] INFO  o.a.s.f.w.b.LogInfoBolt - {word=the, count=599}
24303 [Thread-17-log-executor[3 3]] INFO  o.a.s.f.w.b.LogInfoBolt - {word=seven, count=302}
24303 [Thread-17-log-executor[3 3]] INFO  o.a.s.f.w.b.LogInfoBolt - {word=dwarfs, count=143}
24303 [Thread-25-sentence-spout-executor[4 4]] INFO  o.a.s.s.ShellSpout - ShellLog pid:2436, name:sentence-spout Emiting the cow jumped over the moon
24303 [Thread-30] INFO  o.a.s.t.ShellBolt - ShellLog pid:2438, name:splitter-bolt Emitting cow
^C24303 [Thread-17-log-executor[3 3]] INFO  o.a.s.f.w.b.LogInfoBolt - {word=four, count=160}

Use Ctrl+c to stop the topology.

Run the topology on HDInsight

1. Use the following command to copy the WordCount-1.0-SNAPSHOT.jar file to your Storm on HDInsight cluster:

   scp target\WordCount-1.0-SNAPSHOT.jar [email protected]

   Replace sshuser with the SSH user for your cluster. Replace mycluster with the cluster name.

2. Once the file has been uploaded, connect to the cluster using SSH and use the following command to start the topology on the cluster:

   storm jar WordCount-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux -r -R /topology.yaml

3. You can use the Storm UI to view the topology on the cluster. The Storm UI is located at https://mycluster.azurehdinsight.net/stormui. Replace mycluster with your cluster name.

Once started, a Storm topology runs until stopped (killed.) To stop the topology, use either the storm kill TOPOLOGYNAME command from the command-line (SSH session to a Linux cluster,) or by using the Storm UI, select the topology, and then select the Kill button.

Project code of conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

hdinsight-python-storm-wordcount's Issues

Error: Could not find or load main class org.apache.storm.flux.Flux

I am trying to run this example on mac using storm version 1.2.1 but it generates this error when I try to run the topology : Error: Could not find or load main class org.apache.storm.flux.Flux.

The output I get is as follows:

Running: java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/local/Cellar/storm/1.2.1/libexec -Dstorm.log.dir=/usr/local/Cellar/storm/1.2.1/libexec/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/Cellar/storm/1.2.1/libexec/:/usr/local/Cellar/storm/1.2.1/libexec/lib/:/usr/local/Cellar/storm/1.2.1/libexec/extlib/*:WordCount-1.0-SNAPSHOT.jar:/usr/local/Cellar/storm/1.2.1/libexec/conf:/usr/local/Cellar/storm/1.2.1/libexec/bin -Dstorm.jar=WordCount-1.0-SNAPSHOT.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.flux.Flux -l -R /topology.yaml
Error: Could not find or load main class org.apache.storm.flux.Flux

The pom.xml contains follwing flux dependencies:

<dependency>
           <groupId>org.apache.storm</groupId>
           <artifactId>flux-core</artifactId>
           <version>${storm.version}</version>
</dependency>
       
<dependency>
           <groupId>org.apache.storm</groupId>
           <artifactId>flux-wrappers</artifactId>
           <version>${storm.version}</version>
</dependency>

I have tried using mvn clean install and it generates following output for flux dependencies:
[WARNING] flux-wrappers-1.1.0.jar, flux-core-1.1.0.jar define 3 overlappping classes:
[WARNING] - org.apache.storm.flux.wrappers.spouts.FluxShellSpout
[WARNING] - org.apache.storm.flux.wrappers.bolts.LogInfoBolt
[WARNING] - org.apache.storm.flux.wrappers.bolts.FluxShellBolt

Can anyone please help?

61103 [Thread-18-counter-bolt-executor[2 2]] ERROR o.a.s.util - Async loop died! java.lang.RuntimeException: Error when launching multilang subprocess

Getting this error

61103 [Thread-18-counter-bolt-executor[2 2]] ERROR o.a.s.util - Async loop died!
java.lang.RuntimeException: Error when launching multilang subprocess

at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:94) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.task.ShellBolt.prepare(ShellBolt.java:153) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.daemon.executor$fn__5043$fn__5056.invoke(executor.clj:803) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) [storm-core-1.2.1.jar:1.2.1]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

Caused by: java.io.IOException: Unexpected character (() at position 0.
at org.apache.storm.multilang.JsonSerializer.readMessage(JsonSerializer.java:172) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.multilang.JsonSerializer.connect(JsonSerializer.java:68) ~[storm-core-1.2.1.jar:1.2.1]
at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:90) ~[storm-core-1.2.1.jar:1.2.1]
... 5 more

Can you please help?

TypeError: Object of type 'map' is not JSON serializable

File "resources/splitbolt.py", line 18, in process
storm.emit([word])

Above is causing an error:
TypeError: Object of type 'map' is not JSON serializable

My Python setup is 3.6 with anaconda. Even tried converting word into list before emitting but still ending up with the same error. Any suggestions.

Cannot Import name Counter in countbolt.py

File "countbolt.py", line 4, in
from collections import Counter
ImportError: cannot import name Counter

My python version is 2.7. And the statement "from collections import counter" does not give an error in console. And even if I make a separate python file with only this import it doesn't give the error.

Please help.

Error: Could not find or load main class wordcount.core

Hi, I'm follwing the instructions for the Clojure topology.
After successfully running locally, creating the uberjar, and scp'ing to the head node,
I run:
storm jar wordcount-1.0-SNAPSHOT.jar wordcount.core wordcount
and get:

Running: /usr/lib/jvm/java-7-openjdk-amd64/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.2.4-5/storm -Dstorm.log.dir=/var/log/storm -Djava.library.p.
Error: Could not find or load main class wordcount.core

What am I missing?