data-processing Goto Github PK

akka-streams-example

bloop

A hot bloop for your productivity

cassovary

Cassovary is a simple big graph processing library for the JVM

cdk

Cloudera Development Kit

crawler4j

Open Source Web Crawler for Java

crunch

Crunch is an Apache TLP now, and lives at http://crunch.apache.org/

dask

Task scheduling and blocked algorithms for parallel processing

dataduct

DataPipeline for humans.

disque

Disque is a distributed message broker

dpark

Python clone of Spark, a MapReduce alike framework in Python

druid

Real²time Exploratory Analytics on Large Datasets

emr-bootstrap-actions

This repository hold the Amazon Elastic MapReduce sample bootstrap actions

exelixi

Exelixi is a distributed framework based on Apache Mesos, mostly implemented in Python using gevent for high-performance concurrency. It is intended to run cluster computing jobs (partitioned batch jobs, which include some messaging) in pure Python. By default, it runs genetic algorithms at scale.

faust

Python Stream Processing

fireant

Data analysis and reporting tool for quick access to custom charts and tables in Jupyter Notebooks and in the shell.

flink

Mirror of Apache Flink

fluid

gain

Web crawling framework based on asyncio for everyone.

googlescraper

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, Baidu and others) by using proxies (socks4/5, http proxy) and with many different IP's, including asynchronous networking support (very fast).

grill

grpc-java

The Java gRPC implementation. HTTP/2 based RPC

hibench

HiBench is a Hadoop benchmark suite.

hydra

Hydra is a framework for elegantly configuring complex applications

impatient

source examples to support the "Cascading for the Impatient" blog post series

incubator-samza

Mirror of Apache Samza

kafka-embedded

Runs embedded, in-memory Apache Kafka instances. Helpful for integration testing.

kafka-manager

A tool for managing Apache Kafka.

kafka-spark-consumer

kafka-storm-starter

Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+, while using Apache Avro as the data serialization format.

kangaroo

Hadoop utilities for Kafka