amplab/keystone

Simplifying robust end-to-end machine learning on Apache Spark.

keystone's Issues

Lemmatization

May be unnecessary for Release 0.1

TIMIT Features -> Solve

No public features - but we can achieve the solve result!

Grayscale

With proper NTSC standard weights

Linear Solver/Mapper

First cut may or may not do label centering.

HMM

Used for speech pipeline. May be unncessary for release 0.1.

Standardize main() format and use scopt

Pipelines executed from the command line should have a uniform CLI. We should use scopt for argument parsing and be consistent with option names across pipelines.

POS Tagging

May be unnecessary for Release 0.1

Signed Hellinger Mapper

See: https://github.com/etrain/pipelines/blob/fishertesting2/src/main/scala/pipelines/FVVOC2007.scala#L128

GMM

Should fit a GMM on a sample of 1m features locally - call out to VGG GMM library with JNI.

HOG Featurizer

mAP

Mean average precision.

Fisher Vector

Should call VGG fisher vector C++ code via JNI.

ImageNet Pipeline

Should have accuracy >60% with 4k features.

LCS

Local Color Statistic

Create Keystone Repo

That is this repo!

BSD Licensed. Permissions, labels, and milestones set up.

MFCCs

Multi-frequency Cepstral Coefficients - a common set of speech features.

Random Features

Should support gaussian and cauchy random maps. Useful if there's a lazy interface as well.

Pooling

CIFAR Pipelines from AMP Camp 5

These should also run single node

1. Pixel version
2. Random Patch/Convolutions Version

Contrast Normalization

Feature Normalization

Subtract feature means and optionally divide by SD.

Tokenization

Standard VOC Pipeline

VOC Pipeline with 59% MAP

Requires

1. SIFT
2. PCA
3. GMM
4. FisherVector

As well as a dataloader for VOC2007.

ZCA Whitening

NER

Named Entity Recognition. May be unnecessary for 0.1.

Text Classification Pipeline

Stop-word removal -> N-Grams -> TFIDF -> Naive Bayes

Should work on 20 Newsgroups and RCV1.

Max Classifier

Image Scaling

FFT Transformer

Whether this belongs in stats or elsewhere is certainly up for debate.

VOC Data Loader

N-Grams

C build script

Create a Makefile for the C components, and integrate with sbt.

SIFT

Should call VLFeat dense SIFT via JNI.

Clean up launch scripts

Bash scripts should be put in a uniform format, and we should settle on some common spark options.

e.g. setting eventLogger to "on" and locality.wait to some reasonable default.

We should also have instructions about how to launch a cluster with spark-ec2 scripts and execute pipelines with this repo.

MNIST Pipeline

Simplest image pipeline - should run single-node with no issues.

Kneser-Ney Language Model Pipeline

Zongheng can fill in details - I think this is a stretch for release 0.1, so I'm not going to assign a milestone.

PCA

Transformer should downsample with PCA matrix.

Add Jenkins Build to Pipelines Repo

Jenkins should build the repo, and run all unit tests. It should also automatically build/test PRs.

KMeans

Call into MLlib

Event Extraction Pipeline

Ideally on the ACE2005 dataset.

Language Model Pipeline

Implemented with Stupid Backoff. Should have a small demo that shows what the language model can do - e.g. evaluate P(text comes from modeled langague).

Accuracy

Compute accuracy of a classifier.

Daisy Featurizer

TIMIT Full Pipeline

Including featurization. Stretch goal - no milestone.

LDA

Linear Discriminant Analysis - as opposed to the topic modeling kind.

Windowing

Clean up build script

First commit contains a reasonable clean build script.

Linear Rectifier

Convolver

Sampling

Downsample a collection.