myzmh/hypertools

A Python toolbox for gaining geometric insights into high-dimensional data

"To deal with hyper-planes in a 14 dimensional space, visualize a 3D space and say 'fourteen' very loudly. Everyone does it." - Geoff Hinton

Overview

HyperTools is designed to facilitate dimensionality reduction-based visual explorations of high-dimensional data. The basic pipeline is to feed in a high-dimensional dataset (or a series of high-dimensional datasets) and, in a single function call, reduce the dimensionality of the dataset(s) and create a plot. The package is built atop many familiar friends, including matplotlib, scikit-learn and seaborn. Our package was recently featured on Kaggle's No Free Hunch blog. For a general overview, you may find this talk useful (given as part of the MIND Summer School at Dartmouth).

Try it!

Click the badge to launch a binder instance with example uses:

or

Check the repo of Jupyter notebooks from the HyperTools paper.

Installation

To install the latest stable version run:

pip install hypertools

To install the latest unstable version directly from GitHub, run:

pip install -U git+https://github.com/ContextLab/hypertools.git

Or alternatively, clone the repository to your local machine:

git clone https://github.com/ContextLab/hypertools.git

Then, navigate to the folder and type:

pip install -e .

(These instructions assume that you have pip installed on your system)

NOTE: If you have been using the development version of 0.5.0, please clear your data cache (/Users/yourusername/hypertools_data).

Requirements

• python 2.7, 3.5+
• PPCA>=0.0.2
• scikit-learn>=0.18.1
• pandas>=0.18.0
• seaborn>=0.8.1
• matplotlib>=1.5.1
• scipy>=0.17.1
• numpy>=1.10.4
• future
• requests
• deepdish
• pytest (for development)
• ffmpeg (for saving animations)

If installing from github (instead of pip), you must also install the requirements: pip install -r requirements.txt

Troubleshooting

If you encounter an error related to installing deepdish (hdf5) on a MacOS system, try installing hdf5 directly using homebrew:

$ brew tap homebrew/science
$ brew install hdf5

and then re-start the installation.

Documentation

Check out our readthedocs page for further documentation, complete API details, and additional examples.

Citing

We wrote a short JMLR paper about HyperTools, which you can read here, or you can check out a (longer) preprint here. We also have a repository with example notebooks from the paper here.

Please cite as:

Heusser AC, Ziman K, Owen LLW, Manning JR (2018) HyperTools: A Python toolbox for gaining geometric insights into high-dimensional data. Journal of Machine Learning Research, 18(152): 1--6.

Here is a bibtex formatted reference:

@ARTICLE {, author = {Andrew C. Heusser and Kirsten Ziman and Lucy L. W. Owen and Jeremy R. Manning},
title = {HyperTools: a Python Toolbox for Gaining Geometric Insights into High-Dimensional Data},
journal = {Journal of Machine Learning Research}, year = {2018}, volume = {18}, number = {152}, pages = {1-6}, url = {http://jmlr.org/papers/v18/17-434.html} }

## Contributing

[![Join the chat at https://gitter.im/hypertools/Lobby](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/hypertools/Lobby)

If you'd like to contribute, please first read our [Code of Conduct](https://www.mozilla.org/en-US/about/governance/policies/participation/).

For specific information on how to contribute to the project, please see our [Contributing](https://github.com/ContextLab/hypertools/blob/master/CONTRIBUTING.md) page.
## Testing

[![Build Status](https://travis-ci.org/ContextLab/hypertools.svg?branch=master)](https://travis-ci.org/ContextLab/hypertools)


To test HyperTools, install pytest (`pip install pytest`) and run `pytest` in the HyperTools folder

## Examples

See [here](http://hypertools.readthedocs.io/en/latest/auto_examples/index.html) for more examples.

## Plot

import hypertools as hyp hyp.plot(list_of_arrays, '.', group=list_of_labels)

![Plot example](images/plot.gif)

## Align

import hypertools as hyp hyp.plot(list_of_arrays, align='hyper')

### BEFORE

![Align before example](images/align_before.gif)

### AFTER</center>

![Align after example](images/align_after.gif)


## Cluster

import hypertools as hyp hyp.plot(array, '.', n_clusters=10)

![Cluster Example](images/cluster_example.png)


## Describe

import hypertools as hyp hyp.tools.describe(list_of_arrays, reduce='PCA', max_dims=14)

![Describe Example](images/describe_example.png)