metasyn/dreda

dimensionality reduced exploratory data analysis

DREDA

dimensionality reduced exploratory data analysis http://metasyn.pw/dreda/

What

Explore high dimensionality data via reduction and exploration using Three.js

Why

Sometimes you have too many fields, features, columns... dimensions, to your data.

Wikipedia:

In machine learning and statistics, dimensionality reduction or dimension reduction is the process of reducing the number of random variables under consideration, and can be divided into feature selection and feature extraction.

If you reduce your data down, say via SVD or t-SNE, to three dimensions, you can visualize it with this tool. Currently Dreda supports 4 dimensions (x,y,z, and color). Color assignments are chosen randomly.

I had been doing some visualizations in iPython and they left much to be desired. The data input here is just the ouput of a pandas data frame to_json() - a JSON object with an inner object for each column - index and value pairs in each inner object

E.g.

{
  "x":
  {
    "0":117.0501353217,
    "1":63.4789054268,
    "2":-92.4110611211
  },
  "y":
    {
      "0":-33.8277679817,
      "1":120.3959209587,
      "2":172.3790645372
    },
  "z":
    {
    "0":-17.441790351,
    "1":-34.4737315608,
    "2":-224.6172323059
    },
  "cid":
    {
      "0":4.0,
      "1":1.0,
      "2":4.0
    }
}

How

Clone, run python -m SimpleHTTPSever, and open localhost:8000 in your browser.
Open index.html and click "Plot Data".

Authors

• Xander Johnson

License

WTFPL

dreda's Issues

create upload interface

I think a modal should do:

http://semantic-ui.com/modules/modal.html

round xz grid values to deal with incomplete edge

determine point limit

cluster limit will be related but also bound by color choices

upload sample data file

explorer:initialize scene

create renderer, scene, camera, floor (see through grid and y axis vector)

center camera on selected point

write data wrangle tutorial

deal with wrong format scenario

ui: add semantic ui

add tooltip

https://stemkoski.github.io/Three.js/Mouse-Tooltip.html

determine data max/min and grid size function

we need to write a function that looks through the data on load, and decides on the best maximums for plan edge boundaries.

Right now, the edge (global variable) is hardcoded to 500 - if someone's data has a min of -2 and a max of 10, the xz-grid would dwarf the values.

Since the edge needs to be determined before initializing the scene, I'd think that if someone uploads data, this would need to be called before init to set edge to a reasonable size.

ui: create home/about page

determine point limit

cluster limit will be related but also bound by color choices

explorer: create sphere geometry

for basic point

add keyboard controls for exploring

add reset camera button

standardize colors

setup example from iris.json

more classic example with more distinct clusters

Window resizing bug

If the window is resized the 3D rendering gets stretched or smooshed. I'm going to look into using THREEx.WindowResize to fix this.

add x y z labels and scales

loadData() is slow

I found the load takes over 30 seconds for your sample size of 20000.
I've got it down to ~18 milliseconds and will submit a pull request soon. The problem was using underscore to get the size in your loop; it had to execute that code for all 20000 iterations. My fix simply stores the size in a var before starting the loop. Should I use the group, or submit to your repo? I'm fine with your repo being the main one and ditching the group if you want to.

explorer: create colors

30?