wehlutyk/brainscopypaste

Analysis of mutation of quotes on the web

brainscopypaste's Issues

Find a proper visualization for susceptibilities

It's smarter to have susceptibilities as a curve, and feature density as the colors.

Document `mine`

Look at the distribution of features in successive timebags 1, 2, 3...

For the clusters that last at least N timebags. (Or for all clusters, if #25 shows that the duration of a cluster does not depend on its linguistic initial point.)

If there is an evolution, the distributions should gradually go towards more spiked ones (e.g. more words learned earlier, less words learned later).

Licensing

1. Treetaggerwrapper in under GPL, can I release my stuff under any license?
2. Choose license with http://wiki.civiccommons.org/Choosing_a_License

Update matplotlibrc

Do timebag evolution over [0, n-1]

Add word frequency as a feature

See also wehlutyk/brainscopypaste-paper#7

Weird spike

In http://nbviewer.ipython.org/github/wehlutyk/brainscopypaste/blob/master/features_timebags_evolution_recursive_shifting.ipynb , in the MNSyllables plot, there's a growing spike at 4 syllables. What's that? It could also be related to a similar spike in the AoA plots.

Do timebag evolution based on product from substitution mining

Recompute and reinterpret results

After #21 and #22 are done.

Sphinx toctree warning

The error is:

WARNING: toctree contains reference to nonexisting document u'reference/analyze.args.GroupAnalysisArgs.title'

Also decide on the options for automodule (etc.) to get all members of all classes, but not inherited members.

Document `visualize`

Document `analyze`

Great refactoring

I discovered so many new tools since this started. MongoDB, SQLAlchemy joblib, logging, pandas, seaborn to name a few. So there's a great refactoring in sight. Basically, there will be three levels:

Database

Models in SQLAlchemy. This will also solve #33. Basically it eases everything.

Flow

Mining operations are to be generalized. You can mine for substitutions, for evolution of timebags, for other things. One unique command and subcommand brains mine {substitutions | evolution | ...}. Each operation has prerequisites that go through joblib, are done before if needed (with confirmation), a kind of parallelized make. This includes #32.

Creating a new mining operation must be straightforward, because that's the way to go when a new question appears. Maybe also allow for quick immediate tweaking of an analysis by adding variation-ids to analyses?

Viz

In notebooks, and nowhere else. Storify and order them too. Graphs read mined stuff, writing their prerequisites at the beginning.

Check all from __future__ import division

Rename all the online projects to brains-copy-paste

Rename `langid` nltk package to `crubadan`

The language detector relies on this. See here.

IPythonize

Graphing/visualizing (not mining) can be put in ipython with nice views and interfaces.

All the figures from the paper should be included in this.

Update the sphinx doc with the notebooks.

Change licence to permissive

Document `linguistics`

Reread sphinx docs and update reference

Check PEP8 for all files

Will go with the documentation

Rework dependencies (seaborn, pandas, update all)

And rebuild the whole dependency list. See also #19.

Harmonise ids

For Clusters they're ints, in QtStrings in substitutions they seem to be floats, and in Quotes in Clusters they're strings!

Predict evolution of feature distribution by combining susceptibility and feature variation upon substitution

Look at examples

Print out reasonably sized clusters with all their quotes, about 5 clusters per page, on 5-10 pages.

And wee what there is to see.

Wheel install only works with exact same library versions (e.g. libc)

Use feature-to-feature common words for correlation notebook

Set susceptibility to NaN if never substituted

Do timebag evolution with shifting clusters to see if effect changes

Plan documentation

Check what's documented in the code and what isn't. Make a list of things that still need documenting.

Set all the timebagsy models to use timebags of fixed duration

Instead of a duration depending on the cluster duration

Document root scripts

List fact-checks

Many facts stated in the paper need checking. List them. Check them.

Does the distribution of word features in timebag 1 depend (predict) the duration of a cluster?

On python2.7, argparse can't be found in pypi (because included in stdlib?)

So pip install -r requirements.txt fails.

Roll back to directed unweighed FA network

Harmonise lemmatizing policy

PCA on features

Useful?
As an answer to what?