deakkon/techdashboard

techdashboard's Issues

Create HTML interface

Create HTML forntend for web app

NER for extracted content

To try:

1. https://github.com/brendano/stanford_corenlp_pywrapper
2. http://spacy.io/

List of Py Interfaces for Stanfrods NER:
http://nlp.stanford.edu/software/corenlp.shtml

Blabla

balbla demo task

Angular App

TO do for Angular App:

1. figure out $scope and passing variables from Flask to angulars scope to pass a) number of topics in model and b) all data for desired model (data need to be returned in a json format)
   Things to return for individual topic:
   a) last N documents from the topic
   b) top 20 keywords from the topic
   c) a bit tricky but nevertheless, "popularity of the topic in a specific period (e.g. bar chart with all identified topics)

Update gensim models with new documents

Update gensim models with new documents every N minutes (hours). Read the create date of LDA model files and get all articles from DB younger than that date. Add new documents to the model and attach each to one of 20 categories in the generated gensim LDA model.

getpath() ofr xpath values

http://lxml.de/api/lxml.etree._ElementTree-class.html#getpath

Implement LXML getpath function to extract xpath values of each node.

Text preprocessing iwth collocations/n-grams

Currently only minimal text preparation exists, based on NLTK stopwords and tokenizer.

TO DO:

• Improve data preparation (single characters tokens, html tags etc.)
  1 http://www.nltk.org/api/nltk.tokenize.html
• Collocation / n-gram tokenization
  1 http://www.nltk.org/book/
  2 http://www.nltk.org/book/ch03.html
  3 http://n-chandra.blogspot.co.uk/2014/06/collocation-extraction-using-nltk.html
  4 http://nlpwp.org/book/chap-ngrams.xhtml
  5 http://www.nltk.org/howto/collocations.html
• ELIMINATE DUPLICATES FROM EXTRACTED CONTENT (E.G. ARTICLE GET
  EXTRACTED MORE THAN ONCE)
• “LEARN” WHEN THE FREQUENCY OF FEED REFRESH RATE TO CALL RSS FEED
  PARSER ON Y AT SPECIFIC INTERVALS

ANGULAR WATCH VARIABLES (FOR HIDE/SHOW FUNCTIONALITY) INSTEAD OF MANUALLY CODING IT IN

Create filtering on extracted NERs

Use angular routes for redirecting inside app

Angular Graphs - implement number of documents for each topic in database.

Web app

Create web app with Flask (no nginx for now) which will show 20 topics and the articles which belong to the topics.

*Flask as web server
1 http://flask.pocoo.org/docs/0.10/quickstart/

*POLYMER OR ANGULAR FOR FORNTEND
1 https://www.polymer-project.org/1.0/
2 https://material.angularjs.org/latest/
3 dummy text for this task

Create HTML UI for tetch dashboard

To show:

20 topics
for each topic top N keywords
latest M articles which fit best to that category

LXML element to HTML -

IMPORTANT

parsing the text using LXML and its iterative proces has major issues (e.g. skipping all elements in the text). try to convert LXML element to HTML and use BeautifulSoup to extract text.

http://lxml.de/lxmlhtml.html

Create word cloud of extracted NERs

define new ration masure which takes in to account number of xpath elements for a query to normalize the extraction clusters.

//*/div will extract all div elements in the document while a more specific path will be the actual conent path. The idea is to include number of elements which corespond to an xpath to the evaluation of the ratio. Question is how (e.g. some logaritmic or similar range?)

Extract title of document

Based on created topic model assigne most likely topic to each document after extracting

Unify MySQL connections accross framework - use permanent connection

Currently, a few objects are creating their own MySQL objects and are executing MySQL queries wihch can lead to diminished preformances. try to fix that and exclude incremental inclusions of libraries which are in common to specific parts of the API.