nasa-jpl-memex/memex-explorer

Viewers for statistics and dashboarding of Domain Search Engine data

memex-explorer's Issues

Create canned demo for testing, e.g. searching for cute cat images

db_add_crawl bug

This happened when i started a nutch crawl that had no data model:

Traceback (most recent call last):
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask_restful/__init__.py", line 262, in error_router
    return original_handler(e)
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask_restful/__init__.py", line 262, in error_router
    return original_handler(e)
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/brittainchristopherhard/anaconda/envs/memex/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/brittainchristopherhard/Documents/memex-explorer/app/views.py", line 162, in add_crawl
    crawl = db_add_crawl(project, form, seed_filename)
  File "/Users/brittainchristopherhard/Documents/memex-explorer/app/db_api.py", line 73, in db_add_crawl
    data_model_id=form.data_model.data.id,
AttributeError: 'NoneType' object has no attribute 'id'

@kdodia, @chdoig

Move #TODOs to GH issues

Create dashboards overview page

Add basic ache crawler test

Check that an explorer install ache runs and crawls correctly. Will need to set up dummy web service to check against.

More responsive, indicative interface to start/stop crawls

Add user accounts

Improve coverage on database interaction tests

Proofread, make docs current

Backbone Integration

So there are a lot of ways this application can be streamlined through the use of Backbone. Setting up projects, for example, can be done on one page instead of on multiple pages that require multiple page refreshes.

It might be worthwhile to discuss (1) the extent to which we want Backbone to supplant Flask in areas such as project creation and (2) areas of the project that could benefit from the inclusion of Backbone.

Just something to discuss for the near future.

Refactor Image Space

After talking with @chdoig, we decided to

• Remove the Image Space model
• Adapt the crawl <--> image_space relation to crawl <--> image

Images will be stored in a central directory, indexed by ID. Each image should have a parent project and at least one linked crawl.

Clicking on the "Image Space" application in a particular project should return all corresponding images.

Add ability to upload Data Model to Register Crawl page

And be able to select data model as well.

Clip long text fields to sane length in Jinja

alt tag (tooltips) for all buttons

Turn app buttons into lists of current apps, + button to add new

Add edit and remove functionality in the db_api

(For crawls and projects.)

Rename Data Models to Crawl Models

Remove Data Explorer (for the time being)

Create dropdown of current projects in header nav bar

Add HTTP Auth to deploy

Remove/reduce exposure to Data Sources

Make functioning project/crawl/dashboard delete views

We need a way to get rid of these things.

Create "About" page

Tasks:

• Link "Learn More" button to About page

Link source button to GitHub repository

Flash message is floating under sidebar

Enable image dump from Nutch

Add basic nutch testing

Check that an explorer install nutch runs and crawls correctly. Will need to set up dummy web service to check against.

Make sidebar icons more like project dropdown

Fix project tabs stacking on index page.

Crawl models should be per project

Slugify URL components

Add tests to continuous build system

Either TravisCI or BinStar Build

Push private resources to S3 to ease deploy

Add link to gh-pages for Docs

Should link to http://continuumio.github.io/memex-explorer/

Tasks:

• CSS, static views are not loading properly on GitHub Pages

  thanks @quasiben!

Linearize tabbed interface ("Description, Summary Statistics, etc.")

Add upload picture to compare in image_space

Create an Index table/model

Delete project behavior

Update the delete project behavior so that, when you delete a project, all the resources of that project, move to your home directory. This involves changing all the project_id to the project with project.name=="Home" and just removing that project from the project table.

Get new integrated resources.zip file into server

Dashboard take too long start

We are currently not displaying any plots with frontierpages.csv data, but we still depend on that file existing and having data for the dashboard page to display. The frontierpages file takes a long time to be generated (~10min, on a test). I tried removing the frontierpages.csv dependency on the code, but was still getting an error.

The ultimate solution probably involves digging into the ACHE code and providing a better output, a streaming API instead of a bulk write to files every X seconds. For now remove frontier pages dependency.

Responsive design?

The application looks odd when someone looks at it on a small screen. It's not really built for "Responsive Design" ©.

So, do we want to make it so it looks good on everything? Phones, tablets, etc? This shouldn't be too difficult to do, but it will take me some time.

A good way to look at this is with the web inspector in chrome (command + alt + j, then click the button that looks like a phone).

Let me know what you guys think. Do we want our customers to be able to use this thing on tablets and/or smartphones?

image space table link to image compare not working

Error message:

  File "/Users/cdoig/anaconda/envs/foo2/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/cdoig/work/test_memex/memex-explorer/app/views.py", line 417, in compare
    img = get_image(image_name)
TypeError: get_image() takes exactly 2 arguments (1 given)

Include sidebar in missing templates

Add padding to "app" buttons

Integrate nutch dump images to app

Install not complete

When I followed through the install today, I get the following error upon visiting the entry page:

http://0.0.0.0:5000/
 * Running on http://0.0.0.0:5000/
 * Restarting with reloader
http://0.0.0.0:5000/
127.0.0.1 - - [07/Nov/2014 08:56:51] "GET / HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/aterrel/workspace/apps/memex/memex-viewer/app/views.py", line 64, in index
    return render_template('index.html')
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/templating.py", line 126, in render_template
    ctx.app.update_template_context(context)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/flask/app.py", line 716, in update_template_context
    context.update(func())
  File "/Users/aterrel/workspace/apps/memex/memex-viewer/app/views.py", line 41, in inject_crawls
    crawls = Crawl.query.all()
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2320, in all
    return list(self)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
    return self._execute_and_instances(context)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception
    exc_info
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/Users/aterrel/workspace/apps/anaconda/py3/envs/memex-viewer/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (OperationalError) no such table: crawl u'SELECT crawl.id AS crawl_id, crawl.name AS crawl_name, crawl.endpoint AS crawl_endpoint, crawl.description AS crawl_description \nFROM crawl' ()
127.0.0.1 - - [07/Nov/2014 08:56:51] "GET /?__debugger__=yes&cmd=resource&f=style.css HTTP/1.1" 200 -
127.0.0.1 - - [07/Nov/2014 08:56:51] "GET /?__debugger__=yes&cmd=resource&f=jquery.js HTTP/1.1" 200 -
127.0.0.1 - - [07/Nov/2014 08:56:51] "GET /?__debugger__=yes&cmd=resource&f=debugger.js HTTP/1.1" 200 -
127.0.0.1 - - [07/Nov/2014 08:56:51] "GET /?__debugger__=yes&cmd=resource&f=ubuntu.ttf HTTP/1.1" 200 -
127.0.0.1 - - [07/Nov/2014 08:56:51] "GET /?__debugger__=yes&cmd=resource&f=console.png HTTP/1.1" 200 -
127.0.0.1 - - [07/Nov/2014 08:56:51] "GET /?__debugger__=yes&cmd=resource&f=source.png HTTP/1.1" 200 -
127.0.0.1 - - [07/Nov/2014 08:56:52] "GET /?__debugger__=yes&cmd=resource&f=console.png HTTP/1.1" 200 -

Nutch Image Dump Integration

Add Summary Statistics

Ideas:

• How long the crawl has been running
• How many pages have been fetched so far
• Average harvest rate

Add build ACHE model functionality

Providing a folder of positive examples and a folder with negative examples, create an ACHE page classifier model in the app.

Add front-end testing of page loading, redirections, and html