openannotation/annotator-store

A backend store for the Annotator

Annotator Store

This is a backend store for Annotator.

The functionality can roughly be separated in two parts:

1. An abstraction layer wrapping Elasticsearch, to easily manage annotation storage. It features authorization to filter search results according to their permission settings.
2. A Flask blueprint for a web server that exposes an HTTP API to the annotation storage. To use this functionality, build this package with the [flask] option.

Getting going

You'll need a recent version of Python (Python 2 >=2.6 or Python 3 >=3.3) and ElasticSearch (>=1.0.0) installed.

The quickest way to get going requires the pip and virtualenv tools (easy_install virtualenv will get them both). Run the following in the repository root:

virtualenv pyenv
source pyenv/bin/activate
pip install -e .[flask]
cp annotator.cfg.example annotator.cfg
python run.py

You should see something like:

* Running on http://127.0.0.1:5000/
* Restarting with reloader...

If you wish to customize the configuration of the Annotator Store, make your changes to annotator.cfg or dive into run.py.

Additionally, the HOST and PORT environment variables override the default socket binding of address 127.0.0.1 and port 5000.

Store API

The Store API is designed to be compatible with the Annotator. The annotation store, a JSON-speaking REST API, will be mounted at /api by default. See the Annotator documentation for details.

Running tests

We use nosetests to run tests. You can just pip install -e .[testing], ensure ElasticSearch is running, and then:

$ nosetests
......................................................................................
----------------------------------------------------------------------
Ran 86 tests in 19.171s

OK

Alternatively (and preferably), you should install Tox, and then run tox. This will run the tests against multiple versions of Python (if you have them installed).

Please open an issue if you find that the tests don't all pass on your machine, making sure to include the output of pip freeze.

annotator-store's Issues

Tests fail

Hi, I'm trying to run the new store locally. After running through the installation, I get the following test failures:

E....E.........FFFF......................E.............
======================================================================
ERROR: Failure: ImportError (cannot import name db)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/pyenv/lib/python2.6/site-packages/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/pyenv/lib/python2.6/site-packages/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/pyenv/lib/python2.6/site-packages/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/annotator/model/__init__.py", line 4, in <module>
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/annotator/model/consumer.py", line 4, in <module>
ImportError: cannot import name db

======================================================================
ERROR: tests.test_annotation.TestAnnotation.test_save
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/tests/test_annotation.py", line 24, in test_save
    assert_in('id', a)
NameError: global name 'assert_in' is not defined
-------------------- >> begin captured logging << --------------------
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "PUT /annotator_test HTTP/1.1" 200 31
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "PUT /annotator_test/annotation/_mapping HTTP/1.1" 200 31
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "POST /annotator_test/annotation HTTP/1.1" 201 102
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: tests.test_store.TestStore.test_search
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/tests/test_store.py", line 250, in test_search
    assert_in(res['rows'][0]['id'], [anno.id, anno2.id])
NameError: global name 'assert_in' is not defined
-------------------- >> begin captured logging << --------------------
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "PUT /annotator_test HTTP/1.1" 200 31
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "PUT /annotator_test/annotation/_mapping HTTP/1.1" 200 31
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "POST /annotator_test/annotation HTTP/1.1" 201 102
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "POST /annotator_test/annotation HTTP/1.1" 201 102
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "POST /annotator_test/annotation HTTP/1.1" 201 102
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "POST /_all/_refresh HTTP/1.1" 200 61
pyes.urllib3.connectionpool: DEBUG: "GET /_cluster/health HTTP/1.1" 200 228
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "GET /annotator_test/annotation/_search HTTP/1.1" 200 1202
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "GET /annotator_test/annotation/_count HTTP/1.1" 200 59
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "GET /annotator_test/annotation/_search HTTP/1.1" 200 477
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "GET /annotator_test/annotation/_count HTTP/1.1" 200 59
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "GET /annotator_test/annotation/_search HTTP/1.1" 200 846
pyes: DEBUG: Connecting to 127.0.0.1:9200
pyes: INFO: Starting new HTTP connection (1): 127.0.0.1
pyes.urllib3.connectionpool: DEBUG: "GET /annotator_test/annotation/_count HTTP/1.1" 200 59
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: tests.test_auth.TestAuthenticator.test_verify_request
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/tests/test_auth.py", line 89, in test_verify_request
    assert self.auth.verify_request(request), "request should have been verified"
AssertionError: request should have been verified

======================================================================
FAIL: tests.test_auth.TestAuthenticator.test_verify_request_mixedcase_headers
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/tests/test_auth.py", line 101, in test_verify_request_mixedcase_headers
    assert self.auth.verify_request(request), "request with mixed-case headers should have been verified"
AssertionError: request with mixed-case headers should have been verified

======================================================================
FAIL: tests.test_auth.TestAuthenticator.test_verify_token
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/tests/test_auth.py", line 84, in test_verify_token
    assert self.auth.verify_token('Consumer', tok, 'alice', issue_time), "token should have been verified"
AssertionError: token should have been verified

======================================================================
FAIL: tests.test_auth.test_verify_token
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/home/ewald/programming/texas-sprint/okfn-annotator-store/tests/test_auth.py", line 38, in test_verify_token
    assert auth.verify_token(MockConsumer, tok, 'alice', issue_time), "token should have been verified"
AssertionError: token should have been verified

----------------------------------------------------------------------
Ran 55 tests in 17.301s

FAILED (errors=3, failures=4)

The output from pip freeze gives

Flask==0.8
Flask-SQLAlchemy==0.15
Flask-WTF==0.5.2
Jinja==1.2
Jinja2==2.6
SQLAlchemy==0.7.4
WTForms==0.6.3
Werkzeug==0.8.3
-e git+https://github.com/okfn/annotator-store.git@53aebc3e5b77c2ce2ba292e22f72eaff8a1bc589#egg=annotator-v0.6.1
distribute==0.6.10
importlib==1.0.2
iso8601==0.1.4
mock==0.7.2
nose==1.0.0
pyes==0.16.0
thrift==0.8.0
wsgiref==0.1.2

I'm running Ubuntu 10.10 32bit with python 2.6.6

AnnotateIt bookmarklet should be upgradable over the air

At the moment, upgrading the bookmarklets of extant users is going to require updating the deployed version of Annotator on http://assets.annotateit.org to display an alert() box telling users to visit http://annotateit.org and upgrade their bookmarklet.

What we should do is have the bookmarklet load and execute a URL such as http://assets.annotateit.org/bookmarklet/bootstrap.js which in turn loads the latest version of Annotator.

Restrict changes to account_id attribute of annotations on update

At the moment people could theoretically change the account_id of an annotation on update re-assigning it to another account.

Should we not allow this at all?

Should we limit it only to an annotation's admin?

Always return CORS headers regardless of the response status

Currently CORS headers are only returned if the status is less than 300. This means the browsers XMLHttpRequest object has not access to the response body or status if an error occurs.

Removing this line resolves the issue.

No anonymous read-only access to API

Mockup homepage for annotateit.org

Navigation

• About

• Sign up

• Login

  Content

• What annotateit.org provides as a service.

  • Storage of annotations on user's website with Annotator JavaScript
  • Bookmarklet for general use

• Information on the annotator? Should this be included here?

• Sign Up box?

• Setting up the Annotator on personal site.

Perhaps fit the whole site into a single page with navigation at the top that scrolls to the relevant areas.

Decide what on what to switch to from CouchDB (either Relational DB or Elasticsearch)

We should switch from CouchDB because of its painful search functionality (we could use Couch + ES but why bother when ES does storage as well ...)

Options:

• RDBMS
  • (+) tried and tested, will integrate with other backends, SQLAlchemy is excellent and robust, migrations
  • (-) poor full text search. Not JSON oriented and have to hack in various ways to support arbitrary data.
• ElasticSearch We need arbitrary search anyway and elasticsearch does most of what couchdb does (see rgrp work in hypernotes)
  • (+) supports json out of box. Good FTS
  • (-) Less well known, harder to deploy and test etc

Restrict changes to annotation user field

Currently allow anyone with update permission to change user field.

Should we limit this to those with admin permission.

[discuss] Move to pure SQL backed-store

pyes.exceptions.NoServerAvailable

When I run "python run.py", the message shows:

No handlers could be found for logger "pyes" Traceback (most recent call last): File "run.py", line 72, in main() File "run.py", line 37, in main annotation.Annotation.create_all() File "/var/www/okfn-annotator-store-9bb70a6/annotator/elasticsearch.py", line 72, in create_all cls.es.conn.create_index_if_missing(cls.es.index) File "/usr/local/lib/python2.7/dist-packages/pyes-0.16.0-py2.7.egg/pyes/es.py", line 298, in create_index_if_missing return self.create_index(index, settings) File "/usr/local/lib/python2.7/dist-packages/pyes-0.16.0-py2.7.egg/pyes/es.py", line 289, in create_index return self._send_request('PUT', index, settings) File "/usr/local/lib/python2.7/dist-packages/pyes-0.16.0-py2.7.egg/pyes/es.py", line 208, in _send_request response = self.connection.execute(request) File "/usr/local/lib/python2.7/dist-packages/pyes-0.16.0-py2.7.egg/pyes/connection.py", line 174, in _client_call raise NoServerAvailable pyes.exceptions.NoServerAvailable

Deploy staging.annotateit.org for development use

Deploy staging.annotateit.org for development use

401 Permission Denied response when updating an annotation created by another user

The following annotation was received from the store:

{
  "account_id": "39fc339cf058bd22176771b3e3029975", 
  "created": "2011-06-29T15:14:37.953645", 
  "text": "This is a new annotation", 
  "_rev": "1-defd8815e8b813a720c25744942521e7", 
  "tags": [
    "annotated"
  ], 
  "uri": "http://www.alistapart.com/articles/humantohuman/", 
  "ranges": [
    {
      "start": "/div[2]/div/div[2]/p", 
      "end": "/div[2]/div/div[2]/p", 
      "startOffset": 0, 
      "endOffset": 149
    }
  ], 
  "annotator_schema_version": "v1.0", 
  "user": {
    "id": "annotator-demo", 
    "name": "Annotator Demo"
  }, 
  "quote": "It\u2019s not new to say that we now live in an age in which survival in business depends on your ability to communicate effectively through the internet.", 
  "_id": "39fc339cf058bd22176771b3e320d59d", 
  "type": "Annotation", 
  "id": "39fc339cf058bd22176771b3e320d59d", 
  "permissions": {
    "read": [], 
    "admin": [
      "annotator-demo"
    ], 
    "update": [], 
    "delete": [
      "annotator-demo"
    ]
  }
}

The following request was made to update it:

PUT /api/annotations/39fc339cf058bd22176771b3e320d59d HTTP/1.1
Host: uat.annotateit.org
Connection: keep-alive
Referer: http://www.alistapart.com/articles/humantohuman/
Content-Length: 798
Origin: http://www.alistapart.com
X-Annotator-Auth-Token: xxxxx
X-Requested-With: XMLHttpRequest
X-Annotator-User-Id: demo
X-Annotator-Account-Id: 39fc339cf058bd22176771b3e320d53d
Content-Type: application/json; charset=UTF-8
Accept: application/json, text/javascript, */*; q=0.01

{"account_id":"39fc339cf058bd22176771b3e3029975","created":"2011-06-29T15:14:37.953645","text":"This is an edited annotation","_rev":"1-defd8815e8b813a720c25744942521e7","tags":["annotated"],"uri":"http://www.alistapart.com/articles/humantohuman/","ranges":[{"start":"/div[2]/div/div[2]/p","startOffset":0,"end":"/div[2]/div/div[2]/p","endOffset":149}],"annotator_schema_version":"v1.0","user":{"id":"annotator-demo","name":"Annotator Demo"},"quote":"It’s not new to say that we now live in an age in which survival in business depends on your ability to communicate effectively through the internet.","_id":"39fc339cf058bd22176771b3e320d59d","type":"Annotation","id":"39fc339cf058bd22176771b3e320d59d","permissions":{"read":[],"admin":["annotator-demo"],"update":[],"delete":["annotator-demo"]}}

The following response was returned:

HTTP/1.1 401 UNAUTHORIZED
Server: nginx/0.7.67
Date: Wed, 29 Jun 2011 15:22:42 GMT
Content-Type: application/json
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With, Content-Type, X-Annotator-Account-Id, X-Annotator-User-Id, X-Annotator-Auth-Token-Valid-Until, X-Annotator-Auth-Token
Access-Control-Expose-Headers: Location
Access-Control-Allow-Methods: GET, POST, PUT, DELETE
Access-Control-Max-Age: 86400
Content-Length: 71

"Could not authorise request (permissions change). No update performed"

GET /annotations only returns 20 annotations

The description of the Annotator API suggests that all annotations should be returned when GETting /annotations. This doesn't seem to be the case. Rather, it appears that only twenty results are returned.

authorize() needs to match the user id in the header against the user id on the annotation

Currently it matches the user id (a string) sent in the X-Annotator-UserId header against the annotation.user property.

However the annotation.user is an object with an id property and so the match fails. The annotation object should look like this:

annotation = {
  user: {
    id: 'aron',
    name: 'Aron'
  }
}

This bug arose due to a recent change in the way the user is stored on the annotation. See: openannotation/annotator@9f6743d.

WUI for Account creation

HTTP/1.0 404 NOT FOUND

hi Team, Could you pls help - Trying to run the below:

The annotation store, a JSON-speaking REST API, will be mounted at http://localhost:5000/api/annotations. You can test this by running:

$ curl -i http://localhost:5000/api/annotations

Error msg -
127.0.0.1 - - [01/Nov/2011 16:40:49] "GET /api/annotations HTTP/1.1" 404 -
HTTP/1.0 404 NOT FOUND
Content-Type: text/html
Content-Length: 238
Server: Werkzeug/0.8.1 Python/2.7.2+
Date: Tue, 01 Nov 2011 23:40:49 GMT

<title>404 Not Found</title>

Not Found

The requested URL was not found on the server.

If you entered the URL manually please check your spelling and try again.

Regards,
Papitha

nosetest failed (errors=10, failures=8)

=== PIP FREEZE ===
CouchDB==0.8
Flask==0.6.1
Flask-WTF==0.5.2
Jinja2==2.6
WTForms==0.6.3
Werkzeug==0.8.2
iso8601==0.1.4
nose==1.0.0
wsgiref==0.1.2

=== TEST RESULTS ===

....FFF...................EEEEEEEEEEFFFFF

ERROR: tests.annotator.test_store.TestStore.test_cors_preflight

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_create

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_delete

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_delete_notfound

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_index

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_read

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_read_notfound

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_search

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_update

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_update_notfound

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/util.py", line 478, in try_run
return func()
File "/var/www/annotator/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

FAIL: tests.annotator.test_auth.TestAuth.test_verify_request

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/var/www/annotator/tests/annotator/test_auth.py", line 68, in test_verify_request
assert auth.verify_request(request), "request should have been verified"
AssertionError: request should have been verified

FAIL: tests.annotator.test_auth.TestAuth.test_verify_request_mixedcase_headers

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/var/www/annotator/tests/annotator/test_auth.py", line 80, in test_verify_request_mixedcase_headers
assert auth.verify_request(request), "request with mixed-case headers should have been verified"
AssertionError: request with mixed-case headers should have been verified

FAIL: tests.annotator.test_auth.TestAuth.test_verify_token

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/var/www/annotator/tests/annotator/test_auth.py", line 53, in test_verify_token
assert auth.verify_token(tok, 'testAccount', 'alice', expiryTime), "token should have been verified"
AssertionError: token should have been verified

FAIL: tests.annotator.test_store.TestStoreAuth.test_get_allowed

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/var/www/annotator/tests/annotator/test_store.py", line 166, in test_get_allowed
assert response.status_code == 200, "GET should be allowed"
AssertionError: GET should be allowed

FAIL: tests.annotator.test_store.TestStoreAuth.test_reject_post_request

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/var/www/annotator/tests/annotator/test_store.py", line 172, in test_reject_post_request
assert response.status_code == 401, "response should be 401 NOT AUTHORIZED"
AssertionError: response should be 401 NOT AUTHORIZED

FAIL: tests.annotator.test_store.TestStoreAuthz.test_read

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/var/www/annotator/tests/annotator/test_store.py", line 202, in test_read
assert response.status_code == 401, response.status_code
AssertionError: 404

FAIL: tests.annotator.test_store.TestStoreAuthz.test_update

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/var/www/annotator/tests/annotator/test_store.py", line 216, in test_update
assert response.status_code == 401, response.status_code
AssertionError: 404

FAIL: tests.annotator.test_store.TestStoreAuthz.test_update_change_permissions_not_allowed

Traceback (most recent call last):
File "/var/www/annotator/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/var/www/annotator/tests/annotator/test_store.py", line 235, in test_update_change_permissions_not_allowed
assert response.status_code == 401, response.status_code
AssertionError: 404

---

Ran 41 tests in 4.976s

FAILED (errors=10, failures=8)

Stored annotations not immediately available

ElasticSearch doesn't make indexed documents available instantly, instead waiting for a database refresh operation (typically every 30s).

Because I'm an idiot, and am using a search index as a persistence layer, we need to work around this.

Ignore created field on updates

Ignore incoming created attribute on updates as created should not be changed.

Is it worth it?

producing error after command python run.py

ERROR:root:creating index annotator
ERROR:root:creating index annotator
ERROR:root:creating index annotator
ERROR:root:creating index annotator

What could be the possible reason?

Thanks in advance

How to store annotations in a SQL Server 2005 database using Asp.net?

I want to store the annotations in a SQL Server 2005 (Backend) database and retrieve them on user's request using Asp.Net as frontend. It would be great if I could get some example or documentation to achieve it.

Save account id on all annotations

Save account id on all annotations (and set it from x-annotator-account-id header).

Search API should return raw or rawish ES rather than our format

At the moment we have our own hand-crafted results format. We may as well just return ES because:

• It's not that different
• We get everything ES has (e.g. facets) without having to keep modding our format ...

Move homepage header and footer into base.html

Allow specification of abstract user groups in permissions

This was originally proposed way back when I wrote the permissions plugin, but again, there seems to have been some confusion over implementation. @rgrp has suggested:

This is great Nick. To note some discussion on irc. I think we should designate some reserved usernames and then use them in the action lists i.e.

__anonymous__ = any user (logged in or o/w)
__logged_in__ = logged in user (!)

{ read: [ '__anonymous__', ...] }=> any user logged in or o/w can read
{ update: [ '__logged_in__', ...] } => any logged in user can update (v. unusual but possible!)

I'm going to propose something very similar, but a little different, namely that these abstract classes of users are represented by groups:

{
  "read": ["alice", "group:__world__"],
  "update": ["alice", "group:__authenticated__"],
  "delete": ["alice", "group:__consumer__"],
  "admin": ["alice"]
}

In the above scenario, I distinguish between three potential user groups. In decreasing size of group:

1. group:__world__ - absolutely anyone, regardless of logged-in status
2. group:__authenticated__ - anyone who is authenticated to AnnotateIt, which could be an AnnotateIt registered user or the user of another consumer
3. group:__consumer__ - anyone who is authenticated to AnnotateIt with an authToken issued by the same consumer as the creator of the annotation

Under this scenario, I think the implementation of the Permissions plugin should be such that the default permissions are (assuming the current user is "alice")

{
  "read": ["alice", "group:__consumer__"],
  "update": ["alice"],
  "delete": ["alice"],
  "admin": ["alice"]
}

and the role of the "Allow anyone to..." checkboxes should be to add or remove group:__consumer__ to/from the relevant field. (NB: strictly, the "alice" in each permissions entry should probably not be needed, as the owner of the annotation should always have full permissions)

Question: if we implement this, should we restore the role of "action": [] to denote "no-one (other than the annotation owner) has permission"?

Sysadmin accounts for store set in annotator.cfg

Sysadmins would be able to do anything. Could specific them as space separated strings of account ids in annotator.cfg.

Implement new authorization / permissions framework

Implement authorization / permissions in store as per description in https://github.com/okfn/annotator/issues/closed#issue/22

Use proper HMAC signing for authToken generation

Strictly speaking, the simple concatenate-and-hash algorithm currently used to calculate the authToken isn't a very good idea. For reference, see:

• http://benlog.com/articles/2008/06/19/dont-hash-secrets/
• http://rdist.root.org/2009/10/29/stop-using-unsafe-keyed-hashes-use-hmac/

as well as the problems Flickr got into when they did (almost) the same thing as us:

• http://netifera.com/research/flickr_api_signature_forgery.pdf

I'm proposing that we use itsdangerous here, which could also result in a drastic simplification of the auth protocol -- namely that we might end up requiring only two headers: X-Annotator-Auth-Token and X-Annotator-Consumer-Key, with everything else (including arbitrary extensions) bundled up in the authToken.

How to store annotations in a Microsoft SQL Server 2005 database?

How can I use Microsoft SQL Server 2005 database to store these annotations?

Thanks.

elasticsearch exception when persisting dc.date

When the document plugin is enabled it attempts to find Dublin Core metadata to persist as part of the annotation. When it finds a dc.date it persists it as annotation.document.dc.date which ElasticSearch tries to helpfully parse as a date. This is nice except when the date isn't in a format that ElasticSearch recognizes, and it throws an exception, which causes the annotation save to fail.

For example when annotating this page which has a dc.date of "2006-03" which throws this exception in the ElasticSearch:

Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [03/2006], tried both date format [dateOptionalTime], and timestamp number
    at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:426)
    at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:378)
    at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:167)
    at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:385)
    ... 14 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "03/2006" is malformed at "/2006"

Sort annotations search results by most recently updated

Proper search including full-text search

Current search is inadequate:

• Currently based on temporary views in couchdb (could move to predefined views but how do we deal with fact we allow arbitrary keys/columns)
• No full-text search
• No searching on permissions

Suggest using elasticsearch (cf #24)

Revision annotations

Do copy-on-write revisioning of all annotations. Store type of change (create/delete/update) as well as account-id + user-id of person making change.

Get bookmarklet code and API key

Once signed up with an Account an Account holder can generate a bookmarklet and obtain their API key.

Should also provide instructions for use.

How to configure Store Backend into window OS

Hello Annotator,
Could you please explain how can I configure annotateit into window enviroment

nosetests fail and input pip -E pyenv install -e . not work

Hello,

I got an issue; my python version is 2.6.5

(pyenv)root@haitao-desktop:~/workspace/annotations/annotator-store# python run.py

• Running on http://127.0.0.1:5000/
• Restarting with reloader

nosetests

EE

ERROR: Failure: ImportError (No module named pyes)

Traceback (most recent call last):
File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName
addr.filename, addr.module)
File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/haitao/workspace/annotations/annotator-store/annotator/init.py", line 1, in
from .elasticsearch import ElasticSearch
File "/home/haitao/workspace/annotations/annotator-store/annotator/elasticsearch.py", line 5, in
import pyes
ImportError: No module named pyes

ERROR: Failure: ImportError (No module named flask)

Traceback (most recent call last):
File "/usr/lib/pymodules/python2.6/nose/loader.py", line 379, in loadTestsFromName
addr.filename, addr.module)
File "/usr/lib/pymodules/python2.6/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/lib/pymodules/python2.6/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/haitao/workspace/annotations/annotator-store/tests/init.py", line 2, in
from flask import Flask, g, request
ImportError: No module named flask

---

Ran 2 tests in 0.009s

FAILED (errors=2)

by the way, I input pip -E pyenv install -e . it shows no -E command, Can you tell me why;

Appreciate for any helps;

regards

uat.annotateit.org is refusing CORS requests

Getting the following error in Chrome when running the bookmarklet and saving an annotation

 Request header field X-Requested-With is not allowed by Access-Control-Allow-Headers

Store is checking for Authentication headers regardless of AUTH_ON config flag

See lines 66, 80 & 97 of store.py. The call to authorize() should either not be made or return True if AUTH_ON == False

nosetests fail, ubuntu 10.10 using latest git clone.

After running these commands:

git clone https://github.com/okfn/annotator-store.git
cd annotator-store
pip -E pyenv install -r requirements
source pyenv/bin/activate
cp annotator.cfg.example annotator.cfg

I get the following:

(pyenv)ewald@ewald-VirtualBox:~/programming/annotator-store$ nosetests

..........................EEEEEEEEEEFFFFF

ERROR: tests.annotator.test_store.TestStore.test_cors_preflight

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_create

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_delete

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_delete_notfound

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_index

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_read

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_read_notfound

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_search

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_update

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

ERROR: tests.annotator.test_store.TestStore.test_update_notfound

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 371, in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/util.py", line 478, in try_run
return func()
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 13, in setup
assert app.config['MOUNTPOINT'] == '', "MOUNTPOINT config option is incorrect for tests. should be ''"
AssertionError: MOUNTPOINT config option is incorrect for tests. should be ''

FAIL: tests.annotator.test_store.TestStoreAuth.test_get_allowed

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 166, in test_get_allowed
assert response.status_code == 200, "GET should be allowed"
AssertionError: GET should be allowed

FAIL: tests.annotator.test_store.TestStoreAuth.test_reject_post_request

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 172, in test_reject_post_request
assert response.status_code == 401, "response should be 401 NOT AUTHORIZED"
AssertionError: response should be 401 NOT AUTHORIZED

FAIL: tests.annotator.test_store.TestStoreAuthz.test_read

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 202, in test_read
assert response.status_code == 401, response.status_code
AssertionError: 404

FAIL: tests.annotator.test_store.TestStoreAuthz.test_update

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 216, in test_update
assert response.status_code == 401, response.status_code
AssertionError: 404

FAIL: tests.annotator.test_store.TestStoreAuthz.test_update_change_permissions_not_allowed

Traceback (most recent call last):
File "/home/ewald/programming/annotator-store/pyenv/lib/python2.6/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/home/ewald/programming/annotator-store/tests/annotator/test_store.py", line 235, in test_update_change_permissions_not_allowed
assert response.status_code == 401, response.status_code
AssertionError: 404

---

Ran 41 tests in 6.117s

FAILED (errors=10, failures=5)

pip freeze gives:

CouchDB==0.8
Flask==0.6.1
Flask-WTF==0.5.2
Jinja2==2.6
WTForms==0.6.3
Werkzeug==0.7
distribute==0.6.10
iso8601==0.1.4
nose==1.0.0
wsgiref==0.1.2

Expose public annotations at canonical URL

Each public annotation should have a canonical URL where it can be rendered as HTML, and itself annotated.

(From a discussion with Randall Leeds and Dan Whaley from Hypothes.is)

Return correct total counts in queries as well as results

Currently still return a total figure but it is not correct -- just number of rows returned not the actual count in the db.

This is due to change to couchdb. Will need to use map/reduce for this.

TypeError: ResultSet indices must be integers, not str

HI,

I input python run.py

it show

127.0.0.1 - - [12/Oct/2012 00:15:06] "GET /search?uri=http%3A%2F%2Flocalhost%2Fannotation%2Fdev.html HTTP/1.1" 500 -
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1701, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1689, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1687, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1360, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1358, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1344, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/skeleton/workspace/annotations/annotator-store/annotator/store.py", line 147, in search_annotations
results = Annotation.search(**kwargs)
File "/home/skeleton/workspace/annotations/annotator-store/annotator/elasticsearch.py", line 118, in search
docs = res['hits']['hits']
File "/usr/local/lib/python2.6/dist-packages/pyes/es.py", line 1756, in getitem
self.class.name, val.class.name))
TypeError: ResultSet indices must be integers, not str

store.py code;

import json

from flask import Blueprint, Response
from flask import g
from flask import request

from annotator.annotation import Annotation

store = Blueprint('store', name)

CREATE_FILTER_FIELDS = ('updated', 'created', 'consumer')
UPDATE_FILTER_FIELDS = ('updated', 'created', 'user', 'consumer')

We define our own jsonify rather than using flask.jsonify because we wish

to jsonify arbitrary objects (e.g. index returns a list) rather than kwargs.

def jsonify(obj, _args, *_kwargs):
res = json.dumps(obj, indent=None if request.is_xhr else 2)
return Response(res, mimetype='application/json', _args, *_kwargs)

@store.after_request
def after_request(response):
ac = 'Access-Control-'

response.headers[ac + 'Allow-Origin']      = request.headers.get('origin', '*')
response.headers[ac + 'Expose-Headers']    = 'Content-Length, Content-Type, Location'

if request.method == 'OPTIONS':
    response.headers[ac + 'Allow-Headers']  = \
    'Content-Length, Content-Type, ' \
    'x-annotator-auth-token, x-requested-with, x-annotator-auth-token-issue-time, ' \
    'x-annotator-consumer-key, x-annotator-auth-token-ttl, x-annotator-user-id'
    response.headers[ac + 'Allow-Methods']  = 'GET, POST, PUT, DELETE, OPTIONS'
    response.headers[ac + 'Max-Age']        = '86400'

return response

ROOT

@store.route('/')
def root():
return jsonify("Annotator Store API")

INDEX

@store.route('/annotations')
def index():
consumer, user = g.auth.request_credentials(request)
annotations = Annotation.search(_user_id=user, _consumer_key=consumer)
return jsonify(annotations)

CREATE

@store.route('/annotations', methods=['POST'])
def create_annotation():
consumer, user = g.auth.request_credentials(request)

# Only registered users can create annotations
if not (consumer and user):
    return _failed_authz_response('create annotation')

if request.json:
    annotation = Annotation(_filter_input(request.json, CREATE_FILTER_FIELDS))

    annotation['consumer'] = consumer
    if _get_annotation_user(annotation) != user:
        annotation['user'] = user

    annotation.save()

    if hasattr(g, 'after_annotation_create'):
        g.after_annotation_create(annotation)
        annotation.save()

    return jsonify(annotation)
else:
    return jsonify('No JSON payload sent. Annotation not created.', status=400)

READ

@store.route('/annotations/')
def read_annotation(id):
annotation = Annotation.fetch(id)
if not annotation:
return jsonify('Annotation not found!', status=404)

failure = _check_action(annotation, 'read')
if failure:
    return failure

return jsonify(annotation)

UPDATE

@store.route('/annotations/', methods=['POST', 'PUT'])
def update_annotation(id):
annotation = Annotation.fetch(id)
if not annotation:
return jsonify('Annotation not found! No update performed.', status=404)

failure = _check_action(annotation, 'update')
if failure:
    return failure

if request.json:
    updated = _filter_input(request.json, UPDATE_FILTER_FIELDS)
    updated['id'] = id # use id from URL, regardless of what arrives in JSON payload

    if 'permissions' in updated and updated['permissions'] != annotation.get('permissions', {}):
        failure = _check_action(annotation, 'admin', message='permissions update')
        if failure:
            return failure

    annotation.update(updated)

    if hasattr(g, 'before_annotation_update'):
        g.before_annotation_update(annotation)

    annotation.save()

return jsonify(annotation)

DELETE

@store.route('/annotations/', methods=['DELETE'])
def delete_annotation(id):
annotation = Annotation.fetch(id)

if not annotation:
    return jsonify('Annotation not found. No delete performed.', status=404)

failure = _check_action(annotation, 'delete')
if failure:
    return failure

annotation.delete()
return None, 204

SEARCH

@store.route('/search')
def search_annotations():
kwargs = dict(request.args.items())

consumer, user = g.auth.request_credentials(request)

kwargs['_consumer_key'] = consumer
kwargs['_user_id'] = user

if 'offset' in kwargs:
    kwargs['offset'] = _quiet_int(kwargs['offset'])
if 'limit' in kwargs:
    kwargs['limit'] = _quiet_int(kwargs['limit'], 20)

results = Annotation.search(**kwargs)
total = Annotation.count(**kwargs)
return jsonify({
    'total': total,
    'rows': results
})

def _filter_input(obj, fields):
for field in fields:
obj.pop(field, None)

return obj

def _get_annotation_user(ann):
"""Returns the best guess at this annotation's owner user id"""
user = ann.get('user')

if not user:
    return None

try:
    return user.get('id', None)
except AttributeError:
    return user

def _check_action(annotation, action, message=''):
consumer, user = g.auth.request_credentials(request)

if not g.authorize(annotation, action, user, consumer):
    return _failed_authz_response(message)

def _failed_authz_response(msg=''):
consumer, user = g.auth.request_credentials(request)
return jsonify("Cannot authorize request{0}. Perhaps you're not logged in as "
"a user with appropriate permissions on this annotation? "
"(user={user}, consumer={consumer}".format(' (' + msg + ')' if msg else '', user=user, consumer=consumer),
status=401)

def _quiet_int(obj, default=0):
try:
return int(obj)
except ValueError:
return default

I am confused about that, anyone help will be appreciated

regards

Nosetests failures

Hello! I run nosetests as per the documentation actually online and I got (with Python 2.7.3):

.............................................EE...EE.......................

ERROR: tests.test_store.TestStore.test_create_disable_refresh

Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(_self.arg)
File "/usr/lib/python2.7/dist-packages/mock.py", line 566, in patched
return func(_args, *_keywargs)
File "/opt/annotator-store/tests/test_store.py", line 134, in test_create_disable_refresh
headers=self.headers)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/test.py", line 745, in post
return self.open(_args, **kw)
File "/usr/local/lib/python2.7/dist-packages/flask/testing.py", line 102, in open
follow_redirects=follow_redirects)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/test.py", line 675, in open
rv = run_wsgi_app(self.application, environ, buffered=buffered)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/test.py", line 818, in run_wsgi_app
app_iter = app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1701, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1689, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1687, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1360, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1358, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1344, in dispatch_request
return self.view_functionsrule.endpoint
File "/opt/annotator-store/annotator/store.py", line 113, in create_annotation
annotation['consumer'] = g.user.consumer.key
TypeError: 'Mock' object does not support item assignment
-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: DEBUG: "PUT /annotator_test HTTP/1.1" 200 31
urllib3.connectionpool: DEBUG: "PUT /annotator_test/annotation/_mapping HTTP/1.1" 200 31
--------------------- >> end captured logging << ---------------------

ERROR: tests.test_store.TestStore.test_create_disable_refresh

Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 384, in tearDown
try_run(self.inst, ('teardown', 'tearDown'))
File "/usr/lib/python2.7/dist-packages/nose/util.py", line 478, in try_run
return func()
File "/opt/annotator-store/tests/test_store.py", line 25, in teardown
self.ctx.pop()
File "/usr/local/lib/python2.7/dist-packages/flask/ctx.py", line 262, in pop
% (rv, self)
AssertionError: Popped wrong request context. (<RequestContext 'http://localhost/api/annotations?refresh=false' [POST] of tests> instead of <RequestContext 'http://localhost/' [GET] of tests>)
-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: DEBUG: "PUT /annotator_test HTTP/1.1" 200 31
urllib3.connectionpool: DEBUG: "PUT /annotator_test/annotation/_mapping HTTP/1.1" 200 31
--------------------- >> end captured logging << ---------------------

ERROR: tests.test_store.TestStore.test_create_refresh

Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(_self.arg)
File "/usr/lib/python2.7/dist-packages/mock.py", line 566, in patched
return func(_args, *_keywargs)
File "/opt/annotator-store/tests/test_store.py", line 124, in test_create_refresh
headers=self.headers)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/test.py", line 745, in post
return self.open(_args, **kw)
File "/usr/local/lib/python2.7/dist-packages/flask/testing.py", line 102, in open
follow_redirects=follow_redirects)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/test.py", line 675, in open
rv = run_wsgi_app(self.application, environ, buffered=buffered)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/test.py", line 818, in run_wsgi_app
app_iter = app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1701, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1689, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1687, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1360, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1358, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1344, in dispatch_request
return self.view_functionsrule.endpoint
File "/opt/annotator-store/annotator/store.py", line 113, in create_annotation
annotation['consumer'] = g.user.consumer.key
TypeError: 'Mock' object does not support item assignment
-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: DEBUG: "PUT /annotator_test HTTP/1.1" 200 31
urllib3.connectionpool: DEBUG: "PUT /annotator_test/annotation/_mapping HTTP/1.1" 200 31
--------------------- >> end captured logging << ---------------------

ERROR: tests.test_store.TestStore.test_create_refresh

Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 384, in tearDown
try_run(self.inst, ('teardown', 'tearDown'))
File "/usr/lib/python2.7/dist-packages/nose/util.py", line 478, in try_run
return func()
File "/opt/annotator-store/tests/test_store.py", line 25, in teardown
self.ctx.pop()
File "/usr/local/lib/python2.7/dist-packages/flask/ctx.py", line 262, in pop
% (rv, self)
AssertionError: Popped wrong request context. (<RequestContext 'http://localhost/api/annotations?refresh=true' [POST] of tests> instead of <RequestContext 'http://localhost/' [GET] of tests>)
-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: DEBUG: "PUT /annotator_test HTTP/1.1" 200 31
urllib3.connectionpool: DEBUG: "PUT /annotator_test/annotation/_mapping HTTP/1.1" 200 31
--------------------- >> end captured logging << ---------------------

---

Ran 73 tests in 73.657s

FAILED (errors=4)

---

On my machine, pip freeze gives:

---

[...]
Flask==0.9
Jinja2==2.6
Werkzeug==0.8.3
distribute==0.6.24dev-r0
virtualenv==1.8.4
[...]

---

What should I look after?

Thank you very much for any hint.

run.py fail

giving this error stack:

No handlers could be found for logger "pyes.urllib3.connectionpool" Traceback (most recent call last): File "run.py", line 71, in <module> main() File "run.py", line 37, in main annotation.Annotation.create_all() File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/annotator/elasticsearch.py", line 72, in create_all cls.es.conn.create_index_if_missing(cls.es.index) File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/es.py", line 298, in create_index_if_missing return self.create_index(index, settings) File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/es.py", line 289, in create_index return self._send_request('PUT', index, settings) File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/es.py", line 208, in _send_request response = self.connection.execute(request) File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/connection_http.py", line 167, in _client_call return getattr(conn.client, attr)(*args, **kwargs) File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/connection_http.py", line 59, in execute response = self.client.urlopen(Method._VALUES_TO_NAMES[request.method], uri, body=request.body, headers=request.headers) File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/urllib3/connectionpool.py", line 294, in urlopen return self.urlopen(method, url, body, headers, retries-1, redirect) # Try again File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/urllib3/connectionpool.py", line 294, in urlopen return self.urlopen(method, url, body, headers, retries-1, redirect) # Try again File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/urllib3/connectionpool.py", line 294, in urlopen return self.urlopen(method, url, body, headers, retries-1, redirect) # Try again File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/urllib3/connectionpool.py", line 294, in urlopen return self.urlopen(method, url, body, headers, retries-1, redirect) # Try again File "/var/www/os2dagsorden.bellcom.dk/public_html/sites/all/modules/annotator/lib/store-plugin/pyenv/lib/python2.6/site-packages/pyes/urllib3/connectionpool.py", line 255, in urlopen raise MaxRetryError("Max retries exceeded for url: %s" % url) pyes.urllib3.connectionpool.MaxRetryError: Max retries exceeded for url: /annotator

best guess that it is because of the Python (2.6) version
Tests (run_tests.py) however run fine.

AssertionError: request should have been verified

FAIL: tests.annotator.test_auth.TestAuth.test_verify_request

Hi Team,

Could you pls help with fixing this error -

Traceback (most recent call last):
File "/home/tanya27/okfn-annotator-abd7cac/okfn-annotator-store-97aaff6/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/home/tanya27/okfn-annotator-abd7cac/okfn-annotator-store-97aaff6/tests/annotator/test_auth.py", line 68, in test_verify_request
assert auth.verify_request(request), "request should have been verified"
AssertionError: request should have been verified

FAIL: tests.annotator.test_auth.TestAuth.test_verify_request_mixedcase_headers

Traceback (most recent call last):
File "/home/tanya27/okfn-annotator-abd7cac/okfn-annotator-store-97aaff6/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/home/tanya27/okfn-annotator-abd7cac/okfn-annotator-store-97aaff6/tests/annotator/test_auth.py", line 80, in test_verify_request_mixedcase_headers
assert auth.verify_request(request), "request with mixed-case headers should have been verified"
AssertionError: request with mixed-case headers should have been verified

FAIL: tests.annotator.test_auth.TestAuth.test_verify_token

Traceback (most recent call last):
File "/home/tanya27/okfn-annotator-abd7cac/okfn-annotator-store-97aaff6/pyenv/local/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/home/tanya27/okfn-annotator-abd7cac/okfn-annotator-store-97aaff6/tests/annotator/test_auth.py", line 53, in test_verify_token
assert auth.verify_token(tok, 'testAccount', 'alice', expiryTime), "token should have been verified"
AssertionError: token should have been verified

---

Ran 41 tests in 7.160s

FAILED (failures=3)

Regards,
Papitha

error while running run.py

Hi I have followed all the steps
virtualenv pyenv
source pyenv/bin/activate
pip install -e .
cp annotator.cfg.example annotator.cfg
python run.py

it gives following error while last step. Please describe me .

ERROR:root:creating index annotator
WARNING:urllib3.connectionpool:Retrying (3 attempts remain) after connection broken by 'error(111, 'Connection refused')': /annotator
WARNING:urllib3.connectionpool:Retrying (2 attempts remain) after connection broken by 'error(111, 'Connection refused')': /annotator
WARNING:urllib3.connectionpool:Retrying (1 attempts remain) after connection broken by 'error(111, 'Connection refused')': /annotator
WARNING:pyes:Removed server http://127.0.0.1:9200 from active pool
ERROR:pyes:Client error: 3 retries left
Traceback (most recent call last):
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/pyes/connection_http.py", line 91, in execute
response = conn.urlopen(*_kwargs)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 488, in urlopen
release_conn=release_conn, *_response_kw)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 488, in urlopen
release_conn=release_conn, *_response_kw)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 488, in urlopen
release_conn=release_conn, *_response_kw)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/urllib3/connectionpool.py", line 472, in urlopen
raise MaxRetryError(self, url, e)
MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=9200): Max retries exceeded with url: /annotator (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
Traceback (most recent call last):
File "run.py", line 74, in
main()
File "run.py", line 38, in main
annotation.Annotation.create_all()
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/annotator/elasticsearch.py", line 91, in create_all
cls.es.conn.create_index_if_missing(cls.es.index)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/pyes/decorators.py", line 46, in __inner
return fun(_args, *_kwargs)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/pyes/es.py", line 720, in create_index_if_missing
return self.indices.create_index_if_missing(index=index, settings=settings)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/pyes/managers.py", line 161, in create_index_if_missing
return self.create_index(index, settings)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/pyes/managers.py", line 150, in create_index
return self.conn._send_request('PUT', index, settings)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/pyes/es.py", line 583, in _send_request
response = self.connection.execute(request)
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/pyes/connection_http.py", line 85, in execute
self._local.server = server = self._get_server()
File "/root/Desktop/work_related/PDF_R&D/anno_store/annotator-store-master/pyenv/lib/python2.7/site-packages/pyes/connection_http.py", line 121, in _get_server
raise NoServerAvailable
pyes.exceptions.NoServerAvailable

Update annotation without annotation id in the payload

Allow updating an annotation without annotation id in the payload (and ensure we cannot change id from update)

MockAuthenticator needs a request_credentials method

Test Authentication with Current Annotator Store Backend

At the moment not sure Authentication is working with Annotator Store backend. Therefore need to:

1. Deploy latest Annotator Store (locally)
2. Check this works with Annotator js
3. If it does not create tests to reflect failure and fix

Rename Consumer to Account

Will rename Consumer to Account in store as this is more meaningful to average users / devs (and covers both bookmarklet and openshakespeare use case better)