superdesk/eve-elastic

Elasticsearch data layer for eve framework

Superdesk

Superdesk is an open source end-to-end news creation, production, curation, distribution and publishing platform developed and maintained by Sourcefabric with the sole purpose of making the best possible software for journalism. It is scaleable to suit news organizations of any size. See the Superdesk website for more information.

Looking to stay up to date on the latest news? Subscribe to our monthly newsletter.

The Superdesk server provides the API to process all client requests. The client provides the user interface. Server and client are separate applications using different technologies.

Find more information about the client configuration in the README file of the repo: github.com/superdesk/superdesk-client-core

Run Superdesk locally using Docker

You can start superdesk using the docker-compose.yml file:

$ docker-compose up -d

This will start superdesk on http://localhost:8080. On the first run you also have to initialize elastic/mongo and create a user:

# Initialize data
$ docker-compose run superdesk-server run python manage.py app:initialize_data
# Create first admin user
$ docker-compose run superdesk-server run python manage.py users:create -u admin -p admin -e admin@localhost --admin

Then you can login with admin:admin credentials.

The Docker images are hosted on Dockerhub for the client and server.

Manual installation

Requirements

These services must be installed, configured and running:

• MongoDB
• ElasticSearch (7.x)
• Redis
• Python (>= 3.6)
• Node.js (with npm)

On macOS, if you have homebrew installed, simply run: brew install mongodb elasticsearch redis python3 node.

Installation steps:

path=~/superdesk
git clone https://github.com/superdesk/superdesk.git $path

# server
cd $path/server
pip3 install -r requirements.txt
python3 manage.py app:initialize_data
python3 manage.py users:create -u admin -p admin -e '[email protected]' --admin
honcho start
# if you need some data
python manage.py app:prepopulate

# client
cd $path/client
npm install
npm run build
npx grunt server

# open http://localhost:9000 in browser

⚠️ macOS users

All the above commands need to run inside the Python Virtual Environment, which you can create using the pyvenv command:

• Run pyvenv ~/pyvenv to create the files needed to start an environment in the directory ~/pyvenv.
• Run . ~/pyvenv/bin/activate to start the virtual environment in the current terminal session.

Now you may run the installation steps from above.

Questions and issues

• Our issue tracker is only for bug reports and feature requests.
• Anything else, such as questions or general feedback, should be posted in the forum.

A special thanks to...

Users, developers and development partners that have contributed to the Superdesk project. Also, to all the other amazing open-source projects that make Superdesk possible!

License

Superdesk is available under the AGPL version 3 open source license.

eve-elastic's Issues

Error when update or delete an item

Update: Problem with DELETE resolved after dowgrading to Eve 0.7.10 where common.get_document() still passed only 3 arguments to app.data.find_one() but still persist with PUT/PATCH.

---

In my simple app, I can create and get new items just fine but when I tried a PATCH/PUT/DELETE request then I got the error:

TypeError: find_one() takes 3 positional arguments but 5 were given

The error doesn't appear when Mongodb is used instead of Elasticsearch.

My following setup and code:

• Python version: 3.6
• Elasticsearch version: 2.4.6
• Eve version: 0.8.1
• Eve-Elastic version: 2.5.0

Source code

# settings.py
import os

ELASTICSEARCH_URL = os.environ.get("ELASTICSEARCH_URL", "http://localhost:9200")
ELASTICSEARCH_INDEX = os.environ.get("ELASTICSEARCH_INDEX", "eve")

RESOURCE_METHODS = ["GET", "POST"]
ITEM_METHODS = ["GET", "PATCH", "PUT", "DELETE"]

DOMAIN = {
    "test": {
        "schema": {"foo": {"type": "string"}},
        "item_url": 'regex("[\w,.:_-]+")',
        "datasource": {"backend": "elastic"},
    },
}

# run.py
from eve import Eve
from eve_elastic import Elastic

if __name__ == "__main__":
    Eve(data=Elastic).run()

API Call

curl -X DELETE \
  http://localhost:5000/test/AWjlAAzY-Wkn3SL9OgCa \
  -H 'If-Match: 16e79eb4b035c2d25b08f5bb46eeadd8002e4230' 

Traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/eve/flaskapp.py", line 1105, in __call__
    return super(Eve, self).__call__(environ, start_response)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/site-packages/eve/endpoints.py", line 100, in item_endpoint
    response = deleteitem(resource, **lookup)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/common.py", line 317, in rate_limited
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/eve/auth.py", line 80, in decorated
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/common.py", line 1251, in decorated
    r = f(resource, **combined_args)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/delete.py", line 46, in deleteitem
    return deleteitem_internal(resource, concurrency_check=True, **lookup)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/delete.py", line 97, in deleteitem_internal
    original = get_document(resource, concurrency_check, original, **lookup)
  File "/usr/local/lib/python3.6/site-packages/eve/methods/common.py", line 88, in get_document
    resource, req, check_auth_value, force_auth_field_projection, **lookup
TypeError: find_one() takes 3 positional arguments but 5 were given

Invite to join the @pyeve org

Hi, FYI I created the @pyeve organization and moved all eve repositories over there. If you feel like doing that, you can join the org and move eve-elastic there too (you'd still be maintainer of course)

It was tested with new versions of Elasticsearch like 6 and 7?

Is there any support for nested documents?

And if yes, is there any tests/examples.

thanks!

Schema: object hierarchy

Hi,

I have defined a schema in Eve to be used with a collection of products.
In this schema i added an 'images' attribute of type list of objects. It is expressed as follows :

    'name': {
        'type': 'string',
        'minlength': 3,
        'maxlength': 150,
        'required': True,
    },
    'images': {
        'type': 'list',
        'schema': {
            'type': 'dict',
            'schema': {
                'name': {
                    'type': 'string',
                    'required': True
                },
                'url': {
                    'type': 'string',
                    'required': True
                }
            }
        }
    },

This is a valid Cerberus format in Eve.
Inserting an object with a POST request works fine with a mongodb backend.

Request example:

POST on http://localhost:5000/products
with body containing :

{
  "name": "bird",
  "images": [{
    "name": "bird1",
    "url": "http://localhost/images/bird1.jpg"
  },
  {
    "name": "bird2",
    "url": "http://localhost/images/bird2.jpg"
  }]
}

With mongodb i receive a 201 response status whereas with elasticsearch i get a 500 response error and the data is not inserted nor the index created.

Am i missing something ?
Is any additional configuration required to make this work ?

Thanks

"where" searches

When using the
/endpoint?where=field=="value"
or
/endpoint?where={"field":"value"}
The data returned does not appear filtered. However, using the q=value syntax, the search appears to work normally.

I can do the same filters using mongoDB as the backend with the same data.

Does eve elastic work together with mongodb?

What I mean is:
Is this extension meant to REPLACE mongodb when using eve or does it work TOGETHER? Something like when data is inserted in mongo, it is sent to be indexed by elastic...

Thanks!

Error on dependency ciso8601

When I install, its throw following error on ciso8601 package.

Collecting ciso8601>=1.0.2 (from Eve-Elastic==2.1->-r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/61/d8/e17159b6037a26e3facdfeec8750dd17312284a915c5e0da79562b982035/ciso8601-2.0.1.tar.gz
    Complete output from command python setup.py egg_info:
    error in ciso8601 setup command: 'tests_require' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in unittest2 ; python_version < '3' at  ; python_version < '3'

404 & 405 Errors

I've encountered two big issues when attempting to make relatively simple use of this Eve backend. First, it seems items are not available at their endpoint: "/{resource}/{_id}". And second, it seems that only GET requests work.

With issues this obvious, I'm inclined to say that I'm using this wrong. So, below is a simple setup that replicates the issues:

# settings.py

RESOURCE_METHODS = ['GET','POST']
ITEM_METHODS = ['GET','PUT','PATCH','DELETE']
PUBLIC_RESOURCE_METHODS = ['GET','POST']
PUBLIC_ITEM_METHODS = ['GET','PUT','PATCH','DELETE']

DOMAIN = {
    'test': {
        'schema': {
            'foo': {
                'type': 'string'
            } 
        },
        'datasource': {
            'backend': 'elastic'
        }
    }
}

# app.py
from eve_elastic import Elastic
from eve import Eve

Eve(data=Elastic).run()

Also, here is a quick demostration with curl:

$ # POST a resource
$ curl -XPOST -H 'Content-Type: application/json' localhost:5000/test -d '{foo: bar}'
{
  "_status": "OK",
  "_updated": "Sat, 16 May 2015 20:25:53 GMT",
  "_etag": "3014835b5fe61de1d5995d3b5487d6f85dba6c68",
  "_created": "Sat, 16 May 2015 20:25:53 GMT",
  "_id": "AU1eZ4v3w-8a2P9WOYVU",
  "_links": {
    "self": {
      "title": "Test",
      "href": "test/AU1eZ4v3w-8a2P9WOYVU"
    }
  }
}
$ # Now, check that it exists
$ curl localhost:5000/test
{
  "_links": {
    "parent": {
      "title": "home",
      "href": "/"
    },
    "self": {
      "title": "test",
      "href": "test"
    }
  },
  "_items": [
    {
      "_type": "test",
      "foo": "bar",
      "_id": "AU1eZ4v3w-8a2P9WOYVU",
      "_links": {
        "self": {
          "title": "Test",
          "href": "test/AU1eZ4v3w-8a2P9WOYVU"
        }
      },
      "_etag": "3014835b5fe61de1d5995d3b5487d6f85dba6c68",
      "_created": "Sat, 16 May 2015 20:25:53 GMT",
      "_updated": "Sat, 16 May 2015 20:25:53 GMT"
    }
  ],
  "_meta": {
    "page": 1,
    "max_results": 25,
    "total": 1
  }
}
$ # Ok, now try to GET the item endpoint
$ curl localhost:5000/test/AU1eZ4v3w-8a2P9WOYVU
{
  "_status": "ERR",
  "_error": {
    "code": 404,
    "message": "The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again."
  }
}
$ # Ok, that doesn't work??

So, I tried adding an additional lookup:

# New settings.py
RESOURCE_METHODS = ['GET','POST']
ITEM_METHODS = ['GET','PUT','PATCH','DELETE']
PUBLIC_RESOURCE_METHODS = ['GET','POST']
PUBLIC_ITEM_METHODS = ['GET','PUT','PATCH','DELETE']

DOMAIN = {
    'test': {
        'schema': {
            'foo': {
                'type': 'string'
            } 
        },
        'additional_lookup': {
            'url': 'regex("[\w]+")',
            'field': 'foo'
        },
        'datasource': {
            'backend': 'elastic'
        }
    }
}

Now, the item endpoint works. But, attempting to PUT, PATCH, DELETE doesn't.

$ # Now the item endpoint works
$ curl localhost:5000/test/bar
{
  "_status": "OK",
  "_updated": "Sat, 16 May 2015 20:25:53 GMT",
  "_etag": "3014835b5fe61de1d5995d3b5487d6f85dba6c68",
  "_created": "Sat, 16 May 2015 20:25:53 GMT",
  "_id": "AU1eZ4v3w-8a2P9WOYVU",
  "_links": {
    "self": {
      "title": "Test",
      "href": "test/AU1eZ4v3w-8a2P9WOYVU"
    }
  }
}
$ # But, attempting to PUT, PATCH, DELETE doesn't
$ curl -XPUT -H 'Content-Type: application/json' -H 'If-Match: 3014835b5fe61de1d5995d3b5487d6f85dba6c68' localhost:5000/test/bar -d '{"foo: rab"}'
{
  "_status": "ERR",
  "_error": {
    "message": "The method is not allowed for the requested URL.",
    "code": 405
  }
}

Correct spelling

In features list, please correct word "aggragations" to "aggregations"

can't pass query parameters to elastic search

I installed the latest eve via pip install, when I try the example curl http://localhost:5000/items?q=foo&df=name i noticed the query parameter foo was never send to elastic search. By tracing the code, I found this line 'args = getattr(req, 'args', {})' in method find() returns null, actually there is no 'args' attribute of ParsedRequest req. It seems eve only accept parameter 'where' or the alias defined by QUERY_WHERE. Anything necessary configuration that I missed or this is a defect?

How does one go about integrating eve with sqlalchemy with flask?

How does one go about integrating eve with sqlalchemy with flask?

getting connection error

my code is shown below.

import eve
from eve_elastic import Elastic
app = eve.Eve(data=Elastic)

error

(venv)weber@weber-desktop:/var/www/lunar-cloud-web-ui/kukunako$ python test_server.py
Traceback (most recent call last):
File "test_server.py", line 4, in
app = eve.Eve(data=Elastic)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/eve/flaskapp.py", line 136, in init
self.data = data(self)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/eve/io/base.py", line 100, in init
self.init_app(self.app)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/eve_elastic/elastic.py", line 149, in init_app
self.put_mapping(app)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/eve_elastic/elastic.py", line 195, in put_mapping
indices.put_mapping(index=index or self.index, doc_type=resource, body=mapping, ignore_conflicts=True)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(_args, params=params, *_kwargs)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 276, in put_mapping
params=params, body=body)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 301, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/var/www/lunar-cloud-web-ui/venv/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 78, in perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ProtocolError(('Connection aborted.', error(111, 'Connection refused')))

How to connect in a elasticsearch cluster?

How to run eve-elastic search?

I am running python eve successfully. then I am imported eve-elastic and then just added data = Elastic,
and then run my server.
Then getting connection refused error.
My eve localhost machine running on 127.0.0.1:8000 port.
Why I am getting connection refused error

Mention PyMongo dependency

Apparently, this package needs PyMongo (https://pypi.python.org/pypi/pymongo), but doesn't say so. And from looking at the code (which imports bson.objectid.ObjectID) it's not evident that another package named bson is not the one needed (https://pypi.python.org/pypi/bson), which causes confusion. The README then should also point out, why PyMongo is really needed, since some people (including me) would not expect this if they want to use Eve with Elasticsearch instead of MongoDB.

There is no tag for release 2.4

Error with simple first app

I am trying to run simple basic app to get eve_elastic running.
I have following settings.py and app.py files.
When I do a request:
curl localhost:5000/test

I get following error, here I am assuming eve_elastic would create test index in ES

# settings.py

RESOURCE_METHODS = ['GET','POST']
ITEM_METHODS = ['GET','PUT','PATCH','DELETE']
PUBLIC_RESOURCE_METHODS = ['GET','POST']
PUBLIC_ITEM_METHODS = ['GET','PUT','PATCH','DELETE']

DOMAIN = {
    'test': {
        'schema': {
            'foo': {
                'type': 'string'
            } 
        },
        'datasource': {
            'backend': 'elastic'
        }
    }
}

# app.py
from eve_elastic import Elastic
from eve import Eve

Eve(data=Elastic).run()

Call

a@a-virtual-machine ~/api/eve/app1 $ curl localhost:5000
{"_links": {"child": [{"href": "test", "title": "test"}]}}
a@a-virtual-machine ~/api/eve/app1 $ 

a@a-virtual-machine ~/api/eve/app1 $ curl localhost:5000/test
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

Error Serverside

(env) a@a-virtual-machine ~/api/eve/app1 $ python app.py 
WARNING:elasticsearch:GET /eve/test/_search [status:400 request:0.054s]
[2018-01-29 12:03:34,248] ERROR in app: Exception on /test [GET]
Traceback (most recent call last):
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/a/api/env/local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/endpoints.py", line 54, in collections_endpoint
    response = get(resource, lookup)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/common.py", line 284, in rate_limited
    return f(*args, **kwargs)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/auth.py", line 79, in decorated
    return f(*args, **kwargs)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/common.py", line 1040, in decorated
    r = f(resource, **combined_args)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/get.py", line 39, in get
    return get_internal(resource, **lookup)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/get.py", line 112, in get_internal
    return _perform_find(resource, lookup)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve/methods/get.py", line 184, in _perform_find
    cursor = app.data.find(resource, req, lookup)
  File "/home/a/api/env/local/lib/python2.7/site-packages/eve_elastic/elastic.py", line 481, in find
    hits = self.elastic(resource).search(body=query, **args)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 539, in search
    doc_type, '_search'), params=params, body=body)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 110, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/a/api/env/local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 114, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, u'parsing_exception', u'no [query] registered for [filtered]')