atkinson/rawes

Low level elasticsearch driver for Python

rawes

About

rawes is an elasticsearch driver for Python. It provides a small level of abstraction above the requests library - enough abstraction to be useful, but not so much to obscure elasticsearch's great native api

Features

• elasticsearch native API support
• gzip over HTTP support
• Thrift support

Installation

$ pip install rawes

Usage

Create a connection to elasticsearch

import rawes
es = rawes.Elastic('localhost:9200')

Search for a document

es.get('tweets/tweet/_search', data={
    'query' : {
        'match_all' : {}
    }
})

The rawes.Elastic constructor takes the following parameters (defaults shown):

rawes.Elastic(
    url='localhost:9200', # Path and port to elasticsearch service.  Ports 9500-9600 will use thrift; others http
    path='', # http url path (for example, 'tweets/tweet/_search')
    timeout=30, # Timeout in seconds
    connection_type=None, # Set to 'http' or 'thrift' to explicitly set a protocol
)

An instance of rawes.Elastic ('es' in this case) has methods for get, post, put, delete, and head (for each http verb). Each method takes the following parameters (defaults shown):

es.get(
    path='', # HTTP URL path
    data='', # http body.  can be either a string or a python dictionary (will automatically be converted to JSON)
    params={}, # HTTP URL params passed as a python dictionary
    headers={}, # HTTP headers as a python dictionary
    **kwargs # HTTP only: any additional parameters you wish to pass to the python 'requests' library (for example, basic auth)
)

Examples

Create a new document in the twitter index of type tweet with id 1

es.put('tweets/tweet/1', data={
    'user' : 'dwnoble',
    'post_date' : '2012-8-27T08:00:30',
    'message' : 'Tweeting about elasticsearch'
})
es.put('blogs/post/2', data={
    'user' : 'dan',
    'post_date' : '2012-8-27T09:30:03',
    'title' : 'Elasticsearch',
    'body' : 'Blogging about elasticsearch'
})

Search for a document, specifying http params

es.get('tweets/tweet/_search', data={
    'query' : {
        'match_all' : {}
    }
}, params= {
    'size': 2
})

Search for a document with a JSON string

es.get('tweets,blogs/_search', data="""
{
    "query" : {
        "match_all" : {}
    }
}
""")

Update a document

es.put('someindex/sometype/123', data={
    'value' : 100,
    'other' : 'stuff'
})
es.post('someindex/sometype/123/_update', data={
    'script' : 'ctx._source.value += value',
    'params' : {
        'value' : 50
    }
})

Delete a document

es.delete('tweets/tweet/1')

Bulk load

bulk_body = '''
{"index" : {}}
{"key":"value1"}
{"index" : {}}
{"key":"value2"}
{"index" : {}}
{"key":"value3"}
'''

es.post('someindex/sometype/_bulk', data=bulk_body)

bulk_list = [
    {"index" : {}},
    {"key":"value4"},
    {"index" : {}},
    {"key":"value5"},
    {"index" : {}},
    {"key":"value6"}
]

# Remember to include the trailing \n character for bulk inserts
bulk_body_2 = '\n'.join(map(json.dumps, bulk_list))+'\n'

es.post('someindex/sometype/_bulk', data=bulk_body_2)

Alternate Syntax

Instead of setting the first argument of a es.<http verb> call to the HTTP URL path, you can also use python attributes and items to build up the url path. For example:

es.post('tweets/tweet/', data={
    'user' : 'dwnoble',
    'post_date' : '2012-8-27T09:15:59',
    'message' : 'More tweets about elasticsearch'
})

Becomes:

es.tweets.tweet.post(data={
    'user' : 'dwnoble',
    'post_date' : '2012-8-27T09:15:59',
    'message' : 'More tweets about elasticsearch'
})

Or using [] notation. This can be useful for characters that are not allowed in python attributes:

es['tweets']['tweet'].post(data={
    'user' : 'dwnoble',
    'post_date' : '2012-8-27T09:15:59',
    'message' : 'More tweets about elasticsearch'
})

More examples:

Searching tweets index for documents of type tweets

es.tweets.tweet._search.get(data={'query' : {'match_all' : {} }})

Searching tweets and blogs index for documents of any type using a JSON strings

es['tweets,blogs']._search.get(data='{"query" : {"match_all" : {}}}')

Thrift support

Before thrift will work with rawes, you must install the thrift python module

$ pip install thrift

By default, connections on ports between 9500 and 9600 will use thrift

import rawes
es_thrift = rawes.Elastic('localhost:9500')

If you are using thrift on a non standard port, specify connection_type='thrift' as a parameter

import rawes
es_thrift = rawes.Elastic('localhost:8500', connection_type='thrift')

Run Unit Tests

rawes' unit tests require the python thrift module to run:

$ pip install thrift

Run tests:

$ python -m unittest tests

License

Apache 2.0 License

Contact

@dwnoble