aio-libs-abandoned / aioes Goto Github PK
View Code? Open in Web Editor NEWasyncio compatible driver for elasticsearch
Home Page: http://aioes.readthedocs.org
License: Apache License 2.0
asyncio compatible driver for elasticsearch
Home Page: http://aioes.readthedocs.org
License: Apache License 2.0
aioes is not able to json encode datetime objects.
File "/home/dev/.local/lib/python3.6/site-packages/aioes/connection.py", line 52, in perform_request
raise exc_class(resp.status, resp_body, extra)
aioes.exception.TransportError: TransportError(406, '{"error":"Content-Type header [application/octet-stream] is not supported","status":406}')
With test this code:
import asyncio
from aioes import Elasticsearch
async def go():
es = Elasticsearch(['localhost:9200'])
ret = await es.create(index="users",
doc_type="id",
id=42,
body={"str": "data", "int": 1})
assert (ret == {'_id': '32',
'_index': 'users',
'_type': 'id',
'_version': 1})
answer = await es.get(index="users",
doc_type="id",
id=42)
assert answer['_source'] == {'str': 'data', 'int': 1}
loop = asyncio.get_event_loop()
loop.run_until_complete(go())
I think should contains
connection.py
43 headers = {"Content-type":"application/json"}
44 resp = yield from self._session.request(
45 method, url, params=params, data=body, headers=headers)
Can one use aioes
with ca_certs
passed in as a parameter like elasticsearch-py:
es = Elasticsearch(
['localhost', 'otherhost'],
http_auth=('user', 'secret'),
port=443,
use_ssl=True,
ca_certs='/path/to/cacert.pem',
client_cert='/path/to/client_cert.pem',
client_key='/path/to/client_key.pem',
)
Hello,
In the current state - the transport handles the dumping and loading of documents internally - using json.dumps
.
Id like to use ujson for example, but without editing the sources of aioes.
the official ES client implements a Serializer interface - which conviniently supports changing the serializer backend (for example - mentioned here - elastic/elasticsearch-py#365).
Would you consider implementing such an interface?
Hi,
It seem the query need to be a POST on _delete_by_query:
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html
But aioes use DELETE on _query
https://github.com/aio-libs/aioes/blob/a739c156704159fd2729188d2d192baa2c31ee97/aioes/client/__init__.py#L971
I'm not sure if the parameters are the same and in which version they have changed.
It appears the fix is trivial and a few pull requests have already been submitted. I prefer the approach in:
#48
The search with scan, scroll and clear_scroll primitives are implemented in the client, but a scan helper is still lacking. It would work like http://elasticsearch-py.readthedocs.io/en/master/helpers.html#scan
For Python 3.6+, it could be an asynchronous generator, which would be closest to the official ES client.
For Python 3.5+, it could be an async for
:
async for hit in Scan(
client=client, index=..., doc_type=...,
query={"query": {"bool": {"filter": ...}}}):
...
The only way to support Python 3.4 would be to add a callback, I guess, but that's not a good fit with asyncio's model, so maybe 3.4 can be dropped for this feature.
Thoughts?
If Elasticsearch created outside coroutine it produces aiohttp warning
ERROR [asyncio] Creating a client session outside of coroutine
due
I cant find it anywhere in the docs or the source code. Does this support the 1.X and 2.X series of ES?
Just instantiate Elasticsearch
es = Elasticsearch(["localhost:8000"])
When terminate:
Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x7f64807de208>
Should aioes->connection.py->class Connection has a destructor ?
`
def __del__(self):
self._session.close()
`
aiohttp (2.0.4)
calling es.create without await seems to not work. Testing for py>=3.5.2.
This will probably not belong here since it's more an implementation detail, so apologies in advance! I'm slowly porting my code to asyncio as it's mostly I/O based but am hitting an issue with bridging it with synchronous modules, in this case I'm looking at how aioes
could be coupled with the excellent elasticsearch_dsl
module. Wrapping aioes
as a connection is no problem, but at one point one needs to bridge the regular connection call made in elasticsearch_dsl
to the asynchronous aioes
equivalent, somewhat like this -
import asyncio
@asyncio.coroutine
def target(value):
return "{} world".format(value)
def intermediary(value):
"""Return result from coroutine target()
"""
@asyncio.coroutine
def call():
result = intermediary('hello')
print(result)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(call())
I've been pulling my hair out about this one, but whatever I can think of I end up with something blocking. Would there be any pattern applicable to these type of situations?
As aiohttp
changes ClientSession.close()
to coroutine, we need to change Elasticsearch.close
to one too or deprecate it in favor of a new finallize
when usingELasticSearch.create(refresh=True)
gives an error ('Invalid variable type: mapping value should be str or int, got True') because in latest version of aiohttp the params can only be str or int.
so it failing in all requests where bool arguments are accepted
yarl (0.8.1) aio-libs/yarl#9
File "/usr/local/lib/python3.5/dist-packages/aioes/connection.py", line 41, in perform_request
url = self._base_url.with_path(url)
AttributeError: 'URL' object has no attribute 'with_path'
I update aiohttp to 2.0.3 slove the "with_path" problem
And
I am not able to add http headers when I perform http request in es operation
I want to pass header as argument when instantiate the es client
headers = {"Host": u"***", "key": "***"}
es = Elasticsearch(['localhost:8888'], headers=headers)
TypeError: __init__() got an unexpected keyword argument 'headers'
with elasticsearch==5.1.0, I can do that
Now
I need to reserve header argument in connection.py->Connection->perform_request, and reserve for every method/class using it
It would be nice to have support rivers feature. As far as I see it currently missed.
You can clearly see here conversion of parameter to boolean type.
https://github.com/aio-libs/aioes/blob/master/aioes/client/__init__.py#L133
Then it fails here
https://github.com/aio-libs/yarl/blob/master/yarl/__init__.py#L673
Considering this, there's no way to use refresh
parameter at all.
It looks like there is no chance to use HTTP Basic Auth for now?
I'd like to make new release, version 0.2 should support Python 3.5
Is anything left to be done before the release publishing?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.