graphaware / graph-aided-search Goto Github PK
View Code? Open in Web Editor NEWElasticsearch plugin offering Neo4j integration for Personalized Search
Elasticsearch plugin offering Neo4j integration for Personalized Search
EDIT: I just find the error after few hours of research, an old instance from an other version of elasticsearch was still running in background. I close this mistake. Thanks for your work here ;)
Hello,
I get this error using the node.js official plugin of elasticsearch with neo4j.
Elasticsearch TRACE: 2016-11-02T22:26:57Z
-> POST http://localhost:9200/neo4j-index/School%2CEmployer%2CUserFacebook%2CUserEmail%2CPlace%2CCompanyDomain/_search
{
"query": {
"query_string": {
"fields": [
"facebookName^10",
"goodGmailName^7",
"gmailEmail^10",
"country^15",
"city^15",
"schoolName^30",
"employerName^20",
"placeName^15",
"zip^5",
"street^7",
"domain^5"
],
"query": "allan",
"use_dis_max": true
}
},
"gas-booster": {
"name": "SearchResultCypherBooster",
"query": "MATCH (n) WHERE n:UserEmail or n:UserFacebook WITH n AS n, (CASE WHEN exists(n.facebookEmail) or exists((n)-[:STUDIED_IN]->()) or exists(n.hometown) or size((n)-[]->())>30 THEN 1 else 0 end) as score RETURN n.uuid AS uuid, score",
"identifier": "uuid",
"scoreName": "score",
"operator": "+"
},
"gas-filter": {
"name": "SearchResultCypherFilter",
"query": "MATCH (n:UserEmail) WITH n MATCH (n)<-[:HAS_ACCOUNT]-(father) WITH n, father WHERE exists((father)<-[:KNOWS]-()) RETURN n.uuid AS id UNION MATCH (n:UserFacebook) WITH n MATCH (n)<-[:HAS_ACCOUNT]-(father) WITH n, father WHERE exists((father)<-[:KNOWS]-()) RETURN n.uuid AS id"
}
}
<- 400
{
"error": {
"root_cause": [
{
"type": "search_parse_exception",
"reason": "failed to parse search source. unknown search element [gas-booster]",
"line": 1,
"col": 243
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "neo4j-index",
"node": "TafSfQzVT1CTuW2gRv7Acw",
"reason": {
"type": "search_parse_exception",
"reason": "failed to parse search source. unknown search element [gas-booster]",
"line": 1,
"col": 243
}
}
]
},
"status": 400
}
Elasticsearch DEBUG: 2016-11-02T22:26:57Z
Request complete
Trace: [search_parse_exception] failed to parse search source. unknown search element [gas-booster], with { line=1 col=243 }
at /Users/bondot_e/Workspace/shortouch-api/app/controllers/users/users.controller.js:65:11
at tryCallOne (/Users/bondot_e/Workspace/shortouch-api/node_modules/elasticsearch/node_modules/promise/lib/core.js:37:12)
at /Users/bondot_e/Workspace/shortouch-api/node_modules/elasticsearch/node_modules/promise/lib/core.js:123:15
at flush (/Users/bondot_e/Workspace/shortouch-api/node_modules/elasticsearch/node_modules/asap/raw.js:50:29)
at _combinedTickCallback (internal/process/next_tick.js:67:7)
at process._tickCallback (internal/process/next_tick.js:98:9)
What's giving me a headache is that I'm currently working locally. But if I request elasticsearch hosted on my distant server everything is working fine...
I used elasticsearch v2.3.2 :
Version: 2.3.2, Build: b9e4a6a/2016-04-21T16:03:47Z, JVM: 1.8.0_60
on my distant server :
Version: 2.3.2, Build: b9e4a6a/2016-04-21T16:03:47Z, JVM: 1.8.0_91
and I tried with graph-aid-search v2.3.2.0 (the version which is working fine on my distant server), v.2.3.2.2 (recommended on the README) and the last release v2.4.1.3 without any success.
elasticsearch logs:
[2016-11-02 23:16:02,826][INFO ][node ] [Dirtnap] version[2.3.2], pid[30136], build[b9e4a6a/2016-04-21T16:03:47Z]
[2016-11-02 23:16:02,827][INFO ][node ] [Dirtnap] initializing ...
[2016-11-02 23:16:03,544][INFO ][plugins ] [Dirtnap] modules [reindex, lang-expression, lang-groovy], plugins [graph-aided-search], sites []
[2016-11-02 23:16:03,596][INFO ][env ] [Dirtnap] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [32gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2016-11-02 23:16:03,597][INFO ][env ] [Dirtnap] heap size [990.7mb], compressed ordinary object pointers [true]
[2016-11-02 23:16:03,598][WARN ][env ] [Dirtnap] max file descriptors [10240] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-11-02 23:16:06,129][INFO ][node ] [Dirtnap] initialized
[2016-11-02 23:16:06,130][INFO ][node ] [Dirtnap] starting ...
[2016-11-02 23:16:06,227][INFO ][transport ] [Dirtnap] publish_address {127.0.0.1:9301}, bound_addresses {[fe80::1]:9301}, {[::1]:9301}, {127.0.0.1:9301}
[2016-11-02 23:16:06,231][INFO ][discovery ] [Dirtnap] elasticsearch/F78p3vSDST-83RfYIVaG5Q
[2016-11-02 23:16:09,274][INFO ][cluster.service ] [Dirtnap] new_master {Dirtnap}{F78p3vSDST-83RfYIVaG5Q}{127.0.0.1}{127.0.0.1:9301}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-11-02 23:16:09,315][INFO ][http ] [Dirtnap] publish_address {127.0.0.1:9201}, bound_addresses {[fe80::1]:9201}, {[::1]:9201}, {127.0.0.1:9201}
[2016-11-02 23:16:09,315][INFO ][node ] [Dirtnap] started
[2016-11-02 23:16:09,334][INFO ][gateway ] [Dirtnap] recovered [0] indices into cluster_state
So graph-aided-search
is apparently successfully loaded.
I think I am following well my setup:
curl -XPOST 'localhost:9200/neo4j-index/_close'
curl -XPUT 'http://localhost:9200/neo4j-index/_settings?index.gas.neo4j.hostname=http://localhost:7474&index.gas.enable=true'
curl -XPUT 'http://localhost:9200/neo4j-index/_settings?index.gas.neo4j.user=neo4j&index.gas.neo4j.password=xxx'
curl -X PUT 'localhost:9200/neo4j-index/_settings' -d '{
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 4,
"max_gram": 20
}
},
"analyzer": {
"autocomplete": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"autocomplete_filter",
"asciifolding"
]
}
}
}
}'
curl -XPOST 'localhost:9200/neo4j-index/_open'
And then I performed my mapping.
Any suggestion ?
Hi,
This is a useful concept because it can improve understanding of complex networks... do you plan to come back to it?
Just curious because I prefer active codebases!
sent request "curl -X POST http://localhost:9200/neo4j-index/Movie/_search -d '{
"query" : {
"bool": {
"should": [{"match": {"title": "love"}}]
}
},
"gas-filter" :{
"name": "SearchResultCypherFilter",
"query": "MATCH (n:User)-[r:LIKES]->(m) WITH m, avg(r.rate) as avg_rate where avg_rate < 3 RETURN m.objectId as id",
"exclude": true,
"keyProperty": "objectId"
}
}'"
get an error : "{"error":{"root_cause":[{"type":"runtime_exception","reason":"Failed to parse a search response."}],"type":"runtime_exception","reason":"Failed to parse a search response.","caused_by":{"type":"runtime_exception","reason":"Cypher Execution Error, message is : {code=Neo.ClientError.Security.Unauthorized, message=No authentication header supplied.}"}},"status":500}"
first,i wouldn't setting dbms.security.auth_enabled=false,
help me!!!!
Hi,
We're currently giving GAS ago for filtering search queries based on information returned from the graph. It's all very simple, so I'm not sure why it's not working. If I add "exclude": true
to the query it does return results that don't match, like I would expect. As crazy as it sounds, if I run the graph query manually and run the search without the gas-filter
clause, I can see the documents returned do contain the id
s of the documents that are returned by the Cypher query. I'm stumped. Any help you could provide would be appreciated.
(:Product {id: <int>})-[:SIMILAR_TO {weight: <int>}]->(other:Product)
GET /products/product/_mapping
{
"products" : {
"mappings" : {
"product" : {
"properties" : {
"id" : {
"type" : "integer"
}
}
}
}
}
}
Result when running the Cypher query directly
> MATCH (:Product {id: 1})-[r:SIMILAR_TO]->(other) RETURN other.id as id ORDER BY r.weight DESC
╒═══╕
│id │
╞═══╡
│ 5 │
├───┤
│ 2 │
├───┤
│ 4 │
├───┤
│ 3 │
└───┘
The GAS query that returns nothing:
{
"query": {
"match_all": {}
},
"gas-filter": {
"query": "MATCH (:Product {id: 1})-[r:SIMILAR_TO]->(other) RETURN other.id as id ORDER BY r.weight DESC",
"protocol": "bolt",
"name": "SearchResultCypherFilter"
},
"size": 10,
"fields": ["id"]
}
Query Output for above:
{
"took": 602,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 0,
"max_score": 0,
"hits": []
}
}
UPDATE: Added verbose logging to ES
.......
search_1 | [2017-02-16 16:29:27,934][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Boosting results: {
search_1 | "took" : 14,
search_1 | "timed_out" : false,
search_1 | "_shards" : {
search_1 | "total" : 5,
search_1 | "successful" : 5,
search_1 | "failed" : 0
search_1 | },
search_1 | "hits" : {
search_1 | "total" : 1,
search_1 | "max_score" : 7.918695,
search_1 | "hits" : [ {
search_1 | "_index" : "products",
search_1 | "_type" : "product",
search_1 | "_id" : "636279",
search_1 | "_score" : 7.918695,
search_1 | "fields" : {
search_1 | "id" : [ 636279 ]
search_1 | }
search_1 | } ]
search_1 | }
search_1 | }
search_1 | [2017-02-16 16:29:27,935][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Reading headers...
search_1 | [2017-02-16 16:29:27,935][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Reading hits...
search_1 | [2017-02-16 16:29:27,981][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Reading aggregations...
search_1 | [2017-02-16 16:29:27,981][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Reading suggest...
search_1 | [2017-02-16 16:29:27,981][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Creating new SearchResponse...
search_1 | [2017-02-16 16:29:27,981][DEBUG][com.graphaware.es.gas.wrap.WrappingActionListener] [Brass] Rewriting overhead time: 62 - 14 = 48ms
.......
Is the gas-filter should support a multi-type query?
something like GET index1,index2/_search
?
Will there be a future release to support Elastic v2.4.6 as there are bug fixes to ES since v2.4.4? Also, the readme says v2.4.4 is supported but there is no release for that version. Will this be available soon?
Is it possible to query neo4j first and get relevant views, which can be used to get the appropriate documents and search via elasticsearch?
If i run this query over TCP with the Java Api (port 9300), i get this error:
SearchParseException[failed to parse search source. unknown search element [gas-filter]];
However, if i run this query over HTTP it works.
Any ideas? It seems like this plugin doesn't work with the Java API.
Cheers,
Clay
> {
> "size" : 100,
> "query" : {
> "bool" : {
> "must" : {
> "term" : {
> "disabled" : false
> }
> }
> }
> },
> "gas-filter" : {
> "name" : "SearchResultCypherFilter",
> "query" : "MATCH (a)-[:KNOWS*]->(d{referenceId:'522d8b956819338605a53948c5dcef1a'}) RETURN distinct a.referenceId as guid",
> "exclude" : true
> }
> }
Currently the SearchResultCypherBooster
class is cognizant of the node ids which are to be boosted which significantly improves performance as one can leverage indexes, i.e., if one wanted to rank the search results by most recently created,
curl -X GET localhost:3988/nodes/_search?pretty -d '{
"query": {
"query_string": {
"query": "foo"
}
},
"gas-booster": {
"name": "SearchResultCypherBooster",
"query": "MATCH ()-[r:CREATED]->(n:Entity) USING INDEX n:Entity(uuid) WHERE n.uuid IN {ids} RETURN n.uuid AS id, r.timestamp AS score",
"operator": "replace",
"protocol": "bolt"
}
}'
however for the SearchResultCypherFilter
class it is not, i.e.,
curl -X GET localhost:3988/nodes/_search?pretty -d '{
"query": {
"query_string": {
"query": "foo"
}
},
"gas-filter": {
"name": "SearchResultCypherFilter",
"query": "MATCH ()-[:CREATED]->(n:Entity) USING INDEX n:Entity(uuid) WHERE n.uuid IN {ids} RETURN n.uuid AS id",
"exclude": false,
"protocol": "bolt"
}
}'
{
"error" : {
"root_cause" : [ {
"type" : "runtime_exception",
"reason" : "Failed to parse a search response."
} ],
"type" : "runtime_exception",
"reason" : "Failed to parse a search response.",
"caused_by" : {
"type" : "runtime_exception",
"reason" : "Cypher Execution Error, message is : {code=Neo.ClientError.Statement.ParameterMissing, message=Expected a parameter named ids}"
}
},
"status" : 500
}
I presume that the node ids would be present as the gas-filter
condition should augment the same result set returned from ElasticSearch.
Removing the WHERE clause works, however performance is somewhat suboptimal, and about 10 times slower than an equivalent boost query i.e.,
time curl -X GET localhost:3988/nodes/_search?pretty -d '{
"query": {
"query_string": {
"query": "foo"
}
}
}'
real 0m0.016s
user 0m0.003s
sys 0m0.003s
time curl -X GET localhost:3988/nodes/_search?pretty -d '{
"query": {
"query_string": {
"query": "foo"
}
},
"gas-booster": {
"name": "SearchResultCypherBooster",
"query": "MATCH ()-[r:CREATED]->(n:Entity) USING INDEX n:Entity(uuid) WHERE n.uuid IN {ids} RETURN n.uuid AS id, r.timestamp AS score",
"operator": "replace",
"protocol": "bolt"
}
}'
real 0m0.104s
user 0m0.006s
sys 0m0.000s
time curl -X GET localhost:3988/nodes/_search?pretty -d '{
"query": {
"query_string": {
"query": "foo"
}
},
"gas-filter": {
"name": "SearchResultCypherFilter",
"query": "MATCH ()-[:CREATED]->(n:Entity) RETURN n.uuid AS id",
"exclude": false,
"protocol": "bolt"
}
}'
real 0m1.148s
user 0m0.006s
sys 0m0.000s
It is possible to use multiple filters in one query to the Elastic?
Hello, i tried run module, but i have not seen any changes in elastic.
What i did:
~\Documents\Neo4j\default.graphdb\plugins
elasticsearch.host_name=http://localhost:9200
elasticsearch.index_spec=people:Person(name).
My elastic used this default port .
Than run elastic and neo4j.
I used
CREATE (ee:Person {name: "Emil"})
MATCH (ee:Person) WHERE ee.name = "Emil" SET ee.name= ee.name
curl -XGET localhost:9200/* and get nothing
Why my command
MATCH (ee:Person) WHERE ee.name = "Emil" SET ee.name= ee.name cannot create ne record in elastic
Configuration:
Does this work with version 5?
Any roadblocks or issues?
Hi, is this elastic search plugin compatible to communicate to neo4j 3.0.x?
I've duplicated the Movie
database of Neo4j
on Elasticsearch
and it's indexed with the index nodes
. It has two types Movie
and Person
. I am trying to make a simple Result Filtering
with Graph-Aided Search
using this curl command line:
curl -X GET localhost:9200/nodes/_search?pretty -d '{
"query": {
"match_all" : {}
},
"gas-filter": {
"name": "SearchResultCypherfilter",
"query": "MATCH (p:Person)-[ACTED_IN]->(m:Movie) WHERE p.name= 'Parker Posey' RETURN m.uuid as id",
"shouldExclude": true,
"protocol": "bolt"
}
}'
But I get as results all the 171 nodes of both types Movie
and Person
in my index nodes
. However, as my query says I want to return only the type Movie
by its uuid
. So basically it doesn't look to the gas-filter
part.
Also when I put false
as the value of shouldExclude
I am getting the same results.
I am working with:
Result that should be returning:
The uuid
of the movie titled You've Got Mail
.
I tried to follow this tutorial for the configuration, and I found out that index.gas.enable
had the value false
so I changed it and finished the configuration just like in the tutorial:
mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.neo4j.hostname=http://localhost:7474
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.enable=true
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/indexname/_settings?index.gas.neo4j.user=neo4j
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/indexname/_settings?index.gas.neo4j.password=mypassword
{"acknowledged":true}
After that I tried to add the settings of boltHostname
and bolt.secure
but it didn't work and I had this error:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't update non dynamic settings[[index.gas.neo4j.boltHostname]] for open indices [[nodes]]"}],"type":"illegal_argument_exception","reason":"Can't update non dynamic settings[[index.gas.neo4j.boltHostname]] for open indices [[nodes]]"},"status":400}
So I closed my index to configure it and then opened it again:
mac$ curl -XPOST http://localhost:9200/nodes/_close
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.neo4j.boltHostname=bolt://localhost:7687
{"acknowledged":true}
mac$ curl -XPUT http://localhost:9200/nodes/_settings?index.gas.neo4j.bolt.secure=false
{"acknowledged":true}
mac$ curl -XPOST http://localhost:9200/nodes/_open
{"acknowledged":true}
After finishing the configuration I tried again on Postman
the same gas-filter
query that I was executing with curl
and now I am getting this error:
{
"error": {
"root_cause": [
{
"type": "runtime_exception",
"reason": "Failed to parse a search response."
}
],
"type": "runtime_exception",
"reason": "Failed to parse a search response.",
"caused_by": {
"type": "client_handler_exception",
"reason": "java.net.ConnectException: Connection refused (Connection refused)",
"caused_by": {
"type": "connect_exception",
"reason": "Connection refused (Connection refused)"
}
}
},
"status": 500
}
I don't know which connection the error is talking about. I am sure I passed the correct password of Neo4j
in the configuration. I've even stopped and restarted again the servers of Elasticsearch
and Neo4j
but still the same errors.
Any ideas?
Is there any way to use it with Elasticsearch version 6.x? I tried to install the Elasticsearch plugin in Elasticsearch version 6.X and failed.
Hi:
I tried following the steps outlined here.
http://graphaware.com/neo4j/2016/04/20/graph-aided-search-the-rise-of-personalised-content.html
The website uses older neo4j version which is 2.x,, so I replaced all downloads with neo4j 3.x
Now, I'm currently at the step where we are to Configure Graph-Aided Search by defining the URL of the Neo4j server instance and enabling it.
When we execute the following command:
$ curl -XPUT "http://localhost:9200/neo4j-index/_settings?index.gas.neo4j.hostname=http://localhost:7474&index.gas.enable=true"
we got the following error:
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"neo4j-index","index":"neo4j-index"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"neo4j-index","index":"neo4j-index"},"status":404}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.