Comments (9)
Hi @IvRRimum,
We do use 2 separate ES indices as drawn in the schema, but currently we make them share the same ES instance, for simplicity. It will stay this way in local development, but probably not in production.
If you wanted, you could already spin 2 different ES clusters and point to them separately in config.py
from cosr-back.
How can i create those 2 ES instances ?
Going in AWS -> Elasticsearch -> create new ?
And then point to the IP and it will work ?
Thats all that is needed ?
Also i dont see any examples of cosr-config.json ( https://github.com/commonsearch/cosr-back/blob/master/cosrlib/config.py files tells me that.
from cosr-back.
Yes, any ES instance will work. cosr-ops
contains tools to spin up your own cluster but you could use the AWS hosted instances, or even https://www.elastic.co/cloud ; Let me reiterate that you can reuse the same one for simplicity.
cosr-config.json
is just a JSON file with the same keys as the config.py file (which only contains the default values).
from cosr-back.
Okey, i created 2 clusters in Elasticsearch services section in AWS.
How do i deploy test data in there ?
from cosr-back.
Depends which part of common search you want to work on?
For cosr-back
, you should do make import_local_data
to get started and then make reindex1
For cosr-front
, you can use the devindex in the local ES cluster started by Docker, see here: https://github.com/commonsearch/cosr-front/blob/master/INSTALL.md
from cosr-back.
I will do both! I will leave an update if i encounter error i sucessfully set-everything up.
After i set-everything up, i think i will create guide for complete deployment of commonsearch and make pull request on the bugs i descovered.
Also thanks for response, means a lot!
from cosr-back.
That's great, thanks!
from cosr-back.
Hello,
I created 2 Es clusters, i made the make import_local_data, but when i run make reindex1, i get the following error:
(venv) root@ip-172-30-0-177 back]$ make reindex1
./scripts/elasticsearch_reset.py --delete
No handlers could be found for logger "elasticsearch"
^@Traceback (most recent call last):
File "./scripts/elasticsearch_reset.py", line 19, in <module>
indexer.empty()
File "/cosr/back/cosrlib/indexer.py", line 42, in empty
self.es_docs.create(empty=True)
File "/cosr/back/cosrlib/es.py", line 103, in create
self.empty()
File "/cosr/back/cosrlib/es.py", line 57, in empty
if self.indices().exists(index=self.index_name):
File "/cosr/back/venv/lib/python2.6/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
return func(*args, params=params, **kwargs)
File "/cosr/back/venv/lib/python2.6/site-packages/elasticsearch/client/indices.py", line 226, in exists
params=params)
File "/cosr/back/venv/lib/python2.6/site-packages/elasticsearch/transport.py", line 329, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/cosr/back/venv/lib/python2.6/site-packages/elasticsearch/connection/http_urllib3.py", line 102, in perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError((<urllib3.connection.HTTPConnection object at 0x2742d10>, 'Connection to 52.3.91.82 timed out. (connect timeout=10)')) caused by: ConnectTimeoutError((<urllib3.connection.HTTPConnection object at 0x2742d10>, 'Connection to 52.3.91.82 timed out. (connect timeout=10)'))
make: *** [reindex1] Error 1
My config:
_defaults = {
# HTTP URL of both ElasticSearch servers
"ELASTICSEARCHTEXT": "http://52.23.11.145",
"ELASTICSEARCHDOCS": "http://52.3.91.82",
# Host:port of the URLserver instance, or "local" for direct import on the same node
"URLSERVER": "local", # "192.168.99.100:9702"
# Host:port of the Explainer instance
"EXPLAINER": "0.0.0.0:9703", # "127.0.0.1:9703"
# Environment type: prod, staging, local, ci, ...
"ENV": "local",
# Should we use files in tests/testdata/ as datasources? ("0" or "1")
"TESTDATA": "0",
# Path to the parent directory of cosrlib
"PATH_BACK": os.path.dirname(os.path.dirname(__file__)),
# Path to the local-data directory
"PATH_LOCALDATA": os.path.join(os.path.dirname(os.path.dirname(__file__)), "local-data")
}
Why cant it connect ? Maybe its the difference in ES versions ?
from cosr-back.
And when i open kibana, it asks me to: Please specify a default index pattern
from cosr-back.
Related Issues (20)
- Questions on deployment HOT 1
- Spark-submit uses only 1 core. HOT 4
- Errors During Installation HOT 3
- Speed up Travis builds HOT 1
- Improve host-level PageRanks HOT 1
- Investigate MyHTML parser
- Add GDELT document source
- Add a Github document source HOT 2
- Add a Reddit data source
- Add Stack Overflow document source HOT 1
- Add new Malware/Phishing Blacklists HOT 1
- Advertising Lists HOT 2
- Add Makefile commands to save/load elasticsearch snapshots
- PageRank & other jobs: check if output directory already exists
- Integrate the new Common Crawl News dataset HOT 1
- Add docker-compose for the local tests
- Deduplicate URLs in backlinks plugin results
- Investigate annotating all the code with mypy
- Getting error running 'make virtualenv' HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cosr-back.