Web Crawler - the service, which allow you to search the crawled websites in Wikipedia
- node v10.x.x
- npm v7.x.x
- docker v20.x.x
- docker-compose v1.25.x
- Web crawler service - to crawl and store the links and page data
- Search backend api - service to search the links
- search frontend - frontend for search
For windows you'll need to increase memory for ES via wsl
wsl -d docker-desktop
then
_sysctl -w vm.max_map_count=262144_
For linux _sysctl -w vm.max_map_count=262144_
and chmod 777 .docker/elasticsearch -R
to allow access to elasticSearch folder from docker
In web-crawler-service folder run sudo docker-compose run php composer install
In web-crawler-service run docker-compose up
to initialize running
Start the crawling proceess via sudo docker-compose run php bin/console crawl
. It will proccess of adding message with pages metadata to RabbitMQ
You need to check redis port in the log and directly change it in "web-crawler-service/src/Service/CrawlService.php". This is because Redis-clusted docker issue bug
You need to activate workers to store and indexing the data from the RabbitMQ. For this you need to run docker run worker
with docker-compose running in the background
In search-service/search-backend-api run docker-comppose up
and install the node modules
In search-service/search-fronend install node modules and run npm start
with your local nodejs
You are done :)
- RabbitMQ Admin - http://localhost:15672/
- Redis Admin ( RedisInsight ) - http://localhost:8001/
- Elastic Search Admin ( elasticsearchHQ )- http://localhost:5000/
- App frontend ( http://localhost:4222/)
- App backend ( http://localhost:3000/)
- Page Rank ( sort pages by count of links, which leads to this particular link )
- Add software load balancer