ahmia-crawler's Introduction

Ahmia is the search engine for .onion domains on the Tor anonymity network. It is led by Juha Nurmi and is based in Finland. This repository contains crawlers used by Ahmia search engine

Prerequisites

Ahmia-index should be installed and running

Installation guide

Install dependencies:

Ubuntu 16.04

apt-get install build-essential python-pip python-virtualenv
apt-get install libxml2-dev libxslt1-dev python-dev libffi-dev libssl-dev
apt-get install tor polipo

Fedora 23

dnf install @development-tools redhat-rpm-config python-pip python-virtualenv
dnf install libxml-devel libxslt-devel python-devel libffi-devel openssl-devel
dnf install tor polipo

Install requirements in a virtual environment

virtualenv /path/to/venv
source /path/to/venv/bin/activate
(venv)pip3 install -r requirements/prod.txt

Prefer own python HTTP proxy

Look fleet installation here.

Usage

In order to execute the crawler to run permanently:

source /path/to/venv/bin/activate
./run.sh &> crawler.log

Specific run examples

Primary
scrapy crawl ahmia-tor -s DEPTH_LIMIT=3 -s ROBOTSTXT_OBEY=0 -s FULL_PAGERANK_COMPUTE=True
or
scrapy crawl ahmia-tor -s DEPTH_LIMIT=5 -s LOG_LEVEL=INFO
or
scrapy crawl ahmia-i2p -s DEPTH_LIMIT=100 -s LOG_LEVEL=DEBUG
or
scrapy crawl ahmia-i2p -s DEPTH_LIMIT=1 -s ROBOTSTXT_OBEY=0
or
scrapy crawl ahmia-tor -o items.json -t json
or
scrapy crawl ahmia-tor -s DEPTH_LIMIT=1 -s ALLOWED_DOMAINS=/home/juha/allowed_domains.txt -s TARGET_SITES=/home/juha/seed_list.txt -s ELASTICSEARCH_TYPE=targetitemtype

Recommend Projects