kevinlynx/dhtcrawler

dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents.

dhtcrawler

dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents. The program save all torrent info into database and provide an http interface to search a torrent by a keyword.

Usage

• Download mongodb and start it with text search, i.e:

    mongod --dbpath db --setParameter textSearchEnabled=true
  

• Download dhtcrawler source code

• Use rebar to download and install all dependent libraries

    rebar get-deps
  

• compile

    rebar compile
  

• starting the erlang shell with the ebin path added

    erl -pa ebin
  

• start dhtcrawler

    crawler_app:start()
  

• start the http front-end

    crawler_http:start().
  

• Open a web browser and point to localhost:8000/index.html

Config

see priv/dhtcrawler.config.

NOTE, when you change node_count value in dhtcrawler.config, you'd better delete all files saved in dhtstate directory.

dhtcrawler's Issues

Testing on ubuntu16, and crashed.

OS version

root@aliyun2 dhtcrawler  # lsb_release -a
LSB Version:	core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial

Command History

   18  cd /mnt/app/dhtcrawler
   19  ls
   20  ll
   21  rebar get-deps
   22  rebar compile
   23  sudo service mongodb start
   24  erl -pa ebin
   25  ps -ef|grep erlang
   26  ls
   27  ll
   28  ls priv
   29  ls priv/dhtcrawler.config
   30  vi priv/dhtcrawler.config
   31  netstat -tupln
   32  mongo
   33  ll
   34  vi crash.log
   35  rebar get-deps
   36  tree .
   37  rebar compile
   38  erl -pa ebin
   39  cat crash.log
   40  lsb_release -a

Crash log

root@aliyun2 dhtcrawler  # cat crash.log 

=ERROR REPORT==== 30-Mar-2018::14:18:10 ===
** Generic server db_conn_pool terminating 
** Last message in was timeout
** When Server state == {"localhost",27017,5}
** Reason for termination == 
** {'module could not be loaded',
       [{mongo_connection,start_link,[{"localhost",27017}],[]},
        {db_store_mongo,init,2,[{file,"src/db_store_mongo.erl"},{line,24}]},
        {db_conn_pool,'-init_conn/1-lc$^0/1-0-',3,
            [{file,"src/db_conn_pool.erl"},{line,73}]},
        {db_conn_pool,init_conn,1,[{file,"src/db_conn_pool.erl"},{line,73}]},
        {db_conn_pool,handle_info,2,[{file,"src/db_conn_pool.erl"},{line,66}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

=INFO REPORT==== 30-Mar-2018::14:18:10 ===
    application: dhtcrawler
    exited: {{shutdown,
                 {failed_to_start_child,torrent_index,
                     {undef,
                         [{mongo_connection,start_link,
                              [{"localhost",27017}],
                              []},
                          {db_store_mongo,init,2,
                              [{file,"src/db_store_mongo.erl"},{line,24}]},
                          {db_conn_pool,'-init_conn/1-lc$^0/1-0-',3,
                              [{file,"src/db_conn_pool.erl"},{line,73}]},
                          {db_conn_pool,init_conn,1,
                              [{file,"src/db_conn_pool.erl"},{line,73}]},
                          {db_conn_pool,handle_info,2,
                              [{file,"src/db_conn_pool.erl"},{line,66}]},
                          {gen_server,try_dispatch,4,
                              [{file,"gen_server.erl"},{line,615}]},
                          {gen_server,handle_msg,5,
                              [{file,"gen_server.erl"},{line,681}]},
                          {proc_lib,init_p_do_apply,3,
                              [{file,"proc_lib.erl"},{line,240}]}]}}},
             {crawler_app,start,[normal,[]]}}
    type: temporary

=ERROR REPORT==== 30-Mar-2018::14:18:30 ===
** Generic server db_conn_pool terminating 
** Last message in was timeout
** When Server state == {localhost,27017,2}
** Reason for termination == 
** {'module could not be loaded',
       [{mongo_connection,start_link,[{localhost,27017}],[]},
        {db_store_mongo,init,2,[{file,"src/db_store_mongo.erl"},{line,24}]},
        {db_conn_pool,'-init_conn/1-lc$^0/1-0-',3,
            [{file,"src/db_conn_pool.erl"},{line,73}]},
        {db_conn_pool,init_conn,1,[{file,"src/db_conn_pool.erl"},{line,73}]},
        {db_conn_pool,handle_info,2,[{file,"src/db_conn_pool.erl"},{line,66}]},
        {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},
        {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}