cryshana/crycrawler

Cross-platform distributed multi-threaded web crawler

crycrawler's Issues

--version

All version show CryCrawler 1.0.0.
Don't find version string on Source code (zip).

[request] TimePause between send request ro site.

Some sites have a limit on the number of requests for a period of time or do not withstand constant requests. I suppose we need an option to limit request retry. MaxConcurrency limits only the number of simultaneous requests.

DomainBlacklist (third and deep domen) with DomainWhitelist

W: YYY.zz
B: ccn.YYY.zz
Download with blacklist domain.

[request] URL download mask list

Download URL mask.
Example whitelist:
/original/
Example blacklist:
domen.yy/profile/*

Say I want to crawl all images from a chan.sankakuchan tag, How would I use this program to download those images? I miss your api program

Debug mode

Can you public example debug output?

I create test configs, but latest variant config.zip can't start work.
Debug mode don't change log, because i can't find stop error.

C:\0\CryCrawler>C:\0\CryCrawler\CryCrawler.exe -d
[16.08 03:54:28] INFO Program Loading plugins...
[16.08 03:54:28] INFO WorkManager Using local Url source
[16.08 03:54:28] INFO WebGUI WebGUI listening on 127.0.0.1:6001
[16.08 04:00:06] INFO Program Shutting down...
[16.08 04:00:06] DEBUG WebGUI WebGUI listener stopped.
[16.08 04:00:06] INFO WorkManager Dumped 0 backlog items to cache