iwikal/iwicrawl

An indexer for open directories

iwicrawl's Issues

Throttling

This will take some design work. Many behaviours are possible. Leave a comment if you have an idea for how it should work.

Bintray ":" urls causes 404 errors

iwicrawl https://jcenter.bintray.com/com/facebook/soloader/soloader/0.5.0

ERROR - 'https://jcenter.bintray.com/com/facebook/soloader/soloader/0.5.0/:soloader-0.5.0-javadoc.jar': 404 Not Found
ERROR - 'https://jcenter.bintray.com/com/facebook/soloader/soloader/0.5.0/:soloader-0.5.0-sources.jar': 404 Not Found
ERROR - 'https://jcenter.bintray.com/com/facebook/soloader/soloader/0.5.0/:soloader-0.5.0.aar': 404 Not Found
0                    https://jcenter.bintray.com/com/facebook/soloader/soloader/0.5.0/
ERROR - 'https://jcenter.bintray.com/com/facebook/soloader/soloader/0.5.0/:soloader-0.5.0.pom': 404 Not Found

Add timeout

Right now, it will wait forever, which should probably still be an option, but by default it should timeout and maybe retry?

Retry on certain errors

Such as 404's and 500's.

Rewrite with async/await

I'm not sure, but I think it will let me get rid of some of the lifetime cheats I used.