Add Documentation about new features of the spider

[mgifford]

Add Documentation about new features of the spider

[j-mendez]

Hi @mgifford , I put a message that the spider is more leaning towards an indexer. You can add something like this with a fork. We have the features documented on the README and with the crate. You can take the foundations of the crawling and build variations that is needed using the library.

[mgifford]

No worries. I just read "j-mendez closed this as completed 1 hour ago" as "I built it, done!" vs "this is out of scope, sorry".

I also like the idea of a fork.. Lots of good reasons to move in that direction. Think it helps for the spider-rs project's viability to show folks experimenting with it.

[j-mendez]

@mgifford Here is a bit more direction on the missing links, you can adjust this line here and use the scraper to parse out the link info https://github.com/spider-rs/spider/blob/main/spider/src/page.rs#L186.

testcase: https://github.com/spider-rs/spider/blob/main/spider/src/page.rs#L232.

I hope this helps!