pop-os/url-scraper

Rust crate for scraping URLs from HTML pages

url-scraper

Rust crate for scraping URLs from HTML pages.

Example

extern crate url_scraper;
use url_scraper::UrlScraper;

fn main() {
    let directory = "http://phoronix.com/";

    let scraper = UrlScraper::new(directory).unwrap();
    for (text, url) in scraper.into_iter() {
        println!("{}: {}", text, url);
    }
}

url-scraper's Issues

Support Async I/O with Futures

The current API only exposes a synchronous API. An asynchronous API based around reqwest::async::Client would be ideal as well.

Cannot Build: Error Compiling CSS Parser

I have tried to reproduce the code exactly in the url_scraper documentation, but am unable to build the program.

I think it is an error with the cssparser dependency. Any ideas how to fix?

Error:

   Compiling cssparser v0.24.1
error[E0506]: cannot assign to `self.input.cached_token` because it is borrowed
   --> /home/jared/.cargo/registry/src/github.com-1ecc6299db9ec823/cssparser-0.24.1/src/parser.rs:572:17
    |
547 |     pub fn next_including_whitespace_and_comments(&mut self) -> Result<&Token<'i>, BasicParseError<'i>> {
    |                                                   - let's call the lifetime of this reference `'1`
...
560 |             Some(ref cached_token)
    |                  ---------------- borrow of `self.input.cached_token` occurs here
...
572 |                 self.input.cached_token = Some(CachedToken {
    |                 ^^^^^^^^^^^^^^^^^^^^^^^ assignment to borrowed `self.input.cached_token` occurs here
...
584 |         Ok(token)
    |         --------- returning this value requires that `self.input.cached_token.0` is borrowed for `'1`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0506`.
error: could not compile `cssparser`.
warning: build failed, waiting for other jobs to finish...
error: build failed

To Reproduce:

My main.rs is:

extern crate url_scraper;
use url_scraper::UrlScraper;

fn main() {
    let directory = "http://phoronix.com/";

    let scraper = UrlScraper::new(directory).unwrap();
    for (text, url) in scraper.into_iter() {
        println!("{}: {}", text, url);
    }
}

My Cargo.tom dependencies are:

[dependencies]
url-scraper = "0.1.1"