maddiem4/scrap

Scraper written in Golang for high-performance site testing/validation.

scrap

Scraper written in Golang for high-performance site testing/validation.

See http://godoc.org/github.com/campadrenalin/scrap for more information. This library was originally developed to be part of the automated testing infrastructure for www.inspire.com.

scrap's Issues

More builtin bucket types

We have a few in the ABCC scraper that deserve to be in mainline, with full test suites and such. There are also complementary types that deserve to be created.

// Bucket that contains buckets - only returning true if all sub-buckets
// return true. Buckets are tested in order.
type AllBucket struct {
    Children []scrap.Bucket
}

// Bucket that contains buckets - only returning true if any sub-bucket
// returns true. Buckets are tested in order.
//
// Behavior is short-circuiting. A later bucket will not be checked if an
// earlier bucket succeeds. So while the first bucket will be tried every
// time, the last bucket may never be tried, depending on the results
// of earlier buckets.
type AnyBucket struct {
    Children []scrap.Bucket
}

// Bucket that rejects all URLs with a specific prefix, and
// accepts everything else.
type RejectPrefixBucket string

// Reject URLs that are exact matches
type RejectExactBucket string

// Reject URLs with a specific postfix
type RejectPostfixBucket string

Record redirects

Use the HTTP.Client.CheckRedirect pluggable callback to record redirect chain information, so that the RouteActions will be able to access that data.

For now, include this in ScraperRequest.Stats. At some point, though, we'll probably want a separate ServerResponse object, with a Parse function and raw response data, etc.

Backup and Reset for resp.Response.Body

Should be able to, optionally, create a copy of resp.Response.Body as a *bytes.Buffer, such that you can both Grep and Parse.

ServerResponse.Grep

Search the content of the response for the given pattern.

Should be compatible with the backup copy mechanism.

Support 401 Auth for all requests

I could implement a more fine-grained solution, but I think the bulk solution will be more than enough for now, and I'd rather not implement complexity that I'm not actually going to use.

Node.Text()

Get the text content of a node and its descendents.

Add referer info to ScraperRequest object

Frankly, it would be nice to have a generic request context provided by the queuer, which could be basic, or involve more detailed construction of the request to convey even more contextual information (which could also be useful for bucketing).

Important thing, right now, is referer info. So maybe just a struct with a Referer string and a Misc map[string]interface{}.

Change RouteAction signature to use ServerResponse

We should be handing our downstream code a more flexible object to work with, but still not far from current convenience. Something that does not parse HTML when the downstream code doesn't WANT to parse it.

func Handler(req ScraperRequest, resp ServerResponse) {
    root := resp.Parse() // Existing use case, consumes resp.Body to make Node
    title_element := root.Find("title")[0]
    req.Remarks.Printf("title = %s\n", title_element.Text())
    req.Remarks.Printf("Started %v, lasted %v\n",
        req.Stats.Started,
        req.Stats.Duration,
    )

    expected_bytes := resp.Response.ContentLength
    got_bytes := resp.BytesRead
    if got_bytes != expected_bytes {
        req.Remarks.Printf("Expected %d bytes, got %d\n",
            expected_bytes,
            got_bytes
        )
    }

    root.Find("a").Queue()
}

var my_regexp = regexp.MustCompile("foo")
func GreppyHandler(req ScraperRequest, resp ServerResponse) {
    data := ioutil.ReadAll(resp.Body) // Or, resp.ReadAll()
    matches := my_regexp.FindAll(data, -1)
    req.Remarks.Printf("Found %d matches\n", len(matches))
}

This also demonstrates some unrelated, but planned, API improvements (for example, Node.Text()).