gusty/scrapem

A monadic web scraping library

ScrapeM

A monadic web scraping library

This library makes web scraping easier by providing ways to automatically maintain state through different request, handling cookies, form submission and http headers.

One function to scrap'em all

This is essentially a single-function library which integrates many existing libraries and present several ways to approach web scraping by using different monads.

All other common functions used here come from different libraries like FSharp.Data, Http.fs and F#+

Scrapes the web with category

It's possible to create stateful linq-style queries which simulates basic user interaction with form submission by using different flavours of State monads. Also sequences expressions are available to integrate the data being extracted from multiple webpages in the same query.

Getting started

Important: At the moment this library is in a 'Prototype' stage

Recommended: Visual Studio 2017 to avoid slow compile time of generic code

In order to try the examples run:

> build.cmd // on windows    
$ ./build.sh  // on unix

Now you can try the sample files:

• Basic query with state handling - Extracts a text from a website with login.
• Basic query with multiple results - Extracts many tables from a website, using a type provider.
• Advanced query with state handling and multiple results - Extracts many texts from a website by using different logins.

scrapem's Issues

Samples are outdated, they don't work anymore.

Description

The websites used for samples already changed, so sample scrapes don't work anymore.

Repro steps

Run the sample.

Expected behavior

Work.

Actual behavior

Doesn't work.

Known workarounds

No workaround. We need to either update the samples or create a sample site which we can control to do the scrapes over that site.

Related information

• Operating system
• Branch
• .NET Runtime, CoreCLR or Mono Version
• Performance information, links to performance testing scripts