shawnshaligram/scraper

Configurable Python Web Scraper

Scraper

Minimalist Python DOM Scraper

Description

This module is an easy to use HTML/XML scraper. It supports both XPath and Regular Expression retrieval.

Once you have a file you want to extract information from, you can extract multiple pieces of information with a simple function call.

You should obtain the files you want to scrape by your own ways.

Installation

pip install scraper

Usage

Scrape using xpath:

import scraper
import requests

content = requests.get('https://github.com/explore').content

conf = {'trending-repos' : {'xpath' : '//ol/li/h3/a[2]/@href'}}

scraper.scrapes(content, conf)

>>> {'trending-repos': ['/jamescryer/grumble.js', '/dominictarr/JSON.sh', '/JamieLottering/DropKick', '/harvesthq/chosen', '/velvia/ScalaStorm']}

Scrape using regexp:

import scraper
import requests

content = requests.get('http://wiki.nomasnumeros900.com/Air_Liquide').content

conf = {
        'numbers':
            {'regexp': '91[\s\d]+',
             'transf': [lambda x: x.strip()],
             'encoding': 'utf-8'}
        }

scraper.scrapes(content, conf)

>>> {'numbers': [u'915 029 300', u'915 029 560', u'915 029 330', u'91']}