rohanbk/mountain-project-scraper

Python scrapy-based repository for mining information associated with MountainProject

mountain-project-scraper's Issues

Create a new spider to scrape all route information NOT area information

This is for future work, but it will help solve some interesting data-science ideas.

Create ReadTheDocs entry

https://readthedocs.org/

Add Travis.ci infrastructure

https://travis-ci.org/getting_started

Capture parents and children for each area

Each area-page has knowledge of their parents (i.e. breadcrumbs) and children (e.g. the list of all sub-areas in the links side-menu). Capture this information using the ID of each area.

URLs and selectors are outdated

    domain = 'https://www.mountainproject.com'

    # URL should be preceded by a /
    # e.g. /destinations or /v/STATENAME/ID
    relativeURL = '/v/hawaii/106316122'

    start_urls = [domain + relativeURL]
    allowed_domains = ['mountainproject.com']
    rules = [
        Rule(
            LinkExtractor(allow='v/(.+)'),
            callback='parse',
            follow=True
        )
    ]

The /v/ URLs redirect to a new scheme:

2023-01-15 11:19:22 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.mountainproject.com/viewer-old/106316122> from <GET https://www.mountainproject.com/v/hawaii/106316122>
2023-01-15 11:19:22 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://www.mountainproject.com/area/106316122/hawaii> from <GET https://www.mountainproject.com/viewer-old/106316122>

        if self.relativeURL != '/destinations':
            # use the following links variable if testing from an individual state page (e.g. WA states routes)
            links = response.css('#viewerLeftNavColContent a[target="_top"] ::attr(href)').extract()

<div id="viewerLeftNavColContent" class="rspCollapsedContent"> was present in old pages: https://web.archive.org/web/20161122233413/http://www.mountainproject.com/v/alabama/105905173

but no longer.

        else:
            # use the following links variable if testing from the homepage
            links = response.css('span.destArea a::attr(href)').extract()

<span class="destArea"> was present on old homepage:

https://web.archive.org/web/20171016232313/https://www.mountainproject.com/

but no longer.

Create a unique ID for each entry using URL ID

All Mountain Project entities (e.g. states, larger geographical areas, micro-zones, routes, boulder problems, etc.). have a unique URL ID associated with them. This ID should be captured in each Scraped entry.

Fix README.md

Convert parents + child areas to be links instead of human-readable names

Update process_latitude_longitude function to skip pages with no lat/long