ilias-ant/american-alpine-club-articles

Kaggle dataset: American Alpine Club articles - climbing accidents and major new climbs.

american-alpine-club-datasets

AAC: climbing accidents and major new climbs.

---

Articles from the American Alpine Club's publications: AAJ and ANAC.

This work has been published as a Kaggle dataset.

The project consists of the following components:

• collectors: a Scrapy project, responsible for scraping articles from publications.americanalpineclub.org.
• opensearch-cluster: an OpenSearch cluster, where the scraped articles are indexed.
• publishers: functionality responsible for the publication of the articles index (e.g. as Kaggle dataset).
• notebooks: a collection of Jupyter notebooks, for various dataset-based explorations and applications.
• dataset: the raw dataset, in CSV format.

Citation

@misc{ilias antonopoulos_2022, 
      title={AAC: climbing accidents and major new climbs}, 
      url={https://www.kaggle.com/dsv/4457812}, 
      DOI={10.34740/KAGGLE/DSV/4457812}, 
      publisher={Kaggle}, 
      author={Ilias Antonopoulos},
      year={2022} 
}

american-alpine-club-articles's Issues

fix: a lot of `'publication_year': '201'` scraped

reduce no. of dropped items

From the Scrapy stats of the initial run:

2022-11-07 04:06:04 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 72,
 'downloader/exception_type_count/twisted.internet.error.TimeoutError': 59,
 'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 13,
 'downloader/request_bytes': 16174589,
 'downloader/request_count': 29300,
 'downloader/request_method_count/GET': 29300,
 'downloader/response_bytes': 561325860,
 'downloader/response_count': 29228,
 'downloader/response_status_count/200': 29228,
 'dupefilter/filtered': 6,
 'elapsed_time_seconds': 107792.388109,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2022, 11, 7, 2, 6, 4, 242959),
 'item_dropped_count': 217,
 'item_dropped_reasons_count/DropItem': 217,
 'item_scraped_count': 27617,
 'log_count/DEBUG': 139779,
 'log_count/INFO': 29423,
 'log_count/WARNING': 218,
 'memusage/max': 78548992,
 'memusage/startup': 65867776,
 'request_depth_max': 2,
 'response_received_count': 29228,
 'retry/count': 72,
 'retry/reason_count/twisted.internet.error.TimeoutError': 59,
 'retry/reason_count/twisted.web._newclient.ResponseNeverReceived': 13,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 29299,
 'scheduler/dequeued/memory': 29299,
 'scheduler/enqueued': 29299,
 'scheduler/enqueued/memory': 29299,
 'start_time': datetime.datetime(2022, 11, 5, 20, 9, 31, 854850)}

we have:

...
'item_dropped_count': 217,
'item_dropped_reasons_count/DropItem': 217,
...

Can we reduce this number?

scrape Article referer as well

i.e. the URLs of the form: /articles?page=x

This will ease out debugging, since some of the scraped elements occur in the page-level as well (not only in the article-level)

remove `scraped_at` from final dataset