pranftw/openreview_scraper

Scrape papers from OpenReview using OpenReview API

OpenReview Scraper

Scrape papers from top conferences like ICML, ICLR, NeurIPS, etc using OpenReview API, by searching for specific keywords in title, abstract or keywords in the submissions and save them to a CSV file.
Brings down the time taken to gather papers from several hours to a few minutes through automation

Installation

git clone https://github.com/pranftw/openreview_scraper.git # clone repo
python -m venv venv # create virtual environment
source venv/bin/activate # activate virtual environment
pip install -r requirements.txt # install requirements
cp config.py.example config.py # enter your OpenReview credentials in config.py

Example

from scraper import Scraper
from extract import Extractor
from filters import title_filter, keywords_filter, abstract_filter
from selector import Selector
from utils import save_papers, load_papers


years = [
    '2023',
    '2022',
    '2021'
]
conferences = [
    'NeurIPS'
]
keywords = [
    'generalization'
]

def modify_paper(paper):
  paper.forum = f"https://openreview.net/forum?id={paper.forum}"
  paper.content['pdf'] = f"https://openreview.net{paper.content['pdf']}"
  return paper

# what fields to extract
extractor = Extractor(fields=['forum'], subfields={'content':['title', 'keywords', 'abstract', 'pdf', 'match']})

# if you want to select papers manually among the scraped papers
# selector = Selector()

# select all scraped papers
selector = None

scraper = Scraper(conferences=conferences, years=years, keywords=keywords, extractor=extractor, fpath='example.csv', fns=[modify_paper], selector=selector)

# adding filters to filter on
scraper.add_filter(title_filter)
scraper.add_filter(keywords_filter)
scraper.add_filter(abstract_filter)

scraper()

# if you want to save scraped papers as OpenReview objects using pickle
save_papers(scraper.papers, fpath='papers.pkl')
saved_papers = load_papers(fpath='papers.pkl')

openreview_scraper's Issues

Missing papers from ICLR2023

Hi, I tried your package on ICLR2023 with the keyword "disentangling" but the paper "Continual Unsupervised Disentangling of Self-Organizing Representations" is not returned, which seems wrong. The paper is clearly accepted https://openreview.net/forum?id=ih0uFRFhaZZ.

Could you please look into this issue? Thanks!

The code does not work for ICML.

https://openreview.net/group?id=ICML.cc/2023/Conference

The code does not work for ICML.

Add reviewer comments and author response

This repo is really nice. I would like to add reviewer comments, rating score, confidence and author response to analysize it. How can I do for it. Thank you.

Doesn't work for ICLR 2024?

Great repo!

I realize it doesn't seem to work for ICLR 2024 when I tried:

years = [
    '2024'
]

conferences = [
    'ICLR'
]
keywords = [
    'language model'
]

def modify_paper(paper):
  paper.forum = f"https://openreview.net/forum?id={paper.forum}"
  paper.content['pdf'] = f"https://openreview.net{paper.content['pdf']}"
  return paper

# what fields to extract
extractor = Extractor(fields=['forum'], subfields={'content':['title', 'keywords', 'abstract', 'pdf', 'match']})

# if you want to select papers manually among the scraped papers
# selector = Selector()

# select all scraped papers
selector = None

scraper = Scraper(conferences=conferences, years=years, keywords=keywords, extractor=extractor, fpath='examples.csv', fns=[modify_paper], selector=selector)

# adding filters to filter on
scraper.add_filter(title_filter)
scraper.add_filter(keywords_filter)
scraper.add_filter(abstract_filter)

scraper()

But it works fine when I change the year from 2024 to 2023. Any idea why?