intelcomph2020/semanticscholarimporter

Project to import and save to Postgres / Parquet data from SemanticScholar.

semanticscholarimporter's Issues

Add option to create indexes, and don't do it while creating schema

If indexes are created from the beginning, data loading on the postgres db will be very slow. It is better to load data first, and then create indexes once the data is ready. Two choices are given:

• Preferred: Create indexes on each table after ingesting all data relevant to that table
• Other: Create a new option in the menu to create indexes at the very end

Create option to enrich citation table

Use Semantic Scholar API to enrich citation table with the following information:

• isInfluential
• BackIntent
• MethodIntent
• ResultIntent

This information is very useful, but probably costly to obtain. So:

1. Check if part of this information could be obtained online, or if we have access to the software they use and it is feasible to run it
2. Implement the possibility of enriching using the API (is this already implemented?). Add for this second case the possibility of filtering with some paper metadata. E.g., enrich only information from papers beloinging to a certain S2 Field.

Is it really necessary to have a different class for postgres than for MySQL?

The problem of having to classes, one for mysql and another for postgres is that changes done in one of them should be migrated to the other. One ends up having two different classes with many functions in common.

Would it be feasible to have a parent class with common functions, and then inherit with two different classes ?

Priority of this issue is low ... we need to discuss when and if this is implemented at all

Enrich Author metadata with other information

We can enrich author information with:

• influentialCitationCount: Very soundy, but probably costly to obtain, since they need to be crawled from SS using their API.
• ORCID data
• Affiliation and/or whether the author has Spanish affiliation or not

The priority of this enhacement is low