anapaulagomes/peneira

It's time to sift through some articles 🤭

peneira

It's time to sift through some articles 🤭

With this CLI you can search for papers for your research in different sources and export the results.

DISCLAIMER: This is a work in progress. The code is under active development and it's not ready for production use.

Available sources

• OpenAlex
• Semantic Scholar

...and many more to come! Feel free to contribute. There is a world of papers out there!

OpenAlex

Here are some details about this source:

• OpenAlex data sources
• OpenAlex search syntax
• Open Alex filters
• Open Alex rate limits and authentication

This library obeys the rate limits of the OpenAlex API (10 requests per second).

Usage

CLI

You can interact with the CLI using peneira. For example, to search for papers on "artificial intelligence" and "syndromic surveillance" and save the results to a file, you can run:

peneira -s open_alex -s semantic_scholar --filename my-papers.json

You will be prompted to enter the search query for each source. The lib will search for papers in OpenAlex and Semantic Scholar and store it in a file named my-papers.json. If no filename is provided, the results will be stored to results.json.

You have also the option of export it to a bibtex file:

peneira -s open_alex -s semantic_scholar --format bibtex --filename my-papers.bib

peneira's Issues

Publish to PyPI

TODO:

• Send a message to https://pypi.org/project/pn/ owner ✅
• Send a message to https://pypi.org/project/pnr/ owner ✅
• Publish from Github Actions
• Add badges to repo

Export to CSV

Export to files like JSON or CSV.

• JSON
• CSV

Useful links:

https://github.com/Tinche/aiofiles

Remove file from bibtex export

It shows the local file path, which is not helpful for a review database file.

Add support to CORE API registered API KEY

References:
https://api.core.ac.uk/docs/v3#section/Rate-limits

• Registered Personal users: 10,000 tokens per day, maximum 10 per minute
• Registered Academic (Supporting / Sustaining members) and Non-academic users: 200k tokens per day
• Monitor your current api limit by looking for our customised HTTP headers: X-RateLimitRemaining, X-RateLimit-Retry-After, X-RateLimit-Limit

What needs to be done here:

• Read the API Key from the env var
• If there is a key, increase the rate limits to a maximum of 10 per minute
• If there is a key, check the headers, calculate if it is close to the limit and show warning messages

Publish to PyPi from Github Actions

Automate the process of publishing new releases.

Add log

It will be helpful to debug errors and warnings.

Add support to Semantic Scholar registered API KEY

Reference:
https://www.semanticscholar.org/product/api#api-key
https://api.semanticscholar.org/api-docs/
https://www.semanticscholar.org/product/api/tutorial#key

What needs to be done here:

• Read the API Key from the env var
• If there is a key, set the header x-api-key (case-sensitive)
• If there is a key, increase the rate limits to (received via email):

1 request per second for the following endpoints: 

		*	/paper/batch
		*	/paper/search
		*	/recommendations

	*	10 requests / second for all other calls

Add CORE as source

URL: https://api.core.ac.uk/docs/v3
Rate limits: https://api.core.ac.uk/docs/v3#section/Rate-limits
Search syntax: https://api.core.ac.uk/docs/v3#section/How-to-search

Some authors are not exported to bibtex

Source: open alex

Whenever the file is imported to tools like Zotero, the author's names are not there. This issue is for verifying this.

Export results to bibtex

It will bring compatibility with tools like Zotero and Research Rabbit.

https://www.bibtex.org/Format/
https://bibtexparser.readthedocs.io/en/main/quickstart.html

Month should be an integer in bibtex entry

legacy month field 'Apr' in entry 'Oyebode_Ndulue_Adib_Mulchandani_Suruliraj_Orji_Chambers_Meier_Orji_2021' is not an integer - this will probably not sort properly.

Instead of Apr it should be 4.

Export items to a database

A simple and easy solution would be sqlite.

https://docs.python.org/3/library/sqlite3.html
https://github.com/omnilib/aiosqlite

Configure a CHANGELOG

Adapt search string using LLM

Each source has its own way of writing a search string. To avoid adding a query for each one, LLM generates the queries for each source, using few-shot prompting to train it with examples.

We can also ask the user for confirmation and offer the possibility of passing any queries or inputting a fix for the one generated by AI.

peneira aborts

I have errors when I try to run the module.

C:\Users\niels>peneira -s open_alex -o bibtex -f my-papers.bib
Please enter the search string for open_alex: 'ceramic spheres' uhpc
open_alex: Fetching articles for OPEN_ALEX... 24 papers distributed in 1 pages.
Executing the search...

I here attach the logging of the error messages.

Traceback (most recent call last).txt

/Frank

Add CrossRef

https://api.crossref.org/swagger-ui/index.html

Add PubMed as source

Add Semantic Scholar

URL: https://api.semanticscholar.org/

Add PubMed

Docs: https://www.ncbi.nlm.nih.gov/books/NBK25499/#_chapter4_ESearch_
Example of an URL: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&retmode=json&retmax=1000&term=cost+effectiveness+of+mirtazapine[title]

Create configuration file

This will be useful to config things about the source and the outputs.

Useful links:
https://docs.python.org/3/library/configparser.html