urppchusj/impactpharmacie

Impact Pharmacie

Impact Pharmacie

We identify high-quality publications demonstrating the impact of pharmacists on health outcomes. Impact Pharmacy is transparent, reproducible, and evidence-based.

---

Context

The amount of scientific literature describing the impact of pharmacists on health outcomes is increasing. Pharmacists have limited time to keep up with current evidence. We offer a curation service for these publications by identifying high-quality publications through a rigorous, transparent and reproducible methodology. We broadcast these publications weekly in our mailing list. Through this methodology, we are building a dataset of publications related to pharmacy practice. The features of this dataset are the title and abstracts of all publications retrieved by our search strategy. The labels indicate whether or not this publication met our inclusion criteria (as determined indepentently by two reviewers and through consensus when there is disagreement), the study design, the pharmacy practice field for which this publication is applicable as well as the practice setting (see details in methodology). This dataset grows every week. The complete dataset is available as csv files updated weekly within the present repository under the data/second_gen directory. Alternatively, the complete extraction logs and machine learning predictions and ground truths, as well as ratings for the current week, can be found in a Google Spreadsheet here. We develop machine learning models to help determine if a given abstract meets our inclusion criteria, and to predict its design, field and setting labels. We plan to improve upon these models as our new dataset grows. This repository includes

• The raw data extracted from the first generation platform
• The dataset from our current generation platform
• The code used to transform the original data into a machine learning compatible dataset
• The code to train and evaluate our machine learning models
• The code to build our current dataset as well as to update our website and to generate our mailing list.

Files

Files contained in the credentials directory

These files are examples (containing no actual credentials) of how to create json files containing the authentication credentials for each API acessed by our scripts.

Files contained in the data/first_gen directory

These files are the raw HTML scraped from the first generation Impact Pharmacie website and are used by the create_impact_dataset.ipynb notebook to generate the machine learning dataset.

Files contained in the data/second_gen directory

These CSV files contain:

• A log of the data extractions made from PubMed (extraction_log.csv).
• The titles, abstract texts and ratings of all papers evaluated with our methodology (ratings.csv).
• The machine learning tag predictions and ground truths for papers which met our inclusion criteria (predictions.csv).
• The exclusion thresholds for the model that predicts if a paper can be automatically excluded or needs to be reviewed manually, as well as associated metrics computed when determining the threshold (thresholds.csv).

create_impact_dataset.ipynb

This notebook creates a dataset containing as features the titles and abstracts from publications included in the original Impact Pharmacie website, as well as titles and abstracts from publications that were not included.

inclusion_basic_models.ipynb

This notebook uses scikit-learn to train and evaluate various classifier models on the second generation dataset to determine if the paper should be included in Impact Pharmacie.

inclusion_transformers.ipynb

This notebook uses Hugging Face Transformers to train and evaluate transformer models on the second generation dataset to determine if the paper should be included in Impact Pharmacie. The notebook also includes the training of production models.

labels_basic_models.ipynb

This notebook uses scikit-learn to train and evaluate a large number of "classic" machine learning models on the original dataset for label predictions (design, field and setting of included papers).

labels_transformers.ipynb

This notebook uses Hugging Face Transformers to train and evaluate transformer models on the original dataset for label predictions (design, field and setting of included papers). These models performed better than the basic models, therefore the notebook also includes the training, evaluation and explainability testing of models on the test set as well as the training of production models.

update_data.py

This script is used to build our new dataset by performing an automated PubMed search.

update_site.py

This script is used to update our website from our dataset and to generate our newsletter.

update_inclusion_model.py

This script is used to update the model that predicts if abstracts can be excluded automatically or need to be reviewed for inclusion.

Prerequisites

Developed using Python 3.9.7

Requires:

• BeautifulSoup
• Gspread
• Hugging Face Datasets
• Hugging Face Transformers
• Numpy
• Pandas
• PyTorch
• Ray Tune
• Scikit-learn
• Scipy
• Shap
• Tqdm

Contributors

Maxime Thibault Cynthia Tanguay

References

• First generation Impact Pharmacie platform
• New Impact Pharmacie website

License

GNU GPL v3

Copyright (C) 2022 Maxime Thibault, Cynthia Tanguay.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

impactpharmacie's Issues

test google spreadsheet is now irrelevant, remove from code

Integrate Google credential verification into the scripts

Currently the script to update Google credentials ( ./credentials/get_gspread_creds.py ) is separate from the data update and site update scripts. We need to move this into the scripts by verifying that the credentials are still valid and if not go through OAuth to update the credentials.

Data verification step in update_site.py script will not work if labels are placed in "consensus_reason" fields by mistake

Integrate two new verification steps for each label:

1. to make sure that consensus reason is documented only if consensus field is used
2. if consensus is empty, ground_truth_1 and ground_truth_2 are not empty.

Adjust code to enable data sources other than PubMed

LinkedIn publication for Impact Briefing

In publication code for impact briefing posts, add linkedin post (french only)

Move data storage away from Google spreadsheet to a CSV file into the repo

At present the data is stored entirely in a Google spreadsheet, but the data volume will soon make this inconvenient. We need to move the data storage into a CSV file integrated within this repo, and use the Google Spreadsheet only to store and rate the present week's data.

Review inclusion criteria

• Adapt criteria following deployment of machine learning model for inclusion prediction

Add potentially predatory journals

• Update code to include:

  • When updating site, obtain and store predatory publisher and journal list
  • Compare lists to previously stored versions and alert if change.
  • When obtaining pubmed data for publication, follow DOI link and compare result to list.
  • If potentially predatory, alert to determine if tag or not
  • If tagged, include Wordpress tags, link in impact briefing and publication post to method page

• Update website

  • New page to explain
  • Link in method

Update the prediction code in update_site.py

The code used to make predictions in update_site.py is wonky. Update to make better use of Transformers Pipelines like in the new version of update_data.py.

Add new themes (diabetes, pneumology)

• Retrospectively add tags to previously included papers.
• Update methods to include new tags
• Blog post

Ratings check step will not identify a selected paper that was not tagged

Add a verification step to detect included articles without tags

PubMed API will sometimes return incomplete data, most notably the journal in MedlineTA field

Add condition to retry pubmed retrieval when incomplete until complete or timeout.

Update website methodology to account for data storage in the repo

Update the methodology page on the website to reflect that:

1. The dataset is now "officially" stored as CSV files in the GitHub repo (under the /data/second_gen directory)
2. The spreadsheet now only contains ratings for the current week, other ratings will only be found in the CSV files in the repo.

Periodically check if previously evaluated papers have changed

• Explore with notebook if we have previously evaluated papers have changed

Add inclusion model into workflow

Add following enhancements:

-When updating data, call predictions from the inclusion model and log the inclusion score, suggestion based on thresholds and model version.
-For now, hide the three new columns after updating the data to run these predictions silently for prospective model evaluation.
-After updating the website, create a new inclusion model version with the newly updated inclusion data.