martinomensio/claimreview-collector

dataset processing part of https://github.com/MartinoMensio/MisinfoMe

Welcome to my GitHub profile!

I am Martino, PhD Student at the Open University. I graduated in Software Engineering and I am interested in Machine Learning and Explainability. In my PhD I am exploring how different news source are using propaganda to present related events.

I like to understand how things work and I think that Artificial Intelligence should be more than algorithms that learn to perform complex tasks; we have to focus about explainable methods that enable the crowds to understand why things happen, to be able to face and remove bias, correct models and enabling the humans to improve their learning capabilities.

My main contributions here are:

• models for SpaCy:
  • creator of spacy-sentence-bert, a SpaCy wrapper for Sentence Transformers
  • creator of spacy-universal-sentence-encoder, a SpaCy wrapper for Universal Sentence Encoder
  • creator of spacy-dbpedia-spotlight, a SpaCy wrapper for DBPedia Spotlight
  • mantainer of spaCy-entity-linker, a pipeline for spaCy that performs Linked Entity Extraction with Wikidata on a given Document
  • Italian GloVe, an Italian version of GloVe trained on Wikipedia
• MisinfoMe: an app to analyse twitter profiles. It is made of four backend services and one frontend:
  • credibility: aggregating information about credibility of URLs, websites, tweets
  • twitter-connector: managing interaction with Twitter API
  • claimreview-collector: retrieving ClaimReview from public websites, providing the manually annotated ground truth
  • backend: orchestrating the different services, providing data to the Frontend
  • frontend-v2: the user interface available at https://misinfo.me/
• My Master's thesis project: Joint contextual NLU and paper
• Supports for Master's courses at Politecnico di Torino: see the complete list here

To get in contact in the GitHub style, you can create an issue!

https://martinomensio.github.io/

claimreview-collector's Issues

teyit.org

I noticed a bug in teyit.org, I don't know the solution yet. When flatting it from claimreview, claim_url is null, even though the actual website has that information.

Bug in fix page

Found a bug in "re.sub('"claimReviewed": ""([^"]*)"', r'"claimReviewed": "\1"', page)". If claim review is empty, it removes one of double quote and it causes exception.

Page before executing following line:
page = re.sub('"claimReviewed": ""([^"]*)"', r'"claimReviewed": "\1"', page)
Result:

"claimReviewed": "",

After execution:
Result:

"claimReviewed": ",

Test url is: url

Archive.today not allowing scraping

With normal requests, you get HTTP 429.
With Flaresolverr, timeout error.

See https://github.com/wabarc/archive.is#archivetoday-is-unavailable as it is probably related.

Affects archive.vn, archive.is, ... (many domains)

FactCheck Explorer cookie

I'm trying to download all ClaimReviews from Google FactCheck Explorer using this tool, but need some more guidance on obtaining and setting the GOOGLE_FACTCHECK_EXPLORER_COOKIE.

Can you please clarify in simple steps how to use the web inspector to find this cookie in the .env file? And where exactly to set it to download the ~28000 items?

(I was already granted access to FactCheck Explorer.)