sjbitcode/daca-news

A news aggregation site specifically for DACA related articles, updated hourly. Built with Django ❤️

daca.news

A news aggregation site specifically for DACA related articles, updated hourly. Check it out here!

APIs used:

• NewsAPI
• Bing News Search

Built with:

• Django
• Huey (for periodic tasks)
• Javascript

Deployed:

• to DigitalOcean
• using Dokku

daca-news's Issues

Implement fuzzy matching for similar titled articles

Problem

As of 12/28/20, I came across three similarly titled articles, the difference is that some titles have the source included in it, ex:

• In the new year, take a new look at immigration – starting with DACA
• In the new year, take a new look at immigration – starting with DACA | Charlotte Observer
• In the new year, take a new look at immigration – starting with DACA | Raleigh News & Observer

In this function of the article pipeline, I implement a function to check article titles 15 days prior and ahead for the same title.

Some solutions:

• use Python's fuzzy wuzzy package
• Use a regex filter in Django ORM, possibly stripping between characters like | and -

Dockerize project

Summary

Sketch out project structure with Docker. The two main objectives for this project is to have the email service and a simple web interface; together these are two processes that will need to be managed with Docker.

Source articles from NewsAPI

Summary

The API we are using is called NewsAPI.
Good news, they have a Python client and a Top Headlines endpoint.

Task

• Test endpoint(s)
• Parse data
• Brainstorm search terms for endpoint(s)
• Use Cryptobrief as an example on how to use the NewsAPI

Discoveries

• curl https://newsapi.org/v2/everything\?apiKey\=b49ff08901ba439fbb219b3e8ad37a48\&q\=daca | jq

Periodic tasks

Setup periodic task to pull data from NewsAPI.

What task queue will we use?

Some options are:

• rq
• huey
• APScheduler

Use Bing for data backfill

Pull from Microsoft Azure Bing News Search to populate from articles, since NewsApi only allows a month prior.

Update search UI

Trigger a search from clicking source tags, in addition to search bar input.

Integrate Django

Convert this project to a Django project.

Setup database

Decide on a db for the project (sqlite or postgres)

Mock up table schemas that we might need to use.

Maybe something like:

Article  # content that is pulled from news sources
  - title (char)
  - url (char)
  - category (fk?)
  - created_at (datetime)

Digest  # email that is sent. Can reference many articles
  - sent_at (datetime)

Recipient  # a user that receives the email digest.
  - name (char)
  - email (char)