jazzify/newsify_back

Newsify Backend

Newsify Backend (Deprecated)

Newsify Backend is an API that scrapes some of the most important news platforms such as The Washington Post, El Pais and El Tiempo to get the most recent news.

Try the API it on Newsify Back

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

What things you need to install the software and how to install them

• Clone the project:

  git clone https://github.com/jazzify/newsify_back.git
  

• Install Python/Pip:

  You must have Python 3 and pip3 installed on your machine in order to run the project.

• Database:

  Newsify is using PostgreSQL on a Docker container with docker-compose

• Virtual environment (Optional):

  I like to use Conda as a VE manager but you can also use virtualenv or virtualenvwrapper

Installing

A step by step series of examples that tell you how to get a development env running.

Go to the project directory

cd path/to/newsify_back/

Install the dependencies:

pip install -r requirements.txt

Install the development dependencies (optional):

pip install -r requirements-dev.txt

If you already have docker and docker-compose in your machine, just run the following command to get your database up and running:

docker-compose up -d

Apply the basic migrations:

python manage.py makemigrations
python manage.py migrate

Check everything works by starting the server:

python manage.py runserver

Running the tests

TODO

Deployment

TODO

Built With

• Django - The web framework used
• ReDoc - API Docs
• Sphinx - Project Docs

Docs at:

https://newsify-back.readthedocs.io/en/latest/index.html

https://newsify-back.herokuapp.com/api-docs/

Authors

• Jazzify

License

This project is licensed under the BSD 3-Clause License - see the LICENSE.md file for details

newsify_back's Issues

Add support for The washington post

https://www.washingtonpost.com/world/

Order posts by creation date

Order posts by created_at and group them by post_type

Add/config django-debug-toolbar

We should add django-debug-toolbar to work in local and dev stages

Add tests

Add unit tests

Check:

• https://docs.djangoproject.com/en/dev/topics/testing/
• https://www.django-rest-framework.org/api-guide/testing/
• https://docs.pytest.org/en/latest/
• https://github.com/joke2k/faker
• https://factoryboy.readthedocs.io/en/latest/

Better error handling

It would be great to add a feature when the crawl fails, to notify any developer, this app is based on the HTML stability structure of the news platforms, if this change Newsify won't be able to retrieve any post, we need a way to identify this errors to make the respective hotfix.

Possible solutions when the crawler fails:

• Email the backend developers
• Implement a push notification service

Implement CI/CD

Add permissions and auth systems

Improve security 👍

Config and deploy to heroku

Change DB

We can check relational DBs:

• MySQL
• PostgreSQL

Maybe NoSQL:

• MongoDB

Plus check:

• GrahpQL

Enhance the readme and add changelog

Find a better way to handle errors

Create categories for each type

Add categories for every post_type in order to retrieve the last 10 posts for each news platform.

• GET: /api/twp/posts/ (last 10)
• GET: /api/etp/posts/ (last 10)
• GET: /api/eps/posts/ (last 10)
• etc...

Automatically scrape websites

Make a cronjob or something to automatically scrape the news platforms

Add api docs and read-the-docs

https://docs.readthedocs.io/

We can use one of the recommended options given by DRF for the api doc:
https://www.django-rest-framework.org/topics/documenting-your-api/

I would recommend use Swagger or Redoc

Enhance API docs

API docs are autogenerated by DRF we must generate it ourselves to keep the data structure mirroring the actual API response.