uponthesky/techlog-article

My blog for personal technical articles

techlog-article-service

Introduction

• This is the repository of the APIs that my TechLog blog will use in order to do the CRUD operations on the article data

History

• Initially this project was planned to be implemented as an Express.js backend service.
• However, having decided to be a Python/ML expert, I have decided to use FastAPI.
• Hence this project has been migrated from the previous TS codebase into the Python codebase using FastAPI, which you currently see as of now.

Tech Stack

• This project uses the following list of technologies

  • Backend Framework: FastAPI with Python version 3.11

  • Database: we use PostgreSQL with SQLAlchemy 2.0

    • for the driver adapter, I use asyncpg; this is because I wanted to implement the I/O operations as asynchrous ones as much as possible
    • for database migration, I use alembic 1.10.3
    • why choose alembic over other ORM-agonostic tools such as flyway or liquibase?: see this link on how we usually organize a microservice and its DB

  • Python related tools

    • package managing tool: poetry
    • code formatter: black
    • linter: ruff
    • unit testing: pytest
    • I wrote an article on these tools

  • CI/CD tools

    • Github Actions(cloud vendor specific deployment workflows are to be added in the production code)
    • Terraform is used for managing the overall infrastructure

• These are the tools that I will adopt in the near future

  • Cache layer: Redis
  • Monitoring tool: Sentry
  • test coverage tool

Project Design

Architecture

This project tries to adopt what is called Hexagonal pattern. This may be a little bit of overkill for this small project. However, this pattern allows us to follow the SOLID principle easier, and the code becomes more readable and testable.

Above of all, I am very happy with this pattern since it forces the developer to think the business logic(the "domain" layer) first, so that we can quickly adjust to any frequent changes in the codebase.

Serverless

This project is based on the serverless technologies, such as AWS Lambda and serverless database services.

techlog-article's Issues

use case: user

• user sign up

  • the user inputs form data: username, email, and password(POST)
  • data validation check(format) at the web adapter
  • the application checks whether there is a user with the same email or name(including deleted)
  • if there is, raise 400 error, otherwise, respond with 201 created
  • the server stores the values in the DB(password => hashed)

• user sign out

  • the user provides his/her token
  • if the token is valid, the server sets the value of user.deleted_at and auth.deleted_at as datetime.now(timezone.utc)

• user account update

  • email, username, or password only(PATCH)
  • data validation check at the web adapter
  • the application checks whether there is a user with the same email or name(including deleted)
  • if there is, raise 400 error, otherwise respond with 200
  • the server updates the values in the DB(password => hashed)

Test code for the DB

web adapter integration testing(easy)

Do the web controller testing with some mocks

• references

  • https://fastapi.tiangolo.com/tutorial/testing/
  • https://fastapi.tiangolo.com/advanced/async-tests/
  • https://fastapi.tiangolo.com/advanced/testing-dependencies/

• user

• auth : not necessary

• article

unit testing

• only domain logics are our targets of unit testing

• article

  • create: pass all the data to db
  • read single => db 404 response / list => list + count (check call)
  • update: pass only unset values to db
  • delete: call db

• user

  • sign up: hash password functions are invoked
  • update account: db 404 / hash password if password is reset / exclude unset
  • sign out: call db

• auth

  • token check: invalid, expired
  • login: 404 / password not valid / issue access token & update the auth
  • logout: call db

user security(jwt token)

We should fetch a given user's data from the database and authenticate a user.

We will follow the security guide provided by FastAPI docs.

• security - 1
• security - 2
• security - 3
• security - 4
• security - 5

use case: auth(finalize)

Related to #29

use case: auth

User Story

1. a user(already signed up) logs in by sending username and password to the server

• checks whether the given data is valid
• if valid, issues a jwt token to the user(just token, not expiry data)
• if not, raises an 400/404 error

2. a user logged in sends its token to the server

• checks whether the token is valid => dependency
• either invalid or expired: 401/400 error respectively
• if valid: allows the user to do what is allowed for her/him

3. a user tries to log out

• deactivate the token

• implement public interface of 1

• implement public interface of 2

• implement public interface of 3

• revisit: after DB adapter

• revisit: after common utils

• (optional) implement a cache server for the user tokens(Redis) => currently, we only focus on the overall implementation of the application.

backend - api/article

Implement the backend api for the individual article pages

Setup Dev Environment

On branch develop, finish the basic development environment setup.

In detail:

• initialize node.js project via yarn --init
• initialize TypeScript project via tsc --init
• configure the project in package.json and tsconfig.json
• install basic packages:
  • TS developement: typescript, ts-node-dev(instead of nodemon)
  • linter: eslint(with pre-commit hook)
  • testing: ts-jest(with pre-commit hook)
  • formatting: prettier(with pre-commit hook)

postgres DB: neon.tech

TODO: setup a cloud-vendor specific deploy pipeline

/api/logout: Add Redis cache for token blacklist

References:

• https://github.com/redis/node-redis

• install redis locally

• install redis client

project documentation

DB migrations

frontend: vercel

Originally intending to work with GCP. However due to the time limit, we'll deploy it to vercel

system testing

make some senarios and do e2e testings

Currently, we would like to do manually. But in the future we want to convert them into code.

• #58

• user signs up

  • unhappy case: already existing user / form validation error

• user logs in

  • unhappy: password, username wrong

• user reads an article

  • unhappy: article doesn't exist

• user reads a list of articles

• user creates an article

• user updates an article

  • unhappy: not an owner of this article

• user deletes an article

  • unhappy: not an owner of this article

• user logs out

• user signs out

backend - api/admin

Implement the backend api for the login page, solely dedicated to the owner user(the author)

database setting(async)

• Microservice & DB: https://www.prisma.io/dataguide/managing-databases/microservices-vs-monoliths#how-do-microservices-affect-database-architecture
• Alembic tutorial
• postgresql install
• DB auth:
  • https://www.prisma.io/dataguide/mysql/authentication-and-authorization
  • https://www.prisma.io/dataguide/mysql/short-guides/creating-deleting-users
• Transactional decorator
  • dead simple python -> class finish
  • implementation
• async logic implementation(sqlalchemy)

Python Dev environment setup

• poetry & pyenv
• install pytest, ruffpylint, blackyapf with precommit
• install fastapi

backend: Cloud Run

DB adapter integration testing(relatively hard)

Use testcontainers

• relying on this tutorial: https://mariogarcia.github.io/blog/2019/10/pytest_fixtures.html

• user

• auth

• article

backend - api/me

Implement the backend api for the self introduction(about me) page

This page consists mainly of static data, so not much works to be done: the root directory is public/me

• main picture: /profile
• a short introduction about myself: /shortIntro
• education: /education
• work experience: /workExperience
• comp.sci: /compSci
• hobbies: /hobbies
• (possibly in a near future) related pictures

Merge branches done so far into api/base

This is kind of intermediate check - so that we don't have to deal with more complex collisions in the future.

test wrapup & API metadata

• API metadata & middleware(db_session)
• applying @transactional & testing

Migrate into FastAPI & Having only CRUD APIs

Separate the logic, such that this service only handles CRUD APIs for the admin user only.

Furthermore, this service will be migrated into FastAPI.

articles - create

But first, test the docker image

separate the postgres module

Deploy

• #56

• choose one of the followings and deploy:

  • render: https://render.com/pricing
  • fly.io: https://fly.io/
  • railway: https://railway.app/pricing
  • supabase: https://aunicorndev.hashnode.dev/supa-fast-api-supabase-fastapi-building-the-api

• Questions: does this service...

  • support serverless functions in docker image format?
  • support Postgres?
  • support securities? - api gateway, secret manager
  • support loadbalancer?
  • run at reasonable cost?

Edit(May 18th, 2023)
We're going to choose the following path:

• #63

• #64

• #65

• #66

router

Roadmap: Read and implement the FastAPI docs basic tutorial part, before Error handling.

Article

• create
• read
• update
• delete

User

• create
• read
• update
• delete
• anything involving security issues related to the router part

add github action for cicd

• set a precommit hook to generate requirements.txt
• setup a basic cicd pipeline
  • checkout
  • basic setup: python setup => install dependency
  • test
• #61
  • db migration
  • backend deploy

backend - api/main

Implement the backend api for the main page

API implementation cycle

1. write tests, and check that the test fails
2. implement functionality designed in the test code
3. pass the tests

remark: do DB settings simultaneously

• install: Prisma MySQL tutorial
• .env setting: https://www.prisma.io/docs/guides/development-environment/environment-variables/using-multiple-env-files
• unit testing(for services): https://www.prisma.io/docs/guides/testing/unit-testing#singleton
• integrationo testing(for controllers): https://www.prisma.io/docs/guides/testing/integration-testing
• Database Design(MDN tutorial): https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/mongoose#designing_the_locallibrary_models
• Primsa Advanced Tutorial:
  • https://www.prisma.io/blog/backend-prisma-typescript-orm-with-postgresql-data-modeling-tsjs1ps7kip1
  • https://www.prisma.io/blog/backend-prisma-typescript-orm-with-postgresql-auth-mngp1ps7kip4

integration testing

This divides into two parts:

• #47

• #48

make a firestore wrapper

Rewrite the DB to Firestore

Replace the current postgres DB part with Firestore due to the high cost of either AWS RDS or GCP Cloud SQL.

• #76
• implementing the code
• make the transactional annotation abstract
• #77
• #79
• #80
• Setup the infra using terraform

redis: upstash

Recheck data dtos and change some(including tests)

Including:

• MeResponse
• Url type
• distinguishing between image and markdown data

articles - read

Jest making collision in the test DB

We have the following error when we run all the tests for confirmation:

  → 19 await prismaClient.article.createMany(
    Unique constraint failed on the constraint: `PRIMARY`

A solution might be this: https://stackoverflow.com/a/67935087

e2e Testing with FE

Testing with FE locally => testing on AWS

backend deploy: use AWS SAM

• make a separate private repo
• AWS SAM config file
• other secret info => AWS parameter store
• neon.tech

use case: article

Basic CRUD jobs

• create

  • input: title(str), content(Optional[str]), and user(author) id(dependency injected)
  • validate at the web adapter => service stage: title should be not null
  • store the data into the database(update both article and user using relationship)
  • return None(201 created)

• read

  • input: article id
  • db query: both article and the author
  • output: title, content, created_at, updated_at, user.name, user.email(200 ok)

• read all

  • input: query string -> offset & limit(pagination)
  • db query: both article and the author(join)
  • output: a list of articles with the same dto as "read"(200 ok)

• update

  • input: article_id, title(optional), content(optional)
  • update the article(update)
  • output: None(200 ok)

• delete

  • input: article_id
  • set deleted_at = now
  • output: None(204 no content)

refactor the structure before unit testing

Check:

• naming & responsibilities

• duplicates in code

• article

  • in port
  • out port
  • service
  • web adapter
  • persistence adapter

• user

  • in port
  • out port
  • service
  • web adapter
  • persistence adapter

• auth

  • in port
  • out port
  • service
  • web adapter
  • persistence adapter

read through the docs: https://fastapi.tiangolo.com/deployment/

• create a Dockerfile

Mid-term check

Mid-term check: read through the following tutorials and check whether we're missing anything

• fullstackopen

Addressing CORS issues

Need to spend some time to write about this issue in general

• Read through this article: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
• Could be useful to watch these Youtube videos:
  • https://www.youtube.com/watch?v=4KHiSt0oLJ0
  • https://www.youtube.com/watch?v=bW31xiNB8Nc

backend - deploy to GCP

Currently the backend service has very limited performance & devops features.

We will move to GCP for better features.

• frontend: Cloud Run
• #73
• postgres: Neon
• api gateway
• connect all the components together

TBA:

• cloud storage
• redis

Determine the file structure(overall)

Need to consult:

• fullstackopen
• MDN Express tutorial
• Express MVC example
• Nest.js's basic structure
• Urisign Assignment (in Korean)
• Prisma Example

service layer

• Finish implementing the service layer:
  • article service
  • user service
  • unit testings

Directly read static data in sync and return the data to the request

For providing static data such as .txt or .md, we need to read them and send them right away to the user of request.

Consult the following references:

• https://www.geeksforgeeks.org/javascript-program-to-read-text-file/
• https://www.sohamkamani.com/nodejs/file-system/