rog-golang-buddies/api-hub_storage-and-update-service

API hub service to provide CRUD operations with API definitions and keep data up to date.

Storage and update service

Description

This service represents API Specification document (ASD) data access interface (RabbitMQ to save/update, gRPC to get). It provides gRPC API to search and get ASD documents, queue event listener to save ASD documents. Also, this service is supposed to update ASD models. (request update from data scraping service)

Main functions (To Do)

Proof of Concept stage

1. Provide gRPC API with methods:
   • search(search string, page int) (Page[ApiSpecDocShort], error) to search by domain and description. Return a list of short descriptions.
   • get(id long) (ApiSpecDef, error) to get the full API Spec document by id.
2. Process events from Rabbit MQ with ASD model:
   • retrieve events from Rabbit MQ with ASD model
   • save/update data in DB (set created_at/updated_at)
   • notify API Gateway on update if required (event need to contain flag - is notification needed)
3. Check last updated dates and request update if necessary:
   • get expiration time from the environment variable
   • perform periodically task and get records that require update
   • send them to data scraper service with queue

Future plans

1. Add caching.

Data flows involving service

1. Usual UI request (search/get) API Gateway -gRPC-> storage and update service -> DB
2. Add new/update existing ASD data scraper service -RabbitMQ-> storage and update service (-RabbitMQ->, -RabbitMQ->) (API Gateway, DB)
3. Update stale data storage and update service -RabbitMQ-> data scraper service -RabbitMQ-> storage and update service -> DB

Additional resources

1. Database schema https://dbdiagram.io/d/630287b1f1a9b01b0fb26570 / https://imgur.com/a/evcpXry

api-hub_storage-and-update-service's Issues

Create ApiSpecDoc event listener

This service is supposed to retrieve ApiSpecDoc models from storage and update the service.
The message body will also contain a flag - is notification required.

Here you need to create a RabbitMQ event listener similar to task rog-golang-buddies/api-hub_data-scraping-service#11 .
EventListener has to retrieve a message from the queue and pass it to some update service method (currently you may just add a comment to the place where the method should be called, have to be created in the issue #8 ).

Create proto files and generate server code.

Create .proto files with API methods to search and retrieve ApiSpecDocs

1. Search
2. Get by Id

As base representation for the full view, you have to use copied model from the issue #6
(take a look readme for the details)

Test manually and add unit tests for service layer

Due to not enough time service is going to be merged without test.
Add tests and keep coverage about 80 %

Fix/Setup pipelines to docker hub.

Currently pipeline fails due to absence of variables for sonar auth.
Also need to add required data to provide building and pushing image to docker hub.

Copy ApiSpecDoc model from data scraping service.

We need to copy model to this service.

Add base infrastructure

You need to add base interfaces and project structure to provide the possibility to:

1. Update ApiSpecDoc - check if that model is really new, and if it is - calls update in DB.
2. Add middleware (in the future probably will contain cache) and services to search and get ApiSpecDoc.
3. Add entities and repositories draft.

(probably we'll need some discussion here, but we need some draft to start)

Choose database

We need to choose database for storage and update service.
I don't see some specific requirements. Db is supposed to provide quite simple actions.
I propose mongodb as more "cloud friendly" (cause of shared architecture).

Database setup

Here you need:

1. Add database configuration properties.
2. Add gorm db connection with the postgresql driver.
3. Add db connection at the application startup (internal/app.go)
4. Add Postgres db container to docker/docker-compose-dev file and check that connection establish successfully.
5. Setup migration mechanism using go-migrate.