godspeedsystems/gs-node-service

Godspeed microservice framework

gs-node-service's Issues

CRUD API generation from datasource

• Generate events and workflows from Prisma datastores. P0

• Use prisma annotations to decide what API to generate. P1

gs_service: Issue observed in executing sub workflow

gs_service: Issue observed in executing sub workflow

Check for wrong datasource name gracefully handle

Check for wrong datasource or event name gracefully handled

Auto generated test cases

Auto generated test cases

• The cli will be godspeed generate tests This will ensure a folder called tests exists. And create YAML based API tests written in our YAML testing compatible format.
• Test cases must have name from the name of each event file
• If a test already exists, don't touch it

Handle gs_service error/exception gracefully

• Loading exception. If there is any exception during loading, then process.exit
• JSONNET <%%> exceptions for args expansion, transform scripts: Catch the err.message and handle properly.

Jmeter profiling of hello world in gs_service

invoking sub workflow

invoking sub workflow

• Emit event
• Create new context with right inputs and passing it

Could be creating new ctx and set inputs and outputs and at the end of the flow, copy to the parent flow

Enhance Response structure

Enhance gs_service Pino logging

Logging needs to be improved to provide the complete flow/story of the request.

onError handling in tasks within Series, parallel, switch control flows

IN the yaml of a task, developer can specify:

onError:
  continue: boolean # Default value is set in the mappings/default. The project template's default is true.
  response: <%JSONNET SCRIPT%> #optional. Default is the GSStatus for error. which we are already doing.
  log: 'LOG_LEVEL_ENUM'  # Default value is set in the mappings/default. Values can be debug, error, severe. The project template's default for onError.log is 'Error'. If project default is null, and developer also does not specify. No logging should be done. Don't add to the GSLogEvents.

Handling for series, parallel, switch task

When a task fails in series or parallel, depending on its onError handling, it must stop the execution of the control task or continue with the execution.

Important!

Add errors events as GSLogEVent via ctx.addLog() (For telemetry/logs)

Thoughts

Something like this in GSSeriesFunction

 ctx.outputs[this.id] = ctx.outputs[childId] //where id is of the child failed
 if (!ctx.outputs[child.id]?.success && !child.onError.continue) {
    
    ctx.addLogEvent(new GSLogEVent()) // Telemetry itnegration will later log these tasks.
    return; //Exit the series or parallel right away
  }

Mock server with mock data generated using faker

Mock server with mock data generated using faker

Jmeter profiling of hello world in gs-service

gs_service: proper error handling required, in some cases requests just hangs

gs_service: proper error handling required, in some cases requests just hangs

Test || Add Test case: utils functions

Test Coverage

• getAtPath()
• setAtPath()
• checkDatasource()
• checkFunctionExists()
• prepareScript()
• compileScript()
• removeNulls()

Prisma Integration

gs_service: Throw error on load time if any function, specified in events, doesn't exist

gs_service: Throw error on load time if any function, specified in events, doesn't exist

Training module

The following parts/steps make the module

• Should be a PPT
• Framework explanation with documentation
  • Abstractions: event, workflow (YAML, JSONNET, handlers: native/YAML), datasources (Datastores, API)
  • Overview: folder structure and walkthrough
• Dev setup
  • CLI
    • Explain the CLI commands
    • Follow CLI driven process #57
  • Explain folder structure and walkthrough
  • Testing
    • Show sample integration testing YAML
    • Tell about pre-commit test hook
    • Tell about how to auto-generate test cases #15
  • Documentation
    • godspeed generate docs will create/override folder with OpenAPI 3 spec compliant JSON file
    • Show them the url on which they can see the api specifications /api-spec
  • Lint setup with Visual studio
    • Auto linting on file save in VS Code
    • Test Lint ensure on commit
  • Run the project using dev container
  • Explain and see logs
  • Explain datastore setup with schema file
    • godspeed datastore pull {schema_name}
    • or create a new schema from scratch
  • Explain watching. Just for their information.
  • Postman collection
• Assignments
• Projects

Get apiversion and pino log level variables from config

Environment variables to get from config:

• GSResponse: apiVersion
• Pino Logging: level

Test: app functions

Test Coverage

• JsonnetSnippet()
• loadEvents()
• httpListener()
• main()

Auto generated api documentation

• godspeed generate docs will
  • Create the documentation files
  • make it automatically live on {microservice_url}/api-specs

Profiling the gs-project-template of prisma

• Setup a Nodejs/Express project with same prisma schema as in Godspeed project
• Nodejs project should have validation of incoming API request (use same schema as gsproject_template) (use Express Schema Validator)
• Profile both services with a simple get /user call

Test: yamlLoader: function loadYaml()

Test Coverage function loadYaml()

Create Dockefile of gs_service

Unit test case coverage for gs_service

Test case scenarios for GS-Service:

• 1. Invoking an event should execute the workflow
  • 1 Request/ Response schema validation for events
• 2. Failure (if below does not exist)
  • 1. Function(within a function, within an event)
  • 2. Event
  • 3. DataSource
• 3. DataSource Schema Function Schema, Event schema validation (Json Schema for all YAML's using orderly)
• 4. Framework functions
  • 1. Parallel
  • 2. Series
  • 3. Combination
  • 4. HTTP
  • 5. Transform
  • 6. Return
  • 7. Switch
  • 8. OnError
  • 9. Calling one YAML fn from another YAML fn
• 5. GS-service src test cases.
• Loading, functionality, fixtures

Note: Faker-Mockdata Generator >> Microservice testing

Datasource,event, workflow schema based validations

• First load datasource with right structure (schema validation)
• Then load functions with right structure (schema validation)
• Then load events with right structure (schema validation)

Note

• Every step depends on previous to work. Process.exit(1) if any step fails.
• Use Orderly for doing this.

Test: functionLoader: function loadFunctions() and createGSFunction()

Test Coverage

• loadFunctions()
• createGSFunction()

Environment based configuration

Environment based configuration, mapping of environmental variables in framework.

Test Coverage of GS-service

gs_service: Error handling in jsonnet errors

gs_service: Error handling in jsonnet errors

Test: configLoader: function iterate_yaml_directories()

Test Coverage configLoader: function iterate_yaml_directories()

Test: com.gs.return and com.gs.transform

Test Coverage

• com.gs.return
• com.gs.transform

Test: src/plugins add test case

Test: src/plugins add test case

• randomString()
• randomInt()

Developer setup process

Handle OnError at runtime at workflow level and task level

.Dev-container: setup mongodb single node as replicaset

It seems there is a keyFile setting needed. Following docker-compose is not enough.
For refererence

version: "3.7"
services:
  mongodb:
    image: mongo:5.0
    ports:
      - 27017:27017
    volumes:
      - mongodb-data:/data/db
    command: [--auth, --replSet, my-replica-set]
    environment:
      - MONGO_INITDB_ROOT_USERNAME=root
      - MONGO_INITDB_ROOT_PASSWORD=root
volumes:
  mongodb-data:

gs_service: configLoader is not giving proper error in case file doesn't have .yaml extension

gs_service: configLoader is not giving proper error in case file doesn't have .yaml extension
Error thrown when a simple file without .yaml extension was present in functions directory
`node:internal/fs/utils:344
throw err;
^

Error: ENOTDIR: not a directory, scandir '/home/gurjot/gs_project_template/gs_service/dist/core/../functions/com/pinelabs/get_loan_application_status'
at Object.readdirSync (node:fs:1392:3)
at iterate_yaml_directories (/home/gurjot/gs_project_template/gs_service/dist/core/configLoader.js:33:20)
at iterate_yaml_directories (/home/gurjot/gs_project_template/gs_service/dist/core/configLoader.js:67:47)
at iterate_yaml_directories (/home/gurjot/gs_project_template/gs_service/dist/core/configLoader.js:67:47)
at iterate_yaml_directories (/home/gurjot/gs_project_template/gs_service/dist/core/configLoader.js:67:47)
at iterate_yaml_directories (/home/gurjot/gs_project_template/gs_service/dist/core/configLoader.js:67:47)
at loadSources (/home/gurjot/gs_project_template/gs_service/dist/core/loader.js:12:45)
at Object. (/home/gurjot/gs_project_template/gs_service/dist/core/loader.js:15:3)
at Object. (/home/gurjot/gs_project_template/gs_service/dist/core/loader.js:17:4)
at Module._compile (node:internal/modules/cjs/loader:1103:14) {
errno: -20,
syscall: 'scandir',
code: 'ENOTDIR',
path: '/home/gurjot/gs_project_template/gs_service/dist/core/../functions/com/pinelabs/get_loan_application_status'
}
`

Test: com.gs.http function Retry/OnError

• Retry

Add Redis plugin in gs_service

Add Redis plugin in gs_service

Logging with Pino

Add Pino Logger and replace all the console statements with logger (info, warn, debug, error)

Logging with Pino

Replace the console log with pino logger in gs_service

Test: http_listener: app object

Test Coverage http_listener: app object

Test: Telemetry test Kafka for context propogation

Make sure incoming and outgoing Kafka messages and HTTP calls are able to go through proper context propogation, via the headers

Test: codeLoader: function loadModules()

Test Coverage codeLoader: function loadModules()

setup gs_project_template fixing

Test: datasourceLoader: function loadDatasources()

Test Coverage datasourceLoader: function loadDatasources()

Test: loader: config object

Test Coverage loader: config object

Test: jsonSchemaValidation functions

Test Coverage

• loadJsonSchemaForEvents()
• validateRequestSchema()
• validateResponseSchema()

Finding new hacker

Test: com.gs.http function with/without datasource schema

Test Coverage com.gs.http function

• http request with datasource schema=true
• http request with datasource schema=false

GS_PROJECT_TEMPLATE: run integation tests in YAML framework on commit