hwsc-org/hwsc-app-gateway-svc

Web application gateway management service

hwsc-app-gateway-svc's Issues

Epic/VerifyEmailToken

VerifyEmailToken service is used to verify that the user has entered a valid email address belonging to the user. It is used to validate users creating account for first time and existing users wanting to update their email.

High level process

• User clicks on verification link from their email
• Chrome opens this link
• Chrome will validate the url to ensure proper format to prevent abuse
  • i.e: does it contain the proper stub token=someToken
• Chrome will extract the token string from the url
• Chrome will send this extracted token to gateway using token authentication dial in the format of "authorization": "Email Token " + token as metadata in the context
• gateway has to determine the token type (this is to differentiate between VerifyEmailToken and VerifyAuthToken
• gateway will send this token to user-svc VerifyEmailToken
• VerifyEmailToken will look up token in user_svc.email_tokens table

If token does not exist, user-svc will:

• send back error with token does not exist
• Chrome will tell user that this link is no longer valid and user must re-sign up or login

If token exists and is NOT expired, user-svc will:

• delete this token from user_svc.email_tokens table
• grab user row from user_svc.accounts table by looking up UUID grabbed from user_svc.email_tokens
• modify user column's is_verified to TRUE
• send this user row back with OK message

If token exists but is expired, user-svc will:

• delete this token from user_svc.email_tokens table
• grab user row from user_svc.accounts table by looking up UUID grabbed from user_svc.email_tokens table
• generate a new email token and insert into user_svc.email_tokens
• resend a verification email to the email based on two criteria:
  1. resend email under prospective_email column if its column is not null/empty
  2. resend email under email column if prospective_email column is null/empty
• send back a message saying token has expired, and we have sent a new verification email
• is the above considered an error? or an OK with the above message?

Note: For new users, if the token existed in the table but was expired, this means that the maintenance-svc hasn't gotten around deleting the token and user from the database. We might as well re-use this information and allow new user to just click on a resent verification link??

Feature/ Handle refreshing in case auth secret is missing

Description

As a developer, I want app-gateway-svc to handle refreshing invalid or missing auth secret

func init() {
	userSvc = &userService{}
	if err := refreshConnection(userSvc, consts.UserClientTag); err != nil {
		// TODO once docker container is runnable
		log.Error(consts.UserClientTag, err.Error())
		//log.Fatal(consts.UserClientTag, err.Error())
	}
	// NOTE:
	// app-gateway-svc does not start if all the services are not ready
	// this is ONLY on app-gateway-svc startup
	resp, err := userSvc.getStatus()
	if err != nil {
		// TODO once docker container is runnable
		log.Error(consts.UserClientTag, err.Error())
		//log.Fatal(consts.UserClientTag, err.Error())
	} else {
		log.Info(consts.UserClientTag, resp.String())
	}

	// TODO handle refreshing in case auth secret is missing
	if err := userSvc.refreshCurrAuthSecret(); err != nil {
		// NOTE: Check PSQL migration or if there is an active secret
		log.Error(consts.UserClientTag, err.Error())
	} else {
		log.Info(consts.UserClientTag, "AuthSecret obtained")
	}

	// Handle Terminate Signal(Ctrl + C)
	c := make(chan os.Signal)
	signal.Notify(c, os.Interrupt, syscall.SIGTERM)
	go func() {
		<-c
		if err := disconnect(userSvc.userSvcConn, consts.UserClientTag); err != nil {
			log.Error(consts.UserClientTag, err.Error())
		}
		log.Info(consts.UserClientTag, "hwsc-app-gateway-svc terminated")
		serviceWg.Done()
	}()

}

Story Points

5

Definitions of Done

• implementation
• unit test

Feature/Incorporate E-mail Service

Description

As a user, I want to be notified with the following:

• Succesful registration
• Password changes
• E-mail changes
• Forget Password
• more later...

Story Points

8

Definitions of Done

• Research available e-mail service
• Implementations of the following:
  • Succesful registration
  • Password changes
  • E-mail changes
  • Forget Password
• Unit Test
• Integration Test

Epic/hwsc-app-gateway-svc AddDocument

Let us say Chrome wants to add a new document. At this point Chrome has the current User object, and it knows the user's uuid already.

1. The user fills up the form in the website.
2. Grabs the fields in the form and make a Document object. image_urls_map, audio_urls_map, video_urls_map, & file_urls_map are not filled up because there is no means of generating duid and fuids in the browser, therefore we have to send lists of urls.
3. Chrome calls app-gateway-svc's AddDocument() with the following

message AppGatewayRequest{
    hwsc.User user = N; //this is not a complete User object, it may only have uuid
    hwsc.Document doc = N; //this is not a complete Document object, it may only have duid
    repeated string image_urls = N;
    repeated string audio_urls = N;
    repeated string video_urls = N;
    repeated string file_urls = N;
}

4. app-gateway-svc calls document-svc's CreateDocument() which returns the newly Created Document with a now generated duid and fuids.
5. app-gateway-svc calls user-svc's GetUser, which returns a full User object A with password as empty string.
6. app-gateway-svc modifies the User object A, and adds the DUID in the:

map<string, UserDocumentMetadata > user_documents = 7;

7. app-gateway-svc calls user-svc's UpdateUser, which returns a full User object with password as empty string.
8. 1. OPTION 1 - app-gateway-svc returns 2 objects (updated User and updated Document) to Chrome
   2. OPTION 2 - app-gateway-svc calls document-svc ListUserDocumentCollection, returns a list of user documents.
9. Chrome renders page.

Feature/Email-Password Authentication

Description

As a user, I want to login using email and password.

Epic: #23

Story Points

5

Definitions of Done

• app-gateway-svc accepts dialing using email and password
• app-gateway-svc authenticates with user-svc
• unit test
• integration test

Feature/Connect Services

Description

As a DevOps, I want to deploy and to connect services using Docker or Kubernetes

Story Points

13

Definitions of Done

• Deployed hwsc-document-svc
• Deployed hwsc-user-svc
• Deployed hwsc-file-transaction-svc
• Deployed hwsc-app-gateway-svc
• hwsc-gateway-svc & hwsc-user-svc, hwsc-document-svc, hwsc-file-transaction-svc are communicating through hwsc-api-blocks

Feature/CreateUser app-gateway-svc

Description

As a user, I want to be able register using my browser such as Chrome

Story Points

2

Definitions of Done

• implementation
• unit test
• integration test hwsc-org/hwsc-api-blocks#105

Epic/CreateUser

Chrome wants to create a user.

1. The user fills up the registration form in the website
2. Chrome will check for user input validity
3. Chrome gathers this info into a userObject

- firstName: string
- lastName: string
- email: string
- password: hashed string
- organization: string

4. Chrome calls app-gateway-svc CreateUser(userObject) CreateUser(UserResponse)
5. app-gateway-svc will call user-svc GetStatus()
6. Two results from user-svc:

- Service is unavailable: return error with this message
- Service is available: proceed with the following

7. app-gateway-svc will call user-svc CreateUser(userObject)
8. user-svc will take this userObject and check if email already exists in mongoDB (unique email)
9. Two results:

- Email is taken: return error with this message
- Email not taken: proceed with the following

10. user-svc will create a unique user ID using ulid
11. user-svc will hash the hashed password using bcrypt
12. user-svc will create and insert new document with userObject information in user collection
13. user-svc sends back OK
14. app-gateway-svc receives any of the following message from user-svc and sends it back to Chrome:

1: Service is down
2: Email is taken
3: OK

15. Chrome can take any of the following action:

1. If service is down: Chrome displays error message to user that service is unavailable atm and try again later
2. If email is taken: Chrome displays error message to user to use a different email
3. If OK: Chrome will redirect user to logged in search page

Test/make a copy of MongoDB before testing with it

Description

As a tester, when testing with MongoDB's, I want to be able to revert back to MongoDB's original state.

Story Points

1

Definitions of Done

Copies of MongoDB's

Epic/Frontend-Gateway Contract

Description

As a developer using hwsc-app-gateway-svc, I need to know the services provided by hwsc-app-gateway-svc

Story Points

3

Definitions of Done

• Define the services needed by the frontend

Feature/Setup Azure Devops Pipeline

Description

As a developer, I want to integrate unit testing in the pipeline.

Story Points

1

Definitions of Done

• Perform unit test
• Passes Build
• Generate artifact
• Code coverage report

Feature/Enable client_hwsc_test.go for other services

Description

Enable client_hwsc_test.go to test document-svc and file-transaction-svc
docker compose for document-svc and file-transaction-svc are currently disabled.

Story Points

1

Definitions of Done

• uncomment cases in client_hwsc_test.go
• test must pass

Feature/return specific rpc code errors

Description

As a developer, I want to return rpc code specific to the error to provide consistency in error decoding in front end and back end.

Requires refactoring everything that returns errors including unit tests and integration tests.

Story Points

5

Definitions of Done

• rpc codes are specific to error
• unit tested to ensure test passes
• integration tested to ensure test passes

Feature/Configuration

Description

As a user, I want to configuration and environment variables to safely and securely run hwsc-app-gateway-svc

Story Points

2

Definitions of Done

• implementation
  - [ ] unit test
• save config in team channel

Feature/Determine specific error codes

Description

As a developer, I want to be able to send Chrome specific error messages to make it easier for it to determine the type of error to display to users.

Story Points

TBD

Definitions of Done

TBD

Epic/Authentication & Authorization

Objective

The Chrome user has to be authenticated within hwsc cluster.

Purpose

For every single request within a cluster a token is passed for authorization.

Prerequisites

• gRPC with Basic Auth/Token guideline
• Further research is needed to perform the required Token tasks. See TODO at the bottom
• User password are encrypted twice, browser to app-gateway-svc and user-svc to DB

• States

  • AUTHENTICATED - Chrome is connected to app-gateway-svc, but not authorized
  • DISCONNECTED - Chrome is disconnected to app-gateway-svc
  • AUTHORIZED - Chrome is connected and authorized to app-gateway-svc

Procedure

1. Chrome goes to login page - DISCONNECTED
2. Chrome logins with email and password using basic auth GRPC dial - DISCONNECTED
3. app-gateway-svc parses the header - DISCONNECTED
4. app-gateway-svc calls AuthenticateUser from the user-svc - DISCONNECTED
5. If the email and password are:
   • valid - then return the User without password + code.Ok - DISCONNECTED
   • invalid - then return code.Unauthenticated - DISCONNECTED
6. If AuthenticaUser returns code.Unauthenticated to app-gateway-svc:
   a. app-gateway-svc returns code.Unauthenticated to Chrome - DISCONNECTED
   b. Chrome renders failed login - DISCONNECTED
7. If AuthenticateUser returns code.Ok + Identification to app-gateway-svc, then app-gateway-svc return code.Ok + token_string as metadata in the context to Chrome - AUTHENTICATED
8. In order to be AUTHORIZED, Chrome has to call GetAuthToken from app-gateway-svc or parse the token_string from the context's metadata.
   • token string is defined below this document
   • User that is not AUTHORIZED cannot perform RPC calls, therefore this has to be called immediately after GRPC dial
9. If Chrome calls GetAuthToken from app-gateway-svc with email and password -AUTHENTICATED
10. app-gateway-svc calls GetAuthToken from user-svc - AUTHENTICATED
11. user-svc validates from the DB. - AUTHENTICATED
12. If the email and password are invalid:
    a. user-svc returns code.Unauthenticated to app-gateway-svc - AUTHENTICATED
    b. app-gateway-svc returns code.Unauthenticated to Chrome - AUTHENTICATED
    c. Chrome renders failed login - AUTHENTICATED
13. If the email and password are valid, then user-svc generates and returns Identification, to `app-gateway-svc - AUTHORIZED
    • Identification contains a Secret and token string
    • user-svc encodes the token string using hwsc-lib NewToken
    • token string expires in 2 hours
    • user-svc has to record everything in a DB
14. app-gateway-svc returns the token string to Chrome - AUTHORIZED
15. Chrome and services exchange this token_string to authorized the actor or user - AUTHORIZED
16. Services would use hwsc-lib to decode and validate the token string, and if it is invalid would return code.Unauthenticated (remember that the secret key also expires) - AUTHORIZED
17. Services checks if the token string is not expired using hwsc-lib- AUTHORIZED
18. Chrome checks if the token string is about to expire within 15 minutes, and calls GetToken from app-gateway-svc to get a new token. AUTHORIZED
19. If the token string has expired, Chrome redirects to login page. DISCONNECTED
20. if the token string is valid, services provide the appropriate RPC- AUTHORIZED
21. In the event that Chrome loses connection, Chrome should redial using token auth GRPC dial to app-gateway-svc with token_string from the Token - DISCONNECTED
22. app-gateway-svc calls VerifyToken from user-svc - DISCONNECTED
23. user-svc verifies the token string from DB - DISCONNECTED
24. If the token_string is:
    • invalid then:
      • a. user-svc returns code.Unauthenticated to app-gateway-svc` - DISCONNECTED
      • b . app-gateway-svc returns code.Unauthenticated to Chrome - DISCONNECTED
      • c. Chrome redirects to login page - DISCONNECTED
    • valid, user-svc returns Identification + code.Ok to app-gateway-svc AUTHORIZED
25. If the Secret has expired, a new Token is required (THIS IS DIFFICULT)

token string, Identification

• Not using third party library for generating Token
• Feel free to use this repo as reference on how to implement our codes
• Building your own https://www.codementor.io/murphyisiwele/build-a-jwt-library-in-golang-ev6qmbbyh
• token string is exchanged by Chrome and app-gateway-svc
• a struct Secret contains string key and created_timestamp
• a struct Identification contains struct Secret and token string
• Identification is utilized in the requests for user-svc, document-svc, and file-transaction-svc
• Identification is also returned in the response of user-svc to app-gateway-svc
• token string expires in 2 hours
• token string has 3 levels permission (feel free to add more)
  • `NO_PERM
  • USER - can only perform CRUD on his/her own page
  • ADMIN - can perform CRUD with on any user's page
• algorithm used to hash the token_string are the following:
  • HS256
  • HS512
• encoding supported at the moment is JWT
  An example AUTHORIZED token will look like the following:
  Changed from:

"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJ1dWlkIjoiMTIzNDU2Nzg5MCIsInBlcm1pc3Npb24iOiJUb2tlbi5BRE1JTiIsImV4cGlyYXRpb25fdGltZSI6MTU0OTA5MzkxMH0.OZFQ_zU1F2BJm6kyYzsBns5qmOxbVbUnQV2SU1B_kyPfXPOmUd0fddRvF0I3IqaDz-55H7Q80w8zQyldMQ7AAg"

• A struct called Secret that contains the secret key as a string, and created date. and expiration date timestamp is required to generate a new secret key every week on Monday 3am EST UTC.
• The Identification will be defined in the api-blocks as a message
• user-svc has to provide GetAuthToken, VerifyAuthToken, GetSecret, MakeNewSecret
• app-gateway-svc has to implement BasicAuth and TokenAuth, and to provide GetAuthToken

Epic/GetStatus

Objective

Chrome wants to get the status of the app-gateway-svc

Purpose

To know if Chrome can receive services from `app-gateway-svc

The current app-gateway-svc states are the following:

• available
• unavailable
  Additional states will be added in the future.

Procedure

1. Chrome call app-gateway-svc GetStatus to confirm if service is reachable.
2. app-gateway-svc returns OK if app-gateway-svc is available.
3. app-gateway-svc returns UNAVAILABLE if app-gateway-svc is unavailable
4. If app-gateway-svc returns OK, Chrome will invoke services from app-gateway-svc`.
5. If `app-gateway-svc returns UNAVAILABLE, Chrome will retry for 5 times until Chrome renders service unavailable to user.

Feature/Dependency hwsc-app-gateway-svc & hwsc-frontend

Description

• As a frontend dev, I want to use hwsc-app-gateway-svc as a npm dependency.
• As a DevOp, I want to use hwsc-frontend as a GoLang dependency

Story Points

5

Definitions of Done

• successful npm install of hwsc-app-gateway-svc for hwsc-frontend
• successful dep ensure -v or dep ensure -update of hwsc-frontend "dist" for hwsc-app-gateway-svc

Feature/Email Verification

Description

As a user, I want to be able to verify my email

Story Points

3

Definitions of Done

• implementation
• unit test
• integration test

Feature/Token Authentication

Description

As a user, I want to login using token.

Epic: #23

Story Points

5

Definitions of Done

• app-gateway-svc accepts dialing using an existing token
• app-gateway-svc authenticates with user-svc
• unit test
• integration test

Epic/ListDistinctFieldValues

Objective

Chrome wants to get the following unique values for the following field names:

• Publisher (lastname & firstname)
• Study Site (city, state, province, & country)
• Call Type Name
• Ground Type
• Sensor Type
• Sensor Name
• Record Timestamp (Between min and max bounds)

Purpose

To generate available values to search.

Procedure

1. Chrome goes to Search Filter Page
2. hwsc-frontend calls hwsc-app-gateway-svc's rpc ListDistinctFieldValues (AppGatewayServiceRequest) returns (AppGatewayServiceResponse) {}
   • Required args for AppGatewayServiceRequest
     • NONE
3. hwsc-app-gateway-svc calls hwsc-document-svc's rpc ListDistinctFieldValues (DocumentRequest) returns (DocumentResponse) {}
   • Required args for DocumentRequest
     • NONE
4. hwsc-document-svc queries MongoDB
5. hwsc-document-svc's returns to hwsc-app-gateway-svc from rpc ListDistinctFieldValues (DocumentRequest) returns (DocumentResponse) {}
   - Status: &pb.DocumentResponse_Code{Code: uint32(codes.OK)}
   • Message: codes.OK.String()
   • QueryResults: queryResult
6. hwsc-app-gateway-svc returns to hwsc-frontend from rpc ListDistinctFieldValues (AppGatewayServiceRequest) returns (AppGatewayServiceResponse) {}
   - Status: &pb.AppGatewayServiceResponse_Code{Code: uint32(codes.OK)}
   • Message: codes.OK.String()
   • QueryResults: queryResult
7. hwsc-frontend checks Status or Message
8. If OK hwsc-frontend renders Search Filter Page, else a default error page with error 500.

Feature/Localize Unit Test app-gateway-svc

Description

As a dev, I want to localize all unit tests using containers

Story Points

8

Definitions of Done

Containers for the following:

• mongodb
• psql
• user-svc
• file-transaction-svc
• document-svc

Epic/hwsc-app-gateway-svc ShareDocument

Let us say Chrome(User) wants to share a Document. At this point Chrome has the User object already.

1. Chrome calls app-gateway-svc's ListUsers.
2. app-gateway-svc calls user-svc's ListUsers.
3. user-svc returns a list of users to app-gateway-svc
4. app-gateway-svc returns a list of users
5. Chrome picks the users he/she wants to share a document(document name goes with the callTypeName)
6. Chrome calls app-gateway-svc's ShareDocument with the following:
   message AppGatewayRequest{
   hwsc.User user = N; //this is not a complete User object, it may only have uuid
   hwsc.Document doc = N; //this is not a complete Document object, it may only have duid
   repeated string uuids_to_share_doc
   }
7. app-gateway-svc calls user-svc's ShareDocument with the following:

message UserRequest {
    User user = 1;
    string duid = 2;
    repeated string uuids_to_share_duid = 3;
}

8. user-svc's ShareDocument has to do the following:
   1. GetUser
   2. Modify themap<string, UserDocumentMetadata > user_documents = 7
   3. UpdateUser
   4. Traverse and GetUser each friend fromuuids_to_share_doc in UserRequest.
   5. Modify each friend's map<string, UserFriendMetadata> shared_to_me = 8
   6. UpdateUser for each friend

message User {
    string uuid = 1;
    string first_name = 2;
    string last_name = 3;
    string email = 4;
    string password = 5;
    string organization = 6;
    // Key = duid in string
    map<string, UserDocumentMetadata > user_documents = 7;
    // Key = friend's uuid
    map<string, UserFriendMetadata> shared_to_me = 8;

9. user-svc returns the updated User who shared the document to app-gateway-svc.
10. app-gateway-svc returns the User to Chrome.

Feature/AuthSecret

Description

As a app-gateway-svc, I want to get/update the current auth Secret

Story Points

5

Definitions of Done

• implement getting auth secret
• implement updating auth secret
• test by forcing to make new auth secret
• rename user-svc GetSecret to GetAuthSecret
• rename user-svc MakeNewSecret to MakeNewAuthSecret