buildlyio/buildly-core

Buildy-Core is a component based gateway for cloud native architectures.

buildly-core's Issues

Update Django version to 2.2.6

Proposal

A new bugfix release was issued by the Django project and the fix included in it JSONField that we use in two different models. So, we need to upgrade the installed Django version.
https://docs.djangoproject.com/en/2.2/releases/2.2.6/

Create sub-repository and separate Django APP for Workflows (Templates)

Feature Request

Each component should be added as an optional at build or post build component that can be toggled on or off.

Proposal

For Workflow we will need to move it out of the primary buildly-core repository into it's own repo that can be added as a sub-repository when enabled in the admin UI or CLI.

Bug report

N/A

Current behavior

Workflow is included with each clone of the repository as part of the main repo not a sub repository and can not be toggled off. All dependencies are required across models and views.

Expected behavior

User should be able to remove or comment out the inclusion of the app from the settings file and the app should still function but without workflow functionality.

Enviroment

Buildly version : x.y.z
Logs: ``

Create CRUD functionality for the Gateway

Remove Celery and RabbitMQ integration

Proposal

The celery and RabbitMQ integration isn't required for Buildly core to operate properly. These are extra dependencies don't bring any benefit to Buildly so they can be removed permanently.

Update Buildly Core to get API specifications from the database

Feature Request

Update Buildly Core routing process to get API specifications from the database

Proposal

In order to reduce the number of requests sent to a service and redirect faster the requests that come to Buildly Core, we need to update the gateway routing to use API specifications from the database. A field called api_specification of each logic module holds the service API specification and should be used during the routing process.

Manage all generated refresh tokens

Proposal

We should be able to list and/or remove all generated refresh tokens. We should also be able to filter the refresh tokens by the user's username.
The resources are available via the path /oauth/refreshtoken/.
The following fields should be shown:

• user (First & Last name, username)
• token (refresh toke value)
• access_token (access token value the refresh token is bounded to)
• revoked (timestamp of when this token was revoked)

Remove old implementation of Buildly Core gateway

Feature Request

We have three different implementations of the gateway and for obvious and security reasons, we should remove the old one and maintain only two of them.

Proposal

We have to remove the URL of the old gateway, rename the beta views to be the new one. We will need to update and remove some tests.
We also need to remove dependencies that aren't needed anymore.

Integrate Prometheus and Graphana to Monitor the services

Reduce Docker image size

Feature Request

It's better to reduce the size of Buildly Core docker image, so the build process will be faster. The image can also be downloaded and updated faster.

Proposal

When building a docker image of the application we should add only source code, static files and important scripts inside of the docker image

Allow users to be assigned to more than one organization

Proposal

Currently, there is a one-to-one relationship between users and organizations but we should support multi Tennant architectures. Therefore, we need to change this relation to a one-to-many relationship (i.e., one user can be assigned to two or more organizations).

OAuth User endpoint returns forbidden

Bug report

I cannot retrieve OAuth user data from the /oauthuser/ endpoint.

Current behavior

When a GET request is sent to the endpoint /oauthuser/ with a valid JWT, the server responds with 403 Forbidden.

Expected behavior

When a GET request is sent to the endpoint /oauthuser/ with a valid JWT, the server will return the user data of the user related to JWT sent in the request.

Environment

Buildly version : 0.3.0
Logs: Request

curl -X GET \
  http://localhost:8080/oauthuser/ \
  -H 'Authorization: JWT ...' \
  -H 'Host: localhost:8080'

SPIKE: Test Nomad with Buildly Core

Feature Request

Based on our discussion we should be able Test Nomad

Proposal

Will write together with the team

Buildly Core

Replace old data mesh with new

Proposal

Modify current data mesh and remove the old one.

Allow users to specify the swagger.json endpoint for a LogicModule

Feature Request

Allow users to specify the endpoint where their LogicModule's 'swagger.json' is located.

Many auto-generating Swagger services for APIs do not give a lot of flexibility with naming the endpoint where 'swagger.json' is located, or a user may have already implemented their Swagger documentation at a different endpoint from /docs/swagger.json. Giving them the option to specify where this is located reduces the amount of time needed for them to connect their service to Buildly.

Proposal

Add a 'Swagger Docs Endpoint' field to the LogicModule model and make it optional. If it is blank, Buildly Core will still search for the Swagger docs at <endpoint_name>/docs/swagger.json. Otherwise, it will use what the user has specified.

Create API endpoints to manage Authentication process

Proposal

We need to provide API endpoints, so admin users (only them) can create applications, they represent a client on the authorization server, and manage them. They should be able to set the following fields:

• redirect_uris: The list of allowed redirect URI. The string consists of valid URLs separated by space.
• client_type: security level at which some communications
  • confidential
  • public
• authorization_grant_type: Authorization flows available to the Application. Options:
  • authorization code
  • implicit
  • password
  • client credentials
• name: Friendly name for the Application

The following fields will be read-only and autogenerated by the Backend:

• client_id: The client identifier issued to the client during the registration process
• client_secret: Confidential secret issued to the client during the registration process

Configuration of the service discovery

Feature Request

Whenever a user wants to discover a service then the specific configuration is required in order to connect with the Buildly Core

Proposal

Bug report

Describe the current behavior and what is the expected behavior. Include logs and screenshots if possible.

Current behavior

Expected behavior

Enviroment

Buildly version : x.y.z
Logs: ``

Service Discovery

Feature Request

All stories and task related to the service discovery should be part of this epic.

Proposal

Bug report

Describe the current behavior and what is the expected behavior. Include logs and screenshots if possible.

Current behavior

Expected behavior

Enviroment

Buildly version : x.y.z
Logs: ``

Create an endpoint to refresh API specifications

Feature Request

Users have to have a way to update the services' API specification, otherwise, they will out-dated.

Proposal

Create an endpoint that will trigger the gathering of the given service API specification and replace the current one.

Fix Travis pipeline when PR comes from an external repo

Current behavior

Encrypted environment variables are removed for security reasons from PRs coming from another repository. So, tests that use RSA keys are failing because they are encrypted data.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions

Expected behavior

So, tests that use RSA keys pass/are green because the keys aren't encrypted data anymore.
We need to use sample RSA keys for testing purpose.

Make password of admin user during creation dynamic

Proposal

The current admin user created during deployment of the application has a hard-coded password but it would be better to have it more dynamic, in other words, it would be better if users can define the password when deploying it.

Convert Documentation from Markdown to reStructured

Proposal

reStructured has more functionalities and works better with Read the Docs. So, it would be better to migrate all documentation files from Markdown to reStructured.

Make UUID field as primary key for all models

Proposal

A universally unique identifier (UUID) is used to identify information. All objects will be reachable by their UUID field and have it as their primary key. It helps to easily identify and retrieve objects from multiple services and databases.

Create a tutorial explaining how to deploy Buidlly to Minikube

Proposal

We need to have a tutorial explaining how to deploy Buildly to a minikube instance, so they will also use the created Helm charts for that.

Make Buildly Core work as an Authentication server

Proposal

We want Buildly Core to allow us to separate the Authentication Server and the Resource Server. Django OAuth Toolkit provides a rfc-compliant introspection endpoint, based on the RFC 7662. Django OAuth Toolkit also allows to verify access tokens by the use of an introspection endpoint.

In order to make Buildly Core behave as an authentication server, we need to implement and configure an introspection endpoint in it.

Further info

https://django-oauth-toolkit.readthedocs.io/en/latest/resource_server.html

Sending files with POST requests fail with the async gateway

Bug report

POST requests with file to the async implementation of the gateway fail.

Current behavior

When users send a file in a POST request, the server returns a 500 error,

Expected behavior

When users send a file in a POST request, the server routes the request to the right service and return the response without any problem.

Environment

Buildly version: 0.3.0
Logs: Request

curl -X POST \
  http://localhost:8080/async/documents/documents/ \
  -H 'Authorization: JWT ...' \
  -H 'Content-Length: 853' \
  -H 'Content-Type: multipart/form-data' \
  -H 'Host: localhost:8080' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -F file_type=gif \
  -F file=@/images/test2.gif \
  -F file_name=test2.gif

Response

{
	"file": [
		"The submitted data was not a file. Check the encoding type on the form."
	]
}

Allow Buildly Core to integrate Serverless

Store the services' API specification in the database

Feature Request

Store the services' API specification in the database

Proposal

When API specifications are retrieved from their services, they should be stored in the api_specification field of each LogicModule. So, it can be retrieved faster from the database and be used in the gateway routing.

New app or migrate question and response

Service Level Permission

Whenever a user wants to use any service, we should be able to implement the service level permissions such as Read, Write, Update, Delete. Based on permissions the user should be able to use/review it.

Optimize the Buidlly Core Gateway routing

Story

The gateway routing has to be optimized and one of the first points we will tackle is the gathering of OpenAPI specs. The main idea is to store services' specifications in the Buildly Core database, so it will be easier and faster to retrieve it.

Tasks

1. Rename the relationship field to store API specs #253
2. Store the services' API specification in database #254
3. Update Buildly Core to get API specifications from the database #255
4. Create an endpoint to refresh API specifications #256
5. Remove any remaining dead code

Services

DELETE requests fail with async gateway

Bug report

DELETE requests using the async implementation of the gateway fails.

Current behavior

When we send a DELETE request using the async version of the gateway implementation, a 500 error is returned but the object is deleted.

Expected behavior

When a DELETE request is sent to the async implementation of the gateway, the response from the service should be returned and the object deleted.

Enviroment

Buildly version : 0.3.0
Logs: Request

curl -X DELETE \
  https://localhost:8080/async/inventory/produtcs/2/ \
  -H 'Authorization: JWT ...' \
  -H 'Host: localhost:8080'

Core Lean or Scale

Remove User, Organization and Groups and Related tables to Core APP

Feature Request

The User and groups models should be moved out of workflow into the Gateway app.

Proposal

The isolation of the workflow app means the user management should be handled inside of the gateway. It's a common enough request to be able to manage permissions in a gateway that we should generate the models and tables each time.

Bug report

N/A

Current behavior

Currently part of workflow model

Expected behavior

We should be able to disable workflow and still operate gateway and basic permissions.

Enviroment

Buildly version : x.y.z
Logs: ``

Create Buildly Core Operator for K8S

Proposal

As a Buildly user, I’d like to have a way to specify Buildly as a CRD on Kubernetes and have an operator installed in my cluster, so Buildly core will be deployed or updated based on the buildly definition created by me.

https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/

Create custom Health Check to check services availability

Proposal

We should be able to check the health of services connected to Buildly Core and for that, we need to implement a custom health check with the current library used in the project. Users should also be able to choose if a service will be checked or not, enable/disable.

Further info

Django health check library documentation:
https://django-health-check.readthedocs.io/en/latest/readme.html#writing-a-custom-health-check

Add user group permission check to service level

Proposal

We would like to define what group of users will have access to which services, so we can have services available only for admins, guests and so on. In case a service isn't assigned to a role, the role is applied to all services.

• Superuser can access any service
• Services without a core group can be accessed by anyone
• Services with org core group mean that users from the specified organization will have the specified permissions in the core group
• Services with global core group mean that all users will have the specified permissions in the core group

Implement invitation token invalidation

Current behavior

After a user registers with an invitation link the invitation token another person can use it for a second time and create another user.

Expected behavior

After a user registers with an invitation link the invitation token should be invalidated to avoid being used a second time and create another user.

Rename the relationship field to store API specs

Feature Request

As part of the optimization of the gateway routing performance, we need to store API specifications in the database.

Proposal

Rename the relationship field to api_specification because the relationship field isn't used anymore. It was initially being used by the old implementation of Mesh but it's deprecated and will be removed soon.

OpenAPI doc shows error if an instance of a logic module isn't available

Current behavior

When I add one logic module with an URL that isn't available yet and go to the OpenAPI documentation, an error is shown instead of showing the available endpoints/services but the non-available one.

Expected behavior

When I add one logic module with an URL that isn't available yet and go to the OpenAPI documentation, all available endpoints/services are shown but the non-available one.

Environment

• Buildly version: 0.2.3
• Logs:

Create sub repository for Datamesh

Feature Request

Each component should be added as an optional at build or post build component that can be toggled on or off.

Proposal

For Datamesh we will need to move it out of the primary buildly-core repository into it's own repo that can be added as a sub-repository when enabled in the admin UI or CLI.

Bug report

N/A

Current behavior

Datamesh is included with each clone of the repository as part of the main repo not a sub repository and can not be toggled off. All dependencies are required across models and views.

Expected behavior

User should be able to remove or comment out the inclusion of the app from the settings file and the app should still function but without workflow functionality.

Enviroment

Buildly version : x.y.z
Logs: ``

CLI stepwise wizard for creating a buildly core application and service.

Implement the connection between Service discovery and the Gateway

Make Gateway routing more flexible

Proposal

The Gateway has to be able to handle different path structures with different names, for example: /documents/<file_id>/ and /documents/<file_id>/thumbnails/<thumbnail_id>/; but currently, it accepts only one "identifier" in the path and it has to be <id> or <uuid>.

Update API landing page

The current landing page for the api has a link to the documentation and api endpoints but the layout and design looks like TolaData circa 1998. We need to update it to look a little more Buildly but still make it easy to configure.

Log application to the standard output by default

Proposal

The easiest and most embraced logging method for containerized applications is to write to the standard output and standard error streams. Therefore, we should log to stdout by default and add some configuration to enable logging to file.

Create endpoints to manage services

Proposal

We should expose the LogicModule model via endpoints so administrators can manage all services.

Create API endpoints to manage access tokens

Proposal

We need to create API endpoints that admin users (only them) can list and/or delete generated access & refresh tokens. We should also be able to filter the access tokens by user. All the fields should be shown.

Remove GraphQL integration

Proposal

As we decide to not have a GraphQL integration, for now, there’s no need to keep schemas and endpoints for it. We need to remove the code that is still in the project because it also exposes data without authentication and authorization.