kyma-incubator/compass

A flexible and easy way to register, manage and group your applications

compass's Issues

Define a contract between Runtime Agent and Compass

Description

AC

• Define communication between Runtime Agent and Compass. Provide documentation as an architecture diagram
  • take into account that Director needs to know EventGateway URL
  • take into account that Director needs to know for all API spec, that runtime has been created (empty credentials)
  • Establishing a trusted relation based on client certificates (Agent -> Connector)
  • Configuring the runtime with configuration provided from Compass (Agent -> Director)
• Define how to setup and configure Runtime Agent. Document as an architecture diagram

Pickup storage type for compass components

AC

• easy operations / no operations (Priority: super high)
• ability to search Applications and Runtimes by labels (e.g. groups) (Priority: super high)
  • list all labels
  • list labels by key
  • list applications by label
• the same storage type for the Runtime and Management Plane mode (Priority: high)
• store files around 10MB (Priority: high)
• cross-region replication (Priority: medium)
• no vendor lock-in (Priority: medium)
• decide which go libs we will use for storage
  • take into account migration of schemas
  • take into account the creation of schema support

Implement Director's mocks for Application

Description

Implement mocks for the Application domain. Prepare implementation for:

• resolvers
• services
• converters
• in-memory repositories
• models

Remember about unit tests for all these parts.

Investigate compatibility with Kyma's Headless CMS component

As we discussed, we want our agent to be able to create Kubernetes' custom resources that will be consumed directly by Kyma's Headless CMS component. We don't want to rely on existing application-registry component to avoid having two registries in Kyma at the same time.

We should decide which Headless CMS features we want to support. For example:

1. Do we want to support multiple source types (single, package, index)? At the moment, application-registry component in Kyma supports only single files.
2. If we want to support packages, do we also want to support filtering by regex?
3. Which DocsTopic properties do we want to keep in our schema (displayName, description)?

Current definition of DocsTopic custom resource, that is consumed by Headless CMS can be found here and here is the documentation and example of said custom resource.

AgentAuth is not populated when creating Runtime

When I execute the following query:

mutation {
  result: createRuntime(
    in: { name: "runtime-1", description: "runtime-1-description" }
  ) {
    id
    name
    description

    agentAuth {
      credential {
        ... on BasicCredentialData {
          username
          password
        }
        ... on OAuthCredentialData {
          clientId
          clientSecret
          url
        }
      }
    }
  }
}

I got the following error:

{
  "errors": [
    {
      "message": "must not be null",
      "path": [
        "result",
        "agentAuth"
      ]
    }
  ],
  "data": null
}

Additionally, I do not see any errors in the director logs which is strange because of this is a server side error.

Create an overview section about Compass

AC:

• Create an overview section about Compass in the main Compass readme.
• Improve the structure of the main readme.

Install PostgreSQL

AC:

• Copy PostreSQL chart and paste as compass subchart - #131
• Add ability to toggle local PostreSQL installation and use GCP PostgreSQL instance

Hint: See Use embedded etcd flag in service-catalog chart in Kyma, which works similarly to what we want to achieve.

Custom scalars implementation

Description

Implement custom scalars with user-defined types

Reasons

Part of #3

Define DB schema

AC:

• Create pre-install Job for compass chart, which creates and upgrades database tables (for now, only one Runtime table). Prepare separate entry point for Director or even whole new Docker image with upgrade job.
• Use migration tool from investigation: #69

Implement Auth in Director

Description

Figure out how to implement Auth, which contains Auth. Auth has circular reference to itself.

Remember about RuntimeAuth collection for APIs, agentAuth for Runtime and Auth for FetchRequest, ApplicationWebhook.

Do we want to store Auth in a separate table? If so, agentAuth and all auth fields should be resolvers and there should be separate services.

Anyway, we will need one converter, so maybe it would be better to create auth pseudo-domain.

Implement Director's Webhooks mocks

Description

Implement mocks for the Application domain. Prepare implementation for:

• resolvers
• services
• converters
• in-memory repositories
• models

Remember about unit tests for all these parts.

Set Auth while creating Runtime and Application

Description

Set AgentAuth for Runtime and auth for Application while creating them.
Figure out the API and what to generate - We should allow providing which credentials type should be generated. User has to be able to choose which auth type to generate

• Add input for user to pick auth type (for Runtime/Application creation add new field for AUTH_TYPE enum)
• We need to generate:
  • client_id/client_secret pair
  • basic username/password out of scope
  • one-time token
• Should we use hydra-maester (operator for generating client_id/client_secret pair)? Ask corresponding team

Tenancy support for Director

Description

For compass as an experimental module in the Kyma, we need to support already a way to persist and fetch data in a given tenant.

AC

• Implement a header: compass-tenant that will be taken by gateway and proxied to director
  -[x] if tenant not provided - fail request (think about status code)
• remove tenant attribute which is returned in APIs
  • remove tenant from GraphQL schema, but keep it in db
• add integration tests

APIs mock implementation

AC

• create list of test-cases showing full flow, implement main ones (when test-cases ready ask for review @PK85 )
• provide interfaces for APIs defined in #2 and provide mock implementation in gateway component
• store data in-memory database
• interfaces needs to be already nicely defined that will help us in next issues to replace mock with production implementation

Add pagination to GraphQL API and fix CSRF tokens

AC

• add pagination to graphQL
• fix CSRF tokens

Use database for Runtime domain in Director

AC:.

• Implement repository for Runtime, which uses PostreSQL. Remember about filtering by labels in List method. Skip pagination for now.
• Use tools picked in toolbox investigation: #69
• Create real database model types for Runtimes. Treat the existing models as domain types. Move it to a proper directory and figure out type conversion.

Pagination will follow in the next task: #53

Support Application Connector service REST API in the API-gateway

AC

• Define API as a graphQL (CERT support) and add to Connector Component (JWT skipped, for now, we need additional research) @crabtree PR: #32
• architecture diagram added
  • REST to graphQL @crabtree PR: #42
  • for many types of auth, like CERT, JWT
• Implementation on gateway side for REST and proxying to GraphQL

Agent MVP implementation

Description
Implement Agent and fulfill the contract between Agent and Director/Connector.

AC

• Add the Agent component into the Compass, in addition:
  • helm chart update
  • release pipeline
• Keep in mind that all calls can fail so retries are mandatory
• The agent takes configuration from Compass and manages Application CRs and all their dependent objects
  • more or less behave identically as we have right now in application-registry component in kyma
  • Compass exposes specs via blobs, agent takes it and stores in minion internally in runtime
• The agent periodically applies changes of the configuration from Compass on to the runtime.

Stretch (depends on the status of connector migrated to compass):

• The agent is able to establish a trusted relation between runtime and Compass based on client certificates
• The agent is able to renew the client certificate for the runtime before it expires

Add Application/Runtime checks to appropriate Director services

Description

Some Director mutations (related to the creation of Application subresources, e.g. addApplicationWebhook) don't check if an Application with the specified ID actually exists. Right now this happens with Webhooks, Documents, APIs and EventAPIs.

There is also a case where when adding AuthRuntime to API, Runtime should be checked.

Those services should get Application/Runtime service injected and use it to check if Application/Runtime with received ID exists.

Expected result

Attempt to create a subresource of Application (eg. Webhook), when there's no application with specified ID should result in an error.

Actual result

Subresource is created even when an Application with passed ID doesn't exist.

Steps to reproduce

1. Make sure there is no Application with ID foo
2. Create Webhook using addApplicationWebhook mutation with applicationID parameter set to foo

Define a contract between RuntimeHelper and Compass

RuntimeHelper - is a new name for RuntimeProvisioner. As we found that, that agent could be installed into already existing Runtime.

AC

• create a diagram, how to apply a trust between the RuntimeHelper and the Compass components

Notes

• We need a way to register new trusted RuntimeHelpers into compass

Implement Director's EventAPI mocks

Description

Implement mocks for the Application domain. Prepare implementation for:

• resolvers
• services
• converters
• in-memory repositories
• models

Remember about unit tests for all these parts.

Design Registry API

AC

• API layer for storage access
• update architecture document

Prepare simple Gateway implementation

Description

• Prepare simple Reverse Proxy implementation for Gateway.
• Configure charts for Gateway and Director, to work properly (pass requests to Director)

Reasons

Implement repository for Runtime domain

AC

• PostgreSQL installation (should be configurable to use GCP PostgreSQL instance) (1)
• schema definition (2)
• implementation itself according to investigation from previous iteration (8)
• paging implemented and working e2e from API to storage
• filtering by labels implemented

Implement Director's API mocks

Description

Implement mocks for the Application domain. Prepare implementation for:

• resolvers
• services
• converters
• in-memory repositories
• models

Remember about unit tests for all these parts.

Implement Director's Documents mocks

Description

Implement mocks for the Application domain. Prepare implementation for:

• resolvers
• services
• converters
• in-memory repositories
• models

Remember about unit tests for all these parts.

Multi-tenancy in compass

AC

• decide how to map or not tenants from given Cockpits into compass

Implement repository for Application and subdomains

Description
Repository layer for application and other subdomains.

AC

• schema created
• implementation
• documentation
• paging implemented and working e2e from API to storage

That story will make all compass APIs to be persisted in the DB

Structure Director project

Description

Structure Director project similarly to Console Backend Service

Decide how components communicate each other

AC

• proposal and decision about what is a list of components and how they communicate each other
  • decision about where we will have graphQL layer,
    • in dedicated service (not gateway)
    • in every service
  • decision about what gateway responsibility is about (e.g. just token validation)
  • decision about protocol that services will communicate each other

Concept for storing credentials in the Kyma Runtime only

Description
In our first architecture phase we designed compass as a single source of truth for credentials configuration between Applications and Runtimes. See: for example API credentials flow: https://github.com/kyma-incubator/compass/blob/master/docs/architecture/api-credentials-flows.md

AC

• create a proposal about credentials exchange between Application and Runtime without storing them in compass
• provide architecture diagram
• update compass API to reflect that functionality
• have a clear message if:
  • only compass holds Application Credentials
  • only Kyma Runtime holds Application Credentials
  • we allow to mix two above points

S3 as storage for App Registry (CRUD + search by groups)

AC

• request AWS subscription
• create POC which shows how to query data stored in S3
  • includes query via minio gateway and directly to S3
  • show what will be a structure of data in S3 to allow nice query capabilities
  • show investigation of Apache Parquet output format with AWS S3
• check a status of GPC and Azure storages and Parquet support
• provide a short guide how setup and how work cross-region replication for AWS S3

Inconsistency in Director GraphQL API between input and types

1. DocumentInput defines 2 fields, that are missing in the Document:

• displayName
• description

2. APISpecInput defines format as a required field, but APISpec defines it as an optional field.
   The same for EventAPISpec.

AC:

• fix inconsistency
• update integration tests to check if fields are properly populated

Return proper GraphQL errors for end user in Director

Description

Currently we're returning all Go errors to the end user. We should improve them.

AC

• Find format in which we want to return errors that will fulfill all kind of errors (e.g. tenant or entity not found, etc.)
• Figure out how to document errors for GraphQL operations
• Define and implement minimal set of error types (e.g. TENANT_NOT_FOUND, ENTITY_NOT_FOUND, etc.)

Technical requirements:

• Reuse gqlerrors package from Console Backend Service or write something similar, but better
• return machine-readable error codes using https://gqlgen.com/reference/errors/
• Log internal errors in Resolvers using logrus
• Remember about wrapping errors
• unification of error handling across all layers
• suggestion: write possible errors in comments for GraphQL operations

Define a contract between ApplicationHelper and Compass

AC

• create a diagram, how to apply a trust between the ApplicationHelper and the Compass components
  Notes
• document a way how to register new trusted ApplicationHelpers into compass
• ApplicationHelper could register many applications

Get rid of Tenant from Director GraphQL API

From a user perspective, Director GraphQL API should be visible as a single-tenant API.

Create static documentation for compass APIs

AC

• invesitage possible solutions, see links section
• create static documentation for compass APIs
• add to compass/docs and mention in the main readme file
• host on gateway and bind "try it now" action with our existing graphQL playgrounds

Links

• https://github.com/wayfair/dociql
• https://github.com/2fd/graphdoc
• take a look how github document their APIs: https://developer.github.com/v4/

Prepare Director mocks for Runtime domain

Description

• Prepare interfaces for Runtime API defined in #2
• Prepare mocks that use an in-memory database

Update examples for mock implementation

Description

AC:

• Adjust examples in internal/graphql/examples to match current API with mock implementation.
• Full flow for applications (including Documents, EventAPI, API, Webhooks, etc.) and runtimes
• Include link in docs to the examples
• Add index document for GraphQL query/mutations examples (md file as an index)
• Figure out how to keep examples in sync (maybe use them in tests?)

Create models for Labels and Annotations in Director

Description

• Create models for Labels and Annotations
• Use them in services for Runtime and Application, instead of passing key and values, like here: https://github.com/kyma-incubator/compass/blob/master/components/director/internal/domain/runtime/service.go#L90

Reasons

Labels and Annotations could be extended very soon. We need to have flexibility with the types

API - Multiple credentials for APISpec support

AC

• Our Application API must support a way to register application with many APISpec credentials. That means, for example: we have one application and that application is registered to two runtimes. They should be able to use different credentials to call given Application API like PromotionsAPI
• Also. make a modification in APIDefinition versioning - to know when an APIDefiniton is a new version for another APIDefinition.

Connector service implementation with GraphQL API

AC:

• Implementation of connector component for issuing client certificates for any client, for example, applications, runtimes
• Communication should be done via GraphQL API (initial schema available here https://github.com/kyma-incubator/compass/blob/master/components/connector/internal/gqlschema/schema.graphql)

Implement pagination

Description

• Implement generic pagination for GraphQL queries

The following link may be useful: https://github.com/stratumn/graphql-pagination-go/blob/master/array_list.go

Use MILV for Markdown link validation

Description

Use MILV for link validation in docs. Run the tool as CI step for all changed Markdown files.

Reasons

• we will make sure that all links are valid
• we have to unify approach if we want to move to kyma-project organization

API for labeling and scenarios implementation

AC

• document Label and LabelsDefinitons

  • how they work, how they depend each other
  • what LD delete does (is that possible when label exist)
  • what label key deletion means (removes all occurrances from apps and runtimes)

• implement LabelDefinition qGL Type and provide CRD (Update out of scope). Deletion removes a label from all entities (Apps and Runtimes).

  • read also support getting list of schemas for given keys
  • provide LabelDefinition if you need something more than string. String is a default, and type is not required for addAppLabel and addRuntimeLabel in that case but internally new LabelDefiniton will be created. Has fields:
    • key
    • schema for value

• default scenario already exists, plus every app and runtime (if not specified) has that label in. LabelDefiniton already exists, we support only GET operation.

• implement labels qGL Type to mutate and query scenarios

  • create a scenario by adding new app or runtime with scenario label
  • create a scenario by labels endpoint
  • above ways are synced
  • scenarios for application and runtimes has same scope (could be reused)

• validation during Label creation

• remove annotations (we dont see a need for them right now)

• filtering by labels implemented on runtimes and applications

Out of scope:

• system specific labels. We can introduce that feature with getting scopes (together with tennat) when calling API.

Add testing framework to compass project and make it runnable

Description

Add testing framework to compass project.

Reasons

Allow to execute integration test againts k8s cluster.

AC

• tests directory where we have all integration tests for all compass components #20
• release pipeline for tests image #1084
• octopus used as a testing framework #23
• add a document how to run compass #23
  • run on Kyma latest released version with minimal required modules in, eg. essential-data, test-runner, istio #23
  • investigate if we can use kymaCLI, or even kymaCLI utils, if not we have only minikube.sh script with all others from Kyma #23
• update main Readme with how to run and test compass points to detailed documents in /docs #23

Setup repository, create components structure and and provide APIs

AC

• apply https://github.com/kyma-project/community/tree/master/guidelines/internal-guidelines/repository-template into compass repository - @kfurgol #5
• Github repo setup - @kfurgol
  • Enabled required reviews for pull requests (at least one review)
  • License check required
  • Set up labels
  • Add compass to prow config #1009
• create project structure, define and add proper components catalogs - @kfurgol #6
• add release pipelines per component - @tgorgol kyma-project/test-infra#1011
• add helm charts for the components - @kfurgol #10
• investigate file upload with graphql - @kfurgol #11
• define API per components and place specification under /component/api directory
• all stakeholders can do a review of those APIs

Architecture diagrams for compass

Description
We need a place in compass project where user could understand how APIs interact wit each other. We need to provide some type of diagrams + nice descriptions about them, see details in below AC.

AC

• component diagram for compass components that are a part of management plane @tgorgol
  • basic diagram, advanced one not a part of that AC cause we still need to decide how components communicate each other
• high level systems diagram with application, management plane and runtimes @aszecowka
  • make sure we have n to n between apps and runtimes
• 1:n registration flow (sequence diagram) @pkosiec
  • user registers application (fully with provided apiSpec, credentials etc) in the management plane OR user pre-registers application, and then application registers APIs, Events etc in the management plane
  • spin many (2 is fine) runtime and map to default group to them
• n:1 registration flows (sequence diagram) @pkosiec
  • two application registered
  • they are in one new group
  • spin, configure runtime and assign a group
• diagram for "Allow specify different credentials per runtime", we need to describe two ways of provising APIDefinition credentials with and without webhook, see details: #19 @PK85
• update main Readme with architecture and "why compass" then point to detailed documents in /docs

Follow Kyma diagram guidelines: https://github.com/kyma-project/community/blob/master/guidelines/content-guidelines/diagrams.md

Compass Cockpit - POC

Description

As a first step towards standalone compass cockpit UI, provide a UI application ("Compass Cockpit").

It should be part of compass chart :

• deployment for web application
• temporarily - micro frontend definition for kyma console (cluster scope, external link (same use case as "graphana" micro frontend))

The Compass Cockpit should only expose Management Plane API :

• CRUD for Applications - (***)
• Read (List and Get) for Runtimes
• Group management

The compass cockpit should operate in multi tenant mode - All 3 Entities (Application definitions, Runtimes, Groups ) belong to a tenant and the User should be able to switch tenants in the cockpit.

The Compass Cockpit cannot expose any:

• k8s resources (pods, deployments, namespaces, etc..),
• kyma managed resources (lambdas, application CR, service catalog etc)

AC:

• View for managing Applications (***) should be reusable as micro frontend in the context of kyma console for the "Management Plane on the Kyma runtime" scenario

Extend EventAPIDefinition with additional information about custom event system

Description

We need to support two ways of using Eventing. Application can:

• not use event messaging system, and then Runtime EventService will be used
• use specific event messaging system, and then Runtime kNativeEventing must be configured properly

Reasons

We need that to support system with already configured eventing system

AC

• extends EventAPIDefinition with additional fields, like URL, Type, Auth, Channel, etc

type EventAPIDefinition {
    id: ID!
    spec: EventSpec!
    version: Version
}

• Add architecture diagram about those two possibilities how to Eventing works with Kyma Runtime