redhat-developer/app-services-sdk-go

RHOAS SDK for Go

[DEPRECATED]

SDK moved to https://github.com/redhat-developer/app-services-sdk-core/app-services-sdk-go

This repository has been replaced with app-services-sdk-core and should not be used. If you are using this SDK take a look at how to transition your project to the new SDK here.

RHOAS SDK for Go

Go packages and API clients for Red Hat OpenShift Application Services (RHOAS)

Prequisites

• Go 1.15 or above

Installation

Install the RHOAS SDK with go get:

$ go get github.com/redhat-developer/app-services-sdk-go

Import:

import "github.com/redhat-developer/app-services-sdk-go"

Management SDKs

NOTE: Some of these APIs are under development and may sometimes cause backwards-incompatible changes.

API	Status	Package
KafkaManagement	beta	github.com/redhat-developer/app-services-sdk-go/kafkamgmt/apiv1
Service Account Management	alpha	github.com/redhat-developer/app-services-sdk-go/serviceaccountmgmt/apiv1
ServiceRegistryManagement	alpha	github.com/redhat-developer/app-services-sdk-go/registrymgmt/apiv1
ConnectorManagement	alpha	github.com/redhat-developer/app-services-sdk-go/connectormgmt/apiv1

Instances SDKs

API	Status	Package
KafkaInstance	beta	github.com/redhat-developer/app-services-sdk-go/kafkainstance/apiv1
RegistryInstance	beta	github.com/redhat-developer/app-services-sdk-go/registryinstance/apiv1internal

Documentation

Documentation

Examples

Examples

Contributing

Contributions are welcome. See CONTRIBUTING for details.

app-services-sdk-go's Issues

Automate, Document and Generate SRS Data plane SDK

Rename hack/scripts to scripts folder.

hack is golang specific but scripts will be used across the entire fleet of sdks

[Proposal] Break down API client generation into 2 x requests

This proposal spawns out of a conversation in redhat-developer/app-services-sdk-js#54 (comment)

There are cases where the OpenAPI spec file needs to be modified by the SDK maintainer before merging. This would be more common with data plane APIs.
A way to do this would be to create an initial PR (through automation) with the OAS, where the dev can make changes. A second PR is created then which generates the API client.

1. A pull-request with the new OpenAPI changes is created.
2. The maintainer either a) merges it or b) makes necessary changes to the file then merges it.
3. When the pull request is merged another pull request is auto-created with the generated API client.
4. The maintainer can review and test this pull request, then merge.
5. Create a release.

cc @wtrocki @secondsun

Autoupdate of the Apicurio registry schema

Apicurio registry schema evolves and updates frequently. We need to have autoupdate feature

Integrate Kafka Admin API

Needs to be done after the Expose Kafka instance API to end users epic

Error helpers should be part of the SDK's

Examples:
https://github.com/redhat-developer/app-services-cli/blob/srsdata/pkg/api/srs/error.go#L97-L122

Provide (copy) documentation for authentication and other useful elements that are cross cutting concerns.

See SDK-js for example on how this can be done.

Replicate OpenAPI update changes to other SDK's

Go seems to have better messages:

Provide common table/list of the supported APIs

We can technically generate that somehow based on the API descriptions etc.
Instead of duplicating effort for go, js and java we can do that first for go SDK and then replicate it across others

Columns

• Name
• Package name
• Stability level
• Latest Release
• Docs Link

Investigate if we can override or customise the generated API client READMEs

The generated README for each API client is very unclear and convoluted - it is aimed at an API client which is generated inside your own code - not as an externally consumed package.

Documentation for API Endpoints onwards is useful so this is not just a case of not generating the README and writing our own.

OpenAPI Generator has customization options through templates so this is a possible approach we should investigate further.

Add linting for OpenAPI documents on PR level

Linting will help to see if the change that was merged to master is valid.

Document steps required to integrate new services

Define the packages which should be added to the SDK

There are some packages which will be useful/required to use the SDK library:

• Config - for configuration file management.
• Authentication
• Authorization
• Logging

Add examples that can be used when building to actually verify if there are breaking changes

Examples will be used for testing SDK etc.

Update readme to include Instances SDK's

It will be good to get split between Management SDK's and instances SDK's.
I wanted to reference someone to our readme (and documentation format)

./scripts folder contains unused scripts

fetch_api and generate, generate_all etc. are no longer used causing confusion on how to generate codebase.
Scripts arent working anymore.

Introduce go submodules to use independent versioning

For projects that use multiple SDK sometimes we need to update one and stay with old version of the other.

Imagine scenario:
v0.5.0

• kafka-mgmt
• registry-mgmt

Now we want received 2 changes for those apis independently but only want to use registry change.
Users should be able to import SDK's individually with different versions

• kafka-mgmt 0.5.1
• registry-mgmt 0.5.0

Use a tool like openapi-diff to generate an informative changelog

https://www.npmjs.com/package/openapi-diff groups changes into:

• Breaking changes
• Non Breaking Differences
• Other

This could be used to generate a clear diff of what has changed between releases and could be added to the release notes.

Discussion - Security for Automation

Reminder to discuss security implications

Set up CI workflow to trigger API client generation

It should more or less follow this:

SDK calls returning files are improperly decoded as nil

See:

https://github.com/redhat-developer/app-services-sdk-go/blob/dates/registryinstance/apiv1internal/client/api_artifacts.go#L1311

Mixture of upstream/product folder names in the SDK

kafkamgmt - uses common world
srsmgmt - uses srs which is internal name, same as kas.

kafka vs kas
srs vs serviceregistry vs registry

It will strongly suggest renaming package to registrymgmt before we get more usage.

Integrate KAS Fleet manager API client

Add changelog for releases and in the main repository.

We commit messages like in JS-SDK etc.

Add ability to release all SDKs at once

We need ability to release all SDK at once by automation.

[Discussion] changes to the automation workflow

We possibly need to revisit the automation workflow of the SDKs - the initial vision was based on the assumption that API clients would be divided into sub modules and sub packages with individual releases. This fit the current flow perfectly as we could merge changes when an OpenAPI update happened to give client side consumers the ability to use the pre-release versions.

We diverted from the original approach to have a single release per SDK for simplicity as we faced some challenges with multiple packages - plus it is easy to add sub-packages the future but not easy to remove them once embedded in clients.

Now that we are doing one single release the main problem is that we are merging changes from different clients but they all don't follow the same release cycle. KAS Fleet Manager pushes to production a few hours or a day after merging. Kafka Admin API however is when the developers are ready, so it could be any time. This is a problem because we have merged Kafka Admin API changes which may include a new endpoint which is not deployed yet, and we need to release the more recent KAS Fleet Manager changes - so we have now released the SDK with changes to the kafka admin client which will not work.

We could revert to the original approach suggested by updating the SDK whenever a release is made - the problem with that is that currently there is no consistent release procedure across all APIs and the process for automation of this has not been finalised.

An option is to require all APIs to use tags/releases on GitHub to indicate a release to production. This tag/release can be a visual thing only and does not need to hook into the deployment process but could do so in the future. It can be a manual thing for now, which could be done in a few seconds on the API side.

The production deployment process is still manual and the only other option I could see is polling api.openshift.com every few hours to see if the OAS version has changed but this would only work for control plane APIs and we would need a different method for data plane APIs.

Registry data plane SDK

Add SDK for service registry

Change commit messages in all repositories

Currently our commits have:

chore(openapi) ...

I think we should include name of the component inside the commit and use fix.
This way we can ensure that we will get meaningful changelog.

fix(kafka-admin) ...

Generate should run as PR check to see if is returning 0 status code.

We should be able to know if introduced change will still allow us to generate code etc.

Example of the problem:
https://github.com/redhat-developer/app-services-sdk-go/actions/runs/947881577

SDK UX issues and problems with packages

My further recommendation for all apis will be to:

• Drop apiv1 etc. We do not actually have the means to keep API's versions like Kubernetes does and it is confusing for people
• Use the same package name as a folder - currently we are importing apiv1 but the package name is different which is confusing

Double imports (generated code and wrapper can be fixed by having both in the same package.
We will only need to add ignorefile for the generator in that instance.

Related issues:
#125

Remove all god mod hacks from submodules

It was a required temporary hack to migrate packages to sub modules. Now that the initial releases have been completed we can remove this hack.

This will remove the dependency on the root module.

This has been completed for one module already in #243

For further reading on this hack see https://github.com/golang/go/wiki/Modules#is-it-possible-to-add-a-module-to-a-multi-module-repository

Generate all locally is broken

Previously we were basing on the file names to perform generation. Now generation uses ID.
We need to change scripts to be able to generate all things locally.

I think we need to change scripts to look into package structure to match id per file first?

Error:

Error: Unknown api file. Check if API is defined in .openapi/""

@craicoverflow Any ideas yourself?

SDK build on PR should happen over the generated code

SDK build on PR should happen over the generated code rather than current codebase.
This way we can actually check if PR is even valid and compiles.

I have applied this flow in redhat-developer/app-services-sdk-js#77

Unable to update partitions for a topic

Staging server throws the error "Unrecognized field \"partitions\". It expects a field numPartitions.

Here's the payload in UI where it works fine:

Create tests based on examples that will help us to setup proper conventional commits when merging

If our API dosn't compile or lint after generation we should change tests and create BREAKING commit/changelog.
Otherwise we might create fix or feature depending on original commit that came from the source repo.

Getting information about commit or PR of the source repository will be additionally very useful when dealing with this problems

[Idea] Create script to automate fleet operations.

There will be a number of common elements across SDK's like README/docs generation, Platform releases.

Since we have 4 languages (Go, Java, Python and JS) this will trigger a number of manual work required for them.

Let's say we want to regenerate all packages for the new version of the codegeneratior - that could be automatically done using checkout operation and generate from #41

Define unique identifier for each API client

Currently the metadata config uses the name of the OpenAPI spec file as the key - but this is not always unique.

{
  "kas-fleet-manager.yaml": {
    "apiGroup": "kafkamgmt",
    "apiVersion": "v1",
    "release_level": "beta"
  },
  "srs-fleet-manager.json": {
    "apiGroup": "serviceregistrymgmt",
    "apiVersion": "v1",
    "release_level": "alpha"
  },
  "rest": 😞 
}

We need to define a unique identifier format for this, something like:

{{ github.repo }}/{{service-name}}

{
  "github.com/bf2/kafka-mgmt": {
    "apiGroup": "kafkamgmt",
    "apiVersion": "v1",
    "release_level": "beta"
  },
  "github.com/bf2/serviceregistry-mgmt": {
    "apiGroup": "serviceregistrymgmt",
    "apiVersion": "v1",
    "release_level": "alpha"
  },
  "github.com/bf2/kafka-admin": 😄 
}

cc @wtrocki @secondsun

Initial architecture

Before moving to the automation phase we should set up an initial project architecture following best practices.

Proposed layout

The kubernetes Go client has a good layout which we can adopt. See this example.

Example layout:

├── apis
│   ├── accountmgmt
│   │   └── v1
│   │       └── go.mod
│   ├── kafka
│   │   ├── admin
│   │   │   └── v1beta1
│   │   │       └── go.mod
│   │   └── fleetmanager
│   │       └── v1beta1
│   │           └── go.mod
│   └── serviceregistry
│       └── v1alpha1
│           └── go.mod
├── auth
├── config
├── go.mod
├── logging

Sub-modules can be used to allow users to import specific packages without having to import the entire SDK into their project:

import (
   "github.com/redhat-developer/apis/kafka/admin/v1beta1"
   "github.com/redhat-developer/apis/kafka/fleetmanager/v1beta1"
)

All apis should be regenerated all the time and removing previous files

I have noticed that service registry have files that are not present on the openapi.
In other SDK's we remove generated code before generation.
Go might need that as well.

Phantom models (like BlahAPI.go) will cause major problems in the future.

Rewrite generation scripts in Go

The Bash script are difficult to work with and maintain. We need to use a programming language to give proper error handling.

Use a custom template to create an API client

Currently the SDK has a custom wrapper and README to give us control over some of the generated APIs we do not like. It makes more sense to use a custom template for this.

These can be added here: https://github.com/redhat-developer/app-services-sdk-go/tree/main/scripts/templates

This would also resolve #54

SDK should include every sequential change of the OpenAPI

Having 2 changes to the same API should be two different PR's and commits and changelogs.
Getting bulk change because of lack of PR review causing some problems as we missing what commits changed what and it is really hard to reliably change API.
This way we can actually ensure an informative changelog.

Solution for this problem is to create separate PR and branch for every event: #126

Generate interface mocks dynamically based on tags

The OpenAPI spec files should have a root level tags object specified if there are tags. This will let us generate interface mocks based off the tag names.

Generation is overriding current PR's

When 2 changes in the generation will happen my commits were lost as automation forcepushes changes to the same branch.
Additionally branch name should include id of the API.

Solution will be to have suffix added to branch with apiID and some timestamp?
Without that we are likely to loose changes.

TL;DR - chore/add-openapi gets overwritten every time changes happen - We have lost update of one api that were overridden by other.

Add accountmgmt service

This would need to be pulled rather than pushed - maybe with a daily cron job or something?

cc @wtrocki the JS-SDK may need this also.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

github-actions

.github/workflows/build_and_validate.yaml

• actions/checkout v4
• dorny/paths-filter v3@de90cc6fb38fc0963ad72b210f1f284cd68cea36
• actions/setup-go v5
• golangci/golangci-lint-action v6@a4f60bb28d35aeee14e6880718e0c85ff1882e64

.github/workflows/generate.yaml

• actions/setup-go v5
• actions/checkout v4
• peter-evans/create-pull-request v6@c5a7806660adbe173f04e3e038b0ccdcd758773c

.github/workflows/generate_errors.yaml

• actions/checkout v4
• actions/setup-node v4
• peter-evans/create-pull-request v6@c5a7806660adbe173f04e3e038b0ccdcd758773c

.github/workflows/push_security_sdk.yaml

• actions/checkout v4
• peter-evans/repository-dispatch v3@ff45666b9427631e3450c54a1bcbee4d9ff4d7c0

.github/workflows/update_openapi.yaml

• actions/setup-go v5
• actions/checkout v4
• peter-evans/create-pull-request v6@c5a7806660adbe173f04e3e038b0ccdcd758773c

gomod

accountmgmt/go.mod

• go 1.15
• github.com/redhat-developer/app-services-sdk-go v0.14.0
• golang.org/x/oauth2 v0.21.0

auth/go.mod

• go 1.15
• golang.org/x/oauth2 v0.21.0

connectormgmt/go.mod

• go 1.15
• github.com/redhat-developer/app-services-sdk-go v0.14.0
• golang.org/x/oauth2 v0.21.0

go.mod

• go 1.15
• golang.org/x/oauth2 v0.21.0

kafkainstance/go.mod

• go 1.15
• github.com/redhat-developer/app-services-sdk-go v0.14.0
• golang.org/x/oauth2 v0.21.0

kafkamgmt/go.mod

• go 1.15
• github.com/redhat-developer/app-services-sdk-go v0.14.0
• golang.org/x/oauth2 v0.21.0

registryinstance/go.mod

• go 1.15
• github.com/redhat-developer/app-services-sdk-go v0.14.0
• golang.org/x/oauth2 v0.21.0

registrymgmt/go.mod

• go 1.15
• github.com/redhat-developer/app-services-sdk-go v0.14.0
• golang.org/x/oauth2 v0.21.0

serviceaccountmgmt/go.mod

• go 1.15
• github.com/redhat-developer/app-services-sdk-go v0.14.0
• golang.org/x/oauth2 v0.21.0

smarteventsmgmt/go.mod

• go 1.15
• github.com/redhat-developer/app-services-sdk-go v0.14.0
• golang.org/x/oauth2 v0.21.0

---

• Check this box to trigger a request for Renovate to run again on this repository

Move config package from CLI

This is actually very important to have in the SDK repo as authorization and authentication heavily rely on it and need to persist the access token somewhere.

kafkaMgmtAPI := kafkamgmt.NewClientFromConfig()

Without it, the SDK is useless to other tools such as the upstream Terraform Provider.

KAS Fleet Manager doesn't meet validator rules

https://github.com/redhat-developer/app-services-sdk-go/runs/3251943870?check_suite_focus=true

CC @craicoverflow what is the process for this?

Automate, Document and Generate COS Management and Instances SDK

Integrate Service Registry API