raystack/stencil

Stencil is a schema registry that provides schema management and validation dynamically, efficiently, and reliably to ensure data compatibility across applications.

stencil's Issues

Embed database migrations

Embed database migrations files

Non existing namespace delete is happening

Describe the bug
Stencil is successfully deleting the namespace which does not exist

To Reproduce
Steps to reproduce the behavior:

1. Run stencil namespace delete random
2. This will show that namespace has been successfully deleted

Expected behavior
It should show that the namespace does not exist.

Could we support CockroachDB, so we get clustering and region clustering in order to support the privacy laws.

support CockroachDB.
Its got multi region clustering and so conforms to the various privacy laws. A lot easier.

explanation: https://www.cockroachlabs.com/blog/introducing-super-regions/

Nature of bug

The Issue is the migrate command and pg_advisory_lock:
golang-migrate/migrate#703

stencil server migrate fails:

db migrator: try lock failed in line 0: SELECT pg_advisory_lock($1) (details: pq: unknown function: pg_advisory_lock())

The driver ( github.com/jackc/pgx/v4 ) is fine and works great with cockroach.

source of bug:

package postgres

import (
	"context"
	"embed"
	"fmt"
	"log"
	"net/http"

	"github.com/golang-migrate/migrate/v4"
	_ "github.com/golang-migrate/migrate/v4/database/postgres"
	"github.com/golang-migrate/migrate/v4/source/httpfs"
	"github.com/jackc/pgconn"
	"github.com/jackc/pgx/v4"
	"github.com/jackc/pgx/v4/log/zapadapter"
	"github.com/jackc/pgx/v4/pgxpool"
	"github.com/pkg/errors"
	"github.com/raystack/stencil/internal/store"
	"github.com/raystack/stencil/pkg/logger"
)
``

## Reproduction of bug

here is single server.

.env
```sh
COCKROACH_SRC_DATA_FSPATH=$(PWD)/.data/cockroach
COCKROACH_SRC_LISTEN_ADDRESS=localhost:26257
COCKROACH_SRC_HTTP_ADDRESS=localhost:8080
# cluster ( i you want it )
COCKROACH_SRC_JOIN_ADDRESS=localhost:26257,localhost:26258,localhost:26259

start ( single cockroach node ):
cockroach runs on all desktops btw. you don't have to use docker.

cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background

config:

# Raystack Stencil Configuration
#
#
# !!WARNING!!
# This configuration file is for documentation purposes only. Do not use it in production.
#
# Stencil can be configured using a configuration file and passing the file location using `--config path/to/config.yaml`.
# Per default, Stencil will look up and load file ~/.stencil.yaml. All configuration keys can be set using environment
# variables as well.
#

# server controls the configuration for the GRPC server.
# The port to listen on. Defaults to 8080
port: 8080
# Timeout represents graceful shutdown period. Defaults to 60 sec
timeout: 5s
# Configuration for profiling application with new relic
newrelic:
  appname: example
  enabled: false
  license: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
# Database configurations for stencil backend
db:
  # Connection string for postgres database
  #connectionstring: "postgres://postgres@localhost:5432/db"
  connectionstring: "postgres://root@localhost:26257/defaultdb?sslmode=disable"

client: java: add a generic serialzation api

Currently we only support parsing bytes to protobuf dynamic message We need a geneic serlization api that can help us do the reverse.

One idea is to use JSON as the intermeddiate format but it can have performance issues.

Add python client for stencil

Add support for AVRO format

Problem
Not able to upload/download Avro schemas in stencil

Proposed solution
Add support for avro schemas.

additional information

• Stencil should be able to handle a fully contained avro schema.
• Analyze how references can be used in avro (similar to imports in protobufs)

Improve stencil schema commands

List schemas in a given namespace.

$ stencil schema list -n namespace

View schema metadata for a given schema.

$ stencil schema info <name> -n namespace

List versions of a given schema.

$ stencil schema version <name> -n namespace

View a schema document/file for a given schema id.

$ stencil schema view <name> -n namespace -v version 

Download a given schema to a specified path.

$ stencil schema download <name> -n namespace -v version -o ./file

Delete a given schema.

$ stencil schema delete <name> -n namespace

Edit metadata of a given schema.

$ stencil schema edit <name> -n namespace -f format  -c compatibility

Show diff between two schemas that are already available on the server.

$ stencil schema diff <name> -n namespace --source earlier-version --target <later version> --prefix 

Check the compatibility of a schema.

$ stencil schema check <path> --against <name> -v version -n namespace

Generate the graph of relationships in a given schema.

$ stencil schema graph <name> -n namespace  -v version

Upload a particular schema version

$ stencil schema upload <name> -n namespace -f path_to_file

add search API

problem
Unable to find if schema exists in stencil schema registry across namespaces

proposed solution
Provide search API that takes query and returns matching schemas and field names

additional information

• show the properties in search results that's common across different schema types

Execute protoc from stencil CLI

Is your feature request related to a problem? Please describe.
We need to run protoc to generate a descriptor set. We can enhance Stencil CLI to do the job so that all things related to proto come under a single ecosystem of Stencil.

Describe the solution you'd like
We will enhance Stencil CLI to add one more command which will delegate the description set generation to protoc executable (which must be installed in the PATH)

StencilClient configured with ondemandRefresh doesn't refresh for nested protos

Describe the bug
Stencil Client when configured for ondemand refresh it uses ProtoParserWithRefresh which checks for any unknownfields and refreshes the cache, but unknownfields does check only in the root proto, so for nested protos if there are new fields being added it won't refresh the cache.

To Reproduce
Create a proto & add a new field at a nested level or modify an enum this should be reproduced in that case while deserializing

Add ruby client for stencil

Provide CLI defaults

Problem:

All Stencil commands require host and namespace. This makes using stencil verbose.

Solution

Stencil CLI should load the default host and namespace from the Stencil CLI config. This will allow the stencil host and namespace to become the default for most of the commands. The user should also be able to override both of these by providing a flag.

Schema validation for avro, json

Problem
No validation is in place for avro and json schemas. User can accidentally upload schemas.

Proposal
Have a validation check for both avro, json schemas to verify if they are valid schemas.

Json deserialization support in java client

Is your feature request related to a problem? Please describe.
We want to add support to deserialize json data based on the provided schema

Describe the solution you'd like
As of now the client only has protobuf deserialization support, need to create generic interface contract for Deserializer which will be implemented for various data formats. On the lines of

1. KafkaJsonSchemaDeserializer.java
2. KafkaAvroDeserializer.java
3. KafkaProtobufDeserializer.java
   We need to do some poc on return type, whether to return actual class object or generic json object. But regardless of it, it should validate whether the given input data is compatible with the schema or no.

Additional context
We need this because we want to support different input data types in firehose, which currently only supports protobuf.

client: java: Expose refresh jitter as a config

As of now there is no jitter when the cache ttl config is set.
When using the default ttl config the jitter is of 30 mins.
This should be exposed as a config so that it can be set to reduce request spikes on the stencil server side.

replace protobuf specific rules to schema type independent rules

Problem
Existing backward compatibility rules are specific to protobuf schema. To support multiple schema types compatibility rules should be generalised.

Proposal
Move existing protobuf rules to following rules. More info on these rules can be found here

• BACKWARD
• FORWARD
• FULL
• FULL_TRANSTIVE

Add search command

Add a search command in stencil which allows searching for schema files.

Add custom help message

Add a custom help message for stencil commands.

POST /descriptors API gives 400 instead of 409

Ref: post_v1_namespaces{namespace}_descriptors

With both dryryn true & false, the API returns a 400 Bad Request while the documentation clearly states 409 Conflict.

Is this a bug or a feature?

Add support for JSON schema

Problem
Not able to upload/download JSON schemas in stencil

Proposed solution
Add support for json schemas which follows json-schema 2020-12 specification.

additional information

• verify if previous specifications are backward compatible with 2020-12 specification. Based on that take a call on supporting multiple specifications.
• $ref tag should be handled(if not in first cut, should be handled in later releases.)

Stencil Upload allows upload of filedescriptorset without imports for the first time.

Describe the bug
Stencil Upload allows upload of filedescriptorset for the first time

To Reproduce
Steps to reproduce the behavior:

1. Create a FileDescriptorFileSet without without --include_imports
2. Upload the same it succeeds for the first time.

Expected behavior
Should fail

Diff between 2 versions of message descriptor

Is your feature request related to a problem? Please describe.
I want to check the schema changes between the 2 versions. Manual ways to do this is to dig files from version control which can be automated on Stencil.

Describe the solution you'd like
Given name, namespace, earlier_version, later_version, and fullname Stencil CLI gives the diff.

Describe alternatives you've considered
Manual effort.

Additional context
No additional context

Move stencil to GRPC apis

For better compatibility and stability, we should move stencil APIs to GRPC with fallback support for REST using GRPC-gateway. All IDL definitions should be defined in proton, so that client generation can be much easier.

Add feature to merge uploaded descriptor with existing descriptor

The intention for this feature is to maintain backward compatibility by automatically deprecate removed fields, handle renamed fields, and handle the addition of new fields.

Not able to install stencil from source

Describe the bug
Not able to install stencil from the source when I execute the make build command to build the stencil binary file getting issues as mentioned below

generating ui build
make[1]: Entering directory '/home/hemanth/projects/go/stencil/ui'
make[1]: yarn: No such file or directory
make[1]: *** [Makefile:10: dep] Error 127
make[1]: Leaving directory '/home/hemanth/projects/go/stencil/ui'
make: *** [Makefile:35: ui] Error 2

To Reproduce
Steps to reproduce the behavior:

1. Clone the repository
2. Navigate to the project directory
3. Run the make build command

Expected behavior
The build should complete without any errors

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [Ubuntu 22.04]
• Go[1.18.7]

Add version command

Add a version command which returns the client and server version.

client: java: introduce checkstyle

Move to salt for logger package

Salt is the common logger for ODPF ecosystem products. The goal of this talk is to move to stencil to using log package from salt.

Stencil Fails to Download the latest protodescriptor with version number.

Describe the bug
Through Stencil Client shell, download of latest proto descriptor with version number fails, the db query construction is wrong for this scenario.

To Reproduce
Steps to reproduce the behavior:

1. Upload a new descriptor through cli which is promoted to latest
2. Try to fetch the same with version number & it fails.

Expected behavior
Should fetch.

Add download descriptor command

Stencil CLI should provide a stencil download command to download the descriptor from stencil server.

Guava Cache returns stale values during refresh

Stencil uses Guava cache for caching protobufs.
On a get() on a key that is expiring for a refresh, the CacheLoader Async reload from the LocalCache(guava) tries to reload the cache for key, however stale values are returned with the cache not updated.

The reload provides a callable future task which looks like a race condition to cache's parallel get calls. A simple solution is to return a new value synchronously.

add compatibility checks for protobuf oneof fields

Is your feature request related to a problem? Please describe.
There are no compatibility checks implemented for oneof fields.

Describe the solution you'd like
Don't allow removal of fields from oneOf
Don't allow adding existing fields to oneOf
Single existing field can be moved to new oneOf
.

[Clojure client] Unable to pass custom headers

Describe the bug
Unable to pass custom headers to stencil client in clojure. Clojure lib does accept custom headers field. But these headers are not getting used to create a stencil client.

ProtoUpdateListener not called for changes in nested protobuf descriptor

DescriptorCache's logic to check if the protobuf descriptor has changed doesn't work for changes in nested types.
The current check to compare old and updated protobuf descriptor only compares the root level fields and doesn't check nested types.

For example:
Old descriptor:

message TestNestedType {
    string sample_string = 1;
    TestType test_type = 2;
}

message TestType {
    string nested_string = 1;
    int nested_int = 2;
}

And new descriptor:

message TestNestedType {
    string sample_string = 1;
    TestType test_type = 2;
}

message TestType {
    string nested_string = 1;
    int nested_int = 2;
    float nested_float = 3;
}

Equality check between these descriptor returns false even when the nested field has changed.

Move to salt for config package

Migrate stencil config package to using salt config package.

Installing on Windows via Scoop is not working

Describe the bug

PS C:\Users\username> scoop install stencil
WARN  Purging previous failed installation of stencil.
ERROR 'stencil' isn't installed correctly.
Removing older version (0.5.0).
'stencil' was uninstalled.
Installing 'stencil' (0.5.0) [64bit] from stencil bucket
The remote server returned an error: (404) Not Found.
URL https://github.com/raystack/stencil/releases/download/v0.5.0/stencil_Windows_x86_64'.zip is not valid

To Reproduce
Steps to reproduce the behavior:

1. scoop bucket add stencil https://github.com/raystack/scoop-bucket.git

Checking repo... OK
The stencil bucket was added successfully.

2. scoop bucket list

Name    Source                                       Updated               Manifests
----    ------                                       -------               ---------
main    https://github.com/ScoopInstaller/Main.git   2023-12-01 7:32:57 PM      1265
stencil https://github.com/raystack/scoop-bucket.git 2023-11-22 2:10:13 AM         0

3. scoop install stencil

WARN  Purging previous failed installation of stencil.
ERROR 'stencil' isn't installed correctly.
Removing older version (0.5.0).
'stencil' was uninstalled.
Installing 'stencil' (0.5.0) [64bit] from stencil bucket
The remote server returned an error: (404) Not Found.
URL https://github.com/raystack/stencil/releases/download/v0.5.0/stencil_Windows_x86_64'.zip is not valid

Expected behavior
stencil should be installed and can be called in the terminal

Add upload descriptor command

Stencil CLI should provide a stencil push command to upload descriptor to stencil server.

[Docs] [golang client] Data parsing example

In golang client, consumers get dynamic message instance after parsing. Add an example in the docs on how to use dynamic message instance to get values of different types in consumer code.

List of protobuf types to cover

• timestamp
• duration
• bytes
• message type
• struct
• map