automate generation and addition of go code

WHAT

The process of adding go code to this repository should be automated.

WHY

Manual steps can be forgotten by devs
Non-go devs should not be obliged to understand the realm of go, especially since go is not a native programming language in our team / project (there are actually very few devs that know and code go)

core: BodyNodes re-implementation

See https://github.com/orgs/stroeer/teams/api/discussions/1

Dockerfile

we should put the sources of stroeer/protoc-dockerized and scripts to build it somewhere/in this repo

release proto files with packages

Node: create executable client

We need a quick way to perform load tests for the @grpc-js/grpc and grpc client of tapir. This is also some kind of documentation for other teams.

⚠️ Missing requirements for advertisement configuration

should ad container placements be configured channel agnostic (apps vs www)?

possible Blocker
requirements engineering necessary

GetArticlePageResponse vs. GetSectionPageResponse

Actual

message GetArticlePageResponse {
  stroeer.core.v1.Article article = 1;
  repeated RelatedArticle related_articles = 2;
}

message GetSectionPageResponse {
  stroeer.pages.section.v1.SectionPage section_page = 1;
}

Expected either

a)

message GetArticlePageResponse {
  stroeer.pages.article.v1.ArticlePage article_page = 1;
}

message GetSectionPageResponse {
  stroeer.pages.section.v1.SectionPage section_page = 1;
}

b)

message GetArticlePageResponse {
  stroeer.core.v1.Article article = 1;
  repeated RelatedArticle related_articles = 2;
}

message GetSectionPageResponse {
  repeated stroeer.core.v1.Stage stages = 1;
}

[Release] Also create a lib/zip/jar containing the proto sources only

As one example of a less standard JAVAish use case, AkkaGRPC has a quite good gRPC code generation. The recommended way is to reference a proto-files from a lib/jar.

Also new languages like Rust may benefit from this approach.

bundle Jackson De-/Serializer with java lib

Improve build/release process

make to build all the files locally, for all the supported languages
make release to create a release and git-tag (ideally auto-incremented by either minor/major/patch)
make test to locally run tests

stick to one build tool (e.g. no additional scripts such as build_docker.sh)

Remove 'web' APIs

The web APIs have been replaced by their pages counterparts. Changes of #50 need to be merged into the SectionPageService and then the web folder can be removed.

stages: references

We need a concept for links (<a href />) as part of a stage ("Mehr zu Sport").

Add documentation generator

At some point we need some documentation for some code conventions. Examples:

list of valid BodyNode.type strings
list of valid fields keys

Requirements

generate HTML documentation based on code comments
deploy & host to GitHub pages (latest published version)

Links

https://github.com/pseudomuto/protoc-gen-doc

Future

How do we deal with API changes? Versioned documentation?

data type of article ids

we currently use string as the data type for article ids (in core and web). If we keep on using numeric database ids, we should migrate this to something more efficient like uint64.

Document versioning

As soon as we approach a stable v1 of our proto files, we should document our approach to versioning the APIs and the generated packages for node and java:

semantic and guarantees in a vX package
semantic of the GitHub tags used for releasing packages and tagging generated go code

page: SectionPage with References for canonical link

What

Currently only the Article has a References with the canonical link. For SEO reasons we also need this on all section pages.

Proposal

message SectionPage {
   repeated Stage stages = 1;
+  References references = 2;
}

message ArticlePage {
   stroeer.core.v1.Article article = 1;
   repeated RelatedArticle related_articles = 2;
+  References references = 3; // because of same logic for `Pages` in FE
}

The only downside of this is that the Article and ArticlePage has the same References (duplication).

What

Bump all projects + docker image to protoc 3.15.0

See:
https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0

⚠️ define future 3rd party integration inside of content stages

possible blocker to get the new WebSectionPage into production

e.g. stage with two editorial teasers, and an Ad on 3rd position, but maybe on 1st position tomorrow
e.g. inhouse teaser placements
e.g. widgets rendered as teasers (Google Maps widget, finance widget, news from other publishers,....)

requirements engineering necessary

replace hardcoded with dynamic docker tag

https://github.com/stroeer/tapir/blob/master/.github/workflows/docker-release.yml#L24
currently each docker push will overwrite tag 3.13.0

Rule of thumb: Enums vs strings

enums for stable values (e.g. subtypes, maintypes,..)
strings for changing and temporary things (e.g. stage layouts)

Look Ahead: Package Naming Concept

What

Looking ahead for what will probably be added to tapir within the next months, I tried to create a concept for the future package structure

Why

Soon we will open our API to other teams, until then we should have a plan for the api and reach some kind of stability

Proposal

* page (ehem. pages)
    * article
        * ArticlePage
        * ArticlePageService
        * 
    * section
        * SectionPage
        * SectionPageService
            * GetSectionPage
                * page:1
    * theme
        * ThemePage
        * ThemePageService
    * author
        * AuthorPage
        * AuthorPageService
    * simple?
        * LandingPage?
        * 
* config {naming TBD - see proposals below}
    * curation
        * CuratedStage
        * CurationService
    * statistics
        * TrackingService
            * TrackingReport
            * GetArticleIdByReport
    * section
        * Section
        * SectionConfiguration
        * SectionConfigurationService
            * GetSectionConfig
            * ModifySectionConfig
            * AddSectionConfig
            * DeleteSectionConfig
    * personalization
        * PersonalizationService

* api?
    * auth
        * client
            * User
            * UserService
        * token
            * Token
            * TokenService
                * ValidateToken
                * GetToken
        * limits
            * RateLimit
            * ScopeLimit
            * 
    * 
* content (ehem. core)
    * article
        * Article
        * ArticleService
    * teaser
        * article
            * TeaserService
                * // content für paper um einzelne Bühnen nachzuladen
                * // z.B. Regionaler Content für Regionales Bühne
                * GetNewestArticlesForSection
        * advertisement
            * DisplayAd
                * Format
                * …
            * TelekomAdTeaser
        * page
            * LandingPageTeaser
            * TopicPageTeaser
            * AuthorPageTeaser
            * 
    * embed
        * Oembed
        * OembedService

candidates for the `config` pagackage name:

config
page.building / pagebuilding / page_building, 
page.management / pagemanagement / page_management, 
page.control / pagecontrol / page_control, 
site.control / sitecontrol / site_control
site.building / sitebuilding / site_building
 …

package naming specs proto3

As shown in the specs a package declaration follows the pattern:
package = "package" fullIdent ";"

i.e. the keyword package followed by a fullIdent followed by one ; semicolon, where fullIdent is defined as:
fullIdent = ident { "." ident }

i.e. at least one ident followed by zero or more other ident's preceded by ., where ident is defined as:
ident = letter { letter | decimalDigit | "_" }

i.e. at least one letter, followed by zero or more letters ([A-Z], [a-z]), digits ([0-9]), or the underscore _ character.

Motivation

I want to change the coremedia package name in my PR #83

coremedia is a system and should not be included in Service Definitions

RFC: breaking(api/web): naming inconsistencies

Here is list of all inconsistencies I found while migrate to 0.3.0. And some thoughts about the services: There is a Request object (GetArticlePageRequest) but no Response object (only WebArticlePage). This looks odd.

`web_article.proto`

- message RelatedArticle {
-   WebArticle article = 1;
-   RelatedArticleRole role = 2;
- }
+ message RelatedWebArticle {
# or main_article
+   WebArticle web_article = 1;
+   RelatedWebArticleRole role = 2;
+ }

`web_article_service.proto`

GetArticlePageRequest -> GetWebArticlePageRequest
rpc GetArticlePage -> rpc GetWebArticlePage

`web_section_service.proto`

GetSectionPageRequest -> GetWebSectionPageRequest
GetSectionPageResponse -> WebSectionPage

`article.proto`

message Element: the field content_id -> article_id (the message type is Article not Content

enum ContentType: CONTENT_TYPE_ARTICLE -> CONTENT_TYPE_NEWS_ARTICLE. There will be a OPINION_ARTICLE

node: bump to latest node LTS (14)

https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V14.md#14.15.1

conventional commits

Proposal

https://www.conventionalcommits.org/en/v1.0.0/

Example

feat(node): add esm bundle
feat(proto): add new model
ci: use ubuntu-latest

Why

https://www.conventionalcommits.org/en/v1.0.0/#why-use-conventional-commits

RFC: SectionPage with `ContentType` and `ContentSubType`

What

Currently, we only have:

ArticlePage.Article.type
ArticlePage.Article.sub_type

I would like to introduce the following addition (naming tbd):

SectionPage.Section.type = 'section'
SectionPage.Section.sub_type = 'news' | 'frontpage' | 'advertorial' | 'themepage'

In paper is already a type section. This makes it much easier for us to calculate standard info/properties/values in some places. We have an interface Page shared between Article- and Section-Pages.

More Examples

/test-homepage/ could also have the sub_type frontpage.
/themen/* with sub_type themen for easier cache header
/authors/ with sub_type author for custom commercial settings

[node] try to connect bufbuild/protobuf-es with @grpc/grpc-js

See: bufbuild/protobuf-es#252 (comment)

Pros:

no dependency on ts-protoc-gen (dead?), grpc-tools, google-protobuf

Element relations as enum

Currently only the relation field of Element is not an enum. When things about the article body are clear we should migrate here to an enum as well.

message Element {
  ElementType type = 1;
  string guid = 2;
  repeated string relations = 3;
  repeated Asset assets = 4;
  string article_id = 5;
}

Document release process

use one dockerized protoc for all languages and on CI

We currently use a dockerized protoc for generating java and a binary protoc installation for node (which is also used on GitHub actions for both languages).

We should use a dockerized version for all languages we want to support which also works on CI.

Asset.relations not needed?

Assets.relations seems not necessary.

Use Google.Protobuf.WellKnownTypes.Timestamp for dates?

Currently we save date fields in UTC strings, we could also use Google.Protobuf.WellKnownTypes.Timestamp to maybe skip all the necessary parsing of string to date in the clients.

clear import paths

absolut paths in imports currently don't work for example with BloomRPC and the grpc cli
in service
import "article.proto";
vs
import "article/v1/article.proto";

Cleanup `ContentSubType` enum

Some of our ContentSubType values are deprecated and can be removed (see internal ticket).

Remove Asset.index

we don't need this anymore

Deprecate node package: tapir-v1-legacy

On npm there is a deprecating setting for packages: https://docs.npmjs.com/deprecating-and-undeprecating-packages-or-package-versions. Currently, GitHub packages did not have these setting.

In case GitHub packages get it -> enable it for: https://github.com/stroeer/tapir/pkgs/npm/tapir-v1-legacy

Distinct types for articles and elements

Currently Article.type and Element.type share the same enum ContentType, after checking the requirements we might come up with two distinct enums here since not all elements might be an article.

`make test` without node/node-legacy

tapir/Makefile

Lines 79 to 85 in b03b09f

    
           .PHONY: test 
        
           test: generate go-mod ## Runs all tests 
        
           	@echo "+ $@" 
        
           	cd java && ./gradlew clean build 
        
           #	cd node && nvm use && npm run checks 
        
           #	cd node-legacy && nvm use && npm run checks 
        
           	cd go && go test -v .

The nvm use command did not work on my machine. The root cause is that we need to run both checks in a different node version locally. This would work with nvm

Metadata.publication_date?

This seems like a bug - publication_date is part of the fields map.

evaluate Buf Schema Registry (BSR)

The Buf Schema Registry (BSR) could be an interesting enhancement to our toolchain:

The BSR stores and manages Protobuf files as versioned modules, so that individuals and organizations can consume and publish their APIs easier than ever before.

The BSR comes with a browsable UI, dependency management, API validation and versioning, autogenerated documentation, and an extensible plugin system which enables remote code generation.

ContentType: Do we need IMAGE and AGENCY?

Is there a reason why both of them are main content types? In paper we ignore them completely.

tapir/stroeer/core/v1/article.proto

Lines 225 to 249 in ee2c8ae

    
           enum ContentType { 
        
             // Not specified. 
        
             CONTENT_TYPE_UNSPECIFIED = 0; 
        
             // A text article. 
        
             ARTICLE = 1; 
        
             // An image article (not available as standalone article). 
        
             IMAGE = 2; 
        
             // A video article. 
        
             VIDEO = 3; 
        
             // A gallery article. 
        
             GALLERY = 4; 
        
             // An embed article including an oembed or edge_side_include element 
        
             EMBED = 5; 
        
             // An author article (not available as standalone article). 
        
             AUTHOR = 6; 
        
             // An agency article (not available as standalone article). 
        
             AGENCY = 7; 
        
           }

Status and type of core.Article.Metadata.state

core.Article.Metadata.state is currently a string field which isn't used anywhere in the API (we delete articles which do not return a 200 from the CMS).

Questions:

shall we keep the field? We could (re-) add it later but unfortunately it's currently on position 1 (breaking change)
if we decide to keep it - shouldn't it be an enum (UNSPECIFIED, PUBLISHED, ...)?

bug: fundoc generation is broken

Running make fundoc fails with

------
 > [2/3] RUN cargo install fundoc --locked:
0.304     Updating crates.io index
291.8  Downloading crates ...
292.9   Downloaded fundoc v0.5.0
293.1   Installing fundoc v0.5.0
293.2 error: failed to select a version for the requirement `clap = "^4.1.8"`
293.2 candidate versions found which didn't match: 3.2.25, 3.2.24, 3.2.23, ...
293.2 location searched: crates.io index
293.2 required by package `fundoc v0.5.0`
------
Dockerfile:3
--------------------
   1 |     FROM rust:1.54.0
   2 |     
   3 | >>> RUN cargo install fundoc --locked
   4 |     WORKDIR /opt
   5 |     
--------------------

Add example of nested elements to documentation

Adding an example would help here:

A author element might have an image element child
A gallery element have image element children

Originally posted by @bobaaaaa in #111 (comment)

Technical timestamps in metadata

Many of the date fields (last_edited, last_published, created) vanished from the CMS lately. After discussing this internally, we should cleanup those fields.

In addition, we can and should decide on whether to enforce: https://linter.aip.dev/142/time-field-names

Originally posted by @thisismana in #111 (comment)

section path on GetWebSectionPageResponse

To generate breadcrumbs we need the section_path on the GetWebSectionPageResponse message.

Currently a WebArticle has a section_path field. Maybe it make more sense if both GetWebSectionPageResponse and GetWebArticlePageResponse includes a more structured section_path field.

Proposal

message GetWebSectionPageResponse {
  // ...
  repeated string section_path = ?
}

const sectionPath = ['', 'sport', 'fussball'];

	.PHONY: test
	test: generate go-mod ## Runs all tests
	@echo "+ $@"
	cd java && ./gradlew clean build
	# cd node && nvm use && npm run checks
	# cd node-legacy && nvm use && npm run checks
	cd go && go test -v .

	enum ContentType {
	// Not specified.
	CONTENT_TYPE_UNSPECIFIED = 0;

	// A text article.
	ARTICLE = 1;

	// An image article (not available as standalone article).
	IMAGE = 2;

	// A video article.
	VIDEO = 3;

	// A gallery article.
	GALLERY = 4;

	// An embed article including an oembed or edge_side_include element
	EMBED = 5;

	// An author article (not available as standalone article).
	AUTHOR = 6;

	// An agency article (not available as standalone article).
	AGENCY = 7;
	}

stroeer / tapir Goto Github PK

tapir's People

Contributors

Stargazers

Watchers

Forkers

tapir's Issues

WHAT

WHY

Actual

Expected either

a)

b)

Requirements

Links

Future

What

Proposal

What

What

Why

Proposal

candidates for the config pagackage name:

package naming specs proto3

Motivation

web_article.proto

web_article_service.proto

web_section_service.proto

article.proto

Proposal

Example

Why

What

More Examples

Proposal

Recommend Projects

Recommend Topics

Recommend Org

candidates for the `config` pagackage name:

`web_article.proto`

`web_article_service.proto`

`web_section_service.proto`

`article.proto`