otto-de/api-guidelines

A set of rules to build consistent and high quality REST and Async APIs at OTTO.

api-guidelines's Issues

[define] evaluate possibilities for SNS/SQS CloudEvents binding

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

SNS/SQS is a popular async protocol within OTTO.

With this ticket, it should be evaluated if an SNS/SQS protocol binding for CloudEvents can be specified.

Reference

• Closed (not implemented) issue on CloudEvents repo: cloudevents/spec#435

[new rule] producers must specify order guarantee in AsyncAPI spec

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

If the producer guarantees that messages are not sent out-of-order, consumers can more easily consume topics.

This ticket explores if this needs to be part of the contract between the producer and consumer (i.e. AsyncAPI spec).

[new rule] must use `partitionkey` context attribute (CloudEvents)

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

Many transport use partitioning as a way to perform scaling (see https://www.instaclustr.com/blog/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/).

As a CloudEvent may be passed between multiple intermediaries before being consumed, the intermediaries need to have a way to partition event streams to scale efficiently.

The partition extension provides a common way to provide a context attribute containing the partition key.

A typical partitionkey for events originating from one source is the source-id. Depending on the business case, the partitionkey may also be something else.

[new rule] add a rule(set) to clearly specify how data change events should be used

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

Data change events usually do not contain semantical information of why a change happened. This is why the usage of data change events should be limited. Deducting business decisions from data change events implicitly requires domain knowledge of a foreign domain, thus reducing encapsulation.

A rule (or a set of rules) should be created to clearly state how data change events should be used.

Possible rules:

• SHOULD use data change events for data replication
• SHOULD be used for updating otherwise read-only data sources. The only purpose of these read-only data sources is for consumption (e.g. by UIs, report generation, statistics).
• MUST NOT be used to implement policies (automation rules) that issue business side effects (commands) to any API.
• MUST NOT deduct intention from data change events

[update] relax "MUST change type context attribute to indicate incompatible changes" to allow deprecation cycle on property basis

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

Currently, API providers are enforced to introduce a new event type whenever incompatible changes are made.

The following alternative approach should be allowed too:

1. Add a new property
2. Communicate the planned deprecation with all parties and settle on a sunset date
3. Deprecating the old property with the specific sunset date
4. Removing the old property after sunset date, making sure that no consumer uses the old property beforehand.

The advantage of this approach is that no additional versions need to be maintained. The disadvantage is that one can not use the Galapagos subscription model to verify that no consumer uses an old schema version before sunsetting it.

[orga] Provide script for "next free rule number"

Context

Each guideline rule has an ID that is displayed in the UI:

These IDs are used in the URLs and to clearly reference rules.

When creating a new rule, contributors must add a rule ID at the very beginning of their file:

Currently, they do not know which ID to put / which ID is up next / is already used.
We would like to motivate people to contribute to our guidelines.
Thus, we should keep the effort to suggest a new rule as low as possible.

Goal

We should at least provide some minimal guidance.

•  Provide a script users can manually run to get the next free rule ID.
•  Add the script to the Contributing file.

[update] definition of domain event

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

Notes from discussion of 2022/07/08:

• domain events can contain the whole data structure. They are still domain events because they have been triggered by an occurrence in the domain that is contained within the domain event.
• Add a decision table / example table when to use what (e.g., log compaction)
• Consumers of data events should not care about the order of events

[new rule] create rule for formatting prices

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

We currently have no rule defining the format of a money amount in an API.

As money amounts are very common in our APIs (checkout, order-positions, transactional-communication), we would like to introduce a rule that ensures that money amounts are always formatted the same.

Currently, a money amount is formatted as eurocents (integer) in the APIs checkout, order-positions, and transactional-communication.

[new rule] must provide compatibility mode for produced messages

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

Messages may evolve over time. In order to signal a consumer what a compatible change is, the compatibility mode must be documented for produced messages.

The following modes exist:

Mode	Changes allowed	Compatible against	Description
FORWARD	- Add fields - Delete optional fields	previous version	Older clients can read events written with the next newer schema.
FORWARD_TRANSITIVE	see FORWARD	all previous versions	Older clients can read events written with all newer schemas.
FULL	- Add optional fields - Delete optional fields	previous version	Older clients can read events written with the next newer schema. Newer clients can read events written with the previous schema version.
FULL_TRANSITIVE	see FULL	all previous version	Older clients can read events written with a newer schema. Newer client can read events written with all previous schema versions.

Other common modes such as backward and none should not be allowed, as they break existing clients.

Analyze if exemptions can be made to this rule if all consumers are prepared for a breaking change.

Reference

• asyncapi/spec#727
• https://docs.confluent.io/platform/current/schema-registry/avro.html#

[update] add example for MAY provide sequence

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

A specific example may help to understand how a sequence can be used. Coming from the DeepSea Common Event Format, the association between sequence and source is often not directly understood.

[new rule] guidelines for avro format

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

Multiple teams expressed the desire to use something other than JSON to encode messages on Kafka. While the Event-Guidelines does not restrict the usage of another format, it also provides no guidance how to properly use another format.

As most of the kafka ecosystem is working with avro, we will focus on the evaluation of guidelines for the avro format.

Tasks

• Check the existing guidelines for JSON if they also apply to the avro format.
• Check if additional rules make sense
• Update the guidelines accordingly to include rules for Avro

Reference

• https://avro.apache.org/

[new rule] write domain events originating from the same entity type to the same kafka topic

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

The rule https://api.otto.de/portal/guidelines/r200017 implicitly states that domain events from the same entity type should be written to the same topic. This should be made explicit in a new rule.

[update] remove rule MUST NOT validate audience of the JSON Web Token

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

The rule describes how the "aud" claim is used within api.otto.de. The usage and validation of the "aud" claim can make sense for other OAuth2 processes.

Instead of having a rule, we should improve api.otto.de's documentation of the OAuth2 server, as this rule is tightly bound to how api.otto.de's OAuth2 server works.

[define] evaluate proposed w3c standard "Content Negotiation by Profile"

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

W3C proposed a standard to use profiles in content negotiation: https://w3c.github.io/dx-connegp/connegp/

It differs from the way we use profiles. Instead of using the profile as a media type parameter, it is used as a separate header Accept-Profile and Content-Profile in content negotiation.

Depending on the outcome of w3c/dx-connegp#41, we should evaluate if it makes sense to adapt to the w3c proposal.

[update] rework ETag usage

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

Follow up of #1193:

Goal is to describe/clarify the usage of ETags for:

cache management

• good match for strong/weak etags
• Care should be taken, which headers, embedding, etc are part of the ETag calculation

concurrency control

• might not be suitable, due to strong ETags being constraint to the entity's representation and not to it's logical state
• Problems with intermediate proxies, that convert strong to weak ETags
• Introduction of a separate token as an alternative

Reference

Details can be found in the discussion of #1193.

[define] usage of CloudEvents' `dataschema` conflicts with the Confluent wire format

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

CloudEvents' dataschema should contain a reference to the schema that has been used to write the cloud event. If the Confluent schema registry and it's SerDes is used, this information is encoded using the Confluent wire format directly in the key/value of the record.

Currently, it is not possible to write the schema ID into a custom record header when using the Confluent Schema Registry SerDes. An open issue exists: confluentinc/schema-registry#881

Other SerDe Libraries support writing to a custom header: https://www.apicur.io/registry/docs/apicurio-registry/2.2.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-config-props_registry

Therefore we need to define how this should be handled.

[update] use link shortener

To do

Use link shortener for links leading to internal pages.

[new rule] compacted topics must always be ordered

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

If a compacted topic is used, it must always be ordered so that the last written data event is the most current for a used key. If a situation might occur in which the last written data event is not the most current for the specified key, the compaction process will delete the most current event.

[define] support for commands

(i) This issue has been manually transferred from a former internal repository, as a private repository issue cannot be transferred to a public repository.

Context

This ticket contains the use cases and requirements for Commands in the context of asynchronous API.

[orga] add issue template

To do

• add issue template to the project for predictable and easy-to-handle structure.
• consider security/data protection hints