triggermesh/triggermesh

TriggerMesh is the open-source AWS EventBridge alternative. It provides a unified eventing experience, a developer-friendly CLI, and runs on Docker or natively on Kubernetes.

The TriggerMesh Cloud Native Integration Platform consists of a set of APIs which allows you to build event-driven applications. Implemented as a set of Kubernetes CRDs and a Kubernetes controller, it gives you a way to declaratively define your event sources and event targets, in addition to potential actions needed in your applications: content-based event filtering, event splitting, event transformation and event processing via functions.

Getting Started

• Guides
• Documentation

Installation

To install TriggerMesh, follow the installation instructions.

TL;DR

Register TriggerMesh APIs by deploying the Custom Resources Definitions:

kubectl apply -f https://github.com/triggermesh/triggermesh/releases/latest/download/triggermesh-crds.yaml

Deploy the platform:

kubectl apply -f https://github.com/triggermesh/triggermesh/releases/latest/download/triggermesh.yaml

Namespaced installation

By default, TriggerMesh works with resources across all namespaces in the Kubernetes cluster. This requires permissions provided by ClusterRoles and ClusterRoleBindings. If cluster-wide permissions are not desired, TriggerMesh can be configured to run in a single namespace. To get the namespace-scoped installation, first, deploy the CRDs:

kubectl apply -f https://github.com/triggermesh/triggermesh/releases/latest/download/triggermesh-crds.yaml

And then, using kubectl's "kustomization" feature, install customized manifests:

kubectl apply -k config/kustomization/namespaced/

Controllers created with this command won't reconcile resources outside the triggermesh namespace, ClusterRoleBindings will have the namespace scope.

Contributing

Please refer to our guidelines for contributors.

Commercial Support

TriggerMesh Inc. offers commercial support for the TriggerMesh platform. Email us at [email protected] to get more details.

License

This software is licensed under the Apache License, Version 2.0.

Additionally, the End User License Agreement included in the EULA.pdf file applies to compiled executables and container images released by TriggerMesh Inc.

triggermesh's Issues

Implement common webhook verifications

HTTPSource should implement common webhook verifications by sources.
The problem is that they might differ slightly on the process or where the signed payload is informed, but we can come up with some abstraction that fits most implementations

Shopify: https://shopify.dev/tutorials/manage-webhooks#verify-webhook
Stripe: https://stripe.com/docs/webhooks/signatures
Slack: https://api.slack.com/authentication/verifying-requests-from-slack
Github: https://developer.github.com/webhooks/securing/

Add Functions

https://github.com/triggermesh/function

Handle more event types than just the creation of new tickets

Right now, the Zendesk source only handles the creation of new tickets (see source code).
We would like to allow users to subscribe to different types of updates (TBD: which ones?)

Acceptance criteria:

1. We need to allows users to select multiple event types
   This could be materialized as a list of strings in the spec.

2. We need a way to differentiate event types from the payload.
   I can think of 2 ways to do this:

   • either expose multiple HTTP endpoints (/ticket.created, /ticket.updated, ...) and send requests from different triggers to different endpoints.
   • or include the type in the Trigger's payload as a type attribute (similar to the way CloudEvents does it).

3. We need the controller to auto-register event types with the Knative event registry.
   The cloudAttributes of each instance of the source need to be populated with the event types it emits.

4. We need to read event types from a single location in zendesk/pkg/apis.
   Right now the adapter declares its own constant for event types, while another set of constants is declared inside zendesk/pkg/apis. There should be a single source of truth.

Update AWS Go SDK to v2

The SDK v2 is available and brings really nice improvements: https://aws.amazon.com/blogs/developer/aws-sdk-for-go-version-2-general-availability/

Cut release v1.10.1

GoogleSheetTargetStatus uses the wrong condition set

currently it uses AwsCondSet

Target CE Responses are Documented/Implemented wrong

The majority of Targets either respond with the wrong event type (per crd documentation) or are documented to return an event type and return nothing at all. I think its time that we move to implement the Replier on all Targets and get them all responding with properly documented responses.

Movement of upstream constants from knative.dev/pkg to knative.dev/eventing

In attempt to clean up serving & eventing specific metrics we've moved some of the packages to their respective repos.

I notice you import some of the metricskey so when you adopt Knative 0.25 you'll need to change the import path

Related issue: knative/pkg#608

The eventing specific metrickeys were moved to there:
https://github.com/knative/eventing/blob/main/pkg/metrics/metrics.go

And the sources stats reporter has moved from pkg to:
https://github.com/knative/eventing/blob/main/pkg/metrics/source/stats_reporter.go

Should UIPath be Removed?

Support Basic auth

ref. https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html

Allow fan-out over SNS topic

Problem

Currently, there is no possibility for multiple subscribers to create a notification configuration in the same bucket. AWS rejects the request with the error Configurations on the same bucket cannot share a common event type..

Even if we enforced a mechanism such as allowing multiple subscribers only if they define non-overlapping event types, we would be limited by the feature of the SQS queue, which doesn't allow fanning messages out to multiple consumers.

Proposal

We could enable an alternative mode of notification delivery where a user could select a SNS topic (which must exist) instead of a SQS queue, and the source would then create and subscribe one SQS queue to that topic, for each AWSS3Source instance which uses that mode.

Unlike the current SQS-based mode of delivery, external subscribers would now be able to consume bucket notifications as well (fan out).

Possible API change

edit: API change implemented in #269

apiVersion: sources.triggermesh.io/v1alpha1
kind: AWSS3Source

# ...

spec:

  # ...

  destination:  # Optional. Default to 'sqs' with managed queue.

    # Only *one* possible choice per AWSS3Source instance:

    sqs:
      queueARN: arn:aws:sqs:us-west-2:123456789012:triggermeshtest  # Optional. Managed by the source if not provided.

    sns:
      topicARN: arn:aws:sns:us-west-2:123456789012:triggermeshtest

External links

• SNS - Fanout to Amazon SQS queues

Collect logs after failed e2e tests

The kind of failure below happens occasionally in e2e tests but, because all receive-adapter Pods are ephemeral, we don't have a chance to inspect their logs and see if some error occurred and caused the failure.

STEP: creating test REST clients
STEP: creating test namespace with base name github-eventdisplay
STEP: creating a GitHub repository
STEP: creating a Kubernetes Secret containing the GitHub API token
STEP: creating a Bridge object
STEP: checking whether the initial ping event was received
STEP: creating a Git commit
STEP: waiting for an event to be received
STEP: marking test namespace github-eventdisplay-4238 for deletion
STEP: deleting GitHub repository triggermesh/e2e-github-eventdisplay-4238


• Failure [39.282 seconds]
GitHub to Event-Display
/home/circleci/project/test/e2e/bridges/githubeventdisplay/main.go:59
  receives events generated by the source [It]
  /home/circleci/project/test/e2e/bridges/githubeventdisplay/main.go:111

  Timed out after 10.000s.
  A push event should have been received
  Expected
      <[]event.Event | len:0, cap:0>: nil
  to have length 1

  /home/circleci/project/test/e2e/bridges/githubeventdisplay/main.go:152

Dangling SNS subscriptions left behind

e2e tests are leaving a bunch of subscriptions behind

I suspect this happens because the topic is removed (in the test's AfterEach) before the event source (due to namespace termination), so the finalizer doesn't get a chance to match subscriptions to a topic.

Adjust processor:receiver ratio to avoid filling the processing queue under load

As shown in the dashboard below, the processing queue is always full under load because the event processors can't cope with the number of concurrent message receivers. This comes from the current ratio of 1:1 which isn't enough because processors process events one by one, while receivers receive them in batches of 10.

The buffer's depth is expected to remain low under normal circumstances (see the deletion queue's buffer below as a comparison), and only fill if the network latency between the adapter and the event sink become abnormally high, to apply some backpressure on the message receivers.

Migrate container release pipeline to Artifact Registry

Google is slowing down their commitment to Container Registry and recommends migrating to Artifact Registry:

A migration guide is available at https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr.

Use a `Tracker` instead of a filtered `EventHandler` to track changes to multi-tenant source adapters

We currently track changes to multi-tenant receive adapters by attaching an event handler with a custom filter (hasAdapterLabelsForType()) to Deployment/KnService informers.
The filtering occurs on a set of common labels such as

app.kubernetes.io/name: zendesksource
app.kubernetes.io/component: adapter
app.kubernetes.io/part-of: triggermesh
app.kubernetes.io/managed-by: triggermesh-controller

Code location, for reference:

triggermesh/pkg/reconciler/base.go

Lines 170 to 173 in d208c4b

	serviceinformerv1.Get(ctx).Informer().AddEventHandler(cache.FilteringResourceEventHandler{
	FilterFunc: hasAdapterLabelsForType(typ),
	Handler: controller.HandleAll(adapterHandlerFn),
	})

While this approach works well for us, there has been some effort upstream, especially since Knative v0.26, to use a Tracker for that purpose. Here is an example in the upstream event source for GitHub:

https://github.com/knative-sandbox/eventing-github/blob/ee07cd159049aff7e0a2d7eb062e67fee5d45f57/pkg/reconciler/source/controller.go#L78-L85

We should consider evaluating whether using a Tracker may indeed simplify the filtering of multi-tenant receive adapters for us.

The conclusion may be that a Tracker can only filter by name and not by labels, in which case we may simply close this issue as "won't do".

Consider dropping requirement for region and account ID in bucket ARN

Users are currently expected to pass a non-standard bucket ARN to be able to use the S3 source.

To quote our documentation (ref. triggermesh/docs#121):

The ARN displayed in the AWS Console, which has the format arn:aws:s3:::{bucket_name}, lacks some essential
information: the AWS region and the account ID. Both must be included in the ARN by using the more complete
format below:

arn:aws:s3:{aws_region}:{aws_account_id}:{bucket_name}

Without the region and account ID, this event source would be unable to set an accurate identity-based access policy
on the SQS queue described in the SQS Queue section of this document.

While the last sentence holds true (partially, read on), some possible fallback exists:

• The account ID can be retrieved via an API call: GetCallerIdentity.
• The bucket's region is actually irrelevant for the policy, and is only used for creating the SQS queue. We could default to the bucket's region, which can as well be retrieved via an API call: GetBucketLocation.

Remove usage of ObserveFinalizeKind

This interface was removed.
Here is some context: knative/pkg#2021

Create source for Azure Cosmos DB

Acceptance criteria:

1. Enables a change feed on a Cosmos DB container (if needed).
2. Consumes changes from the change feed using the pull model.

Make Webhook source multi-tenant

Checklist, based on the example of the AWS SNS source.

Generic changes:

• Implement the MultiTenant micro-interface (example)
  The generic reconciler asserts it to take a few reconciliation decisions.
• Initialize the Knative Service reconciler in multi-tenant mode (example)
  This ensures all sources are being rechecked whenever the receive adapter changes, instead of just the owner since there is no more single owner.
• Remove all instance-specific env vars from the adapter builder (example)
  There is a helper for it, only the kind-specific env vars need to be injected: image, and observability related configs.
• Add a controller constructor to the adapter package (example)
  This should be completely generic, but the event handler needs to be added to the correct informer.
• Call the multi-tenant shared main (example)
  It sets the namespace scope, enables HA, and injects the controller constructor before calling the "regular" main.

Source-specific:

• Define the adapter contract available to the reconciler via an interface (example)
  In the case of the HTTP source, the contract might actually be identical to the SNS source. Propagating a status condition is optional.
• Implement reconcile/observe/finalize using the methods defined above (example)
  Reconcile ensures a HTTP handler is registered for the source. Finalize removes that handler when a source is deleted.
• Perform the required changes to the adapter type (example)
  For example, if the source exposes a HTTP endpoint, its adapter should at the very least embed a router.Router.

Duplicate subscriptions for the same source instance

We often notice in our end-to-end tests that more than one subscription are created for a single instance of a AWSSNSSource. Not only this leads to dangling subscriptions, but it also causes multiple message deliveries.

It appears that attempts to subscribe fail for a little while because the receive adapter isn't yet ready to accept HTTP requests, then, after a few minutes (!), the error Subscription is pending confirmation, will retry is reported multiple times, before the subscription eventually succeeds:

  Normal   CreateAdapter    3m43s                  awssnssource-controller  Created adapter Service "awssnssource-adapter"
  Warning  FailedSubscribe  3m35s (x3 over 3m36s)  awssnssource-controller  Error subscribing endpoint "http://awssnssource-adapter.e2e-awssnssource-1787.k.triggermesh.io/e2e-awssnssource-1787/test-nsbrz" to SNS topic "arn:aws:sns:eu-central-1:123456789012:e2e-awssnssource-1787": InvalidParameter: Invalid parameter: Unreachable Endpoint
  Warning  FailedSubscribe  14s (x5 over 16s)      awssnssource-controller  Subscription is pending confirmation, will retry
  Normal   Subscribed       13s                    awssnssource-controller  Subscribed to SNS topic "arn:aws:sns:eu-central-1:123456789012:e2e-awssnssource-1787"

During these rapid retries, multiple subscriptions are occasionally created:

$ aws sns list-subscriptions | jq -r '.Subscriptions[] | select(.SubscriptionArn | test("e2e-awssnssource")) | .SubscriptionArn'
arn:aws:sns:eu-central-1:123456789012:e2e-awssnssource-1787:9aeab061-932c-49c9-8ef0-9d1746240a1a
arn:aws:sns:eu-central-1:123456789012:e2e-awssnssource-1787:c0ba1d42-b2d3-434b-ac47-d60f05b99016

Of course, the source reports only one subscription in its status:

subscriptionARN: arn:aws:sns:eu-central-1:123456789012:e2e-awssnssource-1787:c0ba1d42-b2d3-434b-ac47-d60f05b99016

Add Azure and Google Cloud sources

triggermesh/event-sources

First user creation doesn't generate an event

The logic below is incorrect, causing the issue described in the title:

https://github.com/triggermesh/aws-event-sources/blob/27cce63bb0c09a5a0eb377718d2f3af12f6becf7/pkg/adapter/awscognitouserpoolsource/adapter.go#L148-L150

Fix failing tests for source reconcilers

As of revision c2a5502, tests are currently failing for the following packages:

• pkg/sources/reconciler/azureblobstoragesource
• pkg/sources/reconciler/azureeventgridsource
• pkg/sources/reconciler/googlecloudauditlogssource
• pkg/sources/reconciler/googlecloudpubsubsource
• pkg/sources/reconciler/googlecloudstoragesource

Add CI/CD integration

Build and publish releases using CircleCI infra

Leverage Azure SDK's detection of Event Hubs instance and token provider

The Azure SDK is capable of detecting the correct Event Hubs instance and token provider (auth method: SAS or AAD) based on the presence of some specific environment variables:

https://pkg.go.dev/github.com/Azure/azure-event-hubs-go/v3#NewHubFromEnvironment

We should leverage this mechanism. It would allow us to remove our own logic here:

triggermesh/pkg/sources/adapter/azureeventhubsource/adapter.go

Lines 141 to 161 in b4b1d5d

	func newHubWithConnStr(env *envConfig) (*eventhub.Hub, error) {
	return eventhub.NewHubFromConnectionString(env.ConnStr)
	}
	func newHubWithKey(env *envConfig) (*eventhub.Hub, error) {
	provider, err := sas.NewTokenProvider(sas.TokenProviderWithKey(env.HubKeyName, env.HubKeyValue))
	if err != nil {
	return nil, err
	}
	return eventhub.NewHub(env.HubNamespace, env.HubName, provider)
	}
	func newHubWithSP(env *envConfig) (*eventhub.Hub, error) {
	provider, err := aad.NewJWTProvider(aad.JWTProviderWithEnvironmentVars())
	if err != nil {
	return nil, err
	}
	// get an existing hub for dataplane use
	return eventhub.NewHub(env.HubNamespace, env.HubName, provider)
	}

Add guide for contributors

We need a document (CONTRIBUTE.md/DEVELOPMENT.md/insert name) which lists what contributors need to know so they can start hacking on this codebase, and what they should be paying attention to.

Non exhaustive list of things it should contain:

• Technical pre-requisites
  • Go toolchain (+ recommended version)
  • ko
  • a Kubernetes cluster with Knative (+ minimum versions)
  • ...
• How to run things locally
  • ko publish individual adapters
  • go run the controller with the correct environment variables set
  • how to go run a single component in isolation (same as controller, what env vars, etc.)
  • ...
• Use the .local/ directory for custom manifests
  • avoids pushing secrets accidentally
  • avoids destroying config/ accidentally
• How to add a new integration
  • pkg/apis types
  • codegen
  • keep alphabetical order while enumerating components
• Testing
• ...

Add Routing

https://github.com/triggermesh/routing

Allow persistence of event offset for AzureEventHubs

When the adapter restarts, some intermediate events can be missed because the offset is kept in memory.

We need a way to store the offset in a storage account instead.

The Hub struct can be customized to use an persist.CheckpointPersister. By default, a Hub uses an in-memory
CheckpointPersister, but accepts anything that implements the persist.CheckpointPersister interface.

-- https://github.com/Azure/azure-event-hubs-go

Standardize naming scheme of programs cmd and config manifests

The naming convention of their prior homes were ported over to initially get the code in place first. This task ensures that it is addressed.

Replace insecure library: CVE-2020-26160

We have a direct dependency and many indirect ones to https://github.com/dgrijalva/jwt-go
All of them should be replaced with non vulnerable libraries.

Add support for CORS headers

I received a request for the ability have the HTTP Source send CORS headers during preflight requests where the invoker may require the header for verification.

Convert deprecated Zendesk HTTP target to Zendesk Webhook

In February 2022, HTTP targets will be deprecated and in favour of webhooks: https://support.zendesk.com/hc/en-us/articles/4404321217690

Create a source for AWS CloudTrail

https://aws.amazon.com/cloudtrail/

Add Knative Targets

Support alternative auth methods such as Kube2IAM and IAM Roles for Service Accounts

I would like to use the credentials built into the POD via Kube2IAM instead of cred file

Add Bumblebee

https://github.com/triggermesh/bumblebee

Remove empty `ClusterRoles` used for event sources

Context

Currently, each event source kind gets a dedicated ServiceAccount managed by their controller (e.g. foosource-adapter is used by all instances of FooSource).

Controllers try to bind these ServiceAccounts to a ClusterRole of the same name using a RoleBinding:

triggermesh/pkg/sources/reconciler/common/reconcile.go

Lines 423 to 425 in b4b1d5d

	if _, err = r.syncAdapterRoleBinding(ctx, currentRB, desiredRB); err != nil {
	return nil, fmt.Errorf("synchronizing adapter RoleBinding: %w", err)
	}

Problem

Not all source adapters require permissions to communicate with Kubernetes. Yet, a missing ClusterRole would cause a binding failure:

Events:
  Type     Reason            Age    From                  Message
  ----     ------            ----   ----                  -------
  Warning  FailedRBACCreate  2m20s  foosource-controller  Failed to create adapter RoleBinding "foosource-adapter": clusterroles.rbac.authorization.k8s.io "foosource-adapter" not found

Because of that design, our release manifests include a bunch of ClusterRoles, most of which with an empty set of rules, just to accommodate the minority of sources which do require access to Kubernetes (only 2).

Proposed solution

Either add a new interface, or simply reuse the existing multiTenant interface, to determine whether or not the controller should bind the ServiceAccount to a ClusterRole.

Replier: improve response error types

The replier object helps composing replies at targets.

Functions Ok and Error at the replier are used on success and error scenarios:

• Ok will set a new type to the response and also add a category: success extension.
• Error will set a new type to the response and also add a category: error extension. The response will be compliant with this schema.
• By default a new ID is generated for the response CloudEvent.

Problems:

• category is a weak and not meaningful attribute.
• It is expected by registries and users that a type would match a schema. Letting people choose the Error type but forcing the schema to be the one linked above does not sound like a good idea.
• After emitted there is no way to link the response to the incoming event. If two same typed events are received at the target at the same time, it would be hard to distinguish which response belongs to which request.

Proposal:

• category extension should be gone. A successful processing will return a explicitly typed response, and an error will return an io.triggermesh.error.
• by default all responses enable ResponseWithProcessedHeaders which will add a reference to the incoming type, source and ID to the response:
  

  triggermesh/pkg/targets/adapter/cloudevents/response_options.go

  Line 80 in d60ec28

  func ResponseWithProcessedHeaders() EventResponseOption {

Example:

incoming event:
 - type: me.icecream.request
 - source: dani.doe
 - id: 12345
- data: {...}

on success response event:
- type: io.benjerry.icecream
- source: benjerry.target
- id: abcde
- processedtype: me.icecream.request
- processedsource: dani.doe
- processedid: 12345
- data: {"flavor": "shrimp","size":"s"}

on failure response event:
- type: io.triggermesh.error
- source: benjerry.target
- id: 1a2b3c
- processedtype: me.icecream.request
- processedsource: dani.doe
- processedid: 12345
- data: {"code": "auth","description":"credentials are not valid"}

This should cover most response filtering scenarios I can think of.

• capture all successful responses from target
• capture all error responses from target
• capture all error responses from any target
• capture the response to this request event
• capture the response to this request event if the outcome is successful.
• capture the response to this request event if the outcome is an error.
  ...

Not included, since for the time being Knative filters do not support not:

• capture all successful responses from any target (but this one can be tackled using TriggerMesh's Filter).

Address linter warnings

$ golangci-lint run ./pkg/...

pkg/sources/secret/secret.go:63:49: S1019: should use make(map[string]*corev1.Secret) instead (gosimple)
        secretCache := make(map[string]*corev1.Secret, 0)
                                                       ^
pkg/transformation/pipeline/handler.go:62:19: SA1019: cloudevents.NewDefaultClient is deprecated: Please use NewClientHTTP with the observability options. (staticcheck)
        ceClient, err := cloudevents.NewDefaultClient()
                         ^
pkg/routing/adapter/splitter/adapter.go:168:16: Error return value of `newCE.SetData` is not checked (errcheck)
                newCE.SetData(cloudevents.ApplicationJSON, v.Raw)
                             ^

A failure occuring in a cleanup task prevents to execution of subsequent cleanup tasks

In the following AfterEach, if the call to srcClient.Delete fails, the call to e2esns.DeleteTopic is never executed, which leaves dangling resources after the execution of the test suite (in this case, a SNS topic).

AfterEach(func() {

	By("deleting the AWSSNSSource object", func() {
		err := srcClient.Delete(ctx, src.GetName(), metav1.DeleteOptions{})
		Expect(err).ToNot(HaveOccurred())
	})

	By("deleting SNS topic "+topicARN, func() {
		e2esns.DeleteTopic(snsClient, topicARN)
	})
})

All AWS resources are tagged with e2e_instance: <test name> which allows us to run periodic garbage collections, but it would also be nice if we could gracefully handle errors inside AfterEach blocks instead of failing right away.

Create a source for Azure Service Bus Topics

The design looks very close to Google Pub/Sub, therefore the source should:

• Take the resource ID of a topic to subscribe to.
• Reconcile a subscription for pulling messages from that topic.

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-overview

Controller panics when `TRANSFORMER_IMAGE` environment variable is not set

The following panic occurs when the TRANSFORMER_IMAGE environment variable is not set:

panic: unable to process Transformation required environment variables (missing TRANSFORMER_IMAGE)

goroutine 1 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000234240, {0x0, 0x0, 0x0})
        go.uber.org/[email protected]/zapcore/entry.go:232 +0x446
go.uber.org/zap.(*SugaredLogger).log(0xc000362760, 0x4, {0x0, 0x2}, {0xc000ab5848, 0x3, 0x1a}, {0x0, 0x0, 0x0})
        go.uber.org/[email protected]/sugar.go:227 +0xee
go.uber.org/zap.(*SugaredLogger).Panic(...)
        go.uber.org/[email protected]/sugar.go:123
github.com/triggermesh/triggermesh/pkg/transformation/reconciler/controller.NewController({0x2ab7e08, 0xc000a5eea0}, {0xc000a55ea0, 0xc00070c840})
        github.com/triggermesh/triggermesh/pkg/transformation/reconciler/controller/controller.go:61 +0x26d
knative.dev/pkg/injection/sharedmain.ControllersAndWebhooksFromCtors({0x2ab7e08, 0xc000a5eea0}, 0x1, {0xc000ab5d90, 0x3c, 0x77359400})
        knative.dev/[email protected]/injection/sharedmain/main.go:350 +0x15c
knative.dev/pkg/injection/sharedmain.MainWithConfig({0x2ab8ce8, 0xc00000e030}, {0x26ccba1, 0x16}, 0xc00057c6c0, {0xc000aa1d90, 0x3c, 0x3c})
        knative.dev/[email protected]/injection/sharedmain/main.go:192 +0x5da
knative.dev/pkg/injection/sharedmain.MainWithContext({0x2ab8ce8, 0xc00000e030}, {0x26ccba1, 0x16}, {0xc00089fd90, 0x3c, 0x3c})
        knative.dev/[email protected]/injection/sharedmain/main.go:131 +0xe7
knative.dev/pkg/injection/sharedmain.Main({0x26ccba1, 0x16}, {0xc00089fd90, 0x3c, 0x3c})
        knative.dev/[email protected]/injection/sharedmain/main.go:105 +0x8f
main.main()
        github.com/triggermesh/triggermesh/cmd/triggermesh-controller/main.go:80 +0x3d4

We should instead default to something, like the public "latest" image name, because most ko:// entries may be commented during development.

The culprit:

triggermesh/pkg/transformation/reconciler/controller/controller.go

Lines 56 to 62 in b4b1d5d

	env := &envConfig{}
	if err := envconfig.Process("", env); err != nil {
	logger.Panicf("unable to process Transformation required environment variables: %v", err)
	}
	if env.Image == "" {
	logger.Panic("unable to process Transformation required environment variables (missing TRANSFORMER_IMAGE)")
	}

Spawn and terminate receivers dynamically

We should spawn and terminate receivers dynamically based on the current amount of messages being processed (internal queue lengths should be a good indicator of this) to optimize costs generated by ReceiveMessage API requests (https://aws.amazon.com/sqs/pricing/).

Migrate the common code used in the AWS Targets to a shared directory

There is a lot of re-used code in the split-up AWS Targets. We should move this code to a shared common folder.

Split out AWS Target Components

Various AWS Targets were originally created using a single target adapter to leverage common code specific to AWS. This should be split up where there is a 1:1 mapping between adapter and their CRD.

Add makefile support

Create a common makefile based on the previous sources/targets.

Debug why bringing up a target results in a `not implemented` message

I'm able to build and deploy the source and target controller and adapters using ko, however attempting to run the adapter results in the following log messages:

            {"severity":"EMERGENCY","timestamp":"2021-09-22T21:09:23.358612296Z","logger":"slack-target-adapter","caller":"v2/main.go:205","message":"Start returned an error","error":"not implemented","stacktrace":"knative.dev/eventing/pkg/adapter/v2.MainWithInformers\n\tknative.dev/[email protected]/pkg/adapter/v2/main.go:205\nknative.dev/eventing/pkg/adapter/v2.MainWithEnv\n\tknative.dev/[email protected]/pkg/adapter/v2/main.go:105\nknative.dev/eventing/pkg/adapter/v2.MainWithContext\n\tknative.dev/[email protected]/pkg/adapter/v2/main.go:80\nknative.dev/eventing/pkg/adapter/v2.Main\n\tknative.dev/[email protected]/pkg/adapter/v2/main.go:76\nmain.main\n\tgithub.com/triggermesh/triggermesh/cmd/slack-target-adapter/main.go:25\nruntime.main\n\truntime/proc.go:204"}

Update container build/release pipeline to use `ko` instead of Dockerfiles

Our Dockerfile-based image building process has been painfully slow due to inefficient caching, leading to build failures becoming the norm.

There are two problems to address:

• Cloud Build runs every individual image build as a separate job.
  For each of our 56 components, we either rebuild the same packages over and over again, or have to wait for very long cache loads, whereas we could build everything in one pass instead, and then create all 56 images very fast.

• Our builds can't leverage Docker's caching mechanism due to their nature.
  We don't install OS packages, or pull large files as part of our builds. The only thing we do is copy our source code, and compile our code, which changes on every commit. Therefore, the only step which hits the cache it go mod download, which would only take a few seconds without cache anyway.

Since we already use ko during development, we could as well leverage it to release our images.

---

A few things we should pay attention to:

1. Images naming
2. Images tagging
3. Releasing manifests

Images naming

By default, ko will name images gcr.io/triggermesh/confluenttarget-adapter-<md5>, where <md5> is the MD5 hash of the full import path:

$ echo -n 'github.com/triggermesh/triggermesh/cmd/confluenttarget-adapter' | md5sum
0931029195c247eb6a799674b2f21e28  -

$ ko publish ./cmd/confluenttarget-adapter
2021/10/09 11:27:45 Using base gcr.io/distroless/base:nonroot for github.com/triggermesh/triggermesh/cmd/confluenttarget-adapter
2021/10/09 11:27:47 Building github.com/triggermesh/triggermesh/cmd/confluenttarget-adapter for linux/amd64
2021/10/09 11:28:13 Publishing gcr.io/triggermesh/antoineco/confluenttarget-adapter-0931029195c247eb6a799674b2f21e28:latest

Our components' names are unlikely to conflict because we don't use generic cmd names such as "controller", "adapter" or "webhook" anywhere, so we can get rid of that hash with the -B flag:

$ ko publish -B ./cmd/confluenttarget-adapter
2021/10/09 11:42:35 Using base gcr.io/distroless/base:nonroot for github.com/triggermesh/triggermesh/cmd/confluenttarget-adapter
2021/10/09 11:42:38 Building github.com/triggermesh/triggermesh/cmd/confluenttarget-adapter for linux/amd64
2021/10/09 11:42:43 Publishing gcr.io/triggermesh/antoineco/confluenttarget-adapter:latest
...

Images tagging

ko allows setting a list of image tags during builds, but by default will return the digest version of the image:

(last line in the output is stdout, the rest is stderr)

$ ko publish -B -t v0.0.0 ./cmd/confluenttarget-adapter
...
2021/10/09 11:58:21 Building github.com/triggermesh/triggermesh/cmd/confluenttarget-adapter for linux/amd64
2021/10/09 11:58:26 Publishing gcr.io/triggermesh/antoineco/confluenttarget-adapter:v0.0.0
...
2021/10/09 11:58:29 gcr.io/triggermesh/antoineco/confluenttarget-adapter:v0.0.0: digest: sha256:b73f38cc141ab4a8feb4fff5a6d3b53600d39d2d775ac9d67f7ec39a6b04d49d size: 1115
2021/10/09 11:58:29 Published gcr.io/triggermesh/antoineco/confluenttarget-adapter:v0.0.0@sha256:b73f38cc141ab4a8feb4fff5a6d3b53600d39d2d775ac9d67f7ec39a6b04d49d
gcr.io/triggermesh/antoineco/confluenttarget-adapter:v0.0.0@sha256:b73f38cc141ab4a8feb4fff5a6d3b53600d39d2d775ac9d67f7ec39a6b04d49d

In case we want to consume that output for any reason, and are not interest in the digest, we will need to use the --tag-only flag:

(last line in the output is stdout, the rest is stderr)

$ ko publish -B -t v0.0.0 --tag-only ./cmd/confluenttarget-adapter
...
2021/10/09 12:02:42 Building github.com/triggermesh/triggermesh/cmd/confluenttarget-adapter for linux/amd64
2021/10/09 12:02:47 Publishing gcr.io/triggermesh/antoineco/confluenttarget-adapter:v0.0.0
...
2021/10/09 12:02:50 gcr.io/triggermesh/antoineco/confluenttarget-adapter:v0.0.0: digest: sha256:b73f38cc141ab4a8feb4fff5a6d3b53600d39d2d775ac9d67f7ec39a6b04d49d size: 1115
gcr.io/triggermesh/antoineco/confluenttarget-adapter:v0.0.0

Releasing manifests

ko supports label selectors, so the following commands do what we expect.

For CRDs:

$ ko resolve -f config/ -l 'triggermesh.io/crd-install' > triggermesh-crds.yaml

For everything else:

$ ko resolve -f config/ -l '!triggermesh.io/crd-install' > triggermesh.yaml

⚠️ The negated label selector above has a strange edge case which should be resolved in ko-build/ko#467

Do not pass CloudEvent `type` attribute via ceOverrides

The two sources labeled on this issue hack the ceOverrides spec in order to override the type attribute of CloudEvents sent by their receive adapter:

https://github.com/triggermesh/event-sources/blob/7dfd216b238e65228c081729fe858c36065ce6a6/pkg/reconciler/azureactivitylogssource/adapter.go#L129-L132

This practice is strongly discouraged, and the only reason why it currently works is due to a bug in the client library:

When spec.ceOverrides.extensions is specified and non-empty, the event sent to the sink is augmented or modified with the specified extension attributes. The source controller MUST reject extensions overriding the required Cloud Events attributes.

-- https://github.com/knative/eventing/blob/5c08bd3d/docs/spec/sources.md#duckspec

We should fix this ASAP by using either

• message processors (e.g. setting it to "activity-logs" enables knowledge about Activity Logs payloads)
• a plain env var such as CE_TYPE, which may work when only 1 type is expected
• other suggestions