googleapis/google-cloudevents-go

CloudEvent Types for Go

Google CloudEvents - Go

This library provides Go types for Google CloudEvent data.

Features

• Simple import and interface
• Inline documentation for Go structs
• Automatic decoding of base64 data
• Enum support
• Protobuf bindings

Installation

To install this package, run:

go get -u github.com/googleapis/google-cloudevents-go

This library requires Go 1.17+ and is tested with Go 1.19.

Usage

Unmarshal a CloudEvent data payload from raw bytes.

package examples

import (
	"fmt"
	"log"
	"time"

	"github.com/googleapis/google-cloudevents-go/cloud/storagedata"
	"google.golang.org/protobuf/encoding/protojson"
)

// cloudEventPayload is initialized with an example CloudEvent data payload.
// Source: github.com/googleapis/google-cloudevents/tree/main/examples/binary/storage/StorageObjectData-simple.json
var cloudEventPayload = []byte(`
{
	"bucket": "sample-bucket",
	"contentType": "text/plain",
	"crc32c": "rTVTeQ==",
	"etag": "CNHZkbuF/ugCEAE=",
	"generation": "1587627537231057",
	"id": "sample-bucket/folder/Test.cs/1587627537231057",
	"kind": "storage#object",
	"md5Hash": "kF8MuJ5+CTJxvyhHS1xzRg==",
	"mediaLink": "https://www.googleapis.com/download/storage/v1/b/sample-bucket/o/folder%2FTest.cs?generation=1587627537231057\u0026alt=media",
	"metageneration": "1",
	"name": "folder/Test.cs",
	"selfLink": "https://www.googleapis.com/storage/v1/b/sample-bucket/o/folder/Test.cs",
	"size": "352",
	"storageClass": "MULTI_REGIONAL",
	"timeCreated": "2020-04-23T07:38:57.230Z",
	"timeStorageClassUpdated": "2020-04-23T07:38:57.230Z",
	"updated": "2020-04-23T07:38:57.230Z"
}`)

func Example() {
	data := storagedata.StorageObjectData{}
	if err := protojson.Unmarshal(cloudEventPayload, &data); err != nil {
		log.Fatal("protojson.Unmarshal: ", err)
	}

	updated := data.Updated.AsTime().Format(time.UnixDate)
	fmt.Printf("Bucket: %s, Object: %s, Updated: %s", data.Bucket, data.Name, updated)

	// Output: Bucket: sample-bucket, Object: folder/Test.cs, Updated: Thu Apr 23 07:38:57 UTC 2020
}

Contributing

Contributions to this library are always welcome and highly encouraged.

See CONTRIBUTING for more information how to get started.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See Code of Conduct for more information.

google-cloudevents-go's Issues

[Policy Bot] found one or more issues with this repository.

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• GitHub detects a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

[golang] auditv1.AuditLogWrittenEvent{} unable to unmarshall CloudRun AuditLog Event

AuditLogs for events for Cloud Run cannot be unmarshalled using the cloud-events SDK's event.DataAs{} with a struct of type auditv1.AuditLogWrittenEvent{}

For example, the if the snippet below is received either as a pubsub or run event as a GCP AuditLog, the code throws a parsing error as shown below

		audit_data := &auditv1.AuditLogWrittenEvent{}
		if err := event.DataAs(audit_data); err != nil {
			fmt.Printf("DataAs Error %v", err)
			return fmt.Errorf("Got Data Error: %s\n", err)
		}
		fmt.Printf("Audit ResourceName %s\n", audit_data.ProtoPayload)

2020/09/23 18:12:28 projects/mineral-minutia-820/logs/cloudaudit.googleapis.com%2Factivityyopiilehxskr1600899141625294
DataAs Error [json] found bytes "{"protoPayload":{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":...
, but failed to unmarshal: json: cannot unmarshal string into Go struct field AuditLogWrittenEvent.severity of type int64

i suspect there is a core mismatch either on the event from GCP or the source proto thats used to generate the AuditLogWirttenEvent.

---

What i can state is the following work to unmarshall the LogEntry first as DataAs and then as an AuditLog messagge

import (
      	logging "google.golang.org/api/logging/v2"
	svc "google.golang.org/api/servicecontrol/v1"
)
		loggingData := &logging.LogEntry{}
		if err := event.DataAs(loggingData); err != nil {
			return fmt.Errorf("Got Data Error %v", err)
		}
		audit_data := &svc.AuditLog{}

		err := json.Unmarshal(loggingData.ProtoPayload, audit_data)
		if err != nil {
			return fmt.Errorf("Got Data Error %v", err)
		}
		fmt.Printf("Audit ResourceName %s\n", audit_data.ResourceName)

pls comare the combined logging.LogEntry{} with svc.AuditLog{} struct as provided by google.golang.org/api/servicecontrol/v1 and the logging library agasint auditv1.AuditLogWrittenEvent{}. You can verify/repo this by subcribing to a cloud run audit event.

Evaluate replacement of realpath with string concatenation

Current Behavior

CONTRIBUTING.md, tools/setup-generator.sh, and GitHub Workflows use realpath to derive an absolute path to the parent repository because the absolute path is needed for validation testing.

Preferred Behavior

Use simpler string concatenation, such as "${PWD}/${relative_path}". See if this means we can remove the dedicated GitHub Workflow step for dynamically deriving the path.

Remove multiple small packages

Expected Behavior

A single package that contains all cloud event types

Actual Behavior

Directories like cloud/audit, cloud/cloudbuild, cloud/firestore, etc

Details

Maybe all of the events could be in the same package and just different files?

Add protobuf example & README documentation

Problem

We now support receiving events in binary protobuf format, and expect Firestore events to be primarily used in protobuf. Current examples only show JSON parsing.

Proposal

Add examples of how to parse protobuf event payloads with the library in two forms:

• Standalone
• In conjunction with CloudEvent SDK for Go: https://github.com/cloudevents/sdk-go

Rough Example (Untested)

import (
    "log"
    cloudevent "github.com/cloudevents/sdk-go/v2"
    _ "github.com/cloudevents/sdk-go/binding/format/protobuf/v2"
    "github.com/googleapis/google-cloudevents-go/cloud/firestoredata"
)

func handler(w http.ResponseWriter, r *http.Request) {
    event := cloudevent.NewEventFromHTTPRequest(r)
    var target firestoredata.DocumentEventData
    if err := event.DataAs(&target); err != nil {
        log.Fatal("event.DataAs: %v", err)
    }
    fmt.Println(target.Value.Name)
}

remove v1 suffixes on import paths

Expected Behavior

A v0/v1 module that contains no import paths with version numbers.

Actual Behavior

Imports like: "github.com/googleapis/google-cloudevents-go/cloud/audit/v1"

Details

Directories named vX have a meaning with Go and its module ecosystem. For v0/v1 no suffix should be added. It is for this reason that when we generate GAPICs we do so with apiv1/api2beta.

Cannot unmarshall logEntry.Severity

I'm trying unmarshall and inbound cloudEvent Audit entriy into github.com/googleapis/google-cloudevents-go/cloud/audit/v1.LogEntryData as shown below.

However, the Severity field that is sent is just a string while the auditv1 below defines it

type Severity struct {
	Enum    *InsertID
	Integer *int64
}

while google eventarc send over just a string value

 "{"protoPayload":{"status":{},"authenticationInfo....

 "severity":"NOTICE",              <<<<<<<<<<<<<

which leads to the exception

 json: cannot unmarshal string into Go struct field LogEntryData.severity of type audit.Severity 

---

import (

	cloudevents "github.com/cloudevents/sdk-go/v2"
	auditv1 "github.com/googleapis/google-cloudevents-go/cloud/audit/v1"
)

func Receive(ctx context.Context, event cloudevents.Event) error {
	fmt.Printf("  EventID: %s\n", event.ID())
	fmt.Printf("  EventType: %s \n", event.Type())
	fmt.Printf("  Event Context: %+v\n", event.Context)
	fmt.Printf("  Protocol Context: %+v\n", pscontext.ProtocolContextFrom(ctx))
	switch event.Type() {
	case auditLogEventType:
		auditData := &auditv1.LogEntryData{}
		if err := event.DataAs(auditData); err != nil {
			fmt.Printf("DataAs Error %v", err)
			return fmt.Errorf("Got Data Error: %s\n", err)
		}
		fmt.Printf("Audit auditLogEventType ResourceName %s\n", *auditData.ProtoPayload.ResourceName)

	default:
	}
	return nil
}
func main() {
	c, err := cloudevents.NewDefaultClient()
	err = c.StartReceiver(context.Background(), Receive)
}

---

see also #22

Library generation failed (dd2ac79)

See failed job

Use Pub/Sub event as input to README sample

Expected Behavior

Sample demonstrates how to unmarshal a Pub/Sub event payload.

Actual Behavior

Sample will break unmarshalling a non-JSON []byte

Steps to Reproduce the Problem

Create go.mod

This module should have a go.mod and go.sum to be used by other Go programs.

Error when parsing google.cloud.storage.object.v1.finalized events using storage.UnmarshalStorageObjectData

Expected Behavior

We're able to parse the event and populate data into StorageObjectData object.

Actual Behavior

Got the following error instead:

{"level":"error","ts":1610564646.3508415,"logger":"fallback","caller":"client/invoker.go:66","msg":"call to Invoker.Invoke(...) has panicked: json: cannot unmarshal string into Go struct field StorageObjectData.generation of type int64","stacktrace":"..."}

pretty printed stacktrace:

github.com/cloudevents/sdk-go/v2/client.(*receiveInvoker).Invoke.func2.1
        /go/pkg/mod/github.com/cloudevents/sdk-go/v2@v2.3.1/client/invoker.go:66
runtime.gopanic
        /usr/local/go/src/runtime/panic.go:969
main.Receive
        /app/main.go:35
reflect.Value.call
        /usr/local/go/src/reflect/value.go:476
reflect.Value.Call
        /usr/local/go/src/reflect/value.go:337
github.com/cloudevents/sdk-go/v2/client.(*receiverFn).invoke
        /go/pkg/mod/github.com/cloudevents/sdk-go/v2@v2.3.1/client/receiver.go:86
github.com/cloudevents/sdk-go/v2/client.(*receiveInvoker).Invoke.func2
        /go/pkg/mod/github.com/cloudevents/sdk-go/v2@v2.3.1/client/invoker.go:69
github.com/cloudevents/sdk-go/v2/client.(*receiveInvoker).Invoke
        /go/pkg/mod/github.com/cloudevents/sdk-go/v2@v2.3.1/client/invoker.go:71
github.com/cloudevents/sdk-go/v2/client.(*ceClient).StartReceiver.func2.1
        /go/pkg/mod/github.com/cloudevents/sdk-go/v2@v2.3.1/client/client.go:233

Steps to Reproduce the Problem

Event (in go object format)

Context Attributes,
  specversion: 1.0
  type: google.cloud.storage.object.v1.finalized
  source: //storage.googleapis.com/projects/_/buckets/events-quickstart-danglu-knative-dev-290519
  subject: objects/go-test-123.txt
  id: 1911674720947509
  time: 2021-01-13T19:03:20.8Z
  dataschema: https://raw.githubusercontent.com/googleapis/google-cloudevents/master/proto/google/events/cloud/storage/v1/data.proto
  datacontenttype: application/json
Extensions,
  knativearrivaltime: 2021-01-13T19:03:21.568106765Z
  knsourcetrigger: link0.8966907269185795
  traceparent: 00-31c5577593932217af33e2c0ab5dcf12-32037e59dd69f8b4-00
Data,
  {
    "kind": "storage#object",
    "id": "events-quickstart-danglu-knative-dev-290519/go-test-123.txt/1610564600595212",
    "selfLink": "https://www.googleapis.com/storage/v1/b/events-quickstart-danglu-knative-dev-290519/o/go-test-123.txt",
    "name": "go-test-123.txt",
    "bucket": "events-quickstart-danglu-knative-dev-290519",
    "generation": "1610564600595212",
    "metageneration": "1",
    "contentType": "application/octet-stream",
    "timeCreated": "2021-01-13T19:03:20.603Z",
    "updated": "2021-01-13T19:03:20.603Z",
    "storageClass": "STANDARD",
    "timeStorageClassUpdated": "2021-01-13T19:03:20.603Z",
    "size": "1682",
    "md5Hash": "DqB0N1pmMRX4mjv1llfoWQ==",
    "mediaLink": "https://www.googleapis.com/download/storage/v1/b/events-quickstart-danglu-knative-dev-290519/o/go-test-123.txt?generation=1610564600595212&alt=media",
    "contentLanguage": "en",
    "crc32c": "mA33eA==",
    "etag": "CIyGoNfMme4CEAE="
  }

Code

go.mod

module xxxxx

go 1.13

require (
	github.com/cloudevents/sdk-go/v2 v2.3.1
	github.com/googleapis/google-cloudevents-go v0.0.0-20210109004349-be1f906e2061 // indirect
)

main.go

import (
	"context"
	"fmt"
	"log"

	cloudevents "github.com/cloudevents/sdk-go/v2"
	"github.com/googleapis/google-cloudevents-go/cloud/storage/v1"
)

func Receive(event cloudevents.Event) {
        fmt.Printf("%+v\n", event)
	e, err := storage.UnmarshalStorageObjectData(event.Data())
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", e)
	fmt.Printf("Detected change in GCS bucket: %s\n", event.Subject())
}

func main() {
	c, err := cloudevents.NewDefaultClient()
	if err != nil {
		log.Fatalf("failed to create client, %v", err)
	}
	log.Fatal(c.StartReceiver(context.Background(), Receive))
}

Add space between copyright and package name

Expected Behavior

A space between copyright and package name.

Actual Behavior

There is no space today

Details

There should be a space between the copyright and the package name or else the copyright shows up in godoc: see https://pkg.go.dev/github.com/googleapis/google-cloudevents-go/cloud/audit/v1 as an example.

[Policy Bot] found one or more issues with this repository.

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• There is a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): update dependency @types/node to v14.17.14
• chore(deps): update dependency typescript to v4.4.2
• fix(deps): update dependency node-fetch to v3

---

• Check this box to trigger a request for Renovate to run again on this repository

Use GAPIC Generator instead of Protoc

Expected Behavior

The GAPIC generator is more updated. Use that instead of protoc:
https://github.com/googleapis/gapic-generator-go

Actual Behavior

Using protoc.

Improve error reporting in protoc-gen-go-googlecetypes

Current Behavior

Each function in the generator uses log.Fatal or panic on encountering an error.

Improved Behavior

Return an error from the function and centralize error handling.

Preferred Behavior

Aggregate all errors from the parsing process and report at once. This has two facets:

1. When assembling data for use in code generation, assemble in advance, aggregate all errors, then flush at the end.
2. Instead of halting the generate-code.sh script on any error, continue processing but ensure to fail the script if any event type has failures

Cannot parse LogEntryData (google.cloud.audit.log.v1.written type) with latest google-cloudevents-go

Latest commit to github.com/googleapis/google-cloudevents-go seems to break parsing GCP audit log types

In the following snippet, if i use github.com/googleapis/google-cloudevents-go v0.0.0-20201014214533-8f932ccdc752, i can use the cloud-events go SDK event.DataAs() fine but if i upgrade to github.com/googleapis/google-cloudevents-go v0.0.0-20201106055002-2d7dd1dcf649 , i'll see the error shown below about LogEntryData.severity

---

require (
	cloud.google.com/go v0.71.0 // indirect
	github.com/cloudevents/sdk-go/protocol/pubsub/v2 v2.3.1
	github.com/cloudevents/sdk-go/v2 v2.3.1
	github.com/googleapis/google-cloudevents-go v0.0.0-20201014214533-8f932ccdc752  // works
	github.com/googleapis/google-cloudevents-go v0.0.0-20201106055002-2d7dd1dcf649  // does not work
	google.golang.org/api v0.35.0
)

import  (
	auditv1 "github.com/googleapis/google-cloudevents-go/cloud/audit/v1"
)
		auditData := &auditv1.LogEntryData{}
		if err := event.DataAs(auditData); err != nil {
			fmt.Printf("DataAs Error %v", err)
			return fmt.Errorf("Got Data Error: %s\n", err)
		}
		log.Printf("Audit v ResourceName %s\n", *auditData.ProtoPayload.ResourceName)

gives

DataAs Error [json] found bytes "", 
but failed to unmarshal: json: 
   cannot unmarshal string into Go struct field LogEntryData.severity of type audit.NumResponseItems

Add install instructions

Expected Behavior

We should have basic install instructions for this library. Something like:

go get -u github.com/googleapis/google-cloudevents-go

Actual Behavior

No instructions for installing.

label deprecated fields conventionally

Expected Behavior

// Foo does something.
// Deprecated: use Bar instead.
type Foo struct {}

Actual Behavior

// Deprecated, use bar instead.
type Foo struct {}

Details

For deprecated fields it would be nice to match godoc conventions so users can get a warning in their editor.

Refactor hard to use types map[string]map[string]interface{}

Expected Behavior

type Foo string {
   Bar AHelperType
}

Actual Behavior

type Foo string {
   Bar map[string]map[string]interface{}
}

Details

There are a few instances of map[string]map[string]interface{} being used. I wonder if a nicer type could be generated to make using these fields easier to use and construct? It is unclear to me why things like Claims embed this type. I think Claims itself could just be a map[string]interface{}? Example of where we do something similar in the client libraries: https://pkg.go.dev/google.golang.org/api/idtoken#Payload.

[Policy Bot] found one or more issues with this repository.

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• There is a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

audit LogEntryData.ProtoPayload.Metadata is empty

When I want to get audit.LogEntryData from cloudevents.Event by using event.DataAs, decode itself was success but Metadata, Request, and other several fields are empty.

All of those empty fields have this struct.

type Metadata struct {
	Fields map[string]interface{} `json:"fields,omitempty"` // Unordered map of dynamically typed values.
}

This struct has Fields. This is because the definition of proto file and google.protobuf.Struct has this fields.

However at least my envionment and this sample of cloudevents, request does not have a fields key.

        request: {
          resource: "my-gcp-project-id",
          policy: { bindings: [...], }
        },

So nothing decoded on those properties. When I changed the code, I can get values as expected.

type Metadata map[string]interface{} // Unordered map of dynamically typed values.

process: slow down generator re-runs

Background

The code generator is run nightly, and if there's any diff, a PR is opened.

Currently, the generated tests include the git SHA of the repo as part of declaring the version of the custom generator. As a result, any change to the repo leads to the generator opening a PR / pushing an update.

Proposed Change

The SHA of the code generator still has value, because it allows us to look at generated code and see the specific SHA associated with the file being generated, which may be valuable to contrast library version from latest update to a specific type. However, the value of the overall repo SHA is not so high. Instead of HEAD, let's use the latest commit associated with a change in the generators directory.

Instead of using LibraryVersion:
https://github.com/googleapis/google-cloudevents-go/blob/main/generate-code.sh#L41-L42
https://github.com/googleapis/google-cloudevents-go/blob/main/generators/templates/validationtest.gotpl#L19

We would derive using:

git rev-parse --short HEAD:generators

Add json hints for unmarshal types like int64

I transported Cloud Stroage object.finalize events through Pub/Sub. When I decoded the event from pubsub message body, it failed with error message like this: json: cannot unmarshal string into Go struct field StorageObjectData.generation of type int64

I temporary solved this issue by adding json: ",string" tags. Could it be possible to add the tags? It can help json unmarshal successfully.

cannot unmarshal string into Go struct field StorageObjectData.generation of type int64

I'm writing a gcloud function that handle new file uploaded into storage bucket and it's failed to unmarshall event data into StorageObjectData.

Environment details

• Programming language: Golang
• OS:
• Language runtime version: Go 1.19
• Package version: v0.6.0

Steps to reproduce

My function is simple:

package functions

import (
	"context"

	_ "github.com/GoogleCloudPlatform/functions-framework-go/funcframework"
	"github.com/googleapis/google-cloudevents-go/cloud/storagedata"
	"github.com/sirupsen/logrus"
)

func StorageWrite(ctx context.Context, data *storagedata.StorageObjectData) error {
	log := logrus.WithContext(ctx).WithField("name", data.Name).WithField("data", data)
	log.Info("Executed")
	return nil
}

I deployed with this trigger-event "providers/cloud.storage/eventTypes/object.change"

Error logs:

image_writezdfhzqyqwy14 Error: json: cannot unmarshal string into Go struct field StorageObjectData.generation of type int64, while converting event data: {"bucket":"anhemdev-dummy","contentType":"image/png","crc32c":"RDX58Q==","etag":"CMmpzM/W5f8CEAE=","generation":"1687945399702729","id":"anhemdev-dummy/tokens.png/1687945399702729","kind":"storage#object","md5Hash":"Y2JkMGQxNDExOTRiYTY1NWQzYmUwNzA5YTJhZWViYzY=","mediaLink":"https://storage.googleapis.com/download/storage/v1/b/anhemdev-dummy/o/tokens.png?generation=1687945399702729&alt=media","metageneration":"1","name":"tokens.png","resourceState":"exists","selfLink":"https://www.googleapis.com/storage/v1/b/anhemdev-dummy/o/tokens.png","size":"217170","storageClass":"STANDARD","timeCreated":"2023-06-28T09:43:19.717Z","timeStorageClassUpdated":"2023-06-28T09:43:19.717Z","updated":"2023-06-28T09:43:19.717Z"}

Thank you

Package Should Not Include Version Number

Expected Behavior

The package shouldn't include the version number:

import (
  "github.com/googleapis/google-cloudevents-go/cloud/pubsub/v1"
)
pubsub.UnmarshalMessagePublishedData

A user can use an alias if they want to:

import (
  pubsubv1 "github.com/googleapis/google-cloudevents-go/cloud/pubsub/v1"
  pubsubv2 "github.com/googleapis/google-cloudevents-go/cloud/pubsub/v2"
)

pubsubv1.UnmarshalMessagePublishedData

Re-add Go documentation

Look at Chen's previous work and add docs per language.

Add YAML Linting to validate changes to workflows

The generator workflow and various configuration files in .github use YAML configuration. In most cases, syntax errors in these YAML configurations will not be detected through existing PR checks. Add a linter to catch basic errors.

[Policy Bot] found one or more issues with this repository.

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• GitHub detects a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

Setup Release Tooling

Setup Release Please:
https://github.com/googleapis/release-please/

ci: add action for testing golang compilation

In GitHub Actions, there should be an action that tests to ensure that this library's golang code builds

Have struct docs conform with godoc conventions

Expected Behavior

// Foo: something something something about foo.
type Foo struct {}

Actual Behavior

// Something something something about foo.
type Foo struct {}

Details

It would be nice if the generated docs could follow godoc conventions. For example Auth could start with: “Auth: The third ...”

Enable master branch protection

      This repository does not seem to have master branch
      protection enabled, at least in the way I'm expecting.
      I was hoping for:

      - master branch protection
      - requiring at least one code reviewer
      - requiring at least two status checks
      - enforcing rules for admins

      Please turn it on!

json: invalid use of ,string struct tag, trying to unmarshal unquoted value into *int64

Hello!

We're using Pub/Sub notifications for Google Cloud Storage, which upon receival we json.Unmarshal to storage.StorageObjectData. When enabling parallel composite uploads using gsutil we stumble upon an error.

        [...]
	var e storage.StorageObjectData
        err := json.Unmarshal(msg.Data, &e)

will return: json: invalid use of ,string struct tag, trying to unmarshal unquoted value into *int64 as the componentCount field in the payload is "componentCount": 10 (or some other integer).

Disclaimer: I saw the comment regarding that this is under heavy refactoring, but thought I would lift this to the surface.

Environment details

• Programming language: Go
• OS: Cloud Run instance running a simple go-binary based on gcr.io/distroless/static:nonroot
• Language runtime version: 1.17
• Package version: v2.3.0

Steps to reproduce

package unmarshal_test

import (
	"encoding/json"
	"testing"
        
        "github.com/googleapis/google-cloudevents-go/cloud/storage/v1"
)

func TestUnmarshalComponentCount(t *testing.T) {
	data := `{"componentCount": 16}` 
	var e storage.StorageObjectData
	if err := json.Unmarshal([]byte(data), &e); err != nil {
		t.Errorf("failed: %v", err)
	}
}

Copying the struct definition locally and removing string in the ComponentCount json field tag does not have this problem.

Generate Type Index

Expected Behavior

We generate a type index programmatically:

Package	Struct	Description
pubsubv1	MessagePublishedData	A message that is published by publishers and consumed by subscribers.
auditv1	LogEntryData	This event is triggered when a new audit log entry is written.

Actual Behavior

We don't have any automatic way of generating this.

Consider helpers or different types for pointers to builtins

Expected Behavior

// Idea #1
type Foo struct {
   Bar *string 
}

func main() {
   foo := Foo{
      // a helper that returns a pointer
      Bar: cloudevent.String("test"),
   }
}

// Idea #2
type Foo struct {
   Bar optional.String 
}

func main() {
   foo := Foo{
      Bar: "test",
   }
}

Actual Behavior

type Foo struct {
	Bar *string
}

func main() {
	bar := "test"
	foo := Foo{
                 // Variable needs to be predeclared and a reference taken from it.
		Bar: &bar,
	}
}

Details

I like the choice to make the structs have pointer fields in general, not doing so in the Go apiary libraries has led to much confusion. But at the same time they can be hard to work with. For this reason in the GAPIC libraries we have the optional package that is referenced in structs directly. Another pattern I have seen is to provide some helper functions to convert to and from pointers. Have you gathered any user feedback in this area?

generator: Version string munging cannot handle dataflow's v1beta3

Problem

Output from nightly generator job:

=== RUN   TestParsingJobEventData
    data_test.go:41: No test cases found: os.ReadDir: open /home/runner/work/google-cloudevents-go/google-cloudevents-go/protos-source-repo/testdata/google/events/cloud/dataflowdatav1beta3/v1: no such file or directory

Likely there is regex matching along the lines of v\d+$ not accounting for pre-release suffix.

Proposal

Iterate to a more comprehensive version match.

This is blocking #155 . Since it's a pre-release version without other event types queued up for release, I'm keeping this bug at P2.

Add godoc link

Expected Behavior

README has godoc badge such as

Actual Behavior

No godoc link.

[Policy Bot] found one or more issues with this repository.

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• GitHub detects a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

Remove marshaller/unmarshaller functions

Expected Behavior

Users of this library should work with the json package directly.

Actual Behavior

There are non-standard custom marshaller functions/methods. ex1 ex2

Details

I don’t think there is a need from the customer marshallers/unmarshallers. If anything special is needed that is not directly supported by the json package I would instead implement a custom JSON marshaller that is respected by the json package.

🛠️ Significant Refactoring Underway 🛠️

In the next few weeks this project will be going through significant refactoring.

There will be updates to generate code from protobufs. This will change the resulting data structures and import paths. There will also be updates to help maintainers ensure this library will work reliably to help unmarshal data payloads from CloudEvents.

We do not see usage of this library, which has never been noted as stable or released to a version 1, so there will be breaking changes.

use []byte or custom type for pubsub Message data

Expected Behavior

// Option 1
type Message struct {
	Data        []byte        `json:"data,omitempty"`        // The binary data in the message.
}

// Option 2
type Message struct {
	Data        ByteData        `json:"data,omitempty"`        // The binary data in the message.
}

Actual Behavior

type Message struct {
	Data        *string        `json:"data,omitempty"`        // The binary data in the message.
}

Details

The choice to use a string was also made in our HTTP/REST libraries. We did not respect the ”format”: “byte”. We have an open issue around how it would have been nice to generate these types as []byte.

Option 2 might be nice do you could provide a custom mashaller for the type that auto encodes/decodes base64 values so the user of this library does not have to.

ci: add test for event types

There should be tests for each event type (or at least a few) to ensure that the event type can be correctly parsed.

[Policy Bot] found one or more issues with this repository.

Policy Bot found one or more issues with this repository.

• Default branch is 'main'
• Branch protection is enabled
• Renovate bot is enabled
• Merge commits disabled
• There is a CODEOWNERS file
• GitHub detects a valid LICENSE.md
• There is a CODE_OF_CONDUCT.md
• There is a CONTRIBUTING.md
• There is a SECURITY.md

Add support for Pub/Sub

Current Behavior

Pub/Sub is suppressed from the code generation process in generate-code.sh because Pub/Sub messages are incompatible with protojson parsing. This is because Pub/Sub messages include duplicate fields message_id/messageId, and publish_time/publishTime, which the protobuf spec indicates is invalid.

Expected Behavior

All Event Sources have type libraries.

Approach

Add a custom function that can be used to pre-process Pub/Sub payloads before unmarshalling.

We already have a function in the repository that can facilitate this that was put in place to enable testing:

google-cloudevents-go/internal/testhelper/prepare.go

Lines 11 to 27 in 322c786

	func PreparePubSubMessagePublishedData(b []byte) ([]byte, error) {
	var j map[string]interface{}
	if err := json.Unmarshal(b, &j); err != nil {
	return nil, fmt.Errorf("json.Unmarshal: %w", err)
	}
	m := j["message"].(map[string]interface{})
	// Remove reserved @type field.
	delete(m, "@type")
	// Remove message_id field name conflict with messageId.
	delete(m, "message_id")
	// Remove publish_time field name conflict with publishTime.
	delete(m, "publish_time")
	j["message"] = m
	return json.Marshal(j)
	}

• Move this function to the public API, preferably into the pubsubdata package. This will require a copying/templating process.
• Unblock Pub/Sub from code generation
• Update the code generator to use this function instead of the internal/testheper package

google-cloudevents-go/generators/protoc-gen-go-googlecetypes/generator.go

Lines 146 to 147 in 322c786

	case params.TypePrefix == "google.events.cloud.pubsub.v1" && dataType == "MessagePublishedData":
	params.PrepareFunction[dataType] = "testhelper.PreparePubSubMessagePublishedData"