raystack/depot

Depot contains various common sink implementations and publishes them as a library. This library will be used in firehose, daggers or any other application which wants to send data to destinations.

depot's Issues

setup gradle cache in github workflow and use --no-daemon flag in gradle build

reference https://github.com/actions/cache/blob/main/examples.md#java---gradle

Add HTTP Sink in depot

• Create HttpClient and related classes to include all the HTTP auth and HTTP methods.
• Implement a bare minimum HttpSink which sends data to the HTTP server.
• Add support for parsing of list data type in getFieldByName() method in ParsedOdpfMessage.
• getFieldByName() should return ParsedMessage instead of Object. ParsedMessage should contain more methods like getType(), isReapated(), isStruct(), isPrimitive() etc etc.
• Add ToJson() method to convert struct into JSON string in ParsedOdpfMessage class.
• Create HeaderBuilder class
• Create URI builder and QueryParam class
• Create Raw Payload builder
• Create Json Payload builder
• Create Message Payload Builder
• Create String Payload builder
• Create JsonTemplate Payload Builder
• Create RequestBuilder build() method for Single request
• Create RequestBuilder build() method for Batch request
• Create HttpRecordConverter
• Implement HttpRequestRecord and HttpResponseRecord classes to contain methods like logging based on configs.
• Add methods to parse the HttpResponseRecord to set correct ErrorTypes.
• Instrumentation: Add sink related metrics
• Documentation

redis sink: complex protobuf data type values not getting synced properly

Description
redis sink: complex protobuf data type values not getting synced properly
Steps To Reproduce

1. REDIS SINK
2. With config INPUT_SCHEMA_PROTO_TO_COLUMN_MAPPING: {"2":"topics"}
3. With config SINK_REDIS_KEY_TEMPLATE: patch-subscription:%s,customer_name
4. With protobuf message as
5. message TestBookingLogMessage { string customer_name = 1; message TopicMetadata { string topic = 1; int32 qos = 2; } repeated TopicMetadata topics = 2;

Expected behavior
""[{"topic":"world/pns/1/com.hello.owl/701483858/3ac6a577fbd334b4","qos":1}]""

Additional context
"[topic: "world/pns/1/com.hello.owl/701483858/3ac6a577fbd334b4"\nqos: 1\n]"

Proposed Fix.
Use JsonPrinter in ProtoOdpfParsedMessage.getFieldByName()

Add BigTable sink in Depot

Summary
As part of this feature, we want to add support for Google’s Bigtable sink in Depot.

Tasks

• Implement Bigtable sink with batching with the hardcoded key in the depot
• Add input to column - family - qualifier mapping support
• Define RowKey template and implement RowKeyTemplateParser
• Error parser and error handling for bigtable sink
• Capture Bigtable metrics and add instrumentation(logging and monitoring)
• Add dashboard panels to capture BigTable sink metrics
• Documentation for bigtable sink

Allow ignore unknown value in firehose BQ client

There's issue on starting firehose when proto schema is not matching with BQ table schema. Sample error looks like this:
Provided Schema does not match Table xxx. Field yyy is missing in new schema

Need a feature to ignore this error and fill with default value if proto schema is incomplete.
This can be done by allowing ignoreUnknownValues while instantiating BQ client.
source: https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll#request-body
Firehose version: odpf/firehose:0.7.1