redis-field-engineering/redis-connect-dist

Real-Time Event Streaming & Change Data Capture

Redis Connect 🚀 is a distributed platform that enables real-time event streaming, transformation, and propagation of changed-data events from heterogeneous data platforms to Azure Cache for Redis, Redis Cloud, and Redis Enterprise.

✅ No Code ✅ Seamless Integration ✅ Multi-Tenancy ✅ Linear-Scalability ✅ High-Availability ✅ Support

End-to-End demos

Video Tutorials

Table of Contents

• Background
• Quick start
• Requirements

Background

Cloud-Native Architecture Redis Connect has a cloud-native and shared-nothing architecture which allows any node to operate stand-alone or as a cluster member. Its platform-agnostic and lightweight design requires minimal infrastructure and avoids complex dependencies on 3rd-party platforms. All you need is Redis Enterprise.
	Multi-Tenancy | Partitioning | Linear Scalability Redis Connect manages multi-tenant replication pipelines. A pipeline from source to sink is known as a job. Jobs with different source types can be collocated without becoming noisy neighbors. Jobs can be partitioned for linear scalability across one or more cluster nodes.
High-Availability | Recovery Redis Connect jobs update their checkpoint upon each committed changed-data event within a transactional scope. In the event of a node failure or network split, in-flight jobs will fail over to another node and seamlessly begin replication from the last committed checkpoint. Data is not lost, and order is preserved. Redis Connect works in container orchestration environments such as Kubernetes and OpenShift.
	Custom Transformations Redis Connect jobs support user-defined business logic. You can create custom workflows that include user-defined stages for proprietary business rules, custom transformations, de-tokenization, and more. You can also extend the supported list of target sinks.
REST API | CLI | Swagger UI Redis Connect is entirely data-driven and relies on Redis Enterprise as its metadata store. You can configure, start, stop, migrate, and restart jobs using the built-in REST API and interactive CLI. Redis Connect also exposes a Swagger UI to simplify endpoint discovery and operational experience.
	Enterprise-Grade Security Redis Connect jobs are stateless, so changed-data events are always in-transit. Redis Connect benefits from Redis Enterprise’s security, including RBAC, TLS, and more. Credentials, secrets, and trust-store passwords are never stored in Redis Connect; these secrets can be dynamically rotated with minimal disruption to the replication pipeline. Vault integration is supported.

Requirements

Minimum production hardware requirements

• 1 GB of RAM
• 4 CPU cores
• 20 GB of disk space
• 1 Gbps network

Runtime requirements

• JRE 11+ (JRE 17+ version 0.10.7 onwards) e.g. Azul OpenJDK

Quick Start

You can run Redis Connect as a container or by downloading the code and running in your environment of choice.

Docker

You can run and deploy Redis Connect using the Redis Connect Docker image.

Download

Download the latest release for Linux or Windows and unarchive redis-connect-<version>.<build>.[tar.gz|zip] archive

The following subdirectories will be extracted under /redis-connect:

• bin – Startup scripts
• lib – Dependencies
• config – Credentials property files, jobmanager.properties, and job-config (JSON) examples
• extlib – Custom/external dependencies (e.g., custom stages, source-database drivers, etc.)

Launch Redis Connect

Redis Connect includes scripts for launching a single instance. You can run the scripts as follows:

On Linux

redis-connect/bin$ ./redisconnect.sh    
-------------------------------
Redis Connect startup script.
*******************************
Please ensure that the value of REDISCONNECT_JOB_MANAGER_CONFIG_PATH points to the correct jobmanager.properties in /home/viragtripathi/qa/vm/redis-connect/bin/redisconnect.conf before executing any of the options below
Check the value of redis.connection.url and credentials.dir.path in jobmanager.properties e.g.
redis.connection.url=redis://redis-19836.c9.us-east-1-2.ec2.cloud.redislabs.com:19836
credentials.dir.path=/var/secrets/redis
*******************************
Usage: [-h|cli|start]
options:
-h: Print this help message and exit.
cli: init Redis Connect CLI
start: init Redis Connect Instance (Cluster Member)
-------------------------------

On Windows

redis-connect\bin> redisconnect.bat

Prerequisite Configuration ❗
Update credentials.dir.path and redis.connection.url in /config/jobmanager.properties Example - jobmanager.properties

Starting an instance

Linux:

redis-connect/bin$ ./redisconnect.sh start

Windows:

redis-connect\bin> redisconnect.bat start

Swagger UI

Redis Connect Swagger UI is available on port 8282 by default. If you're running locally, you can point your browser to http://localhost:8282/swagger-ui/index.html

For quick start, use 'cdc_job' as jobName

Create Job Configuration - /connect/api/vi/job/config/{jobName}
For quick start, use a sample job configuration: DB2, Files, Gemfire, MongoDB, MySQL, Oracle, PostgreSQL, SQL Server and VERTICA

Prerequisite Configuration ❗
Update credentialsDirectoryPath, databaseURL, database.dbname (database.names for SQL Server), database.hostname, database.port and columns within sample job configuration for source and target, where applicable.

Start Job - /connect/api/vi/job/transition/start/{jobName}/{jobType}
For quick start, use 'stream' as jobType

Confirm Job Claim - /connect/api/vi/jobs/claim/{jobStatus}
For quick start, use 'all' as jobStatus

Once you've configured a job, try inserting some records into the source database. Then confirm that they have arrived in Redis.

Copyright

Redis Connect is developed by Redis, Inc. Copyright (C) 2023 Redis, Inc.

redis-connect-dist's Issues

Grafana dashboard for RedisCDC

If there is an example dashboard for RedisCDC then it's json definition should be provided in this repo.

Oracle connector

l'm interested in the redis connector for oracle but don't see any read.me doc or setup instructions.Will it be available soon?

JobOwner key not updated on Start RedisCDC for MSSQL

Investigate including JAXB-runtime jar

Using MacOS with both JRE 11 and 14.

Scenario:

1. Start the RedisCDC JobManager with ./startup.sh.
2. The following ClassNotFoundException is encountered.

javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.
 - with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory]
	at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:232)
	at javax.xml.bind.ContextFinder.find(ContextFinder.java:375)
	at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:691)
	at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:632)
	at com.ivoyant.cdc.mapper.MapperProvider.processMapperConfig(MapperProvider.java:36)
	at com.ivoyant.cdc.mapper.MapperProvider.<init>(MapperProvider.java:29)
	at com.ivoyant.cdc.mapper.MapperProvider.<clinit>(MapperProvider.java:22)
	at com.ivoyant.cdc.connector.debezium.producers.RDBEventProducer.init(RDBEventProducer.java:84)
	at com.ivoyant.cdc.connector.redis.LJobClaimerImpl.claimJob(LJobClaimerImpl.java:54)
	at com.ivoyant.cdc.connector.redis.JobClaimerImpl.run(JobClaimerImpl.java:111)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at javax.xml.bind.ServiceLoaderUtil.nullSafeLoadClass(ServiceLoaderUtil.java:92)
	at javax.xml.bind.ServiceLoaderUtil.safeLoadClass(ServiceLoaderUtil.java:125)
	at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:230)
	... 15 more

Potential Workaround:

• Adding jaxb-runtime-2.3.0.jar to the RedisCDC lib/ directory resolved the ClassNotFoundException.
• Per this link: jaxb-api is required for compilation, jaxb-runtime is required at runtime.

Windows version can't create a job due to credentialsFilePath

Hello,
I successfully started Redis Connect in my Windows Machine, but when I try to create a job for Oracle integration through API, it throws an error:

"ERROR: Failed to save configuration for JobName: cdc_job Validation Failure: Invalid CredentialsFilePath: C:\credentialscdc".

• The equivalent curl command is listed below:

curl -X POST "http://localhost:8282/connect/api/v1/job/config/cdc_job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@cdc_job.json;type=application/json"

• The credential file exists in credentials path:

• "File" parameter has a JSON file like this:

Link to local SwaggerUI is incorrect

Swagger UI
The [Redis Connect Swagger UI](
https://redis-field-engineering.github.io/redis-connect-api-docs => https://redis-field-engineering.github.io/redis-connect-api-docs
) is available on port 8282 by default.

If you're running locally, you can point your browser to
http://localhost:8282/swagger-ui/index.html => https://github.com/redis-field-engineering/redis-connect-dist/blob/main

As you can see the second href points to github and not to the local swagger

Composite Target key Update is failing on stream transaction

When connecting oracle with redis connect. Table have composite primary key, so targetKey was true for multiple fields in a table.

During stream transaction, update is throwing an error stating keyset cannot be serialized to String.