splitio/split-synchronizer

Golang agent for Split SDKs

Split Synchronizer

Overview

Split Synchronizer

By default, Split’s SDKs keep segment and feature flag data synchronized as users navigate across disparate systems, treatments, and conditions. However, some languages, do not have a native capability to keep a shared local cache of this data to properly serve treatments. For these cases, we built the Split Synchronizer service.

This tool coordinates the sending and receiving of data to a remote datastore that all of your processes can share to pull data for the evaluation of treatments. Out of the box, Split supports Redis as a remote datastore, and so the Split Synchronizer uses Redis as the cache for your SDKs when evaluating treatments. It also posts impression and event data and metrics generated by the SDKs back to Split’s servers, for exposure in the web console or sending to the data integration of your choice. The Synchronizer service runs as a standalone process in dedicated or shared servers and it does not affect the performance of your code, or Split’s SDKs.

Split Proxy

The Split Proxy enables you to deploy a service in your own infrastructure that behaves like Split's servers and is used by both server-side and client-side SDKs to synchronize the flags without connecting to Split's actual backend directly.

This tool reduces connection latencies from the SDKs to the Split server to the SDKs transparently, and when a single connection is required from a private network to the outside for security reasons.

Compatibility

Split Synchronizer supports Go version 1.18 or higher.

Getting started

Below is a simple example that describes the instantiation of Split Synchronizer:

Usage via Go

1. Install dependencies via go build || go mod vendor
2. Then, execute go run main.go -apikey "<YOUR_SDK_KEY>" -redis-host "<YOUR_REDIS_HOST>" -redis-port <YOUR_REDIS_PORT> -redis-prefix "<YOUR_PREFIX>"

Docker

1. You can pull the Docker image from Docker Hub and run it into your container environment.

docker pull splitsoftware/split-synchronizer:latest

2. Run the image:

docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_APIKEY=<YOUR_SDK_KEY> \
 -e SPLIT_SYNC_REDIS_HOST=<YOUR_REDIS_HOST> \
 -e SPLIT_SYNC_REDIS_PORT=<YOUR_REDIS_PORT> \
 -e SPLIT_SYNC_REDIS_PREFIX=<YOUR_PREFIX> \
 splitsoftware/split-synchronizer

Please refer to our official docs to learn about all the functionality provided by Split Synchronizer and this doc for Split Proxy.

Submitting issues

The Split team monitors all issues submitted to this issue tracker. We encourage you to use this issue tracker to submit any bug reports, feedback, and feature enhancements. We'll do our best to respond in a timely manner.

Contributing

Please see Contributors Guide to find all you need to submit a Pull Request (PR).

License

Licensed under the Apache License, Version 2.0. See: Apache License.

About Split

Split is the leading Feature Delivery Platform for engineering teams that want to confidently deploy features as fast as they can develop them. Split’s fine-grained management, real-time monitoring, and data-driven experimentation ensure that new features will improve the customer experience without breaking or degrading performance. Companies like Twilio, Salesforce, GoDaddy and WePay trust Split to power their feature delivery.

To learn more about Split, contact [email protected], or get started with feature flags for free at https://www.split.io/signup.

Split has built and maintains SDKs for:

• .NET Github Docs
• Android Github Docs
• Angular Github Docs
• GO Github Docs
• iOS Github Docs
• Java Github Docs
• JavaScript Github Docs
• JavaScript for Browser Github Docs
• Node Github Docs
• PHP Github Docs
• Python Github Docs
• React Github Docs
• React Native Github Docs
• Redux Github Docs
• Ruby Github Docs

For a comprehensive list of open source projects visit our Github page.

Learn more about Split:

Visit split.io/product for an overview of Split, or visit our documentation at help.split.io for more detailed information.

split-synchronizer's Issues

Feature Request: Configuration Report

As a user of Split Synchronizer
I would like a panel on the Admin Dashboard that reflects the current configuration values the Synchronizer is running with
So that I may be confident that runtime configuration is being properly passed to the Synchronizer
and so that I may know current configuration to determine how to tune the application in a production setting that may be experiencing performance issues.

Proxy and Redis Cache

I see the latest versions that proxy mode was split out. Does Synchronizer currently support both redis and proxy?
My ask is instead of having an sdk connect directly to redis, I would like to have the sdk connect to synchronizer which then fetches from redis as opposed to split cloud.
Is this something that can be done currently?

Upgrade version to 5.0.2 docker image error

We wanted to upgrade from 4.0.2 to 5.0.2 (using djengo_split and split_client sdk)
On local machine the only change needed is updating the version and changing the ENV VAR from SPLIT_SYNC_API_KEY -> SPLIT_SYNC_APIKEY.
And it works!

On AWS the same change result this errors. (the API KEY didn't change)

Split-Sync - ERROR - 2022/02/09 10:23:38 client.go:109: GET method: [https://auth.split.io/api/auth] Status Code: 400 - 400 Bad Request
Split-Sync - ERROR - 2022/02/09 10:23:38 auth.go:29: Error while authenticating for streaming 400 Bad Request
Split-Sync - ERROR - 2022/02/09 10:23:38 manager.go:200: non retryable error in streaming subsystem. Switching to polling until next SDK initialization
Split-Sync - ERROR - 2022/02/09 10:26:38 threshold.go:110: counter 'Segments' has timed out with tolerance=180s
Split-Sync - INFO - 2022/02/09 10:50:47 runtime.go:98:
* Starting graceful shutdown
Split-Sync - INFO - 2022/02/09 10:50:47 runtime.go:99:  * Waiting goroutines stop
Split-Sync - INFO - 2022/02/09 10:50:47 manager.go:169: Stopping all synchronization tasks
Split-Sync - INFO - 2022/02/09 10:50:47 runtime.go:111:  * Shutdown complete - see you soon!

Can you help me please?

Installer hardcodes /usr/local/bin

It should be possible to choose a location for the binary for situations where that destination is not desired.

Synchronizer entrypoint doesn't pick cli flags

When running Synchronizer on Docker you can't send the cli flags

flags := &CliFlags{
		ConfigFile:             flag.String("config", "", "a configuration file"),
		WriteDefaultConfigFile: flag.String("write-default-config", "", "write a default configuration file"),
		VersionInfo:            flag.Bool("version", false, "Print the version"),
		RawConfig:              MakeCliArgMapFor(definition),
	}

If I pass an argument to the container as per documentation, it doesn't work as expected

docker run splitsoftware/split-synchronizer -version
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

     __      ____        _ _ _
    / /__   / ___| _ __ | (_) |_
   / / \ \  \___ \| '_ \| | | __|
   \ \  \ \  ___) | |_) | | | |_
    \_\ / / |____/| .__/|_|_|\__|
       /_/        |_|



Split Synchronizer - Version: 5.4.2 (2fbf91c)
Split-Sync - ERROR - 2023/11/13 14:21:41 main.go:73: Failed to initialize the split sync:  initialization error: error parsing client key from provided SDK key: apikey too short

If I overwrite the entrypoint and run the command with arguments, it does work

docker run --entrypoint /bin/bash splitsoftware/split-synchronizer -c "split-sync -version"
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

     __      ____        _ _ _
    / /__   / ___| _ __ | (_) |_
   / / \ \  \___ \| '_ \| | | __|
   \ \  \ \  ___) | |_) | | | |_
    \_\ / / |____/| .__/|_|_|\__|
       /_/        |_|



Split Synchronizer - Version: 5.4.2 (2fbf91c)

The /info/config path leaks the Redis password

As can be seen on the image, the path /info/config shows the redis password. The apikey is sanitised.

[Question] Configure SplitProxy with HTTPs ?

I would like my clients to use HTTPS instead of HTTP to connect to split_proxy, but I can't find documentation about it.

Feature Idea - Prometheus metrics

It will be very good to expose metrics from /admin/dashboard/stats (and the best from other monitoring endpoints too) on another endpoint (/metrics) in prometheus format.

Then prometheus server can scrape those metrics and with Grafana create dashboard and alerts.

Empty /tmp/split-agent.log file

Hello, I'm running the Synchronizer with the following docker command:

docker run --rm --name split-synchronizer-proxy \
 -p 3000:3000 \
 -p 3010:3010 \
 -e SPLIT_SYNC_API_KEY="MY SPLIT API KEY HERE" \
 -e SPLIT_SYNC_PROXY="on" \
 -e SPLIT_SYNC_LOG_VERBOSE=true \
 -e SPLIT_SYNC_LOG_DEBUG=true \
 -e SPLIT_SYNC_LOG_STDOUT=true \
 -e SPLIT_SYNC_PROXY_SDK_APIKEYS="MY PROXY KEYS HERE" \
 splitsoftware/split-synchronizer

After successfully requesting a treatment from an app running the Split Java SDK client, I execd into the docker container running the Synchronizer and expected to see some logs in /tmp/split-agent.log file, however, the file was empty. Are there any other steps necessary to get more logging to happen?

Support Redis Cluster

Your PHP SDK supports redis cluster, but it appears the synchronizer does not.
https://github.com/splitio/split-synchronizer/blob/master/splitio/storage/redis/client.go#L32

Needs to use NewClusterClient
https://godoc.org/github.com/go-redis/redis#NewClusterClient

synchronizer redis failed reconnection

We have faced with Split-synchronizer issue in Prod using AWS Elasticache Redis 3.2.10 (single mode) when Amazon try to recreate node:

Recovering cache nodes 0001

note: ip address of redis endpoint preserve
https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/ClientConfig.DNS.html
https://forums.aws.amazon.com/thread.jspa?threadID=140869

But in split-synchronizer logs we were seeing Error log for several minutes till we have redeployed split-synchronizer service:
reconnection not handled properly

December 26th 2018, 10:10:08.284 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:10:08 impression.go:35: dial tcp 10.10.100.100:6379: i/o timeout
December 26th 2018, 10:10:03.284 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:10:03 postimpressions.go:63: Error Retrieving
...
December 26th 2018, 10:08:34.282 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:08:34 impression.go:35: dial tcp 10.10.100.100:6379: getsockopt: no route to host
...
December 26th 2018, 10:08:31.281 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:08:31 postimpressions.go:63: Error Retrieving
...
December 26th 2018, 10:08:13.281 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:08:13 postimpressions.go:63: Error Retrieving
...
December 26th 2018, 10:08:01.282 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:08:01 impression.go:35: dial tcp 10.10.100.100:6379: getsockopt: no route to host
...
December 26th 2018, 10:07:41.320 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:07:41 metrics.go:31: read tcp 172.17.0.7:40692->10.10.100.100:6379: i/o timeout
...
December 26th 2018, 10:07:36.321 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:07:36 metrics.go:70: dial tcp 10.10.100.100:6379: i/o timeout
...
December 26th 2018, 10:07:35.365 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:07:35 postimpressions.go:63: Error Retrieving
...
December 26th 2018, 10:07:29.048 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:07:29 fetchsplits.go:58: Error saving till value into storage adapter. read tcp 172.17.0.7:40682->10.10.100.100:6379: i/o timeout
...
December 26th 2018, 10:07:15.364 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:07:15 impression.go:35: read tcp 172.17.0.7:40686->10.10.100.100:6379: i/o timeout
...
December 26th 2018, 10:07:05.364 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:07:05 impression.go:35: read tcp 172.17.0.7:40688->10.10.100.100:6379: i/o timeout
December 26th 2018, 10:07:05.364 | SPLITIO-AGENT \| ERROR: 2018/12/26 10:07:05 postimpressions.go:63: Error Retrieving

I've tried these steps with local redis, but it was not reproduced :(

1)  run redis:
docker run -d --name test-redis -p 6379:6379 redis:3
2) run sync:
docker run -ti --rm --name test-splitio -e SPLIT_SYNC_LOG_STDOUT=on -e SPLIT_SYNC_API_KEY=xxx -e SPLIT_SYNC_REDIS_HOST=<HOST_IP> splitsoftware/split-synchronizer:1.7.1
3) stop redis:
docker rm -f test-redis
4) wait when ERRORS appear
5) back redis container again 
docker run -d --name test-redis -p 6379:6379 redis:3
6) ERRORS disappeared; sync reconnected properly

May be you have some idea?

Connection refused error when initialising Synchronizer but curl works fine

Split Synchronizer - Version: 5.4.0

Unable to use synchronizer service in kubernetes setup.

I'm getting connect-refused error from synchronizer container but if I exec inside the container and do a curl call to same URL it passes with HTTP 200 status code.

===Log snippet from container===

➜  ~/code git:(synchronizer_setup) $ kubectl logs -f split-synchronizer-87856ddf8-v7v9c          

     __      ____        _ _ _
    / /__   / ___| _ __ | (_) |_
   / / \ \  \___ \| '_ \| | | __|
   \ \  \ \  ___) | |_) | | | |_
    \_\ / / |____/| .__/|_|_|\__|
       /_/        |_|

       

Split Synchronizer - Version: 5.4.0 (18b8201) 
Split-Sync - DEBUG - 2023/10/27 00:14:48 client.go:60: [GET]  https://sdk.split.io/api/splitChanges?since=1698365688257
Split-Sync - DEBUG - 2023/10/27 00:14:48 client.go:64: Authorization [ApiKey]:  <api-key>
Split-Sync - DEBUG - 2023/10/27 00:14:48 client.go:73: Headers: map[Accept-Encoding:[gzip] Content-Type:[application/json] Splitsdkmachineip:[<ip>] Splitsdkmachinename:[<ip>] Splitsdkversion:[SplitSyncProducerMode-5.4.0]]
Split-Sync - ERROR - 2023/10/27 00:14:48 client.go:79: Error requesting data to API:  https://sdk.split.io/api/splitChanges?since=1698365688257 Get "https://sdk.split.io/api/splitChanges?since=1698365688257": dial tcp 151.101.3.9:443: connect: connection refused
Split-Sync - ERROR - 2023/10/27 00:14:48 http_fetchers.go:55: Error fetching split changes  Get "https://sdk.split.io/api/splitChanges?since=1698365688257": dial tcp 151.101.3.9:443: connect: connection refused
Split-Sync - ERROR - 2023/10/27 00:14:48 main.go:73: Failed to initialize the split sync:  initialization error: invalid SDK key

===Output from CURL===

➜  ~/code git:(synchronizer_setup) $ kubectl exec -it split-synchronizer-68d9d75f75-c99pk -- bash
split-synchronizer-68d9d75f75-c99pk:/# curl -i  https://sdk.split.io/api/splitChanges?since=1698365757211 -H "Authorization: Bearer $SPLIT_SYNC_APIKEY"
HTTP/2 200 
content-type: application/json; charset=utf-8
..<lot of other headers>...
date: Fri, 27 Oct 2023 00:39:33 GMT
content-length: 56

{"till":1698365757211,"since":1698365757211,"splits":[]}split-synchronizer-68d9d75f75-c99pk:/# 
exit

split-sync impression errors

We have been receiving the following alerts from split-synchronizer today. Can someone explain why this is happening and what can be done to remediate it? Do I need to configure a timeout?

split-synchronizer version - 5.0.2

Split-Sync - ERROR - 2022/05/05 17:54:02 pipelined.go:265: [pipelined/impressions] error posting: Post "https://events.split.io/api/testImpressions/bulk": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Split-Sync - ERROR - 2022/05/05 17:54:02 pipelined.go:270: [pipelined/impressions] bad status code when sinking data: 400
Split-Sync - ERROR - 2022/05/05 17:54:02 pipelined.go:270: [pipelined/impressions] bad status code when sinking data: 400
Split-Sync - ERROR - 2022/05/05 17:54:03 pipelined.go:265: [pipelined/impressions] error posting: Post "https://events.split.io/api/testImpressions/bulk": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Split-Sync - ERROR - 2022/05/05 17:54:03 pipelined.go:270: [pipelined/impressions] bad status code when sinking data: 400
Split-Sync - ERROR - 2022/05/05 17:54:03 pipelined.go:270: [pipelined/impressions] bad status code when sinking data: 400

Environment variables comparison between versions

We recently updated the synchronizer from 2.4.1 to 5.0.2 and noticed that many environment variables have changed.

Can someone please verify if the environment variables used in 2.4.1 are equivalent to the ones I have in 5.0.2?

2.4.1	5.0.2
SPLIT_SYNC_IMPRESSIONS_POST_RATE	SPLIT_SYNC_IMPRESSIONS_POST_SIZE
SPLIT_SYNC_IMPRESSIONS_PER_POST	SPLIT_SYNC_IMPRESSIONS_PROCESS_BATCH_SIZE
SPLIT_SYNC_IMPRESSIONS_THREADS	SPLIT_SYNC_IMPRESSIONS_PROCESS_CONCURRENCY
SPLIT_SYNC_EVENTS_THREADS	SPLIT_SYNC_EVENTS_PROCESS_CONCURRENCY

Unable to pass a username to Redis config

I've looked at the available environment variables, and we can only provide those :

SPLIT_SYNC_REDIS_HOST
SPLIT_SYNC_REDIS_PORT
SPLIT_SYNC_REDIS_DB
SPLIT_SYNC_REDIS_PASS

We need to be able to specify a username as we want to use ACL in Redis 6.

impressionsPostRate cannot be reduced

The Runbook seems to say that one solution for improving impressions throughput is to decrease the impressionsPostRate

In Example 2 and 3, they give examples of decreasing the rate to 30 seconds or 15 seconds.

However, when attempting to configure it to 30 seconds, I get the following error:

Split Synchronizer - Initialization error: ImpressionsPostRate must be >= 60. Actual is: 30

Seems like the documentation does not match the code.

Please release 4.0.3 to docker hub

I believe that the current latest tag on docker hub may be the 4.0.3 image, but there is no 4.0.3 tag. I would like to be able to use 4.0.3 without using the latest tag, since latest could change without warning with future releases.

Feature Idea - Monitor the synchronizer with something like StatsD

The admin dashboard, and the metrics apis available (like /admin/metrics) are interesting to help us investigate the current status synchronizer, but they are not very useful for monitoring the system over time. They don't have a historical view of things like queue size or lambdas, and it also requires you to remember to check that dashboard.

It would be good to have a way to pipe these metrics out of the synchronizer, and into a tool designed for monitoring and alerting, such as Datadog or Prometheus. We would like to monitor the queue size and trigger alerts based on queue size or bad lambda values.

I can think of a few good options for this, what do others think?

• adding a StatsD client metric exporter to the synchronizer (like the one built by etsy or datadog). I know it's easy to pass statsd into Datadog, not sure about Prometheus.
• Creating a custom Datadog Integration (which would probably call the existing admin REST api and convert it to datadog metrics. Downside is it requires separate maintenance and is datadog specific, but might provide a better experience for customers using datadog like us.

Load spikes in Redis when opening dashboard

Synchronizer currently uses "KEYS" Redis operation, which is strongly discouraged for production workloads. See https://redis.io/commands/keys/

Currently it is used here:

split-synchronizer/splitio/admin/controllers/helpers.go

Line 82 in 1cd638d

keys := segmentStorage.Keys(name)

We are observing big load spikes in our Redis (which is not only used by Split.io) whenever anyone opens the split.io dashboard, and KEYS queries clearly flare up in our metrics.

At the very least KEYS should be replaced with SCAN, but ideally entire storage shouldn't be scanned at all - maybe some lookup maps could be utilized?

[SCALING] Deployment with multiple pod behavior

Recently, we have been implementing split synchronizer within our K8s environment, and we have encountered a scenario where we need some clarification.

Could you please provide some insights on how the split synchronizer works when there are multiple instances running concurrently in a K8s deployment? Specifically, we are interested in understanding the following:

1. How does the split synchronizer handle synchronization conflicts when multiple instances attempt to modify the same resource simultaneously (e.g., updating redis key)?
2. Are there any limitations or known issues related to deploying multiple instances of the split synchronizer within a K8s cluster?

Thanks!

Feature suggestion: Modify the shutdown message

There's two features that would be nice to have.

First one would be to customize the shutdown message. Right now we're getting this *[IMPORTANT]* @here Shutting Split-Sync down - see you soon! in Slack and a lot of people are being alerted by @here which is not something we want.

Second feature that would be nice to have is to have slack alerts for application startup, right now we only get alerts when it shuts down but not when it starts up.

v5.0: Breaking change: api-key becomes apikey

in versions through 4.x, the environment variable for the api key was 'SPLIT_SYNC_API_KEY' but apparently it changes to 'SPLIT_SYNC_APIKEY' in 5.x.

The docs here on github still all reference the old name.

But running the new split-synchronizer docker container 5.0.0 with the old value gives:
Split-Sync - ERROR - 2021/11/22 19:36:52 main.go:72: Failed to initialize the split sync: initialization error: error parsing client key from provided apikey: apikey too short

If I rename the variable to SPLIT_SYNC_APIKEY then 5.0.0 starts up fine.

Error decoding event JSON invalid character

Running the split synchronizer using docker in a kubernetes cluster. After some time running the service, we get a number of logs showing the following error message:

split-synchronizer-preprod-7b8d94966-c2sk5 split-synchronizer SPLITIO-AGENT  - ERROR - 2021/09/01 16:48:46 impressions.go:142: Error decoding event JSON invalid character 'a' looking for beginning of value

Unfortunately, we don't have much of an idea why this error happens or how to fix it.

Split Sync version 4.0.2

Warning in logs when synchronizer updates empty segment

Just noticed the following line in splitio-synchronizer logs:

SPLITIO-AGENT | WARNING: 2019/11/04 11:45:18 common.go:221: Error fetching last update for segment <<segment name>>

Opened that segment in Split.io dashboard and it turns out that it is just an empty segment (exists but contains no keys).
I understand that an empty segment looks a bit strange, but I think it's a normal situation, no need to issue Warning in logs.

splitio-synchronizer is version 2.5.2, running via an official docker image.

Net::ReadTimeout with #<TCPSocket:(closed)> Errors

Hi,

We are running the split proxy in our K8s clusters and are seeing sporadic timeouts between the Ruby SDK and the proxy. These occur at the 5s mark, which I see is the default http_timeout_config, however I'm curious if anyone has run into this issue before and has been able to solve it at the root cause?

Our stack is EKS/Istio/Two deployments one for admin and one for split proxy

Multiple environments

Does the Split synchronizer support multiple environments or should there be a synchronizer per environment?

TLS support

We need to connect to Redis over a TLS connection. This is a supported option in the Redis client library github.com/go-redis/redis that Synchronizer uses, but this isn't exposed as a configuration paramter.

My suggestion would be to add it to the SPLIT_SYNC_ADVANCED_PARAMETERS as -redis-tls=true.

See the PR I've created - #58