valmi-io/valmi-activation

⚡ valmi.io reverse ETL (data activation) is the open source ( OSS ) data activation platform to load data from warehouses into Webhooks and SaaS tools like Klaviyo, Facebook Ads, Salesforce, Braze etc. Valmi.io Customer Data Platform (CDP) helps track and ingest user activity events from websites, shopify, serverside events. https://cloud.valmi.io

valmi-activation's Issues

Task: Sync runs api output change

Currently there is no order for displaying metrics in sync runs page
Need to add display_order for each metric

Old format:

{
  "src": {
    "total$$0": 4935,
    "new$$0": 4935,
    "success": 4935,
    "valid$$0": 4935
  },
  "dest": {
    "succeeded": 290
  }
}

New format:

{
  "src": {
    "total$$0": { "value": 4935, "display_order": 0},
    "new$$0":  { "value": 4935, "display_order": 4},
    "success":  { "value": 4935, "display_order": 16},
    "valid$$0":  { "value": 4935, "display_order": 8},
  },
  "dest": {
    "succeeded":  { "value": 290, "display_order": 0},
  }
}

Hubspot upsert mode needs query for existence first and then insert/update accordingly. Also needs support for just update

Support Templated messages for Slack

without jinja templates , slack destination will be useless. It also means extending ValmiSink to contain templated messages in addition to mapping.

bug: duplicate runs on making a sync active:inactive

Bug: Long live access token not retrieving from fb when trying to authenticate while configuring facebook ads

bug: If one connector fails, other connector gets stuck.

Story: CI/CD pipeline

Connectors:

• Docker build for linux/amd64 and linux/arm64 platforms
• Add commands for building and pushing dockers images for connectors in Makefile

Valmi Activation:

• Makefile rules for building valmi-repo, valmi-dagit and valmi-dagster-daemon

Valmi App:

• Makefile rules

Valmi App Backend:

• Makefile rules

---

• Docker compose files for prod and dev environments

Story: Documentation

Connectors

• Hubspot
• Customer.io
• Slack
• Stripe
• Facebook Ads

Introduction Topics

• Intro
• Concepts
• Architecture
• Deployment

Story: Facebook Ads

• Oauth Testing
• Verify Facebook App
• Change the code to use new discover & write functionality

Story: Ability to build models in the app?

• hi I wanted to start by thanking the team for this great piece of software
• I was wondering if there is any plan to build models in valmi, i.e., ala grouparoo/census
• intention would be to build groups/segments/audiences based off models which can be rETL'd and also shared across the org
  • eg: marketing build a segment for LTV>100, which can also be targetted by product for a different product experience
• thanks!

Bug: Metrics are not finalised when a sync is aborted. The UI showed empty metrics as expected.

Stripe Finalise these issues

#TODO:

• finalize upserts and updates

• Handle errors along with rate limits - customize back-off behavior code

Task: CI/CD pipeline

• Remove .version files
• Add github actions

Task: Adding `run_end_at` column to `sync_runs` table

Bug: Dagster terminate run is failing

When will it occur?

• Update the sync's status from "active" to "inactive," resulting in the termination of any currently running jobs associated with valmi-activation.
• The failure of the Dagster client to terminate the job creates an endless loop of termination calls.

Why is this happening?

• The default Dagster TerminateRunPolicy is set to SAFE_TERMINATE, causing failure when no container is available for termination.
• Containers are self-terminated by examining the flag in syn_runs.extra.run_manager.

Story: Helm charts to deploy on k8s

Story: Slack Connector

• Template Messages
• Change the code to use new discover & write functionality
• Documentation

Bug: Sync runs got stopped when switching from active to Inactive

Story: Google Sheets

• Change the code to use new discover & write functionality
• Verify Google App Application

valmi-activation Repository Shutdown Exception

05/25/2023 03:21:38 PM INFO JobCreator requests::execute >>> {"query": "mutation ($repositoryLocationName: String!) {\n shutdownRepositoryLocation(repositoryLocationName: $repositoryLocationName) {\n __typename\n ... on PythonError {\n message\n }\n ... on RepositoryLocationNotFound {\n message\n }\n }\n}", "variables": {"repositoryLocationName": "valmi-repo"}}
05/25/2023 03:21:38 PM DEBUG JobCreator connectionpool::_new_conn Starting new HTTP connection (1): dagit:3000
05/25/2023 03:21:38 PM ERROR SyncRunner run_manager::_run Error while handling syncs in run manager
Traceback (most recent call last):
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 98, in _execute
return self._client.execute(gql(query), variable_values=variables)
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/client.py", line 403, in execute
return self.execute_sync(
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/client.py", line 220, in execute_sync
with self as session:
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/client.py", line 702, in enter
return self.connect_sync()
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/client.py", line 678, in connect_sync
self.transport.connect()
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/transport/requests.py", line 106, in connect
raise TransportAlreadyConnected("Transport is already connected")
gql.transport.exceptions.TransportAlreadyConnected: Transport is already connected

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/src/./orchestrator/run_manager.py", line 106, in _run
dagster_run_id = self.dc.submit_job_execution(
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 319, in submit_job_execution
return self._core_submit_execution(
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 149, in _core_submit_execution
pipeline_info_lst = self._get_repo_locations_and_names_with_pipeline(pipeline_name)
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 106, in _get_repo_locations_and_names_with_pipeline
res_data = self._execute(CLIENT_GET_REPO_LOCATIONS_NAMES_AND_PIPELINES_QUERY)
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 100, in _execute
raise DagsterGraphQLClientError(
dagster_graphql.client.utils.DagsterGraphQLClientError: Exception occured during execution of query

query {
repositoriesOrError {
__typename
... on RepositoryConnection {
nodes {
name
location {
name
}
pipelines {
name
}
}
}
... on PythonError {
message
}
}
}

with variables
None

05/25/2023 03:21:38 PM ERROR SyncRunner retry_decorators::wrapper Failure _run due to error: Exception occured during execution of query

query {
repositoriesOrError {
__typename
... on RepositoryConnection {
nodes {
name
location {
name
}
pipelines {
name
}
}
}
... on PythonError {
message
}
}
}

with variables
None

Bug: Source connector got stopped on ./valmi prod down and the resulting UI shows Success for intermediate results.

Syncs Api response.

[
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "7736700b-9609-4cdc-953f-4cae79dc8e77",
        "run_at": "2023-08-12T12:05:42.872152",
        "status": "running",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 92000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 66300,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 306
                            }
                        },
                        "emitted_at": 1691842378000
                    }
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 300
                                },
                                "chunk_id": 219,
                                "finished": false,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                }
            }
        },
        "run_time_args": null,
        "run_end_at": null
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "b037da8b-833a-42f7-97c5-c323e4b3701a",
        "run_at": "2023-08-12T11:58:30.637649",
        "status": "stopped",
        "metrics": {
            "src": {
                "succeeded": {
                    "value": 90900,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 52600,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 302
                            }
                        },
                        "emitted_at": 1691841862000
                    }
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 300
                                },
                                "chunk_id": 174,
                                "finished": false,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T12:05:40.844190"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "5d3706da-e213-4d27-91ab-c036916ece3a",
        "run_at": "2023-08-12T10:22:36.348108",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691839850000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T11:58:29.392701"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "4fc24fbd-965d-4fdf-91fc-ac035954f348",
        "run_at": "2023-08-12T08:47:15.037642",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691833978000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T10:22:35.073529"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "fd84da71-633f-42e2-9544-33bc947e16ba",
        "run_at": "2023-08-12T07:32:22.202988",
        "status": "stopped",
        "metrics": {
            "src": {
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691826490000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T08:47:13.480052"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "04aca962-a428-47c8-91ab-eae5a45ad394",
        "run_at": "2023-08-12T06:13:53.778872",
        "status": "stopped",
        "metrics": {
            "src": {
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691822039000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T07:32:20.962417"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "d9e8e850-d4a8-4093-b0df-f6f9951bc5fd",
        "run_at": "2023-08-12T04:44:42.612574",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691818126000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T06:13:51.787631"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "bfb1ad26-7478-443e-8bcd-03910403bab7",
        "run_at": "2023-08-12T03:11:57.037993",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691813318000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T04:44:40.819042"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "4f001824-d3a6-4bd2-8d21-c6ea2ebe2df5",
        "run_at": "2023-08-12T01:36:28.291449",
        "status": "stopped",
        "metrics": {
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691808098000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T03:11:55.166514"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "284de56d-2541-4a12-a19f-ed5a3d8dc12d",
        "run_at": "2023-08-12T00:05:37.547739",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691801646000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T01:36:26.813729"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "d36d1060-5806-4a3f-9fe0-f63e44ab1dad",
        "run_at": "2023-08-11T22:35:00.997702",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691796204000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T00:05:36.057233"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "14babb64-a8b4-4682-92c3-eac60f3d0c7c",
        "run_at": "2023-08-11T21:09:50.813134",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691790143000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T22:34:59.409940"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "b0193b98-d979-46ac-8123-a1e220e0e897",
        "run_at": "2023-08-11T19:35:19.264918",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691786181000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T21:09:48.764788"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "4fdca72e-6f0e-4de8-8765-70b15d8f8429",
        "run_at": "2023-08-11T18:00:08.094616",
        "status": "stopped",
        "metrics": {
            "src": {
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691780710000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T19:35:17.881522"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "11cc2feb-a063-416a-97eb-05cc3300f5d4",
        "run_at": "2023-08-11T16:36:24.129758",
        "status": "stopped",
        "metrics": {
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "src": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691773543000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T18:00:05.992098"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "1ed518d9-eefc-4aea-9539-93dcdeca4dc4",
        "run_at": "2023-08-11T15:08:04.803818",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691768969000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T16:36:22.249576"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "4ed455c3-fd85-454d-b643-e9b1b62f4995",
        "run_at": "2023-08-11T13:32:49.986874",
        "status": "stopped",
        "metrics": {
            "src": {
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691764637000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T15:08:03.285073"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "078a3be3-aab6-4e19-b2e1-c61a7cf782bb",
        "run_at": "2023-08-11T12:12:59.446682",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691757278000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T13:32:48.065335"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "93a14e7f-78e7-4fc7-945c-29796eb70554",
        "run_at": "2023-08-11T10:37:05.447218",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691754255000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T12:12:57.274907"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "67dea90b-8ce4-4f3b-9ebb-5756b590efdf",
        "run_at": "2023-08-11T09:01:08.931101",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691748500000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T10:37:04.009195"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "e90b950b-32c7-43ac-996e-201d42ca7140",
        "run_at": "2023-08-11T07:37:06.545194",
        "status": "stopped",
        "metrics": {
            "src": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691741271000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T09:01:07.277233"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "2f769413-2abd-4da5-8e8f-a25b9c73566e",
        "run_at": "2023-08-11T06:05:46.042595",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691737026000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T07:37:05.208067"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "a27707df-56a7-4102-af49-9998ee49fd20",
        "run_at": "2023-08-11T04:30:52.383389",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691732110000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T06:05:44.411593"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "9e845439-1b5f-4f7a-af36-2063c59c19cb",
        "run_at": "2023-08-11T02:55:18.467970",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691726456000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T04:30:50.338333"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "1dc717c9-661e-41f7-9ba2-31cc6c47a38d",
        "run_at": "2023-08-11T01:27:25.689896",
        "status": "stopped",
        "metrics": {
            "src": {
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691719659000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T02:55:16.821350"
    }
]```

Story - [valmi-app] bugs

• Slack - oAuth2 redirecting to login page.
• Login - Not able to login from 1 million sync page.
• Live events - logs - json formatting.
• Sync runs - logs - json formatting.
• Icons - update icons for Activate , Streams & Destinations in events - connections page.
• Logout - Logs out - when clicked on the sidebar routes after logged in from 1 million sync page.

Story - OAuth configurations

Story: Google Ads Connector

• Creating separate offline jobs for delete and create actions
• Implementing back-off for rate limit error
• Documentation

Story: Hubspot Connector

• Finalize upserts and updates - Check for existence before creating
• Change the code to use new discover & write functionality

Story: Jitsu Integration

Connector: Gong - Use Batching to Upsert and update records and use /v2/crm/request-status for success. Now using one request for each record.

Story: Customer IO

Issues

• x out of n in batch mode rejections to be handled & counted as failed

Enhancements

• Change the code to use new discover & write functionality

Story: On update mapping in Sync UI, trigger sync full_refresh

Story: Gong Connector

• Finalize Connector with just API keys - not Oauth

Story: Edit Sync API

Story: Logs collection for the UI display

• Collect rejected samples from sync runs and store them in intermediate store
• API call to retrieve rejected samples

Story: Send certain failed/rejected records on the destination back to the source and they need to be retried on the subsequent syncs

Story: Inject Checkpoint state to connectors

#TODO:

• on connector launch, pass stored checkpoint state.

Story: Separate Metric-Service into a different process : do it by passing a flag --metric_server embedded or --metric_server remote --metric_server_url http://xx/

Story: Start stop functionality for syncs

• Fast API end points in valmi activation
• Backend api points

bug: add privileged to docker- compose for docker-in-docker runs

Story: "valmi figures" for Customer-360

bug: mount metrics db for prod docker compose on a volume

Story: Stripe Connector

• Finalise upserts and updates
• Customise backoff & retry behavior
• Change the code to use new discover & write functionality

Story: Shopify Event Tracking App for valmi.io

Bug: snowflake mirror mode duplicates issue.

Snowflake mirror causes metrics and delivery issues, when there are duplicates.
Reproduce.

1. Sync once
2. Add a duplicate key
3. Now sync.
4. All the metrics are garbled because of the duplicate key.
   Also check in upsert mode,

Doc: what will happend on click on deployment link or looks like there is no deployment section

Story: Android push notifications connector

• Finalise Connector
• Documentation

Bug: Production dockers are not working

Story - Show Sample data for reverse ETL syncs

• UI call to retrieve rejected samples & successful samples

Run not found error

Story: Monitoring stack for valmi-activation

• view Logs, Traces and Metrics in New Relic

use dummy keys all across.

bug: making a sync inactive must first confirm abort from activation engine , then only set the app-backend to inactive. order matters