faros-ai / airbyte-connectors Goto Github PK
View Code? Open in Web Editor NEWAirbyte connectors (sources & destinations) + Airbyte CDK for JavaScript/TypeScript
License: Apache License 2.0
Airbyte connectors (sources & destinations) + Airbyte CDK for JavaScript/TypeScript
License: Apache License 2.0
Please see if this draft is useful to work off of #122
Add converters for Jenkins streams in Faros Destination
Resolve locations using Faros /geocoding/lookup
API - https://dev.api.faros.ai/docs.html#Geocoding.
Example from org import feed - https://github.com/faros-ai/feeds/blob/master/feeds/org/org-import-feed/src/feed.ts#L346
We would like to build a connecter and converters for OpsGenie. To start, this should be a similar connector to that of PagerDuty as we are interested in incident data.
Data to retrieve:
Note: Be aware of rate-limiting when using node.js API client as it does not appear to handle 429 responses
Applies to Faros Community Edition
How to reproduce:
the reset fails with the following error
2022-04-06 17:04:22 ERROR i.a.c.i.LineGobbler(voidCall):82 - Failed to reset vcs_PullRequest data for origin mygithubsrc: [{"extensions":{"path":"$","code":"constraint-violation"},"message":"Foreign key violation. update or delete on table \"vcs_PullRequest\" violates foreign key constraint \"vcs_PullRequestComment_pullRequest_fkey\" on table \"vcs_PullRequestComment\""}]
which is likely because the reset does not delete the data from the removed streams.
Workaround once in that situation: update source schema, reset, delete unwanted streams.
Probably a bug in pagination?
Add a Harness source and converter based on Harness feed
Pull same data as Okta connector (org_*)
Currently its hardcoded to 20000, but some issues have descriptions that cause Squadcast API responses to go past that length. Make this a configurable source option.
Write a source and Faros converter based on the Changeset feed
Please also go through the sources and add descriptions for all the fields in the spec.json files that don’t currently have descriptions.
cc: @cjwooo
In the original Bitbucket feed we write out repository information as follows - https://github.com/faros-ai/feeds/blob/master/feeds/vcs/bitbucket-feed/src/feed.ts#L143-L154
We need to add the same object into Butbucket source as well.
We need to write cicd_BuildCommitAssociation
objects from jenkins when available. See feed as reference on how this is done.
Auth can be tricky, first iteration can just have basic auth
Pull vcs objects from Azure
https://docs.microsoft.com/en-us/rest/api/azure/devops/git/?view=azure-devops-rest-6.0
Hi @huongtn!
As it was discussed with you, please add the start_date
argument to the following connectors (where applicable):
Merged:
Not Merged:
"start_date": {
"type": "string",
"title": "Start Date",
"description": "The date from which you'd like to replicate data for GitHub in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated",
"examples": ["2021-03-01T00:00:00Z"],
"pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$"
}
Example of this argument in GitHub -
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-github/source_github/spec.json#L65-L71
Also, attaching here the initial description of the request from Matthew:
https://devcube-group.slack.com/archives/C01U644P2P4/p1647962158678039
Running build from the root of the project fails with tsc
> [email protected] build /airbyte-connectors/faros-airbyte-cdk
> tsc -p src
src/destinations/destination-runner.ts:87:19 - error TS2339: Property 'message' does not exist on type 'unknown'.
87 e.message ? e.message : e
~~~~~~~
Should be similar to Circle CI (Will to confirm)
A good default is to do the same as our GitHub connector: create a project and a board for each repository. Tasks (issues) should then link to those.
https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/airbyte-faros-destination/src/converters/github/common.ts#L92
On Faros CE, Buildkite connections cannot write cicd_BuildCommitAssociation entries because its converters need to properly set the uid, somewhat similar to this PR:
146cb16#diff-78e89bf564fd6dbf0d497de07baf220468bbee05a9bc73996c698f136970691aL47
This shall be fixed for all converters writing that entity.
2022-04-27 22:59:27 destination > Error processing input: Failed to write cicd_BuildCommitAssociation record: [{"extensions":{"path":"$.selectionSet.insert_cicd_BuildCommitAssociation_one.args.object[0].vcs_Commit.data.vcs_Repository.data","code":"constraint-violation"},"message":"Not-NULL violation. null value in column \"uid\" of relation \"vcs_Repository\" violates not-null constraint"}]
A good default is to do the same as our GitHub connector: create a project and a board for each repository. Tasks (issues) should then link to those.
https://github.com/faros-ai/airbyte-connectors/blob/main/destinations/airbyte-faros-destination/src/converters/github/common.ts#L92
Create Victorops source and converter based on the Victorops feed.
It's a bit convoluted, to work around API quirks, apologies in advance.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.