Comments (5)
I wonder if we could preprocess the organizations
config param with the graphql endpoint, like we do for users
and repos
, but do it in such a way that we end up with 2 lists: "real" organizations, and valid users (and implicitly, invalid input that is discarded, basically whatever was neither org nor user). I think we could manage this with a single graphql query passing the list as both orgs and users, and processing results+errors in the response.
Then:
- for specifically org streams (teams, team members, team roles), we use the "real_orgs" as input,
- for repo streams, we can use both validated orgs and users as inputs, changing the url as required so the tap runs transparently on both.
- And for invalid input, we just log a warning during preprocessing, like we currently do for repos and users: "invalid org name , discarding".
from tap-github.
@laurentS - Just to check here, it is desireable in this case to be able to pull streams for a "personal" org as if it were a "real" org?
If yes, and if there is a parent-child relationship, perhaps we could check a property of the org and skip/omit child streams that are non-applicable.
That would allow a hybrid of orgs and usernames to be provided in the organizations
list.
However, if we'd rather not accept personal usernames as orgs, then (again assuming we have a parent-child relationship) probably a similar check that results in hard failure when iterating on the organizations themselves would do the trick. "Error: Detected that 'laurents' is not a valid organization ID. Personal usernames are not allowed in the organizations list."
What do you think?
from tap-github.
My feeling is that github mixes users and orgs in the web app, to the point where behaviour is basically the same when listing an org's repo or a user's repos. Using the api, I get essentially the same results as well, whether I call github.com/orgs/meltano/repos
or github.com/users/laurents/repos
: a list of repos, so it is possible to follow up with the same child streams.
Taking a step back, the lack of consistency is somewhat annoying here. It is impossible to know if joeblog
in github.com/joeblog
or joeblog/superpackage
is a user or an org without hitting the api and possibly getting an error back (and crashing the tap, abandoning all further processing).
I think having to write code outside the tap to figure this out defeats the purpose of the tap, yet it is the behaviour we currently have (as we can pass users
or organizations
in the config file).
So I'd lean towards making it transparent in the tap as well, so that it is possible to pass "organizations": ["meltano", "laurents"]
as a config option, and the tap figures things out.
This would work for the repositories
stream and its children (which is my use case, and the only use case available in the tap right now), but if we were to add a stream to fetch data about the org itself using the orgs/ORG
endpoint, then we'd have a problem again, and we would need a different behaviour there. For consistency with the "symmetric" users
stream (if you pass "users": ["meltano"]
, it will skip it), we could use the graphql endpoint in a new Organizations
stream to filter out invalid ones.
What do you think?
@ericboucher any further thoughts?
from tap-github.
It is indeed quite annoying. As a first approach I think we could try to make it transparent and run successfully for repos. And log a more explicit warning on "Not Found" for the other streams. Eg. "meltano" was not found, are you sure this GitHub organization exists and it is not a user
?
from tap-github.
@laurentS sounds good, let's try something like this. Random thought - we could imagine another type of input called users_and_orgs
or users_or_orgs
that explicitly has this behavior.
from tap-github.
Related Issues (20)
- KeyError: `commit_timestamp` HOT 5
- Field `fetched_at` in stream `extra-metrics` can be formatted as a date-time string
- Releases stream has 10,000 record limit HOT 3
- The 'pull_number' field not being populated for the 'pull_request_commits' stream HOT 5
- If a member is part of multiple teams, they will only be listed once HOT 2
- ValueError: not enough values to unpack (expected at least 1, got 0) in repository_streams HOT 1
- Incremental replication doesn't respect the current state HOT 1
- Use pre-commit.ci to lint project
- Stream `extra_metrics` fails on repos with large number of issues/PRs HOT 1
- Drop support for python 3.7 HOT 1
- Invalid SCHEMA messages are produced for deselected streams HOT 3
- Replace use of `get_next_page_token` in the tap HOT 2
- Workflow streams incorrectly claim to support incremental loading
- Hard to tell if API token is valid or not HOT 1
- Add `files` property to `CommitsStream` HOT 1
- Experiencing 401 Bad Credentials when credentials are valid
- Document `api_url_base` setting for Enterprise Server installations
- SDK Version pointing to a specific commit HOT 1
- Loader 'target-jsonl' is not known to Meltano.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tap-github.