Using a sqlite database that can be created by for a single project

[chaoss]

Using a sqlite database that can be created by for a single project

[howderek]

Hello @dmgerman! We appreciate your feedback and I apologize for the delayed response. I absolutely agree that GHTorrent is the largest hurdle for using GHData. We are exploring solutions and will include your SQLite suggestion in our discussions :)

[sgoggins]

@dmgerman : there is no coupling to the GHTorrent as a data source. If you mimic the schema from another source, that will work. The challenge, technologically, is that the GHTorrent schema requires queries more complex than sqllite will support in order to operationalize the metrics.

We could refactor some of that with pandas.

Is the main aim here to have a developer version on lighterweight technology?

[dmgerman]

Can you give me an example of a query that SQLite will not be able to handle (that mySQL can)? In my experience it is the other way around (mySQL does not support merge joins, for example).

The goal is to change the need for GHTorrent to a script that does the equivalent mining, but only for one project. That will be significantly easier than having to install GHtorrent.

[howderek]

@dmgerman, GHTorrent appears to already support SQLite and can be configured to only load data for a set list of repositories. If you would like to add SQLite support to ghdata, you could either modify ghtorrent.py to work with both MySQL and SQLite, or add another data source focused on SQLite.

[sgoggins]

I think there are specific challenges being addressed in this issue:

1. Wanting to run GHData against a smaller dataset, or against a dataset that is not captured in GHTorrent (i.e., not a GitHub repository). We think this will be accomplished working the Perceval as a mapper/aggregator. This is something Jesus and I are discussing as part of the CHAOSS project
2. Actual support for SQLLite: SInce GHTorrent now provides .csv files by default, I think a subset could be mapped into SQLLite as @howderek notes above.

[howderek]

Hello @dmgerman,

I just wanted to follow up on your feedback about GHTorrent being burdensome. We have discussed a new architecture that will make it so that Augur does not rely on GHTorrent (which we agree is an issue), which we will work on implementing in the coming months.

Thank you again for your feedback!