fiqus/coophub

Cooperatives repos over the world! 🌈🌎

Coophub - Cooperatives repos over the world!

🔗 coophub.io

This web app uses the GitHub API and GitLab API to fetch, process and nicely display the projects/repositories of any subscribed cooperative from over the world.

Please note that it's strictly limited to cooperative enterprises.

The main goal is to gather in one place all the open source projects that can be used to start others, to be consumed, to learn or to just motivate collaboration.

Add your cooperative

1. Fork this repo or edit this file
2. Add your co-op in the cooperatives.yml file:

key_org_name:
  source: github|gitlab|git.coop
  login: <USERNAME_IN_THE_SOURCE>
  name: <NAME_OF_THE_COOP>
  url: <URL_OF_THE_COOP>
  description: <DESCRIPTION>
  location: <LOCATION>

3. Add-commit-push and send us a Pull Request!

Changelog

See changelog file.

Development

Use GITHUB_OAUTH_TOKEN ENV var in order to authenticate with the GitHub APIv3. Read the guide.

With docker

If you want to set any environment, like GITHUB_OAUTH_TOKEN, create a .env file with the custom values that you want.

Available environments:

environment	Default value
GITHUB_OAUTH_TOKEN
VERSION_ELIXIR	1.14
VERSION_NODE	18

Steps:

• Clone this repository
• Run the command docker compose up -d

Local

It uses the Elixir Phoenix Framework for the back-end and ReactJS for the front-end. Then, to run this app you will need:

• Erlang OTP >= 18
• Elixir >= 1.5
• NodeJS >= 5.0

Run it!

• Install dependencies with mix deps.get
• Install Node.js dependencies with npm install
• Start Phoenix server with mix phx.server

Now you can visit localhost:4000 from your browser.

API Endpoints

• GET api/orgs (all the coops)
• GET api/orgs/:name (detail of a coop)
• GET api/orgs/:name/repos (repos of a coop)
• GET api/repos (all coops repos)
• GET api/search?q=term_to_search
• GET api/topics (all the repos topics)
• GET api/languages (all the programming languages with percentages)
• GET api/languages/:lang (the repos using the lang)

Allowed query params

• limit - Number
• sort - popular or latest (default)
• exclude_forks - boolean (false default)

Releasing a new version

1. Update CHANGELOG.md with latest changes.
2. Go to create a new release and complete the fields:

• Tag version: vx.x.x (like v0.2.2).
• Target: Always against master branch.
• Release title: Same as tag version.
• Description: Just copy/paste the latest changes from CHANGELOG.md.

3. Click the Publish release button and check that a new github action was started for this release.
4. When the github action finishes, a release asset should be attached (like coophub-20200330-034316-0635b9c7.tar.gz).
5. Done! Just wait a few minutes and the new release will be deployed to https://coophub.io (you can check the version at site footer).

coophub's Issues

coophub.io is down?

The server hosting the site appears to be down?

lynx -head -dump  https://coophub.io/

Looking up coophub.io
Making HTTPS connection to coophub.io
Alert!: Unable to connect to remote host.

lynx: Can't access startfile https://coophub.io/

dig coophub.io +short
54.172.94.238

dig -x 54.172.94.238 +short
ec2-54-172-94-238.compute-1.amazonaws.com.

ping 54.172.94.238
PING 54.172.94.238 (54.172.94.238) 56(84) bytes of data.
^C
--- 54.172.94.238 ping statistics ---
88 packets transmitted, 0 received, 100% packet loss, time 89066ms

Add gitlab API

Note that an org could have both github and gitlab.

• We can specify if the coop user is from github or/and gitlab.
• When get the repos for the cache get from gitlab api too
• Show an icon in the repos box in the UI to distinguish between github/gitlab

Fetch tags for each repo

Fetch tags for each repo and will be used in the search box.

Create repos by lang endpoint and link languages tags

Avoid using the /search endpoint to get repos by language, create api/languages/:lang and integrate it to the languages tags/buttons.

Add the endpoint to the README

Repos by lang

Currently we can search by language i.e: /api/search=python but we can improve 2 things:

• search over main languages (if a repo has a 1% of Python it is currently matching)
• order the results by popularity.

Coop profile: show main languages

We may display those above the average percentage (displays "n" langs) or just the most "n" langs (fixed).

Integrate with GITLAB

add org info in the cache

Using: https://developer.github.com/v3/orgs/#get-an-organization

Improve cachex warmer

The idea is to have the data available as soon as it comes from github API for each org.

Currently, the data is available when ALL the orgs and ALL their repos+languages were fetched.
That process can take a lot when starting the server!

Remove repos from organization data

api/orgs endpoint should return organization data without repositories.

Not getting repos for a coop

This is the case, accessing to the GH page they have a lot:
https://coophub.io/orgs/enspiral

Full steam labs organization does not exist in github

https://api.github.com/orgs/fullsteamlabs

Serve all repos created_at to graph the results in the home

It will replace the hardcoded Commits per month

Update deps

Done and released at v0.2.10: https://github.com/fiqus/coophub/releases/tag/v0.2.10

Bug in languages in profile

This profile is showing other 100% instead of the main langs
https://coophub.io/orgs/obran

Wrong Languages % in the index chart

There's a bug with the languages percentage for all the orgs (/api/languages):

1. The calculation is being done with bytes, we should use the sum of org percentages.
   So, it does not affect if there is a repo with a LOT of bytes of one language, instead of the percentage of that language in the org.
2. The GitLab API retrieves language percentages and not bytes, so the number of bytes couldn't be used to calculate totals anymore.

Proposed solution:

• Calculate the total of lang % using the aggregation of org lang %.
• Get rid of bytes where it is not needed.
• Fix the index chart with the languages accumulating the org percentages (it will be impacted in react models, etc)

Jamgo coop organization does not exist in github

https://api.github.com/orgs/jangocoop

Support Gitea API for self-hosted git hosting

Hi folks! Autonomic.zone want to join but we've got our git hosting setup on https://git.autonomic.zone which is a self hosted gitea instance. As far as I know, this app doesn't support reading the Gitea API yet? There seems to be decent docs for it though https://docs.gitea.io/en-us/api-usage. Would be cool to get this sorted! Thanks!

Add release info in README

Add how to make a new version, a release and run a deployment.

Support to declare ignored repos at cooperatives.yml

Cooperatives could decide to not display (ignore) particular repos.
A public repo may receive continuous updates but it's not a relevant repo, and that causes to be listed as latest/popular taking the place for other important repos.

May be good to support regex patterns!

Fetch org members

Add members list to org data.

Add tests for API controller

add tests for api

missing endpoints

Endpoints:

• GET repos/latest
• GET repos/popular
• GET orgs
• GET orgs/:name
• GET orgs/:name/latest
• GET orgs/:name/popular

HTTPS support

The http://coophub.io/ site is only available via HTTP at the moment?

Could HTTPS support be added?

Support GitLab API for self-hosted git hosting

Similarily to #65, me as a user of coophub would like to connect this dashboard to a variety of GitLab communities on different servers, in order to achieve an aggregated higher-level view of the communities I am interacting with.

dynamically limit repos via query

using ?limit=X
default: 3

Support for site subdomains as matching keys for langs

Joining at: https://elixir.coophub.io
Should display results as in: https://coophub.io/languages/elixir

We could even match against topics too or even more, as a search term and simply display its results...
An URL like: https://docker.aws.deploy.coophub.io
Displays the results of: https://coophub.io/search?q=docker+aws+deploy
But that could be too much...! 😄

Just for langs seems to be a good first approach to this feature.
Suggestions welcomed!

Loading cache asynchronously

Try using Task.async and compare times to be sure before merging.

Align GitHub's best practices for rate limits

https://developer.github.com/v3/guides/best-practices-for-integrators/#dealing-with-abuse-rate-limits

https://developer.github.com/v3/#abuse-rate-limits

There are 2 points where we need to work:

Make requests for a single user or client ID serially. Do not make requests for a single user or client ID concurrently.

When you have been limited, use the Retry-After response header to slow down. The value of the Retry-After header will always be an integer, representing the number of seconds you should wait before making requests again. For example, Retry-After: 30 means you should wait 30 seconds before sending more requests.

Open Graph and Twitter metadata

I'd be interested in seeing if some additional Open Graph and Twitter metadata could be added to <head> sections of pages, see:

• The Open Graph protocol
• Facebook: A Guide to Sharing for Webmasters
• Optimize Tweets with Cards
• Twitter Card Validator

Is this the only template for the HTML head?

• lib/coophub_web/templates/layout/app.html.eex

Use github actions for AWS deployment

Setup an AWS instance in order to be used as coophub server.
The deployment should be done through github actions.

Fix DateTime.from_iso8601/2 on get_repo_popularity/1

[error][2022-11-15 06:47:35.792] Task #PID<0.32181.3> started from #PID<0.32178.3> terminating
** (FunctionClauseError) no function clause matching in Calendar.ISO.parse_utc_datetime/1
(elixir 1.12.3) lib/calendar/iso.ex:510: Calendar.ISO.parse_utc_datetime(nil)
(elixir 1.12.3) lib/calendar/datetime.ex:1073: DateTime.from_iso8601/2
(coophub 0.4.3) lib/coophub/repos.ex:173: Coophub.Repos.get_repo_popularity/1
(coophub 0.4.3) lib/coophub/cache_warmer.ex:231: anonymous fn/1 in Coophub.CacheWarmer.put_popularities/1
(elixir 1.12.3) lib/enum.ex:1582: Enum."-map/2-lists^map/1-0-"/2
(elixir 1.12.3) lib/enum.ex:1582: Enum."-map/2-lists^map/1-0-"/2
(coophub 0.4.3) lib/coophub/cache_warmer.ex:214: Coophub.CacheWarmer.get_org_with_repos/2
(elixir 1.12.3) lib/enum.ex:1582: Enum."-map/2-lists^map/1-0-"/2
Function: #Function<14.67629458/0 in Coophub.CacheWarmer.warm_cache/0>
Args: []

Order the org repos using a criteria

Order by popular or latest (or both with a combobox) in:
GET orgs/:org/repos

Can an organization that is part of the cooperative be added?

We at @LibreCodeCoop have created and are maintaining a free-licensed digital signer, @LibreSign, can the LibreSign organization be added to CoopHub?

Add search box and show repo results

Add memoization for all the endpoints

add another cachex key and memoize by uri

Add last updated date in any place

Add the last updated date of the cache warmer in any place.
This is useful for checking next time the warmer will be run and to inform the visitors the last update of the repos if they see any discrepancy.

Complete tests

After #6 that added tests for API controller, we should complete other tests as well:

• ReposTest
• ReposWarmerTest

Fix react-github-btn component when running npm run deploy

It currently breaks the deploy build:

ERROR in app.js from UglifyJs
Unexpected token: name «react_github_btn_GitHubButton», expected: punc «;»

Exclude forks from lang stats

this

Support for git.coop

They are using a self-hosted Gitlab:
https://git.coop/api/v4/groups

See conversation in #4

Use an updated version of uglifyjs-webpack-plugin that don't break npm run deploy

After issue #22 and PR #32 the dep uglifyjs-webpack-plugin v2.2.0 was downgraded to v1.0.0-rc.0 because v2 breaks assets prod deploy when running: npm run deploy.

The idea is to address this in the future, trying to keep all deps at its latests versions.

Dashboard with statics of n° of repos

Add YML format validator

Add it to the CI

One option https://yamllint.readthedocs.io/en/stable/quickstart.html#installing-yamllint

Repo metadata with markdown (gitlab)

Gitlab supports markdown in the repo description, analyze if we will support this and how.

Add location for each coop and a map with all coops

The full-size map could be displayed at: https://coophub.io/map

UI improvements - org page

• Show main languages (issue #7). Click to see more (all) languages.
• Show a badge if the org is_verified or not.
• Sort repos by popularity (as it is now, but horizontal instead of vertical).
• On loading keep footer at bottom.
• Show quantity of repos.

Add filter for deprecated repositories

Feature request: Add filter for deprecated repositories. Button that toggles showing only non-deprecated repos.

Extract github API into its own module and refactor cache warmer to use it

Related to #4, as second step to archieve gitlab support (the first one was the PR #47).

The repos_warmer should be refactored to call behaviourized functions to fetch data from external sources. The first source will be github (and after this, in another issue, the gitlab one).

Each external source module should implement the required functions to fetch the remote data and return structs defined at #47 (Organization and Repository).

UI improvements - home page

• App logo for home page + favicon.
• App description for home page.
• Show how many orgs and repos we have indexed at coophub so far.
  (keep adding)