szabgab/crystal-mine.cr

Analyzing all the source-code using the Crystal programming language

Crystal Mine

Source code of the Crystal Mine

Setup Development environment

• Create a Personal Access token with a name that you can easily recognize, e.g. "Crystal Mine Development Token" using the following rights:

repo
   x public_repo

user
   x read:user
   x user:email

• Save your GitHub username and the token it in a file called config.txt like this:

USERNAME:TOKEN

Docker compose for development

1. Install Docker
2. Install Docker compose (e.g. pip install docker-compose)
3. Copy docker-compose.override.yml.example to docker-compose.override.yml and change it according to your taste.
4. Build: docker-compose build
5. Start docker-compose up
6. Collect some data: docker exec -it crystal-minecr_cron_1 crystal src/mine.cr --repos spec/repos.txt
7. Visit http://localhost:6002

Run tests

docker exec -it crystal-minecr_cron_1 crystal spec

Run test files separately

docker exec -it crystal-minecr_cron_1 crystal spec/web_test_spec.cr
docker exec -it crystal-minecr_cron_1 crystal spec/db_test_spec.cr

Data Collecion:

A single URL

docker exec -it crystal-minecr_cron_1 crystal src/mine.cr --url https://github.com/szabgab/crystal-mine.cr

Or create a file with a list of repo URLs and then call

docker exec -it crystal-minecr_cron_1 crystal src/mine.cr --repos spec/repos.txt
docker exec -it crystal-minecr_cron_1 crystal src/mine.cr --repos spec/many_repos.txt

Web application

crystal run src/webapp.cr

Build sentry:

crystal build --release lib/sentry/src/sentry_cli.cr -o ./bin/sentry

Run with sentry

./bin/sentry -b "crystal build src/webapp.cr -o bin/webapp" -r bin/webapp

CSS

Generate CSS file (public/bulma.css) from SCSS file (sass/mystles.scss):

docker run --rm -it -v$(pwd):/opt --workdir=/opt node:14.15 npm install && npm run css-build

Add fields to collect and display

1. Add to database schema (in db)
2. Add to store_in_db (in db)
3. Run crystal/update.cr that will update the *.sql and *.json files in spec/
4. Add columns to FIELDS and to parse_row
5. Add columns to the display template

crystal-mine.cr's Issues

Disable the downloading of large repositories

Create a list of repositories that are too big

https://github.com/tomato42/ecpp-verifier/ is 2.1 Gb

There are also

https://github.com/vjousse/lucky-blog/ is 150 Mb
https://github.com/mattrberry/crab is 93 Mb

and don't download their source code.

For now, it is ok to download them, but later if the disk-space usage becomes an issue this might need to be considered.

Add /stats page

List which shards use ameba

• What is their ameba configuration? Which rules are turned off usually?
• Run ameba on every shard and collect the results showing which violations are the most frequent ones

Improve deploy: generate the bulma.css during deploy

Processing mode --all (and maybe others) only clone Git if the repo is newer than what we have

Add Google Analytics; Add to Google webmaster, submit the sitemap.xml

Move application inside Docker container and docker-compose

Create sitemap.xml and robots.txt

Is there an .editorconfig in the shard?

Allow user to search database

Searching the name and the description of the project

List dependencies based on the actual usage (import statements)

Link to dependencies, list reverse dependencies

Shards with invalid license strings

• Create an accepted list of license strings
• Listing of the shards with license strings that are not in that list

Analyze the crystal field from the shard.yml to show which shard requires old versions of crystal

Show the most recently updated shards

Have a URL /recent to show the list

We can start by using the date when the entry was updated in our database, but later we should move to the date the repository was updated.

Add a page for /github.com and /github.com/user_name

Menu Hamburger does not work on mobile phone

Create a logo that can also be used as a favicon

Directory browser - don't browse into .git directory, don't even show it

Make it easy to exclude a repo

A config file in this project or a file in the GitHub repository of the projects that want to be excluded?

Save the latest source code and show it on the web site. Index some keywords

Index keywords and expressions

• require
• class
• Set
• Hash
• Array
• NamedTuple
• def initialize
• property
• getter
• def_equals
• include JSON::Serializable

List all the shards on the front page

Improve deploy: compile the webapp and the collector during docker-compose build

Show elapsed time for each web page request

Test the data collection process

Create a cross reference with the authors

Add nice HTML/CSS, add navigation menu to main page and about page

List dependencies based on the shard.yml

link to dependencies, list reverse dependencies

Deploy application using Docker compose

list shards with problematic crystal version requirements

Create searches that will show the shards with "problematic crystal version requirements"

• The ones with totally invalid requirements (e.g. 1-2)
• The ones with upper limits < 2 that are not invalid but are probably unnecessary
• The ones with ~> 1 that also set an upper limit and have the same issue as the pure upper limit ones

Improve development: Make sure "crystal spec" can run inside the Docker container

Fetch all the Crystal projects from Github

waiting for watzon/octokit.cr ( watzon/octokit.cr#13 )

Alternatively use plain GitHub API request

Be able to easily find the documentation or link to documentation of packages.

Both for packages included in the Crystal distribution and for third-party packages
e.g. HTTP::Client::Response

Use templates

Make logging to a file the default and have a --screen flag to log to the screen

Show last-modified based on GitHub

Implement pagination of the results

We need this before we can index all the data and before we can have sub-lists of items linked from the stats page.

In source listings display images properly

When a shard has an image in its source, the images should be displayed properly inside our web page.
e.g.
https://crystal-mine.org/github.com/szabgab/crystal-mine.cr/source/public/crystal-mine-icon.png

Collect all the contributors as well and show them as well

Collect information from git command, show names, show images

What is that "replace" in the data dump?

In spec/data.sql

Display version restrictions

See also https://forum.crystal-lang.org/t/artificial-shard-restrictions-version-2-0-0-considered-harmful/3319

Collect data from Gitlab as well

e.g. https://github.com/icy-arctic-fox/espresso depends on

https://gitlab.com/arctic-fox/spectator
https://gitlab.com/arctic-fox/opengl.cr

So we better start collecting the shards from gitlab as well.

Separate the "Crystal versions" and the "licenses" page from the stats page

e.g. /stats/versions and /stats/licenses

Show the time the collector --recent ran the last time

Maybe show a history of the timestamps, or at least the 10 most recent runs.

Maybe also show the verbose log from each run.

Include the total number of shards after each run so we can have a graph.

Make sure site does not crash if database is missing

Create a page with the pictures of the authors

Add all the details we already have about a shard

crystal mine.cr should display the help end exit