microsoft/ghcrawler-dashboard

Crawler Dashboard to control ghcrawler application.

ghcrawler-dashboard's Issues

Not easy to configure "single-shot" running

The existing crawler is rather designed to be able to scale to crawling many hundreds or thousands of repositories; it's a daemon which carefully schedules its work, and a separate set of utilities to control that daemon and add to its queues and so on. This is great if you want to crawl thousands of repositories. It's pretty serious overkill if you want to crawl, say, three repositories though. Is there some way to run all this in "cut down" mode, where it wakes up and scans a repository and then shuts down, without needing lots of setup and a running background daemon and so on, so it's a bit less heavyweight? (The answer may be "no, that's not what it's designed for", which is fine, but if there's some way to spin it up in a lightweight way that would be useful to know about.)

Where does configuration need to go?

The README says that the bare minimum configuration is (some vars), but doesn't say where they should be put. env/env.json? env.json? config/env.json? Somewhere else?

rename to ghcrawler-dashboard

We've settled on "ghcrawler" as the name for the system. rename this repo to line up

Dashboard API returns 500s if Crawler Service using self-signed certificate

If the endpoint at CRAWLER_SERVICE_URL is using a self-signed (or otherwise untrusted) certificate then the dashboard will fail to perform any actions that require talking to the Crawler Service. The dashboard should support passing a list of CA certificates to request like is done here: https://github.com/Microsoft/ghcrawler/blob/develop/providers/queuing/rabbitQueueManager.js#L48.

How do input to crawler to search for repos with specific contents.

How do input to crawler to search for repos with specific contents.

Deleting/Requeuing dealetters should not fail if one of requests fails

Warning messages should be added to response instead of either failing everything or completely swallowing the errors.

unable to get dashboard working

So I have ghcrawler running and I'm able to manage it using command line.
However dashboard is not coming up.

I cloned this project locally then ran npm install and then npm start (unable to run DEBUG=* npm start)
however then when I go to localhost:4000 it doesn't load anything.