langleu/ma_crawler

GitHub Crawler to find any given document. Built for my master thesis.

MA_Crawler

A distributed crawler built for my master thesis. Goal is to collect data and pre-process it from e.g. GitHub.

Getting Started

Prerequisites

Node.js >= v10

Grakn.ai - 1.8.4

Docker (optional)

Docker-Compose (optional)

Start

Install Packages

npm i

Master

GITHUBACCOUNTS='[{"username": "name", "token": "00000"}]' TYPE=master node app.js

Crawler

node app.js

Docker

docker-compose up --build

Environment Variables

Please look at config.js for a better overview and understanding.

• PORT - (def) '8000'
• TYPE - (def) 'client' || 'master'
• GRAKNURI - (def) 'localhost:48555'
• SENTRY - (def) ''
• MASTER_SOCKET - (def) 'http://localhost:8000' (Socket of the Master)
• GITHUBACCOUNTS - (def) [] - example '[{"username": "name", "token": "00000"},...]'

ma_crawler's Issues

Database Connection

GraphDB (figure out which one)
possible mongodb for metadata?
dive deeper into neo4j and grakn. Take notes and decision making to the other repository.

Add Readme(s)

for MA_Crawler and MA_Thesis

More status reporting and endpoint for master

e.g. how far nodes are with crawling.
how the current status is of nodes (healthy, reachable etc...)

Metadata storage?

could be used to check against duplication.
for error logging of crappy crawled files.

e.g. MongoDB

• check how grakn deals with duplicates when it comes to the same key
  • if one tries to commit duplicates those will be automatically ignored and all uncommitted data will be removed as on every insert we immediately commit only the duplicate should be lost
• maybe just write errors down to disk or sentry?

Client testing

spend a day testing the implementation of the client using various docker files.

Strategy Pattern (Crawler)

Add Swagger for API Documentation

or comparable that works better with koa

any service of name X is linked to deployment

let graqlInsertQuery = `match $deployment isa deployment, has rid "${deploymentId}"; `;
graqlInsertQuery += `$service isa service, has name "${serviceName}"; `;

generate unique hash for services based on name + deployment sha

Basic Routing

Strategy Pattern (Processing)

first GitHub strategy

based on the API

Assign GitHub Tokens/Accounts from the master onConnect

Google crawler strategy

possible first step intitle:docker-compose.yml site:github.com

Create Dockerfile and Docker-Compose

master/worker support

Define Database Model

Grakn to Gephi

convert a grakn graph to gephi.

has nothing directly to do with the crawler, but for now I'll keep it here.

the grakn workbase is just simply too slow to handle multiple hundred thousand entries.

https://gephi.org/gexf/format/schema.html (1.2)

second GitHub strategy

based on the web app via webcrawlers