incquerylabs/repo-crawler

A small utility for crawling model repositories

Repository and revision crawler for Teamwork Cloud

Run the crawler

1. Download twc.repo-crawler-<version>-all.jar from GitHub releases
2. java -jar twc.repo-crawler-<version>-all.jar <args> with arguments (see below)

Example arguments

Running the jar without arguments prints out the usage instructions.

• Running the crawler without workspace, resource, branch, and revision will result in crawling the repository structure of the given scope
• Running the crawler with workspace, resource, branch, and revision will result in crawling the model content of the given revision

Repository structure of OpenSE Cookbook workspace on twc.openmbee.org

-S twc.openmbee.org -P 8111 -ssl -W 9c368adc-10cc-45d9-bec6-27aedc80e68b -C 2000 -u openmbeeguest -pw guest

OpenSE Cookbook model on twc.openmbee.org

-S twc.openmbee.org -P 8111 -ssl -W 9c368adc-10cc-45d9-bec6-27aedc80e68b -R c6bede89-cd5e-487b-aca8-a4f384370759 -B 29110c0f-bdc1-4294-ae24-9fd608629cac -REV 350 -C 2000 -u openmbeeguest -pw guest

TMT on twc.openmbee.org

-S twc.openmbee.org -P 8111 -ssl -W 9c368adc-10cc-45d9-bec6-27aedc80e68b -R 6819171d-1f52-4792-a08d-15d50d47985a -B a95e8bd1-f7d2-433e-a0a1-0c1cd7702e59 -REV 229 -C 2000 -u openmbeeguest -pw guest

Configuring self-signed certificate when using SSL on Teamwork Cloud OSMC API

• Java has to start with -Djavax.net.ssl.trustStore=<path to JKS file> -Djavax.net.ssl.trustStorePassword=<password of JKS file>
  • Note: These are JVM parameters that should be added before the -jar part
  • You can add -Djavax.net.debug=all to check whether Java loaded the certificate correctly
• Make sure that ssl is enabled in CLI arguments

Running from source

1. Clone this repository or download the sources for a release.

IntelliJ IDEA

1. Install IntelliJ IDEA with Kotlin plugin.
2. Open IntelliJ IDEA and open project. File -> Open.. -> <project_path> (Select Gradle if needed.)
3. Build -> Build Project
4. Run -> Run... -> com.incquerylabs.twc.repo.crawler.Crawler.kt (set arguments in Configuration)

Gradle

1. ./gradlew run --args='<args>' with arguments (see above)

Shadow Jar

1. ./gradlew shadowJar
2. java -jar build/libs/twc.repo-crawler-<version>-all.jar <args> with arguments (see above)

repo-crawler's Issues

Print user and session cookie to console to allow logout even in case of JVM process termination

Log the parent element name if an element is not found

We should extend it to log the fully qualified name of the parent element of a missing elements

Usage instructions not printed when starting without arguments

Add basic integration tests for crawler with mocked OSMC API

• introduce interface for mocking API calls or start a mock HTTP server on localhost
• add a basic test that runs the crawler and checks for completion and optionally successful session cleanup

Support configuring OSMC timeout and number of concurrent queries

Print element IDs on error

Upgrade to Vert.x 4

Support reading configuration from file instead of command line

When the repo crawler is executed without parameters, look for a file called config.json in the working directory and parse it instead of getting command line parameters.

The configuration file should contain at least the required fields for the current version (e.g. username, password, server, port, etc.).

Refactor crawling to retrieve several elements at once

TWC provides a method to download elements at once. Issue POST request to /elements, i.e., /osmc/resources//revisions//elements, /osmc/resources//branches//elements. The request body is a list of UUID separated by commas, e.g., 4eeb2e6d-9cbc-4f59-9d74-69263e52ba54,520bc74d-b5b4-40d5-87cb-9d0b8aba5d2e,7a057ae7-6281-462d-9dcd-de9931633f5c and the Content-Type in the request header should be “text/plain”.

Add element content handler

Print name of workspaces, resources, branches when crawling repository structure

We should improve repo-crawler in a way that helps users select a revision to test. This can be done in several ways:

• Print the label/name of visited structure elements to standard output when receiving this information
• Store the structure information in-memory and save it to a file at the end of the execution
• Stream structure information to a file while running