fabiankirstein/hopsworks

HopsWorks - Hadoop for Humans

Hopsworks

Hopsworks is the UI for Hops, a new distribution of Apache Hadoop with scalable, highly available, customizable metadata. Hopsworks lowers the barrier to entry for users getting started with Hadoop by providing graphical access to services such as Spark, Flink, Kafka, HDFS, and YARN. HopsWorks provides self-service Hadoop by introducing two new abstractions: projects and datasets. Users manage membership of projects, which scope access to datasets. Datasets are again managed by users who can safely share them between projects or keep them private within a project. Hopsworks takes the administrator out of the loop for managing data and access to data.

Information

• Follow our Twitter account.
• Join our developer mailing list.
• Checkout the current build status.

Installing Hopsworks

Installation of Hopsworks and all its services is automated with the Karamel software. Instructions on how to install the entire platform are available here.

For a local single-node installation, to access Hopsworks just point your browser at:

  http://localhost:8080/hopsworks
  usename: [email protected]
  password: admin

Build instructions

Hopsworks consists of the backend module which is packaged in two files, hopsworks.ear and hopsworks-ca.war, and the front-end module which is packaged in a single .war file.

Build Requirements (for Ubuntu)

NodeJS server and bower, both required for building the front-end.

sudo apt install nodejs-legacy
sudo apt-get install npm
sudo npm cache clean
# You must have a version of bower > 1.54
sudo npm install bower -g
sudo npm install grunt -g

Build with Maven

mvn install 

Maven uses yeoman-maven-plugin to build both the front-end and the backend. Maven first executes the Gruntfile in the yo directory, then builds the back-end in Java. The yeoman-maven-plugin copies the dist folder produced by grunt from the yo directory to the target folder of the backend.

You can also build Hopsworks without the frontend (for Java EE development and testing):

mvn install -P-web

Front-end Development

The javascript produced by building maven is obsfuscated. For debugging javascript, we recommend that you use the following script to deploy changes to HTML or javascript to your vagrant machine:

cd scripts
./js.sh

You should also add the chef recipe to the end of your Vagrantfile (or Karamel cluster definition):

 hopsworks::dev

For development

You can build Hopsworks without running grunt/bower using:

mvn install -P-dist

Then run your script to upload your javascript to snurran.sics.se:

cd scripts
./deploy.sh [yourName]

Testing Guide

The following steps must be taken to run Hopsworks integration tests:

-Warning: This test will clean hdfs and drop Hopsworks database. So it should only be used on a test machine.

First create a .env file by copying the .env.example file. Then edit the .env file by providing your specific configuration.

   cd hopsworks/hopsworks-ear/test
   cp .env.example .env

Then change the properties in Hopsworks parent pom.xml to match the server you are deploying to:

  <properties>
    ...
    <glassfish.hostname>{hostname}</glassfish.hostname>
    <glassfish.admin>{username}</glassfish.admin>
    <glassfish.passwd>{password}</glassfish.passwd>
    <glassfish.port>{http-port}</glassfish.port>
    <glassfish.admin_port>{admin-ui-port}</glassfish.admin_port>
    <glassfish.domain>{domain}</glassfish.domain>
  </properties>

To compile, deploy and run the integration test:

   cd hopsworks/
   mvn clean install -Pjruby-tests

If you have already deployed hopsworks-ear and just want to run the integration test:

   cd hopsworks/hopsworks-ear/test
   bundle install
   rspec --format html --out ../target/test-report.html

To run a single test

   cd hopsworks/hopsworks-ear/test
   rspec ./spec/session_spec.rb:60

To skip tests that need to run inside a vm

   cd hopsworks/hopsworks-ear/test
   rspec --format html --out ../target/test-report.html --tag ~vm:true

When the test is done if LAUNCH_BROWSER is set to true in .env, it will open the test report in a browser.