esipfed/geoweaver

a web system to allow users to automatically record history and manage complicated scientific workflows in web browsers involving the online spatial data facilities, high-performance computation platforms, and open-source libraries.

Geoweaver

Geoweaver is an in-browser software allowing users to easily compose and execute full-stack data processing workflows via taking advantage of online spatial data facilities, high-performance computation platforms, and open-source deep learning libraries. It provides all-in-one capacity covering server management, code repository, workflow orchestration software, and history recorder.

It can be run from both local and remote (distributed) machines.

Why choose Geoweaver?

1. Safely Store all your progress along the way.
2. Stay organised and productive through out your years-long research
3. Seamlessly connect to external servers with SSH.
4. In-Built Web UI with full support for Python.

GeoWeaver is a community effort. Any contribution is welcome and greatly appreciated!

Software Goals

Only two things basically:

1. Make it time affordable for less-coder scientists (who know nothing about WfMS) to manage their data processing workflows

2. Preserve all the model run history and share them along with the code files

Installation

Upgrade

Tutorial

• Create and Manipulate Hosts

• Create and Run Processes

• Create, Run, and Export Workflows

• Jupyter Recording

Demo

A live demo site is available.

Citation

If you found Geoweaver helpful in your research, please cite:

Sun, Z. et al., "Geoweaver: Advanced cyberinfrastructure for managing hybrid geoscientific AI workflows." ISPRS International Journal of Geo-Information 9, no. 2 (2020): 119.

Dependencies

This project is impossible without the support of several fantastic open source libraries.

d3.js - BSD 3-Clause

graph-creator - MIT License

bootstrap - MIT License

CodeMirror - MIT License

JQuery Terminal - MIT License

Community

License

MIT

geoweaver's Issues

Enable the SSH console

SSH console is important for having real-time instant control over the distributed resources.

Don't close SSH session after the logging window is closed

Create a TOC

TOC is very important when the document is too long.

Cannot download files without suffix

In the file browser function of SSH server, people cannot download files without suffix.

How to reproduce it?

1. Choose one host under "SSH Server" list.
2. Click "Browse File Hierarchy"
3. Enter password in the popup dialog
4. In the file hierarchy, find a file without suffix, such as XYZ
5. Click the file name. The downloading won't start.

Screenshot

workflow "ends too quickly"

When running a workflow, Geoweaver provides the following messages in quick succession:

00:00:01 connected
00:00:01 It ends too quickly. Go to history to check the logs out.
00:00:01 disconnected

However, the workflow executes correctly on the remote server.

Provide option to remember password in one session

It is annoying to input password every time.

The right panel will hide when the screen is small

Keep it on no matter how small the screen is.

Uploading workflow is not working

Error parsing uploaded file
error message: d.source is undefined

The newly created workflow cannot be monitored

A bug it seems.

Add a progress bar list to indicate the progress

Add a progress bar list.

Keep outputing log for workflow

At present, once the first process is finished, the log output will interrupt.

Support running Jupyter Notebook as a process

The process should allow run of jupyter notebooks and show the results.

Fail to install on Lubuntu linux

The following error pops up:

org.h2.jdbc.JdbcSQLNonTransientException: IO Exception: "java.net.UnknownHostException: ggs681-VBox: ggs681-VBox: Name or service not known" [90028-199] at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar!/:na] at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar!/:na]

Add name/notes to the history versions of jupyter notebooks

Allow users to label or tag the version.

Add a button to remove notebooks without tags (non-important)

H2 database empty

H2 database seems to be empty after Linux quick install.

Database files are moved to ~/ by install script. Can connect only using the default H2 credentials:

username: sa
password: [blank]

(username: geoweaver, password: geoweaver does not work)

After connecting, no tables exist.

I am new to Java, H2, etc., so this may be user error.

Thank you.

Add logo

Resize the circle to fit the length of the process name

The text should be completed wrapped inside.

When the log out is too long, the save function has problem

There is limit on the mysql column length.

XSRF cookie does not match POST argument

When trying to login in a jupyter at the first time, the login page will return this message.

Simple solution is clearing all the browser cache and trying to login again.

Better solution is required.

Create a tool to upload files from local to the servers

Now only the downloading channel is supported. The reverse, uploading, should be there as well. It is an essential function for Geoweaver.

How to check the status of running processes

The running process has no sign after it is started. It should has an record in the history showing the status of running.

Make the proxy of JupyterHub work

The oauth doesn't work for authorizing the websocket session. The kernel keeps dying and asks for restart.

Improve the installation package

Use h2 and jetty to make it a downloadable install file on all three mainstream operation systems.

Create installable software release besides the tomcat and docker

Exe for windows, dmg for Mac, deb for Linux.

Improve the user guide to the latest version

Use some library to do the automatic documentation.

JupyterLab doesn't work with Geoweaver

The content is not showing up. All the buttons don't work.

Cannot download/upload big files to jupyter via Geoweaver

When the files are big, e.g., 1gb, Geoweaver will fail.

Collect successful examples for demo and boosting user confidences

One feedback from ESIP summer meeting 2019, one feedback is saying "Geoweaver need more successful example" to attract more community users.

Use NoVNC or Spice to replace the SSH console

NoVNC is more stable and powerful to control the server. But still no sure if this is a good move. Should go deep on what Geoweaver is really pursuing.

Small fix to README

Under Quick Install for Mac, incorrect port in link:

Should be

http://127.0.0.1:3030/Geoweaver/web/geoweaver

Currently

http://127.0.0.1:8080/Geoweaver/web/geoweaver

SSH Server not showing up after being newly added

As title said.

Add Jupyter Notebook Client into Geoweaver

To allow scientists to feel at home when using Geoweaver to run their jupyter notebooks. This function will also enable the support of Pangeo in Geoweaver.

Show the output in a fixed window in the workspace

The output should keep going after the window is minimized.

Default database change from mysql to h2

The database of Geoweaver won't be large. MySQL is a overkill. H2 will be much more lightweight, efficient and scientist-friendly. The change is necessary.

It takes too long to wait for the response of failed login attempts

There is some place stuck.

The process in the workspace is not selectable

Cannot select the circle, the monitor is not working neither.

Fix the frozen jupyter cell on long-running tasks

Priority issue need be fixed.
@andrewmagill @ZihengSun

Build in some usual scripts for installing deep learning libraries

Some bash scripts that are universal to install dependencies and create environment for deep learning will be very useful for everyone.

Test Geoweaver with Dask cluster on Azure Kubernetes

As title said.

issue in windows installation script for windows quick install

the windows installation script employs the use of Expand-Archive which contains extraction path and Destination Path both of which could be encapsulated within Quotation marks if either of the string contains space in between folder name, gives out parameter binding exception error which could be solved by above said method.

C:/Users/test cap/ would give out error which can be solved by
"C:/Users/test cap/" binding it in between quotation marks

A workflow of North Dakota Crop Map generation

Use the trained network on 2016.

Fail to save file in the browser

After click save, it says "saved" but the file doesn't change.

Save the unfinished jobs into the history table

Add builtin function show() to display results and inputs for comparison

Allow to run the code in snippet to get response in time

Create a process execution dialog similar to jupyter notebook, while not requiring the complicated jupyter logic.

cannot upload jupyter notebook

When creating a new process, UI elements allowing uploading or importing of Jupyter Notebooks are not present.

(This was fixed with Ziheng's PR #58, just adding the issue for the record. Please close.)

Documentation about how to use Geoweaver with Dask cluster on Kubernetes

Detailed tutorials are needed to teach people how to use Geoweaver on Dask cluster.

Increase z level for modal windows

I think the desired behavior would be for all modals to overlay on top of non-modal UI elements, please increase z level to bring all modals to the front.

Create a file browser by SFTPClient

A file browser to view the files within the host. Password protected.

Backspace not working in new process dialog name field

When creating a new process, the name field does not seem to recognize the backspace key. To correct an error, you must highlight and replace. I have testing with Chrome and Firefox on Mac and Linux.