esipfed/geoweaver

boost data pipeline's tangibility, enhance research productivity, reduce work anxiety

GitHub Repo: https://github.com/ESIPFed/Geoweaver

Pygeoweaver (Python Bindings): https://github.com/ESIPFed/pygeoweaver

Live Demo: https://geobrain.csiss.gmu.edu/Geoweaver

Geoweaver is an in-browser software allowing users to easily compose and execute full-stack data processing workflows via taking advantage of online spatial data facilities, high-performance computation platforms, and open-source deep learning libraries. It provides all-in-one capacity covering server management, code repository, workflow orchestration software, and history recorder.

It can be run from both local and remote (distributed) machines.

Why choose Geoweaver?

1. Safely Store all your progress along the way.
2. Stay organised and productive through out your years-long research
3. Seamlessly connect to external servers with SSH.
4. In-Built Web UI with full support for Python.

For further insights into Geoweaver, please explore the website at https://geoweaver.dev. GeoWeaver is a community effort. Any contribution is welcome and greatly appreciated!

Features

1. Host Management:

• Register machines via SSH as hosts for running processes.
• Add Jupyter Servers as host resources for interaction and workflow editing.

2. Process Variety:

• Add various types of processes, such as bash scripts for data downloading.

3. Jupyter Notebook Integration:

• Upload or import Jupyter notebooks.
• Intercept websocket traffic to save notebook versions, enabling easy revision history access.

4. Process History and Logging:

• Detailed history of every process run, including logs and outputs, is stored.

5. Workflow Management:

• Link processes to create workflows for parallel or sequential execution across different resources.
• All aspects of workflow management are centralized within GeoWeaver.

6. Boosts Data Pipeline's Tangibility:

• Geoweaver provides an intuitive, interactive interface for visualizing data workflows, making it easier for users to understand and manage complex data pipelines.
• This clear visualization helps users to see the connections and dependencies between different components of their workflows.

7. Enhances Research Productivity and Reduces Work Anxiety:

• Geoweaver has automated scheduling and execution of tasks, researchers can set up their workflows to run at specified times or conditions without manual intervention.
• This automation reduces the burden of monitoring and manual execution, allowing researchers to focus on analysis and innovation.

Geoweaver Installation Guide

Geoweaver is a powerful tool for geospatial data processing, offering a range of features and capabilities. This guide will walk you through the steps to install Geoweaver on your system.

Prerequisites

Before you begin, ensure that you have the following dependencies installed:

• Java 1.8 or higher (OpenJDK 8 or higher)
• Docker (required only for the Docker installation method)

Demo

A live demo site is available.

Documentation

Learn more about Geoweaver in its official documentation at https://esipfed.github.io/Geoweaver/docs/install.html

Creating a New Release

For detailed steps on how to create a new release in Geoweaver, please refer to the release instructions.

PyGeoWeaver

PyGeoWeaver is a Python package that provides a convenient and user-friendly interface to interact with GeoWeaver, a powerful geospatial data processing application written in Java. With PyGeoWeaver, Jupyter notebook and JupyterLab users can seamlessly integrate and utilize the capabilities of GeoWeaver within their Python workflows.

Please do visit the PyGeoWeaver GitHub repository.

Contributors

Thanks to our many contributors!

Geoweaver History

v0.6.7 - v1.0.0 (2018 - 2023)

Key features included:

• Made GitHub zip importable and added .wci.yml.
• Introduced a process stop button, organized the SSH folder, and enabled shell commands to call Python processes.
• Added buffer size control and exit code usage for process status.
• Fixed numerous issues on remote hosts.
• Included new features such as hovering tips and code comparison.
• Added process history and status.
• Added code search function.
• Enabled run button in side panel.
• Allows reset password from terminal.

v1.0.1 - v1.2.8 (2023 - 2024)

After incorporating feedback from the user community, the Geoweaver team released new versions. This major update focused on performance improvements and added several highly requested features:

• Log output is real time and web socket channels are untangled.
• Made the local logging real time.
• Created a macOS App for Geoweaver.
• Ability to Restore workflow.
• Run maven tests on github actions.

These versions solidified Geoweaver's position as a powerful open-source GIS solution and attracted interest from various industries and research institutions.

v1.3.0 - v1.6.1 (2024)

This version focuses on updating features and bug fixing:

• Fixed the chart visibility issue and table actions in side panel.
• Opens dock at bottom by default.
• Updated README.md by including latest features and modern style.
• Navigation fix for process tab and workflow tab in Guide page.
• Added filtering skipped process functionality.
• Responsive Design for lower resultion devices such as ipad / tablets.
• Support for MySQL and PostgreSQL - Production grade DB.
• Autosaves code on Run.
• Added support for Docker. The Geoweaver Docker image can be found here(https://hub.docker.com/repository/docker/geoweaver/geoweaver/general).

For more details, you can check the Geoweaver Releases Page.

Citation

If you found Geoweaver helpful in your research, please cite:

Sun, Z. et al., "Geoweaver: Advanced cyberinfrastructure for managing hybrid geoscientific AI workflows." ISPRS International Journal of Geo-Information 9, no. 2 (2020): 119.

Existing Projects

Sun, Ziheng, Nicoleta C. Cristea, Kehan Yang, Ahmed Alnuaim, Lakshmi Chetana Gomaram Bikshapathireddy, Aji John, Justin Pflug et al. "Making machine learning-based snow water equivalent forecasting research productive and reusable by Geoweaver." In AGU fall meeting abstracts, vol. 2022, pp. IN23A-04. 2022.

Sun, Ziheng, and Nicoleta Cristea. "Geoweaver for Automating ML-based High Resolution Snow Mapping Workflow." In AGU Fall Meeting Abstracts, vol. 2021, pp. IN11C-07. 2021.

Sun, Ziheng, Liping Di, Jason Tullis, Annie Bryant Burgess, and Andrew Magill. "Geoweaver: Connecting Dots for Artificial Intelligence in Geoscience." In AGU Fall Meeting Abstracts, vol. 2020, pp. IN011-02. 2020.

Sun, Ziheng, Liping Di, Annie Burgess, Jason A. Tullis, and Andrew B. Magill. "Geoweaver: Advanced cyberinfrastructure for managing hybrid geoscientific AI workflows." ISPRS International Journal of Geo-Information 9, no. 2 (2020): 119.

Sun, Ziheng, Liping Di, Jason Tullis, Annie Bryant Burgess, and Andrew Magill. "Geoweaver: Connecting Dots for Artificial Intelligence in Geoscience." In AGU Fall Meeting Abstracts, vol. 2020, pp. IN011-02. 2020.

License

The MIT License (MIT)

Copyright (c) 2015 Chris Kibble

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

geoweaver's Issues

The right panel will hide when the screen is small

Keep it on no matter how small the screen is.

Fail to save file in the browser

After click save, it says "saved" but the file doesn't change.

Cannot download files without suffix

In the file browser function of SSH server, people cannot download files without suffix.

How to reproduce it?

1. Choose one host under "SSH Server" list.
2. Click "Browse File Hierarchy"
3. Enter password in the popup dialog
4. In the file hierarchy, find a file without suffix, such as XYZ
5. Click the file name. The downloading won't start.

Screenshot

Improve the user guide to the latest version

Use some library to do the automatic documentation.

Allow to run the code in snippet to get response in time

Create a process execution dialog similar to jupyter notebook, while not requiring the complicated jupyter logic.

Enable the SSH console

SSH console is important for having real-time instant control over the distributed resources.

Fix the frozen jupyter cell on long-running tasks

Priority issue need be fixed.
@andrewmagill @ZihengSun

It takes too long to wait for the response of failed login attempts

There is some place stuck.

Add a progress bar list to indicate the progress

Add a progress bar list.

Increase z level for modal windows

I think the desired behavior would be for all modals to overlay on top of non-modal UI elements, please increase z level to bring all modals to the front.

Collect successful examples for demo and boosting user confidences

One feedback from ESIP summer meeting 2019, one feedback is saying "Geoweaver need more successful example" to attract more community users.

Create a tool to upload files from local to the servers

Now only the downloading channel is supported. The reverse, uploading, should be there as well. It is an essential function for Geoweaver.

workflow "ends too quickly"

When running a workflow, Geoweaver provides the following messages in quick succession:

00:00:01 connected
00:00:01 It ends too quickly. Go to history to check the logs out.
00:00:01 disconnected

However, the workflow executes correctly on the remote server.

Test Geoweaver with Dask cluster on Azure Kubernetes

As title said.

Save the unfinished jobs into the history table

Documentation about how to use Geoweaver with Dask cluster on Kubernetes

Detailed tutorials are needed to teach people how to use Geoweaver on Dask cluster.

Create a TOC

TOC is very important when the document is too long.

Default database change from mysql to h2

The database of Geoweaver won't be large. MySQL is a overkill. H2 will be much more lightweight, efficient and scientist-friendly. The change is necessary.

How to check the status of running processes

The running process has no sign after it is started. It should has an record in the history showing the status of running.

XSRF cookie does not match POST argument

When trying to login in a jupyter at the first time, the login page will return this message.

Simple solution is clearing all the browser cache and trying to login again.

Better solution is required.

Small fix to README

Under Quick Install for Mac, incorrect port in link:

Should be

http://127.0.0.1:3030/Geoweaver/web/geoweaver

Currently

http://127.0.0.1:8080/Geoweaver/web/geoweaver

Add builtin function show() to display results and inputs for comparison

The process in the workspace is not selectable

Cannot select the circle, the monitor is not working neither.

Add Jupyter Notebook Client into Geoweaver

To allow scientists to feel at home when using Geoweaver to run their jupyter notebooks. This function will also enable the support of Pangeo in Geoweaver.

The newly created workflow cannot be monitored

A bug it seems.

Improve the installation package

Use h2 and jetty to make it a downloadable install file on all three mainstream operation systems.

Don't close SSH session after the logging window is closed

Backspace not working in new process dialog name field

When creating a new process, the name field does not seem to recognize the backspace key. To correct an error, you must highlight and replace. I have testing with Chrome and Firefox on Mac and Linux.

SSH Server not showing up after being newly added

As title said.

Resize the circle to fit the length of the process name

The text should be completed wrapped inside.

Build in some usual scripts for installing deep learning libraries

Some bash scripts that are universal to install dependencies and create environment for deep learning will be very useful for everyone.

When the log out is too long, the save function has problem

There is limit on the mysql column length.

Add name/notes to the history versions of jupyter notebooks

Allow users to label or tag the version.

Add a button to remove notebooks without tags (non-important)

Make the proxy of JupyterHub work

The oauth doesn't work for authorizing the websocket session. The kernel keeps dying and asks for restart.

Support running Jupyter Notebook as a process

The process should allow run of jupyter notebooks and show the results.

Uploading workflow is not working

Error parsing uploaded file
error message: d.source is undefined

A workflow of North Dakota Crop Map generation

Use the trained network on 2016.

Create installable software release besides the tomcat and docker

Exe for windows, dmg for Mac, deb for Linux.

Provide option to remember password in one session

It is annoying to input password every time.

cannot upload jupyter notebook

When creating a new process, UI elements allowing uploading or importing of Jupyter Notebooks are not present.

(This was fixed with Ziheng's PR #58, just adding the issue for the record. Please close.)

JupyterLab doesn't work with Geoweaver

The content is not showing up. All the buttons don't work.

Add logo

Create a file browser by SFTPClient

A file browser to view the files within the host. Password protected.

Use NoVNC or Spice to replace the SSH console

NoVNC is more stable and powerful to control the server. But still no sure if this is a good move. Should go deep on what Geoweaver is really pursuing.

Keep outputing log for workflow

At present, once the first process is finished, the log output will interrupt.

Fail to install on Lubuntu linux

The following error pops up:

org.h2.jdbc.JdbcSQLNonTransientException: IO Exception: "java.net.UnknownHostException: ggs681-VBox: ggs681-VBox: Name or service not known" [90028-199] at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) ~[h2-1.4.199.jar!/:na] at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ~[h2-1.4.199.jar!/:na]

Show the output in a fixed window in the workspace

The output should keep going after the window is minimized.

issue in windows installation script for windows quick install

the windows installation script employs the use of Expand-Archive which contains extraction path and Destination Path both of which could be encapsulated within Quotation marks if either of the string contains space in between folder name, gives out parameter binding exception error which could be solved by above said method.

C:/Users/test cap/ would give out error which can be solved by
"C:/Users/test cap/" binding it in between quotation marks

H2 database empty

H2 database seems to be empty after Linux quick install.

Database files are moved to ~/ by install script. Can connect only using the default H2 credentials:

username: sa
password: [blank]

(username: geoweaver, password: geoweaver does not work)

After connecting, no tables exist.

I am new to Java, H2, etc., so this may be user error.

Thank you.

Cannot download/upload big files to jupyter via Geoweaver

When the files are big, e.g., 1gb, Geoweaver will fail.