teselagen / fsml.org Goto Github PK
View Code? Open in Web Editor NEWA BioMADE Collaboration Project
Home Page: https://fsml.org
A BioMADE Collaboration Project
Home Page: https://fsml.org
In regards to libFSML https://github.com/TeselaGen/fsml.org/milestone/3 , task indicates that we need to write the library in Python. However I think writing it in Deno and compiling it to WebAssembly (WASM) and then importing WASM into Python would provide us maximum flexibility because then we could consume it in the browser via JavaScript or on the server via JavaScript or Python or any other language that supports WASM (which is growing everyday).
Implement e2e CI tests for CLI tool using GitHub Actions.
Task 2 describes at a high level developing a draft for fermentation data. However for the past 6 months we've been focused on creating a specification for fermentation protocols.
I think we should have a planning discussion to better understand the requirements of the work and ensure our current work is heading in the right direction.
I propose using TypeBox with Deno to create the specification. Then compile that library to WASM for use in other languages such as Python.
The current draft specification is documented on the website fsml.org.
TypeBox - TypeBox is a type validation system built on TypeScript and JSON Schema. It provides the ability to have static (or transpile time) TypeScript validation of types as well as runtime JSON Schema validation of types. It can also generate a sinclairzx81/typebox#143 from the TypeScript types. It's available for both Node.js and Deno.
In addition to "regular" markdown we would like to support MDX v2 (markdown with JSX...aka React Components). I found an example of doing this in Node.js mdx-js/mdx#1985 (reply in thread) . We need to see if we can get the same thing working in Deno.
After careful evaluation, the following tech stack has been chosen:
Setup Repo with Nx #9
Setup an Example Deno CLI project #10
Create any necessary Nx plugins for Deno CLI project #9
Setup GitHub Actions CI/CD #12
Create a Proof of Concept with a trivial example that takes a Deno package, compiles it to WASM, and uses that WASM module in a Python library.
Related to #14
Currently Nx doesn't support Deno out of the box. However it has a flexible plugin system that will allow us to create plugins to support Deno packages. We have two ways to do this:
An inference plugin will crawl the directory structure and find directories that represent project directories and infer the project settings from files in that directory. Currently Deno packages can contain a config file such as deno.jsonc
that is similar to a package.json that we could look for in order to infer projects for Nx. The downside of this is that we're limited to what we can infer from the Deno config.
Instead of inferring projects, you can also place a project.json file in a folder to designate it as an Nx project. To help in creating these and maintaining them Nx provides a set of libraries to create generators and tools that will create and update projects. Instead of inferring projects from a Deno config we could use a generator to create a Deno package or app as well as the config and project file.
We need to start developing relationships with other developers who will be interested in both using and contributing to this schema and tool chain.
This ticket covers the creation of an interface that allows TG users to load a protocol from protocols.io.
The manifest schema is stored in the spec package. We need a package that will contain the functionality to use the schema to both generate and validate the schema.
We want this package to be isomorphic so it needs to not have any server or web specific dependencies such file system libraries. Instead data should be passed to this package as a stream.
The screenshot below shows a 96-96 well plate stamp in Dan Bryce's PAML editor. We should pick a preferred JS-based graph tool (see ticket #2) and generate a simple mockup of how the tool should be laid out... perhaps in analogy with Node-RED.
Here is a screenshot from Dan's PAML editor to give us an idea of its look and feel.
Here is a screenshot from Node-RED to give us an idea about its look and feel and the needed functional elements.
For protocols imported or indexed in our platform... users should have the option to enhance to such an extent that they are fully fleshed out workflows.
This is imagined as a way, for example, of converting protocols.io (or other) protocol "steps" into a set of placeholder tools in a TeselaGen workflow. The user would then convert each step to a proper tool or tools and link them as appropo.
Otherwise, for a simple pdf... it could just be a convenient side-by-side or standard browser tab view that could be torn off so the user could see the protocol steps while creating a workflow... (trivial example).
In any case, the workflow should be cross linked with the protocol.
i.e. in the protocol... a list of linked workflows (where this protocol is utilized).
i.e. in the workflow... a list of linked protocols (that this workflow is based on).
MVP rendering of a "static protocol" from JSON structure based on protocols.io
Protocols.io provides an obvious template for how to layout a protocol and associated metadata. This open-source "protocol viewer" can be developed in analogy with OVE, so it could be easily incorporated into TG as well as other tools, and we can solicit community support for its development.
The MVP is a read-only view of the protocol, but we will no doubt want to add the ability to edit soon thereafter.
We will work with the BioMADE community to develop a draft
standard for fermentation data taken at the manufacturing operating systems level. Datasets will be
represented in a structure that reveals the characteristics of individual data items and the relationships among
them. In addition, we will develop parser generation software that will auto-generate a parser that can accept
relatively unstructured fermentation data and convert to the standard.
The current way of updating the Gatsby site is cumbersome. You have to pull the code locally, edit the markdown, run a build process, then preview it locally, commit it and then merge it into the main branch. Additionally there's no way to share a preview of the changes with anyone else.
The compiled website needs to go in the /docs folder on the main branch. That's where we're serving the content for GitHub pages.
Currently the /website folder contains the Gatsby source.
It was identified that the manifest schema will need to cover fermentation data as well as protocol. Additionally we've identified upstream components such as en experiment definition and other associated information such as the equipment and execution environment needed to run a protocol that may also need be tracked by a manifest.
To accommodate this I propose adding a type field to the core schema as well as then creating a new set of type specific nested core settings for each type.
The utility scores will need to be nested under this type specific schema section since different types will have different utility scores.
We need to layout the command structure for the FSML CLI tool.
Here's what I have so far:
---
Global flags
--config/-c=[./config.json,filepath] #sets flags from config file
--stdin #reads inputs from stdin
---
help
version
create manifest
--type=[data,protocol]
--parser=[...parser plugin name]
--format/-f=[json,yaml,toml]
--write/-w=[filepath] # required if pack set / by default all will be sent to stdout
--pack=[zip,tar,tgz]
--author=[[email protected]] # required if not set in defaults
<filepattern...>
update manifest
--type=[data,protocol]
--parser/-p=[...parser plugin name]
--format/-f=[json,yaml,toml]
--write/-w=[filepath] # required if pack set / by default all will be sent to stdout
--pack/-z=[zip,tar,tgz]
--patch="{}" #inline json
--patch-file=[filepath] #patch json file
--patch-type=[json,json-merge,lodash-merge]
--author=[[email protected]] # required if not set in defaults
<path>
fetch manifest
--registry=<name> / --uri=<download uri>
--format/-f=[json,yaml,toml]
--unpack=[true, false]
--write/-w=[filepath] # required if pack set / by default all will be sent to stdout
<id>
import manifest
--from <registry name/alias> / --from-uri=<download uri>
--to <registry name/alias>
<id>
describe manifest
--output/-o=[record,json (pretty print),yaml,toml]
--write/-w=[filepath] # required if pack set / by default all will be sent to stdout
--select=[comma separated list of property paths]
--summary #only show summary
--section=[supplemental section name] #only show that section
<path>
score manifest
--formula=[latest,vx.x.x]
--dry-run (or --update)
--author=[[email protected]]
<path>
export manifest
--exporter=[...export plugin name]
--write/-w=[filepath] #by default all will be sent to stdout / maybe required by some exporters
[...--exporter flags]
<path>
validate manifest <path>
pack manifest
--pack=[zip,tar,tgz]
<path>
unpack manifest <path>
defaults edit # runs interactive default settings mode
default list # shows default flag values
defaults set <key> <value>
defaults reset <key> # resets key to installed default
defaults reset-all # resets all keys to installed default
--confirm
install plugin
--cache
<path, uri>
list plugin / list plugins
--type=[parser,exporter]
--filter=[glob]
--regex=[regex]
--sort=[asc,desc]
upgrade plugin
--latest / --version=[latest] / --minor / --major / --patch #one of these flags must be provided
--cache
<name>
uninstall plugin <name>
### Plugins will be "installed" by saving a reference to the plugin via a url however
### the plugin won't be actually downloaded until it's used. The cache command will
### force the plugin to be downloaded locally and cached
cache plugin <name>
cache plugins #cache all plugins
delete plugin cache # removes all plugins from local cache
(or clear plugin cache)
run <plugin name> ...args
add registry
--platform=[gh,git,s3,gcs,ftp,fsml-api,fs]
--alias=[alternate name]
<uri / path>
index registry <name>
remove registry <name>
search registry <name>
serve registry
--port=[3090]
--name=[fsml_<shortid>]
--properties=[./server-properties.json]
--middleware=[..list of plugins to use a middleware]
--allow-submissions
--temp-dir=[./tmp]
--index-dir=[./index]
<manifests location>
We've proposed a draft for the exchange protocols and now we need to expand upon that to also support exchanging data generated by those protocols such as execution traces and workflow runs.
We should take our existing Test Data Model and convert it into a standard.
Create a CLI project using Deno and Nx monorepo workspaces.
Deno - Deno is the "next" iteration of Node.js. It provides native TypeScript and es6 support. It's compatible with Node.js and supports isomorphic (runs on back and front end) code. Finally it has the advantage of being able to target all major OS's and compile to a single executable / CLI tool.
Nx - for monorepo support. Nx is fairly platform agnostic however it is missing tooling for nrwl/nx#5823 which we'll need to provide as plugins.
Example Deno CLI projects
Daniel Bryce at SIFT.net has created a nice alpha for a directed graph style protocol editor for PAML (see screenshot). We should investigate tooling that fits better with our pretty-much pure JS development environment. @tgadam has suggested the following:
Let's check them out and pick one to move forward as a provisional choice by the next BioMADE meeting on Feb 25.
Objective 2.1 - Develop a draft standard for fermentation process and protocol data: This
objective includes the creation of a specification, which we provisionally call the Fermentation Science Markup
Language or FSML. The FSML specifications define the syntax and semantics of FSML. These documents will
become the definitive references for FSML. The objective will also include the development and support of a
community website for sharing FSML developments and software along the lines of SBML.
Currently the GitHub pages site is on a separate branch and hosted from the root of the repo. I think in order to facilitate auto generated documentation on the website we should merge it into the main branch and put it in a subfolder.
Additionally I plan to rename the repo to all lowercase to match the patter of our other repos.
We want to have a predictable way to convert markdown to html. Showdown is one of the most popular packages that does this. We need to determine if it will work with Deno.
https://github.com/showdownjs/showdown
Here's the link to the Deno docs on how to use Node.js code in Deno https://deno.land/[email protected]/npm_nodejs
MVP Node-RED Flow for importing a complete protocol from protocols.io into TeselaGen.
protocols.io can provide us with a rich source of user vetted information on what constitutes the essential data and metadata needed to fully describe a scientific protocol. We will use this information as a shortcut to a JSON structure that will serve as a cornerstone for our open-source JSON representation of what traditionally constitutes a protocol, if not a complete workflow. One example of a data model based on protocols.io information, (but slightly extended to cover some other notions around fermentation) is shown:
@tgadam can you flesh this ticket out a bit?
@tgadam can we flesh this out a bit?
The current way in-app links are created in Gatsby makes it easy to create a broken link. We need a crawling test that can be run in a GitHub Action that will find broken links in the website and website previews (depends on #34 )
Develop appropriate tooling for reading, writing, and
manipulating files and data streams in FSML format. Written in Python and provides APIs for JavaScript,
Python and tools for binding standardized XML to Python, JSON objects. An additional objective in support of
the library is to develop an FSML Development Process that details the development process and community
organization procedures that are followed by the FSML community. Also in support of the library we will
develop a FSML Test Suite for conformance testing FSML.
Implement CLI Command to Generate Manifest for PDF with Core Sections. Depends on #35
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.