Jayvee

Jayvee is a domain-specific language (DSL) for automated processing of data pipelines. The Jayvee interpreter allows executing such data pipelines on local machines. Data engineers can use Jayvee and its interpreter to clean and preprocess data for later activities like data science or machine learning.

Roadmap

Explore a glimpse of our upcoming features in the following list. This overview is broad and subject to evolution. We're excited to share our vision of the exciting journey ahead, and we invite you to accompany us on this adventure!

✅ Blocks and pipes
✅ Simple valuetypes and constraints
✅ Natively support table-based data
✅ Column-based transformations
✅ Describe blocks via builtin blocktypes in Jayvee
✅ Compose logic of multiple blocks via composite blocktypes
⌛ Multi-file Jayvee to distribute programs over multiple files (see RFC 0015)
⌛ Improve the syntax of valuetypes (see RFC 0014)
⌛ Jayvee formatter
⌛ Further extractors and sinks
🤔 Reusable libraries (with a package manager)
🤔 Composite valuetypes (with multiple fields)
🤔 Natively support tree data (XML, JSON)
🤔 Valuetypes parsers (to read and write different formats)
🤔 Customizable invalid value handling (default value, average, median, interpolation, ...)
🤔 VSCode Debugger
🤔 Blocktypes with multiple ports (e.g., for merging different data)

Anything missing, or you have ideas how some of the items on the list could be approached? Feel free to create and issue and share your thoughts with us!

You like the project and our vision? Then we'd appreciate your star! ⭐

Contribute

In case you would like to contribute to Jayvee, please have a look at our contribution guide.

Development Quickstart

Run npm ci to install the dependencies.
Run npm run generate to generate TypeScript code from the Jayvee grammar definition and the standard library.
Run npm run build to compile all projects.
In Visual Studio Code, press F5 to open a new window with the Jayvee extension loaded.
Create a new file with a .jv file name suffix or open an existing file in the directory example.
Verify that syntax highlighting, validation, completion etc. are working as expected.
Run node dist/apps/interpreter/main.js to see options for the CLI of the interpreter; node dist/apps/interpreter/main.js <file> interprets a given .jv file.

In case you run into problems, make sure to use the current LTS version of Node.js and npm.

Projects overview

Name	Description	NPM package
`language-server`	Jayvee language definition and language server implementation	@jvalue/jayvee-language-server
`interpreter`	Command line tool for interpreting Jayvee files	@jvalue/jayvee-interpreter
`language-server-web-worker`	Ready-to-use Jayvee language server, bundled as a Web Worker	@jvalue/jayvee-language-server-web-worker
`vs-code-extension`	Visual Studio Code extension for editing Jayvee files	-
`docs`	Website for Jayvee user documentation	-
`monaco-editor`	React component for editing Jayvee files	@jvalue/jayvee-monaco
`execution`	Shared code for Jayvee extensions and the interpreter	-
`extensions/std`	Standard Jayvee extension consisting of the extensions below	-
`extensions/rdbms`	Jayvee extension for relational databases	-
`extensions/tabular`	Jayvee extension for tabular data	-

Scripts

Building all projects

npm run build

Linting all projects

npm run lint

Formatting project files via Nx

npm run format

Testing all projects

npm run test

Generating TypeScript code from the grammar definition

npm run generate

Examples

Load data about cars into a local SQLite DB

npm run example:cars

Load GTFS data into a local SQLite DB

npm run example:gtfs

Load data about electric vehicles into local Postgres and SQLite DBs

Start the Postgres database locally

docker compose -f ./example/docker-compose.example.yml up

Run the Jayvee model

npm run example:vehicles

joluj / jayvee Goto Github PK

jayvee's Introduction