trymdev/springql

SpringQL: Open-source stream processor for IoT devices and in-vehicle computers

SpringQL

A stream processor for IoT devices and in-vehicle computers

What is SpringQL?

SpringQL is an open-source stream processor working with low and hard-limited working memory. It is supposed to run on resource-poor such as IoT devices and in-vehicle computers.

• SQL support

  Like other stream processors, infinite number of rows flow through SpringQL system. You can register SQL-like (continuous) queries to filter, transform, and analyze rows. SpringQL has the ability to make windows from stream of rows so that you can perform GROUP BY, JOIN, and even more operations targeting on finite number of rows.

• Small memory footprint

  SpringQL usually works with as low working memory as around 1MB. Even when processing heavy workload (e.g. with large size windows, or with rapid stream inputs), SpringQL tries to recover from high memory pressure by switching the internal task schedulers.

  You can set maximum memory allowed to consumed by SpringQL by configuration.

Why SpringQL?

• Realtime stream processing completed in edge

  Some computations uses real-time data, such as sensor data, emerged from edge devices themselves. If such computations need to calculate results quickly, you may not have time to send the data to servers and get the result back from them.

  For real-time processing in edge devices, SpringQL provides fundamental tools to time-series calculations including window operations via SQL-like language.

• Reducing data size before sending to servers

  If you would perform complex stream analysis, such as heavy machine learning, visualization, using data from edge devices, you may perform stream processing mainly on resource-rich servers.

  If you have so many edge devices and need to shrink costs for data transfer, storage, and computation in servers, you may want to shrink data size in edges before sending them to servers.

  SpringQL helps you reduce data size by filtering, aggregation, and projection.

How to use SpringQL?

You can use SpringQL currently in 2 ways:

1. Embedded mode: Link libspringql from your application binary and call API functions using a client library for each programming language.
2. Embedded mode (Rust): Almost the same as normal embedded mode but you statically link to springql-core crate from your Rust application.

IPC mode will be also planned. Multiple applications communicate with springqld daemon process so that they can share the same pipeline (stream processing definition).

Documentation

Read https://SpringQL.github.io/ for installation guide, tutorials, and references.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in SpringQL by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Copyright (c) 2022 TOYOTA MOTOR CORPORATION.