functionstream/function-stream

Function Stream - Event-streaming function platform based on Apache Pulsar and WebAssembly

Function Stream

Function stream is an event-streaming function platform based on Apache Pulsar and WebAssembly. It enables efficient and scalable processing of data streams by leveraging the power of WebAssembly. Function Stream provides seamless integration with Apache Pulsar, allowing users to take full advantage of its robust messaging capabilities.

Features

1. Support for Multiple Programming Languages: Function Stream aims to provide the capability to write code using multiple programming languages. This allows developers to use their preferred language and harness its specific strengths while working with Function Stream.
2. High Performance and Throughput: Function Stream is designed to deliver high performance and handle substantial throughput. It strives to optimize resource utilization and minimize latency, enabling efficient execution of code and processing of data.
3. Isolated Environment: Function Stream offers an isolated environment for executing code. This ensures that each function runs independently, without interference from other functions or external factors. The isolation enhances the security, reliability, and predictability of code execution.
4. Scalability and Fault Tolerance: Function Stream focuses on scalability by offering the ability to effortlessly scale up or down based on workload demands. Additionally, it emphasizes fault tolerance, ensuring that system failures or errors do not disrupt the overall functioning of the platform.
5. Support for Complex Data Schema: Function Stream acknowledges the need to handle diverse data types and formats. It provides support for complex data schema, including bytes data and JSON format data, among others. This versatility enables developers to process and manipulate data efficiently within the platform.
6. Stateful/Stateless Computing: Function Stream caters to both stateful and stateless computing requirements. It accommodates scenarios where functions require maintaining state between invocations as well as situations where a stateless approach is more suitable. This flexibility allows developers to implement the desired architectural patterns.
7. Cross-Architecture Platform Execution: Function Stream aims to be a cross-architecture platform capable of executing code across different hardware architectures seamlessly. It provides compatibility and portability, allowing developers to run their code on various platforms without concerns about underlying hardware dependencies.

Architecture and Components

Function Stream is composed of three main components: the WebAssembly runtime engine, the Pulsar client, and the Function Stream service. The following figure shows the overview of the Function Stream architecture.

The WebAssembly runtime engine is responsible for executing the WebAssembly modules that implement the stream processing logic. The runtime engine supports an interface for the underlying wasm runtime library. We use wazero as the WebAssembly runtime library, as they are both fast and lightweight. The WebAssembly runtime engine communicates with the Pulsar client through standard IO and file systems.

The Pulsar client is responsible for consuming and publishing the messages from and to the Apache Pulsar cluster. We use Pulsar Go client, which is a pure go implementation of the pulsar client library, to interact with the Pulsar brokers. The Pulsar client handles the data schema, the message metadata, and the processing guarantees of the messages.

The Function Stream service is responsible for managing the lifecycle and coordination of the WebAssembly instances.

Directory Structure

The Function Stream project is organized as follows:

├── LICENSE                 # The license for Function Stream
├── Makefile                # Contains build automation and commands
├── README.md               # README file for the project
├── benchmark               # Contains benchmarking tools or results
├── bin                     # Contains compiled binary files
├── cmd                     # Contains the command line executable source files
├── common                  # Contains common utilities and libraries used across the project
├── docs                    # Documentation for the project
├── examples                # Example configurations, scripts, and other reference materials
├── go.mod                  # Defines the module's module path and its dependency requirements
├── go.sum                  # Contains the expected cryptographic checksums of the content of specific module versions
├── fs                      # Core library files for Function Stream
├── license-checker         # Tools related to checking license compliance
├── openapi.yaml            # API definition file
├── perf                    # Performance testing scripts
├── restclient              # REST client library
├── server                  # Server-side application source files
└── tests                   # Contains test scripts and test data

Building Instructions

To compile Function Stream, use this command:

make build-all

This creates the function-stream binary program and example wasm files in the bin directory, like bin/example_basic.wasm.

Running Instructions

You have two ways to start the function stream server.

Option 1: Standalone Mode (for development and testing)

Use this command to start the standalone server:

bin/function-stream standalone

Option 2: Server Mode (for production)

First, start an Apache Pulsar service. See this doc for instructions.

Then, use this command to start the server based on Apache Pulsar:

bin/function-stream server

Creating a Function

We'll use example_basic.wasm as an example wasm file. This function increases the money by 1. See the code here.

After starting the server, create a function with this command:

bin/function-stream client create -n example -a "bin/example_basic.wasm" -i example-input -o example-output -r 1

This creates a function named example using example_basic.wasm. It takes messages from example-input, produces messages to example-output, and runs with 1 replica.

Consuming a Message from the Function Output

After creating the function, consume a message from the output topic with this command:

bin/function-stream client consume -n example-output

Producing a Message to the Function Input

In a new terminal, produce a message to the input topic with this command:

bin/function-stream client produce -n example-input -c '{"name":"rbt","money":2}'

You'll see this log:

Event produced

Checking the Output

In the terminal where you consume the message from the output topic, you'll see this log:

"{\"name\":\"rbt\",\"money\":3,\"expected\":0}"

Deleting the Function

After testing, delete the function with this command:

bin/function-stream client delete -n example

Contributing

We're happy to receive contributions from the community. If you find a bug or have a feature request, please open an issue or submit a pull request.

License

This project is licensed under the Apache License 2.0.

function-stream's Issues

[Test] Add e2e test framework for frontend

[CI] Eslint check

We need to add a GitHub action to make eslint check on each PRs.

Function Hub

Frontend

• Function Hub frontend

Backend

• Funciton Hub manager

Refactoring: eslint

eslint

Refactoring: add function

Refactoring new functions

runtime/wazero error in windows environment

When command "bin/function-stream client consume -n example-output" is executed, the following problems occur:

Failed to parse JSON: invalid character 'n' looking for beginning of object key string
2024/03/26 20:15:11 ERROR Error calling process function name=example index=0 runtime=default component=function-instance error="module closed with exit_code(1)\nError calling wasm function\ngithub.com/functionstream/function-stream
/fs/runtime/wazero.(*WazeroFunctionRuntimeFactory).NewFunctionRuntime.func2\n\tD:/Github/function-stream/fs/runtime/wazero/wazero_runtime.go:91\ngithub.com/functionstream/function-stream/fs/runtime/wazero.(*WazeroFunctionRuntime).Ca
ll\n\tD:/Github/function-stream/fs/runtime/wazero/wazero_runtime.go:120\ngithub.com/functionstream/function-stream/fs.(*FunctionInstanceImpl).Run\n\tD:/Github/function-stream/fs/instance_impl.go:112\nruntime.goexit\n\tD:/Golang/src/
runtime/asm_amd64.s:1650"

Function flow controller

Refactoring: functions detail

show functions detail

The ConsumeEvent defines cancel but is not used

Improvement of the FunctionTable page

Currently, the FunctionTable page looks as below

We need to make some improvements on this page:

• Align the columns Status and Options to the right of the table. And make the contents of these two columns compactly display.
• Add a new option in Options columns: Start or Stop. When the function is running, the available options are Trigger | Detail | Stop | Delete. And when the function is down, the available options are Trigger | Detail | Start | Delete.
• Write a mock API test to output both running function and down function in the function table.

About the Implementation Logic of SendToChannel Function

func SendToChannel[T any](ctx context.Context, c chan<- T, e interface{}) bool {
select {
case c <- e.(T): // It will panic if e is not of type T or a type that can be converted to T.
return true
case <-ctx.Done():
close(c)
return false
}
}

I think we need to implement "case" priority control. If chan has a buffer, when "c<- e. (T)" and "<- ctx. Done()" both satisfy simultaneously, "select" will randomly execute one of them. What should our goal be?

1. After ctx is cancelled, "c <- e.(T)" can still be executed, and then "<-ctx.Done()" can be executed to close chan.
2. When ctx is cancelled, priority is given to execution, that is, "<-ctx.Done()" has a higher priority

The ConsumeEvent defines cancel but is not used

https://github.com/FunctionStream/function-stream/blob/main/fs/manager.go#L235
Why is a "cancel" defined but not used? There seems to be only "defer cancel()", and no logic to use "cancel. But the "NewSourceTube" called in "ConsumeEvent" needs to do its job internally via "<-ctx.Done()". Then this "cancel" doesn't seem to control the "goroutine in the "NewSourceTub". So the "ctx" with the "cancel" condition should be passed upstream from the "ConsumeEvent" or complete the "cancel" logic in the "ConsumeEvent".

Function exception handler

Flaky test: BenchmarkStressForBasicFuncWithMemoryQueue

Failure: https://github.com/FunctionStream/function-stream/actions/runs/8130116568/job/22218004872?pr=156#step:8:405

panic: pebble: closed

goroutine 485117 [running]:
github.com/cockroachdb/pebble.(*DB).Close(0xc0002ad200)
	/home/runner/go/pkg/mod/github.com/cockroachdb/[email protected]/db.go:1573 +0xb74
github.com/functionstream/function-stream/fs/statestore.(*PebbleStateStore).Close(0x0?)
	/home/runner/work/function-stream/function-stream/fs/statestore/pebble.go:88 +0x17
github.com/functionstream/function-stream/fs.(*FunctionManager).Close(0xc004c3a200)
	/home/runner/work/function-stream/function-stream/fs/manager.go:282 +0x113
github.com/functionstream/function-stream/server.(*Server).Close(0xc004c3a2a0)
	/home/runner/work/function-stream/function-stream/server/server.go:415 +0x78
github.com/functionstream/function-stream/server.(*Server).Run.func1()
	/home/runner/work/function-stream/function-stream/server/server.go:158 +0x3a
created by github.com/functionstream/function-stream/server.(*Server).Run in goroutine 486374
	/home/runner/work/function-stream/function-stream/server/server.go:156 +0xcb

Function logging support

Motivation

It's an important need to support logging for the function. We need these logs to check if the function is working correctly, and in case of failure, help identify the problem.

Goals

• Support logging in the wasm function
• Support accessing logs from the function-stream

Implementation

API Changes

• We could use the stderr stream to get logs from the wasm function
• Add new rest endpoints for function-stream: /api/v1/function/{function_name}/logs. This will return all the logs from the function

Notes

We need to support accessing the logs while the function is executing.

Refactoring: trigger functions

refactoring trigger functions

• trigger api

Data schema support

Motivation

Our wasm function requires knowledge of the data schema to parse JSON data.

Goals

• Support transfer of the data schema from function-stream host to wasm function

Implementation

We can employ pulsar-client-go to extract the schema in JSON format from the message.

One possible approach is to mount a specific file, say /schema, to the wasm function's wasi. We can then utilize the buffer_reader to write to this file.

When the wasm function wants to access the schema, we could transfer the schema through this file interface.

Refactoring: router controller

Refactoring: router controller

Add tests for different OS env

Motivation

Support running CI tests under different OS env like windows, ubuntu, macos etc.

Multiple closes on the same result channle in MemoryQueueFactory

Why put "defer close (result)" inside the goroutine? This may lead to multiple goroutines attempting to close the result, causing panic. https://github.com/FunctionStream/function-stream/blob/main/fs/contube/memory.go#L80
Perhaps it can be managed through "sync. WaitGroup", for example:

func (f *MemoryQueueFactory) NewSourceTube(ctx context.Context, configMap ConfigMap) (<-chan Record, error) {  
	config := NewSourceQueueConfig(configMap)  
	result := make(chan Record)  
	var wg sync.WaitGroup  
	for _, topic := range config.Topics {  
		wg.Add(1)   
		t := topic  
		go func() {  
			<-ctx.Done()      
			f.release(t)      
		}()  
  
		go func() {  
			defer wg.Done()   
			c := f.getOrCreateChan(t)  
			for {  
				select {  
				case <-ctx.Done():  
					return   
				case event := <-c:  
					result <- event   
				}  
			}  
		}()  
	}  
  
	go func() {  
		wg.Wait() 
		close(result) 
	}()  
  
	return result, nil  
}

Function Flow

Frontend

• #100
• #103
• #108
• Some Function image component for testing
• Generation for the deploy definition
• Context menu

Backend

• #97
• #98
• • Deploy definition parser
• • K8s connector
• • REST handler
• #112
• #113
• Function example images

Function flow manager

Create a function flow staging project in the `front-end` folder

Yarn build error

When run yarn run build, there are some errors. We need to fix these errors.

yarn run v1.22.10
warning package.json: No license field
$ vue-tsc --noEmit && vite build
src/components/Menu.vue:4:12 - error TS2571: Object is of type 'unknown'.

4       :is="item.children ? 'ElSubmenu' : 'ElMenuItem'"
             ~~~~

src/views/function/components/Func.vue:3:30 - error TS2304: Cannot find name '$t'.

3     <el-table-column :label="$t('func.table.Name')" prop="name" />
                               ~~

src/components/Menu.vue:6:13 - error TS2571: Object is of type 'unknown'.

6       :key="item.path"
              ~~~~

src/views/function/index.vue:5:12 - error TS2304: Cannot find name '$t'.

5         {{ $t('func.addFunc') }}
             ~~

src/views/function/components/Func.vue:4:30 - error TS2304: Cannot find name '$t'.

4     <el-table-column :label="$t('func.table.Status')" prop="status">
                               ~~

src/components/Menu.vue:7:15 - error TS2571: Object is of type 'unknown'.

7       :index="item.path"
                ~~~~

src/components/PageHeaderWrapper.vue:5:12 - error TS2304: Cannot find name '$t'.

5         {{ $t(router.name) }}
             ~~

src/components/Menu.vue:10:17 - error TS2571: Object is of type 'unknown'.

10       <i v-if="!item.children" :class="item.meta.icon"></i>
                   ~~~~

src/components/Menu.vue:10:40 - error TS2571: Object is of type 'unknown'.

10       <i v-if="!item.children" :class="item.meta.icon"></i>
                                          ~~~~

src/components/Menu.vue:12:18 - error TS2571: Object is of type 'unknown'.

12         <i v-if="item.children" :class="item.meta.icon"></i>
                    ~~~~

src/components/Menu.vue:12:41 - error TS2571: Object is of type 'unknown'.

12         <i v-if="item.children" :class="item.meta.icon"></i>
                                           ~~~~

src/components/PageHeaderWrapper.vue:9:42 - error TS2304: Cannot find name '$t'.

9       <span class="text-xl font-bold">{{ $t(router.name) }}</span>
                                           ~~

src/components/Menu.vue:13:18 - error TS2304: Cannot find name '$t'.

13         <span>{{ $t(item.name) }}</span>
                    ~~

src/components/Menu.vue:13:21 - error TS2571: Object is of type 'unknown'.

13         <span>{{ $t(item.name) }}</span>
                       ~~~~

src/components/Menu.vue:15:24 - error TS2571: Object is of type 'unknown'.

15       <SubMenu :menus="item.children" />
                          ~~~~

src/components/Header.vue:13:22 - error TS2304: Cannot find name '$i18n'.

13           {{ LangMap[$i18n.locale] }}
                        ~~~~~

src/views/function/components/FunctionDetail.vue:12:56 - error TS2554: Expected 0 arguments, but got 1.

12             <el-button size="small" @click="cancelEdit('info')"> Cancel </el-button>
                                                          ~~~~~~

src/views/function/components/Func.vue:18:30 - error TS2304: Cannot find name '$t'.

18     <el-table-column :label="$t('func.table.Options')" prop="options">
                                ~~

src/views/function/components/FunctionDetail.vue:17:37 - error TS2339: Property 'name' does not exist on type '{}'.

17             <el-input v-model="info.name" readonly />
                                       ~~~~

src/views/function/components/FunctionDetail.vue:30:37 - error TS2339: Property 'className' does not exist on type '{}'.

30             <el-input v-model="info.className" :readonly="!editable" :class="{ editable: !editable }" />
                                       ~~~~~~~~~

src/views/function/components/Func.vue:33:31 - error TS2304: Cannot find name '$t'.

33       <el-empty :description="$t('comp.emptyData')"></el-empty>
                                 ~~

src/views/function/components/FunctionDetail.vue:36:24 - error TS2339: Property 'key' does not exist on type 'never'.

36             :key="item.key"
                          ~~~

src/views/function/components/FunctionDetail.vue:41:37 - error TS2339: Property 'input' does not exist on type 'never'.

41             <el-input v-model="item.input" :disabled="true" :class="{ editable: !editable }" />
                                       ~~~~~

src/views/function/components/FunctionDetail.vue:46:37 - error TS2339: Property 'output' does not exist on type '{}'.

46             <el-input v-model="info.output" :readonly="!editable" :class="{ editable: !editable }" />
                                       ~~~~~~

node_modules/element-plus/lib/el-cascader-panel/src/node.d.ts:4:20 - error TS2304: Cannot find name 'Nullable'.

4     readonly data: Nullable<CascaderOption>;
                     ~~~~~~~~

node_modules/element-plus/lib/el-cascader-panel/src/node.d.ts:22:23 - error TS2304: Cannot find name 'Nullable'.

22     constructor(data: Nullable<CascaderOption>, config: CascaderConfig, parent?: Node, root?: boolean);
                         ~~~~~~~~

node_modules/element-plus/lib/el-cascader-panel/src/types.d.ts:50:20 - error TS2304: Cannot find name 'Nullable'.

50     expandingNode: Nullable<CascaderNode>;
                      ~~~~~~~~

node_modules/element-plus/lib/el-form/src/token.d.ts:28:11 - error TS2304: Cannot find name 'ComponentSize'.

28     size: ComponentSize;
             ~~~~~~~~~~~~~

node_modules/element-plus/lib/el-popper/src/use-popper/defaults.d.ts:7:34 - error TS2304: Cannot find name 'Nullable'.

7 export declare type RefElement = Nullable<HTMLElement>;
                                   ~~~~~~~~

node_modules/element-plus/lib/el-select-v2/index.d.ts:93:116 - error TS2344: Type '"toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | "reduce" | ... 38 more ... | ((<A, D extends number = 1>(this: A, depth?: D | undefined) => FlatArray<...>[]) & string)' does not satisfy the constraint 'number | "toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | ... 17 more ... | "class"'.
  Type '(() => string) & string' is not assignable to type 'number | "toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | ... 17 more ... | "class"'.
    Type '(() => string) & string' is not assignable to type '"class"'.

 93         }> & import("vue").VNodeProps & import("vue").AllowedComponentProps & import("vue").ComponentCustomProps), "length" | "toString" | "toLocaleString" | "concat" | "join" | "slice" | "indexOf" | "lastIndexOf" | "every" | "some" | "forEach" | "map" | "filter" | "reduce" | "reduceRight" | "find" | "findIndex" | "entries" | "keys" | "values" | "includes" | "flatMap" | "flat" | "key" | "ref" | "onVnodeBeforeMount" | "onVnodeMounted" | "onVnodeBeforeUpdate" | "onVnodeUpdated" | "onVnodeBeforeUnmount" | "onVnodeUnmounted" | "class" | "style" | ((() => string) & string) | ((() => string) & string) | ({
                                                                                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 94             (...items: ConcatArray<never>[]): never[];
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
112             (predicate: (value: never, index: number, obj: readonly never[]) => unknown, thisArg?: any): never;
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
113         } & string) | (((predicate: (value: never, index: number, obj: readonly never[]) => unknown, thisArg?: any) => number) & string) | ((() => IterableIterator<[number, never]>) & string) | ((() => IterableIterator<number>) & string) | ((() => IterableIterator<never>) & string) | (((searchElement: never, fromIndex?: number) => boolean) & string) | ((<U_3, This = undefined>(callback: (this: This, value: never, index: number, array: never[]) => U_3 | readonly U_3[], thisArg?: This) => U_3[]) & string) | ((<A, D extends number = 1>(this: A, depth?: D) => FlatArray<A, D>[]) & string)>) | (Partial<{}> & Pick<(Readonly<{
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/element-plus/lib/el-select-v2/index.d.ts:97:128 - error TS2526: A 'this' type is available only in a non-static member of a class or interface.

97             <S extends never>(predicate: (value: never, index: number, array: readonly never[]) => value is S, thisArg?: any): this is readonly S[];
                                                                                                                                  ~~~~

node_modules/element-plus/lib/el-select-v2/index.d.ts:197:116 - error TS2344: Type '"toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | "reduce" | ... 38 more ... | ((<A, D extends number = 1>(this: A, depth?: D | undefined) => FlatArray<...>[]) & string)' does not satisfy the constraint 'number | "toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | ... 17 more ... | "class"'.
  Type '(() => string) & string' is not assignable to type 'number | "toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | ... 17 more ... | "class"'.
    Type '(() => string) & string' is not assignable to type '"class"'.

197         }> & import("vue").VNodeProps & import("vue").AllowedComponentProps & import("vue").ComponentCustomProps), "length" | "toString" | "toLocaleString" | "concat" | "join" | "slice" | "indexOf" | "lastIndexOf" | "every" | "some" | "forEach" | "map" | "filter" | "reduce" | "reduceRight" | "find" | "findIndex" | "entries" | "keys" | "values" | "includes" | "flatMap" | "flat" | "key" | "ref" | "onVnodeBeforeMount" | "onVnodeMounted" | "onVnodeBeforeUpdate" | "onVnodeUpdated" | "onVnodeBeforeUnmount" | "onVnodeUnmounted" | "class" | "style" | ((() => string) & string) | ((() => string) & string) | ({
                                                                                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
198             (...items: ConcatArray<never>[]): never[];
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
216             (predicate: (value: never, index: number, obj: readonly never[]) => unknown, thisArg?: any): never;
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
217         } & string) | (((predicate: (value: never, index: number, obj: readonly never[]) => unknown, thisArg?: any) => number) & string) | ((() => IterableIterator<[number, never]>) & string) | ((() => IterableIterator<number>) & string) | ((() => IterableIterator<never>) & string) | (((searchElement: never, fromIndex?: number) => boolean) & string) | ((<U_3, This = undefined>(callback: (this: This, value: never, index: number, array: never[]) => U_3 | readonly U_3[], thisArg?: This) => U_3[]) & string) | ((<A, D extends number = 1>(this: A, depth?: D) => FlatArray<A, D>[]) & string)>);
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/element-plus/lib/el-select-v2/index.d.ts:201:128 - error TS2526: A 'this' type is available only in a non-static member of a class or interface.

201             <S extends never>(predicate: (value: never, index: number, array: readonly never[]) => value is S, thisArg?: any): this is readonly S[];
                                                                                                                                   ~~~~

node_modules/element-plus/lib/el-select-v2/index.d.ts:394:116 - error TS2344: Type '"toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | "reduce" | ... 38 more ... | ((<A, D extends number = 1>(this: A, depth?: D | undefined) => FlatArray<...>[]) & string)' does not satisfy the constraint 'number | "toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | ... 17 more ... | "class"'.
  Type '(() => string) & string' is not assignable to type 'number | "toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | ... 17 more ... | "class"'.
    Type '(() => string) & string' is not assignable to type '"class"'.

394         }> & import("vue").VNodeProps & import("vue").AllowedComponentProps & import("vue").ComponentCustomProps), "length" | "toString" | "toLocaleString" | "concat" | "join" | "slice" | "indexOf" | "lastIndexOf" | "every" | "some" | "forEach" | "map" | "filter" | "reduce" | "reduceRight" | "find" | "findIndex" | "entries" | "keys" | "values" | "includes" | "flatMap" | "flat" | "key" | "ref" | "onVnodeBeforeMount" | "onVnodeMounted" | "onVnodeBeforeUpdate" | "onVnodeUpdated" | "onVnodeBeforeUnmount" | "onVnodeUnmounted" | "class" | "style" | ((() => string) & string) | ((() => string) & string) | ({
                                                                                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
395             (...items: ConcatArray<never>[]): never[];
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
413             (predicate: (value: never, index: number, obj: readonly never[]) => unknown, thisArg?: any): never;
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
414         } & string) | (((predicate: (value: never, index: number, obj: readonly never[]) => unknown, thisArg?: any) => number) & string) | ((() => IterableIterator<[number, never]>) & string) | ((() => IterableIterator<number>) & string) | ((() => IterableIterator<never>) & string) | (((searchElement: never, fromIndex?: number) => boolean) & string) | ((<U_3, This = undefined>(callback: (this: This, value: never, index: number, array: never[]) => U_3 | readonly U_3[], thisArg?: This) => U_3[]) & string) | ((<A, D extends number = 1>(this: A, depth?: D) => FlatArray<A, D>[]) & string)>) | (Partial<{}> & Pick<(Readonly<{
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/element-plus/lib/el-select-v2/index.d.ts:398:128 - error TS2526: A 'this' type is available only in a non-static member of a class or interface.

398             <S extends never>(predicate: (value: never, index: number, array: readonly never[]) => value is S, thisArg?: any): this is readonly S[];
                                                                                                                                   ~~~~

node_modules/element-plus/lib/el-select-v2/index.d.ts:498:116 - error TS2344: Type '"toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | "reduce" | ... 38 more ... | ((<A, D extends number = 1>(this: A, depth?: D | undefined) => FlatArray<...>[]) & string)' does not satisfy the constraint 'number | "toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | ... 17 more ... | "class"'.
  Type '(() => string) & string' is not assignable to type 'number | "toString" | "toLocaleString" | "concat" | "indexOf" | "lastIndexOf" | "slice" | "length" | "includes" | "map" | "style" | "filter" | "join" | "every" | "some" | "forEach" | ... 17 more ... | "class"'.
    Type '(() => string) & string' is not assignable to type '"class"'.

498         }> & import("vue").VNodeProps & import("vue").AllowedComponentProps & import("vue").ComponentCustomProps), "length" | "toString" | "toLocaleString" | "concat" | "join" | "slice" | "indexOf" | "lastIndexOf" | "every" | "some" | "forEach" | "map" | "filter" | "reduce" | "reduceRight" | "find" | "findIndex" | "entries" | "keys" | "values" | "includes" | "flatMap" | "flat" | "key" | "ref" | "onVnodeBeforeMount" | "onVnodeMounted" | "onVnodeBeforeUpdate" | "onVnodeUpdated" | "onVnodeBeforeUnmount" | "onVnodeUnmounted" | "class" | "style" | ((() => string) & string) | ((() => string) & string) | ({
                                                                                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
499             (...items: ConcatArray<never>[]): never[];
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
517             (predicate: (value: never, index: number, obj: readonly never[]) => unknown, thisArg?: any): never;
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
518         } & string) | (((predicate: (value: never, index: number, obj: readonly never[]) => unknown, thisArg?: any) => number) & string) | ((() => IterableIterator<[number, never]>) & string) | ((() => IterableIterator<number>) & string) | ((() => IterableIterator<never>) & string) | (((searchElement: never, fromIndex?: number) => boolean) & string) | ((<U_3, This = undefined>(callback: (this: This, value: never, index: number, array: never[]) => U_3 | readonly U_3[], thisArg?: This) => U_3[]) & string) | ((<A, D extends number = 1>(this: A, depth?: D) => FlatArray<A, D>[]) & string)>);
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

node_modules/element-plus/lib/el-select-v2/index.d.ts:502:128 - error TS2526: A 'this' type is available only in a non-static member of a class or interface.

502             <S extends never>(predicate: (value: never, index: number, array: readonly never[]) => value is S, thisArg?: any): this is readonly S[];
                                                                                                                                   ~~~~

node_modules/element-plus/lib/el-slider/src/slider.type.d.ts:23:17 - error TS2304: Cannot find name 'Nullable'.

23     firstValue: Nullable<number>;
                   ~~~~~~~~

node_modules/element-plus/lib/el-slider/src/slider.type.d.ts:24:18 - error TS2304: Cannot find name 'Nullable'.

24     secondValue: Nullable<number>;
                    ~~~~~~~~

node_modules/element-plus/lib/el-slider/src/slider.type.d.ts:25:15 - error TS2304: Cannot find name 'Nullable'.

25     oldValue: Nullable<number>;
                 ~~~~~~~~

node_modules/element-plus/lib/el-slider/src/slider.type.d.ts:69:48 - error TS2304: Cannot find name 'Nullable'.

69     [s in 'firstButton' | 'secondButton']: Ref<Nullable<ISliderButton>>;
                                                  ~~~~~~~~

node_modules/element-plus/lib/el-slider/src/slider.type.d.ts:89:18 - error TS2304: Cannot find name 'Nullable'.

89     tooltip: Ref<Nullable<HTMLElement>>;
                    ~~~~~~~~

node_modules/element-plus/lib/el-table/src/table/defaults.d.ts:23:30 - error TS2304: Cannot find name 'Nullable'.

23 declare type HoverState<T> = Nullable<{
                                ~~~~~~~~

node_modules/element-plus/lib/el-time-picker/src/common/props.d.ts:39:24 - error TS2304: Cannot find name 'ComponentSize'.

39         type: PropType<ComponentSize>;
                          ~~~~~~~~~~~~~

node_modules/element-plus/lib/el-tree/src/model/node.d.ts:28:24 - error TS2304: Cannot find name 'Nullable'.

28     get nextSibling(): Nullable<Node>;
                          ~~~~~~~~

node_modules/element-plus/lib/el-tree/src/model/node.d.ts:29:28 - error TS2304: Cannot find name 'Nullable'.

29     get previousSibling(): Nullable<Node>;
                              ~~~~~~~~

node_modules/element-plus/lib/utils/config.d.ts:2:11 - error TS2304: Cannot find name 'ComponentSize'.

2     size: ComponentSize;
            ~~~~~~~~~~~~~


Found 47 errors.

error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Update front-end readme

Docker image for function flow (Integrated in the helm chart)

Better collaboration and more enjoyable development

1. Design basic rules to make the project be managed well.
2. Optimize the code architecture (I recommend the hexagonal architecture).
3. Hold meetings to discuss features and make the requirements clear.
4. Use some good development tools such as Apifox.

Function flow helm chart

Dock menu

[Test] Add e2e test for front-end

Currently, we have e2e test running on our old front-end project: antv-front-end. Since we deprecated antv-front-end. Now we need to move this e2e test framework to new front-end project in the /front-end folder.

Use release and getOrCreateChan multiple times

In NewSourceTube and NewSinkTube, both getOrCreateChan and release are called, which can result in each topic being "GET" twice and "release" twice，such as :

API joint for pulsar backend

We need to make the frontend API caller be compatible with the pulsar backend.

Setup pulsar cluster: https://pulsar.apache.org/docs/en/standalone-docker/
API documentation for pulsar function: https://pulsar.apache.org/functions-rest-api/?version=2.7.2&apiversion=v3#tag/functions

The API includes:

• Get List of function
• Get function info
• Create function
• Update function
• Delete function
• Get function stats
• Get function status
• Start function
• Stop function

Function Node panel and the detail page

Get token function

This function is responsible for validating the user name and password. It will return a token to the user.
Input event is as below:

{
    "userName": "user-a",
    "password": "pwd"
}

Output event is as below:

{
    "token": "xxx"
}

This function should only care about the fields that need to be cared about and should not change fields that are not related to it.
For example, when the input is :

{
    "requestId": "1",
    "userName": "user-a",
    "password": "pwd"
}

It should output the event as below:

{
    "requestId": "1",
    "token": "xxx"
}

As you can see, it did not change the field requestId.
This function validates the user login information through a database like MySQL.

• Implement Get Token function
• Unit tests

Refactoring: add function

Refactoring new functions