infinyon/fluvio

Lean and mean distributed stream processing system written in rust and web assembly.

fluvio's Issues

Support for TLS

External public API should use TLS.

Components impacted: API (Node, Rust), CLI, SC, SPU

problem provisioning topic with replica of 2

topic with 2 replica:

kc get partition
NAME      LEADER   FOLLOWERS   STATUS    LSR   HW    EOF   FOLLOWER OFFSETS
test1-0   0        [0 1]       Offline   0     0     0     []

kc get spu
NAME     ID    STATUS   TYPE      PUBLIC HOST                                                            PRIVATE HOST                  PORT
main-0   0     Online   Managed   map[encryption:PLAINTEXT ingress:[map[ip:10.107.159.255]] port:9005]   flv-spg-main-0.flv-spg-main   9006
main-1   1     Online   Managed   map[encryption:PLAINTEXT ingress:[map[ip:10.106.230.223]] port:9005]   flv-spg-main-1.flv-spg-main   9006

seems like SPU-0 received topic with spu 0 as follower other way around

kc logs flv-spg-main-0
20:28:47.481 INFO  flv_spu::config::cli > extracting SPU from: flv-spg-main-0
20:28:47.481 INFO  flv_spu::config::cli > found SPU INDEX ID: 0
20:28:47.481 INFO  flv_spu::config::cli > using sc endpoint from env var: flv-sc-internal.default.svc.cluster.local:9004
20:28:47.482 INFO  flv_spu::config::cli > overriding log base: /tmp/fluvio
20:28:47.482 INFO  flv_spu::config::cli > overriding log size 1Gi
starting spu server (id:0)
SPU Version: 0.3.0 started successfully
20:28:47.482 INFO  flv_spu::services::public > starting SPU: 0 at public service at: 0.0.0.0:9005
20:28:47.482 INFO  flv_spu::services::internal > starting SPU: 0 at internal service at: 0.0.0.0:9006
20:28:47.483 INFO  kf_service::kf_server       > starting event loop for: 0.0.0.0:9006
20:28:47.483 INFO  kf_service::kf_server       > starting event loop for: 0.0.0.0:9005
20:28:56.882 WARN  flv_spu::controllers::leader_replica::connection > replica leader: test1-0 was not found
20:29:56.882 WARN  flv_spu::controllers::leader_replica::connection > replica leader: test1-0 was not found
20:30:56.883 WARN  flv_spu::controllers::leader_replica::connection > replica leader: test1-0 was not found
20:31:56.883 WARN  flv_spu::controllers::leader_replica::connection > replica leader: test1-0 was not found
20:32:56.884 WARN  flv_spu::controllers::leader_replica::connection > replica leader: test1-0 was not found
20:33:56.887 WARN  flv_spu::controllers::leader_replica::connection > replica leader: test1-0 was not found
20:34:56.886 WARN  flv_spu::controllers::leader_replica::connection > replica leader: test1-0 was not found
20:35:56.887 WARN  flv_spu::controllers::leader_replica::connection > replica leader: test1-0 was not found
20:36:56.888 WARN  flv_spu::controllers::leader_replica::connection > replica leader: test1-0 was not found

kc logs flv-spg-main-1
20:28:50.541 INFO  flv_spu::config::cli > extracting SPU from: flv-spg-main-1
20:28:50.541 INFO  flv_spu::config::cli > found SPU INDEX ID: 1
20:28:50.541 INFO  flv_spu::config::cli > using sc endpoint from env var: flv-sc-internal.default.svc.cluster.local:9004
20:28:50.541 INFO  flv_spu::config::cli > overriding log base: /tmp/fluvio
20:28:50.541 INFO  flv_spu::config::cli > overriding log size 1Gi
20:28:50.541 INFO  flv_spu::services::public > starting SPU: 1 at public service at: 0.0.0.0:9005
20:28:50.541 INFO  flv_spu::services::internal > starting SPU: 1 at internal service at: 0.0.0.0:9006
starting spu server (id:1)
SPU Version: 0.3.0 started successfully
20:28:50.542 INFO  kf_service::kf_server       > starting event loop for: 0.0.0.0:9005
20:28:50.543 INFO  kf_service::kf_server       > starting event loop for: 0.0.0.0:9006

Deleting topic should remove associated storage as well.

When topic is being deleted, it should go thru following stages:

• topic is marked for deletion ( in this case, consume/producer can't be performed). this is similar to pod lifecycle
• For each SPU (leader/follower), when received topic delete
  • ensure all consumer/leader connections have gracefully shutdown
  • delete local leader/follower replica storage.
• When a leader receives that all follow replica has been purged, it should send back deleted status for topic.

• When SC receives, deleted status, it will delete the topic.

CLI: misc cluster install changes

• change description of the SPU to number of SPU in the cluster install.
• for local cluster, change launching SPU as: launching SPU (2 of 4)
• show logs location for SC and SPU for local

built-in installer

built in installer to Fluvio CLI instead of using helm.

Benefits

• remove one more tool that user need to install
• better user experience

1.0.0

First production version

support for websocket

access fluvio using websocket protocol

DOC: Update CLI

We need to update the CLI documentation on the Fluvio docs website.

Review and update

Required for this issue:

• consume
• produce
• topic
• profile
• install
• update
• cluster start
• cluster delete

Future work:

• cluster spu
• cluster spg

Support for AWS EKS

To allow Fluvio to run on AWS EKS environment.

Topic LIVE-REPLICAS shows spu id 0 (but there is no such SPU installed)

$ ./target/debug/fluvio topic create test -p 1 -r 2
topic "test" created


$  ./target/debug/fluvio topic describe -t test
 Name                    :  test 
 Type                    :  computed 
 Partition Count         :  1 
 Replication Factor      :  2 
 Ignore Rack Assignment  :  - 
 Status                  :  provisioned 
 Reason                  :  - 
 Partition Map               
 -----------------           
     ID      LEADER      REPLICAS          LIVE-REPLICAS 
      0       5003       [5003, 5001]      [0, 5001] 


$  ./target/debug/fluvio spu list
 ID    NAME             STATUS  TYPE    RACK  PUBLIC          PRIVATE 
 5001  custom-spu-5001  online  custom   -    localhost:9010  localhost:9011 
 5002  custom-spu-5002  online  custom   -    localhost:9020  localhost:9021 
 5003  custom-spu-5003  online  custom   -    localhost:9030  localhost:9031 

Change "offset" command description in comsumer

The comment:

./target/debug/fluvio consume -h
...
-o, --offset <integer>             optional, offset, negate offset is relative to end offset (either committed or uncommitted)

should be changed to

./target/debug/fluvio consume -h
...
-o, --offset <integer>      Offsets can be positive or negative. (Syntax for negative offset: --offset="-1")

CI: enable workflow for testing k8 cluster

bug: offline replica is chosen as leader

Steps to reproduce:

Create 2 custom SPU:

kc apply -f k8-util/samples/crd/spu_5001.yaml
 kc apply -f k8-util/samples/crd/spu_5002.yaml

Bring up first SPU

 ./dev-tools/log/debug-spu-min 5001 9005 9006

The SPU list shows only first SPU is up

$ flvd spu list
 ID    NAME          STATUS   TYPE      RACK  PUBLIC          PRIVATE 
 5002  custom-spu-2  Offline  "custom"   -    localhost:9007  localhost:9008 
 5001  custom-spu-1  Online   "custom"   -    localhost:9005  localhost:9006

Now create topic with 2 replicas

flvd partition list
 TOPIC    PARTITION  LEADER  REPLICAS  RESOLUTION  HW  LEO  LSR  FOLLOWER OFFSETS 
 message  0          5001    [5002]    Online      0   0    0    [ReplicaStatus { spu: 5002, hw: -1, leo: -1 }] 

shows that online leader is used

Release: change log

Incorporate change log into release process.

Some good practice:

• https://github.com/kubernetes/minikube
• https://github.com/kubernetes/kubernetes
• https://github.com/rust-lang/rust/blob/master/RELEASES.md

Fluvio SC CLI doesn't work with localhost

Steps to reproduce:

fluvio spu list --sc localhost:9003
error: cannot retrieve spus: Connection refused (os error 61)

Note: IP address works fine

fluvio spu list --sc 127.0.0.1:9003
ID NAME STATUS TYPE RACK PUBLIC PRIVATE
5001 custom-spu-1 online custom - localhost:9005 localhost:9006
5002 custom-spu-2 online custom - localhost:9007 localhost:9008
5003 custom-spu-3 online custom - localhost:9009 localhost:9010

Misc items for open source library

• Separate contributor agreement
• Code of conduct

replicas should only contain follower for topic describe

Example:

 Name                    :  test1 
 Type                    :  computed 
 Partition Count         :  1 
 Replication Factor      :  2 
 Ignore Rack Assignment  :  - 
 Status                  :  provisioned 
 Reason                  :  - 
 Partition Map               
 -----------------           
     ID      LEADER      REPLICAS      LIVE-REPLICAS 
      0        0         [0, 1]        [1] 

Live replicas contains follow but replica contain leader and follower. They should be consistent.

move kf-protocol to separate repository

If 2 consumers listen for the same topic/partition, only one of them gets messages

Steps to reproduce:

1. Create topic:
   ./target/debug/fluvio topic create -t my-topic-1 -p 1 -r 2

2. Create first consumer
   ./target/debug/fluvio consume -t my-topic-1 -p 0 -C

3. Create second consumer
   ./target/debug/fluvio consume -t my-topic-1 -p 0 -C

4. Create producer
   ./target/debug/fluvio produce -t my-topic-1 -p 0 -C

type a few lines... only 1 consumer gets the messages.

Fluvio Github action

Github action to install and Run Fluvio

Support in memory metadata store

Build version of SC that works against local in memory metadata store. With in memory metadata store, we can build local cluster suitable for

• smoke test
• local development

The in memory store implements traits in: https://docs.rs/k8-metadata-client/1.0.1/k8_metadata_client/trait.MetadataClient.html.

Implementation should support full fluvio lifecycle:

• Create/Delete local cluster with CLI
• Run smoke test

CLI: show API versions

show client-rs version
show list of kf versions as well.

SPU: implement delay flush

Currently every write incur flush call. Delay flush based on following criteria

• time
• number of writes or bytes transferred

Bonus: should be configurable

Question on Developer.md

Hello,
I'm following process in Developer.md now. Seems:

$ ./dev-tools/log/debug-sc-min
./dev-tools/log/debug-sc-min: line 5: /Users/djKooks/workspace/etc/rust/fluvio/dev-tools/log/../../target/debug/sc-k8-server: No such file or directory
...
$ ./dev-tools/log/debug-spu-min 5001 9005 9006                          ⏎
./dev-tools/log/debug-spu-min: line 9: /Users/djKooks/workspace/etc/rust/fluvio/dev-tools/log/../../target/debug/spu-server: No such file or directory
...
$ ./target/debug/fluvio spu list --sc 127.0.0.1:9003                    ⏎
error: IoError(
    Custom {
        kind: ConnectionRefused,
        error: VerboseError {
            source: Os {
                code: 61,
                kind: ConnectionRefused,
                message: "Connection refused",
            },
            message: "could not connect to 127.0.0.1:9003",
        },
    },
)

Is there something wrong in my process?

add namespace support in the CLI

Rust Client: Improvements

• Better streamlined API
• Add examples
• Rust doc

CLI: change create/delete command to register/unregister

1. Change Spu Group column" NAME" as left justified.
2. Change create/delete command custom-spu to "register/unregister"

CLI: warn if release version is not available for cluster install

Check if version is available in fluvio helm repo before proceeding to install.

provide OS specific default storage location

in Mac OS/BSD, /tmp location are clearly every 3 days. should change default location to appropriate for each OS

SPU reconnect aggressively after couple of minutes.

Steps to reproduce:

• flvd cluster install --local --log DEBUG
• flvd topic create test1

After few minutes, SPU reconnect without delay, it should wait 60 seconds.

support local cluster install using cargo install

CLI should perform local cluster installation using cargo install by default.
So fluvio install --local should expand into

cargo install flv-sc-k8-server,
cargo install flv-spu-server

then, it can search binary in the cargo .bin directory. This requires

• both spu and sc are published in the crates
• can search binaries in the cargo .bin

It should support --path to specify local fluvio repo path. This will invoke cargo build and search binaries in the target directory.

Enforce MAX BYTES for transferring batch records

Fluvio uses zero copy I/o to transfer records from SPU to client.
Records to need to be limit since maximum bytes can be up to 1G for each segment.

Solution:

• SPU: Limit transferring file slice. This will create potentially truncated last batch
• Kf decoder: When decoding DefaultRecords, ignore any batch that can't be decoded
• Client: Use offset in the last batch to retrieve next batches for streaming cases
• SPU-SPU: Apply spu-message-size to limit bytes

HTTP protocol support

Access fluvio using HTTP protocol.
May follow Kafka HTTP Proxy url.

Test: Stress test framework phase 1

stress test configuration

• number of parallel producer
• number of parallel consumer
• number of SPU
• size of the records

Support ARM platform

C Library Client

C based library which can be used by any language.
This means we provide "C" export interface on top of the rust native library.

See: Mesa Link as example

keep cli release tag versioning consistent

CLI release tag should be derived from Cargo.toml

NATS Protocol

Your project is intriguing!

I was looking through your source, and was wondering if you've considered using NATS as a protocol? NATS is lightweight and has cloud native features that would trickle up to making your cli's and operations easier. For a specific example, your users wouldn't need to provision a topic and setup partitions with the cli before sending. You'd also be able to securely provide multi-tenancy from the get-go, which would make it easier to use fluvio in a PaaS/SaaS model.

From an adoption standpoint, you'd be able to reach both the NATS and Kafka ecosystems with the NATS Kafka bridge - best of both worlds. Just a thought....

Full disclosure, I'm a NATS maintainer so have bias, but noticed your project and just wanted to make you aware of NATS.

I wish you luck - we'll be following you guys!

CLI: topic creation should return proper error message

Steps to reproduce in dev

1. cargo build
2. create local cluster target/debug/cluster install --local which only creates 1 SPU
3. create topic fluvio topic create --topic test --replication 2 which requires 2 SPU

This result in error:
topic error 'test2' error

this is not descriptive error. it should return proper error message

High level architecture design

Any high level architecuture desing? I'm interested in contributing this, but it seems there's not enough information about this.

use Kubernetes Local Persistent Volume instead of PVC

Since SPU performs replication, there is no need to use PVC.

Requires changing helm templating to use different storage provider

Support connection multiplexing

Allow multiple concurrent connection over same TCP/IP.

CI: workflow for enforcing cargo fmt

Enforce formatting

Initial port of doc from fluvio.io

only use zero copy in the server

This need updated version of flv_future

make public-server and private-server port the spu optional

for public port: 0.0.0.0:9005,
for private port: 0.0.0.0:9004,

Switched to async-std and Isahc

Upgraded to futures 0.3.
Adopt async-std for futures library.
Isahc for http client.

CLI: Change Spu Group column" NAME" as left justified.

Implement Resync between SC and SPU

Currently, SPU re-connect to SC every 60 seconds to avoid issue with TCP connection. It would better if periodic resync is perform between SC and SPU. if re-sync doesn't not occur, full reconnect will occur

Replica Leaders should be evenly distributed across SPUs

If you add 5 topics, the leader distribution should be spread evenly across SPUs.
In my environment I have 3 SPUs and I created 5 topics with 1 partition and 2 replicas each.

The topics are distributed as follows:
5001 - 3 x leaders
5002 - 2 x leaders
5003 - 0 x leaders

$  ./target/debug/fluvio topic describe
Retrieved 5 out of 5 topics

TOPIC DETAILS
-------------
 Name                    :  stress-test-topic 
 Type                    :  computed 
 Partition Count         :  1 
 Replication Factor      :  2 
 Ignore Rack Assignment  :  yes 
 Status                  :  provisioned 
 Reason                  :  - 
 Partition Map               
 -----------------           
     ID      LEADER      REPLICAS          LIVE-REPLICAS 
      0       5001       [5003, 5001]      [0, 5003] 

TOPIC DETAILS
-------------
 Name                    :  test 
 Type                    :  computed 
 Partition Count         :  1 
 Replication Factor      :  2 
 Ignore Rack Assignment  :  - 
 Status                  :  provisioned 
 Reason                  :  - 
 Partition Map               
 -----------------           
     ID      LEADER      REPLICAS          LIVE-REPLICAS 
      0       5001       [5003, 5001]      [0, 5001] 

TOPIC DETAILS
-------------
 Name                    :  test2 
 Type                    :  computed 
 Partition Count         :  1 
 Replication Factor      :  2 
 Ignore Rack Assignment  :  - 
 Status                  :  provisioned 
 Reason                  :  - 
 Partition Map               
 -----------------           
     ID      LEADER      REPLICAS          LIVE-REPLICAS 
      0       5001       [5001, 5002]      [0, 5002] 

TOPIC DETAILS
-------------
 Name                    :  test3 
 Type                    :  computed 
 Partition Count         :  1 
 Replication Factor      :  2 
 Ignore Rack Assignment  :  - 
 Status                  :  provisioned 
 Reason                  :  - 
 Partition Map               
 -----------------           
     ID      LEADER      REPLICAS          LIVE-REPLICAS 
      0       5002       [5002, 5003]      [0, 5003] 

TOPIC DETAILS
-------------
 Name                    :  test4 
 Type                    :  computed 
 Partition Count         :  1 
 Replication Factor      :  2 
 Ignore Rack Assignment  :  - 
 Status                  :  provisioned 
 Reason                  :  - 
 Partition Map               
 -----------------           
     ID      LEADER      REPLICAS          LIVE-REPLICAS 
      0       5002       [5002, 5003]      [0, 5003] 


 $ ./target/debug/fluvio spu list
 ID    NAME             STATUS  TYPE    RACK  PUBLIC          PRIVATE 
 5001  custom-spu-5001  online  custom   -    localhost:9010  localhost:9011 
 5002  custom-spu-5002  online  custom   -    localhost:9020  localhost:9021 
 5003  custom-spu-5003  online  custom   -    localhost:9030  localhost:9031