spotify/flyte-flink-plugin

Flyte Flink k8s plugin.

flyte-flink-plugin

Flyte Flink k8s plugin.

Contents

• flyte-flink-plugin
  • Contents
  • About
  • Dependencies
  • Building
  • Testing
  • License

About

Current development status:

• MVP features are developed
• Missing user documentation
• Project being tested, and collecting feedback
• No guarantees of API stability

To learn more about Flyte refer to:

• Flyte homepage
• Flyte master repository

Dependencies

To install the project dependencies, run:

go mod install

Building

To build the binary, run:

make build

Testing

To run tests, run:

make test

To run tests with coverage, run:

go test -v -covermode=count  ./...

License

This project is released under the Apache License 2.0.

flyte-flink-plugin's Issues

Add FlinkCluster resource name to FlinkProperties template

Support job resource configuration

Job submitter resources are currently hard coded to:

    resources:
      limits:
        cpu: "1"
        memory: 1Gi

For some jobs this might result in:

java.lang.OutOfMemoryError: Java heap space

Add protobuf definition to define task execution custom info

Add Flyte / Flink generated names can get pretty long

This will makes hit some of the limits k8s has in place:

• name: max allowed size is 253
• label value: max allowed size 63

Generated name with flink:

<flyte-exec-id>-<task-name>-<attempt>-<job | taskmanager | jobmanater>-<hash>

One possible way to address this is to hash flyte generated name:

<flyte-exec-id>-<task-name>-<attempt>

Expose JobManager Ingress generated URL's

Add aws S3 downloader support

Add support for default logging properties

Add support for ingress annotations

Add JobManager and TaskManager sidecar support

Error creating Flink clusters due to label value size

label value max allowed size is 63

"create Pod q7u424vimc-orgflyteexamplesexampleflinktask-0-taskmanager-0 in StatefulSet q7u424vimc-orgflyteexamplesexampleflinktask-0-taskmanager failed error: Pod "q7u424vimc-orgflyteexamplesexampleflinktask-0-taskmanager-0" is invalid: metadata.labels: Invalid value: "q7u424vimc-orgflyteexamplesexampleflinktask-0-taskmanager-76485844df": must be no more than 63 characters"

Duplicate volumeMounts when adding a PersistenceVolumeClaim with same mount path

Generate unique name for init containers emptyDir volumes

Bug: Job task logs are getting the job name instead of the pod name

Add cluster name validation

Names need to follow:

[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*

Conversion from literal to args adding unnecessary quotes

Add remote cluster execution support

Add supplied job artifacts as env var

Rework container resource specification

JobManager and TaskManager should use ResourceRequirements

Resource Quantity not being parsed correctly

Allow setting JobManager accessScope

Add `nodeSelector` support

• Default setting in the config
• (Nice to have) allow override from task definition

Tweak memory settings

Requires #36

Tweak persistent volume claims

• remove volumes
• add io.tmp.dirs

Update flink operator to v1beta2

This new CRD will bring:

• FlinkVersion
• ProcessMemoryRatio