ververica/jupyter-vvp

Jupyter Integration for Flink SQL via Ververica Platform

jupyter-vvp's Issues

Support Flink SQL DDL

I can execute a Flink SQL DDL statement in a Jupyter Notebook cell.

• Supported

  • SHOW TABLES
  • SHOW FUNCTION
  • CREATE/ALTER/DROP FUNCTION
  • CREATE/ALTER DROP TABLE
  • DESCRIBE

• Not supported (example)

  • CREATE TEMPORARY TABLE|VIEW

• On success, the result is displayed in Jupyter Notebook.

• On error, the error message is displayed in Jupyter Notebook.

Something like:

%%flink-sql

SHOW TABLES

Table Name
Test1
Test2

Local Development Environment with Snapshot Version of VVP and SQL

• checkout https://github.com/dataArtisans/daplatform-charts
• add remote https://github.com/dataArtisans/ververica-platform-playground-private preview to Ververica Platform Playground Repostiory
  • checkout branch snapshot-sql branch
  • create “vvp” and “vvp-jobs” namespace
  • add Kubernetes Secret to both Namespaces to be used an ImagePullSecret for private registry (command and Secret content can be found in README of https://github.com/dataArtisans/daplatform-charts - Connect to preview)
  • run setup.sh
• swagger ui available under localhost:8080/swagger

Install Extension and Connect to VVP

I can install the VVP SQL Jupyter “extension” and connect to Ververica Platform by providing its host:port and a Namespace.

• unauthenticated access

Something like:

%%connect-vvp --endpoint localhost:8080 --namespace default

Succesfully connect to Ververica Platform under localhsot:8080.

Code-Completion

%%flink_sql cells offer code-completion for SQL.

DML with Deployment Configuration

In can execute a Flink SQL DML Statement in a Jupyter Notebook. I can configure at least the following fields manually:

• name (metadata.name)
• non-production mode (metadata.annotations.license/testing)
• parallelism (spec.template.spec.parallelism)
• Upgrade Strategy (spec.restoreStrategy)
• Restart Strategy (spec.upradeStrategy)
• flink configuration (spec.template.spec.flinkConfiguration)

These two fields never have a default in the backend:

• name (metadata.name)
• non-production mode (metadata.annotations.license/testing)

Hence, a default needs to be supplied by the frontend. All other fields might have a default in the backend.

Some Ideas:

• Show the YAML representation of the Deployment before submission, so that it can be adapted. Very flexible, but low-level. Only requires one confirmation, if the defaults shall be used.

• Show a form with the minimal set of fields. Don't allow any configuration beyond this. Will we end up adding more and more fields on-demand?

• Show a form with the minimal set of fields, but also the full YAML (like the web user interface). Feels like an overkill.

• Customize the Deployment Spec via comments or similar in-inline in the cell.

Support Flink SQL DQL

Hi, having jupiterlabs as a primary development for FlinkSQL should help with faster development.

Right now, this extension only serves for metadata investigation and table creation.

Support Flink SQL DML with Default Configuration

In can execute a Flink SQL DML Statement in a Jupyter Notebook.
assuming that no additional Deployment Configuration needs to be passed to Ververica Platform (all defaults apply)

• A link to the Deployment is provided after successful submission.
  • Can we show the live status of the Job in the Notebook?

Something like:

%%flink-sql

INSERT INTO Test1 SELECT * FROM Test2

Support Python UDF

Hi,
I don't know if this is possible. But I really want to use Python UDF from notebook with this extension.

Authentication via OpenID Connect

As Flink SQL Jupyter user I would like to be able to connect to a Ververica Platform instance that uses OpenIDConnect for authentication.

Authentication via ApiToken

When Ververica Platform is configured with auth enabled, a Jupyer user can connect to VVP by authenticating with an ApiToken (https://docs.ververica.com/administration/auth/api_tokens.html).

The ApiToken would need to have at least the "Editor" role.