pachyderm/helmchart

Helm Chart for Pachyderm

Pachyderm – Automate data transformations with data versioning and lineage

Pachyderm is cost-effective at scale, enabling data engineering teams to automate complex pipelines with sophisticated data transformations across any type of data. Our unique approach provides parallelized processing of multi-stage, language-agnostic pipelines with data versioning and data lineage tracking. Pachyderm delivers the ultimate CI/CD engine for data.

Features

• Data-driven pipelines automatically trigger based on detecting data changes.
• Immutable data lineage with data versioning of any data type.
• Autoscaling and parallel processing built on Kubernetes for resource orchestration.
• Uses standard object stores for data storage with automatic deduplication.
• Runs across all major cloud providers and on-premises installations.

Getting Started

To start deploying your end-to-end version-controlled data pipelines, run Pachyderm locally or you can also deploy on AWS/GCE/Azure in about 5 minutes.

You can also refer to our complete documentation to see tutorials, check out example projects, and learn about advanced features of Pachyderm.

If you'd like to see some examples and learn about core use cases for Pachyderm:

• Examples
• Use Cases
• Case Studies

Documentation

Official Documentation

Community

Keep up to date and get Pachyderm support via:

• Follow us on Twitter.
• Join our community Slack Channel to get help from the Pachyderm team and other users.

Contributing

To get started, sign the Contributor License Agreement.

You should also check out our contributing guide.

Send us PRs, we would love to see what you do! You can also check our GH issues for things labeled "help-wanted" as a good place to start. We're sometimes bad about keeping that label up-to-date, so if you don't see any, just let us know.

Usage Metrics

Pachyderm automatically reports anonymized usage metrics. These metrics help us understand how people are using Pachyderm and make it better. They can be disabled by setting the env variable METRICS to false in the pachd container.

helmchart's Issues

Ensure that pre-release versions are not listed as release in ArtifactHub

Helm knows that 1.2.3-alpha.beta.3 is a pre-release version due to SemVer, but ArtifactHub appears to rely on annotations. Can these be added in the chart releaser process?

Otherwise, do they need to be added manually to pre-releases and removed manually from releases? That would be … not terribly great.

Replace Pachctl Deploy (Local Dev)

• Reset.py changes
• Square up helm chart with pachctl deploy local -d

Move from implicit Postgres configured Host/Port to explicit (external PG Support)

Right now, postgres is configured implicitly by having a specially named service which corresponds to a kube generated env var (POSTGRES_SERVICE_HOST) This makes using an external postgres difficult/awkward (ie. needing to override an autogenerated Env var) External services didn't work as it requires a DNS name, which won't always be the case for databases.

TODO

• Finish corresponding Pach PR https://github.com/pachyderm/pachyderm/pull/6341/files

Change 'metricsEndpoint' to 'usageMetricsEndpoint' for clarity

Metrics is an overloaded term and could refer to usage metrics, application metric, (prom) etc. Since metrics in charts typically refers to application metrics, rename the metricsEndpoint key to usageMetricsEndpoint for clarity.

Helm bug causing partSize to be converted to a float

Hi all.

This bug took me much longer to track down that I'd like to admit!

Recreate:
Use an "AMAZON" setup. E.g. https://github.com/pachyderm/helmchart/blob/master/examples/aws-values.yaml

Run that code (or just helm template if you don't want to recreate).

You'll get an error in pachd which crashes.

error setting up External Pachd GRPC Server: error setting up PFS API GRPC Server: cannot parse: strconv.ParseInt: parsing "5.24288e+06": invalid syntax

Reason:
This bug: helm/helm#1707

The partSize setting is being converted into a float by the toString method in this line: https://github.com/pachyderm/helmchart/blob/pachyderm-0.4.1-rc.1/pachyderm/templates/pachd/storage-secret.yaml#L35

Because the number is big and converted to engineering notation.

If you use a smaller number, then the toString works, but unfortunately minio requires a value of at least 5242880 to work (the default value).

I think you can work around this issue by accepting a string in your json schema, not an int. Which will be passed through to the k8s env var and interpreted as an int by pach.

Etcd storage class doesn't work in Minikube (only aws/gcp)

Hi there. This todo: https://github.com/pachyderm/helmchart/blob/master/pachyderm/templates/etcd/statefulset.yaml#L63

Prevents etcd starting on Minikube. It expects the storage class to exist and when it doesn't the PVC can't bind.

I propose that it should default to the default storage class, or allow the user to set/override in the values.yaml.

Thanks!

add liveness probe to etcd deployment

From pachyderm/pachyderm#3803

Bring Postgres storage classes in line with etcd for 2.x

Right now etcd storage classes have been fleshed out for each storage provider and are well documented / understood. Copy this over for postgres

Node Affinity

For shared clusters it's essential that Pachyderm not compete for resources with other production applications. There are a number of methods for doing, all of which require additions to the Pod's spec. This is currently impossible to do with this Helm chart.

Please make it possible to restrict Pachyderm to specific nodes as per this official documentation. The quickest implementation of this would be to allow users to specify additional yaml that should be added to the spec.

(Bug) Investigate image pull credentials and add tests

Try passing through docker credentials and make sure they work in a real setting

Add helm chart tests around adding docker credentials

Currently defined twice in chart?

• imagePullSecret
• imageCredentials

Allow Dash ingress TLS to be enabled/disabled

Although definitely a best practice to use TLS with ingress, it's technically not required, and there may be use cases where tls shouldn't be enabled on the ingress.

Set up CI

Set project up in circle CI to run tests

Set up Chart publish

Use https://github.com/pachyderm/chart-releaser-action or other means to publish chart on tag

(Note: mirrored action to pachyderm repo for security)

Move helm into pach core repo

Add tests for storage class options

to make sure they're rendered correctly for each cloud provider

Add Namespaced scoped role/rolebinding option (ie Role vs Cluster Role)

Investigate using pachctl pod (or similar) to setup auth from configmaps

Generate Deployment ID

Add support for prometheus monitoring of etcd

From: pachyderm/pachyderm#3805

Test that imagepullsecret works

Might not be the right data type?

(2.x) Allow local postgres to be disabled

Allow a remote postgres to be specified.

Divorce pachd storage backend from cloud backend for etcd/postrges

For scenarios where we're disabling pachd (it enterprise only) pachd.storage.backend is not specified, but we still need a way to tell etcd/postgres which storageclass to deploy (if necessary)

Make Postgres DB name configurable / Change to pachyderm (or similar)

Right now the DB name is pgc it should be pachyderm (or similar) for clarity

Note: This needs a corresponding task on the Pachyderm side, as pgc is hardcoded in places

PG Bouncer Support

Add CLA Assistant

We need to add the Contributor License Agreement Assistant https://cla-assistant.io/

(2.0x) Update prom annotation to correct port in pachd

Port changed, but need to update annotation

Add more-informative description

The description in helm search repo pachyderm is just ‘A Helm chart for Kubernetes,’ which is hardly useful. Add in some more useful verbiage here.

Enterprise server only support

• Need disable Pachd option #122
• Need enterprise server option #121

Add test faster to run helm chart on live cluster?

@lukemarsden what would the lift be to use test faster in this repo to run a helm install in a fresh minikube cluster?

Move ingress section under dash

The current ingress section could be moved under dash as it only includes dash values

Allow for configurable annotations in ingress

Change the ingress annotations section to a toYaml so you can pass in whatever you'd like (in our case traefik, but folks might want to add their own ingress, cert manager, etc)

See here for an example

Issue with prometheus port name: "must be no more than 15 characters"

Hey Kyle!

Quick issue. The name for the prom port is invalid. Annoyingly it doesn't validate properly on the local side, and passes. But when you apply it (I'm on 1.20 Minikube atm) it fails.

Error: Service "pachd" is invalid: spec.ports[8].targetPort: Invalid value: "prometheus-metrics": must be no more than 15 characters

https://github.com/pachyderm/helmchart/blob/master/pachyderm/templates/pachd/service.yaml#L67

Renaming to prom-metrics should fix. Will create a PR.

Remove hostpath from stateful sets for etcd and postgres

Hostpath is not needed for etcd and postgres (Need to make sure PVC is deleted anyway)

Register with helm hub

Separate TLS configuration for Dash and Pach

Currently, Dash and Pachd share the same TLS configuration, but it would be good to make these separate (with the option to use the same secret) as sometimes folks host dash and pach on different domains or have other use cases to use different certificates

Init checks for pachd

Check if etcd (and postgres in 2.0) are running before booting pachd pod

(Bug) Fix s3 gateway port in load balancer service

target port should be s3gateway-port

TLS configuration Changes [ High prio, blocks chart installation ]

Right now, we half enable TLS by default by specifying a secret name and hoping the user specifies the cert. Default should be either A) fully disabled or B) Enabled, but with an error message for cert

As well, specifying no TLS is a bit awkward

tls:
  certName: "" // Must be empty string
  create: null // must be null

We should change this to be more straightforward

cc @robert-uhl

Pick up Pachyderm version from chart appVersion

The app version is used once in Chart.yaml and once in values.yaml, which is repetition and leads to mistakes. Better would be for the templates to default to the former and only use the latter as an override.

Should be pretty straightforward.

Explore making release process more automated

https://github.com/mbenabda/helm-local-chart-version

https://github.com/semantic-release/semantic-release

Dashboard ingress not working

$ helm -n pachyderm template ./pachyderm -f ./pachyderm/values.yaml \
--set ingress.enabled=true        
Error: template: pachyderm/templates/dash/ingress.yaml:12:20: executing "pachyderm/templates/dash/ingress.yaml" at <.Pach.DashURL>: nil pointer evaluating interface {}.DashURL

Use --debug flag to render out invalid YAML

here:

helmchart/pachyderm/templates/dash/ingress.yaml

Line 12 in f0301ad

- host: {{ .Pach.DashURL }}

IAM Roles for EKS Pods

This chart currently supports IAM roles for GKE but not IAM roles for pods on EKS. This is blocking my company's adoption.

Here are two easy methods to add this functionality:

1. Add Helm values for specifying the IAM role name for the two service accounts, which is as simple as adding the annotation eks.amazonaws.com/role-arn: ...
2. A more universal solution solution is to allow users to specify additional annotations for each of the two service accounts. There are examples of this in a plethora of other popular helm charts.

Liveliness probe for pachd

Pach 2.0 PostgreSQL storage class

Should only use a StorageClass for cloud providers, not locally. Cf. how the StorageClass for etcd.

etcd storageclass not able to provision gp3 disk

In order to provision a gp3 disk for etcd on eks, we need to use the aws ebs csi driver as the provisioner.

The following change has to be made in the storageclass:

# Source: pachyderm/templates/etcd/storageclass.yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  labels:
    app: etcd
    suite: pachyderm
  name: etcd-storage-class
parameters:
  type: gp3
provisioner: ebs.csi.aws.com >>>> I changed this to make it work.

[Hub Enhancement] Change internal service to use ClusterIP instead of Nodeport

Add standard configurability for postgres

• imagePullSecret
• resources
• others?

Simplify dash tls secret check (optimization)

https://github.com/pachyderm/helmchart/blob/master/pachyderm/templates/dash/tls-secret.yaml#L12

Double if statements can be combined into one.

Move 'LoadBalancer' section under pachd, rename to 'externalService' and make ports configurable

• Move the load balancer section under pachd in values.yaml as it only controls pachd
• Rename to externalService instead of loadbalancer for consistency (controls a service object)
• Make external port numbers configurable for this service - Folks might want to be able to access pachd on a custom external port

Add nodeport option back (for local deploy only)