kedacore/keda-external-scaler-azure-durable-functions

External scaler for Durable Functions workloads.

keda-external-scaler-azure-durable-functions's Issues

Upgrade to KEDA 2.0

Create helm chart

As a developer, I want to create a helm chart to deploy scaler so that customer can deploy server in one liner.

Success Criteria

• helm chart works locally can/configure Scale Server
• Discuss which namespace can we use for the helm chart with product team
• Publish the helm chart
• Write a documentation how to deploy it

Dependency

https://github.com/microsoft/keda-durable-scaler/issues/9

Load Testing with Scaling Logic refinement

Have a load testing with sending durable orchestrator request and tuning the scale logic.

Success Criteria

• Have a automated load testing with script
• Design the load testsing and execute it
• Create a report the trend of the scaling action and find it works
• Update the scaling logic to fine tuning the scale decision

Implement health check for gRPC server implementation.

Current version doesn't have liveness/readinesProbe on the yaml/helm chart since the image is for gRPC server not http server.

However, we can do it with this idea.
https://kubernetes.io/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/
We can use grpc-health-probe binary.

Success Criteria

• helm chart is updated to implement health checking liveness/readinessProbe
• deployment is fully tested as automated testing

Dependency

• #2
• #12

Configure Certificate for Scale Object

As a developer, I'd like to enable https connection for gRPC so that customer wants to choose if they try it as http or https for gRPC.

Success Criteria

• Test the Certificate Configuration both on gRPC server and Scale Object
• Write document for enabling certificate

Release Pipeline for the helm chart

As a developer, I want to have a release pipeline for the helm chart so that developer can deploy new version very quickly and safety.

Success Criteria

• Deploy new helm chart with a new version tag.
• Upload new docker version to DockerHub
• (Optional) Automate to create Release Note

Update Azure Functions Core Tools

As a developer, I'd like to update Azure Functions Core Tools template.
Currently, the func kubernetes command doesn't generate correct YAML file. Once this project is approved, It should be updated to fit this project.

Success Criteria

• Azure Functions Core Tools https://github.com/Azure/azure-functions-core-tools behavior should be updated for using this project
• Document is property updated for Azure Functions Core Tools repo
• Document (Getting Started) is property updated.

How to get started with KEDA for durable functions?

I want to use KEDA for durable functions, but in the doc only a few triggers are supported yet in KEDA, and not Durable functions triggers like ActivityTrigger or OrchestratorTrigger etc. I came across this repo, and tried to follow Getting Started doc, but it shows "Please make sure you have the correct access rights and the repository exists" error while trying to clone KEDA and durable scaler repos. Is this project still under development, or is it supposed to be an internal/private project? If yes, is there any other way to use Durable Functions in Kubernetes?

Pods doesn't scale in to zero

The current version of KEDA Durable Scaler has a limitation. The pods of durable functions never goes to zero. The reason is, if you need to get a pod of durable functions by this scaler, you need to put a control/worker queue. In this case, you need additional http/queue trigger implementation that executes DurableOrchestrationClient.

That is why, the current Azure Functions Core Tools (func command) generate following yaml file for durable functions. It includes, AzureFunctionsJobHost__functions__0 that is for filter the functions. However, it seems doesn't work and I can see the issue.

• Azure/azure-functions-host#4412

data:
  AzureWebJobsStorage: <YOUR_STORAGE_ACCOUNT_BASE64> 
  FUNCTIONS_WORKER_RUNTIME: ZG90bmV0
apiVersion: v1
kind: Secret
metadata:
  name: durable-keda
  namespace: default
---
apiVersion: v1
kind: Service
metadata:
  name: durable-keda-http
  namespace: default
  annotations:
    osiris.deislabs.io/enabled: true
    osiris.deislabs.io/deployment: durable-keda-http
spec:
  selector:
    app: durable-keda-http
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: durable-keda-http
  namespace: default
  labels:
    app: durable-keda-http
  annotations:
    osiris.deislabs.io/enabled: true
    osiris.deislabs.io/minReplicas: 1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: durable-keda-http
  template:
    metadata:
      labels:
        app: durable-keda-http
    spec:
      containers:
      - name: durable-keda-http
        image: tsuyoshiushio/durable-keda
        ports:
        - containerPort: 80
        env:
        - name: AzureFunctionsJobHost__functions__0
          value: LoadOrchestration_HttpStart
        envFrom:
        - secretRef:
            name: durable-keda
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: durable-keda
  namespace: default
  labels:
    app: durable-keda
spec:
  selector:
    matchLabels:
      app: durable-keda
  template:
    metadata:
      labels:
        app: durable-keda
    spec:
      containers:
      - name: durable-keda
        image: tsuyoshiushio/durable-keda
        env:
        - name: AzureFunctionsJobHost__functions__0
          value: LoadOrchestration
        - name: AzureFunctionsJobHost__functions__1
          value: LoadOrchestration_Hello
        envFrom:
        - secretRef:
            name: durable-keda
---
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: durable-keda
  namespace: default
  labels:
    deploymentName: durable-keda
spec:
  scaleTargetRef:
    deploymentName: durable-keda
  triggers:
  - type: orchestrationtrigger
    metadata:
      type: orchestrationTrigger
      name: context
  - type: activitytrigger
    metadata:
      type: activityTrigger
      name: name
---

Currently, I make it one deployment and make it minimum scale count as 1. We need to wait until the issue is solved or if there is other way to solve it, I'll take your advice.

---
apiVersion: v1
kind: Service
metadata:
  name: durable-keda
  namespace: default
spec:
  selector:
    app: durable-keda
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: durable-keda
  namespace: default
  labels:
    app: durable-keda
spec:
  replicas: 1
  selector:
    matchLabels:
      app: durable-keda
  template:
    metadata:
      labels:
        app: durable-keda
    spec:
      containers:
      - name: durable-keda
        image: tsuyoshiushio/durable-keda:latest
        ports:
        - containerPort: 80
        env:
        envFrom:
        - secretRef:
            name: durable-keda
---
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: durable-keda
  namespace: default
  labels:
    deploymentName: durable-keda
spec:
  scaleTargetRef:
    deploymentName: durable-keda
  triggers:
  - type: external
    metadata:
      scalerAddress: durable-external-scaler-service.keda.svc.cluster.local:5000
---

Move DockerHub repo to kedacore

As a developer, I'd like to move the container image namespace from tsuyoshiushio to kedacore, so that people can see the image as official.

Success Criteria

• docker image name is changed to kedacore
• All Deployment Script / YAML file is modified and tested.
• Create a Release pipeline to publish the docker image

Design Monitoring Solution for the Scale Server

As a developer, I want to monitoring soltion so that developer and customer can both diagnose the issue easily. We might be able to use Application Insights or Plain Stdout/err to the docker. Or we might need to discuss product team if more telemetery requires.

Success Criteria

• Discuss the monitoring solution and architecture
• Implement the monitoring solution
• Can be dashboarding the current status of the scaler
• Can be search the log to diagnostic the issue

Add Document on KEDA-DOCS

Update the keda-doc repo for adding keda-scaler-durable-functions.
https://github.com/kedacore/keda-docs

Success criteria

• Update the keda-docs to add keda-scaler-durable-functions
• Create PR and getting review for it

Design and Implement Architecture of handling multiple deployment

As a developer I want to have a feature to handle multiple deployment by a single gRPC server, so that customer don't need to deploy server one for a deployment.

The gRPC server will accept request from several deployments. In this case, when the New method on the ExternalScalerService is created, You need to create a DisconnectedPerformanceMonitor object for each deployment. And it should be removed when Close called.

Success Criteria

• DisconnectedPerformanceMonitor object is created for each deployment and removed when it closed.
• Discuss the architecture with product team
• Write unit testing for this feature
• Write integration testing in possible scenario.

Resource

https://github.com/microsoft/keda-durable-scaler/issues/11

Scaling not working, external scaler throws exception

I want to use KEDA for durable functions. I have been following Getting Started doc of this repo to test scaling of durable functions, but can't get to make it work. I have a simple durable function orchestration like this one. Also tried with QueueTriggered orchestration (actually this is what I to implement in my project), but doesn't work for that too.

Steps which I followed:

1. Deployed KEDA to keda namespace.
2. Deployed secret (create_secrets.sh) and external scaler to keda namespace.
3. Deployed the function app to default namespace along with secrets converted to base64.
4. Pods are running fine. I can start the orchestration by hitting the URL (http://localhost/api/Function1_HttpStart).
5. Started load testing. Number of pods didn't scale up as expected.

Logs of events and external scaler pod:

kubectl get events

FailedGetExternalMetric horizontalpodautoscaler/keda-hpa-durable-keda unable to get external metric default/ScaleRecommendation/&LabelSelector{MatchLabels:map[string]string{deploymentName: durable-keda,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: No matching metrics found for scalerecommendation FailedComputeMetricsReplicas horizontalpodautoscaler/keda-hpa-durable-keda invalid metrics (1 invalid out of 1), first error is: failed to get ScaleRecommendation external metric: unable to get external metric default/ScaleRecommendation/&LabelSelector{MatchLabels:map[string]string{deploymentName: durable-keda,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: No matching metrics found for scalerecommendation

kubectl logs keda-durable-external-scaler-86c6f85bcc-s77qz -n keda -f

Request starting HTTP/2 POST http://durable-external-scaler-service.keda.svc.cluster.local:5000/externalscaler.ExternalScaler/GetMetrics application/grpc info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetrics' dbug: Grpc.AspNetCore.Server.ServerCallHandler[1] Reading message. info: Keda.Durable.Scaler.Server.Services.ExternalScalerService[0] Namespace: default DeploymentName: durable-keda GetMetrics() called. fail: Grpc.AspNetCore.Server.ServerCallHandler[3] Error when executing service method 'GetMetrics'. System.NullReferenceException: Object reference not set to an instance of an object. at Keda.Durable.Scaler.Server.Services.ExternalScalerService.GetMetrics(GetMetricsRequest request, ServerCallContext context) in /src/src/Keda.Durable.Scaler.Server/Services/ExternalScalerService.cs:line 112 at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase3.<HandleCallAsync>g__AwaitHandleCall|17_0(HttpContextServerCallContext serverCallContext, Method2 method, Task handleCall)

Specifications: Docker Desktop for Windows (2.4.0.0), Kubernetes (1.18.8)

Please let me know if more information is needed, I'd be happy to dig out as much as I can. I'm still new to Kubernetes so not sure whether this is a bug or I'm just missing something. Thanks!

Add Infrastructure testing and Pipeline

As a developer, I'd like to have an infrastructure integration testing, so that we can validate pull request.
I'm planning to implement it as terratest. The scenario will be:

• Create an AKS cluster with RBAC enabled
• Push new image to private ACR
• deploy kedacore
• deploy keda-scaler-durable-functions with helm. It uses new images.
• deploy sample durable functions
• Start durable functions several times from example directory
• Check if it scare more than two
• Clean up the resources

Success Criteria

• Integration testing is implemented
• PR validation works with GitHub Action when we add full-ci label on the target PR.

Switch default branch from master to main

Switch default branch from master to main