kedacore / keda-external-scaler-azure-durable-functions Goto Github PK
View Code? Open in Web Editor NEWExternal scaler for Durable Functions workloads.
License: Apache License 2.0
External scaler for Durable Functions workloads.
License: Apache License 2.0
As a developer, I want to create a helm chart to deploy scaler so that customer can deploy server in one liner.
Have a load testing with sending durable orchestrator request and tuning the scale logic.
Current version doesn't have liveness/readinesProbe on the yaml/helm chart since the image is for gRPC server not http server.
However, we can do it with this idea.
https://kubernetes.io/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/
We can use grpc-health-probe binary.
As a developer, I'd like to enable https connection for gRPC so that customer wants to choose if they try it as http or https for gRPC.
As a developer, I want to have a release pipeline for the helm chart so that developer can deploy new version very quickly and safety.
As a developer, I'd like to update Azure Functions Core Tools template.
Currently, the func kubernetes
command doesn't generate correct YAML file. Once this project is approved, It should be updated to fit this project.
I want to use KEDA for durable functions, but in the doc only a few triggers are supported yet in KEDA, and not Durable functions triggers like ActivityTrigger or OrchestratorTrigger etc. I came across this repo, and tried to follow Getting Started doc, but it shows "Please make sure you have the correct access rights and the repository exists" error while trying to clone KEDA and durable scaler repos. Is this project still under development, or is it supposed to be an internal/private project? If yes, is there any other way to use Durable Functions in Kubernetes?
The current version of KEDA Durable Scaler has a limitation. The pods of durable functions never goes to zero. The reason is, if you need to get a pod of durable functions by this scaler, you need to put a control/worker queue. In this case, you need additional http/queue trigger implementation that executes DurableOrchestrationClient.
That is why, the current Azure Functions Core Tools (func command) generate following yaml file for durable functions. It includes, AzureFunctionsJobHost__functions__0
that is for filter the functions. However, it seems doesn't work and I can see the issue.
data:
AzureWebJobsStorage: <YOUR_STORAGE_ACCOUNT_BASE64>
FUNCTIONS_WORKER_RUNTIME: ZG90bmV0
apiVersion: v1
kind: Secret
metadata:
name: durable-keda
namespace: default
---
apiVersion: v1
kind: Service
metadata:
name: durable-keda-http
namespace: default
annotations:
osiris.deislabs.io/enabled: true
osiris.deislabs.io/deployment: durable-keda-http
spec:
selector:
app: durable-keda-http
ports:
- protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: durable-keda-http
namespace: default
labels:
app: durable-keda-http
annotations:
osiris.deislabs.io/enabled: true
osiris.deislabs.io/minReplicas: 1
spec:
replicas: 1
selector:
matchLabels:
app: durable-keda-http
template:
metadata:
labels:
app: durable-keda-http
spec:
containers:
- name: durable-keda-http
image: tsuyoshiushio/durable-keda
ports:
- containerPort: 80
env:
- name: AzureFunctionsJobHost__functions__0
value: LoadOrchestration_HttpStart
envFrom:
- secretRef:
name: durable-keda
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: durable-keda
namespace: default
labels:
app: durable-keda
spec:
selector:
matchLabels:
app: durable-keda
template:
metadata:
labels:
app: durable-keda
spec:
containers:
- name: durable-keda
image: tsuyoshiushio/durable-keda
env:
- name: AzureFunctionsJobHost__functions__0
value: LoadOrchestration
- name: AzureFunctionsJobHost__functions__1
value: LoadOrchestration_Hello
envFrom:
- secretRef:
name: durable-keda
---
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
name: durable-keda
namespace: default
labels:
deploymentName: durable-keda
spec:
scaleTargetRef:
deploymentName: durable-keda
triggers:
- type: orchestrationtrigger
metadata:
type: orchestrationTrigger
name: context
- type: activitytrigger
metadata:
type: activityTrigger
name: name
---
Currently, I make it one deployment and make it minimum scale count as 1. We need to wait until the issue is solved or if there is other way to solve it, I'll take your advice.
---
apiVersion: v1
kind: Service
metadata:
name: durable-keda
namespace: default
spec:
selector:
app: durable-keda
ports:
- protocol: TCP
port: 80
targetPort: 80
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: durable-keda
namespace: default
labels:
app: durable-keda
spec:
replicas: 1
selector:
matchLabels:
app: durable-keda
template:
metadata:
labels:
app: durable-keda
spec:
containers:
- name: durable-keda
image: tsuyoshiushio/durable-keda:latest
ports:
- containerPort: 80
env:
envFrom:
- secretRef:
name: durable-keda
---
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
name: durable-keda
namespace: default
labels:
deploymentName: durable-keda
spec:
scaleTargetRef:
deploymentName: durable-keda
triggers:
- type: external
metadata:
scalerAddress: durable-external-scaler-service.keda.svc.cluster.local:5000
---
As a developer, I'd like to move the container image namespace from tsuyoshiushio to kedacore, so that people can see the image as official.
As a developer, I want to monitoring soltion so that developer and customer can both diagnose the issue easily. We might be able to use Application Insights or Plain Stdout/err to the docker. Or we might need to discuss product team if more telemetery requires.
Update the keda-doc repo for adding keda-scaler-durable-functions.
https://github.com/kedacore/keda-docs
As a developer I want to have a feature to handle multiple deployment by a single gRPC server, so that customer don't need to deploy server one for a deployment.
The gRPC server will accept request from several deployments. In this case, when the New method on the ExternalScalerService is created, You need to create a DisconnectedPerformanceMonitor object for each deployment. And it should be removed when Close called.
I want to use KEDA for durable functions. I have been following Getting Started doc of this repo to test scaling of durable functions, but can't get to make it work. I have a simple durable function orchestration like this one. Also tried with QueueTriggered orchestration (actually this is what I to implement in my project), but doesn't work for that too.
Steps which I followed:
Logs of events and external scaler pod:
kubectl get events
FailedGetExternalMetric horizontalpodautoscaler/keda-hpa-durable-keda unable to get external metric default/ScaleRecommendation/&LabelSelector{MatchLabels:map[string]string{deploymentName: durable-keda,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: No matching metrics found for scalerecommendation FailedComputeMetricsReplicas horizontalpodautoscaler/keda-hpa-durable-keda invalid metrics (1 invalid out of 1), first error is: failed to get ScaleRecommendation external metric: unable to get external metric default/ScaleRecommendation/&LabelSelector{MatchLabels:map[string]string{deploymentName: durable-keda,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: No matching metrics found for scalerecommendation
kubectl logs keda-durable-external-scaler-86c6f85bcc-s77qz -n keda -f
Request starting HTTP/2 POST http://durable-external-scaler-service.keda.svc.cluster.local:5000/externalscaler.ExternalScaler/GetMetrics application/grpc info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetrics' dbug: Grpc.AspNetCore.Server.ServerCallHandler[1] Reading message. info: Keda.Durable.Scaler.Server.Services.ExternalScalerService[0] Namespace: default DeploymentName: durable-keda GetMetrics() called. fail: Grpc.AspNetCore.Server.ServerCallHandler[3] Error when executing service method 'GetMetrics'. System.NullReferenceException: Object reference not set to an instance of an object. at Keda.Durable.Scaler.Server.Services.ExternalScalerService.GetMetrics(GetMetricsRequest request, ServerCallContext context) in /src/src/Keda.Durable.Scaler.Server/Services/ExternalScalerService.cs:line 112 at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase3.<HandleCallAsync>g__AwaitHandleCall|17_0(HttpContextServerCallContext serverCallContext, Method2 method, Task handleCall)
Specifications: Docker Desktop for Windows (2.4.0.0), Kubernetes (1.18.8)
Please let me know if more information is needed, I'd be happy to dig out as much as I can. I'm still new to Kubernetes so not sure whether this is a bug or I'm just missing something. Thanks!
As a developer, I'd like to have an infrastructure integration testing, so that we can validate pull request.
I'm planning to implement it as terratest. The scenario will be:
full-ci
label on the target PR.Switch default branch from master to main
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.