Comments (16)
@wesselkranenborg just acknowledging that we are working on adding custom health probe annotation and trying learn the custom health probe through the liveness checks defined for pods. Will update once we have PRs associated with these features.
from application-gateway-kubernetes-ingress.
You can provide host
in readinessProbe
/livenessProbe
.
Here is the merged doc:
https://azure.github.io/application-gateway-kubernetes-ingress/docs/tutorial.html#adding-health-probes-to-your-service
from application-gateway-kubernetes-ingress.
@wesselkranenborg if I understand correctly you want the ingress controller to learn the custom health check endpoint for a given service and configure custom health probes on the application gateway?
If my assumption is correct, could you give an example of the custom helath check endpoint here.
Thanks,
aVinash
from application-gateway-kubernetes-ingress.
That is indeed correct. We for example use a /private/ping endpoint on all our services which are only exposed internally within the cluster (and thus available by the applicatoin gateway).
This service is configured as health check endpoint in the deployment of kubernetes and returns a 200 OK if it is healthy, otherwise an other statuscode. This endpoint is checking if the service is correctly running, has access to the database and other dependent services, etc.
from application-gateway-kubernetes-ingress.
+1 For this feature I am running into the same issue now.
I tried to work around this by manually configuring the custom health check endpoint within the http-settings that are automatically added to the application gateway by the ingress controller. This manual configuration works until your pods are redeployed. When that happens the ingress controller rewrites the http-settings and consequently removes any custom health check endpoint you configured.
from application-gateway-kubernetes-ingress.
+1 for this feature. The default nginx ingress controller supports this feature using a custom annotation: https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/health-checks
from application-gateway-kubernetes-ingress.
We received this notification in our Azure Service Health. This makes this feature even more important to implement.
This is to notify that Application Gateway v2 SKU will be shortly introducing changes in probing behavior which may require customers to take action to prevent any application downtime. Currently v2 SKU does not perform any default health probes. This implies that if customers do not specify a probe configuration, then all backend instances are assumed to be healthy and traffic is sent to them irrespective of the backendโs health status. Application Gateway is planning to change this behavior by introducing default probes similar to the current GA SKU. Once these changes take effect, probing behavior on both V2 SKU and the current GA SKU will be identical. Details on the health probing
behavior are documented here - https://docs.microsoft.com/en-us/azure/application-gateway/application-gatewayprobe-overview.No action is required at this time. We will be sending future communication before introduction of default probes reaches production, along with instructions on how to avoid failures. This communication is advanced notification of intended changes to the default probing behavior. Please feel free to reach out to us at [email protected] for comments.
from application-gateway-kubernetes-ingress.
This notification is now placed in the service health for region South Central US.
We have important information for your Application Gateway or WAF V2 resources. Beginning on May 1 2019, we will roll out default health probes for the Standard_V2 and WAF_V2 SKUs such that the behavior of the health probes becomes identical to that for the Standard and WAF (V1) SKUs, as documented here: https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-probe-overview#default-health-probe
Recommended action: We have identified one or more Application Gateway / WAF V2 gateways that you own that appear to be failing health probes, either due to a non-healthy HTTP status code (i.e not in the range 200 โ 399) or due to a timeout for the health probe.
Since these default health probe failures can potentially cause application downtime (if all backend instances are unhealthy), it is recommended to check your backend endpoint health for these identified gateways and ensure that your backend endpoints are reachable and not returning HTTP failures on the default health probe URL (http://: or https://: depending on whether HTTPs is configured for backend http settings โ please refer to the documentation: https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-probe-overview#default-health-probe).
We also recommend configuring custom health probes (https://docs.microsoft.com/en-us/azure/application-gateway/application-gateway-probe-overview#custom-health-probe) to have more control and flexibility on the health probe configuration.
from application-gateway-kubernetes-ingress.
from application-gateway-kubernetes-ingress.
Please try out the release candidate and let us know if you find any issues.
helm install -f <helm-config.yaml> application-gateway-kubernetes-ingress/ingress-azure --version 0.4.0-rc1
from application-gateway-kubernetes-ingress.
Thanks @akshaysngupta, is there any documentation on how to get this working?
from application-gateway-kubernetes-ingress.
As an update @akshaysngupta I have deployed the latest release and seen that the health probes are created based on my liveness and readiness configuration. However in my setup I am using nginx ingress controller to actually forward on requests that come in from Application Gateway (because I don't want to be locked in to using application gateway fully, it also means I can service multiple namespaces in my AKS enviroment, using one ignress control plane and AG). Because the health probe is configured with the hostname from the ingress manifest, when the health check is executed it is sent to the nginx ingress controller running on the backend node, but nginx then forwards this on to mybackend pod which tries to execute the health check (this is because of the value of the hostname). I can get the behaviour that I want if I could override the hostname that the health probes are being configured with and set it to localhost or 127.0.0.1 which will the execute the health check on the nginx ingress controller itself. Is this possible?
from application-gateway-kubernetes-ingress.
@dan-jackson-github we working on the documentation for health porbes here:
#179
The ask you have is a bit odd since it looks like its very specific for your configuration of having AG ingress controller running with the nginx ingress controller. If you have access to the nginx ingress controller pod spec, adding liveness checks in the spec would be the most straightforward approach here.
from application-gateway-kubernetes-ingress.
Closing this out. #158 covers this.
from application-gateway-kubernetes-ingress.
Hi, can't get this working https://azure.github.io/application-gateway-kubernetes-ingress/troubleshooting/
Deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "5"
creationTimestamp: "2019-11-25T14:07:12Z"
generation: 10
labels:
run: nginx
name: nginx
namespace: default
resourceVersion: "1249139"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx
uid: e0470631-0f8c-11ea-b26f-9e9afe712f2b
spec:
progressDeadlineSeconds: 600
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
run: nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
run: nginx
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: 80
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 3
conditions:
- lastTransitionTime: "2019-11-25T14:07:12Z"
lastUpdateTime: "2019-12-06T12:50:34Z"
message: ReplicaSet "nginx-68cf44b44d" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing - lastTransitionTime: "2019-12-06T12:51:04Z"
lastUpdateTime: "2019-12-06T12:51:04Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 10
readyReplicas: 3
replicas: 3
updatedReplicas: 3
Service:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-11-25T14:08:02Z"
labels:
run: nginx
name: nginx
namespace: default
resourceVersion: "5452"
selfLink: /api/v1/namespaces/default/services/nginx
uid: fe67e689-0f8c-11ea-b26f-9e9afe712f2b
spec:
clusterIP: 10.0.63.93
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
run: nginx
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
Ingress rule
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
appgw.ingress.kubernetes.io/backend-path-prefix: /
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"azure/application-gateway"},"name":"test","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"nginx","servicePort":80}}]}}]}}
kubernetes.io/ingress.class: azure/application-gateway
creationTimestamp: "2019-11-25T14:11:53Z"
generation: 2
name: test
namespace: default
resourceVersion: "1247987"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/test
uid: 884dab27-0f8d-11ea-b26f-9e9afe712f2b
spec:
rules:
- http:
paths:- backend:
serviceName: nginx
servicePort: 80
path: /
status:
loadBalancer:
ingress: - ip: XXX
- backend:
No matter if i change from liveness to readiness probe... in the app-gateway I always see this:
the default configuration... no matter if i scale up or down or if i change the paths... it's always the default.
Using azure ingress 1.0.0
from application-gateway-kubernetes-ingress.
got it working:
Didn't know that the following is required. After specifying this inside the deployment it's working.
ports:
- containerPort: 80
from application-gateway-kubernetes-ingress.
Related Issues (20)
- AGIC Behaviour Change for Managed Identity HOT 1
- Redirect to another domain
- Removal of HTTPS Listener when there is an intermittent issue in secret retrieval HOT 3
- panic in pkg/k8scontext/secrets_handlers.go:73
- Failed to watch *v1beta1.AzureApplicationGatewayRewrite HOT 1
- Setup per-site WAF policy HOT 1
- New custom error pages issue: Application Gateway Custom Error Status Code Is Invalid HOT 1
- Support regex in AzureIngressProhibitedTarget HOT 1
- Public IP HTTPS fails when creating private IP HTTPS ingress HOT 3
- AGIC is cannot list resource "azureapplicationgatewayrewrites" in API group "appgw.ingress.azure.io" at the cluster scope HOT 15
- From ingress annotations, which one helps create backend pool in app gateway
- AGIC does not create dedicated pool & target for shared App Gateway HOT 1
- appgw support - Monitor managed Prometheus
- ingress appgw pod - forced restart necessary to create listener etc. for new frontend ip HOT 1
- [Help request] How to redirect from non-www to www site
- How to remove "defaulthttpsetting" because it raises security issue
- az cli command from install guide evaluates with empty value
- Unable to create AzureIngressProhibitedTarget on AKS HOT 2
- AGIC is wiping out my backend pools
- Terraform wipes out AGIC's backends on each deploy HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from application-gateway-kubernetes-ingress.