containersolutions/k8s-deployment-strategies

Kubernetes deployment strategies explained

k8s-deployment-strategies's Issues

No target for "my-app" on prometheus

Hi,

When I follow the tutorial for the canary deployment, Grafana does not pull any data from Prometheus and thus cannot draw the graph. I've changed the prometheus service to be a NodePort, so that I can access it UI. The /targets path only show http://localhost:9090/metrics, despite there is much more targets definitions on configuration. Is this normal? How is Prometheus able to get data from my-app if there isn't any target to it?

Best Regards.

Graphana visualization doesn't work

For some reason, the visualation isn't working, although the data source is properly configured

.

The repository is still very useful, without the grafana part, but I'd be happy to know how to get grapha working.

This is my configuration for the prometheus chart:

serverFiles:
  prometheus.yml:
    rule_files:
      - /etc/config/rules
      - /etc/config/alerts
    scrape_configs:
    - job_name: 'kube-kubelet'
      honor_labels: false
      scheme: https

      tls_config:
      # This is needed because the kubelets' certificates are not generated
      # for a specific pod IP
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - target_label: __metrics_path__
        replacement: /metrics
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

    - job_name: 'kube-kubelet-cadvisor'
      honor_labels: false
      scheme: https

      tls_config:
      # This is needed because the kubelets' certificates are not generated
      # for a specific pod IP
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - target_label: __metrics_path__
        replacement: /metrics/cadvisor
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

    # Example scrape config for probing services via the Blackbox Exporter.
    #
    # Relabelling allows to configure the actual service scrape endpoint using the following annotations:
    #
    # * `prometheus.io/probe`: Only probe services that have a value of `true`
    - job_name: 'kubernetes-services'
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
        - role: service
      relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
          action: keep
          regex: true
        - source_labels: [__address__]
          target_label: __param_target
        - target_label: __address__
          replacement: blackbox
        - source_labels: [__param_target]
          target_label: instance
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          target_label: kubernetes_name
    # Example scrape config for pods
    #
    # Relabelling allows to configure the actual service scrape endpoint using the following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`.
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: (.+):(?:\d+);(\d+)
          replacement: ${1}:${2}
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
    # Scrape config for service endpoints.
    #
    # The relabeling allows the actual service scrape endpoint to be configured
    # via the following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape services that have a value of `true`
    # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
    # to set this to `https` & most likely set the `tls_config` of the scrape config.
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: If the metrics are exposed on a different port to the
    # service then set this appropriately.
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
        - role: endpoints
      relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: (.+)(?::\d+);(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name # Add your additional configuration here...

Canary by weight is not working properly while having perfect configuration using nginx-ingress

deployment_nginx_canary_conf.zip

we are using nginx version 0.27.0 and we tried it with version 0.30.0 also.

we tried implementing canary first by header value and then we tried to implement canary by header weight but the issue is canary itself is not working.

A little question about Grafana visualization of blue-green strategies.

I am learning Kubernetes deployment strategies with this repository. It's interesting to deploy test App and show the process in Grafana :D

When deploy blue-green (Single-service) the Graph is difference with README.md, maybe the problem is Grafana configure that has nothing to do with this repository, but I try to submit this issue to find out the reason.

Desired

Real

It look like Ramped but actual is Blue/Green deploying.

Look at the terminal:

# Switch version
kubectl patch service my-app -p '{"spec":{"selector":{"version":"v2.0.0"}}}'

# Output of `while sleep 0.1; do curl "$service"; done`
# We can see version was switched immediately
# but the Graph is not, it switch slowly just like ramped.
...
Host: my-app-v1-d9d8fb4c4-7nmhd, Version: v1.0.0
Host: my-app-v1-d9d8fb4c4-ppkz4, Version: v1.0.0
Host: my-app-v1-d9d8fb4c4-mzj6n, Version: v1.0.0
Host: my-app-v2-5856879d68-9kh2m, Version: v2.0.0
Host: my-app-v2-5856879d68-54xd7, Version: v2.0.0
Host: my-app-v2-5856879d68-9kh2m, Version: v2.0.0
...

Environmental

# base env version
macOS 11.2.3 + minikue  v1.18.1
prometheus 2.20.1 + grafana 7.1.1

# Grafana Panel
Metrics: sum(rate(http_requests_total{app="my-app"}[5m])) by (version)
Legend: {{version}} 

Visualization > Graph
Display-Bars > ON
Display-Lines > ON
Stacking and null value - Stack > ON
Stacking and null value - Percent > ON