eslam-gomaa/kptop

CLI tool for Kubernetes that provides pretty monitoring for Nodes, Pods, Containers, and PVCs resources on the terminal through Prometheus metrics

Kube-Prometheus-Top [ kptop ]

A Python tool that provides Monitoring for Kubernetes Nodes, Pods, Containers, and PVCs resources on the terminal through Prometheus metircs

kptop2.mp4

📌 Try out the new beta features

• kptop_0.0.7 (Introducing K8s Pod port-forward connection method))
  • Environment variables

Motivation

The resources metrics provided by the K8s APIs are very limited compared what what's scraped by Prometheus.

This tool is using Prometheus as a data source for metrics to display all the needed informations right on the terminal.

Project Status

• Top Nodes
  • -o cloud // -o json
• Live monitoring for Nodes
• Top Pods
• Live monitoring for Pods/Containers
• Top PVCs

Additional

• Check the connection with Prometheus Server
• Check the existence of the needed Prometheus metrics _{[ good for similar use cases ]}

---

Installation

Compatible with Python 3.6+

on PyPi

pip3 install kptop --upgrade

---

Environment Variables

ENV	Description	Default	Required
KPTOP_PROMETHEUS_SERVER	Prometheus server URL		Yes
KPTOP_BASIC_AUTH_ENABLED	Whether basic authentication is needed to connect to Prometheus	False	No
KPTOP_PROMETHEUS_USERNAME	Prometheus username		No
KPTOP_PROMETHEUS_PASSWORD	Prometheus password		No
KPTOP_INSECURE	Verify SSL certificate	False	No
KPTOP_NODE_EXPORTER_NODE_LABEL	node exporter "node label"	"node"	NO
KPTOP_START_GRAPHS_WITH_ZERO	By default graphs begin with '0' to let the graph take its full hight	True	NO
KPTOP_LOGGING_DIR	Choose a different logging directory	/tmp/	NO
KPTOP_GRAPH_WIDTH	Choose a custom graphs width	45	NO

CLI Arguments

ENV	Description	Default
--namespace, -n	Specify a Kubernetes Namespace	default
--all-namespaces, -A
--container, -c	Specify a container
--interval, -i	Live monitoring update interval	8 [NOTE: the actuall update depends on the Prometheus scaping interval (15s by default)]
--debug, -d	Enable debugging logging mode	False
--verify-prometheus, -V	Verify connectivity to Prometheus server & check the existence of the needed exporters
--sort-by-mem-usage, -s	Sort top result by memory usage	False
--check-metrics , -C	Check the existence of the needed metrics, needs to be done with -V	False

---

Usage

Different ways to connect to Prometheus server:

1. You have direct access to it (Like in dev environments)
2. Prometheus is exposed publically/over-vpn (mostly with an Ingress)
3. You can use kubectl port-forward command
4. You also can run kptop as a Kubernetes pod (to be built))

export KPTOP_PROMETHEUS_SERVER="http://prometheus.home-lab.com"

Top nodes

kptop nodes

NODE      MEM TOTAL    MEM USAGE    MEM FREE      CPU CORES  CPU USAGE%      RUNNING PODS
worker-1  19.6 gb      16.92 gb     2.69 gb               6  9%                        14
worker-2  19.6 gb      9.52 gb      10.08 gb              6  9%                        27

kptop nodes --option / -o 🎉

kptop nodes -o json --colorize-json

Under testing (currently tested with EKS)

kptop nodes -o cloud

Live monitoring for Nodes

kptop node <NODE>

Top pods

kptop pods -n <NAMESPACE>

kptop pods -n elk-stack

NAMESPACE    POD                                      MEM LIMIT    MEM USAGE    MEM USAGE %    MEM USAGE MAX    MEM FREE    CPU LIMIT    CPU USAGE
elk-stack    elasticsearch-master-0                   2.0 gb       1.38 gb      68%            2.0 gb           635.25 mb   1000m        0.04m
elk-stack    elasticsearch-master-1                   2.0 gb       1.49 gb      74%            2.0 gb           522.05 mb   1000m        0.03m
elk-stack    strimzi-filebeat-filebeat-f8ms7          200.0 mb     85.73 mb     42%            174.16 mb        114.27 mb   1000m        0.02m
elk-stack    haproxy-ingress-filebeat-filebeat-pq2wf  200.0 mb     87.73 mb     43%            171.31 mb        112.27 mb   1000m        0.03m
elk-stack    strimzi-filebeat-filebeat-r7dht          200.0 mb     119.12 mb    59%            199.52 mb        80.88 mb    1000m        0.02m
elk-stack    haproxy-ingress-filebeat-filebeat-lzqdt  200.0 mb     98.66 mb     49%            199.57 mb        101.34 mb   1000m        0.02m
elk-stack    my-kibana-kibana-79448f7fb7-wf4t6        2.0 gb       342.87 mb    16%            618.07 mb        1.67 gb     1000m        0.02m
elk-stack    my-logstash-logstash-0                   1.5 gb       1008.22 mb   65%            1.21 gb          527.78 mb   1000m        0.02m

kptop pod -n kube-system

NAMESPACE    POD                                              MEM LIMIT    MEM USAGE    MEM USAGE%    MEM USAGE MAX    MEM FREE    CPU LIMIT    CPU USAGE
kube-system  coredns-558bd4d5db-nfcjq                         170.0 mb     26.0 mb      15%           42.77 mb         144.0 mb    ---          0.0m
kube-system  coredns-558bd4d5db-vcstr                         170.0 mb     17.45 mb     10%           24.04 mb         152.55 mb   ---          0.0m
kube-system  etcd-master                                      ---          85.02 mb     ---           391.7 mb         ---         ---          0.02m
kube-system  kube-apiserver-master                            ---          635.97 mb    ---           731.13 mb        ---         ---          0.09m
kube-system  kube-controller-manager-master                   ---          95.55 mb     ---           145.41 mb        ---         ---          0.03m
kube-system  kube-proxy-q6nr7                                 ---          27.46 mb     ---           58.91 mb         ---         ---          0.0m
kube-system  kube-proxy-q489q                                 ---          21.98 mb     ---           63.0 mb          ---         ---          0.0m
kube-system  kube-proxy-bghp6                                 ---          22.35 mb     ---           64.1 mb          ---         ---          0.0m
kube-system  kube-scheduler-master                            ---          37.04 mb     ---           61.68 mb         ---         ---          0.0m
kube-system  nfs-subdir-external-provisioner-b97f4d9f5-bjp2h  ---          9.43 mb      ---           37.48 mb         ---         ---          0.0m

Live monitoring for Pods

kptop pod <POD> -n <NAMESPACE>

Live monitoring for Containers

kptop pod <POD> -n <NAMESPACE> -c <CONTAINER>

Top PVCs

kptop pvcs <NAMESPACE>

NOTE: in this example, all VPCs have the same capacity because this is a testing environment (using nfs-provisioner)

kptop pvcs --all-namespaces

NAMESPACE    PVC                                          VOLUME                CAPACITY    USED      USED %    FREE      FREE %
elk-stack    elasticsearch-master-elasticsearch-master-0  elasticsearch-master  123.14 gb   21.42 gb  17%       95.43 gb  77%
elk-stack    elasticsearch-master-elasticsearch-master-1  elasticsearch-master  123.14 gb   21.42 gb  17%       95.43 gb  77%
elk-stack    elasticsearch-master-elasticsearch-master-2  elasticsearch-master  ?           ?         ?         ?         ?
kafka        data-0-kafka-cluster-region1-kafka-0         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region1-kafka-1         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region1-kafka-2         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-0         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-1         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-2         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-0         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-1         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-2         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-0         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-1         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-2         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-0         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-1         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-2         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-kafka-cluster-region1-zookeeper-0       data                  123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-kafka-cluster-region2-zookeeper-0       data                  123.14 gb   21.42 gb  17%       95.43 gb  77%
prometheus   my-prometheus-alertmanager                   storage-volume        123.14 gb   21.42 gb  17%       95.43 gb  77%
prometheus   my-prometheus-server                         storage-volume        123.14 gb   21.42 gb  17%       95.43 gb  77%

Verify Prometheus connectivity

kptop --verify-prometheus

Sample output

Verifying Prometheus connection: Connected                     
{
  "connected": true,
  "status_code": 200,
  "reason": "",
  "fail_reason": ""
}

Verifying Prometheus Exporters:

* Node Exporter:  Found             
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_versions": {
      "1.3.1": "2"
    }
  }
}

* Kubernetes Exporter:  Found           
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_git_versions": {
      "v1.21.0": "3",
      "v1.21.14": "1"
    }
  }
}

---

Check Prometheus metrics

kptop --verify-prometheus --check-metrics

Sample output

Verifying Prometheus connection: Connected                     
{
  "connected": true,
  "status_code": 200,
  "reason": "",
  "fail_reason": ""
}

Verifying Prometheus Exporters:

* Node Exporter:  Found             
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_versions": {
      "1.3.1": "2"
    }
  }
}

* Kubernetes Exporter:  Found           
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_git_versions": {
      "v1.21.0": "3",
      "v1.21.14": "1"
    }
  }
}
 
  0:00:00 0:00:00 Checking Metrics  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%  [ container_fs_writes_bytes_total ]
+---------------------------------------------------+---------------+-----------+-----------+
| METRIC                                            | EXPORTER      | STATE     | COMMENT   |
+===================================================+===============+===========+===========+
| node_memory_MemFree_bytes                         | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemAvailable_bytes                    | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemTotal_bytes                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Cached_bytes                          | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Buffers_bytes                         | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapTotal_bytes                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapFree_bytes                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapCached_bytes                      | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_cpu_seconds_total                            | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load1                                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load5                                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load15                                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_physical_cores                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_sockets                               | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| up                                                | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_boot_time_seconds                            | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_size_bytes                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_receive_bytes_total                  | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_transmit_bytes_total                 | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_written_bytes_total                     | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_read_bytes_total                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_cores                                 | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_running_pods                              | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_last_seen                               | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_working_set_bytes                | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_max_usage_bytes                  | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_limit_bytes                 | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_cache                            | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_swap_limit_bytes            | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_load_average_10s                    | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_usage_seconds_total                 | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_system_seconds_total                | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_user_seconds_total                  | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_cpu_quota                          | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kube_pod_spec_volumes_persistentvolumeclaims_info | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_capacity_bytes               | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_used_bytes                   | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_available_bytes              | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_receive_bytes_total             | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_transmit_bytes_total            | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_start_time_seconds                      | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_file_descriptors                        | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_threads                                 | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_processes                               | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_reads_bytes_total                    | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total                   | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total                   | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+

---

Logging

Default log file location is "/tmp/kptop.log"

---

Known Issues

[1] Node Exporter metrics don't return data

• This is NOT an issue, the node exporter NODE label change from version to another, currently we encountered only "kubernetes_node" or "node"
• "node" is the default, to fix the it you can change it with the "KPTOP_NODE_EXPORTER_NODE_LABEL" Environment variables

export KPTOP_NODE_EXPORTER_NODE_LABEL="node" # default
export KPTOP_NODE_EXPORTER_NODE_LABEL="kubernetes_node"

auto detection of exporters verstions can be implemented later (if needed).

---

Reach me anytime on Linkedin

kptop's Issues

automate / allow using a tunnel or port-forward for prometheus endpoint

Hi,

neat tool;)
Have you thought about automating and allowing to use a port-forward or tunnel to the (not-exposed) prometheus-service?

That would ease multi-cluster usage. Allowing to do sth. like export KPTOP_PROMETHEUS_SERVER="http://127.0.0.1:8080 once and just use the current kubectl-context to create the tunnel etc without any more manual steps?

regards,
strowi

rewrite it as kubectl plugin

Hi,
rewrite it as kubectl plugin

Examples

kubectl kptop get nodes

kubectl kptop get pods

.....

change logging mode of queries results to `debug` instead of `info`

Workload metrics

Instead of just containers, pods and nodes, it would be nice to also have views for deployments, daemonsets, statefulsets, etc...

Handle if the mem usage metric gives higher value than the limit

Getting application errors when executing kptop command

I have installed kptop and added env variable for KPTOP_PROMETHEUS_SERVER with prometheus endpoint on AKs cluster, and getting below errors can u please help out here.

File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kptop_tool.py", line 6, in
run()
File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kptop_tool.py", line 3, in run
from kubePtop.cli import Cli
File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kubePtop\cli.py", line 184, in
cli = Cli()
File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kubePtop\cli.py", line 77, in init
pod_metrics.topPodTable(namespace=ns,sort_by_mem_usage=self.sort_by_mem_usage)
File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kubePtop\pod_metrics.py", line 1052, in topPodTable
pods_json = self.topPod(namespace=namespace, sort_by_mem_usage=sort_by_mem_usage)
File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kubePtop\pod_metrics.py", line 967, in topPod
memory_limit = self.run_query(memory_limit_query)
File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kubePtop\session.py", line 120, in run_query
return json.loads(req.text)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

---

PS C:\Users\Mounika.Akurathi> kptop --verify-prometheus

Verifying Prometheus connection: Connected
{
"connected": true,
"status_code": 200,
run()
File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kptop_tool.py", line 3, in run
verify_node_exporter = self.verifyNodeExporter()
File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kubePtop\session.py", line 148, in verifyNodeExporter
File "C:\Users\Mounika.Akurathi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\kubePtop\session.py", line 120, in run_query
return json.loads(req.text)
return _default_decoder.decode(s)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Introduce `--version, -v` to show kptop version

Update methods comments

add an ENV for graphs width

[feature request] Allow setting up dashboards as yaml files

It would be awesome to be able to allow creating dashboards with a config file defining the layout and queries to run.

Maybe a dashboard command that would take the file as an argument or if the files are stored under some specific folder, they would show as sub-commands.

Improve "kptop pvcs" performance

Introduce -o "yaml" / "json"

Add kubernetes to the dependencies

After installation, it doesn't work due to missing module 'kubernetes', should be added to the dependencies

Traceback (most recent call last):
  File "/Users/khaled-abushqear/Library/Python/3.11/bin/kptop", line 5, in <module>
    from kptop_tool import run
  File "/Users/khaled-abushqear/Library/Python/3.11/lib/python/site-packages/kptop_tool.py", line 6, in <module>
    run()
  File "/Users/khaled-abushqear/Library/Python/3.11/lib/python/site-packages/kptop_tool.py", line 3, in run
    from kubePtop.cli import Cli
  File "/Users/khaled-abushqear/Library/Python/3.11/lib/python/site-packages/kubePtop/cli.py", line 6, in <module>
    from kubePtop.session import PrometheusAPI
  File "/Users/khaled-abushqear/Library/Python/3.11/lib/python/site-packages/kubePtop/session.py", line 19, in <module>
    from kubernetes import config
ModuleNotFoundError: No module named 'kubernetes'

finalizing default dashboard

• detect if the pod or node exists

I can't find the kptop command after installing kptop. Is there any step I am missing?

[root@llody-dev ]#pip3 install kptop --upgrade
Collecting kptop
Downloading kptop-0.0.7-py3-none-any.whl (47 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.9/47.9 kB 148.3 kB/s eta 0:00:00
Collecting tabulate==0.9.0
Downloading tabulate-0.9.0-py3-none-any.whl (35 kB)
Collecting rich==13.3.1
Downloading rich-13.3.1-py3-none-any.whl (239 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.0/239.0 kB 18.7 kB/s eta 0:00:00
Requirement already satisfied: requests==2.28.2 in /usr/local/bin/python3/lib/python3.7/site-packages (from kptop) (2.28.2)
Collecting asciichartpy==1.5.25
Downloading asciichartpy-1.5.25-py2.py3-none-any.whl (7.2 kB)
Collecting argparse
Downloading argparse-1.4.0-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: setuptools in /usr/local/bin/python3/lib/python3.7/site-packages (from asciichartpy==1.5.25->kptop) (67.8.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/bin/python3/lib/python3.7/site-packages (from requests==2.28.2->kptop) (3.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/bin/python3/lib/python3.7/site-packages (from requests==2.28.2->kptop) (3.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/bin/python3/lib/python3.7/site-packages (from requests==2.28.2->kptop) (2022.12.7)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/bin/python3/lib/python3.7/site-packages (from requests==2.28.2->kptop) (1.26.15)
Collecting markdown-it-py<3.0.0,>=2.1.0
Downloading markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.5/84.5 kB 18.1 kB/s eta 0:00:00
Requirement already satisfied: pygments<3.0.0,>=2.14.0 in /usr/local/bin/python3/lib/python3.7/site-packages (from rich==13.3.1->kptop) (2.15.1)
Requirement already satisfied: typing-extensions<5.0,>=4.0.0 in /usr/local/bin/python3/lib/python3.7/site-packages (from rich==13.3.1->kptop) (4.5.0)
Collecting mdurl=0.1
Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Installing collected packages: argparse, tabulate, mdurl, asciichartpy, markdown-it-py, rich, kptop
Attempting uninstall: rich
Found existing installation: rich 11.0.0
Uninstalling rich-11.0.0:
Successfully uninstalled rich-11.0.0
Successfully installed argparse-1.4.0 asciichartpy-1.5.25 kptop-0.0.7 markdown-it-py-2.2.0 mdurl-0.1.2 rich-13.3.1 tabulate-0.9.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.0.1 -> 23.2.1
[notice] To update, run: /usr/local/bin/python3/bin/python3.7 -m pip install --upgrade pip
[root@llody-dev ~]#kpa
kpartx kpasswd
[root@llody-dev ~]#pip3 list | grep kptop
kptop 0.0.7

[notice] A new release of pip is available: 23.0.1 -> 23.2.1
[notice] To update, run: /usr/local/bin/python3/bin/python3.7 -m pip install --upgrade pip
[root@llody-dev ~]#export KPTOP_PROMETHEUS_SERVER="http://192.168.1.227:32456/"
[root@llody-dev ~]#kptop nodes
-bash: kptop: command not found

`--verify-prometheus` should check metrics availability

In the cluster I'm trying kptop with, we are filtering which cadvisor metrics we scrape. To get kptop working correctly, I need to run kptop's commands, see what graphs populate and don't populate, check the source code to see which metrics the graphs are looking at, then update the list of scraped metrics.

I would love if --verify-prometheus returned a list of metrics that were missing, so I could more-easily update the allowlist in my scrape config.

It was easy to determine that I needed machine_cpu_cores, since kptop nodes fails noisily:

# kptop nodes
No nodes found
Query did not return any data: machine_cpu_cores

However, for the dashboards, I need to dig through the logfile. It would be nice if all the failing queries were presented at once.

add an ENV Key for logging directory

Fix KeyError exception with "kptop pods --all-namespaces"

implement "kptop nodes"

Handle if failed to get some data while listing the pods