sustainable-computing-io / susql-operator Goto Github PK
View Code? Open in Web Editor NEWa Kubernetes operator that aggregates energy data from tagged resources
Home Page: http://susql.org
License: Apache License 2.0
a Kubernetes operator that aggregates energy data from tagged resources
Home Page: http://susql.org
License: Apache License 2.0
Investigate and if necessary eliminate suspected race between metrics computation and metrics serving
Enable additional parameters to enable deployment in additional environments.
(Essentially port the flexibility added to the github.com/trent-s/susql-operator clone.)
https and http traffic needs to be handled differently and correctly. Here is sample log output:
ERROR [Reconcile]: Querying Prometheus didn't work: read tcp 10.217.0.203:33522->10.217.5.82:9091: read: connection reset by peer
ERROR [GetMetricValuesForPodNames]: Querying Prometheus didn't work: read tcp 10.217.0.203:33524->10.217.5.82:9091: read: connection reset by peer
Much configuration is currently done with environment variables at deploy. Using a configmap approach is preferable. It is less "ad-hoc" and provides a more dynamic method of updating values.
The controller keeps track of how much energy per pod has already been accounted for in main memory. This table is lost on restart of the controller meaning total energy consumption will be incorrectly doubled on restart when the label group is not setup to reset counts on restart.
Currently SusQL aggregates total energy as Joules. It could be useful to enable customization of this metric. For example, financial cost, or carbon footprint might be other options.
Users should be restricted to aggregate metrics they have access to.
We just had 3 new security alerts come in from at https://github.com/sustainable-computing-io/susql-operator/security/dependabot
There is a good chance that work I am engaged in will resolve this as a byproduct. But if not, I will also take care of resolving these alerts.
Currently runtime errors and certain warnings are available from container console output.
However, it would be very nice for debugging, verification, and exploration to adopt a logging framework and use it to be able to customize additional output. e.g., log output as INFO, DEBUG, WARNING, ERROR, etc, and allow customization of desired output level at deploy time.
Inclusion of the logging framework should be trivial, and inclusion of additional output within the code will not be challenging, but may take a day or two to go through the code identifying potentially interesting data to log.
Ensure that image is built and placed in the official public image repository: quay.io/sustainable_computing_io
There are a number of self-references to the old repository github.com/metalcycling/susql
.
Likewise we should probably find a new location to store docker images other than docker.io/metalcycling
.
Reference:
./cmd/main.go: susqlv1 "github.com/metalcycling/susql/api/v1"
./cmd/main.go: "github.com/metalcycling/susql/internal/controller"
./go.mod:module github.com/metalcycling/susql
./test/labelgroups.sh:namespace=metalcycling
./internal/controller/resource_manager.go: susql "github.com/metalcycling/susql/api/v1"
./internal/controller/labelgroup_controller.go: susql "github.com/metalcycling/susql/api/v1"
./internal/controller/suite_test.go: susqlv1 "github.com/metalcycling/susql/api/v1"
./PROJECT:repo: github.com/metalcycling/susql
./PROJECT: path: github.com/metalcycling/susql/api/v1
./deployment/susql-controller/values.yaml:containerImage: docker.io/metalcycling/susql-controller
./deployment/deploy.sh:SUSQL_REGISTRY="docker.io/metalcycling"
Refactoring this project from Kubebuilder to OperatorSDK would enable improved operator packaging and deployment.
There are use cases that require monitoring and aggregating multiple input metrics.
Apparently our build process pulls in oldish packages with a potential security impact. I'm not overly worried about container security, but it would be good if we can easily fix this.
Path : /home/xyz/.local/share/containers/storage/overlay/.../diff/usr/lib/x86_64-linux-gnu/libcurl.so.4.8.0
Installed version : 7.88.1
Fixed version : 8.4.0
Path : /home/xyz/.local/share/containers/storage/overlay/.../diff/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.8.0
Installed version : 7.88.1
Fixed version : 8.4.0
And
Path : /home/xyz/.local/share/containers/storage/overlay/.../diff/usr/bin/curl
Installed version : 7.88.1
Fixed version : 8.4.0
For example:
susql needs to understand user permission [admin view vs normal viewer for single or multiple namespaces)
Ideally this would also reduce energy consumption. This is a long range use case demonstration.
PEAKS integration would be excellent.
Verify SusQL functionality on OpenShift AI cluster.
Make SusQL updates if needed. Document deployment process as necessary.
Enable use authentication protected Thanos Querier.
@metalcycling would you consider migrating the repo? If so, can you add me as the repo admin? Thanks
r.GetPodNamesMatchingLabels
and GetMetricValuesForPodNames
should keep track of namespaces and names instead of just pod names.
When deleting and restarting workloads, there appears to be a double counting bug.
In the following demo, workloads are deleted and restarted.
before restarting susql-demo-1 accurately shows that it is the sum of all susql-demo-1 workloads. However on restarting, it shows unexpected behavior. Details in the comments.
the label that the service monitor is trying to match to is different from that used by the susql-controller.
Currently the "scope of interest" is specified by creating label groups and attaching these labels to pods of interest. Consider approaches to make this easier for users.
Susql is being killed periodically on the gdr-test cluster due to OOM errors.
Checking if Kepler is deployed...
Error from server (Forbidden): error when creating "STDIN": pods "kepler-check" is forbidden: error looking up service account openshift-kepler-operator/prometheus-k8s: serviceaccount "prometheus-k8s" not found
Error from server (NotFound): pods "kepler-check" not found
Error from server (NotFound): pods "kepler-check" not found
Error from server (NotFound): error when deleting "kepler-check.yaml": pods "kepler-check" not found
Kepler service at 'http://prometheus-k8s.monitoring.svc.cluster.local:9090/' was found. Using it to collect SusQL data.
2024-02-01T17:21:09Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: [susql.ibm.com/v1](http://susql.ibm.com/v1): the server could not find the requested resource"}
and
2024-02-01T17:21:39Z ERROR setup problem running manager {"error": "failed to wait for labelgroup caches to sync: timed out waiting for cache to be synced for Kind *v1.LabelGroup"}
internal/controller/prometheus_manager.go contains the following code:
var (
susqlMetrics = &SusqlMetrics{
totalEnergy: prometheus.NewGaugeVec(prometheus.GaugeOpts{
Namespace: "kepler",
Name: "node_platform_joules_total",
Help: "Accumulated energy over time for set of labels",
}, susqlPrometheusLabelNames),
}
Concerns:
Although it is possible to customize various details about how to connect to Prometheus, the referenced metric is hardcoded.
It would be useful to enable this to be specified at deployment time. Additionally, a short (1-2 word?) description of this metric should also be configurable, as this too is hardcoded in multiple locations.
Ideally, ODH would automatically create and apply labelgroups for workloads it creates, and then it would provide a dashboard for susql generated metrics.
Add cluster connection before proceeding into deploying the operator and throwing other unrelated errors.
E0213 16:08:14.306333 2593 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused E0213 16:08:14.317558 2593 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused E0213 16:08:14.318604 2593 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused E0213 16:08:14.319466 2593 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused E0213 16:08:14.320811 2593 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused The connection to the server localhost:8080 was refused - did you specify the right host or port?
Create a CLI tool that displays labeled energy consumption similar to the top command.
How to use existing time series data in SusQL Prometheus to perform an interval based power consumption query on the labels of interest.
[ ] Define query language
[ ] Implement
[ ] Verify
[ ] Document
Once issue 11 is finished:
make bundle
worksmake bundle
d SusQL operator.It would be useful to keep a log of deployment information for debugging, and even as a reference to double check how a current susql implementation is deployed.
.susql-deploy-info.txt
exists copy it to .susql-deploy-info-last.txt
overwriting older file if necessary..susql-deploy-info.txt
in sh
readable format.susql-deploy-info.txt
and .susql-deploy-info-last.txt
to .gitignoreThis information could be useful for either un-deployment or re-deployment.
In the future, it may be useful to include detailed errors, warnings, and other output, perhaps in another file. It is simple enough to redirect stdout/stderr, but since this script is slightly interactive, redirection could be problematic unless we remove interaction.
Enhance and document time series susql data.
Enable operations such as max, min, average, total, etc over custom time ranges along with label/pod selection for cli, api, and dashboard. (Currently only aggregate and label selection supported.)
Reconsider where data is located. e.g., storing data in multiple locations can lead to double counting as reported in issue 3. (Currently, data seems to be stored separately in pod memory, CR, and Prometheus.)
Provide support for metrics used in AI tuning, such as energy consumed per token or request, etc.
Create a version file and automate occurrences of susql version
Fix an installation bug that causes the current working directory to take on an unexpected value on error conditions. Specifically with the original code cd ${SUSQL_DIR} && make manifests && make install && cd - if either of the make commands fail, the the current working directly will stay in ${SUSQK_DIR} and not return to the starting point as intended with cd -. The simplest solution to ensure that the final cd is executed is to separate it out on a separate line as follows:
cd ${SUSQL_DIR} && make manifests && make install
cd -
kubectl get labelgroups -o json --all-namespaces
) once and then extract data with jq
. While we are at it, for extra credit do similar improvement to test/labelgroups.sh
.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.