Giter Site home page Giter Site logo

seal-io / walrus Goto Github PK

View Code? Open in Web Editor NEW
393.0 9.0 38.0 43.98 MB

Walrus is an open-source application management platform based on IaC tools including OpenTofu, Terraform and others. It helps platform engineers build golden paths for developers and empowers developers with self-service capabilities.

Home Page: https://seal.io

License: Apache License 2.0

Makefile 0.11% Go 97.25% Shell 2.22% Dockerfile 0.42%
idp opentofu devops iac terraform platform-engineering multi-cloud cloud-native kubernetes cncf

walrus's People

Contributors

aiwantaozi avatar alexcodelf avatar dependabot[bot] avatar gitlawr avatar linyinli avatar pengjiang80 avatar rainfd avatar shanewxy avatar thxcode avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

walrus's Issues

Add a cost data synchronization status

Version: main
UI: dev-454e8cb
Commit ID: cc89f02
Env: Ubuntu 22.04 amd64

According to the current implementation, if the cost data is not obtained successfully, the information of the cluster cannot be displayed on the cost analysis page. Cost data is synchronized every hour. It may takes up to an hour to get the data.
Users can be confused by not being able to view cost information for new clusters.

Therefore, please add a cost data synchronization status, and attach a corresponding prompt to tell the user approximately how long does it take to get the data.

image

image

Failed to query available provider packages after deleting build-image module and redeploying

Version: main
UI: dev-ae9ee1f
Commit ID: 01407be
Env: MacOS 13.1, Apple M2

Test Steps:

  1. Create an application, add a build-image module and deploy an instance.
  2. Delete the build-image module and add a webservice module. Redeploy the instance created in step 1.
  3. Check the deployment status.

Test Result:

  1. Failed to deploy. View the logs from db:
Initializing modules...
Downloading git::https://github.com/gitlawr/modules.git for flask-server...
- flask-server in .terraform/modules/flask-server/webservice
Downloading registry.terraform.io/terraform-iaac/deployment/kubernetes 1.4.2 for flask-server.deployment...
- flask-server.deployment in .terraform/modules/flask-server.deployment
Downloading registry.terraform.io/terraform-iaac/service/kubernetes 1.0.4 for flask-server.service...
- flask-server.service in .terraform/modules/flask-server.service

Initializing the backend...

Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/kubernetes versions matching ">= 2.1.0"...
- Finding latest version of hashicorp/random...
- Finding latest version of gitlawr/kaniko...
- Installing hashicorp/kubernetes v2.18.1...
- Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
- Installing hashicorp/random v3.4.3...
- Installed hashicorp/random v3.4.3 (signed by HashiCorp)

Warning: Reference to undefined provider

  on main.tf line 27, in module "flask-server":
  27:     kubernetes = kubernetes.connector--452132025961283588

There is no explicit declaration for local provider name "kubernetes" in
module.flask-server, so Terraform is assuming you mean to pass a
configuration for "hashicorp/kubernetes".

If you also control the child module, add a required_providers entry named
"kubernetes" with the source address "hashicorp/kubernetes".


Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
gitlawr/kaniko: no available releases match the given constraints 

Dashboard

Show basic statistics of the system

helm release status is not updated after installing prometheus server timeout

Version: main
UI: dev-454e8cb
Commit ID: ddc7ec3
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Create a Kubernetes connector and view the logs of seal.
    Install chart app seal-system:seal-prometheus timed out.
2023-03-09T08:04:04.410Z	E	error ensure cost tools for connector myseal: error install chart app seal-system:seal-prometheus from /.cache/repository-cache/prometheus-19.7.2.tgz: timed out waiting for the condition
github.com/seal-io/seal/utils/log.DelegatedLogger.Errorf
	github.com/seal-io/seal/[email protected]/log/delegate.go:136
github.com/seal-io/seal/utils/log.Errorf
	github.com/seal-io/seal/[email protected]/log/logger.go:168
github.com/seal-io/seal/pkg/connectors.EnsureCostTools.func1
	github.com/seal-io/seal/pkg/connectors/bus.go:55
github.com/alitto/pond.(*WorkerPool).executeTask
	github.com/alitto/[email protected]/pond.go:454
github.com/alitto/pond.worker
	github.com/alitto/[email protected]/worker.go:32
2023-03-09T08:04:04.410Z	E	failed to update connector myseal: context deadline exceeded
github.com/seal-io/seal/utils/log.DelegatedLogger.Errorf
	github.com/seal-io/seal/[email protected]/log/delegate.go:136
github.com/seal-io/seal/utils/log.Errorf
	github.com/seal-io/seal/[email protected]/log/logger.go:168
github.com/seal-io/seal/pkg/connectors.EnsureCostTools.func1
	github.com/seal-io/seal/pkg/connectors/bus.go:67
github.com/alitto/pond.(*WorkerPool).executeTask
	github.com/alitto/[email protected]/pond.go:454
github.com/alitto/pond.worker
	github.com/alitto/[email protected]/worker.go:32
  1. Wait for seal-prometheus-server pod to become running. Check cost analysis tools.
root@8f3212bf9ef0:/# kubectl get all -n seal-system
NAME                                                      READY   STATUS    RESTARTS   AGE
pod/seal-opencost-7d7fcffc6b-v829d                        1/1     Running   0          37m
pod/seal-prometheus-kube-state-metrics-55959bc5ff-mg2wp   1/1     Running   0          37m
pod/seal-prometheus-prometheus-node-exporter-b7m6v        1/1     Running   0          37m
pod/seal-prometheus-server-69555c46b-vsgq7                2/2     Running   0          37m

NAME                                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/seal-opencost                              ClusterIP   10.43.70.38     <none>        9003/TCP   37m
service/seal-prometheus-kube-state-metrics         ClusterIP   10.43.167.44    <none>        8080/TCP   37m
service/seal-prometheus-prometheus-node-exporter   ClusterIP   10.43.248.228   <none>        9100/TCP   37m
service/seal-prometheus-server                     ClusterIP   10.43.71.117    <none>        80/TCP     37m

NAME                                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/seal-prometheus-prometheus-node-exporter   1         1         1       1            1           <none>          37m

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/seal-opencost                        1/1     1            1           37m
deployment.apps/seal-prometheus-kube-state-metrics   1/1     1            1           37m
deployment.apps/seal-prometheus-server               1/1     1            1           37m

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/seal-opencost-7d7fcffc6b                        1         1         1       37m
replicaset.apps/seal-prometheus-kube-state-metrics-55959bc5ff   1         1         1       37m
replicaset.apps/seal-prometheus-server-69555c46b                1         1         1       37m
  1. View the logs of CostToolsCheckCronExpr task. It returned errors. But it should be ready.
2023-03-09T08:20:00.009Z	E	cost.check-status	error check cost tools for connector myseal: release seal-system:seal-prometheus status is failed, check helm release details
github.com/seal-io/seal/pkg/costs/scheduler.(*ToolsCheckTask).Process.func1
	github.com/seal-io/seal/pkg/costs/scheduler/tools_check_task.go:51
github.com/seal-io/seal/utils/gopool.(*WrapWaitGroup).Go.func1
	github.com/seal-io/seal/[email protected]/gopool/pool.go:76
github.com/alitto/pond.(*WorkerPool).executeTask
	github.com/alitto/[email protected]/pond.go:454
github.com/alitto/pond.worker
	github.com/alitto/[email protected]/worker.go:32

2023-03-09T08:30:00.044Z	E	cost.check-status	error check cost tools for connector myseal: release seal-system:seal-prometheus status is failed, check helm release details
github.com/seal-io/seal/pkg/costs/scheduler.(*ToolsCheckTask).Process.func1
	github.com/seal-io/seal/pkg/costs/scheduler/tools_check_task.go:51
github.com/seal-io/seal/utils/gopool.(*WrapWaitGroup).Go.func1
	github.com/seal-io/seal/[email protected]/gopool/pool.go:76
github.com/alitto/pond.(*WorkerPool).executeTask
	github.com/alitto/[email protected]/pond.go:454
github.com/alitto/pond.worker
	github.com/alitto/[email protected]/worker.go:32

image

Enhancement for first sync cost data

Version: main
UI: dev-454e8cb
Commit ID: cc89f02
Env: Ubuntu 22.04 amd64

Prerequisites:

  1. Create Kubernetes cluster in November 2022.
  2. Create cost analysis tools on March 7, 2023.

Test Steps:

  1. Setup seal container and create a Kubernetes connector.
  2. Click Synchronize cost data icon and view the seal logs.

Test Result:

  1. Sync data from November 6, 2022. But there is no data until March 7, 2023
2023-03-09T02:54:51.062Z	D	cost	collect cost for connector: aws
2023-03-09T02:54:51.071Z	D	cost	connector: aws, current sync costs within 2022-11-06 10:00:00 +0000 UTC, 2023-03-09 02:00:00 +0000 UTC
2023-03-09T02:54:51.071Z	D	cost	connector: aws, step sync within 2022-11-06 10:00:00 +0000 UTC, 2022-11-07 10:00:00 +0000 UTC
2023-03-09T02:54:51.631Z	D	cost	connector: aws, step sync within 2022-11-07 10:00:00 +0000 UTC, 2022-11-08 10:00:00 +0000 UTC
2023-03-09T02:54:52.157Z	D	cost	connector: aws, step sync within 2022-11-08 10:00:00 +0000 UTC, 2022-11-09 10:00:00 +0000 UTC
2023-03-09T02:54:52.742Z	D	cost	connector: aws, step sync within 2022-11-09 10:00:00 +0000 UTC, 2022-11-10 10:00:00 +0000 UTC
2023-03-09T02:54:53.215Z	D	cost	connector: aws, step sync within 2022-11-10 10:00:00 +0000 UTC, 2022-11-11 10:00:00 +0000 UTC
2023-03-09T02:54:53.777Z	D	cost	connector: aws, step sync within 2022-11-11 10:00:00 +0000 UTC, 2022-11-12 10:00:00 +0000 UTC
2023-03-09T02:54:54.328Z	D	cost	connector: aws, step sync within 2022-11-12 10:00:00 +0000 UTC, 2022-11-13 10:00:00 +0000 UTC
2023-03-09T02:54:54.901Z	D	cost	connector: aws, step sync within 2022-11-13 10:00:00 +0000 UTC, 2022-11-14 10:00:00 +0000 UTC
2023-03-09T02:54:55.380Z	D	cost	connector: aws, step sync within 2022-11-14 10:00:00 +0000 UTC, 2022-11-15 10:00:00 +0000 UTC
2023-03-09T02:54:55.974Z	D	cost	connector: aws, step sync within 2022-11-15 10:00:00 +0000 UTC, 2022-11-16 10:00:00 +0000 UTC
...
2023-03-09T02:56:00.365Z	D	cost	connector: aws, step sync within 2023-03-06 10:00:00 +0000 UTC, 2023-03-07 10:00:00 +0000 UTC
2023-03-09T02:56:02.514Z	D	cost	create 8 clusterCosts, 160 allocationResourceCosts for connector:aws, within 2023-03-06 10:00:00 +0000 UTC, 2023-03-07 10:00:00 +0000 UTC
2023-03-09T02:56:02.514Z	D	cost	connector: aws, step sync within 2023-03-07 10:00:00 +0000 UTC, 2023-03-08 10:00:00 +0000 UTC
2023-03-09T02:56:02.554Z	I	cache	expired:       74 B | ServeUiIndex
2023-03-09T02:56:05.745Z	D	cost	create 24 clusterCosts, 483 allocationResourceCosts for connector:aws, within 2023-03-07 10:00:00 +0000 UTC, 2023-03-08 10:00:00 +0000 UTC
2023-03-09T02:56:05.745Z	D	cost	connector: aws, step sync within 2023-03-08 10:00:00 +0000 UTC, 2023-03-09 10:00:00 +0000 UTC
2023-03-09T02:56:07.704Z	D	cost	create 16 clusterCosts, 344 allocationResourceCosts for connector:aws, within 2023-03-08 10:00:00 +0000 UTC, 2023-03-09 10:00:00 +0000 UTC

Wrong date range when requesting data for this month

Version: main
UI: dev-0141927
Commit ID: 73c9d29
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Create a kubernetes connector.
  2. Create a cost analysis perspective and select time range for this month. Dropdowns for Group By, Granularity and Filter have no data.

image

  1. Check the api request. The endTime field is missing a timezone.

image

cost table has no data when group by workload

Version: main
UI: dev-454e8cb
Commit ID: 36795ae
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Create a Kubernetes connector and there are some cost datas.
  2. View the build-in cluster perspective. There is no data on workload table.

image

  1. Check the api request.

image

  1. Check the response.

image

  1. Check custom perspective group by workload.

image

Application deployment

Implement an app deployment framework that works for different infrastructure.

To begin with, make kubernetes deployment works.

Inconsistent data: cost statistics

Version: main
UI: dev-c8b5dd2
Commit ID: 73c9d29
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Create a kubernetes connector called aws and enable cost analysis.
  2. Navigate to Cost Analysis page and create a new perspective as below.

image

  1. View the default perspective. The total cost is inconsistent with the cost of cluster (only one cluster).

image

  1. View the custom perspective created in step 2. The cost is also inconsistent.

image

Issues of custom cost analysis gragh

Version: main
UI: dev-0141927
Commit ID: 73c9d29
Env: Ubuntu 22.04 amd64

Test Steps:

Issue 1: Display daily cost group by Cluster Name. View the cost analysis graph and it shows wrong data.
image
image

Issue 2: Select granularity as cumulative. Missing some information.
image
image

Issue 3: Correct the abscissa name when showing data by monthly and yearly.
image
image

Inconsistent data: daily cost after modifying date range

Version: main
UI: dev-0141927
Commit ID: 73c9d29
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Select the date range from 2023-03-02 to 2023-03-08. Check the cost for March 7.

image

image

  1. Change the date range to 2023-03-02 - 2023-03-07 , and check the cost for March 7 again. Compare with the amount in step 1. They are inconsistent.

image

image

  1. View the cluster perspective and repeat the test steps. It also has this issue.

image

Environment management

Environment is the deployment target of applications which contains infrastructure configurations.

Template management

  1. Implement template management
  2. Define schema for template input variables
  3. Add built-in templates. e.g., webservice from image, webservice from code, Aliyun RDS, Mysql by helm.

No response after clicking save to edit custom perspective

Version: main
UI: dev-a12891a
Commit ID: fc3ae47
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Create a Kubernetes connector and create a perspective for this connector.
  2. Delete the connector.
  3. Re-add the connector.
  4. Edit the custom perspective created in step 1, and click Save.

Test Result:

  1. Failed to save. No response after clicking Save.

image

Configuring shared cost returns some errors

Version: main
UI: dev-0141927
Commit ID: 73c9d29
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Create a cost analysis perspective as below:

image

  1. View the custom perspective created in step 1. It returned some errors

image

api response:

{"message":"error query allocation cost: error query shared allocation cost: model: invalid field \"date_trunc('day', (start_time AT TIME ZONE 'UTC-8'))\" for query","status":500,"statusText":"Internal Server Error"}

Failed to execute CostCollectCronExpr task

Version: main
UI: dev-c8b5dd2
Commit ID: 73c9d29
Env: MacOS 13.1, Apple M2

Test Steps:

  1. Get the kubeconfig file of seal container:
docker exec -i <seal-container-id> cat /root/.kube/k3s.yaml
  1. Create a connector called myseal using the kubeconfig file obtained in step 1.
  2. Make sure the cluster cost analysis tool is available.
  3. Waiting for collecting cost task to be executed.

Test Result:

  1. Failed to execute CostCollectCronExpr task. See the log below for details

logs:

2023-03-07T06:59:59.966Z	D	gopool	state: tasks 1/72 workers 32/40
2023-03-07T06:59:59.966Z	D	cost	collect cost for connector: myseal
2023-03-07T06:59:59.971Z	D	cost	connector: myseal, current sync costs within 2023-03-07 01:00:00 +0000 UTC, 2023-03-07 06:00:00 +0000 UTC
2023-03-07T06:59:59.971Z	D	cost	connector: myseal, step sync within 2023-03-07 01:00:00 +0000 UTC, 2023-03-07 06:00:00 +0000 UTC
2023-03-07T07:00:03.916Z	E	error executing CostCollectCronExpr task: decode response from https://127.0.0.1:6443/api/v1/namespaces/seal-system/services/http:seal-opencost:9003/proxy/prometheusQueryRange?duration=1h&end=2023-03-07T02%3A00%3A00.000Z&query=avg%28avg_over_time%28kubecost_cluster_management_cost%5B1h%3A5m%5D%29%29&start=2023-03-07T01%3A00%3A00.000Z: readObjectStart: expect { or n, but found E, error found in #1 byte of ...|Error runni|..., bigger context ...|Error running query avg(avg_over_time(kubecost_clus|...
github.com/seal-io/seal/utils/log.DelegatedLogger.Errorf
	github.com/seal-io/seal/[email protected]/log/delegate.go:136
github.com/seal-io/seal/utils/log.Errorf
	github.com/seal-io/seal/[email protected]/log/logger.go:168
github.com/seal-io/seal/utils/cron.timeoutTask.Process
	github.com/seal-io/seal/[email protected]/cron/cron.go:127
reflect.Value.call
	reflect/value.go:584
reflect.Value.Call
	reflect/value.go:368
github.com/go-co-op/gocron.callJobFuncWithParams
	github.com/go-co-op/[email protected]/gocron.go:106
github.com/go-co-op/gocron.(*executor).start.func1.2
	github.com/go-co-op/[email protected]/executor.go:93
github.com/go-co-op/gocron.(*executor).start.func1.3
	github.com/go-co-op/[email protected]/executor.go:110
golang.org/x/sync/singleflight.(*Group).doCall.func2
	golang.org/x/[email protected]/singleflight/singleflight.go:189
golang.org/x/sync/singleflight.(*Group).doCall
	golang.org/x/[email protected]/singleflight/singleflight.go:191
golang.org/x/sync/singleflight.(*Group).Do
	golang.org/x/[email protected]/singleflight/singleflight.go:104
github.com/go-co-op/gocron.(*executor).start.func1
	github.com/go-co-op/[email protected]/executor.go:102

prometheus.log

Inconsistent data: Issue for shared cost statistics

Version: main
UI: dev-b2fc910
Commit ID: fc3ae47
Env: MacOS 13.1, Apple M2

Test Steps:

Configuration of perspective as below:

  1. Create a perspective called shared for apportionment.

image

  1. Create a perspective called container without defining shared cost.

image

  1. View the cost analysis of shared. The total cost does not match the sum of the individual costs.

image

  1. Check the total cost of cluster and the total cost of containers:
    Workload cost is not equal to container cost.

image

Application management

Provide status, logs, exec capabilities for application instance. Start with kubernetes & helm resources

Viewing cost analysis of cluster returns an error

Version: main
UI: dev-454e8cb
Commit ID: cc89f02
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Do not create project.
  2. Navigate to cost analysis page. Select Cluster view. It returned an error.

image

{"message":"invalid project: blank","status":400,"statusText":"Bad Request"}

Optimizing table for cost analysis custom perspective

Version: main
UI: dev-a29954e
Commit ID: 73c9d29
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Create a kubernetes connector and enable cost analysis.
  2. Create a cost perspective as below. The granularity is daily.

image

  1. View the perspective created in step 2. It only shows cost for the most recent day in the table below the graph, and it has no date, which would make users confused.

image

Failed to set filter for custom perspective of cost analysis

Version: main
UI: dev-c8b5dd2
Commit ID: 73c9d29
Env: Ubuntu 22.04 amd64

Test Steps:

  1. Create a kubernetes connect.
  2. Navigate to Cost Analysis page, create a new perspective called aws as below and save:

image

  1. Edit the custom perspective created in step, and check the filter. The Namespace field became to empty.

image

  1. View the aws perspective and check the api response. The namespace filter did not take effect.

image

image

Using build-image and webservice modules at the same time returns error

Version: main
UI: dev-ae9ee1f
Commit ID: 01407be
Env: MacOS 13.1, Apple M2

Test Steps:

  1. Create an application and add modules as below.

image

image

  1. Deploy an instance. There is an error returned: version 0.0.0 of module build-image not found.

image

  1. It can be deployed successfully if using build-image module or webservice module separately.

The legend of the custom perspective is overwritten by the previous view

Version: main
UI: dev-a12891a
Commit ID: fc3ae47
Env: MacOS 13.1, Apple M2

Test Steps:

  1. Create two perspectives. One is grouped by container, the other is grouped by namespace.
  2. View perspective in the following order:

image

Test Results:

  1. The legend is the connector name.

image

  1. The legends are namespace names.

image

  1. The legend of connector name is overwritten by namespace name.

image

Connector management

Connector are used to connect to different infrastructures like kubernetes clusters, cloud providers.

Implement connector CRUD and validation for kubernetes, aliyun and custom.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.