pingcap/tidb-dashboard

A Web UI for monitoring, diagnosing and managing the TiDB cluster.

TiDB Dashboard

TiDB Dashboard is a Web UI for monitoring, diagnosing and managing the TiDB cluster.

Documentation

• Product User Manual (English)
• Product User Manual (Chinese)
• FAQ (English)
• FAQ (Chinese)

Question, Suggestion

Feel free to open GitHub issues for questions, support and suggestions.

You may also consider to reach out on the TiDB Internals forum if you encounter any problems about TiDB development.

For Chinese users, you can visit the PingCAP official user forum AskTUG.com to make life easier.

Getting Started

The most easy way to use TiDB Dashboard with an existing TiDB cluster is to use the one embedded into PD: http://127.0.0.1:2379/dashboard. You need PD master branch or 4.0+ version to use TiDB Dashboard.

Note: The TiDB Dashboard inside PD may be not up to date. To play with latest TiDB Dashboard, build it from source (see next section).

Contributing & Developing

Checkout our help wanted issues for a list of recommended tasks, in which we have also marked the difficulty level.

See CONTRIBUTING.md for a detailed step-by-step contributing guide, or steps to build TiDB Dashboard from source.

If you need any help, feel free to reach out on the TiDB Internals forum.

Thank you to all the people who already contributed to TiDB Dashboard!

Architecture

This repository contains both Dashboard HTTP API and Dashboard UI. Dashboard HTTP API is placed in pkg/ directory, written in Golang. Dashboard UI is placed in ui/ directory, powered by React.

TiDB Dashboard can also be integrated into PD, as follows:

License

Apache License

Copyright 2020 PingCAP, Inc.

tidb-dashboard's Issues

Log Export UI and Integration

A manage page for user to export the log should be implemented. Task status should be updated in time.

Fix KeyVisual hoverbox

Currently KeyVisual hoverbox will overflow.

Tech design for cluster info (TiDB, TiKV & PD)

Investigate I18N default language issue

It is reported that in some scenarios the default language cannot be correctly recogized, result in displaying translation keys, instead of translation words.

How to use this?

Sorry...I don't see any deployment guide in this repo.

Report topology info from TiKV

Report grafana & am address in ansible

Add I18N Support

In addition to ZH_CN UI, also support EN UI.

Support Standalone PD Port

In order to allow firewalls blocking PD API without blocking Dashboard API and UI, Dashboard should be supported to run on standalone port when integrated with PD.

Basic Support PD Integration

Support running TiDB Dashboard using PD binary.

Cluster topology frontend

Implement the frontend page of cluster topology.

Cannot connect to PD after PD is started

Reproduce:

1. Start Dashboard
2. Connect PD (fail)
3. Start PD
4. Connect PD (fail)

Provide topology & status API in Dashboard

The topology API should be able to report the topology of TiKV, PD, TiDB, Grafana(if any) and AlertManager(if any), where status should be attached to TiKV, PD and TiDB.

rpc error: code = ResourceExhausted

When searching too many logs:

rpc error: code = ResourceExhausted desc = grpc: received message larger than max (18696402 vs. 4194304)

Support Listing Topology

Support listing the cluster (TiDB, TiKV, PD) topology, including IP, port, status port, store ID (TiKV only), label (TiKV only), deployment path, version.

Tech design for Flamegraph backend

Support Standalone Dashboard Binary

Run TiDB Dashboard in a standalone way.

Introduce better mono font

We can ship SourceCodePro by default and use it as our monospace font to improve user experience.

Elect only one PD to start Dashboard

Now PD natively includes the Dashboard. When PD starts, Dashboard starts. However if there are multiple PD nodes, all PD nodes will start a Dashboard instance, which wastes resource.

We should update PD to elect only one PD node to start Dashboard by default.

Introduce DI

Currently we maintain dependencies manually, which is too verbose. We can use DI to make life simple.

Tech design for cluster info (grafana & alertmanager)

In Dashboard, users should be able to navigate to the Grafana and AlertManager. Thus these addresses should be known by Dashboard.

Finalize a template for cluster report

Report topology info from PD

make dev run tidy twice

$ make dev
scripts/lint.sh
+ Check golangci-lint binary
+ Run lints
+ Clean up go mod
SWAGGER=1 make server
make swagger_spec
scripts/generate_swagger_spec.sh
+ Install swagger tools
+ Clean up go mod
+ Generate swagger spec
...

UCP: Add TLS Support

Description

This task is to make TiDB Dashboard support TLS TiDB cluster. When Dashboard is running in standalone mode, TLS certificates of the TiDB cluster can be supplied by passing cli arguments. When Dashboard is running with PD, TLS certificates can be obtained from PD cli configuration.

You can see more descriptions in #174

Score

• 1680

Note: You can get 2530 in total if #177 is also implemented.

Mentor(s)

• @breeswish

Contact the mentors: #tidb-challenge-program channel in TiDB Community Slack Workspace

Recommended Skills

• Security
• Golang
• Web programming

Cluster Report Fronted Page

Implement the report frontend page.

Log Export API

Implement the log export API according to the tech design document.

Add Gzip Handler

We store assets in gzip now. We can further serve them as gzip as well.

Can not get RegionsInfo

After starting two PDs and use make run to start the dashboard:

[2020/02/24 14:46:37.781 +08:00] [ERROR] [periodic.go:53] ["can not get RegionsInfo"] [error="json: cannot unmarshal string into Go value of type input.RegionsInfo"] [stack="github.com/pingcap/log.Error\n\t/Users/Ryan/PingCAP/go/pkg/mod/github.com/pingcap/[email protected]/global.go:42\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual/input.(*periodicInput).Background\n\t/Users/Ryan/PingCAP/tidb-dashboard/pkg/keyvisual/input/periodic.go:53\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual.(*Service).Start.func2\n\t/Users/Ryan/PingCAP/tidb-dashboard/pkg/keyvisual/service.go:88"]
[2020/02/24 14:47:37.780 +08:00] [ERROR] [periodic.go:53] ["can not get RegionsInfo"] [error="json: cannot unmarshal string into Go value of type input.RegionsInfo"] [stack="github.com/pingcap/log.Error\n\t/Users/Ryan/PingCAP/go/pkg/mod/github.com/pingcap/[email protected]/global.go:42\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual/input.(*periodicInput).Background\n\t/Users/Ryan/PingCAP/tidb-dashboard/pkg/keyvisual/input/periodic.go:53\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual.(*Service).Start.func2\n\t/Users/Ryan/PingCAP/tidb-dashboard/pkg/keyvisual/service.go:88"]
[2020/02/24 14:48:37.782 +08:00] [ERROR] [periodic.go:53] ["can not get RegionsInfo"] [error="json: cannot unmarshal string into Go value of type input.RegionsInfo"] [stack="github.com/pingcap/log.Error\n\t/Users/Ryan/PingCAP/go/pkg/mod/github.com/pingcap/[email protected]/global.go:42\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual/input.(*periodicInput).Background\n\t/Users/Ryan/PingCAP/tidb-dashboard/pkg/keyvisual/input/periodic.go:53\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual.(*Service).Start.func2\n\t/Users/Ryan/PingCAP/tidb-dashboard/pkg/keyvisual/service.go:88"]
[2020/02/24 14:49:37.786 +08:00] [ERROR] [periodic.go:53] ["can not get RegionsInfo"] [error="json: cannot unmarshal string into Go value of type input.RegionsInfo"] [stack="github.com/pingcap/log.Error\n\t/Users/Ryan/PingCAP/go/pkg/mod/github.com/pingcap/[email protected]/global.go:42\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual/input.(*periodicInput).Background\n\t/Users/Ryan/PingCAP/tidb-dashboard/pkg/keyvisual/input/periodic.go:53\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual.(*Service).Start.func2\n\t/Users/Ryan/PingCAP/tidb-dashboard/pkg/keyvisual/service.go:88"]

Version info
dashboard: master
pd: master (bf4aba5)

Shutdown backend services dynamically

After supporting dynamic enabling / disabling Dashboard in PD as in #21 , we need to ensure that when Dashboard is disabled, all Dashboard backend services are shutdown, for example, SQLite should be closed properly and no more resources are consumed.

Cluster report API

The API should be able to generate a cluster report html, using values filled from TiDB diagnose API.

Support TiDB Authentication

Support log into Dashboard using TiDB username and password.

Support Standalone Authentication

Dashboard should be able to sign in use standalone credentials, which is stored in etcd, so that dashboard can auth users even if there is no TiDB.

UCP: Support super user login without TiDB

Description

Now TiDB Dashboard can only be logged in with existing TiDB node. However for pure TiKV deployments there is no TiDB node, makes user unable to use TiDB Dashboard.

For such scenario, we need to provide alternative login method: supply a super user login username and password. The username and hash of the password is stored in PD etcd so that multiple PD nodes can share the same authenticator.

Even if there is a TiDB node in the cluster, when log in use super user, TiDB related functionalities like Statements should not be available (since they requires a valid TiDB connection).

Score

• 1680

You will be rewarded with extra 300 points if UI is also implemented.

Mentor(s)

• @breeswish

Contact the mentors: #tidb-challenge-program channel in TiDB Community Slack Workspace

Recommended Skills

• Security
• etcd
• Go web programming

An error occurred while zooming the page

step:

• Access dashboard
• Switch to any page other than Key Visualizar
• Zoom page

then got "NotSupportedError: Failed to read the 'value' property from 'SVGLength': Could not resolve relative length."

Standalone Dashboard on PD

For easier deployment, we should update PD to support PD binary running Dashboard exclusively, i.e. only Dashboard functionality is provided and no PD functionality.

Discussion: Is it better to use standalone Dashboard binary instead of PD binary to fulfill this functionality?

Tech design for log export backend

Etcd log does not follow log format

In standalone mode, etcd log is using its own format like this:

[2020/02/20 22:06:06.122 +08:00] [WARN] [tidb_requests.go:71] ["get key failed"] [key=/tidb/server/info] [error="context deadline exceeded"]
[2020/02/20 22:06:54.254 +08:00] [ERROR] [periodic.go:53] ["can not get RegionsInfo"] [error="Get http://127.0.0.1:2379/pd/api/v1/regions: dial tcp 127.0.0.1:2379: connect: connection refused"] [stack="github.com/pingcap/log.Error\n\t/Users/breezewish/go/pkg/mod/github.com/pingcap/[email protected]/global.go:42\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual/input.(*periodicInput).Background\n\t/Users/breezewish/Work/PingCAP/tidb-dashboard/pkg/keyvisual/input/periodic.go:53\ngithub.com/pingcap-incubator/tidb-dashboard/pkg/keyvisual.(*Service).Start.func2\n\t/Users/breezewish/Work/PingCAP/tidb-dashboard/pkg/keyvisual/service.go:86"]
{"level":"warn","ts":"2020-02-20T22:06:55.248+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-72fa7705-b76a-47a4-a64e-8ff99c606a99/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}

which is annoying.

Integrate KeyVis with DBaaS

The KeyVis component should be integrated with DBaaS. API endpoints, front-end assets and so on need to be changed accordingly.

Reduce the frequency of KeyVisual get TiDB table information

Key Visual now gets table information from TiDB every minute.
SchemaVersion can be used to skip updates when the table information has not changed.

Implement FlameGraph UI

We need a control page, similar to log search page, which allows user to select components to grab the flamegraph, as well as start collecting the flamegraph.

Direct Access Grafana

When accessing from Dashboard, user should not be prompted for passwords.

Statement should display more detail for `commit`

Currently in optimistic transaction, most of the write time cost will be counted over commit. As in statement, we will see that a single commit occupies a lot of time. It is essential to allow users view the detail of this single commit, e.g. how many time each "prev statement" costs.

Tech design for Statement backend

TiDB statements SQL is ready. We need to design what kind of interfaces Dashboard should expose, as well as how these interfaces will invoke the Statement SQL.

Polish I18N translations

Currently I18N translation is written casually. We need to polish these translations.

PD Dynamic Enable/Disable Dashboard

Users should be able to choose which PD node serves Dashboard dynamically using pd-ctl.

Provide remove TiDB topology API

When TiDB node is removed, its topology remains. We need to allow user hide specific TiDB node from the topology. If the TiDB node is still alive, this operation should not be allowed. If the TiDB node is partitioned, the TiDB itself should be able to re-add the topology back when partition ends.

Navigate to Grafana and AlertManager

When Grafana or AlertManager exists in the topology, the frontend should provide a link for user to navigate.

Report topology info from TiDB

Implement FlameGraph backend

The backend in Dashboard provides an API, accepts an array of targets and sends flame graph requests concurrently to these targets. The targets can be TiDB, TiKV or PD. The result of these requests should be stored in our Sqlite storage and the status (running, done) of each target should be recorded and can be retrieved from another interface.