tessera-metrics / tessera Goto Github PK
View Code? Open in Web Editor NEWA dashboard front-end for graphite.
Home Page: http://tessera-metrics.github.io/tessera/
License: Apache License 2.0
A dashboard front-end for graphite.
Home Page: http://tessera-metrics.github.io/tessera/
License: Apache License 2.0
Basic layout transforms are in place on the client side - having them as part of the JavaScript model works best. Right now a layout class is super-simple; it takes a dashboard item (such as the definition) and returns another dashboard item with the contents re-arranged. Number of column switching is handled this way.
Rename layouts to transforms - the one convention can cover basic re-layout (i.e. column switching) and drill-downs, where we take one graph and get more detail on it.
Dashboard items currently have a query_name
to reference a query stored in the dashboard_definition
.
ds.models.data.Query
needs to have options from
/until
fields as well) for showing the same metric over multiple time scalesDashboardManager
but cleaner) to handle transforms, and managing browser history API.
In order to display a proper breadcrumb navigation trail for drilldowns, we're going to need to move the breadcrumb template client-side as well.
(at this point, it looks like almost all the UI will client-side eventually, leaving the server side code to handle API, persistence, and serving static assets).
Move expansion of graphite queries & template processing of same to the client side. Doing it server side is tidy, but pybars
helper extensions are pretty broken, and the project seems abandoned.
Client side has some definite integration advantages -- adding a clusto tag helper is as simple as including a .js
file that calls Handlebars.registerHelper()
. No muss, no fuss.
Also, this will reduce the amount of duplicated code/hackery for editing.
Much of the rendering & interaction with presentations relies on the "convention over configuration" approach, which I'm not a huge fan of (i.e. the way templates for an item aren't specified by any metadata, but are just assumed to be available at ds.templates.models.<type>
). This also makes it harder to cleanly add new ones, since you need to add a lot of individual objects to many different places.
That should be centralized & driven by simple javascript options.
A presentation needs:
Python packaging, oh joy!
Good references:
With the implementation of issue #201, we'll have two packages - tessera-client
and tessera
.
Add the ability to link to dashboards from any dashboard item (or at the very least from charts - charts will be easiest because they already have an action menu). The action menu would read from the links property to build a submenu to take you to any related dashboards.
This would allow linking from summary dashboards to drilldowns (i.e. a high level board with one summarized chart per sub-system; each chart could link to details for that sub-system, providing easy drill-downs)
The number of application specific CSS classes has grown very quickly without much naming standardization. They should be cleaned up and given a common prefix, to make them easy to distinguish from 3rd party classes (i.e. bootstrap).
Recommend using a short prefix like the bs-classname
convention from bootstrap's documentation CSS.
If you're in display mode, selecting the data range will drop you out of it (because of the current hack of setting window.location without considering all UI state)
Templated queries only get rendered at load time, not during editing.
Any tag system eventually wants to have tag classes. Tag classes would be useful for setting tag colors for a whole set of tags (i.e. tags for services, servers, users), making the visual distinguishable without having to set the colors for all individual tags.
At present, bad things will happen if you have embedded newlines in a query target. This is inconvenient for editing, because it's easier to understand the query if it's indented. Newlines should be preserved in stored dashboard queries, but stripped before building the graphite URL.
random musings on organizing the python code (from chat logs):
[6:56 PM] alpern: - thinking about extracting web.py into a separate package (`tessera-client`, probably), and bundling it with a client class that uses `requests` to communicate w/the tessera API
[6:56 PM] alpern: - both importers could then use the API instead of interacting directly w/the SQLAlchemy model
[6:56 PM] alpern: - which would mean they could be run from anywhere
[6:57 PM] alpern: move into a separate package so that it doesnt pull in flask & sqlalchemy (which needs a C compiler) if you just want to script creation of some dashboards or something
[6:58 PM] alpern: the only thing `web.py` depends on is `json`
[6:58 PM] alpern: Also want to refactor the web.py classes so that they'll still function if parsing encounters an `item_type` value that doesn't map to a python class -- you'd just get a generic base class, but the JSON data would pass through
[6:59 PM] alpern: that would allow for extensions/integrations that add new item type w/o needing to modify the server
[6:59 PM] alpern: well, w/o needing to modify the python server at anyrate
[6:59 PM] alpern: would cut down on the effort to add a new item type
That technique would let us have the tessera.model
and tessera.client
modules live in one package (tessera-client
or tessera-base
), and tessera.server
and tessera.database
live in tessera
proper.
In building out internal dashboards with tessera, I've been finding that I re-use queries quite a lot, particularly when building out related dashboards where there's a high-level summary and a lot of breakouts.
Being able to create global queries, which can be referenced in multiple dashboards, would cut down on redundancies and also allow query updates to be reflected globally without a lot of replicated effort (DRY!)
Add a variant of the summation_table
dashboard presentation type that takes two queries and compares their summations. The second query could be manually assigned, or could be derived from the first with a time range modifier (i.e. compare now to a week ago, etc...)
See http://blog.stathat.com/2014/04/09/web-app-interface-changes-stats.html for good examples.
Instead of having all y-axis options just be "yAxisThis" & "yAxisThat" on the chart.options
attribute, create an axis
model object, and give each chart an array of them.
This should be considered a prerequisite to adding more sophisticated axis options to the edit UI, so they don't have to be re-worked later.
As a python command, like the graphite importer, but read a directory full of JSON files in the tessera schema; or dump all dashboards to JSON (for archival, transport, backup, etc...)
# Import one or more files
./manage.py import_json <file*>
# Export all dashboards, or those with a specific tag to JSON
./manage.py export_json <directory> --tagged [tag]
``
Handling of query options has gotten pretty crufty, and keeps breaking. At the moment the recent range picker doesn't work reliably. the fire_only
flag has been cleaned up a bunch - it's time to clean up the rest. It's probably a prerequisite to cleanly implementing #125 and #126, as well as any time-related transforms (which are also broken right now).
Add a format string to allow formatting of the numbers in the cells of summation_table
.
Just like the comparison summation table (issue #125), add a variant of the ``singlestat` presentation type that compares a single summation value between two queries. Again - queries can both be explicitly selected, or one can be derived from the first plus a time range modifier.
Also timeshift variant (like timeshift_summation_table
)
The <img/>
tag is being added to the ds-graph-holder
with $(e).html(), replacing the contents. The text elements are inside the ds-graph-holder
. They probably should not be. Either that or $(e).append()
will be sufficient in non-interactive mode, since the <svg/>
element will just remain empty.
When editing a dashboard, sometimes the item IDs end up conflicting. Haven't isolated it yet, but it always happens when duplicating & deleting existing elements.
browserify's analysis of require() chains to generate the loading order should cut down on the manual munging of the load order in Gruntfile
A very simple first interactive dashboard editor could simply start with replicating the basics of what the built-in graphite dashboard editor can do, i.e. take an existing graph URL and import it into a grid.
No frills, not many options, but would be a start we can iterate off of.
Riffing on a comment from @bitprophet in the #metrics
channel. Useful dashboard snapshotting doesn't have to mean attempting to render a static version of the dashboard to paste into a ticket system or anything like that. In fact, we can be much more useful than that.
A snapshot should just be a record of:
The definition and data series are both JSON blobs. Storing the data ensures that the snapshot will remain valid after the data ages out of our Graphite storage. Likewise, the definition is needed in case the dashboard changes in the future.
Pretty straightforward to render -- the existing query map just gets the string targets replaced with the ds.models.data.Query
instances, serialized to JSON. load()
just looks to see if it already has data, and fires the DATA_AVAILABLE
event immediately, instead of calling out to graphite. ๐ฅ
Needs 1 new database model object/table (DashboardSnapshot
), a set of API endpoints to manage snapshots (/api/dashboard/<id>/snapshot
), and a new front end template page (dashboard-snapshot.html
) that removes some of the interactive elements of dashboard.html
(i.e. range selector, auto refresh menu, etc...)
Oh, and obviously a little UI chrome to take a snapshot, edit the description, and access them from the dashboard listing.
At present you can re-order dashboard items within a container (sections can be reordered within the dashboard, rows within sections, cells within rows, etc...), but you can't move an item from one container to another (i.e. move a row between sections, or move a chart from one cell to another in a different row & section).
Being able to do so via drag & drop would be ideal, however that's likely to be a rather large undertaking.
Being able to reuse dashboard definitions is desirable, particularly for templatized dashboards. For example, a dashboard could define set of common graphs for Java services (with the actual metrics in the query being determined at runtime via. a call to clusto with a parameter from the URL, for example). That dashboard can be included in other dashboards that add on service-specific metrics.
Something like:
{
"item_type" : "dashboard_reference",
"id" : "d12",
"dashboard_href" : "/api/dashboard/12"
}
/api/dashboard/<id>/<slug>?rendering=true
should walk through the dashboard definition and expand any dashboard references. This should be much faster than doing it with additional HTTP requests from the client side.Need to work out exactly what processing is done to an included dashboard definition. Probably none - nesting a section inside a cell, for example, should still be fine, I think. Dashboard references would be allowed at top level, in a section, or in a cell.
Some links to get started:
StaleDataError: DELETE statement on table 'dashboard_tags' expected to delete 1 row(s); Only 2 were matched.
2014-06-23 02:00:53 [24643] [ERROR] Error handling request
Traceback (most recent call last):
File "/mnt/services/tessera/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 131, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/mnt/services/tessera/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/mnt/services/tessera/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/mnt/services/tessera/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/mnt/services/tessera/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/mnt/services/tessera/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/mnt/services/tessera/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/mnt/services/tessera/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/mnt/services/tessera/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/mnt/services/tessera/lib/python2.7/site-packages/tessera/views.py", line 249, in api_dashboard_update
mgr.store_dashboard(dashboard)
File "/mnt/services/tessera/lib/python2.7/site-packages/tessera/model/database.py", line 147, in store_dashboard
db.session.commit()
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 149, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 765, in commit
self.transaction.commit()
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 370, in commit
self._prepare_impl()
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 350, in _prepare_impl
self.session.flush()
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1903, in flush
self._flush(objects)
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2021, in _flush
transaction.rollback(_capture_exception=True)
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 57, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1985, in _flush
flush_context.execute()
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 370, in execute
rec.execute(self)
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 479, in execute
self.dependency_processor.process_saves(uow, states)
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/dependency.py", line 1083, in process_saves
secondary_update, secondary_delete)
File "/mnt/services/tessera/lib/python2.7/site-packages/sqlalchemy/orm/dependency.py", line 1104, in _run_crud
result.rowcount)
Oh graphite-web, your API is such a mess.
For some dashboards in graphite-web that have multiple targets, the targets array in the API is not, in fact, an array, but A URL encoded query string with multiple target=<target>&etc...
clauses.
The importer needs to detect this case (target array is a string containing "target="
) and parse the query string.
Just that -- upon entering edit mode, a spurious PUT is being issued to store the dashboard metadata. This may be exacerbating #185
The CSS code has ended up w/ a lot of copy & paste; reduce that with SASS. With the theming support, that will also greatly reduce the burden of keeping color themes up to date and creating new ones
My first attempt with browserify wasn't very successful, and it stopped me from actually evaluating gulp because I didn't get past the browserify step.
But gulp could be used w/o browserify for the simple concat case we need, and definitely looks like the good way forward (and is where UA is heading).
https://www.npmjs.org/package/gulp-handlebars actually looks like a really good way to organized the Handlebars templates in a reasonable manner.
Like the Isolate & Time Span transforms which reveal additional detail about a given chart, a transform that displays all available latency measurements for a given root metric.
For example, if you use Coda Hale's Metrics library for java, you may have timer metrics which look like this, depending on how you collect them:
For each timer metric, there are min, max, mean, a raw count, and 5 percentiles. From a dashboard chart that displays any one of those, transforming to display all of them would allow digging down to all the data without overwhelming a dashboard by having it all on immediate display.
This is a placeholder issue for the moment, but:
Simple example:
ds.actions.register({
scope: 'presentation',
mode: 'edit',
action: ds.models.action({
name: 'set-chart-type-stacked-area-chart',
display: 'Stacked Area Chart',
icon: 'fa fa-picture-o',
handler: function(item, action) {
item.set_item_type('stacked_area_chart')
ds.manager.update_item_view(item)
}
})
})
Add a new dashboard presentation type which plots a scatterplot using two data series as axes (rather than assuming time for one axis). nvd3 does very nice scatter plots, as does flot.
I don't think graphite-web can do it, so that would be an interactive-mode only feature (the model object can set the interactive=true override in its constructor)
Add commands to export an entire dashboard as JSON, or to import one from same
Opening this now as a general thread for relevant documentation ides.
Basically, the flot
renderer has been color-tweaked for dark mode, but the light modes are a mess.
The on-mouseover behavior of the property sheets can be convenient, but it's also leading to the property sheet disappearing without being able to edit sometimes, particular when the cell is very narrow and an edit widget (like, ahem, the query menu) ends up outside the bounds of the property sheet div
.
Proposed behavior:
click
instead of mouseenter
(that will cut down on unintended opening that results in confusing quick UI relayout)mouseleave
set a 5 second timer to automatically close the sheet. If the mouse enters the property sheet div again (mouseenter
) before the timer expires, cancel itNeed ability to pick absolute values for from
& until
Currently the map of queries in a dashboard_definition
is just a string -> string map, mapping a query name to a single graphite target string.
That should be changed to use the JSON representation of the ds.models.data.Query
(link) object, which can represent an array of targets and additional data.
Handling the structured query is also a prerequisite to implementing snapshots (issue #65).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.