adriens/schemacrawler-plugin-dashboard

Schemacrawler plugin that exports lint reports as csv

schemacrawler-plugin-dashboard

Schemacrawler additional command (ie. plugin) that exports lint reports as a set of precompiled and aggregated csv, then provides a ready to use Html dashboard compiled from a R, thanks to a flexdashbaord.

Just run the dashboard command, provide an output directory and take a look the newly created index.html.

Prerequisites

Proprer schemacrawler install R runtime installed the schemacrawler-plugin-dashboard jar in schemacrawler lib dir.

Install

Drop the jar in $SCHEMACRAWLER_HOME/lib and you're done.

Usage

Assuming you have a properly setup schemacrawler environment within schemacrawler.config.properties

schemacrawler -c=dashboard

or

schemacrawler -c=dashboard -outputdir=my_output_dir

Output samples

schemacrawler-plugin-dashboard's Issues

csv file contains linters instead of lints

Section "About"

Add an "About" section to the markdown report

Generate additional csv > list of tables

grab following properties :

numRows
tableNode.setProperty("nbColumns", table.getColumns().size());
tableNode.setProperty("definition", table.getDefinition());
tableNode.setProperty("lookupKey", table.getLookupKey());
tableNode.setProperty("name", table.getName());
tableNode.setProperty("remarks", table.getRemarks());
tableNode.setProperty("schemaName", table.getSchema().getName());
tableNode.setProperty("fullName", table.getFullName());
tableNode.setProperty("tableType", table.getTableType().toString());

R > Implement heatmap report on lints hits

on

X -> table
Y -> lint

f(x,y) : nbHits

http://www.htmlwidgets.org/showcase_d3heatmap.html

BONUS : Find the best sort on axis to show emerging communities

Create a Logo

R prerequisites >rmarkdown install

Got error during html compilation :

➜  dashboard ./dashboard_rscript.R 
Error in loadNamespace(name) : there is no package called ‘rmarkdown’
Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
Execution halted
➜  dashboard 

Run infos

Add a section dedicated to runtime infos (date utc and local, time, database url,...)...and job uniqueId

It could be interesting to produce a csv that contains them rather than hardcode it in the Rmd

Generate additional csv > columns

generate csv with tableFullName as first column will be used to make reports between columns csv and table csv

and all other available available datas :

columnNode.setProperty("columnOrdinalPosition", column.getOrdinalPosition());
                        columnNode.setProperty("columnDataType", column.getColumnDataType().toString());
                        columnNode.setProperty("name", column.getName());
                        columnNode.setProperty("fullName", column.getFullName());
                        if (column.getDefaultValue() != null) {
                            columnNode.setProperty("defaultValue", column.getDefaultValue());
                        }

                        if (column.getLookupKey() != null) {
                            columnNode.setProperty("lookupKey", column.getLookupKey());
                        }

                        if (column.getRemarks() != null) {
                            columnNode.setProperty("remarks", column.getRemarks());
                        }

                        if (column.getShortName() != null) {
                            columnNode.setProperty("shortName", column.getShortName());
                        }
                        columnNode.setProperty("size", column.getSize());
                        columnNode.setProperty("width", column.getWidth());

Remove all the R stuff as switching to a much cooler stack !

Upgrade to tablesaw 0.11.0

Dependencies upgrde

https://www.versioneye.com/user/projects/596fe9a16725bd005f47fe8b

Use outputfile parameter instead of filename

R > collapsable tree on lints

table-> column -> lint (severity as color)

severity ->table

an other idea welcome ;(p

https://adeelk93.github.io/collapsibleTree/

R > Implement DataTables view on the raw csv lint output

http://www.htmlwidgets.org/showcase_datatables.html

Rmarkdown report

Implement a Rmarkdown report on the csv output of the lint, using ggplot2 lib for graphs so we can get a dashboard.

1. FIrst, develop the markdown on the repo (histograms, piecharts, ...heatmap)
2. Provide /install in the r the packages into the R report in a dedicated hidden section so the report can easily be compiled
3. At the end, the report should be compiled from command line thanks to knitr.
4. Install test could be done inside a Vagrant vm
5. Give a try to shiny to get a locally running app

NB : If additional datas are required in the csv, please tell and i'll implement it, but the goal would to push all this stuff on the R side

Definition Of Done :

You run the lint command, then, through a command line, you get a pdf, html. For now, we'll manually document the command line...but in a second step it would be possible :

to create a nice to use makefile
an ant make file (think about windows users...)
embed the Rmarkdown inside the jar and trigger (at least try) the Rmarkdown build.

This feature should be easily : installable in the debian package...deployable on Travis