tiagosiebler/mstrweblogparser

A browser-based parser built to process and browse through the log files generated by the MicroStrategy Web and Mobile servers, under MSTR/WEB-INF/log/.

MicroStrategy Log Parser

This is a simple browser-based parser built to process and browse through the log files generated by MicroStrategy Web, under MSTRWeb/WEB-INF/log/. This parser should also be able to process log files from the MicroStrategy Mobile server.

This specific project is built with:

• AngularJS + various submodules
• Bootstrap 3

Live Demo

See the latest code in-action here: https://tiagosiebler.github.io/MSTRWebLogParser/build/

This demo is a live implementation of the latest code in this repository, the link can be bookmarked for easy access.

Supported MicroStrategy Log Formats

• MicroStrategy Web server generated MSTRLog files, found in MSTRWeb/WEB-INF/log.
• MicroStrategy Web server generated AdvMSTRLogwebapi*.log files.
• MicroStrategy Mobile server log files, similar to the above.
• MicroStrategy Web & Mobile Server Kernel XML API Traces.

Issues & Enhancements

Report issues and request enhancements in the issue tracker, or contact Tiago: https://github.com/tiagosiebler/MSTRWebLogParser/issues

View the changelog of the parser: https://github.com/tiagosiebler/MSTRWebLogParser/blob/master/CHANGELOG.md

Error Handling

The logic driving the parser has error handling in place. If anything can't be parsed for any reason, it'll throw a visible error in the interface. Please let me know if this happens.

Getting Involved

Development

Installs dependencies and launches development instance in default browser. Any file-changes automatically update browser instance.

1. Install Ruby, if not yet installed

   $ gem install sass
   

2. Install gulp dependencies

   $ npm install -d
   

3. Run with gulp

   $ gulp
   

Production Build

The production build produces a zip file, or can be accessed directly via the build directory.

1. Install Ruby, if not yet installed

   $ gem install sass
   

2. Install gulp dependencies

   $ npm install -d
   

3. Build with gulp

   $ gulp build
   

Project Structure

src

Contains all source code that is modified and should be optimized in any way:

• HTML
• JavaScript modules
• CSS

build

Contains production-ready project with heavy code optimization.

dev

Contains development instance of project with minimal code processing for easier debugging. Debugger statements are not removed here.

gulpfile.js

Gulp tasks used to automate build & optimization process.

• Clean project (e.g .DS_Store files)
• Minify HTML
• Concatenate and minify JS resources into two files
• Concatenate, optimize and minify custom CSS files
• Archive production-ready zip of project

For information on gulp refer to: https://github.com/gulpjs/gulp

package.json

npm depencenies for project.

Screenshots

To be added later.

mstrweblogparser's Issues

Mark rows in dataset when selecting in list

Add support for Kernel API traces

Add support for MACfgWiz.xml logs

They have a nicely parsable XML format:

<log4j:event logger="MACfgWiz" timestamp="1486397437704" level="ERROR" thread="main">
<log4j:message><![CDATA[Could not load MFHCNat
	class java.lang.UnsatisfiedLinkError: D:\Program Files (x86)\Common Files\MicroStrategy\HealthCenter\jars\MFHCNat.DLL: Can't find dependent libraries
	java.lang.UnsatisfiedLinkError: D:\Program Files (x86)\Common Files\MicroStrategy\HealthCenter\jars\MFHCNat.DLL: Can't find dependent libraries
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1857)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at com.microstrategy.utils.registry.LoadLibrary.loadLibrary(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.microstrategy.tools.mcdgnstc.plugins.utils.registry.LoadLibrary.loadLibrary(LoadLibrary.java:21)
	at com.microstrategy.tools.mcdgnstc.baseobjects.util.FileUtilities.<clinit>(FileUtilities.java:56)
	at com.microstrategy.tools.macfgwiz.ui.common.hccomponents.DiagnosticsConfig.readFromXML(Unknown Source)
	at com.microstrategy.tools.macfgwiz.ui.common.hccomponents.DiagnosticsConfig.<init>(Unknown Source)
	at com.microstrategy.tools.macfgwiz.ui.common.hccomponents.DiagnosticsConfig.<init>(Unknown Source)
	at com.microstrategy.tools.macfgwiz.backbone.base.CfgWizOperationFactory.<init>(Unknown Source)
	at com.microstrategy.tools.macfgwiz.backbone.base.CfgWizOperationFactory.getInstance(Unknown Source)
	at com.microstrategy.tools.macfgwiz.backbone.responsefile.ResponseFileSingleton.<init>(Unknown Source)
	at com.microstrategy.tools.macfgwiz.backbone.responsefile.ResponseFileSingleton.getInstance(Unknown Source)
	at com.microstrategy.tools.macfgwiz.ui.common.ConfigurationProcedure.retrieveFromINIAndRun(Unknown Source)
	at com.microstrategy.tools.macfgwiz.ui.cmdline.MACfgConsole.main(Unknown Source)

]]></log4j:message>
</log4j:event>

Disable blue 'selected' highlight around buttons after clicking

Timestamp column, provide date range filter

Left-right navigation in inspection view uses ordered logs by ID rather than custom sorting

Not sure if this is preferred or not. If logs are custom sorted (e.g. using message col), left-right navigation is still via previous and next logs in order from originally parsed files.

Provide button to dump in-memory logs to flat JSON file

https://github.com/tiagosiebler/MSTRWebLogParser/blob/master/src/js/app/services/data.js#L36

Move search & resize buttons to left of table title.

Too easily missable on smaller screens, where table cuts off screen on right due to min width.

If <trace> is present, use that for stacktrace cell

Filtering: exclude keywords from list

Rather than just filtering for matches, allow building of an exclude list

Data parsing blocks interface updates when using larger datasets

Interesting: http://stackoverflow.com/questions/17299715/angularjs-computation-heavy-tasks
Worst case scenario, although should be avoided: https://parallel.js.org/

Prettify JSON/XML shown in 'inspect kernel log' view

Use json-explorer, similar to https://github.com/tiagosiebler/Angular_JSONAPI_Interface

Add dedicated button for row highlighting/marking

Replace pastebox with paste event

Looks like you can listen to page-level paste events:

  document.addEventListener("paste", function(e) {
    // cancel paste
    e.preventDefault();
    // get text representation of clipboard
    var text = (e.originalEvent || e).clipboardData.getData('text/plain');

    // do stuff
  });

Migrate bootstrap from 3 to 4

This uses an older version of bootstrap. Worth evaluating effort to migrate.

Sorting needs to be 'refreshed' after completely parsing bigger datasets (7k+ messages)

Temporary workaround: double-click on LID column to reset sorting, placing the newest message on top.

Migrate gulp build process to webpack:babel combo

Lots of old tech in the gulp build process. Still works, but who knows for how long. Also means we can't use ES6 syntax.

Worth migrating to https://ccoenraets.github.io/es6-tutorial-data/babel-webpack/

Migrate angularjs to newer version

Add a 'loading' indicator, while the XML files are still being parsed (1GB+ dataset)

Refactor dataset and parsing into dedicated factory

In progress

Make columns resizable by user

Fix placement of resize & search buttons

onclick row, set custom colour - or provide buttons to mark rows

Provide exporting function for marked rows, formats TBD

Add changelog view within parser, next to help button

Should be able to read and render the changelog.md

Present log views in tabs, when parsing more than one log type

As an easier way of seeing kernel and web logs at the same time.

Add support for DSSErrors.log

Horizontal scrollbar on bottom of window instead of table

Use modals for click-to-view