splunk/splunk-3d-graph-network-topology-viz

Plot relationships between objects with force directed graph based on ThreeJS/WebGL.

Splunk 3D Graph Network Topology Visualization

Plot relationships between objects with force directed graph based on ThreeJS/WebGL.

More information around use cases on the following Splunk blogs:

• Chasing a Hidden Gem: Graph Analytics with Splunk’s Machine Learning Toolkit
• Understanding and baselining network behaviour [part 1 / part 2]

Installation

Please refer to Splunk Documentation for guidance on installing this visualization in your environment. The app needs to be installed on the SH tier.

Download is available from either GitHub or Splunkbase.

Usage

<search> | stats count by src dest [color_src] [color_dest] [edge_color] [weight_src] [weight_dest] [edge_weigth]

Replace src and dest with your fields to start. All other fields are optional and default values will be used if not given.

Optional values

Used to customise nodes and edges (or links) styling.

FieldName	Format	Description	Example
color_src	string	Color of source node in HEX	#00DD00
color_dest	string	Color of destination node in HEX	#CC00FF
edge_color	string	Color of edge in HEX	#12FF00
weight_src	numeric	Sphere size of source node	2.5
weight_dest	numeric	Sphere size of destination node	3
edge_weight	numeric	Stroke weight of edge line in pixels	2.1

Besides:

• Field names must correspond to the ones specified above to be properly handled by the visualization
• Any edge_weight value higher than 18 will be normalised to 18

Drilldown

To enrich your dashboards with some interactivity, enable drilldown in your dashboard panel. For example, assuming the configuration below, by clicking on a node in your graph it should populate a token with the value from a field named src.

    <option name="drilldown">all</option>
    <drilldown>
      <set token="tok">$row.src$</set>
    </drilldown>

This token can then be used within your dashboard in another panel or visualisation.

Examples

Lookup tables

• Add a lookup table defining these additional values to your Splunk instance. An example below:

  $~ cat <your_lookup_table>.csv
  source,color,weight
  A,#010101,50
  B,#ff0101,10
  ...
  

• Execute your SPL <search> | stats count by src dest | lookup <your_lookup_table> source AS src | lookup <your_lookup_table> source AS dest OUTPUTNEW color AS color_dest, weight AS weight_dest

Simple SPL

• Execute your SPL <search> | stats count as edge_weight by src dest | eval color_src="#cc0000", color_dest="#00ff00", weight_src=3 | eval edge_color=if(edge_weight < 18, "#0000cc","")

Graph Algorithm Examples

Four dashboards are provided to show graph algorithms in action using NetworkX. Select them by clicking on Graph Example Dashboards dropdown in the app navigation bar.

Requirements

Make sure you have the following Splunk apps installed in your instance to correctly visualize integrated dashboards.

• Splunk's Machine Learning Toolkit (MLTK)
• Python for Scientific Computing Libraries 2.0 or later

Give global permissions to MLTK app if not already done

Hardware Acceleration in Browsers

All modern browsers shall have hardware acceleration turned on by default in order to increase your browser performance and free up CPU by leaving the GPU (Graphics Processing Unit) handling graphical intensive operations.

Chrome

The best way to check whether hardware acceleration is turned on is to type chrome://gpu into the address bar at the top of the browser.

A whole bunch of results will be returned but we're interested on the section titled Graphics Feature Status. Please make sure "Hardware accelerated" is turned on entry WebGL (among multiple others).

To enable hardware acceleration:

• type chrome://settings in the address bar,
• scroll to the very bottom of that page and expand the Advanced link
• scroll down till reaching the section System and enable the option "Use hardware acceleration when available"
• restart the browser if requested

Firefox

To enable hardware acceleration:

• type about:config in the address bar
• search for layers.acceleration.force-enabled and double click on it to turn the value to true
• restart the browser

Other browsers

For other browsers, please refer to their specific documentation on the matter.

Contributing

If you would like to contribute to the visualization, please have a look at our Contribution Guide

LICENSE

Splunk 3D Graph Network Topology Visualization is licensed under the Apache License 2.0. Details can be found in the file LICENSE.

splunk-3d-graph-network-topology-viz's Issues

Fix for three vulnerability

As per warning https://github.com/splunk/splunk-3D-graph-network-topology-viz/security/dependabot/appserver/static/visualizations/3d_graph_network_topology_viz/package-lock.json/three/open

Export algorithms to other apps

update default.meta to export algorithms for use in other apps.

Currently using the algorithms: GraphLabelPropagation & GraphConnectedComponents cause errors when they are run outside the app.

I suggest the following fix to default.meta:

[algos]
access = read : [ * ], write : [ admin ]
export = system

Update underlying libraries

After a year progress on the used libraries those moved from version 1.45 to 1.60 which could be good thing to update from https://github.com/vasturiano/3d-force-graph and test if the viz is working with that

Search bar disappeared

App version: 1.3.0
Splunk version: 8.2
Issue description: Search bar not shown when loading dashboard "Graph Analysis Framework"

Expose more graph layout controls

The underlying layout engine offers more tuning parameters that users want to adjust for their specific use case: https://github.com/vasturiano/3d-force-graph#force-engine-configuration , e.g. d3Force would be worth investigating of attraction and repulsion forces can be exposed with a numeric value text input or slider type of control into the custom viz format options.

Performance improvement 1: canvas switch from 2D to 3D

Research about improving the stability and performance of the app to check if the initialisation of the 2 available renderers (2D and 3D) can be modified to have not both of them initialised to be switched, but to better initialise on demand when the switch happens and preventing of running a non-visible but resource consuming panel.

Investigate node colouring edge cases

In certain situations nodes get coloured if source colour and destination color is different for the same node Id which causes wrong display results in the visualisation. Create a simple reproducible set to validate and investigate this behaviour to find out what fixes or workaround can help in this situation.

Display mode for Directed Graph

As a user I want to display a graph as directed graph with arrows drawn at the edges. The underlying library supports this in 3D mode, so it could be worth checking it out to add it easily: https://vasturiano.github.io/3d-force-graph/example/directional-links-arrows/

ReferenceError: Can't find variable: link_color

When I switch from 3D to 2D, the graph disappears and the error ReferenceError: Can't find variable: link_color is shown in the console.

Add app.manifest file

Even if supporting Search Head Cluster (SHC), to avoid undesired warnings when installing the app in SHC Cloud deployments, re-package the app adding app.manifest.

More info

Strange visualization behavior

Hi team,
First of all allow me thank you for the great app you've built. I find it very useful.

I am facing an issue with it, where I am trying to explore the Eigencentrality within a large group of people (100+).
The resulting dataset contains around 1500 rows with src, dest, count and all the other necessary fields for the visualization.
I actually used the same queries that are available in the app OOTB for the centrality measures.

My problem is that if a use a "| sort - src" I get one result in the visualization. When I use a "|sort src" (i.e. ascending order) then I get different result in the visualization. I.e. different nodes are marked as "influential" or "prestigious" depending on the sort order.

Is this something to be concerned about or is it expected? I am not sure, but I think the results should always be the same, when trying to visualize the eigencentrality of the same group, regardless of the sort order.

Any ideas?

BTW, same thing can be tested if you change the sort order of the graph query of the OOTB dashboard called "graph_analysis_centrality". You will get two different pictures depending on the sort order.

Performance improvement 2: throttling FPS rate

Research about improving the stability and performance of the app to check if the initialisation of the 2 available renderers (2D and 3D) can be modified to have a throttling of the render loop (= how often the visualisation and the lay outing is happening). Typically there is a repeated interval loop that is called often and causes high resource consumption or even instability especially if more than one visualisations are run, e.g. on a dashboard. Either the rendererConfig OR the renderer() should give access to the behaviour of the threejs render system. Implement a "frame rate throttling option" in the viz format options, e.g. 60 frames per second, 30, 20 etc. so people could adjust for their given situation.

Drill down onNodeClick

In order to drill down from the visulization to interact in a dashboard an added drill down behaviour would be great to set a dashboard token with the primary node information (typically the node id that was taken from the search results). Click on a node maintains the default viz behaviour but adds also the setting of a dashboard token.

Labels & token passthrough for nodes

when visualizing data it would be super helpful to have a toggle labels option for the nodes.

when exploring data through graphs/visualization to be able to quickly see a nodes name/value and then click on it to expand it in a new view with a token pass through would make this extremely powerful.

is that possible? It could almost be like some sort of minority report style interface for data set review which would be awesome!
thanks!

Update docs how to check for WebGL support in client browser

Update docs how to check for WebGL support in client browser, e.g. pointer to chrome://gpu/ etc. to have this available in README or splunkbase app docs.