molgenis/graph2vr

Visualising / Exploring Linked Data (SPARQL Construct queries) in Virtual Reality.

Graph2VR

Graph2VR is a PhD project, a prototype for a VR application to Visualize Graphs (SPARQL) as 3D Graphs in Virtual Reality. The idea is to explore, analyze, and interact with the data in the graph using gesture control.

For a hands-on introduction to Graph2VR, we have created a video tutorial series that covers everything from basic navigation to advanced features. The tutorial is designed to help both beginners and experienced users get the most out of Graph2VR.

Graph2VR Tutorial Series on YouTube

Our tutorial is about using Graph2VR - it does not explain much about the basics of Linked Data and SPARQL. If you want to learn more about the background of Linked Data, we can recommend the Ted talk of Sir Tim Berners Lee "The next Web" (from 2009) for some basics.

Graph2VR has been built in Unity and is able to connect to a SPARQL endpoint using dotNetRDF. We got inspired by many different tools to work with Linked data and to visualize Graphs. Virtual Reality offers the user way more space to expand the graph, then a 2D computer screen.

Here is an example of a query that shows the results in form of stacked 2D graphs behind each other:

We tried to write a GUI to explore and interact with the graph in Virtual Reality. A few simple operations to do so are:

• Getting some information about the current node
• Expanding the graph at a certain node using incoming or outgoing predicates
• Deleting a node or collapsing those nodes around it that are not connected to the graph elsewhere
• Comparing different parts of the graph side by side (different visualisations)
• Building visual query patterns to form a SPARQL query
• Adding new nodes and edges
• Interacting with the graph (zoom, move, rotate)
• Using visual queries or a search function to add specific nodes
• Saving the results to ntriple format to be able to reuse it in other programs

Different Layout Algorithms can help to visualize the data.

For more detailed instructions on how to use Graph2VR, check out the Graph2VR User Manual.

We recommend to load it on the Quest2 VR headset via sidequest

Our research paper about Graph2VR can be found at https://doi.org/10.1093/database/baae008 (soon).

graph2vr's Issues

Improvment: Load labels when asking for similar query patterns

When querying similar query patterns and the predicate label is not part of the query, we won't get a label.
This can be astonishing for the user since he selected a node with a label for the query in the first place.

Feature: Node Coloring

We have different kinds of Nodes, e.g. Variable Nodes, URIs, Literals (Text / Numbers).
Since we often use rdfs labels to display a shorter name, it becomes more and more difficult to differentiate between URIs and Literals. This could be fixed by using different colors or shapes for different kinds of Nodes.

Feature: MINUS

Either MINUS or Filter NOT EXISTS (or both), should be a feature in Graph2VR.

Blank nodes - no node menu

We didn't care much for the handling of blank nodes.
However, the fact that not even the default node menu opens when clicking a blank node was quite surprising.
We should at least be able to get a (limited) node menu to, e.g. remove the node again.

It was possible to generate a new edge and remove the blank node by removing nodes around that other node.
But this is not how it should be handled.

Augmented Reality version

We think the future of Graph2VR is augmented reality.
Currently, the graphs only exist in a virtual reality environment, but in the future, especially with the capabilities of the Meta Quest 3 or newer glasses, AR might be a better option.

There are already tools that can handle 3D Graphs in a Mixed Reality environment.
Here some reference to "Big Data Visualization Platform for Mixed Reality Big Data Visualization Platform for Mixed Reality":
https://doi.org/10.25772/6MDD-2B85"

Feature: Count

SPARQL offers the count command to count the number of variables of a certain type in a result set.
We should also offer the count function in Graph2VR.
This is especially interesting for the semantic planes to see how the numbers vary while changing a variable.
We could think of a literal showing this number, or even having some bar chart, to visualize the number.

Performance tweaking using the Unity profiler

This is rather a general todo:

To allow more nodes to be displayed at once, keeping the framerate up and preventing stuttering while the graph is reordered, we planned to have a look at the Unity profiler and see whether we can tweak the overall performance.

Keyboard: Edit string somewhere in the middle

It is possible to delete either the whole string or the last letter.
Testuser comment:
I would expect to be able to set the cursor at a position with the laser and edit the string there.
E.g. add or remove a letter "s" in "https://somethinglong:8890/sparql".

Federated queries

Using the SERVICE keyword allows to combine federated results from different SPARQL Endpoints.
It might be helpful to store where the different nodes and edges came from originally, so we could check wether to do a service query with the respective server or just ask the current endpoint.

Improve Search functionality

The current version of the search is not working too well, not only do we have a bug (see #24), also the results are also somewhat underwhelming.

It is difficult to really fix that and add a better search functionality.
Currently, we do only have 2 ways to search - one with and one without bif:contains.
Other tools give the user the choice of how to search.

E.g. Protege offers 5 options:

• contains
• startswith
• endwith
• exact match
• regex

Those match exactly the ways to search in SPARQL.
Probably we should also offer exactly those search functionalities.

Loading Quicksave changes behaviour of Class Hierarchy View

After saving a quicksave in the Class Hierarchy view and reloading it, the graph was loaded in the 3D Force-directed layout.
It's not a problem to change the layout via the menu.
But in this case, all the classes, subclasses and individuals were displayed in one line.
I was expecting the class hierarchy layout. Not sure what went wrong here.

Filter

One of the most important SPARQL keywords is Filter.
When working with data in Graph2VR it would often be helpful to be able to set up filter rules for a given context to specify and limit the query results.

This feature is still missing in Graph2VR and might need to be splitted up into several subtasks.

• Filtering for certain criteria that need to be excluded
  This is represented with the "Minus" or "Not Exists" keyword in SPARQL.

• Filtering for a certain number or range of numbers in the case of integers.
  We need some (2 sided) sliders to restrict numbers e.g. "FILTER (?age < 65)". This might be context depending.

• Filter for certain strings in literals (Filter ?name = "Bob")

• Some filter can be defined based on query patterns
  E.g. ?disease rdfs:subClassOf http://www.ebi.ac.uk/efo/EFO_0000311.

No support for databases that require username and password so far

Issue:
Some repositories require a username and password for security reasons.
Currently, Graph2VR doesn't support specifying a username and password.

Current Situation:
It is not prepared to specify any password in Graph2VR.

Potential solution:
Allow to specify a username and password.
This could be hardcoded in the settings.txt file (security issues) or requested within the application.

Note: DotNetRDF already has some solutions implemented, see:
https://dotnetrdf.org/docs/2.7.x/user_guide/Configuration-Storage-Providers.html

Slipping down from the circular scrollbar

Users may slip down and close the menu when adjusting the circular scrollbar with the laser.
Usually, clicking in the empty space should close the menu, but in this case, closing the menu isn't very pleasant.

Search for existing nodes

The search function does not work properly.
We had multiple entries for one entity that should only exist once.
URI and info in the infobox did not match.

Text in .apk file flickers (unity version works)

The text in the application is somewhat blurry / snowy since our update of the TextMeshPro setting.
This is only the case when building the apk, not in Unity.

Feature(s): Filter

One of the most important SPARQL keywords is Filter.
Probably Filtering has to be split up into several subtasks.

We want to have some filtering within the tool to slice and select the data.
Here are some examples of filter functions:

• Filtering for certain criteria that need to be excluded
  This is represented with the "Minus" or "Not Exists" keyword in SPARQL.

• Filtering for a certain number or range of numbers in the case of integers.
  We need some (2 sided) sliders to restrict numbers e.g. "FILTER (?age < 65)". (Context-dependent)

• Filter for certain strings in literals (Filter ?name = "Bob")

• Some filter can be defined based on query patterns
  E.g. ?disease rdfs:subClassOf http://www.ebi.ac.uk/efo/EFO_0000311.

Graph beginns to shake

After a while, a graph may start to shake while holding it, even though the user doesn't move it.
The reason for this is the point that is used to calculate the angle between user and graph, which internally will change its position while moving the graph and due to the layout algorithms.

Calculating the middle of the graph and, based on correcting for that point leads to resonance between the two.

"Multiplayer" - or at least a spectator mode

It would be great to have a look at a graph together.
Until now, only one person is able to enter Graph2VR and interact with the data.
It is possible to cast the video feed of the quest, e.g. via Chromecast, but it would be even better if multiple people could join in Graph2VR at the same time.

One simple version would be a spectator mode, but in the long run, multiple people working with the same dataset would be great.

It might be some effort to implement that in the current version of Graph2VR because the internal messages are broadcasted and might affect the operations of other players as well. Graph2VR has not been built for multiplayer in the first place.

Optional visual indicator (dashed line for the edge) is sometimes missing

When marking an edge as optional, it should be indicated as optional with a dashed edge.
In most cases it does work, but we had the case that an edge was marked as optional, but the visual indicator war not there, it was still a straight line. For the query it still counted as optional, so it's just a visual issue.

Loading lables / layout issuse after loading safestate

After loading the savestate some layout issues happened and labels were not shown, instead the URI was shown.
We need to investigate the reason for this issue.

Slowing down, getting stuck or crashing when loading too many images at once.

While loading many images at once (while starting the application or loading a save file), the application freezes.
After loading a save file with more images, the whole application crashes, especially when moving the head around.
Loading the images via the graph expansion is usually not a problem.
We should unblock the application and work with a queue/threads when loading a save file and the initial query.

Feature: Optional

SPARQL supports optional variables.
This means if the variable is present, show it, but it doesn't matter if it is not present.
Including an option to tag to nodes / edges and apply the optional parameter when we include them in a query would be a great feature.

html en-/decoding for URIs

Current behaviour:

When a URI is html-encoded, it gets automatically decoded.

When requesting the URI
n1:countries?name=Netherlands+%28the%29
It gets automatically decoded to
n1:countries?name=Netherlands+(the)

Consequence:
There are no incoming and outgoing predicates for this URI, even though they are present. (Might affect more functionalities)

Expected Behaviour:
URIs are URIs and should not be decoded before reusing them in a query.

Fix:

1. Find out whether it is the C# encoder of the SPARQL query that does that or whether DotNetRDF is responsible for that.
2. Avoid decoding URIs before (re-)using them in a query.

Term "Autolayout" misleading

We should rename "Autolayout" to "Refresh layout" because someone could assume that a Layout would automatically chosen, which is not the case. "Refresh layout" would be more accurate. Should only be a very small fix.

Indicator for current SPARQL Endpoint

When selecting an Endpoint and potentially a graph, it would be helpful to have an indicator of which database we currently are.
Additionally, we could add an indicator of whether it is actually reachable by sending a simple query, but this is a second step.

Offset

The SPARQL keyword Offsett is really helpful to skip a certain amount of results.
Especially in combination with large amounts (501) of ordered results, this might be handy to get all results.

Node placement - literal nodes - Semantic Planes

When asking a query with multiple-layer answers, literal nodes may not receive the position information and be placed in random positions.

Series of quick SPARQL queries when creating semantic planes

When we create a SPARQL query pattern within the graph and request structures that match this pattern, we do multiple requests.
The first select query gives us all the information (content) needed to create the planes.
But we also do construct queries to create the structure of the different graphs - one per plane.
This means we usually do 26 and up to 501 SPARQL queries in a quick series when using this feature.
This is not nice for the SPARQL endpoint and they might consider slowing us down.
Instead, we should try to use the results and calculate the query patterns locally.

Indicating presence of nodes after switching the SPARQL Endpoint

When switching the endpoint, the user currently has no indicator of whether a node (or edge) exists in the current database.
If we could make all nodes semi-transparent that do not exist in the current database, it would help to find the overlapping of the current graph with a different endpoint.

Important here:

• Let the user choose whether to do it or not - in case of sensitive information, sending data to a foreign SPARQL Endpoint would not be good.
• Don't send too many queries to external SPARQL Endpoints if not necessary

Feature: Menu extension to open specific object

Current situation:
When selecting a node, the incoming/outgoing predicates and the count of triples using those predicates are shown in the menu.
Sometimes we don't want to have random 25 nodes of that type, but one specific node.
Therefore we could extend the menu when clicking at the count of one predicate and show a new circle menu layer with the objects.
A slider like in the first layer of the menu would then enable the user to scroll through those objects and select the one(s) to display.

This could be implemented as a new (2D) layer around the menu, or more fancy, by rotating it.
We would also need another knob to scroll like in the current menu.

Query Log

A log of the queries during a session could help to reproduce the results - and they could be helpful to document.

The log should

• contain the queries the user has defined and executed in the current session
• be exportable from the quest as a text file
• be part of the serialized savestates
• have a timestamp

Hierarchical view outgoing rdf:type

The hierarchical view algorithm tries to order the nodes by

1. The subclassof hierarchy building a horizontal tree structure
2. The individuals based on the rdf:type predicate building a vertical stack next to (one of) their class(es)
3. Any other predicates (e.g. label)

If we open an outgoing rdf:type connection (this node is type of a certain class), it is not clear where this class should be positioned and where the current node should be placed, therefore, and next to which class it should be placed.
Therefore the current behaviour is consistent, but somewhat undefined.