ebispot/lodestar

Linked Data explorer and SPARQL endpoint

Lodestar

Lodestar is a Linked Data Browser and SPARQL endpoint. Lodestar is a Java based web app that can wrap any existing SPARQL endpoint to provide a set of additional SPARQL and Linked Data services. Lodestar was developed to provide a consistent set of SPARQL and Linked Data services across the European Bioinformatics Institute (EBI). Some of the service provided by Lodestar:

• Javascript based SPARQL endpoint with configurable example queries and paginated results table
• Read only SPARQL endpoint for protection from write operations
• A single SPARQL endpoint that provides a UI, the service and a SPARQL 1.1 service description
• SPARQL syntax highlighting provided by CodeMirror
• Works with any SPARQL endpoint (Includes Virtuoso JDBC connection option)
• Linked data browser for navigating data from a SPARQL endpoint
• Configurable resource description/linked data pages:
  • Renders resources by label where possible
  • Grouping of related resource by type
  • Set top facts to display, such as labels and descriptions
  • Configurable limits for how many related resources to render in the browser
• Renders depictions of resources
• Handles content negotiation for both SPARQL queries and linked data pages
• CORS enabled for cross domain scripting
• Basic REST API for accessing data in simplified JSON format

To see a demonstration of the Lodestar linked data browser please see the Expression Atlas RDF website. Lodestar has been primarily developed as an internal tool for EBI services deploying RDF, however, the application should be sufficiently generic that others can use it. I can't guarantee any support for the software at this time, but please feel free to use it or adapt for your own purposes and let me know how you get on.

Documentation and stable release at http://ebispot.github.io/lodestar/.

Release Notes

1.3 15th October 2014

• Fixed potential race condition in explorer when using a small number of connection pools
• Javascript fix to support next/prev links preserving inference option
• IE 11 rendering issue fixed
• Added no JNDI implementation for virtuoso connection pooling

1.2 21st August 2014

• Updated to Jena 2.12
• Exposed JSON-LD support from Jena in UI
• Moved virtuoso to separate module, only builds in "virtuoso" profile
• Removed dependencies on virtuoso inferencing rules
• SPARQL endpoint now support application/sparql-query POST requests
• Fixed some browser rendering bugs

1.1 27th November 2013

• Updated to Jena 2.11
• Fixed query limit bug (RDF-10)
• Added config for query timeouts (RDF-15)
• Configurable hide RDFS button (RDF-7)
• Added servlet status monitor
• javascript cleanup

1.0.2 29th August 2013

• Updated to Jena 2.10
• VirtJena JDBC 4 (includes support for SPARQL bind queries). Requires virtuoso 6.1.7.2
• Added CSV and TSV sparql results export
• Fixed sparql results offset caching from previous query
• Fixed virtuoso describe query not returning all triples from all graphs

1.0.1 5th August 2013

• First release

lodestar's Issues

Upgrade to Spring boot

Update the Spring configuration to use Spring boot, or decide not to do so.

Issues concerning URI escape / unescape

In the lode.js, to encode contents for URI, encodeURIComponent is used while unescape is used to decode.
This makes some problems such as handling UTF-8 characters inappropriately.

So, I fixed.
yayamamo@14e55e3

Changes for security of our clients from XSS attacks

Our security team is trying out a new scanner, called netsparker. It found something that IBM AppScan never found, which is that parameters sent to the SPARQL endpoint that contain scripts will be echoed as errors.

In response to this, I made small changes to SparqlServlet so that all exceptions are sent as text/plain content-type. This prevents someone from getting a browser to execute the script. The point is not just to protect our users - the most likely place for someone to put such a malicious URL is in github.com issues for our projects. We click on it, try to reproduce the issue, and bang, they are in.

The changes have the added benefit of improving the SPARQL compliance, because we are now returning simple text on any error, and I made sure that bad type conversions are coming across as status code 400.

Reporting namespaces in URI page

It would be nice to add namespaces to show URIs in single-URI pages (eg, https://www.ebi.ac.uk/rdf/services/atlas/describe?uri=http%3A%2F%2Frdf.ebi.ac.uk%2Fterms%2Fatlas%2FisAbout), the same way it is done in the table reporting SPARQL solutions (or in Pubby: http://dbpedia.org/page/Bologna).

Something like atlas:isAbout could be added to the absolute URI, or the short form could be made expandable by means of a '+' button. It shouldn't be difficult to do that by exploiting namespaces.js.

This would be useful, when Lodestar is used to build queries from examples, or similar tasks.

Lodestar won't build: Unknown host coconut.ebi.ac.uk

There is a build dependency on the repository coconut.ebi.ac.uk: this is an EBI internal server and not visible outside, causing builds to fail. Can we remove this dependency?

left-align table header to reduce necessity to scroll right over

Hello!

I noticed the following little UI bug on https://medium.com/virtuoso-blog/benefits-of-a-semantic-web-of-linked-data-for-bioinformatics-62c2964357c >under "both the EBML-EBI and UniProt databases, and has been semantically combined" > screenshot, or directly in these search results. The seq column header is far off-screen because the cell content (amino acid sequences, right?) can be so large.

Can th in loadstar-results-table be left-aligned only when a column becomes wider than the screen? Maybe in https://github.com/EBISPOT/lodestar/tree/master/web-ui/src/main/webapp/css/lode-style.css#L18-L20?

Provide history of queries

Hi, it would be cool if lodestar saved my queries (maybe in cookies?) ,
so I can redo them when coming back. Similar to the "recent activity"
box in PubMed: http://www.ncbi.nlm.nih.gov/pubmed/?term=EBI+RDF
Yours, Steffen

Bamboo CI/CD integration discussion

Our tooling section is working with me to make our NLM copy ready for Bamboo CI/CD, and while this works with a single implementation, as described in:

http://marcobrandizi.info/mysite/node/146

What are your thoughts on how this could work? Moving dependencies on repositories into properties and settings.xml is relatively easy, but handling the update and management of version numbers is rather hard.

Some more work may need to be done to make it work across forks of the code. Maven really wants to update the repository...

"more..." display options don't work.

When describing a URI, to display all of the "Related to" or "Related from" resources there are "more..." options shown.
However, clicking any of the options doesn't work.
I found bugs in lode.js and fixed.
yayamamo@8ada275

REGEX to know the type of SPARQL is wrong

In lode.js the incoming SPARQL is parsed to see if it's a CONSTRUCT or SELECT and decide whether the output format is compatible with the type: http://tinyurl.com/osmoz6c

The current regular expression used to do that doesn't work when one of the PREFIX contains an hyphen, eg, this: http://tinyurl.com/m6ydzmk doesn't work (defines foo-ns), while this: http://tinyurl.com/pgjs3qh does.

link to stable release gives error

http://www.ebi.ac.uk/fgpt/sw/lodestar

Update to virt_jena3.jar and Apache Jena 3.10.0

Currently, lode-virtuoso-impl depends on an artifact virt_jena. It seems likely to come from
https://github.com/openlink/virtuoso-opensource/tree/develop/7/binsrc/jena, but I'm not 100% sure as it must be injected from elsewhere.

Correspondence on the virtuoso-users mailing list reveals that we should be using the JAR virt_jena2.jar from https://github.com/openlink/virtuoso-opensource/tree/develop/7/binsrc/jena2 for Apache Jena 2.13

This issue requests that we:

• Make it clear through comments in the pom.xml where this comes from, and change the versionId to correspond to releases of the virtuoso-opensource repository.
• Upgrade to virt_jena2.jar and call the artifact virt_jena2
• Upgrade to Apache Jena 2.13.0

Possibly, it would be good to have a shell script or ant build to download these artifacts locally, and add them to a local repository.

Security updates for Lodestar

As part of my release process, I ran the OWASP Dependency Check, https://www.owasp.org/index.php/OWASP_Dependency_Check, maven plugin against lodestar.

I found the following issues in lode-core-api:

[ERROR] Failed to execute goal org.owasp:dependency-check-maven:1.4.3:check (default-cli) on project lode-core-api:
[ERROR]
[ERROR] Dependency-Check Failure:
[ERROR] One or more dependencies were identified with vulnerabilities that have a CVSS score greater then '4.0':
[ERROR] jackson-annotations-2.3.0.jar: CVE-2016-3720
[ERROR] jackson-core-2.3.3.jar: CVE-2016-3720
[ERROR] httpclient-4.2.6.jar: CVE-2015-5262, CVE-2014-3577
[ERROR] jackson-core-asl-1.5.3.jar: CVE-2016-3720
[ERROR] spring-aop-3.2.2.RELEASE.jar: CVE-2014-1904, CVE-2014-0054, CVE-2013-7315, CVE-2013-6429, CVE-2013-4152
[ERROR] spring-core-3.2.2.RELEASE.jar: CVE-2014-3625, CVE-2014-3578, CVE-2014-1904, CVE-2014-0054, CVE-2013-7315, CVE-2013-6429, CVE-2013-4152

Documentation is missing and links are stale

Hi Simon et al. The lodestar docs at http://www.ebi.ac.uk/fgpt/sw/lodestar seem to have gone missing. The lodestar link on https://www.ebi.ac.uk/rdf/documentation/infrastructure to https://www.ebi.ac.uk/rdf/fgpt/sw/lodestar/ has also become stale. Are there any lodestar deployment/setup docs available anywhere?

"Error creating bean with name virtuosoDataSourceProvider" on startup

Hi Simon,

the manual states that "a clean install will deploy a Lodestar instance that has been configured to wrap the DBpedia SPARQL endpoint." However, when I drop the WAR file into the webapps folder, the following error is thrown (seems like port is omitted if blank, but even if I provide 80 in the properties file, it doesn't work):

2014-08-14 15:28:04,283 ERROR [localhost-startStop-1] context.ContextLoader (ContextLoader.java:319) - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'virtuosoDataSourceProvider' defined in ServletContext resource [/WEB-INF/ebi-lode-service.xml]: Could not resolve matching constructor (hint: specify index/type/name arguments for simple parameters to avoid type ambiguities)
  at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:250)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1051)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:955)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:490)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:626)
  at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
  at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:389)
  at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:294)
  at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
  at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4760)
  at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5184)
  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
  at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:724)
  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700)
  at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714)
  at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:919)
  at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1704)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:744)

I expected the tool to just work without configuration. I'd rather not create a JNDI resource and just communicate with the DBpedia SPARQL endpoint using HTTP. After seeing "Works with any SPARQL endpoint" in the list of features I got the impression that this should be possible.

Update to Spring 4 and Java 1.8

Spring version 3 framework will hit end of life Dec 31, 2016.

I've tried building lodestar as follows, and found it still works. not just the build, but accessing it over a network once installed over Tomcat, and running queries.

mvn -Pvirtuoso -Dorg.springframework.version=4.3.3.RELEASE clean package

Any reason not to upgrade?

Better logging with MDC and maybe jsonevent-layout

@simonjupp , I have been working on enhancing logging a bit.

One piece of the puzzle has been getting to a structured lodestar.log - and the simplest way to do that is by adding a dependency for https://github.com/logstash/log4j-jsonevent-layout, e.g.

        <dependency>
            <groupId>net.logstash.log4j</groupId>
            <artifactId>jsonevent-layout</artifactId>
            <version>1.7</version>
        </dependency>

However, another part of it is to make the query and other parameters appear as structured extra variables, something done with Mapped Diagnostic Context, something like

MDC.put("query", query);
...
MDC.remove("query");

I noted some very interesting stuff in the 1.4 branch, which is what I'm running with currently:

lodestar/lode-core-servlet/src/main/java/uk/ac/ebi/fgpt/lode/servlet/SparqlServlet.java

Line 253 in e6f4fe5

log.info("What user information can we retrieve here");

I'm tempted also to upgrade to log4j 2, which would mean using a different Maven package for structured logging. The benefit of upgrading to log4j 2 is that there would be current documentation for that configuration. log4j 1.2 is getting so old that a search on apache log4j is going to find version 2.

Simple tweak to CorsFilter for Chrome

Apparently, Chrome has a stricter implementation than normal for CORS. The spec. says that a pre-flight request should get a response code of 204 (accepted), and a simple request should actually do the work.

I recommend therefore that the uk.ac.ebi.fgpt.lode.servlet.CorsFilter be updated to respond with 204 to an OPTIONS request, which worked for us. Here's my implementation of doFilterInternal:

   @Override
   protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws ServletException, IOException {
        response.addHeader("Access-Control-Allow-Origin", "*");
        response.addHeader("Access-Control-Allow-Methods", "GET, POST, PUT");
        response.addHeader("Access-Control-Allow-Headers", "Cache-Control, Pragma, Origin, Authorization, Content-Type, X-Requested-With");
        response.addHeader("Access-Control-Max-Age", "1800"); // 30 minutes

        if (request.getMethod().equals("OPTIONS")) {
            // CORS "pre-flight" request
            response.setStatus(HttpServletResponse.SC_ACCEPTED);
            return;
        }

        // Simple request
        filterChain.doFilter(request, response);
   }

OWASP plugin needs to be updated, 1.4.3 triggers errors

Just to let you know, I had to migrate the org.owasp plugin in the main POM, cause it was generating HTTP/404 errors, showing the old version is not aligned the server counter-part anymore. Moving to 5.2.4 made it work again.

Cannot work at IE11

When accessing any "/lodestar/describe" resources by IE11, nothing is shown.

lodestar returns HTTP status 200 for resources that do not exist

For example:

http://www.ebi.ac.uk/rdf/services/biomodels/describe?uri=http%3A%2F%2Fidentifiers.org%2Fbiomodels.db%2FBIOMD0000000001%23_000003foobar

Simon, we have code that almost fixes this problem. The issue is how far to go to fix it.

Link to documentation and stable releases

Hi @simonjupp
The link in the README file is no longer valid: http://www.ebi.ac.uk/fgpt/sw/lodestar
Likewise the link in the RDF platform documentation (https://www.ebi.ac.uk/rdf/documentation/infrastructure): https://www.ebi.ac.uk/rdf/fgpt/sw/lodestar/
Has the page moved somewhere else?
Thanks

Wrong media type used for JSON(-LD)

Filing upstream to reflect the fix for HHS/meshrdf#91, because the media type is encoded in the GraphQueryFormats enum (maybe you shouldn't have accepted my pull request ;)

I'm wondering whether to just fix this in GraphQueryFormats or whether to make the MIME type to use loaded from Spring properties, and am seeking suggestions.

Merge Lodestar version

Our newly developed lodestar isn't merged with the current master branch of lodestar, this should be done and potential conflicts fixed

remove/document unreferenced bean

ebi-lode-service.xml explicitly wires the ExplorerServlet (a Spring controller, not a servlet), as follows:

    <bean id="serviceServlet" class="uk.ac.ebi.fgpt.lode.servlet.ExplorerServlet">
        <property name="sparqlService" ref="jenaSparqlService"/>
        <property name="configuration" ref="explorerConfig"/>
        <property name="service" ref="explorerServiceImpl"/>
    </bean>

However, the bean actually used by Spring's DispatcherServlet is autowired through lode-servlet.xml

It took me quite awhile to discover that editing the bean in ebi-lode-service.xml wouldn't do me any good.

log4j vulnerability and Apache Jena 2

I have updated again my branch "develop" from 1.4-release and did some changes to remediate the log4j attack by moving to logback. The problem I have here however is that Apache Jena 2.12 uses log4j, and we would need to update to Apache Jena 3+ to get away from that.

I'm not sure there is a direct way to exploit the vulnerability but thought we might discuss it.

In case there is no longer a Simon Jupp there, NLM also runs a version of lodestar at https://id.nlm.nih.gov/

jenaVirtuosoExecutorService.endpointURL in default ebi-lode-service.xml is wrong

Hi all,
I came to using LODEStar again after some years, and I've had problems with this Spring block in ebi-lode-service.xml:

<!--For direct virtuoso JDBC connection, use this service-->
<bean id="jenaVirtuosoExecutorService" class="uk.ac.ebi.fgpt.lode.impl.JenaVirtuosoExecutorService">
   <property name="endpointURL" value="jdbc:virtuoso://${lode.sparqlendpoint.url}:${lode.sparqlendpoint.port}" />
</bean>

To me the value above is wrong, or at least it is incompatible with the default assignment lode.sparqlendpoint.url=jdbc:virtuoso://localhost:1111, provided by the default lode.properties.

Obviously, the problem is that the two combined yield jdbc:virtuoso://jdbc:virtuoso..., and the worst part is that later, the error message you get is: virtuoso.jdbc4.VirtuosoException: Wrong port number : For input string: "", ie, the Virtuoso driver expects a given format and got the wrong port number from the wrong URL format, but I took a while before realising it from the confusing message.

To me, the solution should be to set value="${lode.sparqlendpoint.url}" in the Spring XML (as I did in my fork).

Moreover, I'm not sure that definitions without = are valid in lode.properties (there are a few that are commented).

Maven config is EBI internal only

External users can not deploy or build this...