massbank/massbank-web

The web server application and directly connected components for a MassBank web server

massbank-web's Issues

Record format update

New CH$LINK tags need to be implemented.

Cleanup redundant files in source code repo

The current git repository contains redundant files
which originated from the old SVN and/or the Installer,

e.g.
./Tomcat/webapps/ROOT/MassBank/mbadmin/RecordRegist.jsp (Version 1.0.19)
and
./modules/tomcat/webapps/MassBank/mbadmin/RecordRegist.jsp (Version 1.0.20)

Typo "fasle".

There is a typo in the display of the RecordList:

find . -type f | xargs grep fasle
./Tomcat/webapps/ROOT/MassBank/mbadmin/RecordList.jsp:      {"CH_LINK", "fasle"},
./modules/tomcat/webapps/MassBank/mbadmin/RecordList.jsp:       {"CH_LINK", "fasle"},

in

MassBank-web/Tomcat/webapps/ROOT/MassBank/mbadmin/RecordList.jsp

Line 86 in 81d0039

{"TREE", "false"}

and

MassBank-web/modules/tomcat/webapps/MassBank/mbadmin/RecordList.jsp

Line 84 in 81d0039

{"CH_LINK", "fasle"},

When fixing, check that this has no unintended side effects.

Commons HttpClient is eol

Massbank uses Commons HttpClient internally for communication with Apache httpd. Commons HttpClient is eol since 2011 and should be replaced by Apache HttpComponents Client.

Enhancement of search

Implement full text search and replace quick search template, needs #9

Sitemap.sh needs to be updated with base DB name and automatically appended to crontab

Improvement and consilidation of html page headers

Taging is not consistent over the different scripts and we may consider better enrichment of

Web hook for record exchange?

Sent: Friday, 7 October 2016 8:39 PM
Subject: Re: Re: [Bitbucket] Issue #164: Direct link InChIKey search (fiehnlab/mona)

Speaking of reimporting massbank data, did your guys ever manage to understand our web hook api, to automate this? We really would love to avoid monitoring the subversion directory for changes.

Also what kind of web services are you missing?

An overview of our current services can be found here and the related documentations (It's not perfect yet, we are working on it.)

http://mona.fiehnlab.ucdavis.edu/swagger-ui.htmlhttp://www.google.com/url?q=http%3A%2F%2Fmona.fiehnlab.ucdavis.edu%2Fswagger-ui.html&sa=D&sntz=1&usg=AFQjCNHVh_D1vElQCoUen3N5QtWIAoraYQ

thanks,

g.

Fix directory paths

root@vagrant-ubuntu-trusty-64:/var/lib/tomcat7/webapps/MassBank/mbadmin# cat admin.conf 
...
##### For MassBankScheduler ##############################################################
schedule=perl /var/www/html/MassBank/script/FileCleaner.pl /usr/local/tomcat/logs/ [log/txt/tmp],0,86400
schedule=perl /var/www/html/MassBank/script/FileCleaner.pl /usr/local/tomcat/temp/ [txt/tmp/lck/mol/html] 30,0,86400
schedule=perl /var/www/html/MassBank/script/FileCleaner.pl /usr/local/tomcat/webapps/ROOT/MassBank/temp/ [gif] 1,0,86400

Dispatcher.jsp

A great disadvantage in the crawler indexes is that always the link with the site ID is indexed, for example: http://massbank.ufz.de/MassBank/jsp/Dispatcher.jsp?type=disp&id=EA016912&site=31

The order of the databases is changed if a database is removed from the server. Hence, those deep links are broken. There are many examples in Google, HBDB etc.

We might find a solution to use the short deep link instead:
http://massbank.ufz.de/MassBank/jsp/FwdRecord.jsp?id=EA016912

I dunno how this works out in the whole dispatching process.

Increase mobile friendliness

Optimise the wrapper script regarding the following issues:
Viewpoint meta tag not set
Clickable elements to close together
Text to small to read
Content wider than screen

Solutions:
https://developers.google.com/web/fundamentals/design-and-ui/responsive/

We may introduce a special CSS style for the mobile devices. Actually, 5% are mobile users, may increase...

Replacement of the internal client-server SVN to a full replication SVN

massbank.eu act as client of massbank.jp. In cases of denial of services of massbank.jp, massbank.eu gets in trouble due to the heavy SVN requests, especially in case of massbank.jp redirection to massbank.eu.

I suggest to implement an independent SVN service at each server. All servers hold then the full databases of assigned DBs (as manifested in massbank.conf) and not only copies.

We could also implement SVN based commitment of new records and mol files using a key based upload mechanism to avoid exchange of passwords.

Update of databases using the SVN system?

Test if the update of existing DBs is possible using the SVN system. Needs test system with running SVN and internal SVN mirroring system.

Create a MassBank-web test suite

start a script with a set of tests to confirm the successful installation
of the massbank server, e.g.

wget -O- http://192.168.35.18/MassBank/jsp/FwdRecord.jsp?id=XXX00001 | grep GABA
Will return exit code 0 if successful.

Tidy up sitemap and statistics script

Link to record(s) via InChIkey

Hi, it would be great to link to records via an InChIkey,
e.g. https://massbank.eu/inchikey/RYYVLZVUVIJVGH-UHFFFAOYSA-N

Ensure MassBank.JP statistics are correct

The current massbank.jp statistics page does not seem to include the correct information where records are contributed via massbank.eu, for example the Eawag and UFZ entries in the statistics page (http://www.massbank.jp/en/statistics.html) do not match the numbers on the Record Index (http://www.massbank.jp/RecordIndex.html). We'll calculate the numbers from a mb.eu dump tomorrow and email the maintainers the corrected numbers to update the page manually, but it would be good to get this right automatically

Replace version 1 records with version 2 records

Regularly create getInfo.R script output

Run the getInfo.R from
https://github.com/MassBank/MassBankAdministrationScripts/tree/master/R%20projects/MBrecordToFile
as regular cron job. Output the result under some static *.CSV URL, e.g.
http://massbank.eu/MassBank/DB/statistics.csv

Also, from the R DataFrame in the export a HTML representation
similar to http://www.massbank.jp/en/statistics.html
which can then be included in dynamically
in http://www.massbank.jp/en/statistics.html

This is also related to
#1

Quality assessment, metadata enrichment and curation of old and new records

Curation tool for the assessment of quality of MB records should be available. Could be based on RMassBank (annotation of peaks based on the peak list?). This should include a quality tag in the records to give users more confidence. This is related to #12, which could be also merged into this tool.

Create Mofiles and structures from records

We need a parsing tool which processes the structure files from the records automatically. This issue is related to discussions in #10 and #12.

Schymane:

We cannot produce structures from InChIKeys, but rather from the SMILES or InChI fields, both of which are compulsory.
If these are “NA” or similar, then the structural information doesn’t exist and thus we can’t plot a structure anyway.
The idea of working off SMILES would be that we may also be able to render from generic SMILES …

Search via Massbank ID broken

no spectrum/structure are displayed on the compound's page.

Renewal of API model (eg REST)

hi Tobias,

this is how the REST service kinda has to look like, we can provide a JAVA based framework for this, which than only needs an implementation how to lookup the actual value in your system.

/rest/splash?key-splash1232323123

which returns true or false depending if a splash has been found in your database.

thanks and till soon,

g.

Close old SVN architecture

Currently, the SVN is used to synchronize between JP and EU.
Data from JP is inserted into SVN, and EU does an svn update
and locally inserts them into massbank.eu

The refactoring should create a standalone SVN server, and all
MassBanks commit to this SVN.

• Use /mbadmin to upload a ZIP and place that into SVN
• Use /mbadmin to import (part of) the SVN into the MassBank system
• Allow these operations also from command line tools (see #28 ).

Meta data enrichment

I suggest a better enrichment of the meta tags of each record site to improvement indexing by the crawlers, for example the InChIkey and SPLASH could be added here. Needs enhancement of the DB model to map more content from the records.

Furthermore, Google suggests to use canonical URLs. The preferred link for indexing should be provided here in the page header:
https://support.google.com/webmasters/answer/139066?hl=en-GB

Google does not guarantee the usage of provided sitemaps (sic!). This was our first strategy to get better visible, it's going very slow...

Rendering by Specktackle is strange

Mail from @schymane
Re the depiction, the structures that are currently on MassBank.EU look strange, with the atoms but no Hs. Is this adjustable?
http://massbank.eu/MassBank/jsp/Dispatcher.jsp?type=disp&id=ETS00115&site=32&dsn=Eawag_Addn
e.g. the CH3 group should not be missing the H3, and I would prefer that Cs are suppressed automatically, it just looks strange.
http://massbank.eu/MassBank/jsp/Dispatcher.jsp?type=disp&id=ETS00019&site=32&dsn=Eawag_Addn
Ditto here, the OHs are missing the Hs, this is NOT a standard representation...and the chain looks weird with a stepped C-C-C-C-C...

Enhancement of DB model

MassBank should be searchable by SPLASH. We may also consider to update the DB model with some more indexes if we grab it anyway?

Activate or build missing tiles on homepage

Create more command line utilities

Currently MassBank is administered via the mbadmin web application,
which is user-friendly, but difficult to automate.

A first example was the command line validator:
https://github.com/MassBank/validator-cli

We also need a record and molecule uploader.

Implement CDK links

Link to record(s) via SPLASH

Hi, it would be great to link to records via a SPLASH,
e.g. https://massbank.eu/splash/splash10-000i-3900000000-73043667076aaf483c6e

Create private vagrant bootstrap settings

Some bootstrap settings are maybe confidential (e.g. passwords). Should be moved to a private bootstrap script which is listed in .gitignore to prevent the upload to a public repository

Replace molfiles upload and gif rendering by CDK rendering of SVGs from records

Exchange the GIF view by SVG, can be rendered using OpenBabel (on system anyway). Needs also changes in /mbadmin/StructureList.jsp etc.

MS Labels are not correctly displays all time

Emma's email:
Another thing: e.g. here:

http://massbank.eu/MassBank/jsp/Dispatcher.jsp?type=disp&id=EQ310401&site=31&dsn=Eawag

If you click “labels”, only one label appears for a tiny peak, but no mass is shown for the largest one. Maybe the margin needs to change slightly?

Needs some hacking in specktackle.

Uploading records to second database broken in current vagrant version

org.apache.jasper.JasperException: An exception occurred processing JSP page /mbadmin/RecordRegist.jsp at line 551

548: 				pw[i] = new PrintWriter( new BufferedWriter(new FileWriter(filePath)) );
549: 				pw[i].println( "START TRANSACTION;");
550: 			}
551: 			SqlFileGenerator sfg = new SqlFileGenerator( baseUrl, selDbName, ver );
552: 			for ( int i=0; i<dataFileList.size(); i++ ) {
553: 				String filePath = registPath + File.separator + dataFileList.get(i);
554: 				sfg.readFile( filePath );


Stacktrace:
	org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:568)
	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:470)
	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

root cause

java.lang.ArrayIndexOutOfBoundsException: 1
	massbank.GetInstInfo.getInformation(GetInstInfo.java:109)
	massbank.GetInstInfo.<init>(GetInstInfo.java:49)
	massbank.admin.SqlFileGenerator.<init>(SqlFileGenerator.java:77)
	org.apache.jsp.mbadmin.RecordRegist_jsp.registRecord(RecordRegist_jsp.java:540)
	org.apache.jsp.mbadmin.RecordRegist_jsp._jspService(RecordRegist_jsp.java:1175)
	org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

Implement https access to MassBank

Deployment of https services on MassBank server; needs a self-certificate for test case and implementation in Apache configuration

MassBank Record Format / MassBank data schema 2.10

Together with innovations in the database model, we may publish a new record format in order to make map the new database structure.

Improve developer friendlieness

Hi, the current massbank-web is a conglomerate of files extracted from the former
binary installer and the old CVS version control system. @meier-rene has already
started work (at https://github.com/meier-rene/MassBank-eclipse) to import all
relevant files into an Eclipse project. Even developers with different IDEs will benefit
from the cleaner structure. This issue can be closed once that file structure is imported.
Yours, Steffen

Move MassBank EU to cloud computing service

Moving MB EU from UFZ would make administration easier as for example Steffen could also access the server for administration issues. I suggest to use for example EGI: https://www.egi.eu/services/cloud-compute/

I recently talked to Jaroslav, he generally agrees. EGI is maybe less expensive than any private provider. And we would upgrade MB EU to real European level using the European Community cloud services.

Link InCHIKey to Google instead of Chemspider

So far Inchikey links to Chemspider. It might be more interesting for the people to link to Google instead in order to get more secondary information?

Accession code restrictions

If the DB is re-factorised any way, we may think about to ditch the accession code restrictions? Of course, we need some conventions such as the first 3 letters or so for the institution. But who cares if in the back are provided any alpha # code of any length? I would like to use our internal chemical codes such as W305 to make management of the codes easier.

Java applets are depreciated - solutions to replace or workarounds

Btw, as you might know that Java applets are depreciated quite soon? I guess, the soon will be end of 2016. For example Mozilla will stop applets support at this time point:

https://blog.mozilla.org/futurereleases/2015/10/08/npapi-plugins-in-firefox/

We should start a discussion how to carry on as soon as possible. One option is to use the Java Web Start technology:

https://blogs.oracle.com/java-platform-group/entry/moving_to_a_plugin_free

http://docs.oracle.com/javase/8/docs/technotes/guides/deploy/applet_dev_guide.html#JSDPG1032

There are described two ways of migration: migration and re-writing. Both cases need re-compilation of the applets and thus another code signing.

The second option is to migrate to Java applet free alternatives as recommended by Steffen some time ago.

The third option to use a mix of the Java web start things for some applets and new alternative techniques where appropriate?

Lets start the discussion, no rush, but urgent.

Add jk-manager and/or jk-status workers

[Tue Sep 01 23:25:41.961 2015] [19925:140405991069568] [error] extension_fix::jk_uri_worker_map.c (564): Could not find worker with name 'jk-manager' in uri map post processing.
[Tue Sep 01 23:25:41.961 2015] [19925:140405991069568] [error] extension_fix::jk_uri_worker_map.c (564): Could not find worker with name 'jk-status' in uri map post processing.

Creating two databases broken in current vagrant development version

How to reproduce:

1. Create a second database with mbadmin
2. Access http://192.168.XX.XX/MassBank/RecordIndex.html

Observe the following stack trace:

message An exception occurred processing JSP page /jsp/RecordIndex.jsp at line 115

description The server encountered an internal error that prevented it from fulfilling this request.

exception

org.apache.jasper.JasperException: An exception occurred processing JSP page /jsp/RecordIndex.jsp at line 115

112: 			val = siteNameList[siteNo] + "\t" + siteNo;
113: 		}
114: 		else {
115: 			val = fields[0].split(":")[1];
116: 		}
117: 		int count = Integer.parseInt( fields[1] );
118: 		if ( key.equals("site") ) {			// Contributor


Stacktrace:
	org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:568)
	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:470)
	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

root cause

java.lang.ArrayIndexOutOfBoundsException: 1
	org.apache.jsp.jsp.RecordIndex_jsp._jspService(RecordIndex_jsp.java:318)
	org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
	org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
	org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

Deleting the second database and going back to record index "fixes" the problem.

Add logfile analyzer

Add a logfile analyzer of the apache logs like awstats or piwik (both are in Ubuntu, piwik preferred,
see http://debian.piwik.org/).
By default, make sure the results are not visible to the outside, but only locally.
Also, enable some privacy protection as mentioned here:
http://piwik.org/docs/privacy/

Results.jsp: Remove buttons to obsolete applets

The results page shown by URL http://massbank.eu/MassBank/jsp/Result.jsp?type=quick includes buttons to execute obsolete applets.

Error in Record List Generator in mbadmin tools

Just start new Massbank (empty except one record) using Vagrant (vagrant up) and go to mbadmin tools (http://192.168.xx.xx/MassBank/mbadmin/)
Go to Record List Generator (http://192.168.xx.xx/MassBank/mbadmin/GenRecordList.jsp)
Select Database Massbank and click Execute.

message An exception occurred processing JSP page /mbadmin/GenRecordList.jsp at line 158
description The server encountered an internal error that prevented it from fulfilling this request.
exception

org.apache.jasper.JasperException: An exception occurred processing JSP page /mbadmin/GenRecordList.jsp at line 158

155: short col2 = (short)(item.length - 1 + n);
156: HSSFCell cell = hsRow.createCell(col2);
157: int pos = idList[n].indexOf(" NAME=");
158: String id = idList[n].substring( 0, pos );
159: String recordName = idList[n].substring( pos + 6 );
160:
161: cell.setCellValue(new HSSFRichTextString(id));

Stacktrace:
org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:568)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:470)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

root cause

java.lang.StringIndexOutOfBoundsException: String index out of range: -1
java.lang.String.substring(String.java:1911)
org.apache.jsp.mbadmin.GenRecordList_jsp._jspService(GenRecordList_jsp.java:243)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

note The full stack trace of the root cause is available in the Apache Tomcat/7.0.52 (Ubuntu) logs.

Breakdown of general architecture to docker containers if feasible

displaying of gif structure images broken on results page

Catch outdated urls

Hi,
I want to catch outdate urls and redirected them to the short link:

An example:
http://massbank.eu/MassBank/jsp/Dispatcher.jsp?type=disp&id=KOX00513&site=5

Ends up to a zombie page, because the internal order of DBs changed and the id cannot be found in the site (=DB) 5.

My idea is to redirected the request to:
http://massbank.eu/MassBank/jsp/FwdRecord.jsp?id=KOX00513

I think the best is too use the Apache vhost.conf by writing a RewriteRule such as:
RewriteRule ^(.*)$ http://massbank.eu$1 [R=301,NE]

As I am a RegEx dummy someone may help me to extract the id from the url:
RewriteRule ^?????$ http://massbank.eu/MassBank/jsp/FwdRecord.jsp?id=$1 [R=301,NE]

Thanks,
Tobias