pbr/qtm

QTLTableMiner++ tool for mining tables in scientific articles

qtm's Issues

Use a single Solr core config file

Add ER diagram for the SQLite DB

Abbreviation arre found but not tagged by Solr

Look into AbbrevtoSynonyms.java

Change data dir for Solr cores

/var/solr/data-> installSolr/solrData

Improve scripts to create Solr cores

related to #34. Maybe using/indexing ontologies in RDF directly (see here)?

Populate QTL db schema from SQL batch file

related to #29

Similar column names for TraitValues and TraitProperty

Use single file for synonyms

Currently, there are four copies of the synonyms.txt file in Solr cores:

./propTerms/conf/synonyms.txt
./statoTerms/conf/synonyms.txt
./solaLyco2/conf/synonyms.txt
./terms/conf/synonyms.txt

Use symlinks? Modify this code (lines: 28-32) to work with X Solr cores

Add bash script (wrapper) for QTM.jar

QTM does not compile

maven (3.0.5).

mvn clean verify

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building XMLTABLE 2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ XMLTABLE ---
[INFO] Deleting /home/arni/gitrepos/candYgene/QTM/target
[INFO] 
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ XMLTABLE ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ XMLTABLE ---
[INFO] Compiling 17 source files to /home/arni/gitrepos/candYgene/QTM/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.909s
[INFO] Finished at: Thu Jun 01 16:13:26 CEST 2017
[INFO] Final Memory: 9M/239M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project XMLTABLE: Compilation failure: Compilation failure:
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/resultDb/qtlDb.java:[229,63] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] 
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/resultDb/qtlDb.java:[233,63] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] 
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/resultDb/qtlDb.java:[468,98] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] 
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/resultDb/qtlDb.java:[577,130] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] 
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/resultDb/qtlDb.java:[619,131] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] 
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/resultDb/qtlDb.java:[705,131] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] 
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/resultDb/qtlDb.java:[749,133] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] 
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/qtm/Table.java:[701,71] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] 
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/solrTagger/Annotate.java:[79,58] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] 
[ERROR] /home/arni/gitrepos/candYgene/QTM/src/main/java/solrTagger/Annotate.java:[154,61] error: package nl.erasmusmc.biosemantics.tagger.recognize does not exist
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Add Europe PMC endpoint to config

Currently, the URL is hard-coded in the PmcMetaReader.java:

String API_PMCXML = "http://www.ebi.ac.uk/europepmc/webservices/rest/" + PMCID + "/fullTextXML";

Add suffices to the output files

CSV -> *.csv
SQLite DB -> *.db

Database Naming Convention

http://codeplex.com/SqlServerSamples

1. Singular names for tables
2. Singular names for columns
3. Schema name for tables prefix (E.g.: SchemeName.TableName)
4. Pascal casing (a.k.a. upper camel case)

Name of the Repository

Name changed from TablesinXMI to QTM

Add NULL type for empty column values in QTL tables

See https://www.sqlite.org/datatype3.html

Example db entry:

select * from Qtl where qtlId = 10
              qtlId = 10
 traitNameInArticle = Fruit weight
traitNameInOntology = fruit weight
           traitUri = http://purl.obolibrary.org/obo/TO_0002746
   chromosomeNumber = 
   markerAssociated = 
          markerUri = 
     geneAssociated = 11.39‡; 
            geneUri = 
      snpAssociated = 
             snpUri = null
              pmcId = PMC4321030
            tableId = PMC4321030_TABLE 2
          rowNumber = 26

change empty '' and 'null' strings to NULL type

Make the config and use of vocabs/ontologies in Solr cores generic

Is the list in the README complete?

Currently, the vocabs/ontologies in the Solr cores are named as follows:
propTerms -> used in column classification (N.B. also using STATO!)
solaLyco -> remove
solaLyco2 includes SGN gene/transcript IDs and URIs
statoTerms-> STATO
terms - includes PATO, SPTO, PO and TO terms

See Solr data dir.

Add synonymous traits from PO and TO

Synonyms need to be added from PO and TO

Configration File

QTM execution fails using the example article

Example given in the README does not seem to work for me. QTM exits prematurely with the following error message:

Error: Could not find or load main class tablInEx.TablInExMainGnr

Rename target binary file

change XMLTABLE-2.0-SNAPSHOT.jar to QTM.jar

Update command-line interface incl. usage info

Revise/normalize db schema

Current db schema/ER diagram here.

• change pmc_id data type TEXT->NUMERIC and store IDs without the PMC prefix (i.e. PMC4266912->4266912)
• change tab_id data type TEXT->NUMERIC and store IDs without the PMC* prefix (i.e. PMC4266912_Table 1->1)
• TRAIT_TABLE columns n_cols and n_rows seems redundant as the counts can be obtained from TRAIT table, these could be removed and generated by a view
• change col_id and row_id data type TEXT->NUMERIC
• both col_id and row_id should (auto)increment (using the app code) || OR || Col id and Row id increment according to tables (tab_id) ||DONE||
• type column in CELL_ENTRY is not really needed as it could be derived from value column
• generate a new ER diagram (e.g. using DbSchema)

Remove hard-coded column names in the CSV file generating code

code lines: 424-427

The use of some config parameters is not clear.

Config file: config/configQtm.properties

By setting textFiles=./ generates [some PMCID].txt file. What's the purpose of this output file?

Fix db schema

Current db schema as found in QTL_PMC4266912.db needs some fixes. See proposed schema below.

QTM executes using the example but triggers an exception

./QTM -pmc PMC4266912 -o QTL_PMC4266912.db

...
Exception in thread "main" java.io.FileNotFoundException: /var/solr/data/terms/conf/synonyms.txt (No such file or directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at java.io.FileInputStream.<init>(FileInputStream.java:93)
	at solrAnnotator.AbbrevtoSynonyms.abbrevToSolrSynonyms(AbbrevtoSynonyms.java:39)
	at qtm.QtmMain.main(QtmMain.java:98)

Organize/clean-up files in the repo

e.g. src, config , doc and data dirs. The original ontology files do not need to be stored per se but rather fetched e.g. using a bash script etc.

Update readme

Add mavn and sqlite installation command

Add all config into a single file

e.g. you could organize these in sections

[solr]
key1 = value1
...

[qtm]
key1 = value1
...

Most params in config unused.

Config file: conf/configQtm.properties

Haven't found a reference in the code for these params:

...
#solr
serverSolr=http://localhost:8983/solr
solrProgram=solr/qtmSolrRun
core1Solr=core1
core2Solr=statoTerms
core3Solr=propTerms
core5Solr=solaLyco2
matchSolr=LONGEST_DOMINANT_RIGHT
typeSolr=dictionary
...

Remove unused Solr core references from schema.xml

solrData/data/propTerms/conf/schema.xml
solrData/data/statoTerms/conf/schema.xml
solrData/data/solaLyco2/conf/schema.xml
solrData/data/terms/conf/schema.xml
solrData/data/solaLyco/conf/schema.xml

Update README

and fix typos:)

Add option to control logging verbosity

See a nice tutorial here.

Add some stats to the summary output

e.g. number of (QTL) tables, number of QTLs...

Streamline the installation procedure

Abbreviation tagging in Solr

Only publish results in Data files

Improve exception handling and reporting

Re-running QTM on the same input raises the following exception:

...
Insert entry to the database.
-------------------------------------------------
java.sql.SQLException: [SQLITE_CONSTRAINT]  Abort due to constraint violation (UNIQUE constraint failed: ARTICLE.pmc_id)
	at org.sqlite.core.DB.newSQLException(DB.java:890)
	at org.sqlite.core.DB.newSQLException(DB.java:901)
	at org.sqlite.core.DB.execute(DB.java:810)
	at org.sqlite.core.DB.executeUpdate(DB.java:847)
	at org.sqlite.jdbc3.JDBC3PreparedStatement.executeUpdate(JDBC3PreparedStatement.java:86)
	at resultDb.QtlDb.insertArticleEntry(QtlDb.java:132)
	at qtm.QtmMain.main(QtmMain.java:128)
*************************************************
Article already exits, Please provide unique entries
*************************************************

A user should be notified as soon as possible rather than waiting for this message after processing input articles.

Add solr starting and stopping in the main file

Compilation fails

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Unnamed - com.wur:XMLTABLE:jar:2.0-SNAPSHOT
[INFO]    task-segment: [compile, install]
[INFO] ------------------------------------------------------------------------
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

Unable to get dependency information: Unable to read the metadata file for artifact 'com.google.code.gson:gson:jar': Invalid JDK version in profile 'doclint-java8-disable': Unbounded range: [1.8, for project com.google.code.gson:gson-parent
  com.google.code.gson:gson:jar:2.6.2

from the specified remote repositories:
  central (http://repo1.maven.org/maven2)

Path to dependency: 
	1) com.wur:XMLTABLE:jar:2.0-SNAPSHOT

Incorrect output dbfile path

./QTM -pmc PMC4266912 -o QTL_PMC4266912.db

the dbfile is in data/QTL_PMC4266912.db instead of ./QTL_PMC4266912.db

Remove extra space from CSV file

header

...gene_associated, geneUri... -> ...gene_associated,geneUri...

SQLite dbfile is empty

QTL_PMC4266912.db has populated schema but none of the tables contain actual data.

Improve installation & deployment

e.g. Docker-ize QTM

Annotate solyc gene id

Add scripts & doc to create Solr cores

Create view to export CSV file from QTL db

Upload Jar

Upload target/.jar

Incorrect file path using config param

For example, setting textFiles=tmp will generatetmp[PMCID].txt instead of tmp/[PMCID].txt.

Do not append entries to an existing database