vjcitn/hmdbquery

utilities for querying Human Metabolome Database

title	author	date	vignette	output
hmdbQuery: working with Human Metabolome Database (hmdb.ca)	Vincent J. Carey, stvjc at channing.harvard.edu	`r format(Sys.time(), '%B %d, %Y')`	%\VignetteEngine{knitr::rmarkdown} %\VignetteIndexEntry{hmdbQuery: working with Human Metabolome Database (hmdb.ca)}	html_document highlight number_sections theme toc pygments true united true

Initial remarks

The human metabolomics database (HMDB, http://www.hmdb.ca) includes XML documents describing 114000 metabolites. We will show how to manipulate the metadata on metabolites fairly flexibly.

suppressMessages({
suppressPackageStartupMessages({
library(hmdbQuery)
library(gwascat)
})
})

Key utilities of the package

The hmdbQuery package includes a function for querying HMDB directly over HTTP:

library(hmdbQuery)
lk1 = HmdbEntry(prefix = "http://www.hmdb.ca/metabolites/", 
       id = "HMDB0000001")

The result is parsed and encapsulated in an S4 object

lk1

The size of the complete import of information about a single metabolite suggests that it would not be too convenient to have comprehensive information about all HMDB constituents in memory. The most effective approach to managing the metadata will depend upon use cases to be developed over the long run.

Note however that this package does provide snapshots of certain direct associations derived from all available information as of Sept. 23 2017. Information about direct associations reported in the database is present in tables hmdb_disease, hmdb_gene, hmdb_protein, hmdb_omim. For example

data(hmdb_disease)
hmdb_disease

Working with the metadata

Disease associations

Some HMDB metabolites have been mapped to diseases.

d1 = diseases(lk1) # data.frame
pmids = unlist(d1["references", 5][[1]][2,])
library(annotate)
pm = pubmed(pmids[1])
ab = buildPubMedAbst(xmlRoot(pm)[[1]])
ab

Biospecimen and tissue location metadata

Note that pre HMDB v 4.0, biospecimens were called biofluids.

There are arbitrarily many biospecimen and tissue associations provided for each HMDB entry. We have direct accessors, and by default we capture all metadata, available through the store method.

biospecimens(lk1)
tissues(lk1)
st = store(lk1)
head(names(st))
length(names(st))
st$protein_assoc["name",]
st$protein_assoc["gene_name",]

hmdbquery's Issues

Error for certain HMDB entries

Hi, I try to do queries for different compounds and I get an error for some HMDB IDs, such as HMDB0000538, HMDB0001201, and HMDB0001273. The error:

Error in if (tissues[1] == "\n ") tissues = NULL :
argument is of length zero

package version: hmdbQuery_1.0.1

traceback:
1: hmdbQuery::HmdbEntry(id = "HMDB0000538")

sessionInfo:
R version 3.5.0 Patched (2018-05-24 r74787)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS

Best,
Hendrik

Error in if (tissues[1] == "\n  ") tissues = NULL :  argument is of length zero

I am attempting to use your hmdbQuery R package to pull metadata for a list of metabolites from hmdb. I am able to do this for about 215/366 metabolites. At this point, I am most interested in taxonomy. 

The most common error I am getting is: 

"Error in if (tissues[1] == "\n  ") tissues = NULL :  argument is of length zero" 
from the following query:

hmdb_result = HmdbEntry(
prefix = "http://www.hmdb.ca/metabolites/",
id = "HMDB0002467",
keepFull = TRUE
)

Thank you! Aaron

How to extract data for all biospecimen or one specific biospecimen?

Greetings!

I was wondering if I want to extract the data for all breast milk or all biospecimen? Is there any code that I could work with?

Thanks in advance!

Errors when extracting metabolites with null info in disease sections

Dear Dr. Carey,

Thanks for providing this useful tool to conveniently query the tissue and biofluid info with HMDB ID. I was trying to search for about 1000 metabolites. This tool will be very helpful to avoid manual check on these metabolites. However, when I was searching, this error kept popping up for more than half of the metabolites. I found the ones triggering errors are the ones which do not have any associated disease info. Is there any way to return an S4vector with null in the corresponding list instead of failing it? I could try to modify the source code if you allow me to. Thank you.

xml Parsing Error

When I run the following code:

HmdbEntry(prefix = "http://www.hmdb.ca/metabolites/", id = "HMDB61677")

I get the following very long error. Please help.

error parsing attribute name
attributes construct error
Couldn't find end of Start Tag n.length line 3
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag x.length line 3
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag n.allowed_origins.length line 3
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag l line 3
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
StartTag: invalid element name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag l line 3
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag n.length-1 line 3
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
StartTag: invalid element name
xmlParseEntityRef: no name
EntityRef: expecting ';'
StartTag: invalid element name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag o.length line 3
EntityRef: expecting ';'
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag t line 3
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag r.length line 3
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
StartTag: invalid element name
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag i line 3
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag f line 3
xmlParseEntityRef: no name
EntityRef: expecting ';'
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag n.length line 3
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag e.length line 3
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
EntityRef: expecting ';'
xmlParseEntityRef: no name
xmlParseEntityRef: no name
error parsing attribute name
attributes construct error
Couldn't find end of Start Tag t line 3
Entity 'times' not defined
Unescaped '<' not allowed in attributes values
attributes construct error
Couldn't find end of Start Tag a line 52
AttValue: " or ' expected
attributes construct error
Couldn't find end of Start Tag div line 52
AttValue: " or ' expected
attributes construct error
Couldn't find end of Start Tag div line 52
AttValue: " or ' expected
attributes construct error
Couldn't find end of Start Tag div line 52
Opening and ending tag mismatch: li line 0 and div
Opening and ending tag mismatch: ul line 0 and div
Opening and ending tag mismatch: ul line 0 and div
Opening and ending tag mismatch: td line 0 and a
Opening and ending tag mismatch: tr line 0 and li
Opening and ending tag mismatch: tbody line 0 and ul
Opening and ending tag mismatch: table line 0 and ul
Unescaped '<' not allowed in attributes values
attributes construct error
Couldn't find end of Start Tag a line 52
AttValue: " or ' expected
attributes construct error
Couldn't find end of Start Tag div line 52
AttValue: " or ' expected
attributes construct error
Couldn't find end of Start Tag div line 52
AttValue: " or ' expected
attributes construct error
Couldn't find end of Start Tag div line 52
Opening and ending tag mismatch: li line 0 and div
Opening and ending tag mismatch: ul line 0 and div
Opening and ending tag mismatch: main line 0 and div
Opening and ending tag mismatch: body line 0 and a
Opening and ending tag mismatch: html line 0 and li
Extra content at the end of the document
Error: 1: error parsing attribute name
2: attributes construct error
3: Couldn't find end of Start Tag n.length line 3
4: xmlParseEntityRef: no name
5: EntityRef: expecting ';'
6: xmlParseEntityRef: no name
7: xmlParseEntityRef: no name
8: xmlParseEntityRef: no name
9: xmlParseEntityRef: no name
10: xmlParseEntityRef: no name
11: xmlParseEntityRef: no name
12: xmlParseEntityRef: no name
13: xmlParseEntityRef: no name
14: xmlParseEntityRef: no name
15: EntityRef: expecting ';'
16: xmlParseEntityRef: no name
17: EntityRef: expecting ';'
18: xmlParseEntityRef: no name
19: xmlParseEntityRef: no name
20: xmlParseEntityRef: no name
21: EntityRef: expecting ';'
22: xmlParseEntityRef: no name
23: xmlParseEntityRef: no name
24: xmlParseEntityRef: no name
25: EntityRef: expecting ';'
26: xmlParseEntityRef: no name
27: EntityRef: expecting ';'
28: xmlParseEntityRef: no name
29: xmlParseEntityRef: no name
30: xmlParseEntityRef: no name
31: xmlParseEntityRef: no name
32: xmlParseEntityRef: no name
In addition: Warning message:
In readLines(.hmxPath(prefix = prefix, id = id, ...)) :
incomplete final line found on 'http://www.hmdb.ca/metabolites/HMDB61677.xml'

Error when retrieving HMDB0014328 & HMDB0060475

Dear Dr. Carey,

I encountered error when using your R package hmdbQuery

Error in validObject(.Object) : 
  invalid class “HmdbEntry” object: 1: invalid object for slot "diseases" in class "HmdbEntry": got class "NULL", should be or extend class "matrix"
invalid class “HmdbEntry” object: 2: invalid object for slot "biospecimens" in class "HmdbEntry": got class "NULL", should be or extend class "character"
The HMDB XML has an unexpected structure.  Please 
report the HMDB ID to [email protected].  Returning a list of all 
retrieved data.[1] "HMDB0014328"
Error in validObject(.Object) : 
  invalid class “HmdbEntry” object: invalid object for slot "diseases" in class "HmdbEntry": got class "NULL", should be or extend class "matrix"
The HMDB XML has an unexpected structure.  Please 
report the HMDB ID to [email protected].  Returning a list of all 
retrieved data.[1] "HMDB0060475"

But actually, the list object is great for retrieving other related information (e.g., molecular weight, monoisotopic mass). Is your package equipped to have the full list object, instead of S4 object retrieved?

I appreciate your help!

Thanks,
Minghao Gong