micom-dev/q2-micom

A Qiime plugin for MICOM.

A QIIME 2 plugin for MICOM.

Installation

You will need an existing QIIME 2 environment. Follow the instructions on (how to install QIIME 2) otherwise. q2-micom is compatible with all QIIME 2 distributions. Let's assume that environment was called qiime2-2024.2 for all further steps.

Add q2-micom to the QIIME 2 environment

This will be the same step for any supported QIIME 2 version but will vary depending on your operating system (similar to the normal QIIME 2 installation).

wget https://raw.githubusercontent.com/micom-dev/q2-micom/main/q2-micom.yml
conda env update -n qiime2-2024.2 -f q2-micom.yml
# OPTIONAL CLEANUP
rm q2-micom-*.yml

Finally, you activate your environment.

conda activate qiime2-2024.2

q2-micom will now install an open source solver that can be used with MICOM. If you use MICOM regularly we do recommend to obtain an academic license for CPLEX or Gurobi which will be faster.

Install a faster solver (recommended but optional)

CPLEX

QIIME 2 versions before 2021.4 are only compatible with CPLEX 12.10 or earlier (later version require at least Python 3.7).

After registering and downloading the CPLEX studio for your OS unpack it (by running the provided installer) to a directory of your choice (we will assume it's called ibm).

Now install the CPLEX python package into your activated environment:

pip install ibm/cplex/python/3.8/x86-64_linux

Substitute 3.8 with the Python version in your QIIME 2 environment, 3.6 for QIIME 2 up to 2021.2 and 3.8 for QIIME 2 2021.4 and newer. Substitute x86-64_linux with the folder corresponding to your system (there will only be one subfolder in that directory).

Gurobi

Gurobi can be installed with conda.

conda install -c gurobi gurobi

You will now have to register the installation using your license key.

grbgetkey YOUR-LICENSE-KEY

Finish your installation

If you installed q2-micom in an already existing QIIME 2 environment, update the plugin cache:

conda activate qiime2-2024.2  # or whatever you called your environment
qiime dev refresh-cache

You are now ready to run q2-micom!

Usage

Here is a graphical overview of a q2-micom analysis.

The best way to get started is to work through the community tutorial.

Supported QIIME 2 versions

q2-micom is tested against:

1. the current QIIME 2 version
2. the previous version

It should also work with

3. the development version
   However, this may occasionally break. Check here for the current status.

References

MICOM: Metagenome-Scale Modeling To Infer Metabolic Interactions in the Gut Microbiota
Christian Diener, Sean M. Gibbons, Osbaldo Resendis-Antonio
mSystems 5:e00606-19
https://doi.org/10.1128/mSystems.00606-19

q2-micom's Issues

Indicate supported CPLEX versions in the installation instructions

Hello, I'm having a difficult time trying to install the python API. I downloaded CPLEX for Linux x86-64 (part # CC8ATML) and it appears to only come with folders for python 3.7 and 3.8, so I cannot run this command from the set-up instructions:

pip install ibm/cplex/python/3.6/x86-64_linux

If I try to do the 3.7 folder I get the error:
Exception: this 'cplex' package (version 20.1.0.0) is not compatible with this version of Python: expected 3.7 but got 3.6

I've tried many ways to get it to work with the Q2 2020.11 conda env (the main setup.py script, adding it to the python path), but is says there are incompatibilities. I was able to just pip install the free version of cplex and build models, but I get this error when trying to use micom-grow and tradeoff:

"All numerical optimizations failed. This indicates a problem "
cobra.exceptions.OptimizationError: All numerical optimizations failed. This indicates a problem with the solver or numerical instabilities. Check that you have CPLEX or Gurobi installed. You may also increase the abundance cutoff to create simpler models."

I assume this is related to having the free version of CPLEX.

Is there some way to access the python 3.6 version of CPLEX? Or another way you recommend I try to install it?

Typo in the installation steps?

Hi,

in the installation steps described did you mean:

conda env update -n qiime2-2024.2 -f q2-micom.yml

as opposed to

conda env update -n qiime2-2024.2 -f q2-micom-linux.yml

Thanks!

update on CarvMe database

Hi!
I saw that the CarvMe database is now available, but the western_diet_gut.qza is only compatible with AGORA. So does this mean I can make a model based on CarvMe, but I cannot run qiime micom grow and any downstream analysis?
Thanks for clarifications!

mismatch of IDs between models and diet

Hi, I want to run micom 0.10.1 on the carveme_refseq84_genus.qza and the western_diet_gut_carveme.qza artifacts.
The model was created without any errors, but when I run the micom tradeoff command:

qiime micom tradeoff --i-models micom_0_10_1_carveme_genus_models.qza --i-medium western_diet_gut_carveme.qza --p-threads 16 --o-results micom_0_10_1_carveme_genus_tradeoff.qza --verbose

I receive the following error: No ID from the medium could be found in the exchange reactions. This means you probably have mismatched IDs...

Is there any known mismatch between the carveme metabolic community models and their respective carveme diet model?

Thanks again for any help and happy Easter!

Installation issues

I had some issues during installation:

FileNotFoundError: [Errno 2] No such file or directory: '/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/tqdm-4.7.2-py3.6.egg'

Solved using:
touch /home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/tqdm-4.7.2-py3.6.egg

Also:

Traceback (most recent call last):
File "/home/said/anaconda3/envs/qiime2-dev/bin/pip", line 10, in
sys.exit(main())
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_internal/cli/main.py", line 75, in main
return command.main(cmd_args)
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 105, in main
return self._main(args)
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 224, in _main
self.handle_pip_version_check(options)
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 146, in handle_pip_version_check
timeout=min(5, options.timeout)
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 91, in _build_session
index_urls=self._get_index_urls(options),
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_internal/network/session.py", line 241, in init
self.headers["User-Agent"] = user_agent()
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_internal/network/session.py", line 159, in user_agent
setuptools_version = get_installed_version("setuptools")
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 619, in get_installed_version
working_set = pkg_resources.WorkingSet()
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py", line 567, in init
self.add_entry(entry)
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py", line 623, in add_entry
for dist in find_distributions(entry, True):
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py", line 1983, in find_eggs_in_zip
if metadata.has_metadata('PKG-INFO'):
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py", line 1414, in has_metadata
return self._has(path)
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py", line 1854, in _has
return zip_path in self.zipinfo or zip_path in self._index()
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py", line 1731, in zipinfo
return self._zip_manifests.load(self.loader.archive)
File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pip/_vendor/pkg_resources/init.py", line 1688, in load
mtime = os.stat(path).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/tqdm-4.7.2-py3.6.egg'

Which I solved using:

conda install tqdm==4.7.2

Maybe you can specify that version in the tutorial.

observed differences of tutorial data visualizations

Hi again,
I observed a stronger clustering of taxa after running the following commands in the community tutorial:
qiime micom exchanges-per-taxon --i-results crc_growth.qza --o-visualization crc_niche.qzv
Were there any modifications with the --p-n-neighbors and --p-min-dist parameters before producing the example plot?

Error when filtering built community models and simulation results

First of all thanks for including the new functionality to filter existing community models and simulation results!
I tried both qiime micom filter-results and qiime micom filter-models, but in each case I get the following error message:

Plugin error from micom:
'Metadata' object has no attribute 'copy'

I tested different metadata files and validated those metadata files with keemei. So I'm kind of stucked at the moment...

By the way there is a typo in the community tutorial on how to filter results - it should be:

qiime micom filter-results --i-results growth.qza
--m-metadata-file metadata.tsv
--p-query "status == 'Cancer'"
--o-filtered-results cancer_results.qza

Thanks again for your help!

Improve validation of medium types on import

Hey, I created an adapted per sample diet model with the following structure for my dataset:

flux	dilution	metabolite	reaction	sample_id

0 0.000135232 0.1 fru_m EX_fru_m ORS18-017
1 0.000329774 0.1 glc_m EX_glc_m ORS18-017
2 4.99876E-05 0.1 gal_m EX_gal_m ORS18-017
3 0.0148986 0.1 man_m EX_man_m ORS18-017
4 0.000161329 0.1 mnl_m EX_mnl_m ORS18-017
5 0.0148986 0.1 fuc_L_m EX_fuc_L_m ORS18-017

However, after import as --type MicomMedium[PerSample] I receive the following error with the qiime micom grow command:

Plugin error from micom:

'DataFrame' object has no attribute 'reaction'

Hope you can help!
Thanks!

Export of CSV from qiime micom fit-phenotype visualization

It seems as if the direct export of .qzv results obtained with the qiime micom fit-phenotype command in CSV format does not work.
I tried with different browsers (chrome, firefox, edge). Either it takes really long or it is not working.
Thanks for fixing - if there is something to fix here ;-)

Error when trying to map tutorial data to species models

Hi there,

I am trying to build community models using the tutorial data provided, but rather than using the genus-specific models I've taken the species-specific ones from the linked provided (https://zenodo.org/record/3755182#.X7lx0M1Kg2w)

I get this error message after running qiime micom build though:

Merged with the database using ranks: species
Traceback (most recent call last):
  File "/home/chris/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2cli/commands.py", line 329, in __call__
    results = action(**arguments)
  File "<decorator-gen-247>", line 2, in build
  File "/home/chris/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/home/chris/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/chris/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2_micom/_build.py", line 67, in build
    tax = build_spec(abundance, taxonomy, models, cutoff, strict)
  File "/home/chris/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2_micom/_build.py", line 53, in build_spec
    round(stats["mean"]), round(stats["std"]))
ValueError: cannot convert float NaN to integer

Plugin error from micom:

  cannot convert float NaN to integer

Building the models at the genus level works fine. Any ideas?

Thanks,
Chris

tuple error

Hi, Christian!

I have a trouble with using micom as a qiime2(2021.2) plugin following the community tutorial.
When I tried to build growth, It's not working.
I attached the Error content.

Traceback (most recent call last):
File "/home/sybirg/miniconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2cli/commands.py", line 329, in call
results = action(**arguments)
File "", line 2, in grow
File "/home/sybirg/miniconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/home/sybirg/miniconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/action.py", line 418, in callable_executor
spec.qiime_type, output_view, spec.view_type, prov)
File "/home/sybirg/miniconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/sdk/result.py", line 267, in _from_view
result = transformation(view, validate_level)
File "/home/sybirg/miniconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/qiime2/core/transform.py", line 70, in transformation
new_view = transformer(view)
File "/home/sybirg/miniconda3/envs/qiime2-2021.2/lib/python3.6/site-packages/q2_micom/_transform.py", line 104, in _15
gr["growth_rates"].to_csv(rd.growth_rates.path_maker(), index=False)
TypeError: tuple indices must be integers or slices, not str

Plugin error from micom:

tuple indices must be integers or slices, not str

See above for debug info.

real 149m16.154s
user 1804m30.939s
sys 7m38.206s

I tried reinstallation qiime2, gurobi, and micom a few times, but It's still not working.
How can I solve this problem?
I look forward to hearing from you. Thanks! :)

Filtering samples from models for fit-phenotype

Hi, back again! I was wondering, is it possible to filter or subset samples from already built models? From what I understand they are built for each sample individually. I ask because I want to run fit-phenotype on different subsets of data (I have longitudinal data and want to look at individual time points). Of course I could just run the whole pipeline for feature tables from each timepoint, but I wanted to see if I could save time by filtering. Thanks!

Download links don't work on view.qiime2.org

See above. The online viewer does not really unpack the data so you can't access the raw CSVs. Will need to create a blob in the HTML.

Tradeoff error

I am using a skin microbiome dataset, I already calculated the models, minimal medium, growth. I tried to validate the trade-off but I got some errors:

Command:

qiime micom tradeoff --i-models models.qza \
	--i-medium minimal_medium.qza \
	--p-threads 40 \
	--o-results tradeoff.qza \
	--verbose

Error:

2020-04-02 17:48:31.066 | WARNING  | micom.solution:solve:217 - solver encountered an error infeasible
Process Process-6:
Traceback (most recent call last):
  File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/micom/workflows.py", line 9, in _process
    res = f(args)
  File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/q2_micom/_tradeoff.py", line 25, in _tradeoff
    rates = sol.members
AttributeError: 'NoneType' object has no attribute 'members'

I followed the process for a while and it seems like it will never end for that Process-6 error.

Gapseq GEMs, 16S QIIME data, western diet

Hi,

I want to run simulations with MICOM, using my own little catalogue of gapseq GEMs (instead of AGORA) and combine it with my 16S data. Western diet is currently used as placeholder.

This catalogue was created by running gapseq on 160 selected bacterial chromosomes, and creating "our160_genus.qza" from the resulting .xmls files via qiime.

Creating a model worked:

qiime micom build --i-abundance table-dada2.qza
--i-taxonomy taxonomy.qza
--i-models our160_genus.qza
--p-cutoff 0.0001
--p-threads 6
--o-community-models our160_models.qza \

When I want to use the model for tradeoff:

qiime micom tradeoff --i-models our160_models.qza
--i-medium western_diet_gut.qza
--p-threads 6
--o-results our160_tradeoff.qza \

An error comes:

Running ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/micom/workflows/tradeoff.py", line 23, in _tradeoff
    com.medium = medium[medium.index.isin(ex_ids)]
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/micom/community.py", line 663, in medium
    raise ValueError(
ValueError: No ID from the medium could be found in the exchange reactions. This means you probably have mismatched IDs...
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/q2cli/commands.py", line 339, in __call__
    results = action(**arguments)
  File "<decorator-gen-464>", line 2, in tradeoff
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    outputs = self._callable_executor_(scope, callable_args,
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/qiime2/sdk/action.py", line 391, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/q2_micom/_tradeoff.py", line 27, in tradeoff
    results = mw.tradeoff(manifest, model_folder, medium, tradeoffs, threads)
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/micom/workflows/tradeoff.py", line 111, in tradeoff
    results = workflow(_tradeoff, args, threads)
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/micom/workflows/core.py", line 55, in workflow
    results = list(it)
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/rich/progress.py", line 166, in track
    yield from progress.track(
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/site-packages/rich/progress.py", line 1119, in track
    for value in sequence:
  File "/home/conda/miniconda3/envs/qiime2-2022.2/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
ValueError: No ID from the medium could be found in the exchange reactions. This means you probably have mismatched IDs...

Plugin error from micom:

  No ID from the medium could be found in the exchange reactions. This means you probably have mismatched IDs...

See above for debug info.

Running the same files, however with AGORA or Carveme catalogue instead of my own, I get functional tradeoff files.

In an earlier version I just used 20 instead of 160 GEMs, but the same error appeared.

I am using the versions:
gapseq version: 1.2 ccd1144
Qiime2-2022.02.
micom: 0.12.1

Best,

qiime micom build KeyError: 'd__Bacteria;p__Bacteroidota;c__Bacteroidia;__;__;__'

Hi Diener,
I tried to run qiime micom build and had the error in below:

version info: q2cli version 2024.2.0 + QIIME 2 Plugin 'micom' version 0.14.0 (from package 'q2-micom' version 0.14.0)
I am wondering if I should remove all the unassigned taxonomy? or anything else was wrong?

many thanks for your help!
Zhaoju

Provide direct download links for tutorial data

Hi Christian,

I installed q2-micom on qiime2 environment following tutorials (https://github.com/micom-dev/q2-micom/blob/master/docs/README.md) and cplex (CPLEX_Studio128/cplex/python/3.6/x86-64_linux) on university hpcc space. After installation by $conda activate qiime2-2020.2, I tried to learn tutorials with provided database.

(qiime2-2020.2) -bash-4.2$ ls
agora103_genus.qza  crc_table.qza  crc_taxa.qza

I got error message after running command qiime micom build with sample datasets. Could you help me what I need to check?

(qiime2-2020.2) -bash-4.2$ qiime micom build --i-abundance crc_table.qza \
>                   --i-taxonomy crc_taxa.qza \
>                   --i-models agora_genus_103.qza \
>                   --p-cutoff 0.0001 \
>                   --p-threads 8 \
>                   --o-community-models models.qza \
>                   --verbose
Usage: qiime micom build [OPTIONS]

  Builds the metabolic community models for a set of samples.

Inputs:
  --i-abundance ARTIFACT FeatureTable[Frequency | RelativeFrequency]
                          The feature table containing the samples over which
                          beta diversity should be computed.        [required]
  --i-taxonomy ARTIFACT FeatureData[Taxonomy]
                          The taxonomy assignments for the ASVs in the table.
                                                                    [required]
  --i-models ARTIFACT MetabolicModels[JSON]
                          The single taxon model database to use.   [required]
Parameters:
  --p-threads INTEGER     The number of threads to use when constructing
    Range(1, None)        models.                                 [default: 1]
  --p-cutoff PROPORTION   Taxa with a relative abundance smaller than this
    Range(0.0, 1.0)       will be dropped.                   [default: 0.0001]
Outputs:
  --o-community-models ARTIFACT CommunityModels[Pickle]
                          The community models.                     [required]
Miscellaneous:
  --output-dir PATH       Output unspecified results to a directory
  --verbose / --quiet     Display verbose output to stdout and/or stderr
                          during execution of this action. Or silence output
                          if execution is successful (silence is golden).
  --citations             Show citations and exit.
  --help                  Show this message and exit.

                  There were some problems with the command:
 (1/3) Invalid value for "--i-abundance": 'crc_table.qza' is not a QIIME 2
  Artifact (.qza)
 (2/3) Invalid value for "--i-taxonomy": 'crc_taxa.qza' is not a QIIME 2
  Artifact (.qza)
 (3/3) Invalid value for "--i-models": 'agora_genus_103.qza' is not a valid
  filepath

error with qiime micom build and qiime micom grow commands

Dear Sean and Christian,

I'm very curious about micom and I tested it as a new plugin for qiime2. However, I experienced several issues:

First I tried to apply the tool on my own example dataset, but received the following error with the qiime micom build command:
Plugin error from micom: Length mismatch: Expected axis has 15 elements, new values have 8 elements

Ok then I thought I should stick to your example data covered in your tutorial (https://github.com/micom-dev/q2-micom/blob/master/docs/README.md)
However, for the crc_table.qza and crc_taxa.qza files I received the following error with qiime micom build:
(1/2) Invalid value for "--i-abundance": 'crc_table.qza' is not a QIIME 2
Artifact (.qza)
(2/2) Invalid value for "--i-taxonomy": 'crc_taxa.qza' is not a QIIME 2
Artifact (.qza)

Hm, then I thought let's try the small dataset of the moving picture tutorial (https://docs.qiime2.org/2019.10/tutorials/moving-pictures/) and the qiime micom build command worked! :slight_smile:

Nevertheless, with the qiime micom grow command I got another error:
Plugin error from micom: 'NoneType' object is not subscriptable
I guess this was due to several warnings I received during the execution that qiime micom grow could not solve cooperative tradeoffs for many samples.

So I was wondering how to proceed from here. Would be great if you could comment on those errors or maybe check the artifacts of the crc files, or recommend other suitable test datasets to get an idea how to get micom work...

By the way I used qiime2 2019.10 and the gurobi license.

Thanks for any support!

Comparability between humans and mice

Hi I was wondering if it makes any sense to use micom also for amplicon data from mouse fecal samples. Did you ever test how far micom's metabolic predictions could be extrapolated to the system of mice?
Thanks for any discussions...

access micom build output

Hello,
Thank you so much for developing micom and for assembling a very good 'community tutorial'.
Although I am using micom for a while, I realized that actually I never inspected the community.pickle file generated as output from micom build. I tried the regular pickle.load but still cannot access the list of models that are inside.
I run micom on some soil communities and my database contains half very good (curated) models and half not so good. I think that knowing what fraction of the database ends up in the community.models before 'growing' step will help.
Thank you so much again,
silviac