ggirelli/pygpseq

Python3 package to analyse GPSeq images and manage TIFF files.

!DISCLAIMER!

1. This package is not currently maintained. A new package that will include all pygpseq features is being implemented at radiantkit.
2. This package has been developed and tested ONLY for Python3.6, which will reach its end of life On December 23rd, 2021.
3. Versions 3.4.* of this package only change package dependencies to fix an issue due to incorrect dependency declaration.

---

---

---

pyGPSeq

A Python3.6 package that provides tools to analyze images of GPSeq samples. Read the Wiki documentation for more details.

Requirements

Python3.6 and compatible tkinter package are required to run pygpseq. On Ubuntu 20.04, you can install them with:

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.6
sudo apt install python3.6-tk

Installation

We recommend installing pygpseq using poetry. Check how to install poetry here if you don't have it yet! Once you have poetry ready on your system, you can install the package in its own virtual environment with:

git clone https://github.com/ggirelli/pygpseq.git
cd pygpseq
poetry install

And then enter the environment with poetry shell.

Alternatively, if you prefer to use conda , you can setup an environment with:

conda create -n pygpseq python=3.6
conda activate pygpseq
conda install pip
conda install -c anaconda libtiff 

Usage

Analyze a GPSeq image dataset

The gpseq_anim (GPSeq analysis of images) analyzes a multi-condition GPSeq image dataset. Run gpseq_anim -h for more details.

Calculate lamin distance of FISH signals

The gpseq_fromfish script characterizes FISH signals identified with DOTTER (or similar tools) by calculating: absolute/normalized distance from lamina and central region, nuclear compartment, allele status,... Run gpseq_fromfish -h for more details.

Merge multiple FISH analyses using a metadata table

Use the gpseq_fromfish_merge script to merge multiple FISH analysis output (generated with gpseq_fromfish). For more details run gpseq_fromfish_merge -h.

Perform automatic 3D nuclei segmentation

Run tiff_auto3dseg -h for more details on how to produce binary/labeled (compressed) masks of your nuclei staining channels

Identify out of focus (OOF) fields of view

Run tiff_findoof -h for more details on how to quickly identify out of focus fields of view. Also, the tiff_plotoof script (in R, requires argparser and ggplot2) can be used to produce an informative plot with the signal location over the Z stack.

Split a tiff in smaller images

To split a large tiff to smaller square images of size N x N pixels, run tiff_split input_image output_folder N. Use the --enlarge option to avoid pixel loss. If the input image is a 3D stack, then the output images will be of N x N x N voxels, use the --2d to apply the split only to the first slice of the stack. For more details, run tiff_split -h.

(Un)compress a tiff

To uncompress a set of tiff, use the tiffcu -u command. To compress them use the tiffcu -c command instead. Use tiffcu -h for more details.

Convert a nd2 file into single-channel tiff images

Use the nd2_to_tiff tool to convert images bundled into a nd2 file into separate single-channel tiff images. Use nd2_to_tiff -h for the documentation.

Contributing

We welcome any contributions to pygpseq. Please, refer to the contribution guidelines if this is your first time contributing! Also, check out our code of conduct.

License

MIT License
Copyright (c) 2017-21 Gabriele Girelli

pygpseq's Issues

Initial Update

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create seperate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖

Better error message when only one channel tiff is missing

This issue affects

• GPSeq image analysis (gpseq_anim)

What did you do (e.g., steps to reproduce)

gpseq_anim -d dapi -s cy5 -t 36 --note "TK GPSeq YFISH IMR90 deconvolved, center as percentile" --center-as-percentile -r '/home/bicro/Desktop/Backup_in_progress/user_folder_HD1/Tomek/microscopy_pipeline/IMR90_111217/IMR90_Data_input' '/home/bicro/Desktop/Backup_in_progress/user_folders_HD2/Tomek/microscopy_pipeline/IMR90_080518/Data_output' --sigma-smooth 0.05 -n

What did you expect to happen?

Get to the end of the pipeline.

What happened instead?

Missing DNA channel broke execution.

Additional information

/home/bicro/Desktop/ggcode/pygpseq/pygpseq/anim/series.py in get_channel(self=<pygpseq.anim.series.Series object>, ch_name=('dapi',), log='    > Reading channel "(\'dapi\',)"...\n', **kwargs={'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/Backup_in_progress/user_fold.../IMR90_111217/IMR90_Data_input/iTK165_021217_001/', 'calc_n_surface': False, 'cdescr': {}, 'center_as_percentile': True, 'cond_name': 'iTK165_021217_001', 'correctCA': False, 'debugging': False, ...})
    311         if None == log: log = ""
    312         log += self.printout('Reading channel "' + str(ch_name) + '"...', 2)
    313 
    314         # Read channel
    315         f = self.find_channel(ch_name)
--> 316         imch = imt.read_tiff(os.path.join(self.basedir, f[0]))
        imch = undefined
        self.basedir = '/home/bicro/Desktop/Backup_in_progress/user_fold.../IMR90_111217/IMR90_Data_input/iTK165_021217_001/'
        f = {}
    317         imch = imt.slice_k_d_img(imch, 3)
    318 
    319         # Deconvolved images correction
    320         if 'rescale_deconvolved' in kwargs.keys():

KeyError: 0

Retain nd2 metadata when converting to tiff in `nd2_to_tiff`.

Crashed for unknown reasons

Before submitting an issue, please be sure to

• Read the contributing instructions
• Update to the latest version (run git pull; sudo -H pip3 install .)

This issue affects

• GPSeq analysis of FISH data (gpseq_fromfish)

What did you do (e.g., steps to reproduce)

Ran as in iFISH-singLoc-snakemake.

What happened instead?

...........................................................................
.../pygpseq/fish/image.py in analyze_field_of_view(sid=4, data=               x            y          z ...   x...6723 ...  564   954  26

[1958 rows x 15 columns], im2fov={1: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_001_cmle.tif', 2: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_002_cmle.tif', 3: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_003_cmle.tif', 4: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_004_cmle.tif', 5: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_005_cmle.tif', 6: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_006_cmle.tif', 7: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_007_cmle.tif', 8: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_008_cmle.tif', 9: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_009_cmle.tif', 10: '/media/MiSo/iFISH-singleLocus-analysis/input/decon/iAM196_20180830_002//dapi_010_cmle.tif', ...}, dilate_factor=5, istruct=array([[[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
     ...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]], dtype=uint8), aspect=[300.0, 130.0, 130.0], mask_dir='/media/MiSo/iFISH-singleLocus-analysis/input/cmask/iAM196_20180830_002/', mask_prefix='cmask_', plotCompartments=True, pole_fraction=0.25, outdir='/media/MiSo/iFISH-singleLocus-DOTTER/output/iFIS...orrected-preFitting/fromfish/iAM196_20180830_002/', noplot=False, labeled=False, compressed=True, dist_type=1, nbins=200, discard_dilation_mode=False, an_type=2, seg_type=2, mask2d_dir=None, verbose=False, debug=False, debug_dir='')
    158 
    159     # Export mask as PNG
    160     if not noplot:
    161         # Create png masks output directory
    162         maskdir = os.path.join(outdir, 'masks/')
--> 163         if not os.path.isdir(maskdir): os.mkdir(maskdir)
        maskdir = '/media/MiSo/iFISH-singleLocus-DOTTER/output/iFIS...ed-preFitting/fromfish/iAM196_20180830_002/masks/'
    164         imbname = os.path.splitext(os.path.basename(im2fov[sid]))[0]
    165 
    166         # Save default mask
    167         msg += printout("Saving default binary mask...", 3, v)

FileExistsError: [Errno 17] File exists: '/media/MiSo/iFISH-singleLocus-DOTTER/output/iFISH-singLoc-snakemake-rootOutput-CAcorrected-preFitting/fromfish/iAM196_20180830_002/masks/'
___________________________________________________________________________

Additional information

Better error report and format check in fromfish_merge

Any ideas about this:

Finished iEG452_007
[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   25.1s remaining:    0.0s
Traceback (most recent call last):
  File "/usr/local/bin/gpseq_fromfish_merge", line 369, in <module>
    delayed(extract_data)(did, sid) for did, sid in uniID)
  File "/home/erikw/.local/lib/python3.6/site-packages/joblib/parallel.py", line 779, in __call__
    while self.dispatch_one_batch(iterator):
  File "/home/erikw/.local/lib/python3.6/site-packages/joblib/parallel.py", line 625, in dispatch_one_batch
    self._dispatch(tasks)
  File "/home/erikw/.local/lib/python3.6/site-packages/joblib/parallel.py", line 588, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/home/erikw/.local/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 111, in apply_async
    result = ImmediateResult(func)
  File "/home/erikw/.local/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 332, in __init__
    self.results = batch()
  File "/home/erikw/.local/lib/python3.6/site-packages/joblib/parallel.py", line 131, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/home/erikw/.local/lib/python3.6/site-packages/joblib/parallel.py", line 131, in <listcomp>
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/usr/local/bin/gpseq_fromfish_merge", line 178, in extract_data
    session = "%03d" % sid
ValueError: cannot convert float NaN to integer

i.e. how I can track it down?

gpseq_fromfish got stuck at a corrupt image

This issue affects

• GPSeq analysis of FISH data (gpseq_fromfish)

What did you do (e.g., steps to reproduce)

gpseq_fromfish -y -t 3 --dilate 5 --compressed -M cmask --dist-type center_max ../../data/dataFromNM/dots_iEG466_001.csv ../../data/dec_3d/iEG463_001/          ../../data/lamin_distance/lamin_distance_all_f_20180628/iEG463_001/

the script got stuck after printing this:

~~ ERROR ~~ ლ(ಠ益ಠლ)
Something went wrong while trying to read a file (possibly corrupt):
../../data/dec_3d/iEG463_001//dapi_045_cmle.tif

Terminated.

[Parallel(n_jobs=3)]: Done  38 tasks      | elapsed: 14.7min
[Parallel(n_jobs=3)]: Done  39 tasks      | elapsed: 15.2min

What did you expect to happen?

Either:
1/ the script returns with -1, or
2/ the script continues to the next field of view and issues a warning
3/ Possibly either of the above with a flag -Werror (warning is error) that turns warnings into errors.

What happened instead?

gpseq_fromfish halted

Missing dependencies to czifile and cython packages in setup.py

double negations are hard for non-super-humans

        Do not plot : False
No compartment plot : False

This issue affects

• [x ] GPSeq analysis of FISH data (gpseq_fromfish)

What did you do (e.g., steps to reproduce)

This is in the list of settings to confirm before the script goes off.

What did you expect to happen?

Something like Plot : Yes, ...

What happened instead?

My brain almost exploded.

Additional information

NA

Undocumented warnings

This issue affects

• Merging FISH data analysed with GPSeq style (gpseq_fromfish_merge)

Start parallel
Working on iEG459_006...
Warning! No copy pairs found in iEG459_006.

Finished iEG459_006
[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    1.9s remaining:    0.0s
Working on iEG458_004...
Warning! No copy pairs found in iEG458_004.

Warning! No copy pairs found in iEG458_004. Can't figure out what that means and if it is a possible problem!

In tiff_splitt, -I has the opposite behaviour from expected.

In gpseq_anim calculate statistics of signal channel per nucleus.

DFE can merge nuclei, add watershed option in tiff_auto3dseg

URGENT: add compatibility with matplotlib>=3.0

In tiff_auto3dseg use `gcc.export_settings`.

Additional options to `tiff_findoof`

-s for silent run (no tdqm).
-d for dry run (only say what it would do without doing it).

warnings from gpseq_fromfish

Warnings when running gpseq_fromfish, however, the script continues without crashing.

This issue affects

• [ x] GPSeq analysis of FISH data (gpseq_fromfish)

What did you do (e.g., steps to reproduce)

Command line:

gpseq_fromfish -t 2 -y  -m    ../../data/seg_3d/iEG458_171304_002  ../../data/dataFromNM/dots_iEG458_002.csv  ../../data/dec_3d/iEG458_171304_002    ../../data/lamin_distance_imr90_all_20180820/iEG458_171304_002 

What happened instead?

...
       Input regexp : ^.*\.tiff?$
              Delim : ','
            Threads : 2
         Debug mode : False
    
  Isotropic dilation of (0, 0, 0) px in ZYX, respectively.
  Missing image for field #5, skipped.
  Missing image for field #6, skipped.
  > Analyzing fields of view... [n.threads=2]
/home/erikw/.local/lib/python3.6/site-packages/skimage/external/tifffile/tifffile.py:1630: UserWarning: axes do not match shape
  warnings.warn("axes do not match shape")
/home/erikw/.local/lib/python3.6/site-packages/skimage/external/tifffile/tifffile.py:1630: UserWarning: axes do not match shape
  warnings.warn("axes do not match shape")
...

Clean `tools.binarization` module.

G1 filter in `gpseq_fromfish`

In gpseq_fromfish, implement gaussian sum fitting for G1 selection instead of current FWHM (roll back the default to a separate option).

Re-factor for better documentation with python `help()`.

Move constants to appropriate sub-modules.

Support for 2D fields in czi/nd2_to_tiff

This might already be working but it has never been tested.

Add flexible nuclear selection tool (aim at sub-population selection).

Standardize XYZ coordinate noation

Standardize XYZ coordinates across submodules. (e.g., anim.nucleus.Nucleus.box_origin has ZYX where Y are the rows and X the columns, while the input of gpseq_fromfish expects X as the rows and Y as the columns).