singularityhub/singularity-python

command line tool for working with singularity hub and singularity containers

Singularity Python

Singularity Python is a python module and command line tool to provide helpers for working with Singularity containers, specifically providing functions to visualize, package, and compare containers.

• If you are looking for the Singularity Python client to pull, build, and otherwise wrap Singularity functions, then please see the repository singularity-cli for the spython module.
• If you are looking for local management and interaction with various storage locations (e.g., to pull and inspect containers in Singularity Hub, Registry, or other cloud resource) then please see the Global Client for the sregistry module.
• If you are looking for the Google Cloud (original) build functions, see the v2.5 branch.

Install

You have the option to install only the dependencies that are needed for your functionality of interest.

# All
pip install singularity

# Metrics and analysis dependencies
pip install singularity[metrics]

# Building on Google Cloud
pip install singularity[google]

We currently require Python > version 3 to use various timezone functions. If you are unable to install version 3.0, we provide a Singularity.container for you to use instead. This is the recommended approach as some older versions of Python do not support generation of the timestamp. See the installation docs for your different options.

License

This code is licensed under the Affero GPL, version 3.0 or later LICENSE. Please see our complete docs

Help and Contribution

Please contribute to the package, or post feedback and questions as issues. For points that require discussion of the larger group, please use the Singularity List

singularity-python's Issues

bug: package image exports string of image path, not image!

add function for visualization of os sims

we will want a scatterplot (colored by os, with points for different versions) for a single image, and more of a heatmap for many.

old tests need to be updated

the old tests (for docker container building, etc.) need to be redone, since the module was re-implemented in singularity-hub. I need to think about this - ideally the functions should be offered in both places, and be used from the same code base, meaning here, but I don't like creating dependencies.

run_client.py faild to pull an image, with "Do you have permission to write to ..." error message

STEPS TO RECREATE:

Create a conda environment in which I pip installed singularity-python:

conda create -n shub-1.1.8 python=2.7 source activate shub-1.1.8 pip install singularity

In order to download from singularity-hub.org the image: alaindomissy/rnaseqtools ,

Create and run a script as follows:

`

create a target directory for download with full permissions

mkdir DOWNLOADS
chmod +777 DOWNLOADS

create the following script, based on

https://github.com/singularityware/singularity-python/blob/master/examples/shub/run_client.py

cat > ~/DOWNLOADS/shub_pull.py <<'EOSCRIPT'
#!/usr/bin/python2
from future import print_function
from singularity.hub.client import Client

shub = Client()
container_name = 'alaindomissy/rnaseqtools'
collection = shub.get_collection(container_name)
container_ids = collection['container_set']
container_id = container_ids.pop()
print("container_id:", container_id)
manifest = shub.get_container(container_id)
image = shub.pull_container(manifest)

EOSCRIPT

make executable

chmod +x ~/DOWNLOADS/shub_pull.py

run it

cd DOWNLOADS
./shub_pull.py

`
This is the error I get:
ERROR:shub_builder:Error downloading https://www.googleapis.com/download/storage/v1/b/singularity-hub-regional/o/singularity-hub-regional%2Fgithub.com%2Falaindomissy%2Frnaseqtools%2Fe173a79b008e7aa3a51301da19328b63aa799b1e%2F42acf3dd57b3570816289ca532352acf.img.gz?generation=1494864381960279&alt=media. Do you have permission to write to /home/adomissy/DOWNLOADS?

NOTE: modifying the script so that the download goes to /tmp instead works:
chnaging the last line of the script to :
image = shub.pull_container(manifest, download_folder='/tmp')

These are the permission on tmp:
ls -ld /tmp
drwxrwxrwt 299 root root 2527232 May 15 19:19 /tmp

And these are the permissions on ~/DOWNLOADS
ll -d DOWNLOADS/
drwxrwxrwx 2 adomissy yeo-group 3 May 15 19:08 DOWNLOADS/

Thanks!

add inspect to client

(with tests)

remove docker2singularity functionality

this should be done with singularity command line software, and singularity-python should be optimized for singularity hub

method to push a singularity image from CI (continuous integration) to other storage

From @chrisfilo

The basic idea is that we could use continuous integration (eg, circle.ci) to do docker2singularity image conversion, as many docker images are hosted on Github and have these kind of checks. It would be triggered by a docker hub hook, or just pushing to a branch, and then push the images to a google drive that would get indexed by Singularity Hub. For more docker (base) images we could use a lightweight repo that would do the same. This would mean that a user could easily add their image to Singularity Hub without doing any kind of manual work (creating an account, logging in, clicking upload) by simply adding a line to their circle.yml (the continuous integration spec) file.

Helpful resources:

• conda forge: would let us define a custom set of installations / setup packages (called a "feedstock") to make it easy for the user to install everything with a conda command (I think)
• google drive client: At least while I'm at Stanford, I have lots of room here :)
• Amazon S3: via boto, since lots of people use S3, right?
• Dropbox: Not sure this is the best idea, but it is a place to store things that is pretty popular.

add client endpoints for getting labels, environment, and arguments

bug: LargeZipFile: Filesize would require ZIP64 extensions

For gcc, the following error was triggered:

  LargeZipFile: Filesize would require ZIP64 extensions

There is some file size threshold that requires a bigger/different zip extension - look into this!

make cleaner docs, hosted in docs folder from master

need method to detect language of script

either simple features, or some kind of machine learning model to detect language would work!

add proper logging to all modules

update continuous integration tests, do for python 2 and 3

update for 2.3 support

make functions for parsing run scripts, extracting meta data about params

add singularity containers to boutiques spec

see here

The boutiques spec is (currently, under development) using the boutiques spec to carry very simple info about inputs. If we have a proper field for the containers themselves, and perhaps the url on singularity hub, it would be meaningful for singularity hub to provide a (boutiques descriptor) of the container, for usage in other applications. This makes sense to test/do once we have the hub up and running, and can give containers index/urls, etc.

run_client.py fails - with apparent permission issue - to pull image into folder even with 777 permissions

STEPS TO RECREATE:

Create a conda environment in which I pip installed singularity-python:

conda create -n shub-1.1.8 python=2.7
source activate shub-1.1.8
pip install singularity

To download the alaindomissy/rnaseqtools image from singularity-hub.org create and run a script as follows:

# create a target directory for download with full permissions
`mkdir DOWNLOADS`
`chmod +777 DOWNLOADS`

# create the following script,  based on 
# https://github.com/singularityware/singularity-python/blob/master/examples/shub/run_client.py

```cat > ~/DOWNLOADS/shub_pull.py <<'EOSCRIPT'
#!/usr/bin/python2
from __future__ import print_function
from singularity.hub.client import Client

shub = Client()    
container_name = 'alaindomissy/rnaseqtools'
collection = shub.get_collection(container_name)
container_ids = collection['container_set']
container_id = container_ids.pop()
print("container_id:", container_id)
manifest = shub.get_container(container_id)
image = shub.pull_container(manifest)

EOSCRIPT

# make executable
chmod +x ~/DOWNLOADS/shub_pull.py

# run it
cd DOWNLOADS
./shub_pull.py

This is the error I get:

ERROR:shub_builder:Error downloading https://www.googleapis.com/download/storage/v1/b/singularity-hub-regional/o/singularity-hub-regional%2Fgithub.com%2Falaindomissy%2Frnaseqtools%2Fe173a79b008e7aa3a51301da19328b63aa799b1e%2F42acf3dd57b3570816289ca532352acf.img.gz?generation=1494864381960279&alt=media. Do you have permission to write to /home/adomissy/DOWNLOADS?
`

NOTE: Modifying the script so that the download goes to /tmp instead works:

changing the last line of the script to :

image = shub.pull_container(manifest, download_folder='/tmp')

these are the permission on tmp:

ls -ld /tmp
drwxrwxrwt 299 root root 2527232 May 15 19:19 /tmp

and these are the permissions on ~/DOWNLOADS

ls -ld DOWNLOADS/
drwxrwxrwx 2 adomissy yeo-group 3 May 15 19:08 DOWNLOADS/

Thanks!

builder scripts served with application not being added

add functions to derive tags, estimate os, and scores

This has been done, and I'm adding an issue here to make sure I detail it in the PR. Specifically, I've added classification functions that can estimate the operating system, return a list of scores to all base linux os (and different versions provided in the Docker Official Library), and generate a dendrogram (tree) for comparing groups of packages. The builder has been updated (again, not live yet!) to return these metrics to Singularity Hub, so each image will have:

• an estimated operating system
• similarity scores to all base OS (and different versions)
• a list of software tags derived from the image

This should make it much more than a registry / file store of images, but a software tool for visualizing, classifying, and learning about containers. Wicked excited for adding this, likely this coming week :)

reproduce/utils.py tweaks

the functions that use in memory tarfiles should have the option to extract by piping into stream, OR by reading file from the system. Right now all are piping into stream, and will have issue with larger images.

singularity hub visualization functions, etc. should be shared here

singularity hub should import functions from here (as was originally done) so the user always has ability to create the same visualizations, run builders, etc. for his or her images

update command line client to match docker2singularity.sh

add logging functionality to builders

we need to have proper logging, with an option to output to a file

singularity python should be source of build templates

Try converting Dockerfiles to singularity specs

We really should be able to store singularity images in github repos - and so it makes sense to have a repo of equivalent "Dockerfiles" - the singularity spec files (can we just call them Singularity?) so I should try to convert Dockerfile --> Singularityfile and then have functions to build from it.

need proper flask server

I was developing views statically, but these visualizations (to be better interactive) should have a proper flask server.

write equivalent views/functions for comparing images/packages

we would want to see either differences or similarity of images (or both).

add md5 sum to package and boutiques as version

we don't have any kind of version (eg, code version or github repo) so it seems reasonable to take an md5 sum of the entire image, and then minimally we can confirm if an image corresponds to a boutiques file / package.

might want to add more meta data to the zip

see here

add functions for proper compression and decompression of images

users typically think of tarballs when we talk about compressed images, but we actually want them to use gzip. Thus, it makes sense to provide functions for easily doing this. Something like:

    def compress(self,image_path):
        '''compress will (properly) compress an image'''
        if os.path.exists(image_path):
            compressed_image = "%s.gz" %image_path
            os.system('gzip -c -9 %s > %s' %(image_path,compressed_image))
        else:
            bot.logger.error("Cannot find image %s",image_path)



    def decompress(self,image_path):
        '''decompress will (properly) decompress an image'''
        if os.path.exists(image_path):
            extracted_file = image_path.replace('.gz','')
            os.system('gzip -d -f %s' %image_path)
            return extracted_file
        else:
            bot.logger.error("Cannot find image %s",image_path)

generate metric for readmes

make run script templates for users

first obvious languages can be bash and python

Question: Roadmap for development

I think we have some use cases for this software to integrate singularity containers into our python-based scientific workflows. I am just curious about current development activity and plans for a "first release" type of thing. I am not trying to hand out work, but just curious what your thoughts are.

Thanks.

bug in similarity calculation

using unique total instead of total between two containers (so resulting in some values > 1)
also should implement with sets so it's speedy!

fixed in development version, will update code soon.

finish writing tests for all modules

update testing to use Singularity spec file (not .def)

need to carefully debug dockerfile conversion

right now, I need to distinguish cmd and entrypoint based on a param (which goes in runscript). These variables also need to be parsed to check for lists of things. Then, I need to try a crapton of examples to make sure it works, and finally, go through the entire pipeline to see if a Dockerfile builds on Singularity Hub.

shub importing non-existent stuff

Looks like the shub script is perhaps mismatched relative to the rest of the code. I am getting import errors related to at least "build_spec" and "singularity.docker".

[client] add client just for singularity hub (not through singularity)

update singularity commands (without docker) to not require sudo

Currently, the strategy for extracting file and folder lists means using the export command (requiring sudo) with Singularity. This can likely be done without needing this, and all steps in the different pipelines (for calculations and visualizations of similarity, and builders) should be updated.

make docker it's own module

so we can do other cool stuff too :)

add singularity cli (python) wrapper so I can be lazy and use it :)

try using multiprocessing for reproducibility functions

add docker2singularity.sh to this package

the current singularity bash util does not work as it should, in the meantime use the docker2singuarity script I wrote.

converter needs to be updated (and tests written) for 2.3 support

[reproducibility] add assessment functions!

add continuous integration with tests

add tests to produce (tree) visualizations

and output as artifacts

views.py: make functions for visualizing image guts

an image can be represented as a graph/tree (based on folder paths, with files as nodes) or as a corpus, with each path being a document. I'm not sure which will be better for algorithms (testing this!) but minimally I think it will be useful (and fun!) to have functions to immediately open up a browser and see what is inside an image.

functions to summarize files / folder list

I'd want to be able to compare images (eg, subtract a base image from a second image that uses it to get added software) along with generating tags to describe the software. Likely the (right) way to go about this is to generate a bunch of singularity images, then generate a list of software common to the bases, maybe some NLP in there...