Qs: is there a good tutorial for using this in singularity?

[sci-f]

Qs: is there a good tutorial for using this in singularity?

[vsoch]

hey @olechnwin - you are close! The tutorials expect that SCIF would be installed as a binary in the container, but Singularity has it implemented natively. So for Singularity, you would want to run an app like this:

singularity run --app hello-world container.sif
singularity shell --app hello-world container.sif
singularity exec --app hello-world container.sif env

If you want to have the same functionality as in the tutorials, then you'd want to build a container with scif installed (internally), or basically the tutorials above. If you have more specific questions please paste the definition files in the issue here, the links you shared are 404.

[olechnwin]

@vsoch
Sorry. I've fixed the links in my post above.
I don't have container.sif?
I tried to do singularity run --app hello-world hello-world.simg and have the same error.

[vsoch]

okay first testing with Singularity 2.2 (when scif apps was written)

$ sudo singularity build container.sif Singularity
Using container recipe deffile: Singularity
Sanitizing environment
Adding base Singularity environment to container
Docker image path: index.docker.io/continuumio/miniconda3:latest
Cache folder set to /root/.singularity/docker
[3/3] |===================================| 100.0% 
Exploding layer: sha256:68ced04f60ab5c7a5f1d0b0b4e7572c5a4c8cce44866513d30d9df1a15277d6b.tar.gz
Exploding layer: sha256:9c388eb6d33c40662539172f0d9a357287bd1cd171692ca5c08db2886bc527c3.tar.gz
Exploding layer: sha256:96cf53b3a9dd496f4c91ab86eeadca2c8a31210c2e5c2a82badbb0dcf8c8f76b.tar.gz
Exploding layer: sha256:9318fe5d63a8c51c1c9666998e60a664d588ad836407893f0834c0ada4f30b1c.tar.gz
hello-world-echo has runscript definition
hello-world-script has runscript definition
Adding custom environment to hello-world-echo
Adding custom environment to hello-world-script
Installing hello-world-script
+ cd /scif/apps/hello-world-script
+ echo echo 'Hello World!'
+ chmod u+x bin/hello-world.sh
Finalizing Singularity container
Calculating final size for metadata...
Skipping checks
Building Singularity image...
Singularity container built: container.sif
Cleaning up...

$ singularity run --app hello-world-script container.sif
Hello World!

Need to run to another computer to test with more recent...

[vsoch]

Testing with more recent singularity version 3.6.0-rc.4+6-gb9c7ca93e-dirty

$ sudo singularity build container.sif Singularity 
INFO:    Starting build...
Getting image source signatures
Copying blob 68ced04f60ab done  
Copying blob 9c388eb6d33c done  
Copying blob 96cf53b3a9dd done  
Copying config c096208585 done  
Writing manifest to image destination
Storing signatures
2020/07/07 17:46:29  info unpack layer: sha256:68ced04f60ab5c7a5f1d0b0b4e7572c5a4c8cce44866513d30d9df1a15277d6b
2020/07/07 17:46:29  info unpack layer: sha256:9c388eb6d33c40662539172f0d9a357287bd1cd171692ca5c08db2886bc527c3
2020/07/07 17:46:31  info unpack layer: sha256:96cf53b3a9dd496f4c91ab86eeadca2c8a31210c2e5c2a82badbb0dcf8c8f76b
INFO:    Running post scriptlet
+ cd /
+ . /scif/apps/hello-world-echo/scif/env/01-base.sh
+ SCIF_APPNAME=hello-world-echo
+ SCIF_APPROOT=/scif/apps/hello-world-echo
+ SCIF_APPMETA=/scif/apps/hello-world-echo/scif
+ SCIF_DATA=/scif/data
+ SCIF_APPDATA=/scif/data/hello-world-echo
+ SCIF_APPINPUT=/scif/data/hello-world-echo/input
+ SCIF_APPOUTPUT=/scif/data/hello-world-echo/output
+ export SCIF_APPDATA SCIF_APPNAME SCIF_APPROOT SCIF_APPMETA SCIF_APPINPUT SCIF_APPOUTPUT SCIF_DATA
+ cd /scif/apps/hello-world-echo
+ cd /
+ cd /
+ . /scif/apps/hello-world-script/scif/env/01-base.sh
+ SCIF_APPNAME=hello-world-script
+ SCIF_APPROOT=/scif/apps/hello-world-script
+ SCIF_APPMETA=/scif/apps/hello-world-script/scif
+ SCIF_DATA=/scif/data
+ SCIF_APPDATA=/scif/data/hello-world-script
+ SCIF_APPINPUT=/scif/data/hello-world-script/input
+ SCIF_APPOUTPUT=/scif/data/hello-world-script/output
+ export SCIF_APPDATA SCIF_APPNAME SCIF_APPROOT SCIF_APPMETA SCIF_APPINPUT SCIF_APPOUTPUT SCIF_DATA
+ cd /scif/apps/hello-world-script
+ echo echo 'Hello World!'
+ chmod u+x bin/hello-world.sh
+ cd /
INFO:    Creating SIF file...
INFO:    Build complete: container.sif

$ singularity run --app hello-world-script container.sif 
Hello World!

So I think it's likely you have a version installed back when there was a bug, this one to be exact. apptainer/singularity#5103 You should try updating your version.

[vsoch]

And if you cannot change your version, you should use an external SCIF recipe (recipe.scif) and install into a Singularity container with entrypoint scif, example is at https://sci-f.github.io/apps/examples/tutorials/getting-started

[olechnwin]

@vsoch
Thanks for your quick reply & help in figuring out this. I am using singularity 3.5 so it's probably not the same bug? Here is my complete build message so you can see.

$ wget https://sci-f.github.io/tutorials/Singularity
--2020-07-08 09:14:18--  https://sci-f.github.io/tutorials/Singularity
Resolving sci-f.github.io (sci-f.github.io)... 185.199.111.153, 185.199.109.153, 185.199.110.153, ...
Connecting to sci-f.github.io (sci-f.github.io)|185.199.111.153|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 678 [application/octet-stream]
Saving to: ‘Singularity’

100%[======================================>] 678         --.-K/s   in 0s

2020-07-08 09:14:18 (39.9 MB/s) - ‘Singularity’ saved [678/678]

$ dzdo singularity build --tmpdir=/data/tmp container.sif Singularity
Build target already exists. Do you want to overwrite? [N/y] y
INFO:    Starting build...
Getting image source signatures
Copying blob 68ced04f60ab skipped: already exists
Copying blob 9c388eb6d33c skipped: already exists
Copying blob 96cf53b3a9dd skipped: already exists
Copying config c096208585 done
Writing manifest to image destination
Storing signatures
2020/07/08 09:15:34  info unpack layer: sha256:68ced04f60ab5c7a5f1d0b0b4e7572c5a4c8cce44866513d30d9df1a15277d6b
2020/07/08 09:15:36  info unpack layer: sha256:9c388eb6d33c40662539172f0d9a357287bd1cd171692ca5c08db2886bc527c3
2020/07/08 09:15:41  info unpack layer: sha256:96cf53b3a9dd496f4c91ab86eeadca2c8a31210c2e5c2a82badbb0dcf8c8f76b
INFO:    Running post scriptlet
+ cd /
+ . /scif/apps/hello-world-script/scif/env/01-base.sh
+ SCIF_APPNAME=hello-world-script
+ SCIF_APPROOT=/scif/apps/hello-world-script
+ SCIF_APPMETA=/scif/apps/hello-world-script/scif
+ SCIF_DATA=/scif/data
+ SCIF_APPDATA=/scif/data/hello-world-script
+ SCIF_APPINPUT=/scif/data/hello-world-script/input
+ SCIF_APPOUTPUT=/scif/data/hello-world-script/output
+ export SCIF_APPDATA SCIF_APPNAME SCIF_APPROOT SCIF_APPMETA SCIF_APPINPUT SCIF_APPOUTPUT SCIF_DATA
+ cd /scif/apps/hello-world-script
+ echo echo 'Hello World!'
+ chmod u+x bin/hello-world.sh
+ cd /
+ cd /
+ . /scif/apps/hello-world-echo/scif/env/01-base.sh
+ SCIF_APPNAME=hello-world-echo
+ SCIF_APPROOT=/scif/apps/hello-world-echo
+ SCIF_APPMETA=/scif/apps/hello-world-echo/scif
+ SCIF_DATA=/scif/data
+ SCIF_APPDATA=/scif/data/hello-world-echo
+ SCIF_APPINPUT=/scif/data/hello-world-echo/input
+ SCIF_APPOUTPUT=/scif/data/hello-world-echo/output
+ export SCIF_APPDATA SCIF_APPNAME SCIF_APPROOT SCIF_APPMETA SCIF_APPINPUT SCIF_APPOUTPUT SCIF_DATA
+ cd /scif/apps/hello-world-echo
+ cd /
INFO:    Creating SIF file...
INFO:    Build complete: container.sif

$ singularity run --app hello-world-script container.sif
/.singularity.d/actions/run: 3: /.singularity.d/env/94-appsbase.sh: SCIF_APPDATA_hello-world-script=/scif/data/hello-world-script: not found
/.singularity.d/actions/run: 4: /.singularity.d/env/94-appsbase.sh: SCIF_APPMETA_hello-world-script=/scif/apps/hello-world-script/scif: not found
/.singularity.d/actions/run: 5: /.singularity.d/env/94-appsbase.sh: SCIF_APPROOT_hello-world-script=/scif/apps/hello-world-script: not found
/.singularity.d/actions/run: 6: /.singularity.d/env/94-appsbase.sh: SCIF_APPBIN_hello-world-script=/scif/apps/hello-world-script/bin: not found
/.singularity.d/actions/run: 7: /.singularity.d/env/94-appsbase.sh: SCIF_APPLIB_hello-world-script=/scif/apps/hello-world-script/lib: not found
/.singularity.d/actions/run: 9: export: SCIF_APPDATA_hello-world-script: bad variable name

$ singularity --version
singularity version 3.5.3-1.el7

[vsoch]

Nope that seems to be the same bug, the all names have - in them. I’ve already provided you instructions for how to address this.

[olechnwin]

Ok. Thank you.

[vsoch]

Great! We've addressed the bug you brought up, so I'm going to close the issue. Please let me know if you have other questions about the tutorials, etc.