opencadc / vostools Goto Github PK

View Code? Open in Web Editor NEW

12.0 12.0 18.0 22.16 MB

VOSpace command line and FUSE clients

Home Page: https://www.canfar.net/en/docs/storage

Python 74.57% Shell 24.01% Makefile 0.71% Batchfile 0.70% HTML 0.01%

vostools's People

Contributors

Stargazers

Watchers

Forkers

drusk jonathansick edwardchapin normanhill igable astrocrazy andamian anweshadas brianmajor c3tp anazalea yeunga aardalath prkar jburke-cadc zonia3000 sharongoliath

vostools's Issues

control-c is ugly.

vos CL apps should capture the 'control-c' stroke and just exit cleanly instead of dumping stack trace.

VOFS doesn't work with MS Word

During the MS Word file save process there is some interaction of opens/flush/delete etc that results in the exception message below. What ever the sequence is, files can not be saved to 'vospace' correctly. Although running Word on a file isn't a requirement for VOFS I suspect that the interaction chain is revealing an error in the VOFS logic that might affect other users.

2017-09-15 12:48:54,052 123145457655808 vos-3.0.0b6 logExceptions.wrapper.18 Exception throw: <type 'exceptions.OSError'> [Errno 16] File refcount indicates files handle busy.
2017-09-15 12:48:54,052 123145457655808 vos-3.0.0b6 logExceptions.wrapper.19 Traceback (most recent call last):
File "/Users/kavelaarsj/anaconda2/lib/python2.7/site-packages/vos/logExceptions.py", line 15, in wrapper
return func(*args, **kwds)
File "/Users/kavelaarsj/anaconda2/lib/python2.7/site-packages/vofs-3.0.0b6-py2.7.egg/vofs/vofs.py", line 618, in flush
fh.cache_file_handle.flush()
File "/Users/kavelaarsj/anaconda2/lib/python2.7/site-packages/vofs-3.0.0b6-py2.7.egg/vofs/CadcCache.py", line 888, in flush
raise OSError(errno.EBUSY, "File refcount indicates files handle busy.")
OSError: [Errno 16] File refcount indicates files handle busy.

mountvofs: set NULL instead of NONE

There are many records in the VOSpace database whose groups are set to the word 'NONE', but
there is no group called 'NONE'. The Python VOSpace wrappers are setting it to 'NONE', rather than NULL. It has to be consistent among all VOSpace...

vmkdir -p does not do anything

vmkdir does not seem to create directories as required:

$ vmkdir -p vos:sfabbro/toto/tata
$ vls vos:sfabbro/toto/tata
ERROR vls [Errno 2] NodeNotFound: vos://cadc.nrc.ca!vospace/sfabbro/toto/tata
$ vmkdir -h | grep -- -p
-p Create intermediate directories as required.

vos reports warning that is too verbose.

This could be fixed up at the same time as issue #103

The first time one is using any VOS commands after a fresh installation, an error is displayed:

Can not read /.config/vos-config.
Although harmless, it is triggering questions.

The second command is executed without any message (.config was then created)

Is first message a debug statement left in the code?

Also, I don’t think we should use something as simple as ‘.config’. It is a bit generic. ;->

VOFS cache can be/is stale when external copies occur.

Below is shell history that demonstrates this issue.

mach87:~ jjk$ mountvofs --vospace vos:jkavelaars --mountpoint ${HOME}/jkavelaars
mach87:~ jjk$ cp ${HOME}/jkavelaars/script.sh ./
mach87:~ jjk$ head script.sh 
#!/bin/bash
# a script

source ${HOME}/.bash_profile

exp1=$1
exp2=$2
exp3=$3
ccd=$4
mach87:~ jjk$ diff ${HOME}/jkavelaars/script.sh script.sh 
mach87:~ jjk$ rm ${HOME}/jkavelaars/script.sh 
mach87:~ jjk$ vi script.shmach87:~ jjk$ head script.sh 
#!/bin/bash
# a script
JUNK

source ${HOME}/.bash_profile

exp1=$1
exp2=$2
exp3=$3
ccd=$4
mach87:~ jjk$ vcp script.sh vos:jkavelaars/
mach87:~ jjk$ diff ${HOME}/jkavelaars/script.sh script.sh 
2a3
> JUNK
mach87:~ jjk$ 
mach87:~ jjk$ vcat vos:jkavelaars/script.sh | head
#!/bin/bash
# a script
JUNK

source ${HOME}/.bash_profile

exp1=$1
exp2=$2
exp3=$3
ccd=$4

Oops. Why didn't the rm clear the cached version of the file?

copy in vos does not compare md5 before downloading file

This causes files to be downloading even if identical copies already exist at the local destination.

unhandled exceptions when opening a file for read/write/append

It seems likely that the meta data, and in particular the bitmap size no longer matches the data file in the cache. I don't believe this is possible in the log shown, but it may have happened earlier.

This may be possible - the allocation of a new CacheMetaData object in flushNode, setHeader, and open are not protected by the same mechanism. It is possible for a race condition to allow the meta data to be set incorrectly.

The conditions which would allo this to happen would be many threads simultaneously opening the same file, appending to it and closing the file.

A sample of the cache data and meta data file are available for further investigation.

There may be other race conditions associated with opening files while they are being flushed, written, or truncated.

Also, there seems to be a bug near line 208 of CadcCache.py:

                    fileHandle.fileSize = os.path.getsize(
                            fileHandle.cacheMetaDataFile)

I believe the size should be set to the size of data file, not the size of the meta data file.

Following is the output from one vofs mount with a CASA application running:

2014-09-04 12:52:28,684 47838481070400 vos-1.10.4fuse.call.845 -> open /test.src.split.ms.contsub.chan/HISTORY/table.f0 (32770,)
2014-09-04 12:52:28,684 47838481070400 vos-1.10.4vofs.open.462 Opening /test.src.split.ms.contsub.chan/HISTORY/table.f0 with flags a+
2014-09-04 12:52:28,685 47838481070400 vos-1.10.4CadcCache.getAttr.520 gettattr /test.src.split.ms.contsub.chan/HISTORY/table.f0:
2014-09-04 12:52:28,685 47838481070400 vos-1.10.4vofs.getNode.390 force? -> False path -> /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,685 47838481070400 vos-1.10.4vofs.getNode.394 requesting node /test.src.split.ms.contsub.chan/HISTORY/table.f0 from VOSpace
2014-09-04 12:52:28,685 47838481070400 vos-1.10.4vos.getNode.1095 Getting node /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,685 47838481070400 vos-1.10.4vofs.getNode.390 force? -> False path -> /test.src.split.ms.contsub.chan/HISTORY
2014-09-04 12:52:28,685 47838481070400 vos-1.10.4vofs.getNode.394 requesting node /test.src.split.ms.contsub.chan/HISTORY from VOSpace
2014-09-04 12:52:28,685 47838481070400 vos-1.10.4vos.getNode.1095 Getting node /test.src.split.ms.contsub.chan/HISTORY
2014-09-04 12:52:28,685 47838481070400 vos-1.10.4CadcCache.init.732 creating a new File Handle for /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,686 47838481070400 vos-1.10.4CadcCache.open.192 Opening file /test.src.split.ms.contsub.chan/HISTORY/table.f0: isnew False: id 247351696
2014-09-04 12:52:28,686 47838481070400 vos-1.10.4CadcCache.readData.1174 Getting data: 0 0 None
2014-09-04 12:52:28,686 47838639360320 vos-1.10.4vofs.readFromBacking.79 reading range: None
2014-09-04 12:52:28,686 47838639360320 vos-1.10.4vofs.readFromBacking.83 Opening a new vo file on /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,687 47838639360320 vos-1.10.4vos.getNode.1095 Getting node /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,687 47838639360320 vos-1.10.4vos.getNodeURL.1142 Getting URL for: vos://cadc.nrc.ca!vospace/dcolombo/SextansA_test/test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,687 47838639360320 vos-1.10.4vos.getNodeURL.1152 do_shortcut=1 method=GET view=data
2014-09-04 12:52:28,687 47838639360320 vos-1.10.4vos.getNodeURL.1167 parts: [scheme: vos, netloc: cadc.nrc.ca!vospace, path: /dcolombo/SextansA_test/test.src.split.ms.contsub.chan/HISTORY/table.f0]
2014-09-04 12:52:28,687 47838639360320 vos-1.10.4vos.getNodeURL.1196 URL: https://www.canfar.phys.uvic.ca
2014-09-04 12:52:28,687 47838639360320 vos-1.10.4vos.get_connection.102 Trying to connect to https://www.canfar.phys.uvic.ca using /home/dcolombo/.ssl/cadcproxy.pem
2014-09-04 12:52:28,687 47838639360320 vos-1.10.4vos.get_connection.124 Opening the connection
2014-09-04 12:52:28,850 47838639360320 vos-1.10.4vos.getNodeURL.1231 Sending short cut url: http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/transfer/bmVpZ2g9Q0FEQyZzaWdleHA9MjAxNDA5MDQyMzUzMDQmYXJjPVZPU3BhYyZpZD0yMTg3MDM5MTcmdXNlcj1kY29sb21ibyZucz1kY29sb21ibyUyRlNleHRhbnNBX3Rlc3QlMkZ0ZXN0LnNyYy5zcGxpdC5tcy5jb250c3ViLmNoYW4lMkZISVNUT1JZJmF1dGhDb2RlPTAmcnVuSUQ9YTg0a2IyYjZwdHdyZDN2ZyZ1cmw9aHR0cCUzQSUyRiUyRnN0b3I2LTA4JTNBODg4OCUyRmFyY2hpdmUlMkY4MWI3ZjExOSUyRnZvc3BhYyUyRmRlZmF1bHQlMkZkZWZhdWx0XzA2NjglMkYwMCUyRjAwJTJGMTglMkYyMTg3MDM5MTclMjUzQiUyNTNCMjE2ODk3NTkwJnVybD1odHRwJTNBJTJGJTJGc3RvcjYtMDclM0E4ODg4JTJGYXJjaGl2ZSUyRjkyYTRhNDg5JTJGdm9zcGFjJTJGZGVmYXVsdCUyRmRlZmF1bHRfMDY2OCUyRjAwJTJGMDAlMkYxOCUyRjIxODcwMzkxNyUyNTNCJTI1M0IyMTY4OTc1OTAmc2VydmljZVVybD1odHRwJTNBJTJGJTJGd3d3LmNhbmZhci5waHlzLnV2aWMuY2ElMkZ2b3NwYWNlJTJGc3luY3RyYW5zJTNGVEFSR0VUJTNEdm9zJTI1M0ElMjUyRiUyNTJGY2FkYy5ucmMuY2ElMjUyMXZvc3BhY2UlMjUyRmRjb2xvbWJvJTI1MkZTZXh0YW5zQV90ZXN0JTI1MkZ0ZXN0LnNyYy5zcGxpdC5tcy5jb250c3ViLmNoYW4lMjUyRkhJU1RPUlklMjUyRnRhYmxlLmYwJTI2RElSRUNUSU9OJTNEcHVsbEZyb21Wb1NwYWNlJTI2UFJPVE9DT0wlM0Rpdm8lMjUzQSUyNTJGJTI1MkZpdm9hLm5ldCUyNTJGdm9zcGFjZSUyNTJGY29yZSUyNTIzaHR0cGdldCUyNnZpZXclM0RkYXRh-ZmlkPXRhYmxlLmYwJmNsbj0xMDg3NDg4JmNtZD05ZjUxOTU1MmE5YjVhMmQyMTA4MjM2YTRmOTJiZTk5NiZsbWQ9MTQwODQ3NjMwNDY4Nw==-3a9a0d07f3afca1d0bc0ef6ca7e3860f/table.f0
2014-09-04 12:52:28,851 47838639360320 vos-1.10.4vos.open.750 Opening http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/transfer/bmVpZ2g9Q0FEQyZzaWdleHA9MjAxNDA5MDQyMzUzMDQmYXJjPVZPU3BhYyZpZD0yMTg3MDM5MTcmdXNlcj1kY29sb21ibyZucz1kY29sb21ibyUyRlNleHRhbnNBX3Rlc3QlMkZ0ZXN0LnNyYy5zcGxpdC5tcy5jb250c3ViLmNoYW4lMkZISVNUT1JZJmF1dGhDb2RlPTAmcnVuSUQ9YTg0a2IyYjZwdHdyZDN2ZyZ1cmw9aHR0cCUzQSUyRiUyRnN0b3I2LTA4JTNBODg4OCUyRmFyY2hpdmUlMkY4MWI3ZjExOSUyRnZvc3BhYyUyRmRlZmF1bHQlMkZkZWZhdWx0XzA2NjglMkYwMCUyRjAwJTJGMTglMkYyMTg3MDM5MTclMjUzQiUyNTNCMjE2ODk3NTkwJnVybD1odHRwJTNBJTJGJTJGc3RvcjYtMDclM0E4ODg4JTJGYXJjaGl2ZSUyRjkyYTRhNDg5JTJGdm9zcGFjJTJGZGVmYXVsdCUyRmRlZmF1bHRfMDY2OCUyRjAwJTJGMDAlMkYxOCUyRjIxODcwMzkxNyUyNTNCJTI1M0IyMTY4OTc1OTAmc2VydmljZVVybD1odHRwJTNBJTJGJTJGd3d3LmNhbmZhci5waHlzLnV2aWMuY2ElMkZ2b3NwYWNlJTJGc3luY3RyYW5zJTNGVEFSR0VUJTNEdm9zJTI1M0ElMjUyRiUyNTJGY2FkYy5ucmMuY2ElMjUyMXZvc3BhY2UlMjUyRmRjb2xvbWJvJTI1MkZTZXh0YW5zQV90ZXN0JTI1MkZ0ZXN0LnNyYy5zcGxpdC5tcy5jb250c3ViLmNoYW4lMjUyRkhJU1RPUlklMjUyRnRhYmxlLmYwJTI2RElSRUNUSU9OJTNEcHVsbEZyb21Wb1NwYWNlJTI2UFJPVE9DT0wlM0Rpdm8lMjUzQSUyNTJGJTI1MkZpdm9hLm5ldCUyNTJGdm9zcGFjZSUyNTJGY29yZSUyNTIzaHR0cGdldCUyNnZpZXclM0RkYXRh-ZmlkPXRhYmxlLmYwJmNsbj0xMDg3NDg4JmNtZD05ZjUxOTU1MmE5YjVhMmQyMTA4MjM2YTRmOTJiZTk5NiZsbWQ9MTQwODQ3NjMwNDY4Nw==-3a9a0d07f3afca1d0bc0ef6ca7e3860f/table.f0 (GET)
2014-09-04 12:52:28,851 47838639360320 vos-1.10.4vos.get_connection.102 Trying to connect to http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca using /home/dcolombo/.ssl/cadcproxy.pem
2014-09-04 12:52:28,851 47838639360320 vos-1.10.4vos.get_connection.124 Opening the connection
2014-09-04 12:52:28,855 47838639360320 vos-1.10.4vofs.readFromBacking.95 reading from http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/transfer/bmVpZ2g9Q0FEQyZzaWdleHA9MjAxNDA5MDQyMzUzMDQmYXJjPVZPU3BhYyZpZD0yMTg3MDM5MTcmdXNlcj1kY29sb21ibyZucz1kY29sb21ibyUyRlNleHRhbnNBX3Rlc3QlMkZ0ZXN0LnNyYy5zcGxpdC5tcy5jb250c3ViLmNoYW4lMkZISVNUT1JZJmF1dGhDb2RlPTAmcnVuSUQ9YTg0a2IyYjZwdHdyZDN2ZyZ1cmw9aHR0cCUzQSUyRiUyRnN0b3I2LTA4JTNBODg4OCUyRmFyY2hpdmUlMkY4MWI3ZjExOSUyRnZvc3BhYyUyRmRlZmF1bHQlMkZkZWZhdWx0XzA2NjglMkYwMCUyRjAwJTJGMTglMkYyMTg3MDM5MTclMjUzQiUyNTNCMjE2ODk3NTkwJnVybD1odHRwJTNBJTJGJTJGc3RvcjYtMDclM0E4ODg4JTJGYXJjaGl2ZSUyRjkyYTRhNDg5JTJGdm9zcGFjJTJGZGVmYXVsdCUyRmRlZmF1bHRfMDY2OCUyRjAwJTJGMDAlMkYxOCUyRjIxODcwMzkxNyUyNTNCJTI1M0IyMTY4OTc1OTAmc2VydmljZVVybD1odHRwJTNBJTJGJTJGd3d3LmNhbmZhci5waHlzLnV2aWMuY2ElMkZ2b3NwYWNlJTJGc3luY3RyYW5zJTNGVEFSR0VUJTNEdm9zJTI1M0ElMjUyRiUyNTJGY2FkYy5ucmMuY2ElMjUyMXZvc3BhY2UlMjUyRmRjb2xvbWJvJTI1MkZTZXh0YW5zQV90ZXN0JTI1MkZ0ZXN0LnNyYy5zcGxpdC5tcy5jb250c3ViLmNoYW4lMjUyRkhJU1RPUlklMjUyRnRhYmxlLmYwJTI2RElSRUNUSU9OJTNEcHVsbEZyb21Wb1NwYWNlJTI2UFJPVE9DT0wlM0Rpdm8lMjUzQSUyNTJGJTI1MkZpdm9hLm5ldCUyNTJGdm9zcGFjZSUyNTJGY29yZSUyNTIzaHR0cGdldCUyNnZpZXclM0RkYXRh-ZmlkPXRhYmxlLmYwJmNsbj0xMDg3NDg4JmNtZD05ZjUxOTU1MmE5YjVhMmQyMTA4MjM2YTRmOTJiZTk5NiZsbWQ9MTQwODQ3NjMwNDY4Nw==-3a9a0d07f3afca1d0bc0ef6ca7e3860f/table.f0
2014-09-04 12:52:28,894 47838639360320 vos-1.10.4vos.checkstatus.727 status 200 for URL http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/transfer/bmVpZ2g9Q0FEQyZzaWdleHA9MjAxNDA5MDQyMzUzMDQmYXJjPVZPU3BhYyZpZD0yMTg3MDM5MTcmdXNlcj1kY29sb21ibyZucz1kY29sb21ibyUyRlNleHRhbnNBX3Rlc3QlMkZ0ZXN0LnNyYy5zcGxpdC5tcy5jb250c3ViLmNoYW4lMkZISVNUT1JZJmF1dGhDb2RlPTAmcnVuSUQ9YTg0a2IyYjZwdHdyZDN2ZyZ1cmw9aHR0cCUzQSUyRiUyRnN0b3I2LTA4JTNBODg4OCUyRmFyY2hpdmUlMkY4MWI3ZjExOSUyRnZvc3BhYyUyRmRlZmF1bHQlMkZkZWZhdWx0XzA2NjglMkYwMCUyRjAwJTJGMTglMkYyMTg3MDM5MTclMjUzQiUyNTNCMjE2ODk3NTkwJnVybD1odHRwJTNBJTJGJTJGc3RvcjYtMDclM0E4ODg4JTJGYXJjaGl2ZSUyRjkyYTRhNDg5JTJGdm9zcGFjJTJGZGVmYXVsdCUyRmRlZmF1bHRfMDY2OCUyRjAwJTJGMDAlMkYxOCUyRjIxODcwMzkxNyUyNTNCJTI1M0IyMTY4OTc1OTAmc2VydmljZVVybD1odHRwJTNBJTJGJTJGd3d3LmNhbmZhci5waHlzLnV2aWMuY2ElMkZ2b3NwYWNlJTJGc3luY3RyYW5zJTNGVEFSR0VUJTNEdm9zJTI1M0ElMjUyRiUyNTJGY2FkYy5ucmMuY2ElMjUyMXZvc3BhY2UlMjUyRmRjb2xvbWJvJTI1MkZTZXh0YW5zQV90ZXN0JTI1MkZ0ZXN0LnNyYy5zcGxpdC5tcy5jb250c3ViLmNoYW4lMjUyRkhJU1RPUlklMjUyRnRhYmxlLmYwJTI2RElSRUNUSU9OJTNEcHVsbEZyb21Wb1NwYWNlJTI2UFJPVE9DT0wlM0Rpdm8lMjUzQSUyNTJGJTI1MkZpdm9hLm5ldCUyNTJGdm9zcGFjZSUyNTJGY29yZSUyNTIzaHR0cGdldCUyNnZpZXclM0RkYXRh-ZmlkPXRhYmxlLmYwJmNsbj0xMDg3NDg4JmNtZD05ZjUxOTU1MmE5YjVhMmQyMTA4MjM2YTRmOTJiZTk5NiZsbWQ9MTQwODQ3NjMwNDY4Nw==-3a9a0d07f3afca1d0bc0ef6ca7e3860f/table.f0
2014-09-04 12:52:28,894 47838639360320 vos-1.10.4CadcCache.setHeader.779 size: 1087488 md5: 9f519552a9b5a2d2108236a4f92be996
2014-09-04 12:52:28,894 47838639360320 vos-1.10.4vofs.readFromBacking.121 Writing: 16384 bytes at 0 to cache for /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,894 47838639360320 vos-1.10.4CadcCache.writeToCache.702 reading to cache aborted for /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,895 47838639360320 vos-1.10.4vofs.readFromBacking.137 Wrote: 0 bytes to cache for /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,895 47838639360320 vos-1.10.4CadcCache.execute.1237 setFullyCached? True 0 None 1087488
2014-09-04 12:52:28,895 47838639360320 vos-1.10.4CadcCache.execute.1267 read thread finished
2014-09-04 12:52:28,899 47838481070400 vos-1.10.4CadcCache.setHeader.779 size: 1087488 md5: 9f519552a9b5a2d2108236a4f92be996
2014-09-04 12:52:28,899 47838481070400 vos-1.10.4fuse.call.854 <- open 247350480
2014-09-04 12:52:28,899 47838618380608 vos-1.10.4fuse.call.845 -> read /test.src.split.ms.contsub.chan/HISTORY/table.f0 (16384L, 0, 247350480L)
2014-09-04 12:52:28,899 47838618380608 vos-1.10.4vofs.read.560 reading range: /test.src.split.ms.contsub.chan/HISTORY/table.f0 16384 0 247350480
2014-09-04 12:52:28,900 47838618380608 vos-1.10.4CadcCache.read.1022 reading 16384 bytes at 0
2014-09-04 12:52:28,900 47838618380608 vos-1.10.4fuse.call.854 <- read <ctypes.c_char_Array_16384 object at 0xea5d200>
2014-09-04 12:52:28,900 47838491560256 vos-1.10.4fuse.call.845 -> getattr /test.src.split.ms.contsub.chan/HISTORY/table.f0 (None,)
2014-09-04 12:52:28,900 47838491560256 vos-1.10.4vofs.getattr.413 getting attributes of /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,900 47838491560256 vos-1.10.4CadcCache.getAttr.520 gettattr /test.src.split.ms.contsub.chan/HISTORY/table.f0:
2014-09-04 12:52:28,900 47838491560256 vos-1.10.4vofs.getNode.390 force? -> False path -> /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,900 47838491560256 vos-1.10.4vofs.getNode.394 requesting node /test.src.split.ms.contsub.chan/HISTORY/table.f0 from VOSpace
2014-09-04 12:52:28,901 47838491560256 vos-1.10.4vos.getNode.1095 Getting node /test.src.split.ms.contsub.chan/HISTORY/table.f0
2014-09-04 12:52:28,901 47838491560256 vos-1.10.4fuse.call.854 <- getattr {'st_ctime': 1408472704.0, 'st_mtime': 1408472704.0, 'st_blocks': 2124, 'st_gid': 500, 'st_nlink': 1, 'st_mode': 33221, 'st_size': 1087488, 'st_atime': 1409860276.4991741, 'st_uid': 500}
2014-09-04 12:52:28,901 47838481070400 vos-1.10.4fuse.call.845 -> read /test.src.split.ms.contsub.chan/HISTORY/table.f0 (8192L, 1081344, 247350480L)
2014-09-04 12:52:28,901 47838481070400 vos-1.10.4vofs.read.560 reading range: /test.src.split.ms.contsub.chan/HISTORY/table.f0 8192 1081344 247350480
2014-09-04 12:52:28,901 47838481070400 vos-1.10.4CadcCache.read.1022 reading 8192 bytes at 1081344
2014-09-04 12:52:28,901 47838481070400 vos-1.10.4fuse.call.854 <- read <ctypes.c_char_Array_6144 object at 0xea5d050>
2014-09-04 12:52:28,902 47838618380608 vos-1.10.4fuse.call.845 -> write /test.src.split.ms.contsub.chan/HISTORY/table.f0 (<vos.fuse.LP_c_byte object at 0xe746a70>, 2816L, 1081856, 247350480L)
2014-09-04 12:52:28,902 47838618380608 vos-1.10.4vofs.write.772 /test.src.split.ms.contsub.chan/HISTORY/table.f0 -> <vos.vofs.HandleWrapper object at 0xebe44d0>
2014-09-04 12:52:28,902 47838618380608 vos-1.10.4vofs.write.773 1081856 --> 1084672
2014-09-04 12:52:28,902 47838618380608 vos-1.10.4CadcCache.write.972 writting 2816 bytes at 1081856 to 8
2014-09-04 12:52:28,902 47838618380608 vos-1.10.4logExceptions.wrapper.15 Exception throw: <type 'exceptions.ValueError'> index range error
2014-09-04 12:52:28,903 47838618380608 vos-1.10.4logExceptions.wrapper.16 Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/vos-1.10.4-py2.6.egg/vos/logExceptions.py", line 13, in wrapper
return func(_args, *_kwds)
File "/usr/lib/python2.6/site-packages/vos-1.10.4-py2.6.egg/vos/vofs.py", line 776, in write
return fh.cacheFileHandle.write(data, size, offset)
File "/usr/lib/python2.6/site-packages/vos-1.10.4-py2.6.egg/vos/CadcCache.py", line 995, in write
self.makeCached(offset, self.fileSize)
File "/usr/lib/python2.6/site-packages/vos-1.10.4-py2.6.egg/vos/CadcCache.py", line 1065, in makeCached
requiredRange = self.metaData.getRange(firstBlock, lastBlock)
File "/usr/lib/python2.6/site-packages/vos-1.10.4-py2.6.egg/vos/CacheMetaData.py", line 100, in getRange
if self.bitmap[i] == 0:
File "/usr/lib/python2.6/site-packages/vos-1.10.4-py2.6.egg/vos/BitVector.py", line 1466, in _getbit
raise ValueError( "index range error" )
ValueError: index range error

vos vrm *(asterisk) expansion command-delete multiple files

VOS' vrm command does not seem to recognize the * (asterisk) as expansion. It would be useful to add e.g., 'vrm *.dat' functionality in order to be able to delete multiple files with the same extension.

vcp should have a progress bar.

Users would like a progress bar (like scp) when vcp is run in verbose mode.

make vsync bi-directional.

Hi - (from Michael Balogh)
vcp has the functionality I want: specifically preserving links and
bidirectional copying, which vsync doesn't do. But it is really slow -
it seems that even if only one file has changed it would take vcp a long
time to go through checking everything in the directory structure.
Compared to rsync which will sync changed files really quickly. Not sure
if vsync works that fast or not; if so then adding bidrectional copying
and link preservation would really help us.

Move pymop to use vospace instead of local file system.

We need to have pymop read/write/lock the .astrom files directly from VOSpace via the vos.Client and vos.File objects. Eventually we will also want to cache the actions on the VOSpace objects locally (somehow) so that we can do work while 'offline'.

No more local files charlie brown. 🎱

mv of open file not handled correctly.

When a file is that has an open fh and a 'mv' action is performed in a separate thread then the mv action should mv the file to the new location and actions on the fh should be on the new file. Currently the content of a file that was open for write during the mv is lost. Here is a test scenario.

Process 1:
python

f=open('/tmp/vospace/dir/test.txt', 'w')
f.write('This is content\n")
f.flush()

Process 2:

mv /tmp/vospace/dir/test.txt /tmp/vospace/dir/new_name.txt

Process 1:

f.write("more content\n")
f.close()

Now, the file new_name.txt should contain:

This is content
more content

But only contains

more content

The content that writen to the open file handle prior to the file mv has been lost. I think this is because the file rename parts are not correct in the mountvofs layer.

vmv returns the wrong error when destination file exists

credentials in vos.Client

voc.Client has an internal mechanism for auto-discoverying of user credentials. That mechanism interferes in library mode where the client already has its own user credentials. A potential solution would be to add a Subject argument to the Client constructor which, if present, disables the auto-discovery mechanism.

ERROR vls bad handshake

I installed the vos package using pip earlier today.
I have run getCert to get certificates in order; that worked fine.
But upon running vls or vcp, I get an error message.

$ vls vos:OSSOS/
ERROR vls bad handshake: Error([('SSL routines', 'SSL3_READ_BYTES', 'tlsv1 alert unknown ca')],)

I get the same error message when trying to use vcp.
I have openssl and pyopenssl installed and up to date.
I have tried googling this error message and subparts of it, but I'm not finding anything that looks like this, so I suspect it is an error from vos and not from openssl.

Any suggestions to possible solutions?

Restart of download on links redownloads existing files.

I would like to note that the vcp command does not check properly for already existing data with the elixir-directory setup - probably because of the links. If I restart the command

vcp -L vos:cfis/elixir .

it redownloads data that I already have. With the pitcairn directory, this works just fine. I wrote a script to only get the data I need and hence this is no issue for me. But I guess that other people will run into it.

mountvofs not working in batch jobs in latest release

The line:
mountvofs --vospace vos:MWSynthesis/ --mountpoint ${TMPDIR}/vos --cache_dir ${TMPDIR}/vos_cache
no longer works when used in a batch job and produces no error in the log. When checked, the mountpoint is empty. getCert runs without errors earlier in the script.

It does work in the batch environment when using an older release. It also works fine when running in an interactive environment (with the newest release).

.add_props(node) failing on single item?

Occurring under vos-2.0.rc2, but not under vos-2.0a0:

File "/Users/bannisterm/Dropbox/OSSOS/ossos-pipeline/src/ossos-pipeline/ossos/storage.py", line 1080, in set_property
vospace.add_props(node)
File "/Users/bannisterm/lib/python2.7/site-packages/vos/vos.py", line 1881, in add_props
url = urls.pop(0)
AttributeError: 'str' object has no attribute 'pop'

vcat with cutouts gives fail

When attempting to use vcat to cat out a cutout of a file like so:

vcat vos:OSSOS/dbimages/1667726/1667726p.fits[1][1:1,1:1] --vos-debug --debug

I get this error message:

DEBUG 2018-02-07 15:26:08,717 140735500096320 vos-3.0 sjisprober.feed.61 SHIFT_JIS Japanese prober hit error at byte 63361

Which is weird.

vcp works fine. But I expected vcat should also work?

Check connection succeeds before using result from VOSPace service.

vos:ngvs/laura/Isophotes/PILOT/NGVSJ12:30:15.76+12:59:54.0/NGVSJ12:30:15.76+12:59:54.0_U.fits
/ephemeral/scratch/NGVSJ12:30:15.76+12:59:54.0_u.fits
Traceback (most recent call last):
File "/home/jroediger/scripts/reduce.py", line 37, in
c.copy("vos:ngvs/laura/Isophotes/PILOT/"+id+"/"+id+""+band_uc+".fits",work_dir+id+""+band_lc+".fits")
File "/usr/lib/python2.6/site-packages/vos-1.12.3-py2.6.egg/vos/vos.py",
line 1184, in copy
buf = fin.read(BUFSIZE)
File "/usr/lib/python2.6/site-packages/vos-1.12.3-py2.6.egg/vos/vos.py",
line 881, in read
if self.resp.status == 416:
AttributeError: VOFile instance has no attribute 'resp'

reported by [email protected]

Readme should use markdown format

If you want the project to look nice on github you should switch the readme to use markdown format.

https://help.github.com/articles/github-flavored-markdown

Recursive option for vls

Request to have a -R option added to vls. The current work around for recursive listing is to mount the file system and use the find command.

tmp file vuln

https://pypi.python.org/pypi/vos

vos is a set of python modules and scripts to enable access to VOSpace.

The default installation of vos is tuned for accessing the VOSpace provided by the Canadian Advanced Network For Astronomical Research (CANFAR)

VOSpace is a Distributed Cloud storage service for use in Astronomy.

vos-1.10.4/vos/md5_cache.py

import sqlite3, logging
READBUF = 8192

class MD5_Cache:

def __init__(self, cache_db="/tmp/#vos_cached.db#"):
    """Setup the sqlDB that will contain the cache table"""
    self.cache_db = cache_db

    ## initialize the md5Cache db
    sqlConn = sqlite3.connect(self.cache_db)
    with sqlConn:
        sqlConn.execute("create table if not exists md5_cache (fname text PRIMARY KEY NOT NULL , md5 text, st_size int, st_mtime int)")
    ## build cache lookup if doesn't already exists

also posted to oss-security to get a CVE #

fhead doesn't work.

Need to add the 'fhead' ability to vcp and vcat commands. fhead is part of the commands that /data understands some some effort might be needed to get fhead to work on VOSpace commands.

This is a high priority item.

Problem with re-mount after bad-address error.

Chrisitan, Pavel, Sam and I spent 1.5hours yesterday to diagnose a tricky and somewhat
illusive issue with
the VOspace mount. We are using it to have our codes access the data on the VOspace
and read it in to
post-process. Everything worked fine until we hit a particular file (the 15th in a series of
approximately 25).
Initially we thought the file was corrupt. When testing, we were not able to do h5ls or cp
("Bad address") the
file from the linux server in our department. Sam had the same behaviour for this file on his
Mac laptop, however,
I was able to do h5ls on my Mac laptop (both through our respective CADC mounts). We
remounted on the linux
server once after deleting the cache which did not help immediately. Eventually after a lot of
testing and checking
we did another remount and then were able to copy the file in question, do h5ls and h5diff
(to a backup copy on our
system) and finally proceed with the post-processing of the file. At that time Sam was able
to see the file correctly
again, without changing his mount.

Admittedly this problem is maybe difficult to diagnose because I am not sure we can
reproduce the problem.
But we have alerted others and we will keep an eye on this and report additional instances
of this problem.

Trap certificate expired warning t give a clear error

When your proxy cert is expired you get way to scarry of a warning:

[igable@elephant junk]$ vcp vos:sfabbro/vostest/1235593p.ccd06 1235593p.ccd06
Traceback (most recent call last):
  File "/home/igable/code/cadc/bin/vcp", line 7, in <module>
    execfile(__file__)
  File "/home/igable/code/cadc/cadcVOFS/vcp", line 103, in <module>
    if not access(source,os.R_OK):
  File "/home/igable/code/cadc/cadcVOFS/vcp", line 12, in access
    return client.access(filename,mode)
  File "/home/igable/code/cadc/cadcVOFS/vofs/vos.py", line 811, in access
    return not self.status(uri,code=[404])
  File "/home/igable/code/cadc/cadcVOFS/vofs/vos.py", line 819, in status
    file=self.open(uri,view='data',head=True)
  File "/home/igable/code/cadc/cadcVOFS/vofs/vos.py", line 773, in open
    return VOFile(URL,self.conn,method=method,size=size)
  File "/home/igable/code/cadc/cadcVOFS/vofs/vos.py", line 474, in __init__
    self.open(URL,method)
  File "/home/igable/code/cadc/cadcVOFS/vofs/vos.py", line 514, in open
    self.httpCon.connect()
  File "/usr/local/lib/python2.7/httplib.py", line 1144, in connect
    self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
  File "/usr/local/lib/python2.7/ssl.py", line 344, in wrap_socket
    ciphers=ciphers)
  File "/usr/local/lib/python2.7/ssl.py", line 121, in __init__
    self.do_handshake()
  File "/usr/local/lib/python2.7/ssl.py", line 283, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [Errno 1] _ssl.c:499: error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired

We could trap this earlier.

vsync fails when the parent directory of the destination is not writeable

On certain occasions, vsync checks whether a directory exists on the destination by trying to create it first and trapping the NodeAlreadyExists error. However, the parent director in the destination might not be writable by user in which case a PermissionDenied error is received instead.

Problem is in vsync.mkdirs where client.mkdir is called.

Below, it's an excerpt of a command with this error occurring.

DEBUG 2017-11-08 12:14:32,915 140688944195328 vos-3.0.2 vsync.mkdirs.201 vos:jkavelaars/TNORecon/pitcairn []
DEBUG 2017-11-08 12:14:32,924 140688944195328 vos-3.0.2 ws.send.430 Sending request <PreparedRequest [PUT]> to server.
Traceback (most recent call last):
File "/usr/local/bin/vsync", line 9, in
load_entry_point('vos==3.0.2', 'console_scripts', 'vsync')()
File "/usr/local/lib/python2.7/site-packages/vos/commands/vsync.py", line 325, in vsync
copy(filename, this_root)
File "/usr/local/lib/python2.7/site-packages/vos/commands/vsync.py", line 243, in copy
mkdirs(dirname)
File "/usr/local/lib/python2.7/site-packages/vos/commands/vsync.py", line 209, in mkdirs
client.mkdir(directory)
File "/usr/local/lib/python2.7/site-packages/vos/vos.py", line 2193, in mkdir
response = self.conn.session.put(url, data=str(node))
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 521, in put
return self.request('PUT', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/cadcutils/net/ws.py", line 435, in send
self.check_status(response)
File "/usr/local/lib/python2.7/site-packages/cadcutils/net/ws.py", line 484, in check_status
raise exceptions.UnauthorizedException(orig_exception=e)
cadcutils.exceptions.UnauthorizedException: PermissionDenied: Write permission denied on vos://cadc.nrc.ca!vospace/jkavelaars/TNORecon

VOFS doesn't notice directory moves.

Doing mv /vospace/dir1 /vospace/dir2 confusing VOFS as ls /vospace/dir1 will show that contents that were dir1 prior to the mv command. This appears to be a case of directory caching.

Add progressbar for `vcp`

I just started using the tools, and it would be very helpful to have a progressbar and ETA for vcp. I think this could be done easily here. We have a simple setup in astroquery that uses astropy's progressbar, but there are other good alternatives available.

cantop crashes

Dear helpdesk,

cantop (part of cadcVOFS) will crash if the terminal is resized, with the following error:

ERROR:astropy:error: addstr() returned ERR
ERROR: error: addstr() returned ERR [vos.cantop]
Traceback (most recent call last):
File "/Users/michele/Library/Enthought/Canopy_64bit/User/bin/cantop", line 15, in
cantop.redraw()
File "/Users/michele/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/vos/cantop.py", line 90, in redraw
self.main_window.addstr(self.get_status())
_curses.error: addstr() returned ERR

Cheers

sudo behaviour causes trouble

When a user runs a 'vos' command using 'sudo' and there isn't a resource-caps file or directory already one is created, but with 'root' as the owner. Subsequent calls to vos commands as normal user fail due to privilege issue.

Two things: The error is not caught so the message is ugly and we should avoid this permission issue during sudo operations, but I'm not sure how. The following is the current error failure:

Traceback (most recent call last):
File "/usr/local/bin/vcp", line 9, in
load_entry_point('vos==3.0.3', 'console_scripts', 'vcp')()
File "/usr/local/lib/python2.7/dist-packages/vos/commands/vcp.py", line 97, in vcp
transfer_shortcut=args.quick)
File "/usr/local/lib/python2.7/dist-packages/vos/vos.py", line 1333, in init
http_debug=http_debug)
File "/usr/local/lib/python2.7/dist-packages/vos/vos.py", line 224, in init
session_headers=session_headers)
File "/usr/local/lib/python2.7/dist-packages/cadcutils/net/ws.py", line 243, in init
base_url = self.caps.get_access_url(SERVICE_AVAILABILITY_ID)
File "/usr/local/lib/python2.7/dist-packages/cadcutils/net/ws.py", line 555, in get_access_url
caps = self._get_content(self.caps_file, self._get_capability_url(), self.last_capstime)
File "/usr/local/lib/python2.7/dist-packages/cadcutils/net/ws.py", line 626, in _get_capability_url
reg = self._get_content(self.reg_file, registry_url, self.last_regtime)
File "/usr/local/lib/python2.7/dist-packages/cadcutils/net/ws.py", line 593, in _get_content
with open(resource_file, 'w') as f:
IOError: [Errno 13] Permission denied: u'/home/swallstrom/.config/cadc-registry/resource-caps'

feature request: vcp creates a directory by default

it would save a few commands for the user if vcp creates a directory on vospace when a directory is passed, similar to "scp.-r"

Current workaround:

vmkdir vos:myvospace/mydir
vcp mydir vos:myvospace/mydir

Desired behaviour:

vcp mydir vos:myvospace : copy the entire directory mydir to vos:myvospace/mydir

I'm not completely convinced of it, and hoping proper recursivity will be implemented in vospace so we can do "vcp -r" and have exact behaviour as scp.

vcp creates a destination directory incorrectly.

The goal is to have vcp mimmic POSIX cp where possible. Currently the command:

vcp ./source/ vos:dest/

will create ./dest/source but it 'cp' places the contents of 'source' into 'dest' without a sub-directory.

vcp should mimmic cp

Thanks to @mtbannister for pointing this out.

vos does not work on public vospaces with expired proxy certs

How to reproduce:

Install vos-1.12.2
Somehow get an expired proxy, or wait 1.1 day with getCert --daysValid=1
List a public VOSpace:

vls vos:sfabbro
 vos: [Errno 1] _ssl.c:499: error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired
vls: _ssl.c:499: error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired

Realize when you remove .ssl/cadcproxy.pem that it works.

Unit test occasionally hangs in Travis

One of the tests occasionally hangs while running on Travis. The output of the failed task is:

'vofs/tests/test_cadc_cache.py ..........................
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated'

Must figure out which is the test in question before attempting to fix it.

vofs can't deal with negative directory sizes

Some oddities in the CADC VOSpace implementation can result in negative directory sizes. vofs reports these as very large directory sizes. It would be better if it reported them as zero.

vrm on a container node does not fail (but it doesn't succeed either)

vrm on a container node does nothing and the return status is 0. The correct behaviour is to report that the target is a container node and fail.

Note that vrmdir on data container works correctly.

vostools needs a vtail and vhead

The vostools set is missing something that serves the function of head/tail commands. They would be easy implement as per the gist below which is a concept implementation of vtail:

#!/usr/bin/env python
from vos import Client
import sys
node_name = sys.argv.pop()
number_of_bytes = 100
if len(sys.argv) > 1:
number_of_bytes = int(sys.argv.pop())
c = Client()
node = c.get_node(node_name)
sys.stdout.write(c.open(node.uri, view='data', byte_range="bytes={}-{}".format(int(node.props['length'])-number_of_bytes,int(node.props['length']))).read())

touch command does not work well with vofs

Problem seems to be related to flushing files of size 0 to the server that takes excessive time. I remember that there is a work around for pushing files of size 0 in vos. Isn't vofs using that mechanism?

To reproduce it just run the touch command on the vofs mount directory and follow the logs.

unique: 4, opcode: LOOKUP (1), nodeid: 2, insize: 48, pid: 28078
LOOKUP /sometest/abc.txt
getattr /sometest/abc.txt
DEBUG 2017-08-18 16:05:58,826 123145506111488 vos-3.0.0b6 vofs.call.265 -> getattr ('/sometest/abc.txt', None)
DEBUG 2017-08-18 16:05:58,826 123145506111488 vos-3.0.0b6 vofs.get_node.410 requesting node /sometest/abc.txt from VOSpace. Force: False
DEBUG 2017-08-18 16:05:58,831 123145506111488 vos-3.0.0b6 ws.send.420 Sending request <PreparedRequest [GET]> to server.
DEBUG 2017-08-18 16:05:58,850 123145506111488 vos-3.0.0b6 ws.send.420 Sending request <PreparedRequest [GET]> to server.
DEBUG 2017-08-18 16:05:58,871 123145506111488 vos-3.0.0b6 ws._get_url.328 Resolved URL: https://www.canfar.phys.uvic.ca/vospace/nodes
DEBUG 2017-08-18 16:05:58,873 123145506111488 vos-3.0.0b6 ws.send.420 Sending request <PreparedRequest [GET]> to server.
DEBUG 2017-08-18 16:05:58,969 123145506111488 vos-3.0.0b6 vofs.call.279 NodeNotFound: vos://cadc.nrc.ca!vospace/CADCRegtest1/atest/sometest/abc.txt

File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/fuse.py", line 495, in _wrapper
return func(*args, **kwargs) or 0
File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/fuse.py", line 511, in getattr
return self.fgetattr(path, buf, None)
File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/fuse.py", line 759, in fgetattr
attrs = self.operations('getattr', self._decode_optional_path(path), fh)
File "/Users/adriand/Documents/work/github/vostools/vofs/vofs/vofs.py", line 270, in call
ret = getattr(self, op)(*args)
File "/Users/adriand/Documents/work/github/vostools/vofs/vofs/vofs.py", line 425, in getattr
return attr is not None and attr or self.getNode(path, limit=0, force=False).attr
File "/Users/adriand/Documents/work/github/vostools/vofs/vofs/vofs.py", line 411, in get_node
node = self.client.get_node(path, force=force, limit=limit)
File "/Users/adriand/Documents/work/github/vostools/vos/vos/vos.py", line 1581, in get_node
vo_xml_string = vo_fobj.read().decode('UTF-8')
File "/Users/adriand/Documents/work/github/vostools/vos/vos/vos.py", line 1005, in read
self.resp = self.connector.session.send(self.request, stream=True, verify=False)
File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/net/ws.py", line 425, in send
self.check_status(response)
File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/net/ws.py", line 472, in check_status
raise exceptions.NotFoundException(orig_exception=e)
File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/exceptions.py", line 134, in init
HttpException.init(self, msg, orig_exception)
File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/exceptions.py", line 98, in init
self._msg = '{}\n{}'.format(self._msg, ''.join(traceback.format_stack()))

DEBUG 2017-08-18 16:05:58,969 123145506111488 vos-3.0.0b6 vofs.call.280 <- getattr 'NodeNotFound: vos://cadc.nrc.ca!vospace/CADCRegtest1/atest/sometest/abc.txt\n\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/fuse.py", line 495, in _wrapper\n return func(*args, **kwargs) or 0\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/fuse.py", line 511, in getattr\n return self.fgetattr(path, buf, None)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/fuse.py", line 759, in fgetattr\n attrs = self.operations('getattr', self._decode_optional_path(path), fh)\n File "/Users/adriand/Documents/work/github/vostools/vofs/vofs/vofs.py", line 270, in call\n ret = getattr(self, op)(*args)\n File "/Users/adriand/Documents/work/github/vostools/vofs/vofs/vofs.py", line 425, in getattr\n return attr is not None and attr or self.getNode(path, limit=0, force=False).attr\n File "/Users/adriand/Documents/work/github/vostools/vofs/vofs/vofs.py", line 411, in get_node\n node = self.client.get_node(path, force=force, limit=limit)\n File "/Users/adriand/Documents/work/github/vostools/vos/vos/vos.py", line 1581, in get_node\n vo_xml_string = vo_fobj.read().decode('UTF-8')\n File "/Users/adriand/Documents/work/github/vostools/vos/vos/vos.py", line 1005, in read\n self.resp = self.connector.session.send(self.request, stream=True, verify=False)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/net/ws.py", line 425, in send\n self.check_status(response)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/net/ws.py", line 472, in check_status\n raise exceptions.NotFoundException(orig_exception=e)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/exceptions.py", line 134, in init\n HttpException.init(self, msg, orig_exception)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/exceptions.py", line 98, in init\n self._msg = '{}\n{}'.format(self._msg, ''.join(traceback.format_stack()))\n'
unique: 4, error: -2 (No such file or directory), outsize: 16
unique: 2, opcode: LOOKUP (1), nodeid: 2, insize: 48, pid: 28078
LOOKUP /sometest/abc.txt
getattr /sometest/abc.txt
DEBUG 2017-08-18 16:05:58,970 123145429950464 vos-3.0.0b6 vofs.call.265 -> getattr ('/sometest/abc.txt', None)
DEBUG 2017-08-18 16:05:58,970 123145429950464 vos-3.0.0b6 vofs.get_node.410 requesting node /sometest/abc.txt from VOSpace. Force: False
DEBUG 2017-08-18 16:05:58,970 123145429950464 vos-3.0.0b6 ws._get_url.328 Resolved URL: https://www.canfar.phys.uvic.ca/vospace/nodes
DEBUG 2017-08-18 16:05:58,973 123145429950464 vos-3.0.0b6 ws.send.420 Sending request <PreparedRequest [GET]> to server.
DEBUG 2017-08-18 16:05:59,063 123145429950464 vos-3.0.0b6 vofs.call.279 NodeNotFound: vos://cadc.nrc.ca!vospace/CADCRegtest1/atest/sometest/abc.txt

DEBUG 2017-08-18 16:05:59,063 123145429950464 vos-3.0.0b6 vofs.call.280 <- getattr 'NodeNotFound: vos://cadc.nrc.ca!vospace/CADCRegtest1/atest/sometest/abc.txt\n\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/fuse.py", line 495, in _wrapper\n return func(*args, **kwargs) or 0\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/fuse.py", line 511, in getattr\n return self.fgetattr(path, buf, None)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/fuse.py", line 759, in fgetattr\n attrs = self.operations('getattr', self._decode_optional_path(path), fh)\n File "/Users/adriand/Documents/work/github/vostools/vofs/vofs/vofs.py", line 270, in call\n ret = getattr(self, op)(*args)\n File "/Users/adriand/Documents/work/github/vostools/vofs/vofs/vofs.py", line 425, in getattr\n return attr is not None and attr or self.getNode(path, limit=0, force=False).attr\n File "/Users/adriand/Documents/work/github/vostools/vofs/vofs/vofs.py", line 411, in get_node\n node = self.client.get_node(path, force=force, limit=limit)\n File "/Users/adriand/Documents/work/github/vostools/vos/vos/vos.py", line 1581, in get_node\n vo_xml_string = vo_fobj.read().decode('UTF-8')\n File "/Users/adriand/Documents/work/github/vostools/vos/vos/vos.py", line 1005, in read\n self.resp = self.connector.session.send(self.request, stream=True, verify=False)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/net/ws.py", line 425, in send\n self.check_status(response)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/net/ws.py", line 472, in check_status\n raise exceptions.NotFoundException(orig_exception=e)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/exceptions.py", line 134, in init\n HttpException.init(self, msg, orig_exception)\n File "/Users/adriand/work/github/vostools/vofs/.tox/py34/lib/python3.4/site-packages/cadcutils/exceptions.py", line 98, in init\n self._msg = '{}\n{}'.format(self._msg, ''.join(traceback.format_stack()))\n'
unique: 2, error: -2 (No such file or directory), outsize: 16
unique: 9, opcode: GETATTR (3), nodeid: 2, insize: 56, pid: 28078
getattr /sometest
DEBUG 2017-08-18 16:05:59,064 123145487331328 vos-3.0.0b6 vofs.call.265 -> getattr ('/sometest', None)
DEBUG 2017-08-18 16:05:59,064 123145487331328 vos-3.0.0b6 vofs.get_node.410 requesting node /sometest from VOSpace. Force: False
DEBUG 2017-08-18 16:05:59,064 123145487331328 vos-3.0.0b6 vofs.get_node.412 Got node sometest
DEBUG 2017-08-18 16:05:59,064 123145487331328 vos-3.0.0b6 vofs.call.279 {'st_mtime': 1503086592.0, 'st_mode': 16837, 'st_uid': 501, 'st_ctime': 1503086592.0, 'st_atime': 1503097294.125806, 'st_blocks': 0, 'st_gid': 20, 'st_size': 6, 'st_nlink': 2}
DEBUG 2017-08-18 16:05:59,064 123145487331328 vos-3.0.0b6 vofs.call.280 <- getattr {'st_mtime': 1503086592.0, 'st_mode': 16837, 'st_uid': 501, 'st_ctime': 1503086592.0, 'st_atime': 1503097294.125806, 'st_blocks': 0, 'st_gid': 20, 'st_size': 6, 'st_nlink': 2}
unique: 9, success, outsize: 136
unique: 7, opcode: CREATE (35), nodeid: 2, insize: 64, pid: 28078
create flags: 0xa02 /sometest/abc.txt 0100644 umask=0000
DEBUG 2017-08-18 16:05:59,064 123145483575296 vos-3.0.0b6 vofs.call.265 -> create ('/sometest/abc.txt', 33188)
DEBUG 2017-08-18 16:05:59,065 123145483575296 vos-3.0.0b6 vofs.create.347 Creating a node: /sometest/abc.txt with flags 33188
DEBUG 2017-08-18 16:05:59,065 123145483575296 vos-3.0.0b6 ws._get_url.328 Resolved URL: https://www.canfar.phys.uvic.ca/vospace/nodes
DEBUG 2017-08-18 16:05:59,069 123145483575296 vos-3.0.0b6 ws.send.420 Sending request <PreparedRequest [PUT]> to server.
DEBUG 2017-08-18 16:05:59,174 123145483575296 vos-3.0.0b6 vofs.get_node.410 requesting node /sometest from VOSpace. Force: False
DEBUG 2017-08-18 16:05:59,174 123145483575296 vos-3.0.0b6 vofs.get_node.412 Got node sometest
DEBUG 2017-08-18 16:05:59,174 123145483575296 vos-3.0.0b6 vofs.open.456 Opening /sometest/abc.txt with flags a
DEBUG 2017-08-18 16:05:59,174 123145483575296 vos-3.0.0b6 vofs.get_node.410 requesting node /sometest/abc.txt from VOSpace. Force: False
DEBUG 2017-08-18 16:05:59,175 123145483575296 vos-3.0.0b6 ws._get_url.328 Resolved URL: https://www.canfar.phys.uvic.ca/vospace/nodes
DEBUG 2017-08-18 16:05:59,177 123145483575296 vos-3.0.0b6 ws.send.420 Sending request <PreparedRequest [GET]> to server.
DEBUG 2017-08-18 16:05:59,271 123145483575296 vos-3.0.0b6 vofs.get_node.412 Got node abc.txt
DEBUG 2017-08-18 16:05:59,271 123145483575296 vos-3.0.0b6 vofs.get_node.410 requesting node /sometest from VOSpace. Force: False
DEBUG 2017-08-18 16:05:59,271 123145483575296 vos-3.0.0b6 vofs.get_node.412 Got node sometest
DEBUG 2017-08-18 16:05:59,271 123145483575296 vos-3.0.0b6 vofs.open.527 IO Proxy initialized:Path:/sometest/abc.txt Size:0 MD5:None condition:<vofs.CadcCache.CacheCondition object at 0x114d5dc50> in backing.
DEBUG 2017-08-18 16:05:59,272 123145483575296 vos-3.0.0b6 CadcCache.setHeader.832 set header called with size: 0 and md5: d41d8cd98f00b204e9800998ecf8427e
DEBUG 2017-08-18 16:05:59,273 123145483575296 vos-3.0.0b6 CadcCache.fullyCached.817 setHeader set to fullyCached to: False
DEBUG 2017-08-18 16:05:59,274 123145483575296 vos-3.0.0b6 CadcCache.fullyCached.817 setHeader set to fullyCached to: True
DEBUG 2017-08-18 16:05:59,274 123145483575296 vos-3.0.0b6 vofs.open.532 Creating file:/sometest/abc.txt in backing.
DEBUG 2017-08-18 16:05:59,274 123145483575296 vos-3.0.0b6 vofs.get_node.410 requesting node /sometest/abc.txt from VOSpace. Force: False
DEBUG 2017-08-18 16:05:59,275 123145483575296 vos-3.0.0b6 vofs.get_node.412 Got node abc.txt
DEBUG 2017-08-18 16:05:59,275 123145483575296 vos-3.0.0b6 CadcCache.setHeader.832 set header called with size: 0 and md5: None
DEBUG 2017-08-18 16:05:59,275 123145483575296 vos-3.0.0b6 vofs.call.279 4517565552
DEBUG 2017-08-18 16:05:59,275 123145483575296 vos-3.0.0b6 vofs.call.280 <- create 4517565552
create[4517565552] flags: 0xa02 /sometest/abc.txt
getattr /sometest/abc.txt
DEBUG 2017-08-18 16:05:59,275 123145483575296 vos-3.0.0b6 vofs.call.265 -> getattr ('/sometest/abc.txt', None)
DEBUG 2017-08-18 16:05:59,276 123145483575296 vos-3.0.0b6 vofs.call.279 {'st_dev': 16777220, 'st_mode': 33261, 'st_ctime': 1503097559.0, 'st_mtime_ns': 1503097559000000000, 'st_gid': 0, 'st_size': 0, 'st_ino': 6888127, 'st_birthtime': 1503097559.0, 'n_unnamed_fields': 3, 'count': <built-in method count of os.stat_result object at 0x10e56e828>, 'st_blksize': 4096, 'st_flags': 0, 'n_sequence_fields': 10, 'st_rdev': 0, 'st_atime_ns': 1503097559000000000, 'st_uid': 501, 'st_atime': 1503097559.0, 'st_blocks': 0, 'st_mtime': 1503097559.0, 'st_ctime_ns': 1503097559000000000, 'n_fields': 22, 'index': <built-in method index of os.stat_result object at 0x10e56e828>, 'st_gen': 0, 'st_nlink': 1}
DEBUG 2017-08-18 16:05:59,276 123145483575296 vos-3.0.0b6 vofs.call.280 <- getattr {'st_dev': 16777220, 'st_mode': 33261, 'st_ctime': 1503097559.0, 'st_mtime_ns': 1503097559000000000, 'st_gid': 0, 'st_size': 0, 'st_ino': 6888127, 'st_birthtime': 1503097559.0, 'n_unnamed_fields': 3, 'count': <built-in method count of os.stat_result object at 0x10e56e828>, 'st_blksize': 4096, 'st_flags': 0, 'n_sequence_fields': 10, 'st_rdev': 0, 'st_atime_ns': 1503097559000000000, 'st_uid': 501, 'st_atime': 1503097559.0, 'st_blocks': 0, 'st_mtime': 1503097559.0, 'st_ctime_ns': 1503097559000000000, 'n_fields': 22, 'index': <built-in method index of os.stat_result object at 0x10e56e828>, 'st_gen': 0, 'st_nlink': 1}
NODEID: 12
unique: 7, success, outsize: 176
unique: 3, opcode: FLUSH (25), nodeid: 12, insize: 64, pid: 28078
flush[4517565552]
DEBUG 2017-08-18 16:05:59,276 123145484111872 vos-3.0.0b6 vofs.call.265 -> flush ('/sometest/abc.txt', 4517565552)
DEBUG 2017-08-18 16:05:59,276 123145484111872 vos-3.0.0b6 vofs.flush.615 flushing 4517565552
DEBUG 2017-08-18 16:05:59,277 123145466736640 vos-3.0.0b6 vofs.writeToBacking.69 PUSHING /tmp/vos_cache/data/sometest/abc.txt to VOSpace @ /sometest/abc.txt
DEBUG 2017-08-18 16:05:59,277 123145466736640 vos-3.0.0b6 vofs.writeToBacking.70 opening a new vo file for /sometest/abc.txt
DEBUG 2017-08-18 16:05:59,277 123145466736640 vos-3.0.0b6 vofs.get_node.410 requesting node /sometest/abc.txt from VOSpace. Force: False
DEBUG 2017-08-18 16:05:59,278 123145466736640 vos-3.0.0b6 vofs.get_node.412 Got node abc.txt
DEBUG 2017-08-18 16:05:59,279 123145466736640 vos-3.0.0b6 ws._get_url.328 Resolved URL: https://www.canfar.phys.uvic.ca/vospace/synctrans
DEBUG 2017-08-18 16:05:59,279 123145466736640 vos-3.0.0b6 ws._get_url.328 Resolved URL: https://www.canfar.phys.uvic.ca/vospace/synctrans
DEBUG 2017-08-18 16:05:59,287 123145466736640 vos-3.0.0b6 ws.send.420 Sending request <PreparedRequest [GET]> to server.
unique: 8, opcode: GETATTR (3), nodeid: 2, insize: 56, pid: 45
getattr /sometest
DEBUG 2017-08-18 16:05:59,348 123145489477632 vos-3.0.0b6 vofs.call.265 -> getattr ('/sometest', None)
DEBUG 2017-08-18 16:05:59,348 123145489477632 vos-3.0.0b6 vofs.get_node.410 requesting node /sometest from VOSpace. Force: False
DEBUG 2017-08-18 16:05:59,349 123145489477632 vos-3.0.0b6 vofs.get_node.412 Got node sometest
DEBUG 2017-08-18 16:05:59,349 123145489477632 vos-3.0.0b6 vofs.call.279 {'st_mtime': 1503086592.0, 'st_mode': 16837, 'st_uid': 501, 'st_ctime': 1503086592.0, 'st_atime': 1503097294.125806, 'st_blocks': 0, 'st_gid': 20, 'st_size': 6, 'st_nlink': 2}
DEBUG 2017-08-18 16:05:59,349 123145489477632 vos-3.0.0b6 vofs.call.280 <- getattr {'st_mtime': 1503086592.0, 'st_mode': 16837, 'st_uid': 501, 'st_ctime': 1503086592.0, 'st_atime': 1503097294.125806, 'st_blocks': 0, 'st_gid': 20, 'st_size': 6, 'st_nlink': 2}
unique: 8, success, outsize: 136
DEBUG 2017-08-18 16:05:59,478 123145466736640 vos-3.0.0b6 vos.get_node_url.1736 Transfer Server said: b''
DEBUG 2017-08-18 16:05:59,487 123145466736640 vos-3.0.0b6 ws.send.420 Sending request <PreparedRequest [PUT]> to server.
CRITICAL 2017-08-18 16:06:29,278 123145484111872 vos-3.0.0b6 vofs.flush.621 'Condition variable timeout'
CRITICAL 2017-08-18 16:06:29,278 123145484111872 vos-3.0.0b6 vofs.flush.622 Push to VOSpace reached FUSE timeout, continuing VOSpace push in background.
DEBUG 2017-08-18 16:06:29,278 123145484111872 vos-3.0.0b6 vofs.call.279 0
DEBUG 2017-08-18 16:06:29,278 123145484111872 vos-3.0.0b6 vofs.call.280 <- flush 0
unique: 3, success, outsize: 16
unique: 6, opcode: RELEASE (18), nodeid: 12, insize: 64, pid: 28078
release[4517565552] flags: 0x0
DEBUG 2017-08-18 16:06:29,278 123145486258176 vos-3.0.0b6 vofs.call.265 -> release ('/sometest/abc.txt', 4517565552)
DEBUG 2017-08-18 16:06:29,278 123145486258176 vos-3.0.0b6 vofs.release.633 releasing file 4517565552
DEBUG 2017-08-18 16:06:29,279 123145486258176 vos-3.0.0b6 vofs.call.279 None
DEBUG 2017-08-18 16:06:29,279 123145486258176 vos-3.0.0b6 vofs.call.280 <- release None
unique: 6, success, outsize: 16
unique: 26, opcode: GETATTR (3), nodeid: 12, insize: 56, pid: 45
getattr /sometest/abc.txt
DEBUG 2017-08-18 16:06:29,279 123145498062848 vos-3.0.0b6 vofs.call.265 -> getattr ('/sometest/abc.txt', None)
DEBUG 2017-08-18 16:06:29,279 123145498062848 vos-3.0.0b6 vofs.call.279 {'st_dev': 16777220, 'st_mode': 33261, 'st_ctime': 1503097559.0, 'st_mtime_ns': 1503097559000000000, 'st_gid': 0, 'st_size': 0, 'st_ino': 6888127, 'st_birthtime': 1503097559.0, 'n_unnamed_fields': 3, 'count': <built-in method count of os.stat_result object at 0x10e56e9e8>, 'st_blksize': 4096, 'st_flags': 0, 'n_sequence_fields': 10, 'st_rdev': 0, 'st_atime_ns': 1503097559000000000, 'st_uid': 501, 'st_atime': 1503097559.0, 'st_blocks': 0, 'st_mtime': 1503097559.0, 'st_ctime_ns': 1503097559000000000, 'n_fields': 22, 'index': <built-in method index of os.stat_result object at 0x10e56e9e8>, 'st_gen': 0, 'st_nlink': 1}
DEBUG 2017-08-18 16:06:29,279 123145498062848 vos-3.0.0b6 vofs.call.280 <- getattr {'st_dev': 16777220, 'st_mode': 33261, 'st_ctime': 1503097559.0, 'st_mtime_ns': 1503097559000000000, 'st_gid': 0, 'st_size': 0, 'st_ino': 6888127, 'st_birthtime': 1503097559.0, 'n_unnamed_fields': 3, 'count': <built-in method count of os.stat_result object at 0x10e56e9e8>, 'st_blksize': 4096, 'st_flags': 0, 'n_sequence_fields': 10, 'st_rdev': 0, 'st_atime_ns': 1503097559000000000, 'st_uid': 501, 'st_atime': 1503097559.0, 'st_blocks': 0, 'st_mtime': 1503097559.0, 'st_ctime_ns': 1503097559000000000, 'n_fields': 22, 'index': <built-in method index of os.stat_result object at 0x10e56e9e8>, 'st_gen': 0, 'st_nlink': 1}
unique: 26, success, outsize: 136

feature request: wildcard/globbing for vls

It would be useful to have vls understand wildcard on vospace

Desired behaviour:
vls vos:myvospace/*

Probably fairly intense on vospace db.

might be a AttributeError in some error handling code

You don't actually trigger this erro unless the exception is thrown. However when it does get thrown it looks like you trigger a different error.

[igable@elephant junk]$ vcp vos:sfabbro/vostest/1235593p.ccd06 1235593p.ccd06
2012-03-05 15:25:15,981 - vcp.copy: ERROR: [Errno 104] Connection reset by peer
2012-03-05 15:25:15,981 - vcp.copy: Retrying in 1 seconds

Traceback (most recent call last):
  File "/home/igable/code/cadc/bin/vcp", line 7, in <module>
    execfile(__file__)
  File "/home/igable/code/cadc/cadcVOFS/vcp", line 140, in <module>
    logging.error("Failed to copy %s to %s:  %s (%d)" %(source, thisDest, e.strerror, e.errno))
AttributeError: 'exceptions.NameError' object has no attribute 'strerror'
(cadc)[igable@elephant junk]$

missing releases on github

Can't see 1.10.4 nor 1.10.4 on github tags, but can see them on pypi.

content length errors in cache.

Sometimes there appears to be a dis-connect between the content written to disk and the content expected. This maybe due to a race condition.

2015-01-05 12:55:48,848 139654004348672 vos-1.11.2fuse._wrapper.411 'NoneType' object has no attribute 'getRange' Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/vos-1.11.2-py2.6.egg/vos/fuse.py", line 407, in _wrapper
return func(_args, *_kwargs) or 0
File "/usr/lib/python2.6/site-packages/vos-1.11.2-py2.6.egg/vos/fuse.py", line 484, in read
data = self.operations('read', path.decode(self.encoding), size, offset, fh)
File "/usr/lib/python2.6/site-packages/vos-1.11.2-py2.6.egg/vos/vofs.py", line 254, in call
return super(VOFS, self).call(op, path, *args)
File "/usr/lib/python2.6/site-packages/vos-1.11.2-py2.6.egg/vos/fuse.py", line 848, in call
ret = getattr(self, op)(path, *args)
File "/usr/lib/python2.6/site-packages/vos-1.11.2-py2.6.egg/vos/vofs.py", line 569, in read
return fh.cacheFileHandle.read(size, offset)
File "/usr/lib/python2.6/site-packages/vos-1.11.2-py2.6.egg/vos/CadcCache.py", line 1030, in read
self.makeCached(offset, size)
File "/usr/lib/python2.6/site-packages/vos-1.11.2-py2.6.egg/vos/CadcCache.py", line 1065, in makeCached
requiredRange = self.metaData.getRange(firstBlock, lastBlock)
AttributeError: 'NoneType' object has no attribute 'getRange'

chmod broken in vofs

chmod does not alter the permission of the cache file (just the permission of the node) and that causes all sort of inconsistencies.

Make vsync bidirectional

Add download option to rsync local directory with vospace

wrong error message on vcp result

For example:
vls -l vos:sgwyn/fredxxx
NodeNotFound: vos://cadc.nrc.ca!vospace/sgwyn/fredxxx

vcp vos:sgwyn/fredxxx .

[Errno 13] Permission denied: vos:sgwyn/fredxxx

vcp --version
1.10.4

error handling in latest 1.10.5 release with python-2.6?

On a brand new SL5 VM, with a valid certificate with vos-1.10.5:

$ vls --debug vos:sfabbro
getting listing of: vos:sfabbro
error

Not quite informative

feature request: mountvofs default actions

It would be nice if mountvofs does not do anything when no arguments are passed.

Desired behaviour:

mountvofs without arguments: same as mountvofs --help
mountvofs vospace : mounts on /tmp/vospace
mountvofs vospace dir: mounts vospace on directory dir

the version of the scripts should be unique.

Currently scripts pull their version number from the library. Actually, the scripts have their own version increments and should check that they are pulling in the library version they are meant to go with .. This is needed since the scripts / applications are separate entities and even when one installs the new package one sometimes ends-up with the old script calling the new library.