dcache/dcap

dCache access protocol client library

dCache

dCache is a system for storing and retrieving huge amounts of data, distributed among a large number of heterogeneous server nodes, under a single virtual filesystem tree with a variety of standard access methods. Depending on the Persistency Model, dCache provides methods for exchanging data with backend (tertiary) Storage Systems as well as space management, pool attraction, dataset replication, hot spot determination and recovery from disk or node failures. Connected to a tertiary storage system, the cache simulates unlimited direct access storage space. Data exchanges to and from the underlying HSM are performed automatically and invisibly to the user. Beside HEP specific protocols, data in dCache can be accessed via NFSv4.1 (pNFS), FTP as well as through WebDav.

Documentation

The dCache book

User Guide

Getting Started

The file BUILDING.md describes how to compile dCache code and build various packages.

The file also describes how to create the system-test deployment, which provides a quick and easy way to get a working dCache. Running system-test requires no special privileges and all the generated files reside within the code-base.

There are also packages of stable releases at https://www.dcache.org/downloads/.

License

The project is licensed under AGPL v3. Some parts licensed under BSD and LGPL. See the source code for details.

For more info, check the official dCache.ORG web page.

Contributors

dCache is a joint effort between Deutsches Elektronen-Synchrotron DESY, Fermi National Accelerator Laboratory and Nordic DataGrid Facility.

Contributions are welcome! Please check out our CONTRIBUTING guide.

dcap's Issues

dcap-libs postuninstall scriptlet problem

The comment at line 127 in dcap/specs/sl6/dcap-fedora.spec ends up in the postuninstall script.

Build failed on Mac (x86_64-darwin) due to `ntohll(x)` and `htonll(x)` being `#defined` in `Libsystem`

Partial build log on OfBorg (the CI of NixOS/nixpkgs) that demonstrate the error:

/nix/store/b0q4wjdw6xwhfsqy8ab106cmhn5xrr8p-bash-5.1-p12/bin/bash ../libtool  --tag=CC   --mode=compile clang -DHAVE_CONFIG_H -I.  -I/nix/store/109pyfb5h9dally7iav62zcsg38i8ydw-dcap-2.47.12-dev/include -DLOCALEDIR=\"/nix/store/iyg74j1agq4iw0pvb1avk93hpfnvzdy4-dcap-2.47.12/share/locale\"   -D_REENTRANT -DLIBC_SYSCALLS -D_GNU_SOURCE -g -O2 -c -o dcap.lo dcap.c
libtool: compile:  clang -DHAVE_CONFIG_H -I. -I/nix/store/109pyfb5h9dally7iav62zcsg38i8ydw-dcap-2.47.12-dev/include -DLOCALEDIR=\"/nix/store/iyg74j1agq4iw0pvb1avk93hpfnvzdy4-dcap-2.47.12/share/locale\" -D_REENTRANT -DLIBC_SYSCALLS -D_GNU_SOURCE -g -O2 -c dcap.c  -fno-common -DPIC -o .libs/dcap.o
In file included from dcap.c:56:
./dcap_functions.h:43:10: error: expected ')'
uint64_t ntohll(uint64_t x);
         ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
                             ^
./dcap_functions.h:43:10: note: to match this '('
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
    ^
In file included from dcap.c:56:
./dcap_functions.h:43:10: error: cannot redeclare builtin function '__builtin_constant_p'
uint64_t ntohll(uint64_t x);
         ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
     ^
./dcap_functions.h:43:10: note: '__builtin_constant_p' is a builtin with type 'int ()'
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
     ^
In file included from dcap.c:56:
./dcap_functions.h:47:10: error: expected ')'
uint64_t htonll(uint64_t arg);
         ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
                             ^
./dcap_functions.h:47:10: note: to match this '('
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
    ^
In file included from dcap.c:56:
./dcap_functions.h:47:10: error: cannot redeclare builtin function '__builtin_constant_p'
uint64_t htonll(uint64_t arg);
         ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
     ^
./dcap_functions.h:43:10: note: '__builtin_constant_p' is a builtin with type 'int ()'
uint64_t ntohll(uint64_t x);
         ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
     ^
dcap.c:130:15: error: expected ')'
    uint64_t  htonll(uint64_t);
              ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
                             ^
dcap.c:130:15: note: to match this '('
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
    ^
dcap.c:130:15: error: cannot redeclare builtin function '__builtin_constant_p'
    uint64_t  htonll(uint64_t);
              ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:141:25: note: expanded from macro 'htonll'
#define htonll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
     ^
./dcap_functions.h:43:10: note: '__builtin_constant_p' is a builtin with type 'int ()'
uint64_t ntohll(uint64_t x);
         ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
     ^
dcap.c:133:15: error: expected ')'
    uint64_t  ntohll(uint64_t);
              ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
                             ^
dcap.c:133:15: note: to match this '('
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
    ^
dcap.c:133:15: error: cannot redeclare builtin function '__builtin_constant_p'
    uint64_t  ntohll(uint64_t);
              ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
     ^
./dcap_functions.h:43:10: note: '__builtin_constant_p' is a builtin with type 'int ()'
uint64_t ntohll(uint64_t x);
         ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll'
#define ntohll(x)       __DARWIN_OSSwapInt64(x)
                        ^
/nix/store/8xkv6d53n2hznyvf1vsyknqz4awj77i7-Libsystem-1238.60.2/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64'
    (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x))
     ^

https://logs.nix.ci/?key=nixos/nixpkgs.156282&attempt_id=62c1024b-7e66-4cd7-ae7c-a69ad6d84d6b

Build expression: NixOS/nixpkgs#156282

New tag?

I'd like to request a new tag.
It is a bit dodgy to change license statements using a patch in packaging.
It would be possible to base the packaging on a git checkout of master of course, but a tagged version would be better. And there are a few other fixes in git that maybe deserve to be part of a tag.
Can the version be bumped in configure.ac and then a tag created?

remove client passive move

The original dcap implementation was based on a model where dcache pool connect to the client for the data path. Now days so called passive mode is preferred, where client initializes both control and data connections, as this is more firewall friendly behavior. The old functionality is mostly not used and/or source of unexpected problems, thus should be removed.

License clarification

I have received a bug report in Debian about that the dcap source contains non-free code:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1029031

The files in question are the source files:

plugins/gssapi/gssTunnel.h
plugins/gssapi/Copyright
plugins/gssapi/tunnelQueue.c
plugins/gssapi/gssIoTunnel.c
plugins/gssapi/tunnelQueue.h

that contain the line

   THIS IS UNPUBLISHED PROPRIETARY SOURCE CODE OF
             DESY Hamburg DMG-Division

Is this statement correctly reflecting the license status of those files?
If the files are intended to be open source, could the statement be changed and then a new tag be made?