mgerdts/ceph

Ceph is a distributed object, block, and file storage platform

ceph's Issues

port bluestore

Bluestore is disabled because FreeBSD said ETOOHARD. For the illumos implementation to be serious it needs to have a compelling zfs implementation or it needs bluestore.

ceph-osd needs to link with some libraries

/home/mgerdts/ceph/build/src/CMakeFiles/ceph-osd.dir/link.txt needs -lresolv -lnsl -lsocket else we get stuff like the following (which was still missing -lresolv).

cd /home/mgerdts/ceph/build/src && /opt/local/bin/cmake -E cmake_link_script CMakeFiles/ceph-osd.dir/link.txt --verbose=1
ccache /opt/local/bin/c++   -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing -fsigned-char -Wno-unknown-pragmas -rdynamic -pthreads -I/usr/local/include -I/opt/local/include -pthreads -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-unknown-pragmas -Wno-ignored-qualifiers -Wstrict-null-sentinel -Woverloaded-virtual -fno-new-ttp-matching -DCEPH_DEBUG_MUTEX -fstack-protector-strong -fdiagnostics-color=auto -pthreads -I/usr/local/include -I/opt/local/include -DCEPH_DEV   -L/usr/local/lib -Wl,-rpath /usr/local/lib -L/opt/local/lib -Wl,-rpath /opt/local/lib -L/usr/lib/mps -Wl,-rpath /usr/lib/mps  -L/usr/local/lib -Wl,-rpath /usr/local/lib -L/opt/local/lib -Wl,-rpath /opt/local/lib -L/usr/lib/mps -Wl,-rpath /usr/lib/mps CMakeFiles/ceph-osd.dir/ceph_osd.cc.o  -o ../bin/ceph-osd -Wl,-R/opt/local/lib/nss:/opt/local/lib/nspr:::::::::::::::::::: ../lib/libosd.a ../lib/libos.a ../lib/libglobal-static.a ../lib/libcommon.a ../lib/libdmclock.a ../lib/libcpu_profiler.a ../lib/libheap_profiler.a ../lib/libkv.a /opt/local/lib/libleveldb.so rocksdb/librocksdb.a /opt/local/lib/libsnappy.so /opt/local/lib/liblz4.so /opt/local/lib/libz.so ../lib/libjson_spirit.a ../lib/libcommon_utf8.a ../lib/liberasure_code.a ../lib/libcrc32.a ../lib/libarch.a /opt/local/lib/libboost_thread.so /opt/local/lib/libboost_chrono.so /opt/local/lib/libboost_atomic.so /opt/local/lib/libboost_random.so /opt/local/lib/libboost_system.so /opt/local/lib/libboost_program_options.so /opt/local/lib/libboost_date_time.so /opt/local/lib/libboost_iostreams.so /opt/local/lib/libboost_regex.so /opt/local/lib/nss/libssl3.so /opt/local/lib/nss/libsmime3.so /opt/local/lib/nss/libnss3.so /opt/local/lib/nss/libnssutil3.so /opt/local/lib/nspr/libplds4.so /opt/local/lib/nspr/libplc4.so /opt/local/lib/nspr/libnspr4.so /opt/local/lib/libcrypto.so /opt/local/lib/libgssapi_krb5.so -ldl -lnsl -lsocket

need proper numa support

The initial hacks ifdef out pretty much everything related to numa.

find or create options to perform more selective builds

It is likely that builds will only want the client libraries. Explore do_cmake.sh -D WITH_<foo> to disable builds of osd, mds, mgr, etc,

Need O_DIRECTORY in common.h

Need

#define O_DIRECTORY (0)

src/common/xattr.c needs more complete implementation

Only some of the extended file attributes operations have been implemented. The implementation details could probably move to another file. Like, libc?

sort out map confusion

Find a way to not need this in a lot of places (ahead of boost includes):

/*
 * boost ends up including asio/detail/socket_types.hpp which includes
 * <net/if.h>.  if.h has a structure that has a "struct map" member which has
 * namespace conflicts with std::map.  That "struct map" has a namespace
 * conflict with std::map.
 */
#ifdef __sun__
struct renamed_map;
#define map renamed_map
#include <net/if.h>
#undef map
#endif

need portable statfs

There are calls to statfs() and fstatfs() in many places. illumos has statvfs() and fstatvfs(). The tricky part is that statfs.f_type needs to be derived from statvfs.f_str.

librados --exclude-libs not working

[ 19%] Linking CXX shared library ../../lib/librados.so
cd /home/mgerdts/ceph/build/src/librados && /opt/local/bin/cmake -E cmake_link_script CMakeFiles/librados.dir/link.txt --verbose=1
ccache /opt/local/bin/c++ -fPIC  -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing -fsigned-char -Wno-unknown-pragmas -rdynamic -pthreads -I/usr/local/include -I/opt/local/include -pthreads -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-unknown-pragmas -Wno-ignored-qualifiers -Wstrict-null-sentinel -Woverloaded-virtual -fno-new-ttp-matching -DCEPH_DEBUG_MUTEX -fstack-protector-strong -fdiagnostics-color=auto -pthreads -I/usr/local/include -I/opt/local/include -DCEPH_DEV  -Wl,--exclude-libs,ALL  -L/usr/local/lib -Wl,-rpath /usr/local/lib -L/opt/local/lib -Wl,-rpath /opt/local/lib -L/usr/lib/mps -Wl,-rpath /usr/lib/mps  -L/usr/local/lib -Wl,-rpath /usr/local/lib -L/opt/local/lib -Wl,-rpath /opt/local/lib -L/usr/lib/mps -Wl,-rpath /usr/lib/mps -shared -Wl,-hlibrados.so.3 -o ../../lib/librados.so.3.0.0 CMakeFiles/rados_c_api.dir/librados_c.cc.o -Wl,-R/home/mgerdts/ceph/build/lib:/opt/local/lib/nss:/opt/local/lib/nspr: ../../lib/librados_cxx.a ../../lib/liblibrados_impl.a ../../lib/libosdc.a ../../lib/libceph-common.so.1 ../../lib/libcls_lock_client.a /opt/local/lib/nss/libssl3.so /opt/local/lib/nss/libsmime3.so /opt/local/lib/nss/libnss3.so /opt/local/lib/nss/libnssutil3.so /opt/local/lib/nspr/libplds4.so /opt/local/lib/nspr/libplc4.so /opt/local/lib/nspr/libnspr4.so /opt/local/lib/libcrypto.so -ldl -lrt /opt/local/lib/libgssapi_krb5.so ../../lib/libjson_spirit.a ../../lib/libcommon_utf8.a ../../lib/liberasure_code.a -ldl ../../lib/libcrc32.a ../../lib/libarch.a /opt/local/lib/libboost_thread.so /opt/local/lib/libboost_chrono.so /opt/local/lib/libboost_atomic.so /opt/local/lib/libboost_random.so /opt/local/lib/libboost_system.so /opt/local/lib/libboost_program_options.so /opt/local/lib/libboost_date_time.so /opt/local/lib/libboost_iostreams.so /opt/local/lib/libboost_regex.so 
ld: fatal: unrecognized option '--exclude-libs'
ld: fatal: use the -z help option for usage information
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/librados/CMakeFiles/librados.dir/build.make:99: lib/librados.so.3.0.0] Error 1
gmake[2]: Leaving directory '/home/mgerdts/ceph/build'
gmake[1]: *** [CMakeFiles/Makefile2:3277: src/librados/CMakeFiles/librados.dir/all] Error 2
gmake[1]: Leaving directory '/home/mgerdts/ceph/build'
gmake: *** [Makefile:141: all] Error 2

Better, removing -Wl,--exclude-libs,ALL:

$ cd /home/mgerdts/ceph/build/src/librados
$ ccache /opt/local/bin/c++ -fPIC  -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing -fsigned-char -Wno-unknown-pragmas -rdynamic -pthreads -I/usr/local/include -I/opt/local/include -pthreads -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-unknown-pragmas -Wno-ignored-qualifiers -Wstrict-null-sentinel -Woverloaded-virtual -fno-new-ttp-matching -DCEPH_DEBUG_MUTEX -fstack-protector-strong -fdiagnostics-color=auto -pthreads -I/usr/local/include -I/opt/local/include -DCEPH_DEV   -L/usr/local/lib -Wl,-rpath /usr/local/lib -L/opt/local/lib -Wl,-rpath /opt/local/lib -L/usr/lib/mps -Wl,-rpath /usr/lib/mps  -L/usr/local/lib -Wl,-rpath /usr/local/lib -L/opt/local/lib -Wl,-rpath /opt/local/lib -L/usr/lib/mps -Wl,-rpath /usr/lib/mps -shared -Wl,-hlibrados.so.3 -o ../../lib/librados.so.3.0.0 CMakeFiles/rados_c_api.dir/librados_c.cc.o -Wl,-R/home/mgerdts/ceph/build/lib:/opt/local/lib/nss:/opt/local/lib/nspr: ../../lib/librados_cxx.a ../../lib/liblibrados_impl.a ../../lib/libosdc.a ../../lib/libceph-common.so.1 ../../lib/libcls_lock_client.a /opt/local/lib/nss/libssl3.so /opt/local/lib/nss/libsmime3.so /opt/local/lib/nss/libnss3.so /opt/local/lib/nss/libnssutil3.so /opt/local/lib/nspr/libplds4.so /opt/local/lib/nspr/libplc4.so /opt/local/lib/nspr/libnspr4.so /opt/local/lib/libcrypto.so -ldl -lrt /opt/local/lib/libgssapi_krb5.so ../../lib/libjson_spirit.a ../../lib/libcommon_utf8.a ../../lib/liberasure_code.a -ldl ../../lib/libcrc32.a ../../lib/libarch.a /opt/local/lib/libboost_thread.so /opt/local/lib/libboost_chrono.so /opt/local/lib/libboost_atomic.so /opt/local/lib/libboost_random.so /opt/local/lib/libboost_system.so /opt/local/lib/libboost_program_options.so /opt/local/lib/libboost_date_time.so /opt/local/lib/libboost_iostreams.so /opt/local/lib/libboost_regex.so

But it still fails on the next gmake. Hack, hack:

cd /home/mgerdts/ceph/build/src/librados
vi $(find . -type f | xargs grep -l exclude-libs)
cd -

manual_fallocate() is misguided

src/common/compat.cc has:

// On FreeBSD, ZFS fallocate always fails since it is considered impossible to
// reserve space on a COW filesystem. posix_fallocate() returns EINVAL
// Linux in this case already emulates the reservation in glibc
// In which case it is allocated manually, and still that is not a real guarantee
// that a full buffer is allocated on disk, since it could be compressed.
// To prevent this the written buffer needs to be loaded with random data.
int manual_fallocate(int fd, off_t offset, off_t len) {
  int r = lseek(fd, offset, SEEK_SET);
  if (r == -1) 
    return errno;
  char data[1024*128];
  // TODO: compressing filesystems would require random data
  memset(data, 0x42, sizeof(data));
  for (off_t off = 0; off < len; off += sizeof(data)) {
    if (off + static_cast<off_t>(sizeof(data)) > len)
      r = safe_write(fd, data, len - off);
    else
      r = safe_write(fd, data, sizeof(data));
    if (r == -1) {
      return errno;
    }   
  }
  return 0;
}

It acknowledges that fallocate() on a CoW file system cannot give the intended results, then goes ahead and tries to implement it. The problem is that the write that follows fallocate() will write to different blocks than manual_fallocate() writes to. A better solution is to either always return success or to check to see if sufficient space is available (fstatvfs()).

A best practice with ZFS would be to use refreservation and quota if reliable allocation guarantees are needed.