bertha/berthad

Simple, fast, no-nonsense blob storage

BerthaD

BerthaD is a simple, fast, no-nonsense TCP server to store blobs of data by its SHA256 hash with just three operations:

1. PUT – Stores the data on disk and return its hash.
2. LIST – Return the hashes of all blobs stored on this server.
3. GET – Given the hash of a blob it returns the content of the blob.

Features

• __GET__s are fast. They are implemented using Linux' splice and FreeBSD's sendfile syscall.
• Small codebase.
• No authentication. No SSL. If you don't need them, they are only an overhead.

Clients

Python

>>> from bertha import BerthaClient
>>> c = BerthaClient('localhost', 1234)
>>> list(c.list())
[]
>>> c.put_str("Example blob")
'a0869d836f643fae5d740ad4407c97e174d03169fd788fce690341d03f8d8f44'
>>> list(c.list())
['a0869d836f643fae5d740ad4407c97e174d03169fd788fce690341d03f8d8f44']
>>> c.get('a0869d836f643fae5d740ad4407c97e174d03169fd788fce690341d03f8d8f44').read()
'Example blob'

See py-bertha.

Building

$ cd release
$ ./build

Running

Usage: berthad-vfs <bound host> <port> <data dir> <tmp dir>

• bound host is address or name of the address to be bound. For instance: localhost or 0.0.0.0.
• port is the port on which to bind. Clients default to 819.
• data dir is the directory which will contain the blobs. It must exist.
• tmp dir is the directory which will contain the blobs while they are streamed to disk during a PUT. The directory must be on the same mounting point as data dir.

Protocol

The protocol is described in PROTOCOL.md.

berthad's Issues

FreeBSD: Neither sendfile() nor select() triggers diskio

sendfile() does not read from the disk and returns EBUSY when the data is not cached. Select() was supposed to flag it as ready when the data became available but it flags it as such even when the data is not yet buffered, so sendfile returns EBUSY.

There should be something that triggers the kernel to actually read from the disk, preferrably in a nonblocking fashion but otherwise in another thread.

`maried` does not recover from network hiccup

Traceback from maried after a short DNS hiccup:

Traceback (most recent call last):
  File "/home/maried/py/maried/gstreamer.py", line 302, in _on_about_to_finish
    self.on_about_to_finish()
  File "/home/maried/py/sarah/event.py", line 8, in __call__
    handler(*args, **kwargs)
  File "/home/maried/py/maried/core.py", line 376, in _player_on_about_to_finish
    self._queue_next()
  File "/home/maried/py/maried/core.py", line 324, in _queue_next
    self.player.queue(media)
  File "/home/maried/py/maried/gstreamer.py", line 230, in queue
    stream = mf.open()
  File "/home/maried/py/maried/_bertha.py", line 13, in open
    return self.store._open(self.key)
  File "/home/maried/py/maried/_bertha.py", line 59, in _open
    return self.c.get(key)
  File "/home/maried/py/bertha/client.py", line 142, in get
    sock = self._connect()
  File "/home/maried/py/bertha/client.py", line 205, in _connect
    socket.SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

berthad gives back an error because the DNS service is unreachable, but maried does not retry to retrieve the file.