dbarray's Introduction

Array stored in database

DBArray is an 2D array stored in database. The purpose of this class is to provide a way to store and access large array which can not be loaded into memory.

Example

import numpy as np
from dbarray import DBArray

# Create random array
nrows = 5
ncols = 8
dtype = np.float32
arr = np.random.random((nrows, ncols))
arr = np.require(arr, dtype)
print arr

""" `DBArray` from scratch
"""
# Create `DBArray`
dba1 = DBArray('test1.db')
# Initialize
dba1.set_shape((nrows, ncols))
dba1.set_dtype(dtype)

# Set rows
for i in range(nrows):
    dba1[i] = arr[i]

# Convert to ndarray
print dba1.tondarray()

""" `DBArray` from `ndarray`
"""
# Create `DBArray`
dba2 = DBArray.fromndarray(arr, 'test2.db')

# Check attributes
print dba2.nrows, dba2.ncols, dba2.dtype

# Convert to ndarray
print dba2.tondarray()

dbarray's People

Contributors

Stargazers

Watchers

dbarray's Issues

lmdb.BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot

The following code produces lmdb.BadRSlotError in the second iteration (idx==1):

import os
import numpy as np
from dbarray import DBArray

os.system("rm -fr test.db")

for idx in range(2):
    print "###################################"

    # Create random array
    nrows = 1
    ncols = 5
    dtype = np.float32
    arr = np.random.random((nrows, ncols))
    arr = np.require(arr, dtype)
    print arr

    """ `DBArray` from scratch
    """
    # Create `DBArray`
    dba1 = DBArray('test.db')
    # Initialize
    dba1.set_shape((nrows, ncols))
    dba1.set_dtype(dtype)
    # Set rows
    dba1[:] = arr[:]
    # Convert to ndarray
    print dba1.tondarray()

    """ Open exsiting `DBArray`
    """
    # Open another DB
    dba2 = DBArray('test.db')
    print dba2.tondarray()

Error message:

###################################
[[ 0.86903572  0.26334628  0.59591901  0.5937745   0.51802516]]
[[ 0.86903572  0.26334628  0.59591901  0.5937745   0.51802516]]
[[ 0.86903572  0.26334628  0.59591901  0.5937745   0.51802516]]
###################################
[[ 0.42391387  0.74991614  0.7317664   0.62704509  0.33670583]]
Traceback (most recent call last):
  File "bug1.py", line 28, in <module>
    print dba1.tondarray()
  File "/usr/local/lib/python2.7/dist-packages/dbarray/dbarray.py", line 353, in tondarray
    return self.get_rows(range(self.nrows))
  File "/usr/local/lib/python2.7/dist-packages/dbarray/dbarray.py", line 229, in get_rows
    resarr[i, :] = self.get_row(v_rid[i])
  File "/usr/local/lib/python2.7/dist-packages/dbarray/dbarray.py", line 257, in get_row
    self._storage.get(pack(PACK_NUM_TYPE, rid)))
  File "/usr/local/lib/python2.7/dist-packages/dbarray/storage.py", line 103, in get
    with self.env.begin() as txt:
lmdb.BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot