newtdb/db

Newt DB is a Python object-oriented database with JSONB-based access and search in PostgreSQL

db's Issues

Possible relstorage incompatibility

Hello. I am trying to get confident with newtdb and followed the "getting started" tutorial on the website
I installed postgresql locally (on a raspberry) and pip installed newt.db inside a virtual environment (python 3.7.3)

This is what happened:

(venv) pi@raspberry:/tmp $ python
Python 3.7.3 (default, Jan 22 2021, 20:04:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import newt.db
>>> connection = newt.db.connection('')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_db.py", line 153, in connection
    ZODB.connection(storage(dsn, **storage_options), **db_options)
  File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_db.py", line 124, in storage
    return relstorage.storage.RelStorage(Adapter(dsn, options), options=options)
  File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_adapter.py", line 13, in __init__
    driver = relstorage.adapters.postgresql.select_driver(self.options)
AttributeError: module 'relstorage.adapters.postgresql' has no attribute 'select_driver'

I modified line 13 of newt/db/_adapter.py to become
driver = relstorage.adapters.postgresql.drivers.select_driver(self.options)
but then another exception occurred:

>>> connection = newt.db.connection('')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_db.py", line 153, in connection
    ZODB.connection(storage(dsn, **storage_options), **db_options)
  File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_db.py", line 124, in storage
    return relstorage.storage.RelStorage(Adapter(dsn, options), options=options)
  File "/opt/xpend/venv/lib/python3.7/site-packages/newt/db/_adapter.py", line 13, in __init__
    driver = relstorage.adapters.postgresql.drivers.select_driver(self.options)
  File "/opt/xpend/venv/lib/python3.7/site-packages/relstorage/adapters/drivers.py", line 386, in <lambda>
    sys.modules[name])
  File "/opt/xpend/venv/lib/python3.7/site-packages/relstorage/adapters/drivers.py", line 50, in _select_driver_by_name
    driver_name = casefold(driver_name)
TypeError: descriptor 'casefold' requires a 'str' object but received a 'Options'

(venv) pi@raspberry:/tmp $ pip freeze
BTrees==4.9.2
cffi==1.15.0
newt.db==0.9.0
perfmetrics==3.2.0.post0
persistent==4.7.0
pkg-resources==0.0.0
psycopg2==2.9.2
pycparser==2.21
RelStorage==3.4.5
six==1.16.0
transaction==3.0.1
zc.lockfile==2.0
ZConfig==3.6.0
ZODB==5.6.0
zodbpickle==2.2.0
zope.dottedname==4.3
zope.interface==5.4.0

Should I downgrade some package? To which version?

Thanks in advance
Enrico

Need fine print and maybe topic on deleting objects.

Python 2 support

Has newtdb dropped Python 2 support? I get a serialization error with a very simple test case under Python 2, but not Python 3:

import logging
logging.basicConfig()

import newt.db
from BTrees._OOBTree import OOBTree

uplinkdb_con = newt.db.connection('postgresql://[email protected]/svelte')

class Profile(newt.db.Persistent):
	def __init__(self):
		self.rank = 'test'

root = uplinkdb_con.root
root.profiles = OOBTree()
root.profiles['somekey'] = Profile()
uplinkdb_con.commit()

Under Python 2 (Python 2.7.14, PostgreSQL 9.6.6) this causes:

ERROR:newt.db.jsonpickle:Failed pickle load, oid: 0, pickle starts: '\x80\x03cpersistent.mapping\nPersistentMapping\nq\x01.\x80\x03}q\x02U\x04'
Traceback (most recent call last):
  File "/home/svelte/venv/lib/python2.7/site-packages/newt/db/jsonpickle.py", line 509, in __call__
    state = unpickler.load()
  File "/home/svelte/venv/lib/python2.7/site-packages/newt/db/jsonpickle.py", line 227, in load
    for op, arg, pos in self.ops:
  File "/usr/lib64/python2.7/pickletools.py", line 1851, in genops
    code))
ValueError: at position 28, opcode 'C' unknown

Under Python 3 (Python 2 (Python 3.6.3, PostgreSQL 9.6.6) I get no output as expected. Everything looks correct in psql:

svelte=> select * from newt;
 zoid |              class_name              |                                       ghost_pickle                                       |                                            state
------+--------------------------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------
    0 | persistent.mapping.PersistentMapping | \x80036370657273697374656e742e6d617070696e670a50657273697374656e744d617070696e670a71002e | {"data": {"profiles": {"::": "persistent", "id": [1, "BTrees.OOBTree.OOBTree"], "::=>": 1}}}
    2 | __main__.Profile                     | \x8003635f5f6d61696e5f5f0a50726f66696c650a71002e                                         | {"rank": "test"}

The only virtualenv difference is Python 2 having trollius and futures.

Output of pip freeze for Python 2:

BTrees==4.5.0
cffi==1.11.5
futures==3.2.0
newt.db==0.9.0
perfmetrics==2.0
persistent==4.2.4.2
psycopg2==2.7.4
pycparser==2.18
RelStorage==2.1.0
six==1.11.0
transaction==2.2.1
trollius==2.2
zc.lockfile==1.3.0
ZConfig==3.2.0
zdaemon==4.2.0
ZEO==5.2.0
ZODB==5.4.0
zodbpickle==1.0
zope.interface==4.5.0

Output of pip freeze for Python 3:

BTrees==4.5.0
cffi==1.11.5
newt.db==0.9.0
perfmetrics==2.0
persistent==4.2.4.2
psycopg2==2.7.4
pycparser==2.18
RelStorage==2.1.0
six==1.11.0
transaction==2.2.1
zc.lockfile==1.3.0
ZConfig==3.2.0
zdaemon==4.2.0
ZEO==5.2.0
ZODB==5.4.0
zodbpickle==1.0
zope.interface==4.5.0

AttributeError: 'Instance' object has no attribute 'update'

Hi, I'm trying to resolve below traceback:

Failed pickle load, oid: 1203, pickle starts: b'\x80\x03ctransistor.io.persistence.scraper_newt\nSplashCo'
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\pythonocc\lib\site-packages\newt\db\jsonpickle.py", line 509, in __call__
    state = unpickler.load()
  File "C:\ProgramData\Miniconda3\envs\pythonocc\lib\site-packages\newt\db\jsonpickle.py", line 242, in load
    getattr(self, op.name)(arg)
  File "C:\ProgramData\Miniconda3\envs\pythonocc\lib\site-packages\newt\db\jsonpickle.py", line 320, in SETITEMS
    for i in range(0, len(marked), 2)
  File "C:\ProgramData\Miniconda3\envs\pythonocc\lib\site-packages\newt\db\jsonpickle.py", line 123, in update
    self.v.update(v)
AttributeError: 'Instance' object has no attribute 'update'

Environment is Python 3.6.5 on Windows 10, at the point in time where the class is being saved to newt, at the final stage of a gevent routine.

The class object being saved is a fairly simple wrapper with class attributes containing basic data types, str(), dict(), and otherwise there is one JSON byte string which contains an html page and also a .png of the html page.

And finally, even though I get this error, I can't really find a side-effect of it, the object is still persisted to newt, and the data from the JSON byte string seems to be useable. For example, I can recreate the .png from the byte string, pulling it back out from the persisted newt.db object.

But after picking through it a few hours today and unable to resolve it, figured I'd report it here to see if it stands out. Just let me know if I can provide more info. Thank you.

Investigate locating Newt/RelStorage tables in a named schema (other than public).

This can probably be accomplished in some SQL executed in making connections.

Problems using blobs

See: https://groups.google.com/forum/#!topic/newtdb/cQaw5d_pjGs

Relational sets

Lean on PostgreSQL to implement sets.

Objects grow _newt__parents__, which is an array of containers. When a container fetches its contents, it does so by querying objects which has it as one of their parents. Of course, the GC has to be aware of this. An object without references to it would still be non-garbage if it has any non-garbage parents.

Containers "store" their contents through these references. Containers have a length. Container conflicts are always resolvable, because they behave like lengths.

A variation on this is approximately ordered collections. Collections have min and max positions that behave like zope.min-max. When an object is added at the front, the min position is decremented and used for the object position. Similar at the back. The position becomes part of the parent pointers. Positions are not unique.