aiotinydb/aiotinydb

asyncio compatibility shim for tinydb

aiotinydb

asyncio compatibility shim for TinyDB

Enables usage of TinyDB in asyncio-aware contexts without slow syncronous IO.

See documentation on compatible version of TinyDB.

Basically all API calls from TinyDB are supported in AIOTinyDB. With the following exceptions: you should not use basic with syntax and close functions. Instead, you should use async with.

import asyncio
from aiotinydb import AIOTinyDB

async def test():
    async with AIOTinyDB('test.json') as db:
        db.insert(dict(counter=1))

loop = asyncio.new_event_loop()
loop.run_until_complete(test())
loop.close()

CPU-bound operations like db.search(), db.update() etc. are executed synchronously and may block the event loop under heavy load. Use multiprocessing if that's an issue (see #6 and examples/processpool.py for an example).

Middleware

Any middlewares you use should be async-aware. See example:

from tinydb.middlewares import CachingMiddleware as VanillaCachingMiddleware
from aiotinydb.middleware import AIOMiddleware

class CachingMiddleware(VanillaCachingMiddleware, AIOMiddlewareMixin):
    """
        Async-aware CachingMiddleware. For more info read
        docstring for `tinydb.middlewares.CachingMiddleware`
    """
    pass

If middleware requires some special handling on entry and exit, override __aenter__ and __aexit__.

Concurrent database access

Instances of AIOTinyDB support database access from multiple coroutines.

On unix-like systems, it's also possible to access one database concurrently from multiple processes when using AIOJSONStorage (the default) or AIOImmutableJSONStorage.

Installation

pip install aiotinydb

aiotinydb's Issues

UnsupportedOperation: fileno in JSONStorage.write()

Defect

AIOTinyDB raises an UnsupportedOperation: fileno error when writing

Cause

tinydb.JSONStorage.write() calls os.fsync(self._handle.fileno()). aiotinydb.AIOJSONStorage implements self._handle as an io.StringIO object.

io.StringIO has no meaningful implementation of fileno(), and raises an UnsupportedOperation error.

Reproducing

Run tox in aiotinydb.

StringIO error:

Python 3.6.5 (default, Apr  1 2018, 05:46:30)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import io
>>> io.StringIO().fileno()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
io.UnsupportedOperation: fileno
>>>

Possible solutions

• Pin TinyDB version at 3.8.1.post1
  • Not recommended as a long-term solution
• Implement AIOJSONStorage.write()
• Use an alternative to io.StringIO() that does have a meaningful implementation of fileno()

Reviving aiotinydb

Hi, according to the commit history and some comments (#6 (comment), #10 (comment)), this project is unmaintained.

Since TinyDB offers what I'm looking for, but I need an async interface, I'd like to revive this project.

There are currently some major problems:

• Incompatibility with TinyDB >= 4.0 (resulting in issues like #9)
• Incompatibility with newer versions of aiofiles
• outdated test dependencies (like nose & rednose)
• missing support for newer packaging/installation workflows (e.g. PEP 518 support)

I applied some changes in aiotinydb-revived:

• support TinyDb >= 4.0 (all tests passing)
• explicit support for Python 3.6-3.10 and PyPy3 (like TinyDB)
• move project metadata and configuration to the universal pyproject.toml format
• use flit for easier publishing
• more accessible automation and multi-version testing using nox (this could also be done using tox, but IMHO the noxfile.py format is more readable, especially when using it for both linting and testing against multiple Python versions)

What I plan to do:

• make pylint ruleset less verbose
• support #6
• improve type hints
• add additional tests (the current tests already offer 100 % code coverage, but I think there are still some untested edge cases)
• use a maintained alternative to bumpversion
• maybe: use black and isort for formatting

As far as I can tell from the issues section, you don't use this library anymore and aren't interested to work on it — correct me if I'm wrong.

If you are interested, I'm willing to maintain this project in the future.

This would require:

• clarification of the project's license
  • according to the LICENSE.md file and a comment, this library is licensed under the terms of LGPL-3.0-only
  • by contrast, the package metadata and the file headers state that it is licensed under the terms of GPL-3.0-or-later
  • I suggest keeping the LGPL and updating the file headers accordingly
• ownership transfer or granting write access for both the repo and the pypi project

If you prefer, I can also create a PR or continue as a standalone project (this would also require addressing the licensing ambiguity).

New PyPI release?

Latest version in the master is 2.0.0, but on PyPI is still 1.2.2, is there a new release planned?

Computationaly-intense search operations should be run in a thread

With the increasing number of entries it gets more and more expensive to perform searches. We should move the computations to a separate thread (or even a process) to get out of main loop's way.

async calls should take loop as kwarg

In a situation where there are 2 or more main loops one should be able to specify, which to use.

Any info on the license choice?

I can't use LGPL on python libraries - not due to my own sensibility fyi - it makes life complicated downstream since it makes the code incompatible with 90% of python projects from a license perspective.

These days I am more hesitant to stray away from the standard library at all. There's 3 reasons I can think of why I felt a bit curious:

1. tinydb itself uses MIT: https://github.com/msiemens/tinydb/blob/master/LICENSE

2. A lot of the https://github.com/aio-libs/ use permissive licenses

3. Compare on PyPI the usage across libraries to see what other open source projects use:

   https://pypi.org/search/?q=&o=&c=License+%3A%3A+OSI+Approved+%3A%3A+GNU+Library+or+Lesser+General+Public+License+%28LGPL%29

   It's hard to do an "or" search, but there's about 5 different LGPL licenses on file

Is there any reason why this license used LGPL rather than the more compatible ones? was there a conversation before somewhere?

What do you think about the license?

PyPI deploy fail

truncate does not work

    async with AIOTinyDB(DATABASE) as db:
            db.truncate()
            self.logger.info(f"Truncated: {db}")
            return True

response:
AttributeError: 'Table' object has no attribute 'truncate'

there does not seem to be any table object to truncate