bmuller/twistar

Twistar is an object-relational mapper (ORM) for Python that uses the Twisted library to provide asynchronous DB interaction.

twistar: Asynchronous Python ORM

The Twistar Project provides an ActiveRecord (ORM) pattern interface to the Twisted Project's RDBMS library. This file contains minimal documentation - see the project home page at http://findingscience.com/twistar for more information.

Installation

pip install twistar

Usage

Your database must be one of: MySQL, PostgreSQL, or SQLite. The only DBAPI modules supported by Twistar are: MySQLdb, psycopg2, and sqlite3 - at least one of these must be installed.

Here's the obligatory TL;DR example of creating a User record, assuming that there is a table named "users" with varchar columns for first_name and last_name and an int age column:

#!/usr/bin/env python
from twisted.enterprise import adbapi
from twistar.registry import Registry
from twistar.dbobject import DBObject
from twisted.internet import reactor

class User(DBObject):
     pass

def done(user):
     print "A user was just created with the name %s" % user.first_name
     reactor.stop()

# Connect to the DB
Registry.DBPOOL = adbapi.ConnectionPool('MySQLdb', user="twistar", passwd="apass", db="twistar")

# make a user
u = User()
u.first_name = "John"
u.last_name = "Smith"
u.age = 25

# Or, use this shorter version:
u = User(first_name="John", last_name="Smith", age=25)

# save the user
u.save().addCallback(done)

reactor.run()

Then, finding this user is easy:

def found(users):
    print "I found %i users!" % len(users)
    for user in users:
        print "User: %s %s" % (user.first_name, user.last_name)

u = User.findBy(first_name="John", age=25).addCallback(found)

This is a very simple example - see http://findingscience.com/twistar for more complicated examples and additional uses.

Testing

To run unit-tests you simply require Tox

To run the tests:

tox

By default, the tests are run with the database driver sqlite3. To change this, set the DBTYPE environment variable:

DBTYPE=postgres trial twistar
DBTYPE=mysql trial twistar

You'll need a database named "twistar" for each of those tests (or you can change the dbname, user, etc in the <db type>_config.py file in the tests folder).

Documentation

If you intent on generating API documentation, you will need pydoctor. If you want to generate the user documentation, you will need to install Twisted Lore.

To generate documentation:

make docs

Then open the docs/index.html file in a browser.

twistar's Issues

toHash() all

How to make obj.toHash(all) without specify all columns?

Unique and only specified fields in .find()

How can I get only specified fields in .find() method? Can I get only unique rows?

toHash and Decimal

.toHash() function return PostgreSQL numeric() type as Decimal, AutobahnPython can't JSON serialize it.

.toHash() result: { 'longitude': Decimal('41.036283') }

Autobahn error:
wamp.error.invalid_payload
WAMP serialization error (Decimal('41.036283') is not JSON serializable)

easy_install twistar fails

(venv)~/work/myproj$ easy_install twistar
Searching for twistar
Reading http://pypi.python.org/simple/twistar/
Reading http://findingscience.com/twistar
Best match: twistar 1.0.macosx-10.8-x86-64
Downloading http://pypi.python.org/packages/any/t/twistar/twistar-1.0.macosx-10.8-x86_64.tar.gz#md5=c35939bda8523deecd9c3c8384e813f4
Processing twistar-1.0.macosx-10.8-x86_64.tar.gz
error: Couldn't find a setup script in /var/folders/5g/_j29fpt55fx79_y642308n880000gp/T/easy_install-vO2hDM/twistar-1.0.macosx-10.8-x86_64.tar.gz

I think this might be because you've uploaded the egg as a binary distribution (bdist). Instead, use:

python setup.py sdist upload

joinWheres should use parenthesis to preserve precedence

hi,

i think in twistar.utils.joinWheres a couple of parenthesis are missing. It should be:

statement = ["(%s) %s (%s)" % (wone[0], joiner, wtwo[0])]

testcase:

@inlineCallbacks
def test_joinWheres_precedence(self):
    # this relies on initDB default test setup
    yield User(first_name="Second").save()

    first = ['first_name = ?', "First"]
    last = ['last_name = ?', "Last"]
    second = ['first_name = ?', "Second"]

    last_or_second = twistar.utils.joinWheres([last, second], joiner='OR')
    where = twistar.utils.joinWheres([first, last_or_second], joiner='AND')

    results = yield User.count(where=where)
    self.assertEqual(1, results)

the result is different wheter you have parenthesis in joinWheres or not.

Idea on how to encapsulate many ORM operations into a transaction ?

I'm thinking a way to implement a transaction handling to protect several operations,
somethink like AR does:

Account.transaction do
balance.save!
account.save!
end

(see http://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html for other examples).

this is very useful when you want to do several operations with the orm and rollback if any of these operations fails. The rollback can be automatic (like twisted txns do) and an exception can be raised back to the application.

what do you suggest can be the best way to implement into twistar ?
I can start looking at it...

Make Twistar available on pypi.python.org

Currently I cannot pip install twistar/easy_install twistar

Dead or Alive ?

What's the future plan for this project. Seems the development halted since a few years. Are there "recommended" alternatives that we should be looking at ?

eager loading relationships

Is it possible to load the model with its relationships directly ? something like User.with('pictures') to avoid a huge callback chain ?

This example can be ok for one relationship, but what about models having 4 or 5 relationships depth:

def foundPicture(picture):
     print picture.file

def foundUser(user):
     user.picture.get().addCallback(foundPicture)

User.find(where=['first_name = ?', "Bob"], limit=1).addCallback(foundUser)

Imagine I have this structure: User [HAS_MANY] Picture [HAS_ONE] Path [HAS_ONE] descriptor (doesn't make sens, just to see how to do this correctly without running 4 queries)

Please clarify code license

Hi,

setup.py describe the project as using GPLv3 and LICENSE files describe a BSD style license.

Could you please resolve this conflict?

Thanks!

twistar.validation.presenceOf is broken

class MyModel(DBObject):
    field = None
MyModel.validatesPresenceOf('field')

assert MyModel().isValid()  # succeeds

this is because presenceOf has:

    if getattr(obj, name, "") == "":

but should have:

    if getattr(obj, name, None) is None:

optionally you can add checking for "" depending on how you want to interpret empty strings:

if getattr(obj, name, None) in (None, ""):

The statement generated by utils.dictToWhere is not suitable for None value

The statement should be IS NULL for None value, not = NULL. As a result BDObject.findBy() returns [] in that case. The following test succeeds if name is a string:

diff --git a/twistar/tests/test_dbobject.py b/twistar/tests/test_dbobject.py
index 6611e90..d7001d9 100644
--- a/twistar/tests/test_dbobject.py
+++ b/twistar/tests/test_dbobject.py
@@ -84,6 +84,12 @@ class DBObjectTest(unittest.TestCase):
         resultids = [result.id for result in results]
         self.assertEqual(ids, resultids)

+    @inlineCallbacks
+    def test_null(self):
+        name = None
+        user = yield User(first_name=name).save()
+        users = yield User.findBy(first_name=name)
+        self.assertEqual(user, users[0])

     @inlineCallbacks
     def test_count(self):

find() variant returning a generator?

Hi,

In my code I sometimes have to go thought all objects of a certain type. By using the DBObject.all() function it seems all objects are instantiated in a large array. Would it be possible to have all() or find() return a generator function, so that code like
for object in Object.all():
object.doSomething()
would involve only one instance of Object at a time, which could be garbage collected after each iteration in the loop?

Jan-Pascal

missing import in dbconfig/base.py

hi,

a very small problem in your code:
in InteractionBase.refreshObj() method you use the CannotRefreshError without importing the exceptions module first.

a trivial from twistar.exceptions import CannotRefreshError fixes the problem

bye,
flavio

Bad column quoting for postgres

twistar uses backticks to quote column names, which doesn't work for postgres.

Non-working BelongsTo.clear

If a BelongsTo relationship is cleared, the foreign-key atrribute is set to None, but such attributes are ignored during save(). The
test_clear_belongs_to is broken because it does not call refresh, it checks only in-memory value. This testcase demonstrates the problem:

@inlineCallbacks
    def test_clear_belongs_to(self):
        picture = yield Picture(name="a pic", size=10, user_id=self.user.id).save()
        yield picture.user.clear()
        user = yield picture.user.get()
        self.assertEqual(user, None)
        yield picture.refresh()
        user = yield picture.user.get()
        self.assertEqual(user, None)

twistar.tests.test_dbobject.DBObjectTest.test_delete_all fails

Just installed twistar via pip and ran the tests.

trial twistar 
twistar.tests.test_dbconfig
  DBConfigTest
    test_colname_escaping ...                                              [OK]
    test_delete ...                                                        [OK]
    test_insert ...                                                        [OK]
    test_insert_many ...                                                   [OK]
    test_insert_obj ...                                                    [OK]
    test_select ...                                                        [OK]
    test_unicode_logging ...                                               [OK]
    test_update ...                                                        [OK]
    test_update_obj ...                                                    [OK]
twistar.tests.test_dbobject
  DBObjectTest
    test_afterInit ...                                                     [OK]
    test_all ...                                                           [OK]
    test_beforeDelete ...                                                  [OK]
    test_count ...                                                         [OK]
    test_count_all ...                                                     [OK]
    test_creation ...                                                      [OK]
    test_delete ...                                                        [OK]
    test_delete_all ...                                                 [ERROR]
    test_find ...                                                          [OK]
    test_findBy ...                                                        [OK]
    test_findOrCreate ...                                                  [OK]
    test_loadRelations ...                                                 [OK]
    test_refresh ...                                                       [OK]
    test_update ...                                                        [OK]
    test_validation ...                                                    [OK]
    test_validation_function ...                                           [OK]
twistar.tests.test_relationships
  RelationshipTest
    test_belongs_to ...                                                    [OK]
    test_clear_belongs_to ...                                              [OK]
    test_clear_habtm ...                                                   [OK]
    test_clear_has_many ...                                                [OK]
    test_clear_jointable_on_delete_habtm ...                               [OK]
    test_clear_jointable_on_delete_habtm_with_custom_args ...              [OK]
    test_habtm ...                                                         [OK]
    test_habtm_count ...                                                   [OK]
    test_habtm_count_with_args ...                                         [OK]
    test_habtm_get_with_args ...                                           [OK]
    test_has_many ...                                                      [OK]
    test_has_many_count ...                                                [OK]
    test_has_many_count_nocache ...                                        [OK]
    test_has_many_count_with_args ...                                      [OK]
    test_has_many_get_with_args ...                                        [OK]
    test_has_one ...                                                       [OK]
    test_polymorphic_get ...                                               [OK]
    test_polymorphic_set ...                                               [OK]
    test_set_belongs_to ...                                                [OK]
    test_set_habtm ...                                                     [OK]
    test_set_habtm_blank ...                                               [OK]
    test_set_has_many ...                                                  [OK]
    test_set_has_one ...                                                   [OK]
    test_set_on_unsaved ...                                                [OK]
twistar.tests.test_transactions
  TransactionTest
    test_doubleInsert ...                                                  [OK]
    test_findOrCreate ...                                                  [OK]
    test_success ...                                                       [OK]
twistar.tests.test_utils
  UtilsTest
    test_joinMultipleWheres ...                                            [OK]
    test_joinMultipleWheres_empty_arg ...                                  [OK]
    test_joinMultipleWheres_single_where ...                               [OK]
    test_joinWheres_precedence ...                                         [OK]

===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: exceptions.TypeError: <bound method DBObjectTest.test_delete_all of <twistar.tests.test_dbobject.DBObjectTest testMethod=test_delete_all>> is a generator function and therefore will never run

twistar.tests.test_dbobject.DBObjectTest.test_delete_all
-------------------------------------------------------------------------------
Ran 56 tests in 4.381s

FAILED (errors=1, successes=55)

Using JOINs

Hi again,

There doesn't appear to be a group for discussing things in Twistar, so I ended up posting this here: Is it possible to mimic behaviour like "SELECT * FROM table1 NATURAL JOIN table2" in Twistar? Couldn't find anything like that in the documentation..

Thanks!

Allow multiple pools

Hi again,

I'm pretty sure this is really not possible in the module: I have an app that works with two different databases (for various reasons), but obviously Twistar only allows one Registry.DBPOOL connection.. Is there anything that could be done to increase this number? Or either way, what would you recommend to do in such a scenario?

Thanks again,
Martin

Add an updateAll method

...to complement deleteAll. Updating stuff en-masse is inefficient right now.

Add a class method as callback

Hi there,

I'm running something like

class MyClass:
  def action(self, data):
    print data

  def fetch(self):
   User.find(where=['uid = ?', self.uid]).addCallback(self.action)

But the self.action method never gets called.. It appears that only callbacks that are accepted are global functions. Is there any way to make this work with the current code and/or is this a bug/feature of Twistar?

Cheers

dbobject.count() breaks dbobject.save()

It appears that when bdobject.count() is called count() gets added to the object schema by some method. After the call to count() is executed calls to save() will throw 'table does not have a column count()' error. I am using the development branch of twistar, twisted 11.0, sqlite, and python 2.7.1.

Debian/Ubuntu packaging

While working on GlobaLeaks and evaluating twistar usage (globaleaks/whistleblowing-software#1551) we are now evaluating to provide support on debian packaging of twistar in order too speed up the possibility to have it both in debian stretch and ubuntu xenial.

This ticket is to keep track of the activities related activities

https://lists.debian.org/debian-python/2015/10/msg00267.html

@bmuller: are you already following this activities?

\cc @fpietrosanti

Please release version 1.2

Thanks for twistar!
Would you mind releasing version 1.2 fixing the ambiguity in setup.py with respect to licensing? I am currently packaging for Debian and need this to be fixed.
BTW the version string in twistar/init.py doesn't match the version in setup.py.
Thanks a lot for considering!

Please add a tag for 1.1

This can help in tracking the code published on PyPi.

Thanks!

mongo db support

it's excellent. any plan for mongo orm support?

afterInit is not called

twistar.util.createInstances is the only place where it's called which means Model() does not invoke afterInit. This is in implicit violation of what the docstring of DBObject.__init__ says:

def __init__(self, **kwargs):
    """
    Constructor.  DO NOT OVERWRITE.  Use the L{DBObject.afterInit} method.

Is this project dead?

Should I call Horatio Caine?

Python3 compatibility

While working on GlobaLeaks and evaluating twistar usage (globaleaks/whistleblowing-software#1551) we are now evaluating the code quality of twistar in relation to python3.

This ticket is to keep track of the activities related to make twistar compatible with python3.

@bmuller: With commit #59 i've now made possible to test against python3 and it should be easy now for you to fix the support by simply looking at the python3 errors at: https://travis-ci.org/evilaliv3/twistar

\cc @fpietrosanti

Changelog?

Could we get proper CHANGELOG please? Currently tracking changes based on commits is prone to errors.

refresh function

Hi!

Maybe I did not understand, but I can not update the object. The procedure is as follows:

1. I do SELECT (find ()), get the object.
2. Change some properties in the properties of this object
3. Run the refresh.

error:
exceptions.AttributeError: 'list' object has no attribute 'keys'

Even when I try to call the refresh an object without changing its properties, I get the same error.

Thank you.

DBObject.__hash__ missing

DBObject instances cannot currently be used as keys of dictionaries that span more than one transaction.

Add support for additional information on HABTM relationships

Twistar currently lacks support on adding information on HABTM relationship assosiations.
Consider the following example: An Actor plays roles in Movies. Each Actor can appear in multiple Movies, but play different Roles on each Movie. The DB table to represent this HABTM relationship would look like this:
Table Actor(name)
|
Table plays_in(movie_id)(role)
|
Table Movie(name)

Such a mapping is currently not possible with Twistar because the Role field cannot be accessed on a HABTM relationship, but it should be.

Twistar calling lastval() causing error in psycopg2

At the end onf an insertion query, twistar calls lastval() and causes the postgres driver to fail. how to solve that?

2016-10-06` 11:08:02+0200 [-] Log opened.
2016-10-06 11:08:02+0200 [-] MAIN: Starting the reactor
2016-10-06 11:08:02+0200 [-] TWISTAR query: SELECT * FROM my_user WHERE user_id = %s LIMIT 1
2016-10-06 11:08:02+0200 [-] TWISTAR args: 009a65e7-a6a8-4de4-ad1a-87ac20e4073e
2016-10-06 11:08:02+0200 [-] TWISTAR query: SELECT * FROM my_user LIMIT 1
2016-10-06 11:08:02+0200 [-] TWISTAR query: INSERT INTO my_user ("username","user_id") VALUES (%s,%s)
2016-10-06 11:08:02+0200 [-] TWISTAR args: myusername,009a65e7-a6a8-4de4-ad1a-87ac20e4073e
2016-10-06 11:08:02+0200 [-] TWISTAR query: SELECT lastval()
2016-10-06 11:08:02+0200 [-] Unhandled error in Deferred:
2016-10-06 11:08:02+0200 [-] Unhandled Error
Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "/usr/lib/python2.7/site-packages/twisted/_threads/_threadworker.py", line 46, in work
        task()
      File "/usr/lib/python2.7/site-packages/twisted/_threads/_team.py", line 190, in doWork
        task()
    --- <exception caught here> ---
      File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 246, in inContext
        result = inContext.theWork()
      File "/usr/lib/python2.7/site-packages/twisted/python/threadpool.py", line 262, in <lambda>
        inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
      File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/usr/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
        return func(*args,**kw)
      File "/usr/lib/python2.7/site-packages/twisted/enterprise/adbapi.py", line 477, in _runInteraction
        compat.reraise(excValue, excTraceback)
      File "/usr/lib/python2.7/site-packages/twisted/enterprise/adbapi.py", line 467, in _runInteraction
        result = interaction(trans, *args, **kw)
      File "/usr/lib/python2.7/site-packages/twistar/dbconfig/base.py", line 348, in _doinsert
        self.insert(tablename, vals, txn)
      File "/usr/lib/python2.7/site-packages/twistar/dbconfig/base.py", line 192, in insert
        return self.getLastInsertID(txn)
      File "/usr/lib/python2.7/site-packages/twistar/dbconfig/postgres.py", line 9, in getLastInsertID
        self.executeTxn(txn, q)
      File "/usr/lib/python2.7/site-packages/twistar/dbconfig/base.py", line 78, in executeTxn
        return txn.execute(query, *args, **kwargs)
    psycopg2.OperationalError: ERRORE:  lastval non è stato ancora definito in questa sessione

last line says "lastval not yet defined in this session"

how to avoid that?

afterInit called twice + errors in it not caught (breaks compat.)

(I will describe both the issue and the solution here because I already have a patch for this)

After I added a call to afterInit in DBObject.__init__, it (now obviously) started to be called twice: once per createinstances and once per __init__. So I wanted to remove the call to afterInit in createInstances. Seeing the code in createInstances though made me realise that my added line in __init__ swallows errors in afterInit. But afterInit needs to be asynchronous (if not for any practical pruposes—but there are—as per the doc, at least), which means it cannot really be called from __init__ because __init__ is not allowed to return anything (and thus not allowed to return a Deferred). However, __new__ does not suffer from this limitation. So the end solution is to put the call to afterInit into DBObject.__new__.

As a bonus, I also updated the usage of DeferredList in createInstances to include fireOnFirstErrback=True, consumeErrors=True so that any exceptions wouldn't go unnoticed.

validatesUniquenessOf triggers Segmentation Fault in a particular case

The seg fault is triggered when you try to validate the uniqueness of a random string ("xn--tan-test-20160307154100-6tc060eub.ro"). validatePresenceOf works well with the same string provided. By looking at the stack trace, I think the problem is related to the Inflector but I am not 100% sure.

I will attach the stack trace below:

Thank you

`(gdb) bt
#0 call_function (f=, throwflag=) at Python/ceval.c:3796
#1 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#2 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x1d18288, globals=, locals=, args=, argcount=1,

kws=0x1ddfdf0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#3 0x0000003cda26ad9d in function_call (func=<function at remote 0x1d25848>, arg=

(<Domain(_config=<PostgreSQLDBConfig(txn=None) at remote 0x1d78ea8>, errors=<Errors(infl=<Inflector(Inflector=<English() at remote 0x1d6ff38>) at remote 0x1d6ffc8>) at remote 0x1ddfce0>, user_id=277, fqdn='xn--tan-test-20160307154100-6tc060eub.ro', _deleted=False, id=None) at remote 0x1d70b50>,), kw={})
at Objects/funcobject.c:524

#4 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x1d25848>, arg=, kw=) at Objects/abstract.c:2492
#5 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107
#6 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493
#7 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7378, globals=, locals=, args=,

argcount=1, kws=0x1e09cc8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#8 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#9 call_function (f=, throwflag=) at Python/ceval.c:3815
#10 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#11 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x1d0d288, globals=, locals=, args=, argcount=1,

kws=0x1e09ae0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#12 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#13 call_function (f=, throwflag=) at Python/ceval.c:3815
#14 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#15 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x1d05c60, globals=, locals=, args=, argcount=1,

kws=0x17e4740, kwcount=0, defs=0x0, defcount=0, closure=(<cell at remote 0x1d7b4e8>,)) at Python/ceval.c:3044

#16 0x0000003cda26ad9d in function_call (func=<function at remote 0x1d73320>, arg=(True,), kw={}) at Objects/funcobject.c:524
#17 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x1d73320>, arg=, kw=) at Objects/abstract.c:2492
#18 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107
#19 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493
#20 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7d50, globals=, locals=, args=,

argcount=1, kws=0x1e08ab8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#21 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#22 call_function (f=, throwflag=) at Python/ceval.c:3815
#23 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#24 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7c60, globals=, locals=, args=,

argcount=2, kws=0x1e088e8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#25 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#26 call_function (f=, throwflag=) at Python/ceval.c:3815
#27 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#28 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7918, globals=, locals=, args=,

argcount=2, kws=0x1e08708, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#29 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#30 call_function (f=, throwflag=) at Python/ceval.c:3815
#31 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#32 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f02583f0, globals=, locals=, args=,

argcount=4, kws=0x11d9880, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#33 0x0000003cda26ad9d in function_call (func=<function at remote 0x7f01f0263410>, arg=

(<DeferredList(finishedCount=2, _chainedTo=None, fireOnOneCallback=False, resultList=[(True, None), (True, None)], _runningCallbacks=True, _canceller=None, callbacks=[((<function at remote 0x1d6ced8>, (), {}), (<function at remote 0x1d6ced8>, (...), {...}))], fireOnOneErrback=False, result=True, _deferredList=[<Deferred(_chainedTo=None, called=True, _canceller=None, callbacks=[], result=None, _runningCallbacks=False) at remote 0x1d6fef0>, <DeferredList(finishedCount=1, _chainedTo=None, fireOnOneCallback=False, resultList=[(True, None)], _runningCallbacks=True, _canceller=None, callbacks=[], fireOnOneErrback=False, result=None, _deferredList=[<Deferred(_chainedTo=None, called=True, _canceller=None, callbacks=[], result=None, _runningCallbacks=True) at remote 0x1d6fea8>], called=True, consumeErrors=False) at remote 0x1d780e0>], called=True, consumeErrors=False) at remote 0x1d78e60>, None, 1, True), kw={}) at Objects/funcobject.c:524

#34 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x7f01f0263410>, arg=, kw=) at Objects/abstract.c:2492

---Type to continue, or q to quit---
#35 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107
#36 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493
#37 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7d50, globals=, locals=, args=,

argcount=1, kws=0x1dcf7b8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#38 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#39 call_function (f=, throwflag=) at Python/ceval.c:3815
#40 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#41 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7c60, globals=, locals=, args=,

argcount=2, kws=0x1e076b8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#42 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#43 call_function (f=, throwflag=) at Python/ceval.c:3815
#44 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#45 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7918, globals=, locals=, args=,

argcount=2, kws=0x1de5008, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#46 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#47 call_function (f=, throwflag=) at Python/ceval.c:3815
#48 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#49 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f02583f0, globals=, locals=, args=,

argcount=4, kws=0x1d4d8b0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#50 0x0000003cda26ad9d in function_call (func=<function at remote 0x7f01f0263410>, arg=

(<DeferredList(finishedCount=1, _chainedTo=None, fireOnOneCallback=False, resultList=[(True, None)], _runningCallbacks=True, _canceller=None, callbacks=[], fireOnOneErrback=False, result=None, _deferredList=[<Deferred(_chainedTo=None, called=True, _canceller=None, callbacks=[], result=None, _runningCallbacks=True) at remote 0x1d6fea8>], called=True, consumeErrors=False) at remote 0x1d780e0>, None, 0, True), kw={}) at Objects/funcobject.c:524

#51 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x7f01f0263410>, arg=, kw=) at Objects/abstract.c:2492
#52 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107
#53 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493
#54 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7d50, globals=, locals=, args=,

argcount=1, kws=0x1de4bc8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#55 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#56 call_function (f=, throwflag=) at Python/ceval.c:3815
#57 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#58 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7c60, globals=, locals=, args=,

argcount=2, kws=0x1de44b8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#59 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#60 call_function (f=, throwflag=) at Python/ceval.c:3815
#61 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#62 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f06e7918, globals=, locals=, args=,

argcount=2, kws=0x1dbab50, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#63 0x0000003cda26ad9d in function_call (func=<function at remote 0x7f01f0264c08>, arg=

(<Deferred(_chainedTo=None, called=True, _canceller=None, callbacks=[], result=None, _runningCallbacks=True) at remote 0x1d6fea8>, None), kw={})
at Objects/funcobject.c:524

#64 0x0000003cda243c63 in PyObject_Call (func=<function at remote 0x7f01f0264c08>, arg=, kw=) at Objects/abstract.c:2492
#65 0x0000003cda2d4460 in ext_do_call (f=, throwflag=) at Python/ceval.c:4107
#66 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2493
#67 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x16db7b0, globals=, locals=, args=, argcount=1,

kws=0x1dcf4a8, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#68 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#69 call_function (f=, throwflag=) at Python/ceval.c:3815
#70 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#71 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x16e0a80, globals=, locals=, args=, argcount=1,

kws=0x1dcde80, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#72 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890

---Type to continue, or q to quit---
#73 call_function (f=, throwflag=) at Python/ceval.c:3815
#74 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#75 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x16e0990, globals=, locals=, args=, argcount=1,

kws=0x1dcdc90, kwcount=0, defs=0x16f7728, defcount=1, closure=0x0) at Python/ceval.c:3044

#76 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#77 call_function (f=, throwflag=) at Python/ceval.c:3815
#78 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#79 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f273fcd8, globals=, locals=, args=,

argcount=5, kws=0x1dcdaa8, kwcount=0, defs=0x7f01f06ce188, defcount=2, closure=0x0) at Python/ceval.c:3044

#80 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#81 call_function (f=, throwflag=) at Python/ceval.c:3815
#82 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#83 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f2745030, globals=, locals=, args=,

argcount=4, kws=0x1dbe958, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#84 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#85 call_function (f=, throwflag=) at Python/ceval.c:3815
#86 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#87 0x0000003cda2d6b7f in fast_function (f=, throwflag=) at Python/ceval.c:3880
#88 call_function (f=, throwflag=) at Python/ceval.c:3815
#89 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#90 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f273ff30, globals=, locals=, args=,

argcount=1, kws=0x140f258, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#91 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#92 call_function (f=, throwflag=) at Python/ceval.c:3815
#93 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#94 0x0000003cda2d6b7f in fast_function (f=, throwflag=) at Python/ceval.c:3880
#95 call_function (f=, throwflag=) at Python/ceval.c:3815
#96 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#97 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f2745a08, globals=, locals=, args=,

argcount=2, kws=0x1416a60, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#98 0x0000003cda2d5a94 in fast_function (f=, throwflag=) at Python/ceval.c:3890
#99 call_function (f=, throwflag=) at Python/ceval.c:3815
#100 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#101 0x0000003cda2d6b7f in fast_function (f=, throwflag=) at Python/ceval.c:3880
#102 call_function (f=, throwflag=) at Python/ceval.c:3815
#103 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2453
#104 0x0000003cda2d7647 in PyEval_EvalCodeEx (co=0x7f01f3e8a558, globals=, locals=, args=,

argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3044

#105 0x0000003cda2d7722 in PyEval_EvalCode (co=, globals=, locals=) at Python/ceval.c:545
#106 0x0000003cda2f1b9c in run_mod (mod=, filename=, globals=

{'run': <function at remote 0x7f01f2745230>, '__builtins__': <module at remote 0x7f01f3f3c868>, '__file__': '/usr/bin/twistd', '__package__': None, 'sys': <module at remote 0x7f01f3f3cc20>, '__name__': '__main__', 'os': <module at remote 0x7f01f3eef9b8>, '__doc__': None}, locals=
{'run': <function at remote 0x7f01f2745230>, '__builtins__': <module at remote 0x7f01f3f3c868>, '__file__': '/usr/bin/twistd', '__package__': None, 'sys': <module at remote 0x7f01f3f3cc20>, '__name__': '__main__', 'os': <module at remote 0x7f01f3eef9b8>, '__doc__': None}, flags=<value optimized out>,
arena=<value optimized out>) at Python/pythonrun.c:1358

#107 0x0000003cda2f1c70 in PyRun_FileExFlags (fp=0x1097630, filename=0x7ffd72c495f3 "/usr/bin/twistd", start=, globals=

{'run': <function at remote 0x7f01f2745230>, '__builtins__': <module at remote 0x7f01f3f3c868>, '__file__': '/usr/bin/twistd', '__package__': None, 'sys': <module at remote 0x7f01f3f3cc20>, '__name__': '__main__', 'os': <module at remote 0x7f01f3eef9b8>, '__doc__': None}, locals=
{'run': <function at remote 0x7f01f2745230>, '__builtins__': <module at remote 0x7f01f3f3c868>, '__file__': '/usr/bin/twistd', '__package__': None, 'sys': <module at remote 0x7f01f3f3cc20>, '__name__': '__main__', 'os': <module at remote 0x7f01f3eef9b8>, '__doc__': None}, closeit=1, flags=0x7ffd72c48520)
at Python/pythonrun.c:1344

#108 0x0000003cda2f315c in PyRun_SimpleFileExFlags (fp=0x1097630, filename=0x7ffd72c495f3 "/usr/bin/twistd", closeit=1, flags=0x7ffd72c48520)

---Type to continue, or q to quit---
at Python/pythonrun.c:948
#109 0x0000003cda2ff892 in Py_Main (argc=, argv=) at Modules/main.c:618
#110 0x00000039cd21ed5d in __libc_start_main (main=0x400710

, argc=6, ubp_av=0x7ffd72c48648, init=, fini=,

rtld_fini=<value optimized out>, stack_end=0x7ffd72c48638) at libc-start.c:226

#111 0x0000000000400649 in _start ()`

Please provide git tags for releases

Hi,

I'm looking into packaging twistar for Debian. Could you please provide git tags at least for the latest 1.3 release to make it clear which commit exactly is the 1.3 release?

Thank!

Jan-Pascal

Support for non "id" primary keys

Currently twistar assumes that the primary key of a table is always called id. This of course not the case and twistar should provide an option to specify the primary key of a table.

about delete() and deleteAll()

Hi,
I'm trying to add a beforeDelete method to the delete chain. In my branch you can see what I've done.
The beforeDelete is fully usable on delete() but I'd like know what you think about adding it on deleteAll(). If I modify deleteAll I need to istantiate all the DBObjects before deletion and call on everyone the delete(). This will degrade performances... is it ok for you?

During my experiments I noticed that HABTM jointable is not cleared when an Object involved in this type of relation is deleted. I have fixed this but the code is mixed with the beforeDelete stuff.

bye

TransactionTest.test_findOrCreate fails with "database is locked"

This is happening at commit c19c222 so before any of my changes. Running on OS X 10.7 with Python 2.7.3 from installed using Homebrew.

[ERROR]
Traceback (most recent call last):
  File "/Users/erik.allik/.virtualenvs/twistar/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.7-x86_64.egg/twisted/python/threadpool.py", line 167, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/Users/erik.allik/.virtualenvs/twistar/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.7-x86_64.egg/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/Users/erik.allik/.virtualenvs/twistar/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.7-x86_64.egg/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/Users/erik.allik/.virtualenvs/twistar/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-macosx-10.7-x86_64.egg/twisted/enterprise/adbapi.py", line 448, in _runInteraction
    result = interaction(trans, *args, **kw)
  File "/Volumes/encrypted/work/twistar/twistar/utils.py", line 27, in _transaction
    raise TransactionError, str(e)
twistar.exceptions.TransactionError: database is locked

twistar.tests.test_transactions.TransactionTest.test_findOrCreate

This only happens from time to time (about 30% of times).

deleteAll() does not trigger beforeDelete()

When using deleteAll() the method beforeDelete() is not called on the affected objects.

Change type checks to use "isinstance" instead of "is"

Hello bmuller,

I'm using twistar in my project, and encountered an issue; I'm using the 2/3 compatibility pip package "future", and it includes a wrapper around 2's list class to behave like 3's. As long as all type-checking is done via "isinstance", this doesn't cause any problems. However, you're using "is list" here: dbconfig/base.py#L392 . This causes AssertionErrors to get thrown when passing in a list constructed from the future.builtins package. Considering changing the other type-checks to use isinstance might not be a bad idea, as it's the preferred type-check is most cases, I've found.

twistar.validation.lengthOf fails when the attribute is None

class `InteractionBase` method `log` fails if received tuple, why it receives a tuple?

Hello, our production code failed suddenly with strange error expected string, list found

[Failure instance: Traceback: <type 'exceptions.TypeError'>: sequence item 0: expected string, list found
/usr/lib64/python2.7/threading.py:812:__bootstrap_inner
/usr/lib64/python2.7/threading.py:765:run
/usr/lib64/python2.7/site-packages/twisted/_threads/_threadworker.py:46:work
/usr/lib64/python2.7/site-packages/twisted/_threads/_team.py:190:doWork
--- <exception caught here> ---
/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py:250:inContext
/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py:266:<lambda>
/usr/lib64/python2.7/site-packages/twisted/python/context.py:122:callWithContext
/usr/lib64/python2.7/site-packages/twisted/python/context.py:85:callWithContext
/usr/lib64/python2.7/site-packages/twisted/enterprise/adbapi.py:475:_runInteraction
/usr/lib64/python2.7/site-packages/twisted/enterprise/adbapi.py:465:_runInteraction
/usr/lib/python2.7/site-packages/twistar/dbconfig/base.py:144:_doselect
/usr/lib/python2.7/site-packages/twistar/dbconfig/base.py:79:executeTxn
/usr/lib/python2.7/site-packages/twistar/dbconfig/base.py:47:log

so, I traced my data all the way in the code and didn't find out where it's turned to a tuple, and then the logger fails.

def log(self, query, args, kwargs):
        """
        Log the query and any args or kwargs using C{twisted.python.log.msg} if
        C{InteractionBase.LOG} is True.
        """
        if not InteractionBase.LOG:
            return
        log.msg("TWISTAR query: %s" % query)
        if len(args) > 0:
            try:
                log.msg("TWISTAR args: %s" % ",".join(args))
            except Exception as e:
                pass
        elif len(kwargs) > 0:
            log.msg("TWISTAR kargs: %s" % str(kwargs))

I have added this try/catch so that the code passes, but! how can I find out why is query being turned to a tuple?

this is the query that is sent to the log function:

print str(query), type(query)
(['192.168.56.31'],): <type 'tuple'>

here is how Im using the data:

d = Attackers.find(where=['ip = ?', ip],limit=1)
        d.addCallback(self._check_attacker, ip)
        d.addErrback(self.handle_err)

where ip is a string of "192.168.56.31". the funny thing is the code has been running up until now, and I can't get what has changed.

I have traced the data as follows:

dbobject.py - > select function 

loger("in find where is: " + str(where))
in find where is: ['ip = ?', '192.168.56.31']

dabse.py -> select function:

select function where is: ['ip = ?', '192.168.56.31']
                                                                           │select function whereToString is: ['ip = ?', '192.168.56.31']and: ['192.168.56
                                                                           │.31']type: <type 'list'>




and then the log is:

   log func query is: [SELECT * FROM attackers WHERE ip = %s LIMIT 1] args: (['192.168.56.31'],)

for : 
    loger("log func query is: [" + str(query) + "] args: ["+ str(args)+"] ")

Currently, my code runs with the following code:

def log(self, query, args, kwargs):
    """
    Log the query and any args or kwargs using C{twisted.python.log.msg} if
    C{InteractionBase.LOG} is True.
    """
    if not InteractionBase.LOG:
        return
    log.msg("TWISTAR query: %s" % query)
    if len(args) > 0:
       + try:
            log.msg("TWISTAR args: %s" % ",".join(args))
       + except Exception as e:
        +    pass
    elif len(kwargs) > 0:
        log.msg("TWISTAR kargs: %s" % str(kwargs))

How would I fix this problem permanently? how would I edit the base class using oop? I don't want to run my own fork of the code! (what i am currently doing )

How to make a join statement

Hi, I read the documentation but there's no sign how to make a join statement (or using implicit notation, select from two different tables using a where clause). is there a proper way rather than directly with low level sql statements?