nameko/nameko-sqlalchemy

SQLAlchemy dependency for nameko services

Nameko

[nah-meh-koh]

A microservices framework for Python that lets service developers concentrate on application logic and encourages testability.

A nameko service is just a class:

# helloworld.py

from nameko.rpc import rpc

class GreetingService:
    name = "greeting_service"

    @rpc
    def hello(self, name):
        return "Hello, {}!".format(name)

You can run it in a shell:

$ nameko run helloworld
starting services: greeting_service
...

And play with it from another:

$ nameko shell
>>> n.rpc.greeting_service.hello(name="ナメコ")
'Hello, ナメコ!'

Features

• AMQP RPC and Events (pub-sub)
• HTTP GET, POST & websockets
• CLI for easy and rapid development
• Utilities for unit and integration testing

Getting Started

• Check out the documentation.

Support

For help, comments or questions, please go to https://discourse.nameko.io/.

For enterprise

Available as part of the Tidelift Subscription.

The maintainers of Nameko and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source dependencies you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact dependencies you use. Learn more.

Security contact information

To report a security vulnerability, please use the Tidelift security contact. Tidelift will coordinate the fix and disclosure.

Contribute

• Fork the repository
• Raise an issue or make a feature request

License

Apache 2.0. See LICENSE for details.

nameko-sqlalchemy's Issues

Mistake in Example

Hi Matt,
In the example, should:

from nameko_sqlalchemy import Session

be

from nameko_sqlalchemy import Database

?

Testing with `Database` dependency provider

Is there an example of how to test a service which uses the new Database dependency provider.
Previously, with the Session we would have a service like this:

class ServiceA:
    name = 'service_a'
    session = Session(Base)

@rpc 
def run(self):
    print(self.session.query(ModelA).all())

which was easily testable using the db_session fixture.

def test_service_a(db_session):
    worker = worker_factory(ServiceA, session=db_session)
    worker.run()

Is there a preferred pattern for using the Database provider? Right now I don't have a nice solution other than chaining a bunch of mock objects together which in the end mock out __enter__ of get_session() to finally replace the result with db_session.

Retry delay

Would it make sense to add an optional delay or backoff to the retry decorator?
Or would the recommendation be to use https://github.com/nameko/nameko-amqp-retry in this case?

Easy Example

Hi,
can you give me a simple example on how to use nameko-sqlalchemy as DependencyProvider for a service that has to connect to a mysql database?

Sorry but from this example I can't figure it out and I'm a little confused. Thank you

ETA on a new release?

Hi @mattbennett

You've gone and done it again.... You've added something that solves a problem I was having! But it's not in the PyPi release.

Do you know when you might be able to cut a new release that includes the transaction_retry code?

Many thanks,

Geoff

Missing kill behaviour

Not managed to verify yet, but I think nameko-sqlalchemy will leave transactions open when the owning service container is killed.

To mitigate this we should probably implement DependencyProvider.kill and have it close any open sessions. This world force us to hold more than weakrefs to sessions though, since the worker_ctx goes out of scope before DependencyProvider.kill is called.

How do I create tables with this?

The example given in the readme doesn't create new tables and so will error out.

This is my test

---

from nameko.rpc import rpc
from nameko_sqlalchemy import DatabaseSession

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String


DeclBase = declarative_base(name='testbase')

class ExampleModel(DeclBase):
    __tablename__ = 'example'
    id = Column(Integer, primary_key=True)
    data = Column(String)

class ExampleService(object):
    name = "example_service"
    session = DatabaseSession(DeclBase)

    @rpc
    def init_service(self, name):
        obj = ExampleModel(data=name)
        self.session.add(obj)
        self.session.commit()
        return "Hello, {}!".format(name)

When launching using nameko run service --config=service.yaml and sending a message it just errors

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: example [SQL: 'INSERT INTO ex│ 11     data = Column(String)

So how do I create the tables?

A way to supply create_engine kwargs through config (pool size)

Currently there is not way to supply connection pool size. And if max_workers is larger that default
pool_size+max_overflow we get

QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/3o7r)

as a result

Use `autoload` or `reflect`

Hi @mattbennett

How would I go about implementing database reflection with this?

Either for an ORM class:

class Users(Base):
    __table__ = Table('Users', metadata, autoload=True)

or the SDL:

messages = Table('messages', meta, autoload=True, autoload_with=engine)

(Examples copy-pasted from the Documentation).

I'm going to try and work it out, but any pointers greatly appreciated.

why pytest for nameko-sqlalchemy delete the tables after test

Even for nameko-example, the test cannot be rerun on the 2nd time because the table was dropped

Instance is not bound to a Session (nameko-sentry DP)

When using new interface inside some service method like this (taken from examples in README)

class Service(object):
    name = "service"

   db = Database(DeclarativeBase)
   sentry = SentryReporter()

    @entrypoint
    def write_to_db(self):
        model = Model(...)
        with self.db.get_session() as session:
            session.add(model)
            raise Exception()

if any exception is raised inside with block and service instance has nameko-sentry DP this will trigger
Instance <Model at 0x7fe4de51b240> is not bound to a Session when raven wants to stringify this object (repr wants instance's primary key) because the session is closed after context block is over.

Is there a reason to close a session in context manager?

Secondly there might be some misunderstanding from the documentation because it says

As a context manager that issues a commit or rollback on exit:

But not a word about that session is closed and all instances are invalidated.

documentation

more documentation is needed for this project

Database session context manager exceptions

Looking at the session context manager here:
https://github.com/onefinestay/nameko-sqlalchemy/blob/fcaef1246681f76176ca6d449c35056809d6c525/nameko_sqlalchemy/database.py#L12
I'm unsure if it is doing what I expect it to do.
I am assuming that its purpose is to allow me to do "safe" transactions knowing that if something goes wrong, it will roll back. But wouldn't most of the things that could go wrong happen during the commit? which in this case would not be covered as far as I can see.

So if I did something like:

with self.db.get_session() as session:
    session.add(new_user_with_duplicate_id_or_something_wrong)

It would not run into the rollback block (since no exception occured adding the user). Instead the exception would be thrown during the commit, it would be raised and no rollback and no close would be triggered.

I would have expected something more like this:

class Session(BaseSession):

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        try:
            if exc_type:
                self.rollback()
            else: 
                try:
                    self.commit()
                except Exception:
                    self.rollback()
                    raise
        finally:
            self.close()

kind of along the lines of http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it
(untested, just to demonstrate my thinking).
Have I misunderstood something?

required positional argument: 'declarative_base'

When following your example for legacy use of session.
i miss the positional argument declarative_base.
which seems logic when i see the code