morepath / reg Goto Github PK

Clever dispatch

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

reg's Introduction

Morepath: Python web microframework with super powers

Morepath is a Python web framework. An application consists of models. Each type of model is published on a URL path. Content is exposed to the web using views.

Documentation.

reg's People

Contributors

Stargazers

Watchers

Forkers

iapilgrim iivvoo pombredanne waytai webmaven jean taschini neilvyas nandoflorestan kgerring sgaist

reg's Issues

different approach to mapply (or lookup_mapply)

In this case:

def foo(a, lookup):
    pass

a foo(a, lookup=lookup) call passes in the lookup. And in this case:

def foo(a):
   pass

a foo(a, lookup=lookup) call doesn't pass in the lookup.

We could wrap the second function during registration time into a function that just throws away lookup. This would likely be faster than lookup_mapply now which needs to inspect arguments.

mapply into Morepath

As discussed in #41, mapply can be moved into Morepath now. It wouldn't be part of Reg's API anymore, though arginfo will be. The reason to do this is to give us more flexibility in reimplementing the part of Morepath on the critical path in the publisher that still uses mapply.

component_key_dict missing in API docs

multiple subscribers

Come up with a way so that multiple functions can be called when a dispatch function is called. This allows things like an event system.

Update supported Python versions

drop support for Python versions < 3.6
add support for Python versions 3.7, 3.8 (if missing)
fix linting errors (if present)

If you are a beginner, and need to help for this issue, especially during Hacktoberfest, please comment here or reach out to me via https://jugmac00.github.io/

events

Some way to build events on top of Reg. We should be able to select all things that match a certain set of predicates, including things that match less strongly. Then fire off events at it.

C version of argextract

It would help dispatch performance quite a bit (and hide from the debugger) if we implemented what is in argextract in C -- getting an argument dictionary from arguments to a function call, and then getting the predicate information out of that for each predicate.

Note that the fastmapply branch already has code for doing argument introspection in C that we can reuse.

Example for supporting zope.interface Interface-based dispatch

I have an example for interfacing reg with zope.interface https://gist.github.com/moshez/cb020469eb3bf7c43240b17f8cf87a62
h
It would be really useful if it was easier to make custom matchers...for example, if index was an optional argument to match_instance so that I could override it without playing these games.

dispatch_external_predicates missing in API docs

document and expose mapping

Expose the 'mapping.py' functionality in the Reg top level init and document it, at least API docs.

decoupling-fastargextract branch

This branch contains a C implementation of the ArgExtractor logic.

While the performance impact on a simple benchmark with Morepath is minimal, the dispatch performance has gone up quite a bit compared to the plain decoupling branch.

The decoupling branch does this when you execute perf.py:

dispatch 0 args
2.50691318512
dispatch 1 args
4.27959012985
dispatch 2 args
6.44276189804
dispatch 3 args
7.31519389153
dispatch 4 args
8.34504699707
Plain func 0 args
0.161817789078
Plain func 4 args
0.481851100922

while the fastargextract branch does this:

dispatch 0 args
2.0502038002
dispatch 1 args
2.98760485649
dispatch 2 args
4.65890598297
dispatch 3 args
5.31285595894
dispatch 4 args
5.77068710327
Plain func 0 args
0.161151885986
Plain func 4 args
0.491057157516

So that shaves off quite a bit of time from dispatch performance, even though it's nowhere near plain function call performance.

registering a class that has a old-style class in its inheritance chain

Fails with an error that this base class has no mro.

Add link to docs

Idea: add a link to https://reg.readthedocs.org as the main url for the project? (I mean the link at the top of https://github.com/morepath/reg, below the title).

skip internals during debugging

It would be nice if the reg internals would be skipped in debugging, so that you can go from call site immediately to called site when debugging.

invocation hook

@taschini wrote:

I would like to have an invocation hook as an argument to dispatch_method. This hook is a callable
that is passed on to the DispatchMethodDescriptor like get_key_lookup and that is invoked on
obj when the DispatchMethodDescriptor.__get__ is invoked:

  class DispatchMethodDescriptor(object):
     ...
     def __get__(self, obj, type=None):
         self.invocation_hook(obj)
         ...

If cache_bound_method is true, than the hook is executed only once.

Could you go into the use case for this, @taschini?

Demonstrate how reg can be used for testing applications

The DI nature of reg allows us to replace the entire system with fakes, mocks, stubs without using unittest.mock.
This means that testing is much easier because you don't have to know the call site.
Maybe we can add a TestCase base class that cleans up the registry after each test.

include the new dispatch API in the API docs

external abc support

Reg currently doesn't support an externally assigned abc in its lookup code. We need to add to support.

document predicates story

Expose documentation for the predicates story, narrative and API.

Redundancy in the API for registering implementations of dispatch methods

Currently, the API to register the implementations of dispatch methods includes:

DispatchMethod.register
DispatchMethod.register_function
DispatchMethod.register_auto
methodify
methodify_auto

Inspection of these reveals that

methodify_auto is conceptually equivalent to methodify with a second argument;
DispatchMethod.register_function is conceptually equivalent to DispatchMethod.register + methodify;
DispatchMethod.register_auto is conceptually equivalent to DispatchMethod.register + methodify_auto, and therefore to DispatchMethod.register + methodify.

The API can therefore be made tighter just by providing:

DispatchMethod.register
methodify

Note that DispatchMethod.register_function is not used anywhere under the Morepath umbrella, DispatchMethod.register_auto is used in a few places in Morepath, and methodify_auto is used a couple of times in Morepath and once in More.static.

Leaking test isolation

test_no_implicit in test_dispatch.py is not well isolated. Running Reg's test-suite in the same batch after Morepath's test-suite fails:

$ cd morepath
$ ./bin/py.test morepath src/reg/reg
____________________________________________________________________________ test_no_implicit _____________________________________________________________________________

    def test_no_implicit():
        @dispatch('obj')
        def target(obj):
            pass

        alpha = Alpha()
        with pytest.raises(NoImplicitLookupError):
>           target.component(alpha)

src/reg/reg/tests/test_dispatch.py:549: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/reg/reg/dispatch.py:58: in component
    return get_lookup(kw).component(self.wrapped_func, *args, **kw)
src/reg/reg/registry.py:424: in component
    key = self.key_lookup.predicate_key(callable, *args, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <morepath.cachingreg.RegRegistry object at 0x10fd38510>, callable = <function target at 0x10fe3cf50>
args = (<reg.tests.test_dispatch.Alpha object at 0x10fda52d0>,), kw = {}, r = None

    def predicate_key(self, callable, *args, **kw):
        """Construct predicate_key for function arguments.

            For a callable and its function arguments, construct the
            appropriate predicate_key. This is used by the dispatch
            mechanism to dispatch to the right function.

            If the predicate_key cannot be constructed from args and kw,
            this raises a :exc:`KeyExtractorError`.

            :param callable: the callable for which to extract the predicate_key
            :param args: the varargs given to the callable.
            :param kw: the keyword arguments given to the callable.
            :returns: an immutable predicate_key based on the predicates
              the callable was configured with.
            """
        r = self.predicate_registries.get(callable)
        if r is None:
>           raise KeyExtractorError()
E           KeyExtractorError

src/reg/reg/registry.py:223: KeyExtractorError
1 failed, 595 passed in 1.35 seconds

Conflict on buildout

Following the instructions in developing.rst, I get the following error on running ./bin/buildout:

Error: There is a version conflict.
We already have: pyflakes 1.1.0
but flake8 2.5.4 requires 'pyflakes<1.1,>=0.8.1'.

mapply should give clearer errors

Right now now there is a TypeError, mapply gives back information like this:

File "/home/faassen/projects/contact/morepath/src/reg/reg/mapply.py", line 24, in mapply
    return func(*args, **new_kw)
TypeError: __init__() takes exactly 2 arguments (1 given)

This is not very helpful; we want to know what func is and where it is defined.

Switch formatting to black

I'd like to switch to the Black source code formatting tool to format code. This requires:

a modification of the developer notes to mention this.
make Black a development dependency.
adjustment of any flake8 configuration so it's compatible with Black.
executing black on all source code.

I like to use Black's integration in vscode and I noticed on large files sometimes Black is rather slow. I suspect we don't have this issue with Reg, though.

I'd also like to configure it to use 80 characters as a line length by default, as I still do prefer this myself.

``inspect.getargspec`` is deprecated in Python 3.5

Python 3.5 deprecates inspect.getargspec in favor of inspect.signature. It is used by the arginfo function in the reg.arginfo module.

Sphinx excludes generic functions from automodule members.

When generating docs using Sphinx, generic functions are not included as members of an automodule.

See http://morepath.readthedocs.io/en/0.15/internals/generic.html (source).

document register_function value hack

There's a hack that sets the 'value' properly on the function that is registered with register_function. This needs to be documented.

a replacement for classmethod?

Right now there's no clean API to do a classmethod equivalent call. Can we come up with something?

register_dispatch for component lookup

If instead of doing a generic function call you do a component lookup and no register_function was done for a particular dispatch function, register_dispatch must be called first. This could potentially lead to bugs.

generated code is hard to pdb through

When I pdb through the generated code obviously it has no context, so I get a lot of stuff like this:

> <string>(2)__call__()
(Pdb) l
[EOF]

I wonder whether we can do something to help pdb-ability.

extra non-dispatch arguments should not play a part in dispatch

Right now if you have a generic function that defines extra arguments you don't want to dispatch on, the system tries to dispatch on them anyway and then fail to find a matching function, falling back to the default implementation. Turning the argument into an explicit keyword argument fixes this.

This whole situation seems wrong; it should just work when you don't want to dispatch on that argument. See whether we can fix this in a backwards compatible way.

tox to test

Figure out using tox for testing at least Python 2.7 and PyPy.

making register a decorator

Currently register works like this::

def implementation(a):
    ...

foo.register(implementation, a=A)

So the implementation registered is the first argument. I propose we change register so it can
be used as a decorator:

@foo.register(a=A)
def implementation(a):
    ...

register returns a function that does the real registration when called. In Morepath we'd not really benefit, we'd have to rewrite:

app_class.foo.register(obj, **predicates)

into:

app_class.foo.register(**predicates)(obj)

but that's doable enough, and it does make Reg's api by itself cleaner.

TypeError: match_instance() takes at least 2 arguments (1 given)

Trying to run example code from your latest blog post:

import reg


@reg.dispatch(reg.match_instance('obj'),
              reg.match_key('request_method',
                            lambda request: request.request_method))
def view(obj, request):
    raise NotImplemented

# an implementation for GET requests of documents
@view.register(obj=Document, request_method='GET')
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

# get representation for document by calling view()
html = view(doc, request)

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    @reg.dispatch(reg.match_instance('obj'),
TypeError: match_instance() takes at least 2 arguments (1 given)

Is this a documentation or implementation mismatch?

fastmapply

There's an implementation on a branch that reimplements mapply (and arginfo) in C for performance (and invisibility from debugging). We should merge this at some point.

The value attribute of methodified functions

Methodified functions (i.e., functions wrapped by methodify, methodify_auto, DispatchMethod.register_function, DispatchMethod.register_auto) provide a way to retrieve the original function.

This is currently done via a value attribute, which is problematic for two reasons:

methodify_auto must wrap functions that wouldn't need it just to be able to set the value attribute.
DispatchMethod.register_auto attaches the value attribute directly to the callable passed as an argument, which might lead to unexpected behaviour if the callable already has a value attribute.

My suggestion is to introduce a separate function, tentatively called inspect_methodified. In principle, it would behave something like the following (implementation details can be ironed out later on):

def inspect_methodified(func):
    return getattr(func, 'value', func)

It will return func.value if the attribute exists, i.e., if the function has been wrapped, and will return func if it has not been wrapped.

In this way, no unnecessary wrapping is done and no potential bugs are introduced.

Any chances python 2.6 can be supported?

I'm using reg for my Django 1.5.x / 1.6.x project. Since Django supports python 2.6.x and python 2.7.x (and to some extend 3.3), my project also supports 2.6.x and 2.7.x

However, currently my builds are failing because reg doesn't seem to support 2.6:

https://travis-ci.org/wheelcms/wheelcms_axle/jobs/23915236

Specifically, it's failing on the dict comprehension syntax

E       new_kw = {name: kw[name] for name in info.args if name in kw}
E                                  ^
E   SyntaxError: invalid syntax

but there may be more issues of course.

I may do an attempt to fix this myself but if you know of any good reasons why 2.6.x will never work (or never be supported) then I should reconsider if I still want to support 2.6 or want to depend on reg, at this point.

arginfo could use inspect more?

Could use arginfo inspect facilities more to get information about keyword arguments and such? Might clean it up a bit.

Dispatchable ABC interface not supported

Hi!
I've tried to create an interface similar to z3 ones, but as an ABC, and register it as the dispatchable callable without creating a separate function, so that I can write IStuff(obj) to retrieve an adapter of obj implementing IStuff. (I really liked this syntax for the lookup).

Something like:

@reg.dispatch('entity')
class IStuff(metaclass=ABCMeta):

    @abstractmethod
    def __init__(context):
        pass

    @abstractmethod
    def foobar():
        """Foobar doc"""

It doesn"t work. The first reason is that get_callable_info does not support abstract classes. It's just a matter of adding this in it (and assuming the __init__ is an abstractmethod):

if inspect.isabstract(callable):
    return get_class_init(callable), callable, False

Once this is fixed, I get another issue in register_function_by_predicate_key because the arginfo(value) is using the Dispatcher constructor instead of the decorated class constructor (the abc interface), so the signature is not the same: 'context' is not found I've not gone further.

Otherwise thanks a lot for reg, it's a very nice replacement for the zca!

dispatch 0 args
2.50691318512
dispatch 1 args
4.27959012985
dispatch 2 args
6.44276189804
dispatch 3 args
7.31519389153
dispatch 4 args
8.34504699707
Plain func 0 args
0.161817789078
Plain func 4 args
0.481851100922

Here's some older benchmark of the arg extraction code (where @taschini already took out argextract, but not yet completely):

dispatch 0 args
1.25924301147
dispatch 1 args
2.03821897507
dispatch 2 args
3.58315300941
dispatch 3 args
4.08871388435
dispatch 4 args
4.67021179199
Plain func 0 args
0.159023046494
Plain func 4 args
0.48424077034

As of today, it's:

dispatch 0 args
1.30515289307
dispatch 1 args
1.78144598007
dispatch 2 args
3.02587485313
dispatch 3 args
3.37043809891
dispatch 4 args
3.69515585899
Plain func 0 args
0.163196086884
Plain func 4 args
0.484107017517

So we currently we're less than 10 times slower than plain Python.