Type incompatibility when using a generic callback protocol

[vxgmichel]

Type incompatibility when using a generic callback protocol

[erictraut]

The approach you're using here won't work, and I don't consider this a bug in pyright.

In general, type variables scoped to a function need to be solved in the context of a call to that function. Type variables scoped to a class need to be solved in the context of a call to the constructor of that class. If they are not solved within that context, they go out of scope and can no longer be solved.

Here you're defining a class-scoped type variable T in the MyCallable class. When decorator is called, the type variable T@decorator takes on the value of T@func. The return result of the call is therefore MyCallable[T@func]. This return result is a callable object, but the type variable T@func is no longer in scope, so it can no longer be solved when calling this object.

I said "in general" above because pyright and mypy contain special-case logic that allows an unsolved type variable within a Callable return type to be "re-scoped" to the Callable. This allows an unsolved type variable from another scope to be solved when the Callable is eventually called. I think this behavior is arguably defensible because type variables are allowed to be scoped to callables, and this is simply re-scoping an existing type variable to a different callable.

For example, you'll see that your code type checks without errors if you replace your decorator function with the following:

def decorator[T](x: Callable[[T], T]) -> Callable[[T], T]:
    return x

Or if you want a more general version:

def decorator[**P, R](x: Callable[P, R]) -> Callable[P, R]:
    return x

The special-case logic that I mentioned above is rather hacky and is not current specified (or even allowed) by the Python typing spec. This is one of numerous areas where discussion is needed, and the community needs to decide on the correct behavior, which should then be specified in a clear manner within the spec.

It appears that mypy also applies this special-case logic to "pure" callback protocols — i.e. protocols that define only a __call__ method with no additional methods or attributes. This is why you're seeing the behavior in the recent mypy issue you filed. The mypy maintainers are likely to tell you that mypy is working as designed in this case. Mypy's behavior here is somewhat harder to justify given that it's rescoping an unsolved type variable from a function to a class-scoped type variable in a protocol class. The logic is probably that a "pure" callback protocol class is similar enough to a Callable that it's justifiable. That's debatable.

In any event, I recommend against using this pattern because you're deep in a gray area of the typing spec. I recommend sticking to a Callable return type rather than creating a protocol class.

I'm going to close this issue for now because I don't plan to make any modifications to this behavior in pyright unless/until the intended behavior is documented in the typing spec.

If you'd like to kickstart a discussion on this topic, you're welcome to do so in the Python typing forum.

[vxgmichel]

Wow thank you a lot for the thorough response, that clears up a lot for me.

In any event, I recommend against using this pattern because you're deep in a gray area of the typing spec. I recommend sticking to a Callable return type rather than creating a protocol class.

My problem is that I couldn't really find an alternative approach, because Callable does not provide the flexibility needed to precisely define a callable prototype. The mypy documentation acknowledges that:

Protocols can be used to define flexible callback types that are hard (or even impossible) to express using the Callable[...] syntax, such as variadic, overloaded, and complex generic callbacks.
https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols

A good example is a callable with variadic arguments. One could use VarArg but the mypy documentation itself discourages its usage:

This feature is deprecated. You can use callback protocols as a replacement.
https://mypy.readthedocs.io/en/stable/additional_features.html#extended-callable-types

For instance you can see this SO post advocating for generic callback protocols, which works with pyright in this specific case:

T_contra = TypeVar('T_contra', contravariant=True)
T_co = TypeVar('T_co', covariant=True)


class SupportsCallWithArgs(Protocol[T_contra, T_co]):

    @abstractmethod
    def __call__(self, *args: T_contra) -> T_co:
        pass

https://stackoverflow.com/a/73648703, code sample in pyright playground

I'm going to close this issue for now because I don't plan to make any modifications to this behavior in pyright unless/until the intended behavior is documented in the typing spec.

No problem. I hope the community will soon manage to specify a way to write flexible and generic callables, as this is very useful for meta programming patterns. I think a common example is a decorator applied to functions that themselves depends on a type var. In this case, we need some tool to express how the decorator transforms a generic callable to another one.

Thanks again for your time.