zmievsa/cadwyn

Production-ready community-driven modern Stripe-like API versioning in FastAPI

Hi there, my name is Stanislav Zmiev 👋

• I am a Platform Engineer with an extensive experience in devtools and backend Development
• I have done a fair share of frontend, mobile, DevOps, and lower level stuff in my spare time
• I have built numerous open source projects over the years, most of which can be found on my GitHub page
• My passion is creating and combining development tools to provide programmers with the best possible support while coding. To increase value without needing to increase the team size
• In my spare time, I contribute to open source and participate in developer meetups and conferences. Over the years I have contributed code to:
  • CPython
  • typeshed
  • tortoise-orm
  • ideas
  • pygount
  • inflect

... any many smaller projects!

cadwyn's Issues

Create a favicon for cadwyn

I am thinking of something git-like with those cool branches. But I'm open to suggestions

Improve error output for field/endpoint not found during code/route generation

Add a test that checks whether `Literal` in route dependencies gets migrated as expected

For example, let's say Literal has enum values as its args. Will we migrate them to older versions of the same enum when generating new routes? I don't think so.

Add full-structure request/response migrations

Request and response migrations have vastly different tooling

So far we have been viewing request and response migrations as vastly different because request migrations much more often affect business logic and because fastapi's handling of requests is vastly different. However, it makes a lot more sense to unify approaches for them. So I present the redesign of how we could write request and response migrations. It gives us a lot more functionality and is a lot easier to understand for the users.

def migration(response):
    response.body["hello"] = True
    response.headers["gamarjoba"] = request.headers.pop("hi")
    response.status_code = 200


def request_migration(request):
    request.body["hello"] = True
    request.headers["gamarjoba"] = request.headers.pop("hi")
    request.query["hello"] = "world"

So in essence we give our users ability to fine-tune how they handle responses and we give our users ability to migrate requests in the same manner. In order to do this for requests, we will have to marshall request.body from pydantic on the first attribute access and then unmarshall it back after we are done with all request migrations. This will also remove the need to have the nasty union schemas.

However, let's say a new version doesn't have some of the fields of the old version. How would we handle that? How do we use that field after a migrated body has been unmarshalled? We add a _-prefixed attribute that is marked as PrivateAttr with a sane default for new version models and delete it in the old version. The prefix will hide the attribute from openapi. For example, if client_id is only passed in an old request model, we add a _migrated_client_id field to the new model and use it from business logic.

It also makes sense to replace "had_property" with "property.was".

Use response schema instead of endpoints for data migration

Add a CI step with ruff + black + pyright checks

Blocked by #30

Add validation that the user uses latest schemas instead of the versioned ones in routes

Build a new callable instead of wrapping an existing one in annotation transformer

Currently, each time we migrate any callable to an older version (for example, a dependency or a route.endpoint), we wrap it in a decorator and then edit the decorator's signature to have the older schemas, enums, dependencies, etc. As a result, the number of decorators is equivalent to the number of versions.

We could either take the original function from __alt_wrapped__ each time we migrate a function to the older version or we could build an entirely new callable with the same __code__ but a different signature.

See annotation transformer for more details.

Refactor requests and responses to have the full structure instead of just the body

Add pre-commit with ruff + black

Delete default api version contextvar and make passing a custom one a requirement

Come up with mechanisms of specifying/figuring out that some or all endpoints do not need to be migrated to the previous version

For example, if an endpoint was not directly affected, there are no request/response migrations for it, its dependencies didn't reference any schemas changed in the current version, its response model was not related to any changed schemas, and its body was not related to any changed schemas.
Additionally, you could do an easy shortcut for versions with purely side effect changes of skipping generation entirely.

This is all closely related to building a graph of changed schemas in #52 .

Add support for getting different versions of objects based on current API version

Is your feature request related to a problem? Please describe.
Often things like error messages and status codes are also versioned. We want some solution that allows us get a different variable in business logic based on the current API version.

Describe the solution you'd like
The following will allow us to preserve the type hints on the module and return the correct version of the value in runtime.

Based on #5

from cadwyn import get_current_version
from versioned_dir.unions import errors

async def my_business_logic():
    raise get_current_version(errors).DOES_NOT_EXIST_ERROR

However, it feels like something that could be abused to add side effects to migrations without side effects like so:

from cadwyn import get_current_version
from versioned_dir.unions import weird_module

async def my_business_logic():
    if get_current_version(weird_module).raise_err:
        raise Exception

So now we have side effects in a version that didn't have any side effects. To solve this, we could:

• disallow using module().variable() syntax in version changes without side effects
• disallow module().variable() syntax altogether and force people to do side effects with "if-s"

NOTE: Make sure that user uses unions. Also, it might be a good idea to make it a method of VersionBundle so that a single service could have multiple packages like this

Is badoo doing the same thing as cadwyn?

https://youtu.be/M2KCu0Oq3JE

It seems like badoo is doing the same strategy but in reverse: the backend also returns all information but the client now knows that the backend does and just chooses the fields/structure/features that it wants. I feel like it's an overall a better approach if you are ok with your API becoming complex.

I'd like to add a note into our docs about this.

Add "recipes" section into docs

It will look like this:

### <Breaking change description>

<Solution>

Use pydantic.Field instead of universi.Field

because defaults are ok and an error-prone interface is more important

Omit `Field` field value from code generation if it's empty

Oftentimes cadwyn will generate fields with empty Field() constructor because there were no arguments passed. Prevent cadwyn from generating unnecessary Field definitions like this one.

Note! This DOES NOT apply to PrivateAttr and FillablePrivateAttr because they are necessary for pydantic to see that something is indeed a private attr.

Add automatic imports for schemas added in-between migrations

If schema is not versioned, use absolute imports
If versioned, use relative imports

Note that we'll have to check whether the symbol is actually not defined in the module. I'd probably propose parsing file and find all variable/class definitions + all imports.

Add support for changing global variables in versions

Is your feature request related to a problem? Please describe.
Often things like error messages and status codes are also versioned. We want some solution that allows us to change any variables in-between API versions.

Describe the solution you'd like
I'm thinking of the following interface:

from cadwyn.structure import module

module(latest.my_module).variable("some_variable").had(value="'hello darkness'", annotation="str")

which will generate the following code:

# within latest.my_module

some_variable: str = 'hello darkness'

Do we really need this though? I am worried that this might be an overkill that will add hidden side effects.

For the sake of simplicity, we won't allow defining variables in-between versions — only changing their values and annotations.

Use paths and schemas for endpoint migrations to prevent circular imports on side effect checks

Incorrect renaming on import override for modified schemas

I believe that we have a really rare bug within cadwyn that has to do with how our renaming works.

For example, if we have schema A, we import schema A into some module, then lower in that module we define A to be something else. If we rename A using cadwyn's tools, then that second variable and all its usages will also be renamed.

Granted, it's a really rare case but it is possible.

Add request body migration

At first, we will only support migrations for single-body-schema routes. I.e. Routes that have a single payload, not a group of payloads that fastapi united into one.

Add an ability to replace dependencies for an arbitrary endpoint

See #11 for more details.

Replacing individual params (query, header, cookie)

endpoint("/v1/users/{user_id}", ["GET"]).dependency("param_name").was(Query()) # 1
endpoint("/v1/users/{user_id}", ["GET"]).dependency("param_name").was(Depends(get_param_dependency)) # 2

@endpoint("/v1/users/{user_id}", ["GET"]).dependency("param_name").was # 3
def apply_some_changes_to_dependency():
    def any_name_here(anything_here: str = Query()):
        # Do any actions with the dependency here
        return anything_here


    return Depends(any_name_here)

1. (accepts an instance of Header/Query/Cookie) Allows you to change the type (header/query/cookie) or config of any dependency. Body should not be supported!
2. (accepts an instance of Depends) Allows you to change some dependency entirely to another function. It's useful when you want to change dependency's logic without affecting your route. However, sometimes type hints of the dependency cause circular imports from the business logic which is why we have Path 3.
3. (accepts a callable that returns a callable) Is same as 2 except that it allows you to import things after the migration has been defined so it solves the problem of circular imports. It's needed extremely rarely.

You can use APIRoute.dependency_overrides_provider for 3. Not sure if you should though...

Rejected solutions

@dependency("api.v1.users:some_dependency").was
def anything_here():
   ...

It is really implicit and prone to errors. It doesn't really give us anything in terms of functionality over the solutions above so I decided that it was a bad idea to include it.

Changing logic for handling params or removing the param completely

At this point it makes a lot more sense to just replace the route function.

Add an ability to have versioned synchronous API routes

Otherwise our users won't be able to use synchronous db/http libraries efficiently.

Use a threadpool like fastapi does probably.

Add a test that uses a lambda instead of a regular function for schema(...).property

See test__codegen_property for more details on how to test properties.

Add tests for pydantic's constrained types

Pydantic has special types that it assigns to fields which have length/size/value constraints such as constring and conint. It's a hack that I think was removed in Pydantic 2, but nevertheless we must check that cadwyn works correctly with such types which will also help us make sure that we will not break when migrating to pydantic v2.

Please, see custom_repr function in codegen.py for more details.

Add changelog to openapi page with versions list

Openapi dashboard with the list of versions could display the summary of changes between versions in a hideable HTML element. This will make it easy to see the difference between versions for anyone within or outside the company (if necessary)

Add support for python 3.9

Split readme into pages and move it to docs

Collect changelog from migrations

Add VersionBundle.generate_changelog(self) that will output a list of versions and changelog for each of them. Here's the proposed format for it:

[
    {
        "version": "2022-11-16",
        "changelog": []
    },
    {
        "version": "2023-04-12",
        "changelog": [
            {
                "description": "Change `POST /v1/users` status code to 201 instead of 200",
                "side_effects": false,
                "endpoint_changes": [
                    {
                        "type": "endpoint.had",
                        "method": "POST",
                        "path": "/v1/users"
                    }
                ],
                "schema_changes": [
                    {
                        "type": "schema.field.had"
                        // More stuff here
                    }
                ]
            }
        ]
    }
]

We could also unite all changes into the same polymorphic payload but I'm still questioning whether that's a good idea.

Add an MkDocs documentation page hosted on github pages

Add service example for docs

Planning request migrations

Add a test to make sure that cadwyn understands the default in: "field: type = None"

Rewrite schema generation to utilize ASTs for determining correct type annotations

This will also require us to understand where the field came from in the mro so we'll need both features.

Figure out how to simplify `_apply_alter_schema_instructions` in codegen.py

Optimize request migration for migration-less cases

Currently, we create request info and perform all logic related to it in versions.py even if we don't encounter any request migrations which means that the majority of endpoints are needlessly slowed down.

Perform this whole logic only at the point of encountering the first migration.

Make a list of attributes we do not support in field.had(...) and make a note of them in the docs

We might not be supporting migrating all fields in Pydantic. Essentially compare cadwyn.structure.schemas.FieldChanges with pydantic.FieldInfo

Add an ability to add and delete validators for pydantic models in version changes

It is likely going to be incredibly similar to schema(...).property so you can use it for reference. Make sure to document it properly too.

Add command-line interface for codegen

Essentially it will simply call regenerate_dir_to_all_versions. Use argparse to limit the number of dependencies our user needs to install. We could call it cadwyn generate-versioned-schemas.

Note to whoever is working on this: the name regenerate_dir_to_all_versions is not great. Please, change it.

Add an ability to add a private field to schema using info=PrivateField or info=FillablePrivateField

Add header-based routing

cadwyn is not complete without a header-based router. Our guides become more complex and the integration becomes tough.

If there was some external library that implemented header-based versioning, cadwyn would be complete and ready for use.

Add an ability to use str | int | float | date as a version instead of just a date

Add request.form data migration support

We are capable of migrating body, cookies, headers, and query params but we also need to be able to migrate forms.

Add a test that checks whether default values in route dependencies get migrated as expected

For example, let's say that some route has an enum value as a default dependency. Will we migrate them to older versions of the same enum when generating new routes? And what happens if enum gets deleted? How should we handle that?

See test__router_generation__updating_request_depends for more details

Improve cicd to share poetry env between test and lint jobs

Or maybe it's unnecessary... Not sure.

Figure out how to simplify `_apply_endpoint_changes_to_router` in routing.py

It has a huge complexity and is really hard to read but all alternatives are even harder to read.

Come up with additional mechanisms of specifying/figuring out what a request schema can be automatically migrated to the next version.

Currently, we automatically skip union generation for request schemas that were mentioned in a @convert_request_to_next_version_for. However, maybe we could figure out other heuristics for determining if a schema can be skipped. For example, maybe we can determine that the schema and everything it depends on hasn't changed in between versions? Need to be careful about this though because a simple "generate json schema and compare" approach might not account for additional constraints on fields or additional validators.

Note that before we add a method to manually mark schemas as skipped, we must exhaust all other options.

It feels like we need an "affected schema graph" for each version. The way I see it: the only way to a schema to be affected is to reference any other affected schema in its definition (MRO, fields, properties, validators, methods)

Add a `schema(...).field(...).had(name=...)` operation

This felt like a useless operation in the beginning but now that I think about displaying diffs to users in an openapi dashboard it suddenly makes a lot more sense.

Migrate to pydantic v2