lmmx/botodto

Pydantic model codegen from AWS OpenAPI schemas generated from the AWS JS/TS SDK (v3)

botodto's Issues

Multipackage releases on CI

Multipackages

• Use 'multipackages': multiple Python packages in one repo
  • Inspired by visidata apps (but I'm not sure that that actually ships them separately)

We typically just have a single setup.py script which uses find_packages("src") and follows a src layout.

To ship multiple packages (let's say for APIs named "a", "b", "c"), we'd want:

botodto-monorepo/
  botodto-a/
    setup.py
    botodto_a/
      __init__.py
      ...
  botodto-b/
    setup.py
    b/
      __init__.py
      ...
  botodto-c/
    setup.py
    c/
      __init__.py
      ...

Releases on CI

• Control package release via CI: only trigger a release when the code for an API changes.
  • Don't want to spam PyPI or have trivial releases. Every package release should have a good reason.

Discussion

name: Publish botodto-a to PyPI

on: 
  push:
    branches:
      - main
    paths:
      - 'botodto-a/**'
      - 'updated_packages.json'
      - '.github/workflows/botodto-a.yml'

jobs:
  build-n-publish:
    name: Build and publish botodto-a to PyPI
    runs-on: ubuntu-latest
    environment:
      name: pypi
      url: https://pypi.org/p/botodto-a
    permissions:
      id-token: write
    steps:
    - uses: actions/checkout@v3
    - name: Cache pip packages
      uses: actions/cache@v3
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
        restore-keys: |
          ${{ runner.os }}-pip-
    - name: Check if package was updated
      id: check
      run: echo "::set-output name=updated::$(jq -r '.botodto-a' updated_packages.json)"
    - name: Set up Python
      if: steps.check.outputs.updated == 'true'
      uses: actions/setup-python@v2
      with:
        python-version: '3.x'
    - name: Install pypa/build
      if: steps.check.outputs.updated == 'true'
      run: >-
        python -m
        pip install
        build
        --user
    - name: Build a binary wheel and a source tarball
      if: steps.check.outputs.updated == 'true'
      run: >-
        python -m
        build
        --sdist
        --wheel
        --outdir dist/
    - name: Publish distribution to PyPI
      if: steps.check.outputs.updated == 'true' && startsWith(github.ref, 'refs/tags')
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        repository_url: https://test.pypi.org/legacy/

• This is a common pattern, similar to the one used in the datamodel-code-generator package

Use generated models to wrap boto3

I might be able to come up with a way to represent ambiguity or identify an undocumented part of the response that will name the error type.

E.g. InvalidArn, surely part of the response somewhere? Even if only as constr Pydantic pattern?

Also get datamodel-codegen into the development dependencies? Not sure. Generate DTOs as documented already from v2 OpenAPI representations anyway.

Create models from v3-only shapes

Create Pydantic models with keys from the members of the StructShape-type shapes in the v3-only set.

• First for the demo files
• Then for the stepfunction service specifically (will require mapping the v2 and v3 together somehow, based on common metadata)
• Then for any given service in the service list

It looks like they're all going to be simple singleton objects with a message of type ErrorMessage (which is a simple string type).

I didn't count the different types of shape but I did begin to whittle down the enum.