Relative imports and non-root import paths crash the app

[FyZyX]

Relative imports and non-root import paths crash the app

[Lancetnik]

@FyZyX python relative imports should start with a dot (doc and doc)

So, your example with

from settings import Settings, init_settings

Is incorrect from the Python view, not the Propan. If you want to use relative import it should looks like

from .settings import Settings, init_settings

And this example works fine.

You can check your imports are correct by running application like a regular Python script by

if __name__ == "__main__":
    app.run()

and

python serve.py

If this way to run application is working fine, but Propan CLI is fault - please, give me information about your application directory tree and broken imports.

[FyZyX]

Right, so I actually do understand relative imports in Python, my issue states that BOTH relative imports AND non-root import paths both crash the app. in this case, settings is a module in the config subpackage, so the absolute (or root) import path is config.settings. Using just settings, while it's not technically a relative import path, is relative to the subpackage, not the root package, which is my point. To your point, I did try with it .settings and that seems to work, but settings alone, which should still be a valid import, fails with ModuleNotFoundError. That on its own is problematic.

I was trying to illustrate the point with the example project so that you could replicate the behavior, but let's consider the situation I actually ran into. Let's say we create a model module in the app package, and we add a dummy class like this:

class MyObject:
    pass

Now let's go to app.apps.handlers and modify it to import the model and instantiate an object. The idea is that our handlers might require a shared data model, and that data model may also be needed by other packages, so the model would ideally live outside the apps subpackage.

from propan import RabbitRouter
from propan.annotations import Logger

from .. import model

router = RabbitRouter()

@router.handle("test")
async def base_handler(body: dict, logger: Logger):
    my_obj = model.MyObject()
    logger.info(body)

If we then attempt to run the project, we get an ImportError. In this case the error is attempted relative import beyond top-level package, because I'm trying to grab a module from outside the package scope. However, this is not specific to such cases, which is evident from the error message because it clearly is trying to import from the root app package (otherwise it wouldn't say it attempted a relative import beyond the top-level). I've tried a couple of other imports, both relative and absolute (but relative to the subpackage) and they all fail. This is the issue I'm trying to illustrate.

Traceback (most recent call last):
  File "/Users/xyz/.pyenv/versions/3.11.3/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/Users/xyz/.pyenv/versions/3.11.3/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/xyz/venv/lib/python3.11/site-packages/propan/cli/supervisors/utils.py", line 47, in subprocess_started
    t(*args)
  File "/Users/xyz/venv/lib/python3.11/site-packages/propan/cli/main.py", line 110, in _run
    propan_app = try_import_propan(module, app)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xyz/venv/lib/python3.11/site-packages/propan/cli/utils/imports.py", line 12, in try_import_propan
    propan_app = import_object(module, app)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xyz/venv/lib/python3.11/site-packages/propan/cli/utils/imports.py", line 40, in import_object
    loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/xyz/example/app/serve.py", line 8, in <module>
    from apps import router
  File "/Users/xyz/example/app/apps/__init__.py", line 1, in <module>
    from apps.handlers import router
  File "/Users/xyz/example/app/apps/handlers.py", line 4, in <module>
    from .. import model
ImportError: attempted relative import beyond top-level package

The directory structure is identical to the example project, except for the added model module.

|-- app/
|---- apps/
|------ __init__.py
|------ handlers.py
|---- config/
|------ .env
|------ __init__.py
|------ settings.py
|-- __init__.py
|-- model.py
|-- serve.py

Clearly there are issues in the way imports are handled, and I know how tricky the import machinery of Python is, so this problem is completely understandable. I would be happy to help work on this problem, but my initial attempts were unsuccessful. I can keep trying, but you may have better luck since I haven't wrapped my head around the whole package architecture yet. It's interesting that .settings worked in the config subpackage, that implies there is some awareness of package structure, but I would still like imports to work the way they would if this was a normal Python package. Again, let me know if I can support in debugging or implementing a fix, I'm happy to help. I really like the Propan package as a whole, and this is my only sticking point so far.

[Lancetnik]

@FyZyX so did you try failing imports with manual python calling, as I said?

python serve.py

[Lancetnik]

Python always looking packages from the root directory, where you call the script.
So, there no subpackage and you can't import subpackages relative it's directory.

You attemption to import

from setting import *

Is invalid python code at all. Please, check it with regular code, not the Propan to be shure.

Using just settings, while it's not technically a relative import path, is relative to the subpackage, not the root package, which is my point.

It's just a your point, but, unfortunatelly, not the Python. It tries to import packages according to sys.path.

Please, give me example, working in python, but not working with the Propan CLI

[FyZyX]

Okay, my apologies, the real problem is having the main script inside the package to begin with. When we run a Python module as a script, it indeed does use the PYTHONPATH for all imports, which means relative imports in subpackages will fail. This is in fact normal Python behavior, so it is not a problem with Propan itself.

In order to get around this issue, I've pulled the main script out of the package and imported the package directly into the main script. This appears to resolve the import problems. Amazingly I'd never run into a situation like this before, probably because I don't usually keep my application scripts inside a package. I appreciate your fast responses and willingness to engage on the issue.

Fundamentally, this is about project structure, so while Propan is working correctly, complex package structures may become problematic for users if they base their project on one of the starter templates. For example, this would happen in they add additional modules in the same app package as the serve script and attempt to import those modules in subpackages as I did. Again, you're right, no need for code changes, but it might help to add some documentation around this, or potentially restructure the starter templates. I can document my findings if it helps you. Thanks again for working through this with me, I appreciate your patience. I'll close this issue as my initial report is indeed incorrect.

[Lancetnik]

@FyZyX thanks for you feedback. Changing the starting packages make a sence. But, I think the same structure is a typical for http-python packages (with the one entrypoin script and directories to import). Do you have any suggestion about different structure?

[FyZyX]

@Lancetnik I'm happy to provide my perspective if it helps! Personally, I think the problem here is that our entrypoint script (serve.py in the above case) was inside the application package.

In my experience with Python applications, the entrypoint is usually at the top level of the project directory and not inside a Python package. That main script then imports the necessary modules or packages and runs the application. This allows the Python import system to function normally and respect the package boundaries and relative imports correctly. Essentially, I'm trying to emphasize the distinction between packages and scripts.

There are a number of project structures I can imagine that would achieve this.

Example 1

I suppose the simplest is something like this:

example/
├── .env
├── .gitignore
├── docker-compose.yaml
├── Dockerfile
├── README.md
├── requirements.txt
├── serve.py
└── app/
   ├── apps/
   │   ├── __init__.py
   │   └── handlers.py
   ├── config/
   │   ├── __init__.py
   │   └── settings.py
   └── __init__.py

In this case, serve.py is the entrypoint, and it imports the app package since it would be directly available in the default PYTHONPATH. This still allows users to run the app locally without any additional configuration. The Dockerfile would need to be updated to copy in the entrypoint script in addition to the app package (and update the run command accordingly), but otherwise the Docker setup remains mostly unchanged.

Example 2

Adding a scripts directory provides a natural place for run scripts, initialization scripts, or other app tools.

example/
├── .env
├── .gitignore
├── docker-compose.yaml
├── Dockerfile
├── README.md
├── requirements.txt
├── scripts/
├───── serve.py
└── app/
   ├── apps/
   │   ├── __init__.py
   │   └── handlers.py
   ├── config/
   │   ├── __init__.py
   │   └── settings.py
   └── __init__.py

This is a nice structure when you have a lot of extra tooling or the app starts to require tools like databases with migration scripts, etc. However, you're now in a situation where your entrypoint script is isolated from the app package, which makes running the project locally slightly more complicated. The easiest thing is to modify the PYTHONPATH to include app package when you run the entrypoint script. I'd do that with environment variables, but you can go the sys.path.insert() route in the entrypoint if you like that kind of thing. You'd also have to decide how you want to handle the Docker setup. I'd probably just copy the entrypoint script into the same directory as the app package inside the container, that way you don't have to mess with the PYTHONPATH, that way you don't copy unnecessary scripts into the container by grabbing the entire scripts folder. But if you did need want to allow access to those scripts in the container, you could just adjust the PYTHONPATH environment variable in the container instead.

Example 3

Throw all the source code into a src directory.

example/
├── .env
├── .gitignore
├── docker-compose.yaml
├── Dockerfile
├── README.md
├── requirements.txt
├── src/
├───── serve.py
└───── app/
      ├── apps/
      │   ├── __init__.py
      │   └── handlers.py
      ├── config/
      │   ├── __init__.py
      │   └── settings.py
      └── __init__.py

This should still work locally, but the code is one layer deeper than the project root. In Docker you can just dump the contents of src into the container. This tends to be my preferred approach if you want to keep all the source code together but still isolate it from other project files (git, docker, docs, etc.).

Additional Thoughts

There are pros and cons to each of these, but the common theme is trying to separate the entrypoint script from the application package. You mentioned that other packages follow a package based approach, and that may be true, I haven't looked into enough to know for sure. These are just my recommendations, and I'm sure you could come up with lots of other options. Ultimately it will depend on the use case of your end users, as their needs should dictate their own project structure. Since these example projects are just template, I don't think you can really go wrong one way or the other, as long as the usage is clear and there is some documentation of the expectations on the user side.

If you decide which way you want to go, I'm happy to write up some additional docs to help clarify the project structure, assumptions, as well as pros and cons so other users know what to expect or how to modify the structure to fit their needs.

Hope this helps!

[Lancetnik]

@FyZyX sorry for leaving this problem for a long time. As you can see, we have an another Issue with relative imports.
Obviously the current template project structure is not clear enough to use it in a comfortable way.

Thus, we need to refactor it and I preffer the 3rd example from the proposed.

If you are interested in a contribution, I will be glad to merge your PR with this changes.