hivesolutions/appier

Joyful Python Web App development

appier's Issues

Instances auto-discovery (multicast)

Description

By using multicast UDP it would be possible to discover other services with the same INSTANCE values and start communicating with them for certain puposes.

Election system to determine the one that is considered the master.

For instance it should be possible to discover a configuration server that way, or for Ripe discover the proper composer instance in the current network and auto-configure it.

References

https://docs.mongodb.com/manual/core/replica-set-elections/

Support for extending Appier sessions

Description

There seems to be no friendly way of extending an Appier session. In some contexts, it's desirable to be able to extend the session indefinitely.

Solution

The expiration date is being set when the session object is being created. This date can be changed only by accessing the internal property directly. Probably there should be an refresh() method to easily refresh the session's expiration date (current date + default timeout).

Also, in RedisSession values are being stored with SETEX, therefore they are deleted by Redis when the timeout is reached. To extend the session in RedisSession, the value's expiration date must be extended using the EXPIRE command to provide a new timeout.

Support for ACL gathering

Description

It should be possible by using code to gather the complete set of ACL tokens that are registered for the currently running application. This way it would be possible to list the complete set of possible permission for the running application context.

Implementation

Should be possible to use appier.get_tokens() to retrieve the complete set of ACL tokens and action methods (from controllers) associated with them.

Support for accessing current date in appier templates

Description

It should be possible for someone to print out the current date or its individual parts (year, month, day, hours, minutes, seconds).

Solution

Making the datetime module acessible in the template should solve the problem.

Invalid escape sequences

SyntaxWarning: invalid escape sequence

Performing system checks...
System check identified no issues (0 silenced).
March 19, 2024 - 17:10:56
Django version 5.0.2
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
/usr/local/lib/python3.12/site-packages/appier/base.py:197: SyntaxWarning: invalid escape sequence '\('
  REPLACE_REGEX = re.compile("(?<!\(\?P)\<((\w+)(\([\"'].*?[\"']\))?:)?(\w+)\>")
/usr/local/lib/python3.12/site-packages/appier/base.py:203: SyntaxWarning: invalid escape sequence '\<'
  INT_REGEX = re.compile("\<int:(\w+)\>")
/usr/local/lib/python3.12/site-packages/appier/base.py:207: SyntaxWarning: invalid escape sequence '\<'
  REGEX_REGEX = re.compile("\<regex\([\"'](.*?)[\"']\):(\w+)\>")
/usr/local/lib/python3.12/site-packages/appier/base.py:220: SyntaxWarning: invalid escape sequence '\('
  CSS_ABS_REGEX = re.compile(b"url\((?!(http:\/\/|https:\/\/|\/\/|\/))([^\)]+)\)")
/usr/local/lib/python3.12/site-packages/appier/validation.py:40: SyntaxWarning: invalid escape sequence '\:'
  SIMPLE_REGEX_VALUE = "^[\:\.\s\w-]+$"
/usr/local/lib/python3.12/site-packages/appier/validation.py:45: SyntaxWarning: invalid escape sequence '\w'
  EMAIL_REGEX_VALUE = "^[\w\d\._%+-]+@[\w\d\.\-]+$"
/usr/local/lib/python3.12/site-packages/appier/validation.py:50: SyntaxWarning: invalid escape sequence '\w'
  "^\w+\:\/\/([^@]+\:[^@]+@)?[^\:\/\?#]+(\:\d+)?(\/[^\?#]+)*\/?(\?[^#]*)?(#.*)?$"
Watching for file changes with StatReloader
/usr/local/lib/python3.12/site-packages/appier/util.py:1101: SyntaxWarning: invalid escape sequence '\`'
  """
/usr/local/lib/python3.12/site-packages/appier/base.py:197: SyntaxWarning: invalid escape sequence '\('
  REPLACE_REGEX = re.compile("(?<!\(\?P)\<((\w+)(\([\"'].*?[\"']\))?:)?(\w+)\>")
/usr/local/lib/python3.12/site-packages/appier/base.py:203: SyntaxWarning: invalid escape sequence '\<'
  INT_REGEX = re.compile("\<int:(\w+)\>")
/usr/local/lib/python3.12/site-packages/appier/base.py:207: SyntaxWarning: invalid escape sequence '\<'
  REGEX_REGEX = re.compile("\<regex\([\"'](.*?)[\"']\):(\w+)\>")
/usr/local/lib/python3.12/site-packages/appier/base.py:220: SyntaxWarning: invalid escape sequence '\('
  CSS_ABS_REGEX = re.compile(b"url\((?!(http:\/\/|https:\/\/|\/\/|\/))([^\)]+)\)")
/usr/local/lib/python3.12/site-packages/appier/validation.py:40: SyntaxWarning: invalid escape sequence '\:'
  SIMPLE_REGEX_VALUE = "^[\:\.\s\w-]+$"
/usr/local/lib/python3.12/site-packages/appier/validation.py:45: SyntaxWarning: invalid escape sequence '\w'
  EMAIL_REGEX_VALUE = "^[\w\d\._%+-]+@[\w\d\.\-]+$"
/usr/local/lib/python3.12/site-packages/appier/validation.py:50: SyntaxWarning: invalid escape sequence '\w'
  "^\w+\:\/\/([^@]+\:[^@]+@)?[^\:\/\?#]+(\:\d+)?(\/[^\?#]+)*\/?(\?[^#]*)?(#.*)?$"
[19/Mar/2024 17:11:00] "GET /api/rgp/list HTTP/1.1" 200 820
/usr/local/lib/python3.12/site-packages/appier/util.py:1101: SyntaxWarning: invalid escape sequence '\`'
  """
/usr/local/lib/python3.12/site-packages/appier/base.py:197: SyntaxWarning: invalid escape sequence '\('
  REPLACE_REGEX = re.compile("(?<!\(\?P)\<((\w+)(\([\"'].*?[\"']\))?:)?(\w+)\>")
/usr/local/lib/python3.12/site-packages/appier/base.py:203: SyntaxWarning: invalid escape sequence '\<'
  INT_REGEX = re.compile("\<int:(\w+)\>")
/usr/local/lib/python3.12/site-packages/appier/base.py:207: SyntaxWarning: invalid escape sequence '\<'
  REGEX_REGEX = re.compile("\<regex\([\"'](.*?)[\"']\):(\w+)\>")
/usr/local/lib/python3.12/site-packages/appier/base.py:220: SyntaxWarning: invalid escape sequence '\('
  CSS_ABS_REGEX = re.compile(b"url\((?!(http:\/\/|https:\/\/|\/\/|\/))([^\)]+)\)")
/usr/local/lib/python3.12/site-packages/appier/validation.py:40: SyntaxWarning: invalid escape sequence '\:'
  SIMPLE_REGEX_VALUE = "^[\:\.\s\w-]+$"
/usr/local/lib/python3.12/site-packages/appier/validation.py:45: SyntaxWarning: invalid escape sequence '\w'
  EMAIL_REGEX_VALUE = "^[\w\d\._%+-]+@[\w\d\.\-]+$"
/usr/local/lib/python3.12/site-packages/appier/validation.py:50: SyntaxWarning: invalid escape sequence '\w'
  "^\w+\:\/\/([^@]+\:[^@]+@)?[^\:\/\?#]+(\:\d+)?(\/[^\?#]+)*\/?(\?[^#]*)?(#.*)?$"

The regex errors come from the use of "invalid escape sequences" in regex patterns defined within the appier library. In Python 3.6 and later, sequences like (, <, and : in regular expressions should be either double escaped (e.g., \() or, preferably, raw string literals should be used by prefixing the string with r, to avoid these warnings. This is because such sequences are not valid Python string escape sequences, and starting from Python 3.6, a SyntaxWarning is raised for these.

Here's an example to illustrate the adjustment:

Original code in appier that might be causing warnings:

REPLACE_REGEX = re.compile("(?<!\(\?P)\<((\w+)(\([\"'].*?[\"']\))?:)?(\w+)\>")

Adjusted code using a raw string literal:

REPLACE_REGEX = re.compile(r"(?<!\(\?P)\<((\w+)(\([\"'].*?[\"']\))?:)?(\w+)\>")

When X-Forwarded-For is presetn, get_address returns the whole list of IPs

The get_address method needs to parse the X-Forwarded-For header and return only the first one.
It also needs to check if other headers are used, at least x-real-ip and x-client-ip.

This is critical to run applications behind proxies.

Datbase migration infra-structure

Description

It should be really cool to create a migration infra-structure like the one present in Migratore.

Description

The idea is to build an infra-structure that is agnostic from the underlying database manager used.

Support for forcing validation type manually

Description

Currently the _validate() method uses the validation methods in validate() or validate_new(), depending on whether the object is new or not. It's useful in some contexts to be able to call the _validate() method an be able to tell it which validation methods it should execute, one or the other, without relying on the object being validated.

Support for recursive serialization to maps

Description

Appier has support for converting a model to a dictionary, however, it has no support for recursively doing so for the model's relations.

Solution

Here's a sample solution (checking for relations is not correct):

def map_full(self):
    cls = self.__class__
    map = self.map()

    for name, value in appier.legacy.items(map):
        _definition = cls.definition_n(name)
        _type = _definition.get("type", None)
        if not _type: continue

        type_s = str(_type)
        if type_s == "<class 'appier.typesf._Reference'>":
            value = getattr(self, name)
            map[name] = value.map_full()
        elif type_s == "<class 'appier.typesf._References'>":
            value = getattr(self, name)
            map[name] = [_value.map_full() for _value in value]

    return map

Reload of application in debug

Description

To be able to efficiently debug an Appier application it's very important to be able to have it reloaded at a development time. Effectively avoiding the need to restart the application & debugger.

Implemenation

Most of the implementation requires the checking of changes in the source files.
The overall algorithm can be defined as:

• Detect the files that belong to a certain project
• Check for changes on those files
• If changes exist reload the associated modules

App instance retrieval on a per-thread basis

Description

While using Appier it's important to keep in mind that many features of it use the concept of global variable to simplify the usage process.

These same global variables may created weird runtime issues while running the App, mainly while running multiple Apps in the same process.

To avoid some of thoses issues a per-thread association should be created between an App and the thread where it's container (web server) is running.

Solutions

The start() and stop() method of an app are called "inside" the web server container and should be used for this kind of registration.

Support for proper data type validation fails

Description

Currently it's possible to store an integer on a string based field using the Appier data model storage engine. This is a serious vulnerability that may lead to corruption of the database.

Solution

Ensure proper validation of the data types on storage/persist operations.

Add documentation for Appier Email client

Description

Appier has an email client but it's currently referenced anywhere in the documentation. A simple example on how to use the client should be added to the docs.

Async support for database access layer (eg: Mongo)

Description

Using a pool of threads and the Future based abstraction it should be possible to adapt the current data layer access to an async fashion.

PyMongo with callback support.

References

https://motor.readthedocs.io/en/latest/examples/callbacks-and-coroutines.html#with-coroutines

Support for remote syslog as a logging handler

Description

Sometimes it's relevant to add a remote log manager (rsyslog) to the main App instance from Appier.

APPIER_RSYSLOG=log.hive:514 or RSYSLOG=log.hive:514

Referneces

https://docs.python.org/2/library/logging.handlers.html#sysloghandler

Async calls seem slow 🐢

Description

After conducting some empirical testing, Appier is underperforming regarding async call handling.

Running the same App through FastAPI handles at least 10x-100x more requests than Appier.

Implementation

Try to find a fix for the issues affecting Appier's async call handling.

Verify: there's some timeout kind of waiting going on, it is something to look into.

Support for including non-defined attributes in map serialization

Description

When calling map() on a model, the attributes that are defined in the model are set in the map, but the undefined ones aren't. It would be desirable to be able to include all attributes by passing a special flag to the method.

Bug: Appier routes don't support resource names with special characters

Description

Appier routes like this:

@appier.route("/api/orders/<int:number>/attachments/<str:attachment_name>", "GET", json = True)

work as expected and match with an URL such as:

https://URL/api/9999/attachments/docname.pdf

However, they do not match the following two:

https://URL/api/9999/attachments/doc%20name.pdf
https://URL/api/9999/attachments/docname(1).pdf

It seems spaces and parenthesis are not supported. The resources are encoded by the likes of a method like JavaScripts' encodeURIComponent and are valid URL syntax, so they should be supported.

Cron Scheduler

Description

Add support for a Cron-like Scheduler that can easily handle the schedule of simple tasks regularly.

This feature has the potential to greatly simplify the way regular job handling is done within the Appier context.

Implementation

The solution should expose an interface similar to:

def schedule(self, task: function, cron: string):
    pass

Tune X-Frame-Options

Description

Be able to set different values for X-Frame-Options.

Related:

appier/src/appier/base.py

Line 141 in c32433f

FRAME_OPTIONS = "SAMEORIGIN"

Task abstraction for async execution control unit

Description

Sometimes it's important to have a proper abstraction for a working unit inside an abstraction system.

Having per example a DownloadTask that allows the downloading of an HTTP based asset and that properly notifies any listening handler should be the aim of this operation.

Implementation

Create the Task class abstraction and then use the websocket infra-structure to make Appier Admin control all of these tasks.

There should be some co-relation between this Task class and the Future class.

References

socket.io chat example

Appier can't fill unpersisted models with relations

Description

Calling new() on a model with relations causes an error like the following:

AttributeError

500 - 'module' object has no attribute 'Wishlist'
Traceback (most recent call last):
File "/repo.extra/appier/src/appier/base.py", line 803, in application_l
result = self.handle()
File "/repo.extra/appier/src/appier/base.py", line 931, in handle
else: result = self.route()
File "/repo.extra/appier/src/appier/base.py", line 1159, in route
else: return_v = method_i(*args, **kwargs)
File "/repo.extra/myswear/src/myswear/controllers/web/base.py", line 104, in do_signup
user = sadapters.models.User.new()
File "/repo.extra/appier/src/appier/model.py", line 354, in new
if fill: model = cls.fill(model)
File "/repo.extra/appier/src/appier/model.py", line 970, in fill
default = _type._default() if hasattr(_type, "_default") else default
File "/repo.extra/appier/src/appier/typesf.py", line 424, in _default
return cls(None)
File "/repo.extra/appier/src/appier/typesf.py", line 373, in __init__
self.__start__()
File "/repo.extra/appier/src/appier/typesf.py", line 417, in __start__
if is_reference: self._target = self.__class__._target()
File "/repo.extra/appier/src/appier/typesf.py", line 428, in _target
if is_reference: return getattr(common.base().APP.models_i, target)
AttributeError: 'module' object has no attribute 'Wishlist'

HTTP calls in App's init or start method corrupt Event Loop

The below sample app starts, runs and answers requests, but some timeout (about 30%):

#!/usr/bin/python
# -*- coding: utf-8 -*-

import appier
import appier_extras


class App(appier.WebApp):
    def __init__(self, *args, **kwargs):
        appier.WebApp.__init__(
            self, name="app", parts=(appier_extras.AdminPart,), *args, **kwargs
        )
        self._load_scales()

    @appier.route("/bundles", "GET")
    def get_bundles(self):
        return self.bundles

    def _load_scales(self):
        url = "https://api.github.com/"
        result = appier.get(url)
        locales = {"en_us": result}
        for locale, bundle in appier.legacy.iteritems(locales):
            self._register_bundle(bundle, locale, context="scales")


if __name__ == "__main__":
    app = App()
    app.serve()

However, the following has no issues:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import appier
import appier_extras


class App(appier.WebApp):
    def __init__(self, *args, **kwargs):
        appier.WebApp.__init__(
            self, name="app", parts=(appier_extras.AdminPart,), *args, **kwargs
        )
        self._load_scales()

    @appier.route("/bundles", "GET")
    def get_bundles(self):
        return self.bundles

    def _load_scales(self):
        locales = {"en_us": {"current_user_url": "https://api.github.com/user"}}
        for locale, bundle in appier.legacy.iteritems(locales):
            self._register_bundle(bundle, locale, context="scales")


if __name__ == "__main__":
    app = App()
    app.serve()

In my local development machine, both work. In my docker deployed app, the first fails and the second works.

We had an hypothesis where the HTTP call in the init method was invalid since the event loop was still warming up. We replaced the appier.get call for a requests.get (python requests lib) and it worked:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import appier
import appier_extras


class App(appier.WebApp):
    def __init__(self, *args, **kwargs):
        appier.WebApp.__init__(
            self, name="app", parts=(appier_extras.AdminPart,), *args, **kwargs
        )
        self._load_scales()

    @appier.route("/bundles", "GET")
    def get_bundles(self):
        return self.bundles

    def _load_scales(self):
        import requests
        url = "https://api.github.com/"
        result = requests.get(url)
        result = result.json()

        locales = {"en_us": result}
        for locale, bundle in appier.legacy.iteritems(locales):
            self._register_bundle(bundle, locale, context="scales")


if __name__ == "__main__":
    app = App()
    app.serve()

This is an issue whether the HTTP remote call is made in the init method or the start method. I would assume Appier's thread pool manager is still warming up and such an early HTTP call is corrupting it in some way.

Controller email method has no support for non-template messages

Description

It would be desirable to be able to send pre-processed messages using this method, instead of only templates, however the method has no support for that.

Remove deprecation warnings in pymongo 3

Description

PyMongo 3 deprecated a series of methods and they must be changed using a properly conditional strategy, to avoid any warning message.

Images

Request Limit

Description

It's important to have a mechanism that allows control of the number of requests per time unit. This will provide a way to control an unintentional (or intentional) DOS.

Implementation

Create a decorator appier.requests that receives the number of requests per minute allowed for the action method in the controller.

The implementation should take inspiration from PreflightPart.

Inspiration

• express-rate-limit - npm