dag/braindump

Ideas and Experiments

braindump's Issues

Adapters

Could maybe be transparent if the adapter issubclass the requested interface and the constructor has a type annotation:

@adapter  # venusian
class UserPerson(PersonInterface):

    def __init__(self, user:UserInterface):
        self.user = user

    @property
    def first_name(self):
        return self.user.given_name

    @property
    def last_name(self):
        return self.user.family_name

    def greet(self, person):
        return '{greeting}, {person.first_name}'.format(
            greeting=self.user.catch_phrase, person=person)


adapt = AdapterRegistry()
adapt.scan()  #  find @adapter

adapt(user_object, PersonInterface)  # UserPerson(user_object)

Generic toolkit for extensible applications

Probably a complete ripoff of existing options such as the ZCA. I should blame moraes for bugging me about it. :)

• Registry of "utilities"
• Event system with generic signals and special signals for things like when a "utility" has been created, with conditional subscription based on instance checking
• Dependency injection of utilities to event subscribers based on instance checking

# an instance of a jinja environment has been created and registered
@created(jinja2.Environment)
def configure_line_statements(env):  # pass the instance as the first argument
    env.line_statement_prefix = '#'

# a request has started
# find all utilities in the registry matching the annotated types in the function signature
# call the function once for each combination of the found utilities
@started(RequestEvent)
def log_request(log: logbook.Logger, request: werkzeug.BaseRequest):
    log.info('{request.method} {request.path}', request=request)

Tools for defensive programming

Embed constraints in code, that can be disabled in production for performance and fault-tolerance, but helpful in development / automated testing. Python supports this via the assert statement, which is stripped out if Python runs with optimizations (__debug__ is false), but assertions are rather low-level. A support package could provide tools for adding type constraints, design-by-contract etc, in a more high-level fashion than simple assertions. Python also strips out branches for code like if __debug__ when optimized which could allow constraints added via for example decorators to only have a slight overhead at "construction time", that is, usually only once on the first import.

Contracts with magical abuse of frames:

def double(x):
    return x * 2

@pre
def double(x):
    assert isinstance(x, numbers.Number)

@post
def double(r, x):
    assert isinstance(r, numbers.Number)
    assert r > x

Or using decorator arguments:

@accepts(x=numbers.Number)
@returns(numbers.Number, x=operator.gt)
def double(x):
    return x * 2

Annotations:

@constrained
def double(x: numbers.Number) -> numbers.Number, dict(x=operator.gt):
    return x * 2

Constrained attributes using descriptors:

class Person(object):

    name = Attribute(basestring)

When running with optimizations it could simply return None.

Metaprogramming Utilities

Tools

Like with #1, some of these exist in my experimental Attest rewrite, and similarly this idea is a result of it.

• Import hooks
  • Context manager for temporarily installing a hook (possibly thread-aware)
  • Abstract base classes like in Python 3 importlib, possibly port from there
  • Create hooks that apply an ast.NodeTransformer to the source code
  • API for injecting global names to imported modules
• AST utilities
  • Pretty printing AST nodes as plain-text trees for debugging
  • Generate source code from AST nodes
  • Programmatically set the argspec of a callable, by generating the AST for a function definition with that argspec, wrapping the callable. This is similar to the "decorator" module but without string programming.
  • Apply transformations to source code without losing whitespace/formatting/comments, by mapping line numbers
    • Could be useful for scripted refactoring, automating porting for backwards-incompatible changes between versions of a package (like Armin likes to do but without the scary regexps) and other "2to3"-like programs
  • Similarly, build a "concrete" syntax tree wrapping an AST and the original source code
    • Could allow pyflakes-like programs to read pragmas in comments otherwise lost in an AST
  • Could include a generic implementation of an assertion rewriter
  • Highlevel wrappers for creating nodes, such as sane defaults for ast.Call arguments and expression contexts
• Highlevel objects for introspection
  • AST nodes
  • Frames
  • Tracebacks
  • Objects corresponding to language constructs (classes, functions etc), mostly wrappers for functions in the inspect module:
    • Comments and docstrings, with whitespace-cleaning and summaries based on the first non-empty line
    • Call signature, including automatic handling of classes as constructors and non-function objects implementing __call__
    • Source code
    • AST of source code (in the highlevel wrapper)
• Identifiers (done)
  • Encode arbitrary strings as syntactically valid Python identifiers, and vice versa. For example import, a reserved keyword, becomes import_ (the standard convention) and string with spaces-and-hyphens becomes string_with_spaces_and_hyphens and so on. This is useful for example when mapping command-line arguments to functions and attributes in Python, such that while --not --master-of-the-universe=me might map to while_(not_=True, master_of_the_universe='me'). It should also be reversible, with a parameter for the underscore substitute.
    • http://docs.python.org/reference/lexical_analysis.html#identifiers
    • http://docs.python.org/dev/reference/lexical_analysis.html#identifiers (unicode in Python 3 identifiers)
    • http://docs.python.org/library/keyword.html
    • Research conventions for initial digits
      • Prepending an underscore seems sensible, though that implies a "private" name
      • Perhaps prepend a single lower-cased letter such as "a" or "x"
• Decorator utilities
  • Helpers for creating decorators that take no arguments but can be parametrized via methods, for example @test and @test.using(parameters) rather than requiring the first to be @test() or ambiguously checking if the arguments are a single callable

Name Ideas

Caution: Asperger humor ahead!

• Blast: programming becomes a blast; it will explode in your face; ends in "AST"
• Crystal: "meta" is slang for methamphetamine in Swedish; if you "import" drugs you might become "hooked"; "crystal clear" introspection

Interfaces

Related to #5, #6 and #7, but only focusing on interfaces.

Terms and Naming Conventions

• IPerson - ugly, ambiguous
• PersonInterface - long and wordy
• PersonType - inconsistent with the stdlib types module
• PersonT - weird, confusing
• PersonBase - misleading (not a proper base class)
• BasePerson - not even sure what this convention means (BasicPerson?)
• Person - beautiful and consistent with collections.abc, but easily conflicts with concrete classes
• AbstractPerson - long and wordy, suggestive of abc more than interface
• PersonProtocol - long and wordy, probable conflict with classes
• Personal - adjective for what the interface provides (weird example); consistent with the convention of classes as nouns and methods as verbs
• Personality - noun for an inherent property rather than the concrete "thing" (person). maybe mix with adjectives.
• People - interfaces as sets: "the person object is in the set of People", "this is how People behave". potential conflict with plural classes.

The longer ones become an annoyance once you start putting them in function annotations in particular. You want the signature to be short and concise, preferably fitting on one line. Also repeating "Interface" or such for every type quickly starts to sound like a broken record.

The stdlib doesn't seem completely self-consistent anyway:

• collections: nouns (Mapping...)
• importlib: nouns (Finder, Loader...)
• formatter: "abstract" (AbstractFormatter, AbstractWriter)
• io: "base" (IOBase)

The general trend though seems to be a preference recently for plain nouns with mixed abstract/concrete methods.

API

class PersonType (metaclass=Interface):

    # typed attribute
    first_name = str

    # untyped attribute using the Ellipsis constant
    last_name = ...

    # special method for asserting invariants about the object
    def __invariant__(self):
        assert self.first_name or self.last_name, 'must set at least one of the names'

    # abstract methods with type annotations
    def greet(self, person: PersonType) -> str:
        # normally "PersonType" isn't available yet when the "def" executes;
        # a workaround could be to have a special "This" type á la Traits,
        # although it could be aliased to the class name using __prepare__ in the metaclass

        # contractual pre-condition assertions in the abstract method body
        assert person is not self, 'not greeting myself, silly'

        # generator coroutines for contractual post-conditions
        res = yield
        assert len(res) < 100, 'i am not a blabbermouth dammit'


# implementations simply inherit from the interface.
# if the implementation is complete, issubclass will be true for the interface,
# if not, instance creation will fail
class Person (PersonType):
    ...

If a class conforms to an interface but does not inherit from it, issubclass should be false but there could be a custom API for checking conformance beyond inheritance. Perhaps calling an interface with an object could return the object if it is conforming, and otherwise raise an error, forming a simplistic adapter.

Alternative assert modes

Not sure if useful, but here goes:

AST/Import hook for transforming assertions to substitute for alternative behavior.

• Throw warnings rather than failing hard. Failing hard is often desirable, but sometimes not, and currently your only option is to disable assertions completely.
• Translate to log records: debug for passing assertions, error or warning for failing.
• Inject useful debug information beyond traceback.
• Backport Python 3 behavior of warning for always-true tuple literals.

Framework for CLI applications

Motivation

Though argparse is an improvement over optparse, it still feels more like a low-level toolkit for building more high-level systems. ConfigParser doesn't handle deep nesting and isn't type-aware, in itself.

Ideas

Many of these ideas come from, and are partially implemented in, my unpublished experimental rewrite of Attest, and the idea itself of a CLI framework is a result of it, after noticing a pattern and experiencing some annoyances.

• At the top sits an "application" object that implements the application logic which includes things like the configuration builder, the command-line argument parser etc
• When an application is run an "environment" object wraps contextual state such as os.environ, sys.args, a built config etc
  • But maybe config building should be up to the environment as a built config is immutable and one will likely build it conditionally based on the os.environ and sys.args etc
  • Maybe the environment thus could be an intermediary step towards a fixed "state" object (or simply put everything in the built config)
  • Overkill? :) My experience is that making objects do "too much" and with mutable state usually results in pain in one form or another.
• System for dependency injection from these objects for easy testing and reuse
• Configuration system based around primitive types
  • Nodes as dict-like bunches (done)
  • A system to merge configurations by recursing into nodes (done)
    • Support for includes in configuration files
  • Backends for reading configuration from files:
    • YAML, could use custom "tags" for directives (done)
    • ConfigParser, using dots in option keys for nesting nodes, and some form of automatic type conversion (done)
    • JSON, "because we can" (done)
    • Python, like Sphinx build configs, using ast.literal_eval (or something like it) in "safe" mode
  • Mapping of backends by filename extension: (done)
    • Smart resolution of config filenames, in order:
      • If the name refers to an existing file it is loaded
      • If the filename doesn't end in a known extension, try appending each known extension
      • If more than one file was resolved from the backend extensions, it could:
        • Load them all, but the order would be arbitrary
        • Load the first, but again the definition of "first" would be arbitrary
        • Skip this config name silently
        • Raise some conflict error - probably the best option
    • Load fully qualified config filenames by delegating to the backend mapped to the file extension
  • Syntax like Pyramid asset specs for loading configs with pkg_resources (done)
  • Schemas for untyped backends? Or alternatively infer types from string values. Allow pluggable converters?
    • Could generate documentation from a schema (sphinx/manage/--help etc)
    • Otherwise research if PyYAML supports reading nearby comments
  • Interpolation using newstyle string formatting
  • Some form of standardized handling of importable dotted names, with support for trying a sequence of dotted names (getting the first available one) and possibly handling of class instantiation and function arguments, similar to the functions in paste.util.import_string
  • API for "safe" reading of configuration, using yaml.safe_load and disable string importing etc.
    • Options for whether to fail for unsafe values or if to simply ignore them (leaving any previous or default values in place)
  • Maybe separate the concept of a configuration "builder" and a resulting immutable configuration "state" (though maybe load importable strings lazily) (done)
• Command-line argument parsing with argparse
  • Integration with the configuration system
    • At the most basic level, provide switches for setting and unsetting config options
    • Maybe optionally build the whole parser from the configuration, based either on a schema or the default values, and provide a means to map short options to long ones
  • Maybe expose a more high-level API
  • Maybe a system to expose functions based on argspec etc
  • Combinable sub-commands á la Paver and setuptools
  • Some way to merge multiple parsers into one (does argparse do this already?)
  • Maybe include an improved help formatter class - I don't quite like any of the built in ones
    • Could use ansi escapes to tty outputs
• XHTML-like markup language for terminal output
  • Either built in templating or use Genshi for dynamic functionality
  • Styling with CSS
  • Attribute on text elements for Pygments lexer
  • Handle things like automatic wrapping of content marked up as paragraphs, based on width of terminal, padding etc
  • All this might be too much work :) but the idea is to make outputting to the terminal less imperative and more declarative and semantically rich, to make it easier to test, refactor, reuse ... XML is used because while it's rather crappy for data serialization, marking up text with semantics is where it shines.
• Support for plugging in configurable...
  • Logging
  • Daemonization
  • Diffing, with support for Pygments and the various formats supported by difflib
  • Paging

Existing Options

• http://packages.python.org/pyCLI/
• http://builtoncement.org/

Name ideas

• "mainly": it's a tool for the "main" function
• words with "cli" in them: climactic/(anti-)climax, clingy, clinic, decline, incline, disincline
• words with "tty" in them: kitty, bratty, chatty, fatty, petty, potty, pretty, shitty, witty
• words with "term" in them: determination, intermediary, mastermind, terminology, termite, watermelon

"Rich" objects

Maybe not a useful idea as good implementations might already exist, but this is called a braindump for a reason.

• ABC tools
• Interfaces/Adapters
• Schemata for Python classes: type checking and validators
• Signals
• Metadata

Existing options include zope.interface, enthought's traits, schema libraries like flatland and colander, less generic libraries for dealing with html forms, and all sorts of implementations of database-mapped "model" objects. So maybe this idea is useless :) though I'm pondering the usefulness of unifying all these different use cases.

Sphinx Extensions

New

• Directive for listing authors based on git shortlog -nse
• Link docs for Python modules to source code on GitHub
  • If building docs for a specific tag, lines could also be linked for module members
• Generate changelogs based on versionadded/changed directives, link commit list on GitHub between tags

(Ideally, Git and Github above should read as "arbitrary version control system" and "arbitrary source code host".)

• Auto-building:
  • By watching for file changes
  • As a web server
    • Building for each page request (but not requests for static files and such), or
    • Generating pages in memory on demand like traditional web applications, unless this would be difficult with Sphinx (the "websupport" new in 1.1 might be relevant)
• Easier theme authoring:
  • Sample documentation demonstrating most directives
  • Try a theme without rebuilding:
    • Could symlink (non-templated) stylesheets, or
    • Simply generate a single HTML file for the stylesheet's source location, from the sample rst document

Existing

• Look into if viewcode could be made aware or origin modules. Currently the documented object is assumed to be defined in the module exposing it, which often isn't the case. It also tends to get out of sync and requiring building with a fresh environment.

Venusian Utilities

• Generic dict-like registry object for recording decorators (done)

Something like:

@somedecorator('argument', keyword='also')
def func():
    pass

registry = Registry()
registry.scan(__name__, param='something')

>>> registry['somedecorator']
[Record(object=<function func at ...>,
        args=('argument',),
        kwargs={'keyword': 'also', 'default': 'overridable'},
        params={'param': 'something'})]

Decorators could be created using a @recorder decorator on functions that return the registry key (falling back on the function name):

@recorder
def somedecorator(arg, keyword, default='overridable'):
    """Register a "some"."""

• Scanning of arbitrary (dict-like) namespaces
• Scanning strings by importing as dotted names
• Scanning of packages non-recursively (submitted to upstream)

Name Ideas

• Cytherean: synonym of "venusian"

Single bundling-friendly module for portable imports

lxml suggests this beast for portably importing the "best" available elementtree:

try:
  from lxml import etree
  print("running with lxml.etree")
except ImportError:
  try:
    # Python 2.5
    import xml.etree.cElementTree as etree
    print("running with cElementTree on Python 2.5+")
  except ImportError:
    try:
      # Python 2.5
      import xml.etree.ElementTree as etree
      print("running with ElementTree on Python 2.5+")
    except ImportError:
      try:
        # normal cElementTree install
        import cElementTree as etree
        print("running with cElementTree")
      except ImportError:
        try:
          # normal ElementTree install
          import elementtree.ElementTree as etree
          print("running with ElementTree")
        except ImportError:
          print("Failed to import ElementTree from any known place")

Flask includes this code to get a simplejson module:

json_available = True
json = None
try:
    import simplejson as json
except ImportError:
    try:
        import json
    except ImportError:
        try:
            # Google Appengine offers simplejson via django
            from django.utils import simplejson as json
        except ImportError:
            json_available = False

And this for blinker:

signals_available = False
try:
    from blinker import Namespace
    signals_available = True
except ImportError:
    class Namespace(object):
        def signal(self, name, doc=None):
            return _FakeSignal(name, doc)

    class _FakeSignal(object):
        """If blinker is unavailable, create a fake class with the same
interface that allows sending of signals but will fail with an
error on anything else. Instead of doing anything on send, it
will just ignore the arguments and do nothing instead.
"""

        def __init__(self, name, doc=None):
            self.name = name
            self.__doc__ = doc
        def _fail(self, *args, **kwargs):
            raise RuntimeError('signalling support is unavailable '
                               'because the blinker library is '
                               'not installed.')
        send = lambda *a, **kw: None
        connect = disconnect = has_receivers_for = receivers_for = \
            temporarily_connected_to = connected_to = _fail
        del _fail

Logbook proposes this for stubbing a fake when not installed:

try:
    from logbook import Logger
except ImportError:
    class Logger(object):
        def __init__(self, name, level=0):
            self.name = name
            self.level = level
        debug = info = warn = warning = notice = error = exception = \
            critical = log = lambda *a, **kw: None

Patterns like these could be located and put in a single, bundle-friendly module, with apipkg-like magic for lazy importing.