Giter Site home page Giter Site logo

bespin's Introduction

Me

Oh hi there and welcome to my personal github!

You'll find here a collection of random projects that either javascript or Python with varying levels of quality and age.

I'll give you a guide to what can be found here:

Photons

I have a tendency to find problems from my job and then making them something fun to do outside work. This is very much one of those, and I'm super proud of what I've made here.

Everything now lives in https://github.com/delfick/photons

But it was moved from https://github.com/delfick/photons-core and https://github.com/delfick/photons-interactor

It's the public version of a framework I started in early 2016 that heavily uses asyncio to talk to LIFX smart lights.

Docs can be found at https://photons.delfick.com

A docker client

I'm equally proud of this next project but in a very different way. I like the code quality of Photons these days, but the code quality in https://github.com/delfick/harpoon is not great.

I'm proud of it though because what I did there paved the way for what has since become https://github.com/delfick/delfick_project and is essentially the foundation of everything that I've written since then.

Also, Harpoon is cool. I haven't met many who agree with that assessment, but it solved a bunch of problems I had with Docker back in 2014 and I still use it in some places to make it easy to create multiple docker containers.

A truly imaginatively named project

https://github.com/delfick/delfick_project is a small monorepo of quite a few different projects I created.

  • Encapsulating patterns I used everywhere to bootstrap an application
  • Helpers to make logging so much more enjoyable
  • An error class that has a bunch of features
  • A class that helps take a tree of dependencies and return those in layers
  • Tools for managing plugins via entry points
  • An object that can be used to treat multiple dictionaries as if they are one
  • The ability to validate and normalise data

Those last two have a lot of features I can't fit into a single sentence but all of this can be read at https://delfick_project.readthedocs.io/

Data normalisation

Part of delfick_project is the input_algorithms code that lets us validate and normalise data to create well structured objects.

In late 2020 I wrote this idea for how I wanted to improve the API it provides: https://github.com/delfick/delfick_project/blob/ideas/rewrite-norm-api/IDEAS.rst

However I never found the time. Recently (middle of 2022) I decided to achieve this a bit differently and I created a layer on top of cattrs that means you can do cattrs transformation whilst also passing down meta information along for the way.

This project is strcs: https://strcs.readthedocs.io/en/latest/

This one is my first attempt at seeing what I can do with Python Type hints and it's kinda cool :)

My oldest project I still use

In 2010 I was looking for a way to write rspec like tests in Python. There is a big blocker to be able to do this in python, and that is you can't create a multi line lambda. This means that you can't just have an "it" function that takes in a string and the function to call for that test.

Existing libraries solved this by saying you should write your tests functions like normal "def it_should_definitely_do_the_thing_i_want_it_to_do(self):" and then you decorate it with a nice string. I really hate writing sentences like that.

Eventually I discovered that you can create a custom codec for python and as long as that codec is registered before you import a file, if the file specifies that codec, python will translate the file before executing it.

Effectively I made it so describe blocks become classes and it blocks become functions. And then a bunch of code to handle nested levels of indentation.

https://noseofyeti.readthedocs.io/

The problem with that is tooling, specifically linting and formatting. In 2020 I managed to fix both those problems and now my editor can auto format and lint those files when I save them. Best of all the worlds!

The most what is happening here

I have no idea what is happening here https://github.com/delfick/app-play

I was clearly hypomanic when I wrote it!

Virtualenvs

I used to put everywhere a bash script that when run would figure out if a virtualenv already exists. If one didn't then it would make one for you, otherwise it would use what is there. The script would then look at the dependencies you want in that virtualenv and if the versions of those didn't match in the virtualenv it would install those to the desired version. Once we have the virtualenv setup, it'll execute a particular program from the virtualenv.

I eventually decided to make that it's own thing: https://venvstarter.readthedocs.io/en/latest/

I find this really useful because I don't have to think about dependency management. I just say what I want and when I run the script it'll make sure I have those. This is especially useful when you provide something to others and what they want is the thing to work, they don't care about making sure dependencies are correct.

The one downside here is there's no lockfile, but there doesn't really exist programmatic APIs for that functionality in current efforts to improve Python dependency management.

Everything else

Clearly I spend way too much of my time programming! But you'll find other things here, like helpers for a web server, a thing for converting dictionaries to xml, A cool tool for making the lambda side of an Alex skill, A pure python gpg decryptor, An alternative asyncio plugin for pytest, a python library for finding the commit times of github files, pure hilarity, and a plugin for gedit that makes it modal.

Also my website and the blog it contains are all written in svelte (code), which is awesome!

I'm more than my programming, but this is my github so I'll only use this to walk you through my projects!

bespin's People

Contributors

atward avatar danvela avatar delfick avatar jonhiggs avatar w0203j avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bespin's Issues

Support for YAML CloudFormation

All the current code refers to stack_json and JSON validation is done.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Something went wrong! -- BadFile
	"bad file. No JSON object could be decoded"

Determining account id requires role

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Something went wrong! -- BespinError
	"There are no roles in your account, I can't figure out the account id"

This can be done without the need for roles via sts:GetCallerIdentity

config_root should be accurate for the config file a value was defined

Through the use of bespin: extra_files other bespin configuration files can be loaded. Non-absolute file definitions in these files (ie: stack_json|yaml, stack_policy, params_json|yaml [essentially anything defined as subclass of bespin.option_spec.bespin_specs.valid_params]) should be loaded relative to the config_root of that config file.

Currently files are just opened (bespin_specs.py#L61) - assumes os.getcwd == {config_root}?

Stack parameters are required

Some CF templates do not require parameters, so omitting params_json and params_yaml should be valid from an end user perspective.

Is there any history behind why stack parameters are required? See:

Defining an empty mapping (params_yaml: {}) is a workaround for anyone affected by this.

Using formatted environment values breaks non-environment commands

Probably best described by example:

$ cat bespin.yml
environments:
  example:
    tags:
      env: "{environment}"

stacks:
  example: {}
$ bespin
15:06:55 INFO    option_merge.collector Adding configuration from /Users/adamward/tmp/test/bespin.yml
15:06:55 INFO    bespin.collector Converting bespin
15:06:55 INFO    bespin.collector Converting environments

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Something went wrong! -- BadSpecValue
	"Bad value"	meta={source=['./bespin.yml'], path=environments}
errors:
=======

	"Bad value"	meta={source=['./bespin.yml'], path=environments.example}
	errors:
	=======

		"Bad value. Expected a value but got none"	meta={path=environments.example.account_id}
	-------
		"Bad value"	meta={source=['./bespin.yml'], path=environments.example.tags}
		errors:
		=======

			"Something wrong with this specification. Spec doesn't know how to deal with this value"	meta={source=['./bespin.yml'], path=environments.example.tags.env}	spec=<input_algorithms.spec_base.formatted object at 0x1106ee310>	val={environment}
		-------
	-------
-------
$ bespin show
#also affected, same output

Also using {region} produces similar results.

Migrate boto to boto3

boto2 does not have support for things like:

  • sts:GetCallerIdentity
  • authentication profiles
  • tags during cloudformation update-stack
  • CloudFormation change sets

Boto3 migration checklist:

  • bespin.amazon.credentials
  • bespin.amazon.cloudformation
  • bespin.amazon.ec2
  • bespin.amazon.s3
  • bespin.amazon.sqs
  • bespin.amazon.kms

Support for large templates

Templates >51200 are currently not supported.

Something went wrong! -- BadStack
	"Bad stack. Amazon says no"	error_code=400	error_message=1 validation error detected: Value '---
<snip large template>
' at 'templateBody' failed to satisfy constraint: Member must have length less than or equal to 51200	filename=/var/folders/yv/1cr1g_vj5fjds555x2412kn40000gn/T/tmpDMPglT

Templates would either have to be uploaded to S3 and/or referenced via URI.

DynamicVariable.stack is always string

I can't work out a situation where var.stack is ever not a string.

if hasattr(var, 'stack') and not isinstance(var.stack, six.string_types):

if isinstance(self.stack, six.string_types):

I've gone through git history and it seems it's always been a string. @delfick, can you shed any light about this?
How would you go about calling Stack.output_yet_to_be_deployed?

Bespin raises errors when rate-limited

Traceback (most recent call last):
  File ".bespin/bin/bespin", line 11, in <module>
    sys.exit(main())
  File ".bespin/lib/python3.4/site-packages/delfick_app.py", line 162, in main
    app.mainline(argv, **execute_args)
  File ".bespin/lib/python3.4/site-packages/delfick_app.py", line 230, in mainline
    self.execute(args_obj, args_dict, extra_args, handler, **execute_args)
  File ".bespin/lib/python3.4/site-packages/bespin/executor.py", line 31, in execute
    collector.configuration["task_runner"](collector.configuration["bespin"].chosen_task)
  File ".bespin/lib/python3.4/site-packages/bespin/task_finder.py", line 20, in task_runner
    return self.tasks[task].run(self.collector, self.stack_finder(task), available_actions, self.tasks, **kwargs)
  File ".bespin/lib/python3.4/site-packages/bespin/option_spec/task_objs.py", line 108, in run
    return task_action(collector, stack=stack, artifact=artifact, tasks=tasks, **extras)
  File ".bespin/lib/python3.4/site-packages/bespin/actions.py", line 106, in deploy
    Deployer().deploy_stack(stack, collector.configuration["stacks"])
  File ".bespin/lib/python3.4/site-packages/bespin/operations/deployer.py", line 44, in deploy_stack
    self.build_stack(stack)
  File ".bespin/lib/python3.4/site-packages/bespin/operations/deployer.py", line 83, in build_stack
    stack.cloudformation.wait(timeout=stack.build_timeout, rollback_is_failure=True)
  File ".bespin/lib/python3.4/site-packages/bespin/amazon/cloudformation.py", line 171, in wait
    events = description.describe_events()
  File ".bespin/lib/python3.4/site-packages/boto/cloudformation/stack.py", line 76, in describe_events
    next_token=next_token
  File ".bespin/lib/python3.4/site-packages/boto/cloudformation/connection.py", line 580, in describe_stack_events
    StackEvent)])
  File ".bespin/lib/python3.4/site-packages/boto/connection.py", line 1186, in get_list
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.BotoServerError: BotoServerError: 400 Bad Request
<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">
  <Error>
    <Type>Sender</Type>
    <Code>Throttling</Code>
    <Message>Rate exceeded</Message>
  </Error>
  <RequestId>xxxxxxxx-0062-11e6-9396-33252275e1d1</RequestId>
</ErrorResponse>

Stacktrace when stack_yaml and stack_json not found.


Something unexpected happened!! Please file a ticket in the issue tracker! None
===============================================================================

Traceback (most recent call last):
  File "/usr/local/bin/bespin", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/delfick_app.py", line 188, in main
    app.mainline(argv, **execute_args)
  File "/usr/local/lib/python2.7/site-packages/delfick_app.py", line 260, in mainline
    self.execute(args_obj, args_dict, extra_args, handler, **execute_args)
  File "/usr/local/lib/python2.7/site-packages/bespin/executor.py", line 31, in execute
    collector.configuration["task_runner"](collector.configuration["bespin"].chosen_task)
  File "/usr/local/lib/python2.7/site-packages/bespin/task_finder.py", line 20, in task_runner
    return self.tasks[task].run(self.collector, self.stack_finder(task), available_actions, self.tasks, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/bespin/option_spec/task_objs.py", line 108, in run
    return task_action(collector, stack=stack, artifact=artifact, tasks=tasks, **extras)
  File "/usr/local/lib/python2.7/site-packages/bespin/actions.py", line 106, in deploy
    Deployer().deploy_stack(stack, collector.configuration["stacks"])
  File "/usr/local/lib/python2.7/site-packages/bespin/operations/deployer.py", line 26, in deploy_stack
    Builder().sanity_check(stack, stacks, ignore_deps=ignore_deps, checked=checked)
  File "/usr/local/lib/python2.7/site-packages/bespin/operations/builder.py", line 17, in sanity_check
    stack.sanity_check()
  File "/usr/local/lib/python2.7/site-packages/bespin/option_spec/stack_objs.py", line 350, in sanity_check
    self.validate_template_params()
  File "/usr/local/lib/python2.7/site-packages/bespin/option_spec/stack_objs.py", line 314, in validate_template_params
    validation = self.validate_template()
  File "/usr/local/lib/python2.7/site-packages/bespin/option_spec/stack_objs.py", line 309, in validate_template
    fle.write(self.dumped_stack_obj)
  File "/usr/local/lib/python2.7/site-packages/bespin/option_spec/stack_objs.py", line 223, in dumped_stack_obj
    return json.dumps(self.stack_obj)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <class 'input_algorithms.spec_base.NotSpecified'> is not JSON serializable

environment can override stack values

User story: Be able to define a set of tags and use them throughout the configuration.
It is a common use case for organisations to define a tagging standard and require that tags are applied to all resources.
Initial implementation from user is to use yaml aliases on mappings (eg: common_tags: &tags) however this is only valid within a single file.

Propose extending Environment to include tags and have that merged over the top of Stack.tags.

Raising this issue to solicit design feedback.

Document formatter and available paths

A large part of user confusion is coming from not knowing what keys/paths are available for the {} config syntax. Recommend documenting a list of what is available, and any additional keys that are derived/appended.

  • account_id
  • config_root
  • environment
  • environments.<env_name>.*
  • region
  • stacks.<stack_name>.*
  • tags.*
  • vars.*

Additionally::

  • __stack_name__
  • __environment__ (ie: environments.environment)
  • __stack__ (ie: stacks.__stack_name__)
  • _key_name_X syntax

Formatters:

  • overview/syntax
  • :env - '${ENV_VAR}'
  • :date - date object from string
  • :underscored- '-' to '_'
  • :config_dir - dirname of the bespin.yml the config option was defined in (NOTE: python2.7+ only)

environment variables

  • declaring before using
  • env
  • build_env
  • stack_name_env

variables (ie: XXX_var_XXX)

  • how this works
  • why it's needed
  • parts of the config where this is valid (params_yaml only?)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.