Giter Site home page Giter Site logo

ryankwilliams / blaster Goto Github PK

View Code? Open in Web Editor NEW
3.0 2.0 1.0 101 KB

Blast off a list of tasks concurrently calling each tasks methods defined

License: GNU General Public License v3.0

Makefile 1.45% Python 98.55%
python python-library python-package blaster concurrent-programming

blaster's Introduction

Blaster

PR-Verification PyPI version

Blaster is a library that provides the ability to blast off a list of tasks and call each of their given methods concurrently. Blaster uses Python's built-in multiprocessing library to run the list of tasks.

Installation

Blaster can be easily installed with a one line command. It is available on pypi. It is recommended (as best practice) to create a virtual environment and install blaster. Please see the commands below to install blaster.

# Create a python virtual environment
python3 -m venv venv

# Activate the virtual environment
source venv/bin/activate

# Install blaster
(venv) pip install blaster

Examples

At the root of blaster project, you will see a examples folder. Within this folder you will find simple examples on how you can use blaster to efficiently run many tasks.

Output

When blaster calls its blastoff method, on completion. It will return back to you a list of task results. Within each task dictionary it will have the original task data passed in along with a couple new keys. A status key which is an integer (0 or 1) to determine pass or fail. If a task failed, it would have a traceback key with the exception raised for helpful troubleshooting.

Terminology

Task

A task is a python dictionary that defines the task to be blasted off. A task must contain three keys name, task and methods. The name key just tells blaster what the task name is. The task key is a Python class reference. Finally the methods key is a list of methods to be run for the given task. You can then define any other key:value pairs that will be passed to the task given when an object is created for that class.

Below is an example task for building a contemporary house. You will see the task key has a value of the House class which contains all the methods defined. Blaster will create a house object (passing any extra data in this case style) and then call the methods defined.

[
    {
        'name': 'House #1',
        'task': House,
        'methods': [
            'foundation',
            'frame',
            'roof',
            'furnish',
            'enjoy'
        ],
        'style': 'contemporary'
    }
]

You also have the ability to set a timeout for each task. If the timeout is reached, the task will be exited.

[
    {
        'name': 'House #1',
        'task': House,
        'methods': [
            'foundation',
            'frame',
            'roof',
            'furnish',
            'enjoy'
        ],
        'style': 'contemporary',
        'timeout': 10
    }
]

The nice feature with blaster is you can have multiple tasks but each one can call various methods within that task class. They do not all need to call the same methods! See the example below:

[
    {
        'name': 'House #1',
        'task': House,
        'methods': [
            'foundation',
            'frame',
            'roof',
            'furnish',
            'enjoy'
        ],
        'style': 'contemporary'
    },
    {
        'name': 'House #2',
        'task': House,
        'methods': [
            'foundation',
            'frame',
            'roof'
        ],
        'style': 'cape'
    }
]

Issues

For any issues that you may find while using blaster library. Please open a new issue or you can open pull request.

blaster's People

Contributors

ryankwilliams avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar

Forkers

dannyb48

blaster's Issues

Task failed but does not return traceback

With the latest release of blaster v0.1.3. If a task method call fails, it captures the traceback and sets it. Just that data is not being returned back to the user at the end of blastoff method.

2017-08-17 10:30:34,187 INFO [blaster.blast.blastoff:47] ..............................
2017-08-17 10:30:34,188 INFO [blaster.blast.blastoff:48] START: BLASTER PREPARATION
2017-08-17 10:30:34,188 INFO [blaster.blast.blastoff:49] ..............................
2017-08-17 10:30:34,189 INFO [blaster.blast.blastoff:68] 1. TASK ~ House, METHODS: ['foundation', 'frame', 'roof', 'furnish', 'enjoy']
2017-08-17 10:30:34,190 INFO [blaster.blast.blastoff:68] 2. TASK ~ House, METHODS: ['foundation', 'frame', 'roof', 'furnish', 'post_build_tasks', 'enjoy']
2017-08-17 10:30:34,190 INFO [blaster.blast.blastoff:68] 3. TASK ~ House, METHODS: ['foundation', 'frame', 'roof', 'furnish', 'post_build_tasks', 'enjoy']
2017-08-17 10:30:34,191 INFO [blaster.blast.blastoff:68] 4. TASK ~ House, METHODS: ['foundation', 'frame', 'roof', 'furnish', 'post_build_tasks', 'enjoy']
2017-08-17 10:30:34,191 INFO [blaster.blast.blastoff:68] 5. TASK ~ House, METHODS: ['foundation', 'frame', 'roof', 'furnish', 'post_build_tasks', 'enjoy']
2017-08-17 10:30:34,191 INFO [blaster.blast.blastoff:73] ..............................
2017-08-17 10:30:34,191 INFO [blaster.blast.blastoff:74] END: BLASTER PREPARATION
2017-08-17 10:30:34,192 INFO [blaster.blast.blastoff:75] ..............................
2017-08-17 10:30:34,193 INFO [blaster.blast.blastoff:81] ..............................
2017-08-17 10:30:34,193 INFO [blaster.blast.blastoff:82] START: BLASTER BLAST OFF
2017-08-17 10:30:34,193 INFO [blaster.blast.blastoff:83] ..............................
2017-08-17 10:30:34,193 INFO [blaster.blast.blastoff:92] Total processes used by blaster is 5.
2017-08-17 10:30:34,195 INFO [blaster.foundation:21] Building the foundation for contemporary house..
2017-08-17 10:30:34,198 INFO [blaster.foundation:21] Building the foundation for colonial house..
2017-08-17 10:30:34,198 INFO [blaster.foundation:21] Building the foundation for cape house..
2017-08-17 10:30:34,198 INFO [blaster.foundation:21] Building the foundation for ranch house..
Traceback (most recent call last):
  File "/home/rywillia/github/blaster/blaster/engine/processor.py", line 49, in run
    getattr(task_obj, method)()
  File "/home/rywillia/github/blaster/examples/build_houses.py", line 23, in foundation
    raise Exception('Cannot build!')
Exception: Cannot build!
2017-08-17 10:30:34,199 INFO [blaster.foundation:21] Building the foundation for split house..
2017-08-17 10:30:35,197 INFO [blaster.frame:27] Frame the house for contemporary house..
2017-08-17 10:30:35,200 INFO [blaster.frame:27] Frame the house for colonial house..
2017-08-17 10:30:35,200 INFO [blaster.frame:27] Frame the house for cape house..
2017-08-17 10:30:35,200 INFO [blaster.frame:27] Frame the house for split house..
2017-08-17 10:30:36,198 INFO [blaster.roof:31] Roof the house for contemporary house..
2017-08-17 10:30:36,201 INFO [blaster.roof:31] Roof the house for cape house..
2017-08-17 10:30:36,201 INFO [blaster.roof:31] Roof the house for colonial house..
2017-08-17 10:30:36,201 INFO [blaster.roof:31] Roof the house for split house..
2017-08-17 10:30:37,200 INFO [blaster.furnish:35] Furnish the house for contemporary house..
2017-08-17 10:30:37,202 INFO [blaster.furnish:35] Furnish the house for cape house..
2017-08-17 10:30:37,202 INFO [blaster.furnish:35] Furnish the house for split house..
2017-08-17 10:30:37,203 INFO [blaster.furnish:35] Furnish the house for colonial house..
2017-08-17 10:30:38,201 INFO [blaster.enjoy:39] Enjoy your new contemporary house :)
2017-08-17 10:30:38,203 INFO [blaster.post_build_tasks:43] Perform post building tasks for split house..
2017-08-17 10:30:38,203 INFO [blaster.post_build_tasks:43] Perform post building tasks for cape house..
2017-08-17 10:30:38,204 INFO [blaster.post_build_tasks:43] Perform post building tasks for colonial house..
2017-08-17 10:30:39,204 INFO [blaster.enjoy:39] Enjoy your new cape house :)
2017-08-17 10:30:39,204 INFO [blaster.enjoy:39] Enjoy your new split house :)
2017-08-17 10:30:39,205 INFO [blaster.enjoy:39] Enjoy your new colonial house :)
2017-08-17 10:30:40,208 INFO [blaster.blast.blastoff:148] Total blast off duration: 0h:0m:6s
2017-08-17 10:30:40,208 INFO [blaster.blast.blastoff:150] ..............................
2017-08-17 10:30:40,209 INFO [blaster.blast.blastoff:151] END: BLASTER BLAST OFF
2017-08-17 10:30:40,209 INFO [blaster.blast.blastoff:152] ..............................
[   {   '_id': '2ba7f0dc-6cf7-4db8-8b1f-762ec351709c',
        'methods': [   'foundation',
                       'frame',
                       'roof',
                       'furnish',
                       'post_build_tasks',
                       'enjoy'],
        'name': 'House 4',
        'status': 1,
        'style': 'ranch',
        'task': <class '__main__.House'>},
    {   'methods': ['foundation', 'frame', 'roof', 'furnish', 'enjoy'],
        'name': 'House 1',
        'status': 0,
        'style': 'contemporary',
        'task': <class '__main__.House'>},
    {   '_id': '60eaa333-6577-4cbc-aac5-1bc339b159e8',
        'methods': [   'foundation',
                       'frame',
                       'roof',
                       'furnish',
                       'post_build_tasks',
                       'enjoy'],
        'name': 'House 5',
        'status': 0,
        'style': 'split',
        'task': <class '__main__.House'>},
    {   '_id': 'df061d03-0080-486e-8379-42359b2134b2',
        'methods': [   'foundation',
                       'frame',
                       'roof',
                       'furnish',
                       'post_build_tasks',
                       'enjoy'],
        'name': 'House 2',
        'status': 0,
        'style': 'cape',
        'task': <class '__main__.House'>},
    {   '_id': '0641fb02-4d22-454d-abd4-63232dca4801',
        'methods': [   'foundation',
                       'frame',
                       'roof',
                       'furnish',
                       'post_build_tasks',
                       'enjoy'],
        'name': 'House 3',
        'status': 0,
        'style': 'colonial',
        'task': <class '__main__.House'>}]

Need to determine why this data is not being returned back properly.

Expand unit tests

This issue is to expand the unit tests. Need to create more tests to gain better code coverage.

RFE: Be able to specify task timeout so that long running tasks to don't run indefinitely

Hi Ryan,

We've had a couple different teams request the capability to have Carbon tasks provide an optional parameter for timeouts so that long running tasks don't just run indefinitely.

My thinking is that this belongs in blaster since carbon is using it as a task scheduler and queue. My initial thinking is that timeout could be a part of the task definition like so

 {
            'name': 'House 1',
            'task': House,
            'methods': [
                'foundation',
                'frame',
                'roof',
                'furnish',
                'enjoy'
            ],
            'timeout': 60
            'style': 'contemporary'
        }

Wanted to see what your thoughts were and if we could collaborate on this?

Save return value from task methods

This issue is to allow blaster to capture the return data from each method called. Currently if a method returns data it is not being capture. This gives users the ability to have the return data of the method in their blaster results.

Reduce duplicated code in blastoff method

The current blaster blastoff method has duplicate code for handling actions such as: start, wait and process results when handling keyboard interrupts. This issue is to further enhance this method to call another method or function to handle these actions. This way then depending on the exception being handled. It can call the one location of code to run through. It will make the blastoff method more robust.

Reference: https://github.com/rywillia/blaster/blob/master/blaster/blast.py

Limit the number of processes per blast off

With the current release of blaster, if you give 50 tasks to be run. It will create 50 processes and then begin to run them all. This is good since you will get your results much quicker. The down side is the system blaster is running on may see a performance impact.

The feature to be added here is to allow blaster to only have X amount of processes to process a list of tasks. Blaster will be smart enough to decide how many processes it can start.

Expand documentation

This issue is to expand the existing documentation (README.md). Maybe look into creating documentation with github pages for better viewing.

Updated README

The main purpose of this issue is to add to the README.md. Currently the README.md has minimal information. The content to be added should contain the following: a detailed description of blaster, the purpose for blaster and how blaster can help running tasks concurrently to save time.

Support only Python >= 3.6

Python 2.7 will be ending support starting in 2020. This issue is to update blaster to drop Python 2.7 support. A future release of blaster (TBD) will be the final supported one with Python 2.7 support. All other releases will be only Python >= 3.6 support.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.