Giter Site home page Giter Site logo

django-superbulk's Introduction

django-superbulk

Django app/view that adds the ability to execute many requests inside of a single HTTP connection. django-superbulk is compatible with tastypie, django-REST or any other django based view system.

###WHY? Transactions! Transactions! Transactions! The major reason is to be able to commit REST operations transactionally. Tastypie allows bulk operations on a single resource, but not accross multiple resources. Superbulk inherits Djano's transactional operations. All of the requests succed or they all fail (and roll back).

Performance is another issue. While not generally critical some applications can benefit from this. Such as bulk write of a large number of items, or fetching diverse data on bootstrap from networked devices with a high lag time (mobile phones?).

###Security: Django-superbulk passes on security to the views, where it is handled normally. No request can be made using superbulk that the user can't make otherwise.

###IE8: Older browsers don't support all of HTTP's new verbs (PATCH is not supported by IE8 and lower). Since requests are wrapped in a POST request, this problem can be solved by using superbulk.

##example client:

  • data is sent as an array of objects, with three fields (always).
  • method is GET, POST, PATCH, UPDATE, DELETE, or any other HTTP verb you use.
  • uri is the absolute path (not including http and domain) to your django-view.
  • body is always a string, but may contain any data. Here it is a serialized JSON object.
   data = [{
      method:'POST',
      uri:'/api/v1/customer/',
      body:JSON.stringify({
         id: 'asdf-asdf-asdf-sadf',
         name: 'Justin'
         })
      },{
      method:'POST',
      uri:'/api/v1/invoice/',
      body:JSON.stringify({
         customer_id: 'asdf-asdf-asdf-sadf',
         invoice_no: '0001'
         })
      }
   ];

   $.ajax({
      url: '/api/superbulk/',
      dataType: "application/json",
      data: JSON.stringify(data),
      type:'POST',
      contentType:'application/json',
      headers: {
         'X-CSRFToken': (document.cookie.match(/csrftoken=([0-9a-zA-Z]*)/) || ['']).pop()
      }
   });

For the failfast version (Eg: 1000000 transactions, but it will stop after the first failed one)

	data = { failfast: True,
            content: [{
               method:'POST',
               uri:'/api/v1/customer/',
               body:JSON.stringify({
                  id: 'asdf-asdf-asdf-sadf',
                  name: 'Justin'
                  })
               },{
               method:'POST',
               uri:'/api/v1/invoice/',
               body:JSON.stringify({
                  customer_id: 'asdf-asdf-asdf-sadf',
                  invoice_no: '0001'
                  })
               }
            ]
           };

	$.ajax({
	   url: '/api/superbulk_transactional/',
	   dataType: "application/json",
	   data: JSON.stringify(data),
	   type:'POST',
	   contentType:'application/json',
	   headers: {
	      'X-CSRFToken': (document.cookie.match(/csrftoken=([0-9a-zA-Z]*)/) || ['']).pop()
	   }
	});

Installing:

#TODO: Replace this with pip instructions when available in pipy
git clone [email protected]:thelonecabbage/django-superbulk.git
cd django-superbulk
python setup.py install

or another option would be to:

pip install git+https://github.com/thelonecabbage/django-superbulk.git
(Note the https form)

Add this url (or any other you prefer) to your urls.py file.

   urlpatterns += patterns('django_superbulk'
      # this is the url for the more permissive
      # handling where some transactions may fail but this
      # returns a list with the results of the execution
      url(r'^api/superbulk/$', 'superbulk'),

      # this will handle all the post data as a single transaction
      url(r'^api/superbulk_transactional/$', 'superbulk-atomic'),

   )

##return result: Errors in any of the items will result in the entire operation failing (Atomic Transactions). Successfull returns return an array of objects in the same order and length submitted.

[{
   status_code: 201,
   headers: {...},
   content: ''
},
{
   status_code: 201,
   headers: {...},
   content: ''
}]

Tests: In order to run the tests you will need nose and lettuce.

They can be run with the usual:

```
    cd superbulk_test && python manage.py harvest
```

django-superbulk's People

Contributors

adaschevici avatar mihaicc avatar thelonecabbage avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

django-superbulk's Issues

Add it to PIP

Need

For more reach, we should add the project to the PIP mirror so users can easily install a specific version or update.

Django test app improvements

Need:

Test running should be clean and accurate.

Deliverables:

Running tests with the command from the README.md doesn't work because of:

  • if you have a version of django_superbulk installed it uses that, so you're not testing what you actually changed in the repo(if you don't re-install after each change)

Solution:

  • in the urls.py of the test app we should import the views from the repo:
+# Assure that the superbulk functions are used from the current repo, not
+# the installed package
+import imp
+django_superbulk = imp.load_source('django_superbulk', '../django_superbulk.py')


 urlpatterns = patterns('',

-    url(r'^api/superbulk/', 'django_superbulk.superbulk', name='superbulk-api'),
-    url(r'^api/superbulk_transactional/', 'django_superbulk.superbulk_transactional',
+    url(r'^api/superbulk/', django_superbulk.superbulk, name='superbulk-api'),
+    url(r'^api/superbulk_transactional/', django_superbulk.superbulk_transactional,

Add superbulk_transactional view for handling the Django requests in a big transaction

https://github.com/thelonecabbage/django-superbulk/blob/master/views.py#L10

Need

As a developer
I want to be able to send multiple requests to the Django API within a database transaction
So that all requests succeed or fail atomically within a transaction
  • I'm not sure what the output should be when the transaction fails
  • our use-case is that of django-tastypie in combination with superbulk to simulate some kind of REST transactions
  • TransactionMiddleware from Django is an option, but it is not granular at all (either for all views or for none)

Deliverables

  • superbulk_transactional view where all operations is wrapped in a transaction (it should actually call the normal superbulk view)
  • each Django view call should be within a try-catch block, because some of them might fail (what is the expected response in this case?)
  • automated test for all of these situations

make proper environment initialisation for tests

In order to run tests sandboxed 
As a developer
I need to have the database created each time tests are run

Solution

Add the database creation before tests are started, and delete database after tests are run.

Add superbulk_transactional view for handling the Django requests in a big transaction

https://github.com/thelonecabbage/django-superbulk/blob/master/views.py#L10

Need

As a developer
I want to be able to send multiple requests to the Django API within a database transaction
So that all requests succeed or fail atomically within a transaction
  • I'm not sure what the output should be when the transaction fails
  • our use-case is that of django-tastypie in combination with superbulk to simulate some kind of REST transactions
  • TransactionMiddleware from Django is an option, but it is not granular at all (either for all views or for none)

Deliverables

  • superbulk_transactional view where all operations is wrapped in a transaction (it should actually call the normal superbulk view)
  • each Django view call should be within a try-catch block, because some of them might fail (what is the expected response in this case?)
  • automated test for all of these situations

Add regression test for bug #4

Need:

In order to not regress to old bugs
As a developer
I need tests to confirm recent/future changes do not open old bugs

Solution:

  • Add unit(without gherkin feature) in the django-superbulk/django_superbulk/atomic_superbulk/tests.py that calls endpoint with scenario described in #4
  • Find solution(maybe make a script) to run in one command both gherkin and unit tests
  • Update readme with new way of running tests

Refactoring:

  • Rename atomic_superbulk app into test_app (we don't need to create separate django apps for each feature we test)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.