captaincodeman/datastore-locker

A lock / lease mechanism to prevent duplicate execution of AppEngine tasks

Datastore Locker

Provides a lock / lease mechanism to prevent duplicate execution of appengine tasks and an easy way to continue long-running processes. Used by datastore-mapper.

Sometimes it's extremely difficult if not impossible to make tasks truly idempotent. e.g. if your task sends an email or charges a credit card then executing it twice could be 'a bad thing'. Actually preventing a task from running more than once can be challenging though ...

Although named tasks can be used to prevent duplicate tasks being enqueued they cannot be used together with datastore transactions which leaves you with the possibility that either the task scheduling or the datastore write could fail.

An unnamed task can be enqueued within a transaction to ensure that both happen or neither happen but that then leaves the possibility of that operation being repeated (if running in it's own task) which could cause duplicate tasks.

Whichever approach is used, the taskqueue only promises at-least-once delivery so there is also the chance that appengine will execute a task more than once.

The solution is to coordinate execution using information in the task with information in a datastore entity. This package aims to make it easy to restrict task execution by using a lease / lock mechanism.

By obtaining a lock on an entity within a datastore transaction we ensure that only a single instance of any task will be executed at once and, once processed, that duplicate execution will be prevented.

It can be used with single tasks or to chain a series of tasks in sequence with the sequence number used to prevent any old tasks being re-executed.

An exceptional situation can occur if a failure happens during processing of a task and the result cannot be communicated back to appengine (this is a platform issue). In this case the lock / lease is already held but the system cannot determine if the task completed or maybe it just failed to clear the lock. The locker will allow a timeout before querying the appengine logs to determine the task status. In the case of a complete failure with no log information, a timeout will prevent deadlock by overwriting the expired lock / lease.

Both overwritten locks and permanently failing tasks (past a configurable number of retries) can be alerted by email as needing further investigation.

Usage

See the example project for a simple demonstration of locker being used.

Embed the locker.Lock field within the struct you want lock on.

Foo struct {
    locker.Struct
    Value string `datastore:"value"`  
}

Create an instance of the locker and configure as required:

l := locker.NewLocker()

l := locker.NewLocker(
  locker.LeaseDuration(time.Duration(5)*time.Minute),
  locker.LeaseTimeout(time.Duration(15)*time.Minute),
  locker.AlertOnOverwrite,
)

l := locker.NewLocker(locker.LogVerbose)

Schedule a task to be executed once:

key := datastore.NewKey(c, "foo", "", 1, nil)
entity := &Foo{Value:"bar"}
err := l.Schedule(c, key, entity, "/task/handler/url", nil)
if err != nil {
  // operation failed (entity not saved and task not enqueued)
}

Handle the task execution:

func init() {
  http.Handle("/task/handler/url", locker.Handle(fooHandler, fooFactory)
}

// the task handler needs a factory to construct an instance of our entity
func fooFactory() interface{} {
  return new(Foo)
}

// the handler for the task will be passed the appengine context, request, datastore key and entity
func foohandler(c context.Context, r *http.Request, key *datastore.Key, entity locker.Lockable) error {
  foo := entity.(*Foo)

  switch foo.Sequence {
    case 1:
      // step 1 processing, e.g. charge credit card
      // schedule another task to follow this one:
      return l.Schedule(c, key, entity, "/task/handler/url", nil)
    case 2:
      // step 2 processing, e.g. send confirmation email
      // mark the task as completed (to prevent the last task re-executing)
      return l.Complete(c, key, entity)
  }

  // returning an error would cause the task to be failed and retried (normal task semantics)
  // a configurable number of retries can be set to prevent endless attempts from happening
  return nil
}