oschmid/cachestore

Drop-in replacement of appengine/datastore that automatically caches objects using memcache

cachestore's Issues

Speed Up Memcache Gob Encoding/Decoding

Benchmark the difference between memcaching a gob of a struct vs the current memcaching a gob of an array of the struct's properties. If the speed benefit of encoding/decoding the whole vs its parts is significant it may be worth it to switch. The only down side is that it deviates slightly from datastore's behaviour (especially for objects implementing PropertyLoadSaver).

Cache Policies

Once #1 is implemented, make cachestore more flexible by registering policies for different types.

Policies are function that decide:

• whether a struct should be cached in context
• whether a struct should be cached in memcache
• a struct's memcache expiration time

Default policy:

• cache in context
• cache in memcache
• max memcache expiration time

Gob type registration might also be handled here, allowing cachestore to abstract away the fact that gob is used for caching structs to memcache.

This is also modelled after a feature of NDB

Get Difference

On a memcache miss, GetMulti currently gets all items from the datastore. Benchmark the difference between this behaviour and getting only the entities that were missing from memcache from the datastore.

panic: reflect: call of reflect.Value.Type on zero Value

I'm getting this error and I'm not sure what's causing it or how to debug it.

panic: reflect: call of reflect.Value.Type on zero Value

I'm getting it when doing a Get or Put with a the entity type Service:

type Service struct {
    Date      time.Time
    Patient   int
    Dentist   *datastore.Key
    Procedure *datastore.Key
    Insurance *datastore.Key
    Created   time.Time
    Updated   time.Time
    Key       *datastore.Key
    Version   int
}

And, this is the code that's failing with cachestore. They both work fine with datastore.Get/Put, but fail with cachestore.

GETTING:

var (
    c     = appengine.NewContext(r)
    query = datastore.NewQuery("Service").Order("-Date").KeysOnly()
    keys  []*datastore.Key
)

if k != nil {
    s := fmt.Sprintf("%s =", k.Kind())
    query = query.Filter(s, k)
}

keys, err := query.GetAll(c, nil)
if err != nil {
    Log(r, 5, "There was an error querying the datastore for %v: %v", query, err)
    return nil
}

for _, k := range keys {
    x := new(Service)
    if err := cachestore.Get(c, k, x); err != nil {
        Log(r, 2, "Couldn't find service: %v", err)
    } else {
        Log(r, 1, "Service found: %v", x)

        if x.Key == nil {
            x.Key = k
        }
        services = append(services, *x)
    }
}
Log(r, 1, "Found %v services: %v", len(services), services)

PUTTING:

var (
    c    = appengine.NewContext(r)
    serv = new(Service)
    err  error
)

date := r.FormValue("date")
if serv.Date, err = validatedate(date); err != nil {
    Log(r, 1, "Date '%s' is not valid: %v", date, err)
    return
}
Log(r, 1, "String is valid: %v", date)

patient := r.FormValue("patient")
if serv.Patient, err = validatepatientid(patient); err != nil {
    Log(r, 1, "Patient ID '%s' is not valid: %v", patient, err)
    return
}
Log(r, 1, "Patient ID is valid: %v", patient)

dentist := r.FormValue("dentist")
if serv.Dentist, err = validatedentistid(c, dentist); err != nil {
    Log(r, 1, "Dentist ID '%s' is not valid: %v", dentist, err)
    return
}
Log(r, 1, "Dentist ID is valid: %v", dentist)

procedure := r.FormValue("procedure")
if serv.Procedure, err = validateprocedureid(c, procedure); err != nil {
    Log(r, 1, "Procedure ID '%s' is not valid: %v", procedure, err)
    return
}
Log(r, 1, "Procedure ID is valid: %v", procedure)

insurance := r.FormValue("insurance")
if serv.Insurance, err = validateinsuranceid(c, insurance); err != nil {
    Log(r, 1, "Insurance ID '%s' is not valid: %v", insurance, err)
    return
}
Log(r, 1, "Insurance ID is valid: %v", insurance)

k := datastore.NewIncompleteKey(c, "Service", nil)

serv.Created = time.Now()
serv.Updated = time.Now()
serv.Version = API_VERSION

if serv.Key, err = cachestore.Put(c, k, serv); err != nil {
    Log(r, 5, "An error occurred while adding the %v procedure to the datastore: %v", k, err)
    return
}
Log(r, 1, "Placed service in the datastore at %v: %v", serv.Key, serv)

Work Around 500 Entity PutMulti Limit

There's a 500 entity limit on PutMulti which can be worked around by breaking large Puts into multiple smaller Puts. Inspired by goon.

In-Context Caching

Create a third level of caching that uses a Go map to store structs. It would be faster even than memcache but would only exist for a single request. See NDB