Comments (3)
Yeah, this is definitely something that can be improved on. I'm always triggering it either from an cron task or a manual request right now which uses the handler here which should give you some pointers:
Line 106 in 38bbd5e
The idea is that the unique job name is auto-generated from the AppEngine request id. Things like the bucket (if used) and overriding the number of shards and queue to use are also extracted from the request so they can be set when kicking off jobs (e.g. using postman).
Most of that could be wrapped up into a much simpler call for triggering from code maybe something like:
InitiateJob(name string, queue string, shards int, bucket string) error
Would that make sense / be what you'd expect?
from datastore-mapper.
re. the public interfaces: I found "godoc", which is enough for me.
re. my use case: I would like to start a long running job for a particular user (nbased.com) and show the progress and whether it ends successfully. I found "MP_job", but I would need a key to access an entry there (e.g. Key(MP_job
, 'all_table.example1/55ce1022')). If the records in "MP_job" are not a part of the public API, then I would need a replacement.
re. "InitiateJob": seems OK, but it should return a handle to the running job so that I could get its status.
from datastore-mapper.
My plan (when I get some free minutes) is to add a REST API that can provide status for jobs together with a nice Polymer UI to display and control things. These should probably be mirrored / back by some convenient functions to fetch the state directly in code.
In the meantime, and because it sounds like you want it programatically anyway, you should be able to use the Lifecycle methods to add hooks for any notifications as jobs / namespaces / shards / slices start and finish. You could update whatever status entities you want from those.
See: https://github.com/CaptainCodeman/datastore-mapper/blob/master/job_spec.go#L48
from datastore-mapper.
Related Issues (20)
- Control and monitoring UI
- RESTful API
- Cloud storage output
- Use separate queue for control tasks
- Optimize output writer HOT 1
- Add examples
- Auto generate BigQuery schema file HOT 1
- Auto import into BigQuery on job completion HOT 1
- Support splitting query without scatter
- Handle splitting with ancestor keys
- Tidy cloud storage credentials
- Pass actual job values to lifecycle events
- screenshots for readme
- Store last key on shard
- Log task identification
- Task entry cleanup
- License
- Breaks with newest google.golang.org/appengine HOT 1
- Passing parameters to "Query" HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datastore-mapper.