iron-io / functions Goto Github PK
View Code? Open in Web Editor NEWIronFunctions - the serverless microservices platform by
Home Page: https://iron.io
License: Apache License 2.0
IronFunctions - the serverless microservices platform by
Home Page: https://iron.io
License: Apache License 2.0
Using log format defined here: https://github.com/iron-io/logspout-statsd
Eg:
metric=someevent value=1 type=count
metric=somegauge value=50 type=gauge
Emit the following metrics (at least):
Aug 30 00:24:54 0850e63e776b functions: time="2016-08-30T07:24:54Z" level=error msg="Could not list routes from datastore" action=server.handleSpecial
request_id=8db8390e-9442-5f7d-b1d4-bd25ea9f68b6
Implement in worker too: https://github.com/iron-io/worker/issues/339
Required to prevent conflicts of creating two containers having the same name
Bolt and Postgres would be good. Bolt for quick start, postgres for production and ability to scale.
I see this before every execution: INFO[1021] Hitting registry to check image access permission.
and it takes way too long.
eg: http://chadsapp1.123.ironfunctions.com/hi?name=poop
How do we pass these into the container though? This would be a GET with no body. Perhaps we can use env vars for query params and headers?
Say you make a directory for each function, the tool will go through each directory and check for changes, if changed, bump a version number (use the same type of script we use in release.sh? or check for a VERSION file and bump that), then run docker build, tag it with version, then docker push, then update route in IronFunctions.
Should think about some way to help with testing too, like build, then run tests against the container?
See: https://github.com/iron-io/functions/tree/master/tool
and/or build this into serverless framework?
Probably building off this: #13
It will build the function image, then run test(s) against it.
First PR #97
This issue is for discussion and feedback on the functionality.
re: #71
Pros:
Cons:
Note: can't be some json structure containing headers and payload, etc. since we can't stream that.
This is tricky because of this: #71
ie: https://github.com/iron-io/functions/pull/74/files#diff-e180f07f5c320b82004c2bb065eedef0R99
The problem is we'd like to query for a single route, but let's say the route is defined as:
/blogs/:blog_id/comments/:comment_id
And a request comes in for:
/blogs/123/comments/456
How do we query the database for the route metadata with only that information? Could query for /blogs/123/comments/456
, which is the static default way. But that wouldn't match. We could try all variations, eg /blogs/123/comments/*
and /blogs/123/*/456
, and /blogs/123/*/*
, and so on, but that wouldn't be very efficient.
To summarize, the problem here is:
Given a path and only a path, eg: /blogs/travisblog/comments/123
what should you query for? Not knowing what routes exist before hand.
Share memory checking functions from Titan.
If queue of requests get too big (100?), start returning 503's.
ALWAYS log the wait time, even if it's zero.
Add current API endpoints + what we need:
Collect some metrics (function wait time probably good?) from logs then launch new server(s) based on that. Use something like Kubernetes or operator to keep it simple. Needs to be added to load balancer automatically too.
Currently returns:
✗ curl -H "Content-Type: application/json" -X POST -d '{
quote> "app": { "name":"myapp5" }
quote> }' http://functions.iron.io/v1/apps
{"name":"myapp5"}
Return the "app" wrapper, same as the input. And also add a "message" or "msg" that says "App created successfully".
eg: /r/myapp/somethings/:something_id/comments/:comment_id
=> PARAM_SOMETHING_ID=X
and PARAM_COMMENT_ID=Y
Also add METHOD=GET/POST env vars.
re: #56
Triggers functions on a repeating schedule.
Use new input format discussed recently and probably CRON format as well.
Basic architecture sketch for each IronFunctions node:
┌────────┐
┌──────────▶│D. Lock │
│ └────────┘
│
┌────────┐ ┌────────┐
│ Cron │──────▶│ DB │◀─────────────┐
└────────┘ └────────┘ │
│ │
│ ┌────────┐ ┌────────┐
└──────────▶│ MQ │◀────────│ HTTP │
└────────┘ └────────┘
▲ │
│ │
│ ▼
┌────────┐ ┌────────┐
│ Async │ │ Sync │
└────────┘ └────────┘
The two additional components are:
D. Lock
is a some sort of distributed lock offered by latest versions of etcd
and Consul. In theory, any distributed lock would be enough in order to achieve
the necessary role in this design.
Cron
is a gear similar to async. It enqueues a cron job while holding a
distributed mutex lock. Its implementation would be similar to the following:
lock()
jobs := cronjobs()
for _, job := range jobs {
ls := laststart(j)
if job.shouldRunAfter(ls) {
ls.touch()
enqueue(j)
}
}
unlock()
In the DB, the new datastructure would be:
[{
"id": "...",
"function": {
"app": "app",
"path": "/fn",
"headers": ["Context-Type: application/json"],
"body": "....",
}
"frequency": "*/10 * * * *"
}, ...]
id
is a unique identifier for the cronjob. It aims to allow repeated entries
for a same given function.
function
is the description of the function to be ran. It will mimic a HTTP
POST
call with headers
and body
. It also adds a User-Agent
with the
specific information that this call is made by the cron
worker.
frequency
is a crond-like string that specifies when should this cronjob be
ran.
Restricting the distributed lock to etcd/consul and alikes, it means they can
also be used as an authoritative key-value source of truth regarding the last
execution.
laststart
in the example code above, is the last time this cronjob was started,
regardless of success or failure.
eg:
curl -H "Content-Type: application/json" -X POST -d '{
"path":"/guppy",
"image":"treeder/guppy:0.0.2",
"config": {"DB_URL": "abc"},
"response_headers": {"Content-Type": "application/json"}
}' http://localhost:8080/v1/apps/myapp/routes
Reuse Titan code.
So code can access query params .
Could use the obvious like nginx or HAProxy, but it might be nice to use and contribute to a Go one. Maybe Traefik, Vulcand or Caddy.
Builds the image, starts the API, then:
Instead of one off execution, keep container running for a while and take requests.
The first version executes a container per function call/request. Which is fine for many things, but we may want a way to run for longer periods of time to increase performance due to things like keeping db connections open, etc.
This would probably require a more lambda like code format and language libraries. See https://github.com/iron-io/lambda for existing work on lambda style code format.
I created a prototype of this years ago that routed traffic into IronWorker containers, should dig that out and look into that implementation.
Should apply to both IronWorker and Functions.
See IronWorker issue: https://github.com/iron-io/titan/issues/297
There was this one: 662216f
And now I get this one after that fix:
RRO[0086] Could not create app action=server.handleAppCreate error=pq: column "config" of relation "apps" does not exist
@pedronasser looks like you may have broken this with your config changes. Can you please fix, thanks.
An example that builds a full API with multiple endpoints, authentication, etc.
Take a blog example or something and build it using IronFunctions.
ERRO[0456] error streaming docker stats for task container=task-a790dc09-7149-58eb-b8ef-bdc1571ad2c5 error=io: read/write on closed pipe task_id=a790dc09-7149-58eb-b8ef-bdc1571ad2c5
ERRO[0459] Handler for POST /containers/task-a790dc09-7149-58eb-b8ef-bdc1571ad2c5/stop returned error: Container task-a790dc09-7149-58eb-b8ef-bdc1571ad2c5 is already stopped
The function appears to execute and respond just fine, but these errors show up every time.
Similar to Titan.
in README and README.io docs. See: #71
Take STDERR from runner containers and add app_name
, path
, function
, request_id
and put the actual log line in msg
. If using logrus, it will automatically use msg.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.