Giter Site home page Giter Site logo

iron-io / functions Goto Github PK

View Code? Open in Web Editor NEW
3.2K 106.0 229.0 6.92 MB

IronFunctions - the serverless microservices platform by

Home Page: https://iron.io

License: Apache License 2.0

Go 96.53% Shell 1.11% Makefile 0.50% Ruby 1.43% PowerShell 0.26% Dockerfile 0.18%
serverless faas docker lambda

functions's Introduction

IronFunctions

CircleCI GoDoc

Welcome to IronFunctions! The open source serverless platform.

What is IronFunctions?

IronFunctions is an open source serverless platform, or as we like to refer to it, Functions as a Service (FaaS) platform that you can run anywhere.

What is Serverless/FaaS?

Serverless is a new paradigm in computing that enables simplicity, efficiency and scalability for both developers and operators. It's important to distinguish the two, because the benefits differ:

Benefits for developers

The main benefits that most people refer to are on the developer side and they include:

  • No servers to manage (serverless) -- you just upload your code and the platform deals with the infrastructure
  • Super simple coding -- no more monoliths! Just simple little bits of code
  • Pay by the milliseconds your code is executing -- unlike a typical application that runs 24/7, and you're paying 24/7, functions only run when needed

Since you'll be running IronFunctions yourself, the paying part may not apply, but it does apply to cost savings on your infrastructure bills as you'll read below.

Benefits for operators

If you will be operating IronFunctions (the person who has to manage the servers behind the serverless), then the benefits are different, but related.

  • Extremely efficient use of resources
    • Unlike an app/API/microservice that consumes resources 24/7 whether they are in use or not, functions are time sliced across your infrastructure and only consume resources while they are actually doing something
  • Easy to manage and scale
    • Single system for code written in any language or any technology
    • Single system to monitor
    • Scaling is the same for all functions, you don't scale each app independently
    • Scaling is simply adding more IronFunctions nodes

There is a lot more reading you can do on the topic, just search for "what is serverless" and you'll find plenty of information. We have pretty thorough post on the Iron.io blog called What is Serverless Computing and Why is it Important.

Join Our Community

Join our Slack community to get help and give feedback.

Slack Status

Quickstart

This guide will get you up and running in a few minutes.

Prequisites

  • Docker 1.12 or later installed and running
  • Logged into Docker Hub (docker login)

Run IronFunctions

To get started quickly with IronFunctions, just fire up an iron/functions container:

docker run --rm -it --name functions -v ${PWD}/data:/app/data -v /var/run/docker.sock:/var/run/docker.sock -p 8080:8080 iron/functions

where ${PWD}/data is the directory where the functions application data files will be stored

This will start IronFunctions in single server mode, using an embedded database and message queue. You can find all the configuration options here. If you are on Windows, check here.

CLI tool

Install the IronFunctions CLI tool:

curl -LSs git.io/ironfn | sh

This will download a shell script and execute it. If the script asks for a password, that is because it invokes sudo.

Once installed close and re-open the terminal so the installed command fn is in your path.

on OSX with HomeBrew:

brew install iron-functions

Write a Function

Functions are small, bite sized bits of code that do one simple thing. Forget about monoliths when using functions, just focus on the task that you want the function to perform.

The following is a Go function that just returns "Hello ${NAME}!":

package main

import (
	"encoding/json"
	"fmt"
	"os"
)

type Person struct {
	Name string
}

func main() {
	p := &Person{Name: "World"}
	json.NewDecoder(os.Stdin).Decode(p)
	fmt.Printf("Hello %v!", p.Name)
}

Make a new folder and cd into that folder then copy and paste the code above into a file called func.go. From that folder run the following commands to build your function and deploy it:

# create func.yaml file, replace $USERNAME with your Docker Hub username.
fn init $USERNAME/hello
# build the function
fn build
# test it - you can pass data into it too by piping it in, eg: `cat hello.payload.json | fn run`
fn run
# Once it's ready, build and push it to Docker Hub
fn build && fn push
# create an app - you only do this once per app
fn apps create myapp
# create a route that maps /hello to your new function
fn routes create myapp /hello

Now you can call your function:

curl http://localhost:8080/r/myapp/hello

Or surf to it: http://localhost:8080/r/myapp/hello

To update your function:

# update a function with a new version and push it
fn bump && fn build && fn push
# then update the route
fn routes update myapp /hello

See below for more details. And you can find a bunch of examples in various languages in the examples directory. You can also write your functions in AWS's Lambda format.

Usage

This is a more detailed explanation of the main commands you'll use in IronFunctions as a developer.

Create an Application

An application is essentially a grouping of functions, that put together, form an API. Here's how to create an app.

fn apps create myapp

Or using a cURL:

curl -H "Content-Type: application/json" -X POST -d '{
    "app": { "name":"myapp" }
}' http://localhost:8080/v1/apps

More on apps.

Now that we have an app, we can route endpoints to functions.

Add a Route

A route is a way to define a path in your application that maps to a function. In this example, we'll map /hello to a simple Hello World! function called iron/hello which is a function we already made that you can use -- yes, you can share functions! The source code for this function is in the examples directory. You can read more about writing your own functions here.

fn routes create myapp /hello -i iron/hello

Or using cURL:

curl -H "Content-Type: application/json" -X POST -d '{
    "route": {
        "path":"/hello",
        "image":"iron/hello"
    }
}' http://localhost:8080/v1/apps/myapp/routes

More on routes.

Authentication

Iron Functions API supports two levels of Authentication in two seperate scopes, service level authentication, (Which authenticates all requests made to the server from any client) and route level authentication. Route level authentication is applied whenever a function call made to a specific route.

Please check Authentication documentation for more information.

Calling your Function

Calling your function is as simple as requesting a URL. Each app has its own namespace and each route mapped to the app. The app myapp that we created above along with the /hello route we added would be called via the following URL: http://localhost:8080/r/myapp/hello

Either surf to it in your browser or use fn:

fn call myapp /hello

Or using a cURL:

curl http://localhost:8080/r/myapp/hello

Passing data into a function

Your function will get the body of the HTTP request via STDIN, and the headers of the request will be passed in as env vars. You can test a function with the CLI tool:

echo '{"name":"Johnny"}' | fn call myapp /hello

Or using cURL:

curl -H "Content-Type: application/json" -X POST -d '{
    "name":"Johnny"
}' http://localhost:8080/r/myapp/hello

You should see it say Hello Johnny! now instead of Hello World!.

Add an asynchronous function

IronFunctions supports synchronous function calls like we just tried above, and asynchronous for background processing.

Asynchronous function calls are great for tasks that are CPU heavy or take more than a few seconds to complete. For instance, image processing, video processing, data processing, ETL, etc. Architecturally, the main difference between synchronous and asynchronous is that requests to asynchronous functions are put in a queue and executed on upon resource availability so that they do not interfere with the fast synchronous responses required for an API. Also, since it uses a message queue, you can queue up millions of function calls without worrying about capacity as requests will just be queued up and run at some point in the future.

To add an asynchronous function, create another route with the "type":"async", for example:

curl -H "Content-Type: application/json" -X POST -d '{
    "route": {
        "type": "async",
        "path":"/hello-async",
        "image":"iron/hello"
    }
}' http://localhost:8080/v1/apps/myapp/routes

Now if you request this route:

curl -H "Content-Type: application/json" -X POST -d '{
    "name":"Johnny"
}' http://localhost:8080/r/myapp/hello-async

You will get a call_id in the response:

{"call_id":"572415fd-e26e-542b-846f-f1f5870034f2"}

If you watch the logs, you will see the function actually runs in the background:

async log

Read more on logging.

Functions UI

docker run --rm -it --link functions:api -p 4000:4000 -e "API_URL=http://api:8080" iron/functions-ui

For more information, see: https://github.com/iron-io/functions-ui

Writing Functions

See Writing Functions.

And you can find a bunch of examples in the /examples directory.

More Documentation

See docs/ for full documentation.

Roadmap

These are the high level roadmap goals. See milestones for detailed issues.

  • Alpha 1 - November 2016
    • Initial release of base framework
    • Lambda support
  • Alpha 2 - December 2016
    • Streaming input for hot functions #214
    • Logging endpoint(s) for per function debugging #263
  • Beta 1 - January 2017
    • Smart Load Balancer #151
  • Beta 2 - February 2017
    • Cron like scheduler #100
  • GA - March 2017

Support

You can get community support via:

You can get commercial support by contacting Iron.io

Want to contribute to IronFunctions?

See contributing.

functions's People

Contributors

adelevie avatar alnutile avatar auyer avatar bupychuk avatar c0ze avatar ccirello avatar denismakogon avatar derekschultz avatar edsrzf avatar ekalinin avatar gouthamve avatar guilhermebr avatar henriquechehad avatar jconning avatar jmank88 avatar kunihiko-t avatar lingxiankong avatar martinpinto avatar merqlove avatar michaelkitson avatar nii236 avatar nikhilm avatar noqcks avatar pedronasser avatar seiflotfy avatar sks avatar thousandsofthem avatar treeder avatar ucirello avatar wjimenez5271 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

functions's Issues

Make "function tool" in ironcli

Say you make a directory for each function, the tool will go through each directory and check for changes, if changed, bump a version number (use the same type of script we use in release.sh? or check for a VERSION file and bump that), then run docker build, tag it with version, then docker push, then update route in IronFunctions.

Should think about some way to help with testing too, like build, then run tests against the container?

See: https://github.com/iron-io/functions/tree/master/tool

and/or build this into serverless framework?

New Postgres bugs

There was this one: 662216f

And now I get this one after that fix:

RRO[0086] Could not create app                          action=server.handleAppCreate error=pq: column "config" of relation "apps" does not exist

@pedronasser looks like you may have broken this with your config changes. Can you please fix, thanks.

Support for params in route paths

eg: /r/myapp/somethings/:something_id/comments/:comment_id => PARAM_SOMETHING_ID=X and PARAM_COMMENT_ID=Y

Also add METHOD=GET/POST env vars.

re: #56

[WIP] CRON like scheduling / scheduler

Triggers functions on a repeating schedule.

Use new input format discussed recently and probably CRON format as well.


Proposal for cron support

Basic architecture sketch for each IronFunctions node:

                 ┌────────┐
     ┌──────────▶│D. Lock │
     │           └────────┘
     │
┌────────┐       ┌────────┐
│  Cron  │──────▶│   DB   │◀─────────────┐
└────────┘       └────────┘              │
     │                                   │
     │           ┌────────┐         ┌────────┐
     └──────────▶│   MQ   │◀────────│  HTTP  │
                 └────────┘         └────────┘
                      ▲                  │
                      │                  │
                      │                  ▼
                 ┌────────┐         ┌────────┐
                 │ Async  │         │  Sync  │
                 └────────┘         └────────┘

The two additional components are:

D. Lock is a some sort of distributed lock offered by latest versions of etcd
and Consul. In theory, any distributed lock would be enough in order to achieve
the necessary role in this design.

Cron is a gear similar to async. It enqueues a cron job while holding a
distributed mutex lock. Its implementation would be similar to the following:

lock()
jobs := cronjobs()
for _, job := range jobs {
	ls := laststart(j)
	if job.shouldRunAfter(ls) {
		ls.touch()
		enqueue(j)
	}
}
unlock()

In the DB, the new datastructure would be:

[{
	"id": "...",
	"function": {
		"app": "app",
		"path": "/fn",
		"headers": ["Context-Type: application/json"],
		"body": "....",
	}
	"frequency": "*/10 * * * *"
}, ...]

id is a unique identifier for the cronjob. It aims to allow repeated entries
for a same given function.

function is the description of the function to be ran. It will mimic a HTTP
POST call with headers and body. It also adds a User-Agent with the
specific information that this call is made by the cron worker.

frequency is a crond-like string that specifies when should this cronjob be
ran.

Restricting the distributed lock to etcd/consul and alikes, it means they can
also be used as an authoritative key-value source of truth regarding the last
execution.

laststart in the example code above, is the last time this cronjob was started,
regardless of success or failure.

Get rid of these errors in the logs or deal with them better so they aren't errors

ERRO[0456] error streaming docker stats for task         container=task-a790dc09-7149-58eb-b8ef-bdc1571ad2c5 error=io: read/write on closed pipe task_id=a790dc09-7149-58eb-b8ef-bdc1571ad2c5
ERRO[0459] Handler for POST /containers/task-a790dc09-7149-58eb-b8ef-bdc1571ad2c5/stop returned error: Container task-a790dc09-7149-58eb-b8ef-bdc1571ad2c5 is already stopped

The function appears to execute and respond just fine, but these errors show up every time.

Test suite to test API

Builds the image, starts the API, then:

  • Test all API endpoints
  • Tests some routes, such as hello world one and checks to see that response is valid (headers and body)
  • Test an error route (iron/error image is a container that just errors out, good for testing). Ensure error status code.

Emit Metrics to Logs

Using log format defined here: https://github.com/iron-io/logspout-statsd

Eg:

metric=someevent value=1 type=count
metric=somegauge value=50 type=gauge

Emit the following metrics (at least):

  • wait time total (all apps)
  • wait time per function
  • execution time total (all apps)
  • execution time per function
  • function request count
  • function error count
  • cpu/memory per function ?

Dynamic routes: Can we optimize the route matching rather than pulling up all routes for an app?

This is tricky because of this: #71

ie: https://github.com/iron-io/functions/pull/74/files#diff-e180f07f5c320b82004c2bb065eedef0R99

The problem is we'd like to query for a single route, but let's say the route is defined as:

/blogs/:blog_id/comments/:comment_id

And a request comes in for:

/blogs/123/comments/456

How do we query the database for the route metadata with only that information? Could query for /blogs/123/comments/456, which is the static default way. But that wouldn't match. We could try all variations, eg /blogs/123/comments/* and /blogs/123/*/456, and /blogs/123/*/*, and so on, but that wouldn't be very efficient.

To summarize, the problem here is:

Given a path and only a path, eg: /blogs/travisblog/comments/123 what should you query for? Not knowing what routes exist before hand.

Async functions

First PR #97

This issue is for discussion and feedback on the functionality.

Full App/Microservice example

An example that builds a full API with multiple endpoints, authentication, etc.

Take a blog example or something and build it using IronFunctions.

  • GET /posts returns list of posts
  • POST /posts adds a new post and stores it in a database (a database that requires auth like a postgres database hosted on heroku or something). Requires authentication (JWT token in header).
  • GET /posts/:id returns a full post.

Docs and example of autoscaling

Collect some metrics (function wait time probably good?) from logs then launch new server(s) based on that. Use something like Kubernetes or operator to keep it simple. Needs to be added to load balancer automatically too.

For STDERR logging, use this format

Take STDERR from runner containers and add app_name, path, function, request_id and put the actual log line in msg. If using logrus, it will automatically use msg.

Long(er) running containers for better performance aka Hot Containers

Instead of one off execution, keep container running for a while and take requests.

The first version executes a container per function call/request. Which is fine for many things, but we may want a way to run for longer periods of time to increase performance due to things like keeping db connections open, etc.

This would probably require a more lambda like code format and language libraries. See https://github.com/iron-io/lambda for existing work on lambda style code format.

I created a prototype of this years ago that routed traffic into IronWorker containers, should dig that out and look into that implementation.

Response on create app should use app wrapper

Currently returns:

✗ curl -H "Content-Type: application/json" -X POST -d '{
quote>     "app": { "name":"myapp5" }
quote> }' http://functions.iron.io/v1/apps
{"name":"myapp5"}

Return the "app" wrapper, same as the input. And also add a "message" or "msg" that says "App created successfully".

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.