fnproject/fn

The container native, cloud agnostic serverless platform.

fn's Issues

Simplify CLI tool commands

In gitlab by @treeder on May 11, 2017, 09:49

Reduce down to the following:

• fn init
  • create func.yaml file
• fn run
  • build and run function
  • ensure piping into function works, eg: cat input.json | fn run
• fn deploy
  • bumps version
  • pushes to docker registry
  • updates (or creates) route with new version

FunctionHub

In gitlab by @carimura on May 22, 2017, 15:12

if anything this can be a simple way to organize examples by tag, search, etc.

• algorithmia
• stdlib

Give each hot container an ID and add hot_id to logs for everything related to that instance

In gitlab by @treeder on May 23, 2017, 13:44

To be able to trace what's going on with a specific hot container.

default README doesn't work seamlessly

In gitlab by @rdallman on May 16, 2017, 11:03

hello(master): vi person.go
hello(master) ✗: fn init rdallman/hello
no supported files found to guess runtime, please set runtime explicitly with --runtime flag
hello(master) ✗: fn init --runtime go rdallman/hello
runtime: go
func.yaml created.
hello(master) ✗: fn run
Unable to find image 'rdallman/hello:0.0.1' locally
docker: Error response from daemon: repository rdallman/hello not found: does not exist or no pull access.
See 'docker run --help'.
exit status 125

run doesn't work at all, init doesn't auto detect

deploy should say need to 'docker login' first

MVP Smart load balancer / API Gateway

In gitlab by @treeder on May 11, 2017, 11:15

In order to optimize various things such as:

• reducing image pulls
• reducing disk space for image cache
• streaming inputs to running containers (#214) - hot functions

the load balancer will need to be smart about routing requests to a specific function to a subset of machines.

IronLB

The Problem

IronFunctions requires a load balancer to route requests to IronFunctions nodes. The problem is if we just use a regular load balancer, the requests will go to all the nodes which is very suboptimal since every machine will need to store all of the image functions, and we can't take advantage of hot/streaming containers.

The Solution

If we route requests for a particular function to a subset of
machines, we get the following benefits:

• reducing image pulls
• reducing disk space for image cache
• streaming inputs to running containers, AKA hot containers (#214)

See iron-io/functions#151

We can extend an existing load balancer like Vulcand
to solve the problem. At a minimum, the load balancer(s) should be able to route function X to a fixed set of nodes (say 3 by default).

Usage

Like any other load balancer, user will start X number of IronLB nodes to route traffic to IronFunctions nodes.
The logic to route traffic to specific nodes will be baked in so there shouldn't be much more configuration than
telling the load balancers where to route traffic.

Will be delivered via a docker image.

High Level Implementation

The Docker image will start the LB and etcd.

• For each request, get app_name and path (can be obtained from URL).
• Check etcd for app_name.path, if exists, send traffic to one node from the list received. Else continue:
• Use consistent hash or similar to route request based on app_name.path to MAX_NODES (3 default) nodes
  • How to consistent hash to multiple nodes?
• Store node app_name.path -> set of IPs in etcd
• Send traffic to one of the nodes.

Try to implement via a Vulcand middleware.
Not sure if that's possible, I don't see a way to route to a specific server, will have to dig in. Otherwise,
fork Vulcand and add this feature to the "Backend" that handles the servers.

Also consider https://github.com/containous/traefik instead of Vulcan.

First Deliverable

Working MVP for use with IronFunctions.

Future Improvements - not part of initial scope

• Additional configuration may be how many nodes to route a functions traffic too, for instance, for really high load
  functions, you may want to say that a particular function can go to 10 nodes, instead of the default 3.
• If we knew stats on particular routes, we could start with putting all requests to a function on one node and
  increase as traffic increases.

Functions pipeline

In gitlab by @denismakogon on May 17, 2017, 17:31

For now we can create pipeline through webhooks (i.e. do HTTP call to another webhook from function). But the problem is that each time function called request goes through control plane (few hops expected), better to have an ability to let function to talk through internal Docker network (maybe define specific network for N functions?), that would allow to reduce networking load.

Define MVP API

In gitlab by @treeder on May 11, 2017, 09:47

Minimal set of endpoints to get to MVP, eg:

• CRUD apps - grouping of routes
• CRUD routes - route definitions containing image names, requirements, etc

Splash page for the project

In gitlab by @carimura on May 25, 2017, 11:39

Others

• https://stdlib.com/
• fission.io
• https://hyper.sh/
• https://kubernetes.io/

Test issue

In gitlab by @treeder on May 10, 2017, 15:48

Something is wrong

test, test, test

Solultion

test, text, test

Write documentation on using private images and registries

In gitlab by @treeder on May 23, 2017, 10:19

Define input/output formats

In gitlab by @treeder on May 11, 2017, 11:07

• Further define inputs and outputs (how to set response status for example)
• Put them all into the same doc, there are currently a few different docs (input/output, open function format, and writing functions). Merge those.

We can support various formats and essentially two modes: one time and streaming. One time executes a container once for each request and streaming enables "hot functions" so you can keep a container running and stream requests in.

All input is via STDIN, response output is via STDOUT and logging output is via STDERR.

We'll start with a couple of different formats:

• "default" invoke once per request, body into STDIN + headers as ENV vars
  • what env vars are set?
  • what part of the request is passed in as ENV vars?
• "http" hot function format
  • same basic set of env vars as default above
  • no per request env vars as they will be in the HTTP header

Default

Env vars

• REQUEST_URL - the full URL for the request
• ROUTE - the matched route, potentially different than the path in REQUEST_URL if dynamic route used
• METHOD - the HTTP method for the request
• HEADER_X - the HTTP headers for this request. Replace X with the upper cased name of the header and replace dashes in the header with underscores.
• X - any configuration values you've set for the Application or the Route. Replace X with the upper cased name of the config variable you set. Ex: minio_secret=secret will be exposed via MINIO_SECRET env var

Input

Stream request body directly in.

Output

Output from STDOUT is returned to client. JSON content-type.

HTTP

Env vars

• ROUTE - just the matching route? Everything else could change per request

Input

Exactly the same as the request?

Output

TODO: How to set content type? Can't do this with Lambda, but would be nice to offer this ability. Response when streaming could have a header section.

Other options

Perhaps an easier to parse format like json could be used.

Make a homebrew installer

In gitlab by @treeder on May 17, 2017, 16:27

http://docs.brew.sh/Acceptable-Formulae.html

brew install fn

Per function logging

In gitlab by @treeder on May 11, 2017, 17:18

User's need to be able to debug their functions so we need to provide logs for a function.

• fn logs in function directory to retrieve logs from the API

Serverless Framework integration

https://serverless.com/framework/docs/providers/

Simple testing tool

In gitlab by @treeder on May 11, 2017, 11:21

• fn test
• read a file of inputs/outputs
• pipe inputs and execute function, compare output with expected output

Eg:

{
    "tests": [
        {
            "input": {
                "body": "dawg"
            },
            "output": {
                "body": "21"
            }
        },
        {
            "input": {
                "body": "some random string that is bad"
            },
            "output": {
                "status": 500
            }
        }
    ]
}

Onboarding experience (list of issues)

In gitlab by @carimura on May 18, 2017, 10:04

initial

• [DONE] initial server run, clean up output (ie errors, warnings, etc. jarring to user. looks like, well, errors.
• [DONE] "where ${PWD}/data is".... makes it seem like I should change that. Not obvious I can copy/paste.

code

• [DONE] The Go program requires stdin which makes the simple program not obviously runnable from command line
• [DONE] add newline to hello world. nicer output. < / ocd >
• [DONE] be much more explicit in docs about naming requirements (func.go, func.js, etc.)
• [DONE] include explanation of stdin, stdout, passed back to screen, log to console
• [DONE] change tutorial endpoints to /hello/ruby or /hello-ruby (instead of all /hello)

fn CLI

• [DONE] "assuming go runtime" --> consolidate to a single line
• [DONE] probably split out from single code block (ie fn init in it's own block, fn run another block, etc.)
• [DONE] move the echo name:johnny | fn run up into the quickstart...?

other

• [DONE] Move everything from Usage and on to a more details readme or advanced?

Simplify functions to the core.... Convention over configuration at the fore front.

In gitlab by @hibooboo2 on May 18, 2017, 09:55

So my thoughts on the whole functions idea of dev is:

• The following should be drop dead simple:
  • Create a function
  • Testing the functions
  • Publishing the function
  • Testing the functions that have been published
  • Creating an app should be easy
    • Apps should be easy to create / deploy multiple functions to create an entire api / application
  • I should be able to create an application / app (eg group of functions)
  • You should only need to run one command to create an application and deploy it.

In order to meet all of these reqs I think that when you run fn init if there is no app / function configured in the current directory we create one.
This discovery should discover all the functions that create an application and publish it all.
That means there needs to be a sane way to do that. Right now we can init to create a functions and auto deploy to an app. However it requires us to specify the name of the app. I think that it should be similar to https://zeit.co/now#get-started in the most basic case.
To accomplish this I want to kind of expand / add to what @treeder Has said in #10

These are the things I think that fn init or deploy should do if there currently exists no function / app in the current git repo or directory depending on which it is. That being said I have two flows I would propose:

I think first it should just try it using the current directory:

• discover all the functions possible and recurse in to directories.
• Naming the app and functions.
  • currently we name the function based on the current folder / what ever the docker image is that we have provided on fn init.
  • I think that there should be a sane way to name the functions and app with no info in the command.
  • to do this I was thinking that we use the folder name as the name of the app and we can discover functions like this:
    • instead of func.go or func.java we do :
      • *.func.langextension
  • this allows us to easily auto discover functions in the same directory and figure out their runtimes easily with no input.

Here is an example:

$ tree
.
├── admin
│   └── allpolls.func.py
├── getpoll.func.go
├── getresults.func.go
├── makepoll.func.go
├── polls.func.rb
└── vote.func.java

1 directory, 6 files

That would give us 6 functions and grouped into an app called strawpoll since the folders name was strawpoll.
The thing I am not sure on is the format for app.yaml and how we would name the func.yamls. Was thinking maybe changing it to .fx.yaml and then we can put the function name in front of the .fx.yaml

And then from there you would be able to change the values in the yamls etc...

Ruby: in PreBuild, move the bundle install into the Dockerfile so Docker can cache it

In gitlab by @treeder on May 25, 2017, 15:38

Should simplify and speed up the Ruby workflow.

Test to make sure that assumption is true first. ;)

eg:

FROM funcy/ruby:dev

WORKDIR /function
ADD Gemfile* /function/
RUN bundle install

ADD . /function/

ENTRYPOINT ["ruby", "function.rb"]

When overwriting a route: prompt for yes/no or require --force

In gitlab by @carimura on May 26, 2017, 11:07

especially if overwriting with a different function name, older version, whatever. Just feels pretty destructive as is.

Clean up output on startup

In gitlab by @treeder on May 22, 2017, 16:43

Currently shows:

+ sleep 3
+ /usr/local/bin/dind.sh
+ grep -Eh -w -m1 overlay|aufs /proc/filesystems
+ cut -f2
+ fsdriver=overlay
+ [ overlay == overlay ]
+ fsdriver=overlay2
+ cmd=dockerd 		--host=unix:///var/run/docker.sock 		--host=tcp://0.0.0.0:2375 	--storage-driver=overlay2
+ eval dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 --storage-driver=overlay2
+ dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 --storage-driver=overlay2
can't create unix socket /var/run/docker.sock: device or resource busy
+ echo Docker crashed with exit code 1.  Respawning..
Docker crashed with exit code 1.  Respawning..
+ pidfile=/var/run/docker/libcontainerd/docker-containerd.pid
+ cat /var/run/docker/libcontainerd/docker-containerd.pid
cat: can't open '/var/run/docker/libcontainerd/docker-containerd.pid': No such file or directory
+ kill -9
sh: you need to specify whom to kill


+ exec ./functions

Should get rid of most of this.

Auto create the app on deploy (if !exist) on server rather than client

In gitlab by @treeder on May 26, 2017, 10:24

Write now it's in the client: https://gitlab-odx.oracle.com/odx/functions/blob/master/fn/routes.go#L368

Support function supplied by URL

In gitlab by @denismakogon on May 29, 2017, 11:49

Support function supplied by URL.

Docker supports image importing through URL, the same can be done at fn layer.

CLI:
fn routes create <app> <route> --container-url
where container-url is a storage URL where image (provided by docker CLI [2]) is stored.

[1] https://docs.docker.com/engine/reference/commandline/import/

[2] https://docs.docker.com/engine/reference/commandline/save/

Functions as middleware

In gitlab by @treeder on May 22, 2017, 16:48

This may just be examples/docs, but would be interesting to use Functions in Middleware. For instance, an auth check could call an auth function.

Functions UI

In gitlab by @treeder on May 11, 2017, 17:26

• simple open source UI (Angular?)
• to interact with most API endpoints
• can create/edit simple functions

Go wrapper

In gitlab by @treeder on May 11, 2017, 17:11

• both default and hot function support

Update all tutorials

In gitlab by @carimura on May 26, 2017, 16:00

done: ruby, node, php, go

todo: rest

Sweet Fn ASCII art (see serverless)

In gitlab by @carimura on May 24, 2017, 17:52

❯ sls create --template hello-world
Serverless: Generating boilerplate...
 _______                             __
|   _   .-----.----.--.--.-----.----|  .-----.-----.-----.
|   |___|  -__|   _|  |  |  -__|   _|  |  -__|__ --|__ --|
|____   |_____|__|  \___/|_____|__| |__|_____|_____|_____|
|   |   |             The Serverless Application Framework
|       |                           serverless.com, v1.14.0
 -------'

Serverless: Successfully generated boilerplate for template: "hello-world"
Serverless: NOTE: Please update the "service" property in serverless.yml with your service name

   ____                 _                     
  / __ \               | |                    
 | |  | |_ __ __ _  ___| | ___                
 | |  | | '__/ _` |/ __| |/ _ \               
 | |__| | | | (_| | (__| |  __/               
  \____/|_|  \__,_|\___|_|\___|               
 |  ____|              | | (_)                
 | |__ _   _ _ __   ___| |_ _  ___  _ __  ___ 
 |  __| | | | '_ \ / __| __| |/ _ \| '_ \/ __|
 | |  | |_| | | | | (__| |_| | (_) | | | \__ \
 |_|   \__,_|_| |_|\___|\__|_|\___/|_| |_|___/

   ____                  __                      
  / __ \_________ ______/ /__                    
 / / / / ___/ __ `/ ___/ / _ \                   
/ /_/ / /  / /_/ / /__/ /  __/                   
\_________ \__,_/\___/_/\____  _                 
   / ____/_  ______  _____/ /_(_)___  ____  _____
  / /_  / / / / __ \/ ___/ __/ / __ \/ __ \/ ___/
 / __/ / /_/ / / / / /__/ /_/ / /_/ / / / (__  ) 
/_/    \__,_/_/ /_/\___/\__/_/\____/_/ /_/____/  
                                                 

Make fn run pass in all the expected inputs

In gitlab by @treeder on May 23, 2017, 11:35

• env vars
• test streaming input (hot)
• ?

Function statuses

In gitlab by @treeder on May 17, 2017, 12:06

Store function status results.

• start_time
• end_time
• state (error,success,timeout)

Add new API endpoint to retrieve these results.

Make client builder tag the git repos with versions for dep manager

[removed]

Perform strict route content validation

In gitlab by @denismakogon on Jun 2, 2017, 12:29

At this moment route can be whatever user will pass through API:

fn routes create newapp /path?something=2 /path?something=2 created with denismakogon/os.environ:latest
which is not valid, so...

This ticket opened for collecting all strict validation rules.

Simple per function monitoring/metrics

In gitlab by @treeder on May 11, 2017, 17:25

• CPU
• memory

Scheduler

In gitlab by @treeder on May 12, 2017, 11:33

• Cron like scheduler to start functions
• Make generic to hit any webhook

A format like Google AppEngine could be good, human readable: https://cloud.google.com/appengine/docs/standard/python/config/cron

The scheduler processes must be coordinated to ensure if multiple are running they don't start the same job.

Hot functions

In gitlab by @treeder on May 11, 2017, 11:09

• Multiple requests per container execution
• Keep container running until idle for X seconds

Node FDK

In gitlab by @treeder on May 11, 2017, 17:11

• our common syntax
• support for lambda syntax too (probably move our current lambda support into the FDK)
• both default and hot JSON functions

For Lambda, could use mostly the same code, but create a handleLambda function to handle AWS specific syntax. Then we don't have two Node libs hanging around.

Allow user to set Content-Type in func.yaml?

In gitlab by @treeder on May 30, 2017, 06:29

Then function doesn't need to set that header, which is probably the main one that someone would need to send.

Figure out how to do dependencies better

In gitlab by @treeder on May 26, 2017, 12:00

I noticed we still have some automatic dependency installations during the build phase (python, ruby, etc) which is just WAY too slow. We removed this from the GoLangHelper already, and should do with the rest. Those need to be moved to either an fn dep command that user can run once (or every time a dependency changes), or move the dependency build into the Dockerfile so docker will cache it.

Test and update all the lambda stuff

In gitlab by @treeder on May 24, 2017, 15:32

The Node Lambda stuff was updated to follow the new workflow, but not Python or Java. Those should be converted to the new style which is essentially just this:

fn init --runtime lambda-node <DOCKER_HUB_USERNAME>/lambda-node

Then they'd follow the regular workflow from that point on.

Also, need to test and ensure the fn import command works still.

CLI: function rollback

In gitlab by @derekschultz on May 19, 2017, 12:03

As a function developer, it would help to be able to easily rollback a function to a previous version or specified version, so that I don't have to manually edit the func.yaml and redeploy.

fn rollback <function name>

Perhaps also:
fn rollback <function name> --version <x.y.z>

handle concurrency of functions across functions nodes

In gitlab by @rdallman on May 30, 2017, 14:44

right now concurrency is only respected on a per node basis. meaning if there are 3 functions nodes running, then with max_c=1 then a function could be running at actual concurrency of 3. this proved a hard problem in the past, it would be really really nice if we only had "fuzzy" concurrency as otherwise we'll need a relatively slow db to track atomic adds and even then there are issues**.

it would seem like in cases where concurrency is being restricted it would be beneficial for the load balancer to be aware of concurrency as well, since attempting to spread load should end up being futile anyway (i.e. with max_c=1 why have functions time out waiting to get a cookie to run on 5 nodes instead of just 1?). it seems like we could just draw some arbitrary lines in the sand and allow X number to be queued per node, but then we start getting into adding a lot of machinery.

lambda does not allow limiting concurrency to a user specified number. i would be delighted to adopt this policy. they only impose a per account per region limit across all functions (default 1000) [1]. there are people that want to limit concurrency, but not doing it saves a lot of hassle and it's doubtful a majority of users.

** if two processes atomic add and go over the max, then they will both need to decrement, and if they don't then things can get stuck over max concurrency and won't run -- i.e. atomic add and then maybe atomic decrement is not atomic. if you CAS then it ends up acting like a lock, but it would be atomic.

[1] http://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html

PHP LangHelper

In gitlab by @carimura on May 26, 2017, 12:00

Create "application" concept for dealing with groups of functions that make up an application/API

In gitlab by @treeder on May 11, 2017, 17:17

• An easy way to define and manage a set of functions that make up an app or API.
• An app.yaml file in the root directory to define an application
• Default could map directory paths to route paths
• can override paths and configuration in app.yaml or func.yaml's.
• fn deploy in root directory deploys all functions

Perhaps after Milestone 1...

Regenerate functions_go library in new repo and change imports to it

In gitlab by @treeder on May 18, 2017, 10:47

Instead of Iron repo.

Broken tests

In gitlab by @denismakogon on May 30, 2017, 03:44

This issue opened to track work being done for fixing tests. Test report attached.
fn_tests.report

Tasks:

• Update image to fix assertion in tests
• Fix LB tests

Versioning and Updating

In gitlab by @hibooboo2 on May 23, 2017, 13:41

Changes between the cli and server may make them incompatible at some point. Need to add a way for the server to only accept requests from a cli that is of the right version and a way for the cli to update itself to the correct version as well as update the server version for the docker image it uses.
Maybe something like:
https://github.com/inconshreveable/go-update

To do the updating and make it so that responses from the server have a header with server version and the cli will req server version / make sure it matches on requests.

setting concurrency to 1000 causes baby unicorn deaths

In gitlab by @rdallman on May 26, 2017, 16:57

it seems like the concurrency stuff is not very intelligent atm, as if given say, 1000 functions, it will spin up 1000 hot functions containers if the route is set to a concurrency of 1000 even if it could have used some of the other hot containers to run a function on. this, inevitably, basically kills docker (i had to restart my machine ;), so along with that we need some base way to limit concurrency (based on some heuristic, not necessarily just a number of tasks) on any single functions server so that we don't end up killing docker if anything but also so that functions have a relatively fair environment to run in. funny enough, if you set max c to 1 and run 1000 [of the same] functions it finishes in about 10 seconds, so it would have been faster to just shove all 1000 of them down the same tube instead of trying to open 1000 tubes ;) anyway, could be a relatively fun project.

Update README to show CLI binary from gitlab

In gitlab by @carimura on May 26, 2017, 17:29

Global string replace s/gitlab\.oracledx\.com/gitlab-odx.oracle.com/g

In gitlab by @carimura on May 29, 2017, 09:43

go package names, etc

Add `fn start` command to start the fn server

In gitlab by @treeder on May 18, 2017, 13:08

Just runs the main docker run command from the quickstart.

• fn start to start functions
• fn start ui to start functions-ui