fnproject / fn Goto Github PK
View Code? Open in Web Editor NEWThe container native, cloud agnostic serverless platform.
Home Page: http://fnproject.io
License: Apache License 2.0
The container native, cloud agnostic serverless platform.
Home Page: http://fnproject.io
License: Apache License 2.0
In gitlab by @treeder on May 23, 2017, 10:19
In gitlab by @treeder on May 12, 2017, 11:33
A format like Google AppEngine could be good, human readable: https://cloud.google.com/appengine/docs/standard/python/config/cron
The scheduler processes must be coordinated to ensure if multiple are running they don't start the same job.
In gitlab by @treeder on May 17, 2017, 12:06
Store function status results.
Add new API endpoint to retrieve these results.
In gitlab by @treeder on May 23, 2017, 11:35
In gitlab by @treeder on May 26, 2017, 12:00
I noticed we still have some automatic dependency installations during the build phase (python, ruby, etc) which is just WAY too slow. We removed this from the GoLangHelper already, and should do with the rest. Those need to be moved to either an fn dep
command that user can run once (or every time a dependency changes), or move the dependency build into the Dockerfile so docker will cache it.
In gitlab by @treeder on May 11, 2017, 09:47
Minimal set of endpoints to get to MVP, eg:
In gitlab by @denismakogon on May 17, 2017, 17:31
For now we can create pipeline through webhooks (i.e. do HTTP call to another webhook from function). But the problem is that each time function called request goes through control plane (few hops expected), better to have an ability to let function to talk through internal Docker network (maybe define specific network for N functions?), that would allow to reduce networking load.
In gitlab by @hibooboo2 on May 18, 2017, 09:55
So my thoughts on the whole functions idea of dev is:
In order to meet all of these reqs I think that when you run fn init if there is no app / function configured in the current directory we create one.
This discovery should discover all the functions that create an application and publish it all.
That means there needs to be a sane way to do that. Right now we can init to create a functions and auto deploy to an app. However it requires us to specify the name of the app. I think that it should be similar to https://zeit.co/now#get-started in the most basic case.
To accomplish this I want to kind of expand / add to what @treeder Has said in #10
These are the things I think that fn init or deploy should do if there currently exists no function / app in the current git repo or directory depending on which it is. That being said I have two flows I would propose:
I think first it should just try it using the current directory:
Here is an example:
$ tree
.
├── admin
│ └── allpolls.func.py
├── getpoll.func.go
├── getresults.func.go
├── makepoll.func.go
├── polls.func.rb
└── vote.func.java
1 directory, 6 files
That would give us 6 functions and grouped into an app called strawpoll since the folders name was strawpoll.
The thing I am not sure on is the format for app.yaml and how we would name the func.yamls. Was thinking maybe changing it to .fx.yaml and then we can put the function name in front of the .fx.yaml
And then from there you would be able to change the values in the yamls etc...
In gitlab by @treeder on May 24, 2017, 15:32
The Node Lambda stuff was updated to follow the new workflow, but not Python or Java. Those should be converted to the new style which is essentially just this:
fn init --runtime lambda-node <DOCKER_HUB_USERNAME>/lambda-node
Then they'd follow the regular workflow from that point on.
Also, need to test and ensure the fn import
command works still.
In gitlab by @carimura on May 25, 2017, 11:39
Others
In gitlab by @treeder on May 30, 2017, 06:29
Then function doesn't need to set that header, which is probably the main one that someone would need to send.
In gitlab by @carimura on May 29, 2017, 09:43
go package names, etc
In gitlab by @treeder on May 11, 2017, 17:26
In gitlab by @rdallman on May 26, 2017, 16:57
it seems like the concurrency stuff is not very intelligent atm, as if given say, 1000 functions, it will spin up 1000 hot functions containers if the route is set to a concurrency of 1000 even if it could have used some of the other hot containers to run a function on. this, inevitably, basically kills docker (i had to restart my machine ;), so along with that we need some base way to limit concurrency (based on some heuristic, not necessarily just a number of tasks) on any single functions server so that we don't end up killing docker if anything but also so that functions have a relatively fair environment to run in. funny enough, if you set max c to 1 and run 1000 [of the same] functions it finishes in about 10 seconds, so it would have been faster to just shove all 1000 of them down the same tube instead of trying to open 1000 tubes ;) anyway, could be a relatively fun project.
In gitlab by @carimura on May 26, 2017, 17:29
In gitlab by @carimura on May 24, 2017, 17:52
❯ sls create --template hello-world
Serverless: Generating boilerplate...
_______ __
| _ .-----.----.--.--.-----.----| .-----.-----.-----.
| |___| -__| _| | | -__| _| | -__|__ --|__ --|
|____ |_____|__| \___/|_____|__| |__|_____|_____|_____|
| | | The Serverless Application Framework
| | serverless.com, v1.14.0
-------'
Serverless: Successfully generated boilerplate for template: "hello-world"
Serverless: NOTE: Please update the "service" property in serverless.yml with your service name
____ _
/ __ \ | |
| | | |_ __ __ _ ___| | ___
| | | | '__/ _` |/ __| |/ _ \
| |__| | | | (_| | (__| | __/
\____/|_| \__,_|\___|_|\___|
| ____| | | (_)
| |__ _ _ _ __ ___| |_ _ ___ _ __ ___
| __| | | | '_ \ / __| __| |/ _ \| '_ \/ __|
| | | |_| | | | | (__| |_| | (_) | | | \__ \
|_| \__,_|_| |_|\___|\__|_|\___/|_| |_|___/
____ __
/ __ \_________ ______/ /__
/ / / / ___/ __ `/ ___/ / _ \
/ /_/ / / / /_/ / /__/ / __/
\_________ \__,_/\___/_/\____ _
/ ____/_ ______ _____/ /_(_)___ ____ _____
/ /_ / / / / __ \/ ___/ __/ / __ \/ __ \/ ___/
/ __/ / /_/ / / / / /__/ /_/ / /_/ / / / (__ )
/_/ \__,_/_/ /_/\___/\__/_/\____/_/ /_/____/
In gitlab by @carimura on May 26, 2017, 12:00
In gitlab by @treeder on May 11, 2017, 11:09
In gitlab by @treeder on May 11, 2017, 17:17
fn deploy
in root directory deploys all functionsPerhaps after Milestone 1...
In gitlab by @treeder on May 11, 2017, 17:11
In gitlab by @hibooboo2 on May 23, 2017, 13:41
Changes between the cli and server may make them incompatible at some point. Need to add a way for the server to only accept requests from a cli that is of the right version and a way for the cli to update itself to the correct version as well as update the server version for the docker image it uses.
Maybe something like:
https://github.com/inconshreveable/go-update
To do the updating and make it so that responses from the server have a header with server version and the cli will req server version / make sure it matches on requests.
In gitlab by @treeder on May 11, 2017, 17:18
User's need to be able to debug their functions so we need to provide logs for a function.
fn logs
in function directory to retrieve logs from the APIIn gitlab by @treeder on May 11, 2017, 11:07
We can support various formats and essentially two modes: one time and streaming. One time executes a container once for each request and streaming enables "hot functions" so you can keep a container running and stream requests in.
All input is via STDIN, response output is via STDOUT and logging output is via STDERR.
We'll start with a couple of different formats:
Stream request body directly in.
Output from STDOUT is returned to client. JSON content-type.
Exactly the same as the request?
TODO: How to set content type? Can't do this with Lambda, but would be nice to offer this ability. Response when streaming could have a header section.
Perhaps an easier to parse format like json could be used.
In gitlab by @treeder on May 17, 2017, 16:27
http://docs.brew.sh/Acceptable-Formulae.html
brew install fn
In gitlab by @treeder on May 11, 2017, 17:25
In gitlab by @carimura on May 26, 2017, 11:07
especially if overwriting with a different function name, older version, whatever. Just feels pretty destructive as is.
In gitlab by @carimura on May 18, 2017, 10:04
initial
code
fn CLI
other
In gitlab by @treeder on May 23, 2017, 13:44
To be able to trace what's going on with a specific hot container.
In gitlab by @treeder on May 10, 2017, 15:48
test, test, test
test, text, test
In gitlab by @derekschultz on May 19, 2017, 12:03
As a function developer, it would help to be able to easily rollback a function to a previous version or specified version, so that I don't have to manually edit the func.yaml and redeploy.
fn rollback <function name>
Perhaps also:
fn rollback <function name> --version <x.y.z>
In gitlab by @treeder on May 26, 2017, 10:24
Write now it's in the client: https://gitlab-odx.oracle.com/odx/functions/blob/master/fn/routes.go#L368
In gitlab by @treeder on May 22, 2017, 16:43
Currently shows:
+ sleep 3
+ /usr/local/bin/dind.sh
+ grep -Eh -w -m1 overlay|aufs /proc/filesystems
+ cut -f2
+ fsdriver=overlay
+ [ overlay == overlay ]
+ fsdriver=overlay2
+ cmd=dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 --storage-driver=overlay2
+ eval dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 --storage-driver=overlay2
+ dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 --storage-driver=overlay2
can't create unix socket /var/run/docker.sock: device or resource busy
+ echo Docker crashed with exit code 1. Respawning..
Docker crashed with exit code 1. Respawning..
+ pidfile=/var/run/docker/libcontainerd/docker-containerd.pid
+ cat /var/run/docker/libcontainerd/docker-containerd.pid
cat: can't open '/var/run/docker/libcontainerd/docker-containerd.pid': No such file or directory
+ kill -9
sh: you need to specify whom to kill
+ exec ./functions
Should get rid of most of this.
In gitlab by @denismakogon on May 29, 2017, 11:49
Support function supplied by URL.
Docker supports image importing through URL, the same can be done at fn layer.
CLI:
fn routes create <app> <route> --container-url
where container-url
is a storage URL where image (provided by docker CLI [2]) is stored.
[1] https://docs.docker.com/engine/reference/commandline/import/
[2] https://docs.docker.com/engine/reference/commandline/save/
In gitlab by @rdallman on May 30, 2017, 14:44
right now concurrency is only respected on a per node basis. meaning if there are 3 functions nodes running, then with max_c=1 then a function could be running at actual concurrency of 3. this proved a hard problem in the past, it would be really really nice if we only had "fuzzy" concurrency as otherwise we'll need a relatively slow db to track atomic adds and even then there are issues**.
it would seem like in cases where concurrency is being restricted it would be beneficial for the load balancer to be aware of concurrency as well, since attempting to spread load should end up being futile anyway (i.e. with max_c=1 why have functions time out waiting to get a cookie to run on 5 nodes instead of just 1?). it seems like we could just draw some arbitrary lines in the sand and allow X number to be queued per node, but then we start getting into adding a lot of machinery.
lambda does not allow limiting concurrency to a user specified number. i would be delighted to adopt this policy. they only impose a per account per region limit across all functions (default 1000) [1]. there are people that want to limit concurrency, but not doing it saves a lot of hassle and it's doubtful a majority of users.
** if two processes atomic add and go over the max, then they will both need to decrement, and if they don't then things can get stuck over max concurrency and won't run -- i.e. atomic add and then maybe atomic decrement is not atomic. if you CAS then it ends up acting like a lock, but it would be atomic.
[1] http://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html
In gitlab by @treeder on May 22, 2017, 16:48
This may just be examples/docs, but would be interesting to use Functions in Middleware. For instance, an auth check could call an auth function.
In gitlab by @treeder on May 11, 2017, 17:11
For Lambda, could use mostly the same code, but create a handleLambda
function to handle AWS specific syntax. Then we don't have two Node libs hanging around.
In gitlab by @treeder on May 11, 2017, 09:49
Reduce down to the following:
cat input.json | fn run
In gitlab by @treeder on May 25, 2017, 15:38
Should simplify and speed up the Ruby workflow.
Test to make sure that assumption is true first. ;)
eg:
FROM funcy/ruby:dev
WORKDIR /function
ADD Gemfile* /function/
RUN bundle install
ADD . /function/
ENTRYPOINT ["ruby", "function.rb"]
In gitlab by @denismakogon on Jun 2, 2017, 12:29
At this moment route can be whatever user will pass through API:
fn routes create newapp /path?something=2 /path?something=2 created with denismakogon/os.environ:latest
which is not valid, so...
This ticket opened for collecting all strict validation rules.
In gitlab by @treeder on May 18, 2017, 10:47
Instead of Iron repo.
In gitlab by @treeder on May 11, 2017, 11:15
In order to optimize various things such as:
the load balancer will need to be smart about routing requests to a specific function to a subset of machines.
IronFunctions requires a load balancer to route requests to IronFunctions nodes. The problem is if we just use a regular load balancer, the requests will go to all the nodes which is very suboptimal since every machine will need to store all of the image functions, and we can't take advantage of hot/streaming containers.
If we route requests for a particular function to a subset of
machines, we get the following benefits:
We can extend an existing load balancer like Vulcand
to solve the problem. At a minimum, the load balancer(s) should be able to route function X to a fixed set of nodes (say 3 by default).
Like any other load balancer, user will start X number of IronLB nodes to route traffic to IronFunctions nodes.
The logic to route traffic to specific nodes will be baked in so there shouldn't be much more configuration than
telling the load balancers where to route traffic.
Will be delivered via a docker image.
The Docker image will start the LB and etcd.
app_name
and path
(can be obtained from URL).app_name.path
to MAX_NODES (3 default) nodes
Try to implement via a Vulcand middleware.
Not sure if that's possible, I don't see a way to route to a specific server, will have to dig in. Otherwise,
fork Vulcand and add this feature to the "Backend" that handles the servers.
Also consider https://github.com/containous/traefik instead of Vulcan.
Working MVP for use with IronFunctions.
In gitlab by @treeder on May 11, 2017, 11:21
fn test
Eg:
{
"tests": [
{
"input": {
"body": "dawg"
},
"output": {
"body": "21"
}
},
{
"input": {
"body": "some random string that is bad"
},
"output": {
"status": 500
}
}
]
}
In gitlab by @treeder on May 18, 2017, 13:08
Just runs the main docker run
command from the quickstart.
fn start
to start functionsfn start ui
to start functions-uiIn gitlab by @rdallman on May 16, 2017, 11:03
hello(master): vi person.go
hello(master) ✗: fn init rdallman/hello
no supported files found to guess runtime, please set runtime explicitly with --runtime flag
hello(master) ✗: fn init --runtime go rdallman/hello
runtime: go
func.yaml created.
hello(master) ✗: fn run
Unable to find image 'rdallman/hello:0.0.1' locally
docker: Error response from daemon: repository rdallman/hello not found: does not exist or no pull access.
See 'docker run --help'.
exit status 125
run doesn't work at all, init doesn't auto detect
deploy should say need to 'docker login' first
In gitlab by @denismakogon on May 30, 2017, 03:44
This issue opened to track work being done for fixing tests. Test report attached.
fn_tests.report
Tasks:
In gitlab by @carimura on May 22, 2017, 15:12
if anything this can be a simple way to organize examples by tag, search, etc.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.