Giter Site home page Giter Site logo

gofn / gofn Goto Github PK

View Code? Open in Web Editor NEW
138.0 14.0 14.0 3.67 MB

Function process via docker provider (serverless minimalist)

License: MIT License

Go 97.76% Shell 0.48% Python 0.68% Makefile 0.95% Dockerfile 0.12%
docker faas functions lambda prometheus serverless swarm service functions-as-a-service

gofn's Introduction

gofn

Build Status GoDoc Go Report Card codecov Join the chat at https://gitter.im/gofn/gofn Open Source Helpers

Function process via docker provider

Install

go get github.com/gofn/gofn

Example

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"os"
	"runtime"
	"sync"

	"github.com/gofn/gofn"
	"github.com/gofn/gofn/iaas/digitalocean"
	"github.com/gofn/gofn/provision"
)

func main() {
	wait := &sync.WaitGroup{}
	contextDir := flag.String("contextDir", "./", "a string")
	dockerfile := flag.String("dockerfile", "Dockerfile", "a string")
	imageName := flag.String("imageName", "", "a string")
	remoteBuildURI := flag.String("remoteBuildURI", "", "a string")
	volumeSource := flag.String("volumeSource", "", "a string")
	volumeDestination := flag.String("volumeDestination", "", "a string")
	remoteBuild := flag.Bool("remoteBuild", false, "true or false")
	input := flag.String("input", "", "a string")
	flag.Parse()
	parallels := runtime.GOMAXPROCS(-1) // use max allowed CPUs to parallelize
	wait.Add(parallels)
	for i := 0; i < parallels; i++ {
		go run(*contextDir, *dockerfile, *imageName, *remoteBuildURI, *volumeSource, *volumeDestination, wait, *remoteBuild, *input)
	}
	wait.Wait()
}

func run(contextDir, dockerfile, imageName, remoteBuildURI, volumeSource, volumeDestination string, wait *sync.WaitGroup, remote bool, input string) {
	buildOpts := &provision.BuildOptions{
		ContextDir: contextDir,
		Dockerfile: dockerfile,
		ImageName:  imageName,
		RemoteURI:  remoteBuildURI,
		StdIN:      input,
	}
	containerOpts := &provision.ContainerOptions{}
	if volumeSource != "" {
		if volumeDestination == "" {
			volumeDestination = volumeSource
		}
		containerOpts.Volumes = []string{fmt.Sprintf("%s:%s", volumeSource, volumeDestination)}
	}
	if remote {
		key := os.Getenv("DIGITALOCEAN_API_KEY")
		if key == "" {
			log.Fatalln("You must provide an api key for digital ocean")
		}
		do, err := digitalocean.New(key)
		if err != nil {
			log.Println(err)
		}
		buildOpts.Iaas = do
	}

	defer wait.Done()
	stdout, stderr, err := gofn.Run(context.Background(), buildOpts, containerOpts)
	if err != nil {
		log.Println(err)
	}
	fmt.Println("Stderr: ", stderr)
	fmt.Println("Stdout: ", stdout)
}

Run Example

cd examples

go run main.go -contextDir=testDocker -imageName=python -dockerfile=Dockerfile

# or using volume
go run main.go -contextDir=testDocker -imageName=python -dockerfile=Dockerfile -volumeSource=/tmp -volumeDestination=/tmp

# or using remote Dockerfile
go run main.go -remoteBuildURI=https://github.com/gofn/dockerfile-python-example.git -imageName="pythonexample"

# you can also send a string that will be written to the stdin of the container
go run main.go -contextDir=testDocker -imageName=python -dockerfile=Dockerfile -input "input string"

# or run in digital ocean
export DIGITALOCEAN_API_KEY="paste your key here"
go run main.go -contextDir=testDocker -imageName=python -dockerfile=Dockerfile -remoteBuild=true

You can also compile with go build or build and install with go install command then run it as a native executable.

Example Parameters

  • -contextDir is the root directory where the Dockerfile, scripts, and other container dependencies are, by default current directory "./".

  • -imageName is the name of the image you want to start, if it does not exist it will be automatically generated and if it exists the system will just start the container.

  • -dockerFile is the name of the file containing the container settings, by default Dockerfile

  • -volumeSource is the directory that will be mounted as a data volume. By default is empty string indicating his not used.

  • -volumeDestination is the path mounted inside the container. By default is empty string indicating his not used but if only omitted, volumeSource is used.

  • -remoteBuildURI is remote URI containing the Dockerfile to build.By default is empty. More details on docker api docs

  • remoteBuild is a boolean that indicates if have to run localally or in a machine in digital ocean Don't forget to export your api key.

  • -input is a string that will be written to the stdin of the container

  • -h Shows the list of parameters

gofn generates the images with "gofn/" as a prefix.

gofn's People

Contributors

avelino avatar cassiobotaro avatar codetriage-readme-bot avatar crgimenes avatar dakerfp avatar felipeweb avatar gitter-badger avatar gtsalles avatar marioidival avatar pablomelo avatar raninho avatar rogeriozambon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gofn's Issues

Run gofn inside a container

I have an executable Go running inside a container and I want to use gofn to command the host's docker.
How can I do this?

Save snapshot image

As a user,
I want to save a snapshot created by gofn,
So I can re-utilize it when I need to run the same stuff

Write in process stdin

I need to send some information to the container.
I do not want to map a volume, instead I want to send the information by typing in the stdin of the process.

Here is how I believe this feature should work:

    gofn.Input = "some text"
    stdout, err := gofn.Run(...)

Internally gofn would pass the contents of the string to the stdin of the container.

At the end of the string it may be interesting to send an EOF character to signal that the string as ended and the process can start.

When execution failed, logs are not captured

How to reproduce?

Files

# Dockerfile
FROM python:3.6-alpine

ENTRYPOINT ["python"]
CMD ["-c", "print(a)"]
//main.go
package main

import (
	"fmt"
	"log"
	"os"

	"github.com/nuveo/gofn"
	"github.com/nuveo/gofn/provision"
)

func main() {
	buildOpts := &provision.BuildOptions{
		ImageName:               "docker_test",
		DoNotUsePrefixImageName: true,
	}
	stdout, stderr, err := gofn.Run(buildOpts, nil)
	if err != nil {
                fmt.Println("Stderr: ", stderr)
	        fmt.Println("Stdout: ", stdout)
		log.Println(err)
		os.Exit(1)
	}
	fmt.Println("Stderr: ", stderr)
	fmt.Println("Stdout: ", stdout)
	os.Exit(0)
}

steps

Build the image docker build -t docker_test

Run go run main.go

expected behavior

some output in stdout or error

actual behavior

stdout and stderr are empty

error trying to kill container

error trying to kill a process that is not running:

2017/10/13 18:40:59 [error] gofn.go:138 error trying to kill container API error (409): {"message":"Cannot kill container: 4871e7a17a70cb7b90ccfdb6483e31a760d96c155e8e578a7d0f64f852d5fff1: Container 4871e7a17a70cb7b90ccfdb6483e31a760d96c155e8e578a7d0f64f852d5fff1 is not running"}

Annoying log

Avoid logging in with standard output without prompting.
image

Needs to cancel task

As a user,
I want to be able to cancel a running task,
Because sometimes I make mistakes.

List Processes

As the Admin,
I need to see a list of the containers that were started by gofn,
So I can have an overview of the state of my services.

trying to destroy container context canceled

{"message":"You cannot remove a running container db42d75d9ffa43ab9b2214c9a568942e55444720678a51e1c44f126d42f7d54e. Stop the containe 
 before attempting removal or force remove"}

docker client is (*docker.Client)(nil)
docker container is (*docker.Container)(nil)

docker client is &docker.Client{SkipServerVersionCheck:true, HTTPClient:(*http.Client)(0xc42026b620), TLSConfig:(*tls.Config)(nil), Dialer:(*net.Dialer)(0xc42cc54fc0), endpoint:"unix:$
//var/run/docker.sock", endpointURL:(*url.URL)(0xc42010b680), eventMonitor:(*docker.eventMonitoringState)(0xc42cc55020), requestedAPIVersion:docker.APIVersion(nil), serverAPIVersion:docker.APIVersion(nil), expec$
edAPIVersion:docker.APIVersion(nil), nativeHTTPClient:(*http.Client)(0xc42026b650)}
docker container is (*docker.Container)(nil)

UUID returning multi-value

When I tried to install gofn, an error was returned by the uuid (satori/go.uuid) dependency.

$ go get -u github.com/gofn/gofn
# github.com/gofn/gofn/provision
gofn/gofn/provision/docker.go:82:48: multiple-value uuid.NewV4() in single-value context

And the problem affects the following parts:

I can create a pull request to fix this.

New options not to create the machine if snapshot does not exist.

Add two new configuration options

  • do not create the machine in IaaS if the snapshot does not exist
  • also do not build the environment in the new machine

Today the gofn tries to create the machine if the snapshot does not exist.

With these new options the gofn should simply return an error if the snapshot does not exist.

Container exited with status code 1, but don't show errors

How to reproduce?

Files

# Dockerfile
FROM python:3.6-alpine

ENTRYPOINT ["python"]
CMD ["-c", "print(a)"]
//main.go
package main

import (
	"fmt"
	"log"
	"os"

	"github.com/nuveo/gofn"
	"github.com/nuveo/gofn/provision"
)

func main() {
	buildOpts := &provision.BuildOptions{
		ImageName:               "docker_test",
		DoNotUsePrefixImageName: true,
	}
	stdout, stderr, err := gofn.Run(buildOpts, nil)
	if err != nil {
		log.Println(err)
		os.Exit(1)
	}
	fmt.Println("Stderr: ", stderr)
	fmt.Println("Stdout: ", stdout)
	os.Exit(0)
}

steps

Build the image docker build -t docker_test

Run go run main.go

expected behavior

print log with error and exit with status 1

actual behavior

don't have error, print stderr(with traceback) and exit with 0

Missing docker-machine support

Today I have tried to running an example via docker-machine but I got:

$ go run main.go -contextDir=testDocker -imageName=python -dockerfile=Dockerfile
2017/07/13 16:12:18 Get http://unix.sock/images/json?filter=gofn%2Fpython: dial unix /var/run/docker.sock: connect: no such file or directory
Stderr:
Stdout:
2017/07/13 16:12:18 Get http://unix.sock/images/json?filter=gofn%2Fpython: dial unix /var/run/docker.sock: connect: no such file or directory
Stderr:
Stdout:
2017/07/13 16:12:18 Get http://unix.sock/images/json?filter=gofn%2Fpython: dial unix /var/run/docker.sock: connect: no such file or directory
Stderr:
Stdout:

Hypervisor

Hypervisor software to control various instances of gofn on different machines.

The hypervisor decides which machine will instantiate an image and control execution queues.

Improve logging

As a SysAdmin,
I want to be able to monitor a process started, ran by gofn.
So I can find errors, track performance and resources usage.

Stdout via streaming

In order to monitor the processes I suggest that Run() receive a function, this function will receive two *io.Writer as parameters and will be called internally by fnRun as a goroutine, before calling client.WaitContainer(container).

This hook will loop, reading stdin and stdout and taking any necessary action.

I'm already using AttachToContainerNonBlocking in fnRun to be able to read stdin, I believe the change is easy at that point, the trickiest thing to do is decide which is a good high-level interface to pass the new function into the API.

A final touch would be to receive errorlevel at the end of the container execution.

Group options in a structure

today:

func Run(contextDir, dockerFile, imageName, remote string, volumeOpts *provision.VolumeOptions) (stdout string, err error) {
    ...
}

proposal:

func Run(buildOptions *provision.BuildOptions, volumeOpts *provision.VolumeOptions) (stdout string, err error) {
    ...
}

Missing dependencies on running examples

Guys,

I installed gofn via go get and after tried to running the concurrent example, I got the following missing dependencies:

$ go run main.go -contextDir=testDocker -imageName=python -dockerfile=Dockerfile
../../iaas/digitalocean/digitalocean.go:24:2: cannot find package "github.com/digitalocean/godo" in any of:
	/home/leroy/.goenv/versions/1.8.3/src/github.com/digitalocean/godo (from $GOROOT)
	/home/leroy/go/src/github.com/digitalocean/godo (from $GOPATH)
../../iaas/digitalocean/digitalocean.go:27:2: cannot find package "golang.org/x/crypto/ssh" in any of:
	/home/leroy/.goenv/versions/1.8.3/src/golang.org/x/crypto/ssh (from $GOROOT)
	/home/leroy/go/src/golang.org/x/crypto/ssh (from $GOPATH)
../../iaas/digitalocean/digitalocean.go:28:2: cannot find package "golang.org/x/oauth2" in any of:
	/home/leroy/.goenv/versions/1.8.3/src/golang.org/x/oauth2 (from $GOROOT)
	/home/leroy/go/src/golang.org/x/oauth2 (from $GOPATH)

So, is that right? I really have to do another go get?

Example is not working.

I try to run the example, but an error occurs: image not found.
But i think that when an image is not found, it should be builded.

Digital Ocean support

gofn must be able to create a Digital Ocean machine (based on a Digital Ocean snapshot that has installed docker).

After creating the machine, run the container inside the machine and gofn get the stdout.

  • define interface for iaas
  • define a digitalocean iaas
  • authenticate in digitalocean
  • create a droplet
  • generate ssh keys
  • if not have a box os snapshot create one
  • remove a droplet*
  • install box os on new machine

*don't forget to shutdown before

Move project gofn to organization gofn

I understand that the project was created by Nuveo (which is why it was born within the organization Nuveo).
Looking for the future of the project we have many seas to navigate (several opportunities to make the project grow and earn its own life).

What's your opinion on moving to gofn organization?

Reuse machine instances to optimize IaaS usage

When a process will run in a machine in a IaaS, we have the possibility to reuse the same instance of the machine for other processes if the execution time is very short and the machine will stay up for less than 1 minute.

Support docker volume

Today we are able to put a JSON as a stdout, the idea to mount a volume is to generate a file (when necessary) to return.

Example:
Do crop in an image (big), so the gofn would be able to process other file type

Support stderr return from container

I suggest changing the FnRun function to be able to return stderr from the container and return error as well.

change from:

func FnRun(client *docker.Client, containerID string) (Stdout *bytes.Buffer)

to:

func FnRun(client *docker.Client, containerID string) (Stdout *bytes.Buffer,Stderr *bytes.Buffer, err error)

Then the Run function can return those values and we could remove the calls to panic().

Integration tests

Write integration tests that are skipped with a flag( -unit maybe).
These tests will comunicate with real docker(travis support this and it already setted in .travis.yml)

Other output formats

How about one parameter to allow other output formats?
JSON is the default but a parameter could allow other formats such as XML, CSV, raw, etc.

ssh key path

I want to set the path of the SSH key (optional)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.