openfunction / builder Goto Github PK

View Code? Open in Web Editor NEW

10.0 10.0 22.0 1.21 MB

A function builder of OpenFunction. Base on https://github.com/GoogleCloudPlatform/buildpacks

License: Apache License 2.0

Starlark 11.34% Go 74.94% Dockerfile 7.22% Shell 6.51%

serverless

builder's Introduction

Cloud native FaaS platform for running Serverless workloads with ease

👀 Overview

OpenFunction is a cloud-native open source FaaS (Function as a Service) platform aiming to let you focus on your business logic without having to maintain the underlying runtime environment and infrastructure. You only need to submit business-related source code in the form of functions.

OpenFunction features include:

Cloud agnostic and decoupled with cloud providers' BaaS
Pluggable architecture that allows multiple function runtimes
Support both sync and async functions
Unique async functions support that can consume events directly from event sources
Support generating OCI-Compliant container images directly from function source code.
Flexible autoscaling between 0 and N
Advanced async function autoscaling based on event sources' specific metrics
Simplified BaaS integration for both sync and async functions by introducing Dapr
Advanced function ingress & traffic management powered by K8s Gateway API
Flexible and easy-to-use events management framework

☸ Architecture

OpenFunction manages its components in the form of Custom Resource Definitions (CRD) throughout the lifecycle of a function, you can find more details in the Concepts section.

OpenFunction Events is OpenFunction's events framework, you can refer to OpenFunction Events for more information.

🚀 QuickStart

Install OpenFunction

To install OpenFunction, please refer to Installation Guide.

Create functions

You can find guides to create the sync and async functions in different languages here

Uninstall OpenFunction

To uninstall OpenFunction, please refer to Uninstallation Guide.

FAQ

When you encounter any problems when using OpenFunction, you can refer to the FAQ for help.

💻 Development

See the Development Guide to get started with developing this project.

🛣️ Roadmap

Here you can find OpenFunction roadmap.

🏘️ Community

Contact Us

Community Call

Events

Landscape

OpenFunction is a CNCF Sandbox project now which also enriches the CNCF Cloud Native Landscape.

📊 Status

builder's People

Contributors

Stargazers

Watchers

builder's Issues

Develop a builder

Based on the existing stack image, we need to add buildpacks such as ruby.bundle and ruby.functions-framework to generate a builder image

The go1.15 download link in dockerfile has expired

Removing intermediate container 579fcc95c12c
 ---> ec850a9ef5fa
Step 8/11 : RUN curl --fail --show-error --silent --location --retry 3 https://golang.org/dl/go1.15.linux-amd64.tar.gz | tar xz --directory /usr/local/ --strip-components=1
 ---> Running in 19e0ea3415ed
curl: (7) Failed to connect to golang.org port 443: Connection timed out

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Fail to build the stack

I tried to follow docs to build the go116 stack and builder, but facing some errors.

❯ pwd
/Users/home/dev/openfunction/builder
❯ bazel run //builders/go116/stack:build
INFO: Analyzed target //builders/go116/stack:build (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //builders/go116/stack:build up-to-date:
  bazel-bin/builders/go116/stack/build
INFO: Elapsed time: 0.478s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
> Extracting licenses tar
> Building base go116common image
[+] Building 3.3s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                   0.0s
 => => transferring dockerfile: 1.49kB                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/openfunctiondev/ubuntu_18_0_4:latest                                                                                                                                        3.1s
 => [auth] openfunctiondev/ubuntu_18_0_4:pull token for registry-1.docker.io                                                                                                                                           0.0s
 => [1/4] FROM docker.io/openfunctiondev/ubuntu_18_0_4@sha256:47202afe7bd8fb1eebfb6175f5eb4afba6747c22599dc04b53689574bb0a4f3b                                                                                         0.0s
 => CACHED [2/4] RUN apt-get update && apt-get install -y --no-install-recommends   libexpat1   libffi6   libmpdec2   libicu60   libc++1-9   tzdata   && apt-get clean && rm -rf /var/lib/apt/lists/*                  0.0s
 => CACHED [3/4] RUN groupadd cnb --gid 1000 &&   useradd --uid 1000 --gid 1000 -m -s /bin/bash cnb                                                                                                                    0.0s
 => CACHED [4/4] RUN curl --fail --show-error --silent --location --retry 3 https://go.dev/dl/go1.16.13.linux-amd64.tar.gz | tar xz --directory /usr/local/ --strip-components=1                                       0.0s
 => exporting to image                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                0.0s
 => => writing image sha256:d72db7c49a80d4c940f335b01f31f1b608b2a6657a7bf9eda958f017f2862082                                                                                                                           0.0s
 => => naming to docker.io/library/go116common                                                                                                                                                                         0.0s
> Building base go116run image
[+] Building 2.9s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                   0.0s
 => => transferring dockerfile: 865B                                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/busybox:1.35                                                                                                                                                        2.8s
 => [auth] library/busybox:pull token for registry-1.docker.io                                                                                                                                                         0.0s
 => [1/2] FROM docker.io/library/busybox:1.35@sha256:0d1d11dbad88f301fe745cf1b3fca8ee44ff5cf1c1be721f36b22bec592d9966                                                                                                  0.0s
 => CACHED [2/2] RUN adduser -u 1000 -D cnb cnb                                                                                                                                                                        0.0s
 => exporting to image                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                0.0s
 => => writing image sha256:f8f8e11174a63f35c447d5ba415a1b3095f5ad75734e38cf76ec80cd8fdc7f1b                                                                                                                           0.0s
 => => naming to docker.io/library/go116run                                                                                                                                                                            0.0s
> Building openfunctiondev/buildpacks-run-go:v2
[+] Building 0.1s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                   0.0s
 => => transferring dockerfile: 678B                                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/go116run:latest                                                                                                                                                     0.0s
 => CACHED [1/1] FROM docker.io/library/go116run                                                                                                                                                                       0.0s
 => exporting to image                                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                                0.0s
 => => writing image sha256:9a946a62545b94c565c12b0414ff6ba9647ecf001be8623a182d363f94376477                                                                                                                           0.0s
 => => naming to docker.io/openfunctiondev/buildpacks-run-go:v2                                                                                                                                                        0.0s
> Building openfunctiondev/buildpacks-go116-build:v2
[+] Building 0.1s (1/2)
 => [internal] load build definition from build.Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 121B                                                                                                                                                                                   0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount085995818/Users/home/dev/openfunction/builder/builders/go116/stack/build.Dockerfile: no such file or directory

Command failed is:

docker build --build-arg "from_image=go116common" -t "openfunctiondev/buildpacks-go116-build:$TAG" -f "${DIR}/build.Dockerfile" "${TEMP}"

if I removed the "${TEMP}", the Dockerfile can be found, but facing another error:

❯ bazel run //builders/go116/stack:build
INFO: Analyzed target //builders/go116/stack:build (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //builders/go116/stack:build up-to-date:
  bazel-bin/builders/go116/stack/build
INFO: Elapsed time: 0.365s, Critical Path: 0.01s
INFO: 2 processes: 2 internal.
INFO: Build completed successfully, 2 total actions
INFO: Build completed successfully, 2 total actions
> Extracting licenses tar
> Building base go116common image
[+] Building 1.7s (8/8) FINISHED                                                                                                                                                           
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 1.49kB                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/openfunctiondev/ubuntu_18_0_4:latest                                                                                                       1.6s
 => [1/4] FROM docker.io/openfunctiondev/ubuntu_18_0_4@sha256:47202afe7bd8fb1eebfb6175f5eb4afba6747c22599dc04b53689574bb0a4f3b                                                        0.0s
 => CACHED [2/4] RUN apt-get update && apt-get install -y --no-install-recommends   libexpat1   libffi6   libmpdec2   libicu60   libc++1-9   tzdata   && apt-get clean && rm -rf /va  0.0s
 => CACHED [3/4] RUN groupadd cnb --gid 1000 &&   useradd --uid 1000 --gid 1000 -m -s /bin/bash cnb                                                                                   0.0s
 => CACHED [4/4] RUN curl --fail --show-error --silent --location --retry 3 https://go.dev/dl/go1.16.13.linux-amd64.tar.gz | tar xz --directory /usr/local/ --strip-components=1      0.0s
 => exporting to image                                                                                                                                                                0.0s
 => => exporting layers                                                                                                                                                               0.0s
 => => writing image sha256:d72db7c49a80d4c940f335b01f31f1b608b2a6657a7bf9eda958f017f2862082                                                                                          0.0s
 => => naming to docker.io/library/go116common                                                                                                                                        0.0s
> Building base go116run image
[+] Building 1.6s (6/6) FINISHED                                                                                                                                                           
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 865B                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/busybox:1.35                                                                                                                       1.5s
 => [1/2] FROM docker.io/library/busybox:1.35@sha256:0d1d11dbad88f301fe745cf1b3fca8ee44ff5cf1c1be721f36b22bec592d9966                                                                 0.0s
 => CACHED [2/2] RUN adduser -u 1000 -D cnb cnb                                                                                                                                       0.0s
 => exporting to image                                                                                                                                                                0.0s
 => => exporting layers                                                                                                                                                               0.0s
 => => writing image sha256:f8f8e11174a63f35c447d5ba415a1b3095f5ad75734e38cf76ec80cd8fdc7f1b                                                                                          0.0s
 => => naming to docker.io/library/go116run                                                                                                                                           0.0s
> Building openfunctiondev/buildpacks-run-go:v2
[+] Building 0.2s (5/5) FINISHED                                                                                                                                                           
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 678B                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/go116run:latest                                                                                                                    0.0s
 => CACHED [1/1] FROM docker.io/library/go116run                                                                                                                                      0.0s
 => exporting to image                                                                                                                                                                0.0s
 => => exporting layers                                                                                                                                                               0.0s
 => => writing image sha256:9a946a62545b94c565c12b0414ff6ba9647ecf001be8623a182d363f94376477                                                                                          0.0s
 => => naming to docker.io/openfunctiondev/buildpacks-run-go:v2                                                                                                                       0.0s
> Building openfunctiondev/buildpacks-go116-build:v2
"docker build" requires exactly 1 argument.
See 'docker build --help'.

Usage:  docker build [OPTIONS] PATH | URL | -

Build an image from a Dockerfile

System: MacOS 12.3.1

❯ docker --version
Docker version 20.10.14, build a224086
❯ bazel --version
bazel 5.1.1-homebrew

Go Builder for v1.22-1.23

It would be great to have a builder for a more updated go version as there are now many packages that rely on at least Go v1.18+.

Alternatively, some documentation on compiling our own builder would be equally as helpful. Thanks!

How to package the function faster

builder: openfunction/builder-java:v2-11
cmd: pack build --path --builder openfunction/builder-java:v2-11 --env FUNC_NAME= --env FUNC_CLEAR_SOURCE="true"

===> BUILDING
[builder] Done "mvn clean package --batch-mode -DskipTests -Dhttp.keepAlive=false --quiet" (8m23.808882339s)
[builder] Done "mvn --batch-mode dependency:copy-dependencies --quiet" (1m30.132562873s)
[builder] Done "mvn help:evaluate -q -DforceStdout -Dexpression=project.artifactId/${project.version}" (1m8.799153597s)

Add a ruby stack image

Add a ruby stack image, which needs to contain the ruby runtime
After executing the command bazel run //builders/ruby26/stack:build ，it will create three images

ruby26common
penghuima/buildpacks-ruby26-run:v1
penghuima/buildpacks-ruby26-build:v1

Some things to note

The whole process may take 10 minutes
Requires base image gcr.io/gcp-runtimes/ubuntu_18_0_4 ,It can also be replaced with other basic mirrors
It may be necessary to add permissions to the script execution file build.sh

OpenFunction Builder for Ruby

We need to add a builder to the OpenFunction builder for building ruby code.

Todo list:

Add a ruby stack image, which needs to contain the ruby runtime
Develop a builder that can handle HTTP and CloudEvents type function (by using functions-framework)

dotnet builder for OpenFunction

we need to add a new buildpack layer for dotnet in the builder to handle the function framework of dotnet functions functions-framework-dotnet .

Add a new buildpack for the dotnet builder that handles the Openfunction
The new builder can build a dotnet function normally
The new builder can build a dotnet OpenFuncAsync function
Complete docs and samples

Nodejs builder for OpenFunction

The functions-framework-nodejs project of OpenFunction already supports functions of type OpenFuncAsync, we need to add a new buildpack layer for nodejs in the builder to handle the function framework of nodejs functions.

Add a new buildpack for the nodejs builder that handles the OpenFuncAsync function
The new builder can build a nodejs function normally
Complete docs and samples

bazel run/build failed with bazel `6.2.1`

➜ bazel run //builders/go115/stack:build
ERROR: /home/mmif/builder/licenses/BUILD.bazel:7:8: //licenses:licenses: no such attribute 'extension' in '_real_pkg_tar' rule
ERROR: /home/mmif/builder/licenses/BUILD.bazel:7:8: //licenses:licenses: no such attribute 'strip_prefix' in '_real_pkg_tar' rule
ERROR: /home/mmif/builder/builders/go115/stack/BUILD.bazel:3:10: Target '//licenses:licenses.tar' contains an error and its package is in error and referenced by '//builders/go115/stack:build'
ERROR: Analysis of target '//builders/go115/stack:build' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.059s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
ERROR: Build failed. Not running target

➜  bazel build //builders/go115:builder.image
ERROR: /home/mmif/builder/builders/go115/BUILD.bazel:5:8: //builders/go115:builder: no such attribute 'extension' in '_real_pkg_tar' rule
ERROR: /home/mmif/builder/builders/go115/BUILD.bazel:5:8: //builders/go115:builder: no such attribute 'deps' in '_real_pkg_tar' rule
ERROR: Skipping '//builders/go115:builder.image': Error evaluating '//builders/go115:builder.image': error loading package 'builders/go115': Package 'builders/go115' contains errors
WARNING: Target pattern parsing failed.
ERROR: Error evaluating '//builders/go115:builder.image': error loading package 'builders/go115': Package 'builders/go115' contains errors
INFO: Elapsed time: 0.064s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded)

OS env:

➜  cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"

Bazel version:

bazel 6.2.1

support FUNC_GOSUMDB env

When using the private go proxy that cannot connect to the internet, we cannot verifying the module via https://sum.golang.org. So we need to set GOSUMDB off.

under /pkg/golang/golang.go can support domestic go mod operation and set goproxy= https://goproxy.cn ,direct

func ExecWithGoproxyFallback(ctx *gcp.Context, cmd []string, opts ...gcp.ExecOption) *gcp.ExecResult {
if SupportsGoProxyFallback(ctx) {
goProxy := os.Getenv(env.GoProxy)
if goProxy != "" {
opts = append(opts, gcp.WithEnv(goProxy))
}
return ctx.Exec(cmd, opts...)
}

result, err := ctx.ExecWithErr(cmd, opts...)
if err == nil {
	return result
}
ctx.Warnf("%q failed. Retrying with GOSUMDB=off GOPROXY=direct. Error: %v", strings.Join(cmd, " "), err)
    
opts = append(opts, gcp.WithEnv("GOSUMDB=off", "GOPROXY=direct"))
return ctx.Exec(cmd, opts...)

}

Link errors

When I click on links such as Go1.16, it returns error report 404

[python] need to provide a python builder that supports the ofn v1beta2 api

Now that functions-framework-python supports the ofn v1beta2 api, we need to provide a corresponding builder to allow ofn to run the new Python functions.

Optimize the image created by cloud-native buildpacks approach to reduce image size

Description

The current build from the source is not so optimized. I tried to build the hello world program from samples,

For image built from dockerfile, the image size is about 4MB. For image built from source, the size is about 130MB. Hope that the size of image built from source can be optimized as well.

Additional context

Screenshot of the image size for both cases

Modify configuration

Modify configuration files so that the ruby builder image can be successfully generated

builders/BUILD.bazel
cmd/config/entrypoint/ build.bazel
cmd/config/entrypoint/buildpack.toml
ruby/bundle/BUILD.bazel
ruby/bundle/buildpack.toml
ruby/functions_framework/BUILD.bazel
ruby/functions_framework/buildpack.toml

Summary of suggestions for improvements to the builder

As we mentioned in the readme, the builder project is based on gcp/buildpacks with some modifications to facilitate support for Chinese users and support for OpenFunction's functions-framework. gcp/buildpacks primarily serves GCP, so we may need to do more work on improving builder customization.

Of course, we are also considering rewriting a Builder project that will still follow the buildpacks lifecycle criteria (which drove us to find out more about the builder's improvement needs).

plugin name bug

Actual Behavior

See https://github.com/OpenFunction/builder/blob/main/cmd/go/of_functions_framework/main.go#L318-L318

if plg.IsDir() && strings.HasPrefix(plg.Name(), "plugin-")

https://github.com/OpenFunction/builder/blob/main/cmd/go/of_functions_framework/main.go#L329-L329
Since the above condition exists, this condition is always True.

if strings.Contains(plg.Name(), "-")

Java builder fail!

When I use pack with java builder on mac, has below error:



*** Invalid JIT return address 0000000706B1FE58 in 0000000000DE7B10

09:01:58.753 0xde7800    j9vm.249    *   ** ASSERTION FAILED ** at ../../../../../openj9/runtime/vm/swalk.c:1602: ((0 ))
JVMDUMP039I Processing dump event "traceassert", detail "" at 2023/03/04 09:01:58 - please wait.
0000000000DE7800: Object neither in heap nor stack-allocated in thread BasicRepositoryConnector-repo.maven.apache.org-3-0
0000000000DE7800:	O-Slot=0000000000E29BF8
0000000000DE7800:	O-Slot value=000000400C465250
0000000000DE7800:	PC=000000401BE92041
0000000000DE7800:	framesWalked=5
0000000000DE7800:	arg0EA=0000000000E29BF8
0000000000DE7800:	walkSP=0000000000E29B50
0000000000DE7800:	literals=0000000000000010
0000000000DE7800:	jitInfo=0000004085628538
0000000000DE7800:	method=0000000000BBD0A0 (java/security/MessageDigest.getInstance(Ljava/lang/String;)Ljava/security/MessageDigest;) (JIT)
0000000000DE7800:	stack=0000000000E27020-0000000000E2A000
0000000000DE7800: Object neither in heap nor stack-allocated in thread BasicRepositoryConnector-repo.maven.apache.org-3-0
0000000000DE7800:	O-Slot=00000042033579E0
0000000000DE7800:	O-Slot value=00000000E0D32F23
0000000000DE7800:	PC=000000401BE92041
0000000000DE7800:	framesWalked=5
0000000000DE7800:	arg0EA=0000000000E29BF8
0000000000DE7800:	walkSP=0000000000E29B50
0000000000DE7800:	literals=0000000000000010
0000000000DE7800:	jitInfo=0000004085628538
0000000000DE7800:	method=0000000000BBD0A0 (java/security/MessageDigest.getInstance(Ljava/lang/String;)Ljava/security/MessageDigest;) (JIT)
0000000000DE7800:	stack=0000000000E27020-0000000000E2A000
JVMDUMP032I JVM requested System dump using '/workspace/core.20230304.090158.131.0001.dmp' in response to an event
JVMDUMP010I System dump written to /workspace/core.20230304.090158.131.0001.dmp
0000000000DE7800: Object neither in heap nor stack-allocated in thread BasicRepositoryConnector-repo.maven.apache.org-3-0
0000000000DE7800:	O-Slot=0000000000E29BF8
0000000000DE7800:	O-Slot value=000000400C465250
0000000000DE7800:	PC=000000401BE92041
0000000000DE7800:	framesWalked=5
0000000000DE7800:	arg0EA=0000000000E29BF8
0000000000DE7800:	walkSP=0000000000E29B50
0000000000DE7800:	literals=0000000000000010
0000000000DE7800:	jitInfo=0000004085628538
0000000000DE7800:	method=0000000000BBD0A0 (java/security/MessageDigest.getInstance(Ljava/lang/String;)Ljava/security/MessageDigest;) (JIT)
0000000000DE7800:	stack=0000000000E27020-0000000000E2A000
0000000000DE7800: Object neither in heap nor stack-allocated in thread BasicRepositoryConnector-repo.maven.apache.org-3-0
0000000000DE7800:	O-Slot=00000042033579E0
0000000000DE7800:	O-Slot value=00000000E0D32F23
0000000000DE7800:	PC=000000401BE92041
0000000000DE7800:	framesWalked=5
0000000000DE7800:	arg0EA=0000000000E29BF8
0000000000DE7800:	walkSP=0000000000E29B50
0000000000DE7800:	literals=0000000000000010
0000000000DE7800:	jitInfo=0000004085628538
0000000000DE7800:	method=0000000000BBD0A0 (java/security/MessageDigest.getInstance(Ljava/lang/String;)Ljava/security/MessageDigest;) (JIT)
0000000000DE7800:	stack=0000000000E27020-0000000000E2A000
JVMDUMP032I JVM requested Java dump using '/workspace/javacore.20230304.090158.131.0002.txt' in response to an event
Done "mvn clean package --batch-mode -DskipTests -Dhttp.keepAlive=..." (6m24.652248509s)
Failure: (ID: 157993e9) 
ERROR: failed to build: exit status 255
ERROR: failed to build: executing lifecycle: failed with status code: 51

Builder cmd as

pack build ${IMAGE_TAG} --trust-builder --path ./ --builder openfunction/builder-java:v2-18 --env FUNC_NAME="dev.openfunction.samples.OpenFunctionImpl"  --env FUNC_CLEAR_SOURCE=true