api7 / apisix-build-tools Goto Github PK
View Code? Open in Web Editor NEWBuild tools to package and release
License: Apache License 2.0
Build tools to package and release
License: Apache License 2.0
I want to revert #65, or submit a new PR to do this.
Now it seems unnecessary to do so,
https://github.com/api7/apisix-build-tools/blob/master/utils/install-common.sh#L86
and
https://github.com/api7/apisix-build-tools/blob/master/utils/install-common.sh#L89
are related and I think it's best to keep it the way it was before.
my mistake.
I used the master branch of apisix to make a package, but I found that the code of the discovery part is missing after the package is installed。
before:
after:
And I found that apisix did make code adjustments to the discovery module:
By the way,my build command:
make package type=rpm app=apisix version=master-copy checkout=master-cvte image_base=centos image_tag=7 local_code_path=apisix-master-copy openresty=apisix-base
Currently, this build tool will always download the apisix and apisix-dashboard code remotely. We should also support the local code.
except to the tools developed by ourselves, we also need to collect wrk, wrk2, etc.
Some time ago I researched some well-known open source projects, compared their packaging methods, and found that the packaging method of Apache APISIX method has many problems.
I think they need to be improved.
https://github.com/Kong/kong-build-tools
https://github.com/apache/apisix-docker/blob/master/all-in-one/apisix-dashboard/Dockerfile
Just look here https://github.com/api7/api7-dashboard/runs/5764328092?check_suite_focus=true#step:5:4512
The environment:
apisix-build-tools/utils/install-common.sh
Line 154 in 50d0733
If we often use it on a foreign machine, such as github action,why we use goproxy.cn
as the proxy service?
after build package with makefile and dockerfile, should optimize it to use dockerfile to deploy fpm directly.
Understanding the GeoIP data for an incoming request is a common use case for many API GWs. I think Apisix should support natively adding the "GeoIP2 NGINX module". Currently, the only way to do that is to build it ourselves.
For reference, have a look at this issue: apache/apisix#3946 (comment)
Also, GeoIP( not Geoip2) NGINX module only works with the legacy database format and MaxMind seems to stop the legacy format from being publicly available in 2019 as it will be officially unsupported in around a year. We should probably go with " the geoip2 module " instead, and it's officially supported by Nginx as we can find the geoip2 module in Nginx doc, so it is definitely supported that's why I didn't see a reason we shouldn't do it then. I think GeoIP2 support would be a good feature for Apisix.
In general, the name of the rpm artifact often contains the distribution of the supported OS. For example, openresty-1.19.3.2-1.el8.x86_64.rpm
shows that it's for CentOS 8. While the rpm files for apisix do not contain the distribution information and users may get confused to know which OS the apisix-2.8-0.x86_64.rpm
is for.
So it would be great if we could make use of the --rpm-dist
flag of fpm
to add the distribution of rpm, such as: apisix-2.8-0.el8.x86_64.rpm
. Furthermore, this improvement may also allow us to make one rpm artifact for each version of one distribution.
BTW: the 1
is often used as the first release version of one rpm. So do we need to change the current 0
to 1
, like: apisix-2.8-1.el8.x86_64.rpm
?
Any feedback is welcome. Thanks.
Alpine images will drastically shrink the image size and network cost and have our services start faster. We are planning to extend alpine image to all since some images are based on alpine already.
Any feedback and suggestions are welcome :)
https://github.com/apache/apisix/blob/0aa97c30f7741eeeb341eed0795c3bd0ef1b7556/bin/apisix#L32
# find the openresty
OR_BIN=$(which openresty || exit 1)
The apisix startup script use which
to determine openresty location , but which
is not part of centos:7
official docker image.
assign me
Currently we only specify the min version requirement of apisix-base, which causes a problem when people try to install an old version of APISIX, because the latest apisix-base will be installed. The latest apisix-base doesn't work with an old version of APISIX.
In some IDCs, we need the apisix server to run in linux/arm64
server, just like our current linux/amd64 server online.
So, I suggest we should supply the arm64
rpm package to meet the future, not only the x86-64 rpm package.
What do you think about it?
./apisix-nginx-module-1.9.0/src/ngx_http_apisix_module.c:190:17: error: implicit declaration of function 'sk_X509_deep_copy' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
new_chain = sk_X509_deep_copy(cert, ngx_http_apisix_x509_copy,
^
rpm打包成功后,使用rpm -ivh安装,报错openresty >= 1.15.8.1 is needed by apisix-0.8-0.noarch
查看openresty 版本,
openresty -v
nginx version: openresty/1.15.8.2
安装时去掉依赖检查,可以安装成功
rpm -ivh apisix-0.8-0.noarch.rpm --nodeps
APISIX imports new dependencies as it iterates, so to ensure proper packaging, we should perform CI regularly.
I wanted to add this feature because APISIX introduced casbin three days ago, and casbin needs pcre, which causes the packaging tool unable to package the master branch properly.
If we add regularly CI, it will ensure that the new dependencies are installed correctly.
Any feedback is welcome. Thanks.
cause by sed
https://github.com/api7/apisix-build-tools/blob/master/utils/install-common.sh#L77
Error build action
https://github.com/api7/apisix-build-tools/runs/5429589069?check_suite_focus=true
Error: Could not load rockspec file /apisix/./rockspec/apisix-master-0.rockspec (Error loading file: [string "/apisix/./rockspec/apisix-master-0.rockspec"]:77: '}' expected (to close '{' at line 33) near '.')
The command '/bin/sh -c /install-common.sh install_apisix && /determine-dist.sh' returned a non-zero code: 1
make: *** [Makefile:121: build-apisix-rpm] Error 1
for https://github.com/apache/apisix/blob/master/rockspec/apisix-master-0.rockspec Added
dependencies = {
"net-url = 0.9-1",
}
replaced with sed result
dependencies = {
"net-url = "./apisix",
}
I think this https://github.com/api7/apisix-build-tools/blob/master/utils/install-common.sh#L77 sed maybe need update
For the convenience of Ubuntu/Debian users, we need to support deb.
Based on the current situation, the following points need to be completed:
And test for them:
according to doc of apisix plugin real-ip (https://apisix.apache.org/zh/docs/apisix/plugins/real-ip/ ), which "requires APISIX to run on APISIX-OpenResty."
and refer to another doc https://apisix.apache.org/docs/apisix/how-to-build/#step-6-build-openresty-for-apache-apisix .
so APISIX is NOT equal to APISIX OpenResty ?
and how to judge whether or not that my installed apisix has include "APISIX OpenResty" feature ,e.g real-ip .
sorry for this somewhat newbie-like question.
For now there's something wrong with get metalink from the epel repository in the CI running on CentOS 7, please see https://github.com/api7/apisix-build-tools/pull/86/checks?check_run_id=3468216581 for details.
+ rpm -ivh epel-release-latest-7.noarch.rpm
warning: epel-release-latest-7.noarch.rpm: Header V4 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Preparing... ########################################
Updating / installing...
epel-release-7-13 ########################################
+ yum install -y yum-utils readline-dev readline-devel
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
One of the configured repositories failed (Unknown),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=<repoid> ...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid>
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again
The command '/bin/sh -c /install-common.sh install_apisix_dependencies_rpm' returned a non-zero code: 1
make: *** [Makefile:74: build-apisix-rpm] Error 1
Error: Process completed with exit code 2.
After installing apache-apisix-repo, apisix can't be installed.
$ sudo yum install -y https://repos.apiseven.com/packages/centos/apache-apisix-repo-1.0-1.noarch.rpm
$ LANG=C sudo dnf install apisix
Last metadata expiration check: 0:03:47 ago on Tue Jun 28 15:29:29 2022.
Error:
Problem: cannot install the best candidate for the job
- nothing provides apisix-base >= 1.19.9.1.6 needed by apisix-2.14.1-0.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
sudo dnf install -y apisix succeeds and apisix and apisix-base package is installed on.
# cat /etc/redhat-release
Rocky Linux release 8.6 (Green Obsidian)
# dnf info apisix
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:10:28 ago on Tue Jun 28 06:24:32 2022.
Available Packages
Name : apisix
Version : 2.14.1
Release : 0.el8
Architecture : x86_64
Size : 2.2 M
Source : apisix-2.14.1-0.el8.src.rpm
Repository : release
Summary : Apache APISIX is a distributed gateway for APIs and Microservices, focused on high performance and reliability.
URL : http://apisix.apache.org/
License : ASL 2.0
Description : Apache APISIX is a distributed gateway for APIs and Microservices, focused on high performance and reliability.
It is strange that it points .el7 package.
# dnf info apisix-base
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:10:43 ago on Tue Jun 28 06:24:32 2022.
Available Packages
Name : apisix-base
Version : 1.19.3.2.0
Release : 0.el7
Architecture : x86_64
Size : 35 M
Source : apisix-base-1.19.3.2.0-0.el7.src.rpm
Repository : release
Summary : APISIX's OpenResty distribution.
URL : http://apisix.apache.org/
License : ASL 2.0
Description : APISIX's OpenResty distribution.
In a few weeks before, it could be installed.
(Before, apisix-base 1.21.4.1.0 was available)
# LANG=C dnf info apisix-base
Last metadata expiration check: 0:00:49 ago on Tue Jun 28 15:52:25 2022.
Installed Packages
Name : apisix-base
Version : 1.21.4.1.0
Release : 0.el8
Architecture : x86_64
Size : 119 M
Source : apisix-base-1.21.4.1.0-0.el8.src.rpm
Repository : @System
From repo : release
Summary : APISIX's OpenResty distribution.
URL : http://apisix.apache.org/
License : ASL 2.0
Description : APISIX's OpenResty distribution.
i found that the official image of apisix will thorw an error of need to build APISIX-OpenResty to support client restriction
when i use client-control plugin
i use this cmd make package type=deb app=apisix-base version=2.8 checkout=v2.8
build a deb .
but after i install this package , i don't find any executeable file on my machine
i event don't know what is the apisix-base mean and which cmd arguments should i pick
please give me some suggestion and help me solve the problem of need to build APISIX-OpenResty to support client restriction
thanks
For now, the apisix rpm will always use openresty
as a dependency. Actually users may use the enhanced one apisix-openresty
to serve apisix. So this build-tool should support users to choose which openresty version to use.
PR #66 only add pcre as a build dependency. We should also specify libpcre as a dependency, both in rpm & in deb.
original comment: #25 (comment)
I have learned some well-known open-source projects and found that their packaging methods have similarities. I wonder if the packaging methods of apisix should learned from them?
tyk | kong | traefik | skywalking | envoy | tidb | |
---|---|---|---|---|---|---|
Project language | golang | lua | golang | java | c++ | golang |
Packaging env | dockerfile | dockerfile | dockerfile | docker | ||
Packaging method | shell | makefile | makefile | mvnw | shell | makefile |
In these projects, the technology stacks of Kong and apisix are the same, and I think in these projects, Kong’s method is better: deploy the packaging environment via docker, so that the environment is clean and unified every time; packaging via makefile
, so that the packaging entry is the same, passing different parameters can make different packages.
What do you think?
When we compile openresty, we need to use openssl. When we use openssl dynamic library, we cannot guarantee that the version of openssl officially maintained by openresty is finally used.
Originally posted by @membphis in #14 (comment)
fpm -f -s dir -t rpm -n apisix -a `uname -i` -v $version --iteration $iteration \
-d 'openresty >= 1.15.8.1' \
-d 'openresty-openssl-devel' \
Why not use openresty-openssl
instead of openresty-openssl-devel
ping @Fuchange @spacewander @membphis
Now build packages of apisix (or dashboard) is using different scripts, should use makefile
to do it.
According to different parameters, supports build apisix or dashboard package.
In apache/apisix@98be489 the upstream project changed the shebang from #!/usr/bin/env lua
to #!/bin/bash
which causes the following error when trying to start it:
$ apisix
/usr/local/openresty/luajit/bin/luajit: /usr/bin/apisix:4: unexpected symbol near '#'
But even when manually changing the shebang back (and the line added afterwards), the application does not start:
root@host:/opt/software# apisix
/usr/local/openresty/luajit/bin/luajit /usr/local/apisix/apisix/cli/apisix.lua
/usr/local/openresty/luajit/bin/luajit: /usr/local/apisix/apisix/cli/apisix.lua:31: module 'apisix.cli.env' not found:
no field package.preload['apisix.cli.env']
no file '/usr/local/apisix/deps/share/lua/5.1/apisix/cli/env.lua'
no file './apisix/cli/env.lua'
no file '/usr/local/openresty/luajit/share/luajit-2.1.0-beta3/apisix/cli/env.lua'
no file '/usr/local/share/lua/5.1/apisix/cli/env.lua'
no file '/usr/local/share/lua/5.1/apisix/cli/env/init.lua'
no file '/usr/local/openresty/luajit/share/lua/5.1/apisix/cli/env.lua'
no file '/usr/local/openresty/luajit/share/lua/5.1/apisix/cli/env/init.lua'
no file '/usr/local/apisix/deps/lib64/lua/5.1/apisix/cli/env.so'
no file '/usr/local/apisix/deps/lib/lua/5.1/apisix/cli/env.so'
no file './apisix/cli/env.so'
no file '/usr/local/lib/lua/5.1/apisix/cli/env.so'
no file '/usr/local/openresty/luajit/lib/lua/5.1/apisix/cli/env.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
no file '/usr/local/apisix/deps/lib64/lua/5.1/apisix.so'
no file '/usr/local/apisix/deps/lib/lua/5.1/apisix.so'
no file './apisix.so'
no file '/usr/local/lib/lua/5.1/apisix.so'
no file '/usr/local/openresty/luajit/lib/lua/5.1/apisix.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'require'
/usr/local/apisix/apisix/cli/apisix.lua:31: in main chunk
[C]: at 0x5562cfe732b0
Version 2.3.0 starts without problems.
after build package with makefile and dockerfile, these duplicate codes of Makefile should be optimized.
The apache/apisix, apache/apisix-dashboard, and some other projects are using this tool for building and packaging. But they seem always rely on the latest code from the master branch, which actully is a non-persistent dependency.
For example, the #67 renamed the rpm name, which caused CI failures in both apache/apisix and apache/apisix-dashboard.
So I would propose to make this repository under versioning control, as well as only the versioned code could be the dependency of other projects. I would recommend the first version/tag to be 2.0.0
.
What do you think? @spacewander @tzssangglass
Need to verify if the output rpm can be installed
After installed apisix-2.12.0-0.el7.x86_64 with yum on CentOS7, I have a look at rpm -q --changelog apisix the changelog, but it shows nothing :-(.
Will the changelog adds to package?
It doesn't specify --changelog at packaging by fpm.
fpm supports changelog flag for rpm dep but not apk:
--rpm-changelog
--dep-changelog
Refers from fpm-cli.
dependent package openresty-*
can find it.
[vagrant@lvs ~]$ rpm -q --changelog openresty-openssl111-devel-1.1.1l-1.el7.x86_64
* Tue May 11 2021 Jiahao Wang 1.1.1k-1
- upgraded OpenSSL to 1.1.1k.
* Thu Dec 10 2020 Yichun Zhang (agentzh) 1.1.1i-1
- upgraded OpenSSL to 1.1.1i.
* Mon May 14 2018 Yichun Zhang (agentzh) 1.1.0h-1
- upgraded openresty-openssl to 1.1.0h.
When trying to test the existing plugin with apisix RPM package, it fails with loading module failure.
(Just to clarify packaged version of apisix works well with it)
$ rpm -q apisix
apisix-2.14.1-0.el8.x86_64
$ rpm -q apisix-base
apisix-base-1.21.4.1.0-0.el8.x86_64
$ rpm -q perl-Test-Nginx
perl-Test-Nginx-0.29-2.el8.noarch
TEST_NGINX_BINARY=/usr/bin/openresty prove -I. -I./t -r t/plugin/example.t
...
Test Summary Report
-------------------
t/plugin/example.t (Wstat: 0 Tests: 91 Failed: 30)
Failed tests: 3-4, 10-11, 17-18, 24-25, 31-32, 42-49
52-53, 59-60, 66-67, 73-74, 80-81, 87-88
Parse errors: No plan found in TAP output
Files=1, Tests=91, 4 wallclock secs ( 0.02 usr 0.01 sys + 0.57 cusr 0.18 csys = 0.78 CPU)
Result: FAIL
TEST_NGINX_BINARY=/usr/bin/openresty prove -I. -I./t -r t/plugin/example.t succeeds.
t/servroot/logs/error.log indicates a loading error like this:
2022/07/01 14:30:40 [error] 81507#81507: *2 [lua] plugin.lua:110: load_plugin(): failed to load plugin [ldap-auth] err: error loading module
'lualdap' from file '/usr/local/apisix/deps/lib/lua/5.1/lualdap.so':
/usr/lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b, context: init_worker_by_lua*
2022/07/01 14:30:40 [error] 81506#81506: *1 [lua] plugin.lua:110: load_plugin(): failed to load plugin [ldap-auth] err: error loading module
'lualdap' from file '/usr/local/apisix/deps/lib/lua/5.1/lualdap.so':
/usr/lib64/libk5crypto.so.3: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b, context: init_worker_by_lua*
Then, disabled ldap-auth from conf/config-default.yaml plugins: section, it succeeds, so it can be a temporary workaround to test it.
NOTE: I expected that /usr/lib64/* is used, but it seems that it is not true in this case (undefined symbol).
$ ldd /usr/local/apisix/deps/lib/lua/5.1/lualdap.so |grep crypto
libcrypto.so.1.1 => /usr/lib64/libcrypto.so.1.1 (0x00007fdbf489e000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007fdbf3d1b000)
$ objdump -TC /usr/lib64/libcrypt.so.1.1.0 |grep EVP_KDF_
$ objdump -TC /usr/lib64/libk5crypto.so.3 | grep EVP_KDF_
0000000000000000 DF *UND* 0000000000000000 OPENSSL_1_1_1b EVP_KDF_ctrl
0000000000000000 DF *UND* 0000000000000000 OPENSSL_1_1_1b EVP_KDF_CTX_new_id
0000000000000000 DF *UND* 0000000000000000 OPENSSL_1_1_1b EVP_KDF_CTX_free
0000000000000000 DF *UND* 0000000000000000 OPENSSL_1_1_1b EVP_KDF_derive
$ objdump -TC /usr/local/openresty/openssl111/lib/libcrypto.so.1.1 | grep EVP_KDF
It seems that there is a mismatch between build time dependency and runtime dependency or something else.
Should be similar to openresty-debug and openresty-asan
Now, we don't test the rpm after building.
I think we should add an action to test this.
Hello, I would like to ask whether apifix has plans based on lua5.3 now?
I'm now trying to get apifix to work in lua5.3, but I'm having some problems.
If there are relevant documents, it will provide a lot of help.
你好, 我想问一下现在有计划将apisix基于lua5.3运行吗?
我正在尝试做这个,但遇到了一些困难。
如果有相关的文档的话将会很有帮助。
I note that apisix-2.1 provided a noarch rpm package,
But it contains some binaries files built on X86_64 and caused the apifix command cant find the file at ARM system.
Like apache/apisix#issue-691844591.
# file ./local/apisix/deps/lib/lua/5.1/lfs.so
./local/apisix/deps/lib/lua/5.1/lfs.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=5de2ad9d73ddbdaa91151ddfee3eef545a010902, not stripped
Would you like to add some options to fpm in run.sh to buind different arch packages?
Or I can provided an spec file to build packages by rpmbuild command ?
通过此项目构建的 apisix rpm 包可执行文件不可用
构建命令:
make package type=rpm app=apisix version=2.10 checkout=2.10 local_code_path=./apisix-2.10.0
得到的安装包安装后,提示:
apisix help
/usr/local/openresty/luajit/bin/luajit: /usr/bin/apisix:4: unexpected symbol near '#'
查看 /usr/bin/apisix 发现:
#! /usr/local/openresty/luajit/bin/luajit
package.path = "/usr/local/apisix/?.lua;" .. package.path
#
...
应该是这个样子才对的吧:
#!/bin/bash
...
看了一下 install_common.sh 中的内容,这一块有问题
if [ "${checkout_v}" = "master" ] || [ "${checkout_v:0:1}" != "v" -a "${checkout_v}" \> "2.2" ] || [ "${checkout_v:0:1}" = "v" -a "${checkout_v:1}" \> "2.2" ]; then
echo 'use shell '
else
bin='#! /usr/local/openresty/luajit/bin/luajit\npackage.path = "/usr/local/apisix/?.lua;" .. package.path'
sed -i "1s@.*@$bin@" /tmp/build/output/apisix/usr/bin/apisix
fi
cp -r /usr/local/apisix/* /tmp/build/output/apisix/usr/local/apisix/
mv /tmp/build/output/apisix/usr/local/apisix/deps/share/lua/5.1/apisix /tmp/build/output/apisix/usr/local/apisix/
if [ "${checkout_v}" = "master" ] || [ "${checkout_v:0:1}" != "v" -a "${checkout_v}" \> "2.2" ] || [ "${checkout_v:0:1}" = "v" -a "${checkout_v:1}" \> "2.2" ]; then
bin='package.path = "/usr/local/apisix/?.lua;" .. package.path'
sed -i "1s@.*@$bin@" /tmp/build/output/apisix/usr/local/apisix/apisix/cli/apisix.lua
else
echo ''
fi
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.