apache/apisix-python-plugin-runner

Apache APISIX Python plugin runner

apisix-python-plugin-runner's Issues

ci: missing ci

We need a way to confirm that the current code is working properly.

Exception Handling

Hello,
What is the intended behaviour of a plugin raising an exception?

e.g.

  "plugins": {
    "ext-plugin-pre-req": {
      "allow_degradation": false,
      "conf": [
        {
          "name": "netsuite-authenticator",
          "value": "this is invalid json config"
        }
       ]
      }
    }

    def config(self, conf: Any) -> Any:
        """
        Parse plugin configuration
        :param conf:
        :return:
        """
        try:
            conf_json = json.loads(conf)
        except ValueError as e:
            raise Exception("Configuration is invalid json")

        return conf_json

2022-10-07 07:11:42,257 - INFO - request type:2, len:1792
2022-10-07 07:11:42,260 - ERROR - execute plugin `netsuite-authenticator` AnyError, ('Configuration is invalid json',)
2022-10-07 07:11:42,260 - INFO - response type:2, len:16

However apisix ignores the plugin and continues with the request to the upstream. This same occurs even if the exception is thrown in the filter() method.

I would prefer the ability to control this behaviour and I was expecting allow_degredation in apisix config to control the behaviour, but it seems this only apply to the plugin runner process itself, not errors from the plugin.

In some cases I also see the following errors immediately after the starting the plugin in dev mode due to the config token caching mechanism.

2022-10-07 07:37:13,104 - INFO - request type:2, len:1792
2022-10-07 07:37:13,106 - ERROR - token `1` cache acquisition failed
2022-10-07 07:37:13,106 - INFO - response type:2, len:16

Again, apisix ignores the plugin and continues with the request to the upstream, but I would rather it didn't.

I believe this is happening because the plugin runner is always returning a successful RPC response, but the "action" in the response isn't set to either stop or rewrite, so apisix just assumes the plugin didn't want to do anything.

bug: phase_func(): failed to connect to the unix socket unix:/usr/local/apisix/conf/apisix-1.sock: permission denied

Current Behavior

I can't use a python plugin as a ext-plugin-post-req. When I deactivate the plugin,the request reaches the upstream service.
Here are the processes in the container

ps 
PID   USER     TIME  COMMAND
    1 root      0:00 {openresty} nginx: master process /usr/local/openresty/bin/openresty -p /usr/local/apisix -g daemon off;
   46 nobody    0:00 {openresty} nginx: worker process
   .....
   75 nobody    0:00 {openresty} nginx: worker process
   76 nobody    0:00 {openresty} nginx: worker process
   77 nobody    0:00 {openresty} nginx: worker process
   78 nobody    0:00 {openresty} nginx: cache manager process
   80 root      0:00 {openresty} nginx: privileged agent process
   81 root      0:00 python3 /usr/share/extras/py-apisix/py-runner start

It seems that both apisix and my plugin run as root so they should be able to communicate through /usr/local/apisix/conf/apisix-1.sock:

ls -la /usr/local/apisix/conf/apisix-1.sock
srwxr-xr-x    1 root     root             0 Sep 14 13:17 /usr/local/apisix/conf/apisix-1.sock

I just can't make sense of this behavior

Expected Behavior

Requests should reach the upstream service enriched with the headers defined in the plugin module

apisix-python-plugin-runner/apisix/plugins/rewrite.py

    def filter(self, conf: Any, request: Request, response: Response):
        """
        The plugin executes the main function
        :param conf:
            plugin configuration after parsing
        :param request:
            request parameters and information
        :param response:
            response parameters and information
        :return:
        """

        # print plugin configuration
        print(conf)

        # Fetch request nginx variable `host`
        host = request.get_var("host")
        print(host)

        # Fetch request body
        body = request.get_body()
        print(body)

        # Rewrite request headers
        request.set_header("X-Resp-A6-Runner", "Python")

        # Rewrite request args
        request.set_arg("a6_runner", "Python")

        # Rewrite request path
        request.set_uri("/a6/python/runner")

Error Logs

2022/09/14 13:47:01 [crit] 47#47: *795708 connect() to unix:/usr/local/apisix/conf/apisix-1.sock failed (13: Permission denied), client: 192.168.1.139, server: _, request: "GET /mock/ HTTP/1.1", host: "localhost:32674"
2022/09/14 13:47:01 [error] 47#47: *795708 [lua] init.lua:726: phase_func(): failed to connect to the unix socket unix:/usr/local/apisix/conf/apisix-1.sock: permission denied, client: 192.168.1.139, server: _, request: "GET /mock/ HTTP/1.1", host: "localhost:32674"
2022/09/14 13:47:01 [warn] 47#47: *795708 [lua] plugin.lua:759: run_plugin(): ext-plugin-post-req exits with http status code 503, client: 192.168.1.139, server: _, request: "GET /mock/ HTTP/1.1", host: "localhost:32674"

Steps to Reproduce

1. Deploy the chart with the following values.yaml

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apisix:
  # Enable or disable Apache APISIX itself
  # Set it to false and ingress-controller.enabled=true will deploy only ingress-controller
  enabled: true

  admin_key:
    - name: admin
      key: edd1c9f034335f136f87ad84b625c8f1  # using fixed API token has security risk, please update it when you deploy to production environment
      role: admin

  # Enable nginx IPv6 resolver
  enableIPv6: true

  # Use Pod metadata.uid as the APISIX id.
  setIDFromPodUID: false

  customLuaSharedDicts: []
    # - name: foo
    #   size: 10k
    # - name: bar
    #   size: 1m
  luaModuleHook:
    enabled: false
    # extend lua_package_path to load third party code
    luaPath: ""
    # the hook module which will be used to inject third party code into APISIX
    # use the lua require style like: "module.say_hello"
    hookPoint: ""
    # configmap that stores the codes
    configMapRef:
      name: ""
      # mounts decides how to mount the codes to the container.
      mounts:
        - key: ""
          path: ""

  enableCustomizedConfig: false
  customizedConfig: {}

  image:
    repository: apisix/apisix #apache/apisix
    pullPolicy: Always
    # Overrides the image tag whose default is the chart appVersion.
    tag: 2.13.3-alpine
    
  # Use a `DaemonSet` or `Deployment`
  kind: Deployment
  # kind is DaemonSet, replicaCount not become effective
  replicaCount: 1

  podAnnotations: {}
  podSecurityContext: {}
    # fsGroup: 2000
  securityContext: {}
    # capabilities:
    #   drop:
    #   - ALL
    # readOnlyRootFilesystem: true
    # runAsNonRoot: true
    # runAsUser: 1000

  # See https://kubernetes.io/docs/tasks/run-application/configure-pdb/ for more details
  podDisruptionBudget:
    enabled: false
    minAvailable: 90%
    maxUnavailable: 1

  resources: {}
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    # limits:
    #   cpu: 100m
    #   memory: 128Mi
    # requests:
    #   cpu: 100m
    #   memory: 128Mi
  hostNetwork: false

  nodeSelector: {}
  tolerations: []
  affinity: {}
  # If true, it will sets the anti-affinity of the Pod.
  podAntiAffinity:
    enabled: false

  # timezone is the timezone where apisix uses.
  # For example: "UTC" or "Asia/Shanghai"
  # This value will be set on apisix container's environment variable TZ.
  # You may need to set the timezone to be consistent with your local time zone,
  # otherwise the apisix's logs may used to retrieve event maybe in wrong timezone.
  timezone: ""

  # extraEnvVars An array to add extra env vars
  # e.g:
  # extraEnvVars:
  #   - name: FOO
  #     value: "bar"
  #   - name: FOO2
  #     valueFrom:
  #       secretKeyRef:
  #         name: SECRET_NAME
  #         key: KEY
  extraEnvVars: []

nameOverride: ""
fullnameOverride: ""

gateway:
  type: NodePort
  # If you want to keep the client source IP, you can set this to Local.
  # ref: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
  externalTrafficPolicy: Cluster
  # type: LoadBalancer
  # annotations:
  #   service.beta.kubernetes.io/aws-load-balancer-type: nlb
  externalIPs: []
  http:
    enabled: true
    servicePort: 80
    containerPort: 9080
  tls:
    enabled: false
    servicePort: 443
    containerPort: 9443
    existingCASecret: ""
    certCAFilename: ""
    http2:
      enabled: true
    sslProtocols: "TLSv1.2 TLSv1.3"
  # L4 proxy (TCP/UDP)
  stream:
    enabled: false
    only: false
    tcp: []
    udp: []
  ingress:
    enabled: false
    annotations: {}
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
    hosts:
      - host: apisix.local
        paths: []
    tls: []
  #  - secretName: apisix-tls
  #    hosts:
  #      - chart-example.local

admin:
  # Enable Admin API
  enabled: true
  # admin service type
  type: ClusterIP
  # loadBalancerIP: a.b.c.d
  # loadBalancerSourceRanges:
  #   - "143.231.0.0/16"
  externalIPs: []
  #
  port: 9180
  servicePort: 9180
  # Admin API support CORS response headers
  cors: true
  # Admin API credentials
  credentials:
    admin: edd1c9f034335f136f87ad84b625c8f1
    viewer: 4054f7cf07e344346cd3f287985e76a2

  allow:
    # The ip range for allowing access to Apache APISIX
    ipList:
      - 127.0.0.1/24

nginx:
  workerRlimitNofile: "20480"
  workerConnections: "10620"
  workerProcesses: auto
  enableCPUAffinity: true

# APISIX plugins to be enabled
plugins:
  - api-breaker
  - authz-keycloak
  - basic-auth
  - batch-requests
  - consumer-restriction
  - cors
  - echo
  - fault-injection
  - grpc-transcode
  - hmac-auth
  - http-logger
  - ip-restriction
  - ua-restriction
  - jwt-auth
#  - kafka-logger
  - key-auth
  - limit-conn
  - limit-count
  - limit-req
  - node-status
  - openid-connect
  - authz-casbin
  - prometheus
  - proxy-cache
  - proxy-mirror
  - proxy-rewrite
  - redirect
  - referer-restriction
  - request-id
  - request-validation
  - response-rewrite
  - serverless-post-function
  - serverless-pre-function
  - sls-logger
  - syslog
  - tcp-logger
  - udp-logger
  - uri-blocker
  - wolf-rbac
  - zipkin
  - traffic-split
  - gzip
  - real-ip
  - ext-plugin-pre-req
  - ext-plugin-post-req
stream_plugins:
  - mqtt-proxy
  - ip-restriction
  - limit-conn

pluginAttrs:  {}

extPlugin:
  enabled: true
  cmd: ["python3","/usr/share/extras/py-apisix/py-runner", "start"]

# customPlugins allows you to mount your own HTTP plugins.
customPlugins:
  enabled: false
  # the lua_path that tells APISIX where it can find plugins,
  # note the last ';' is required.
  luaPath: "/opts/custom_plugins/?.lua"
  plugins:
    # plugin name.
    - name: ""
      # plugin attrs
      attrs: {}
      # plugin codes can be saved inside configmap object.
      configMap:
        # name of configmap.
        name: ""
        # since keys in configmap is flat, mountPath allows to define the mount
        # path, so that plugin codes can be mounted hierarchically.
        mounts:
          - key: ""
            path: ""
          - key: ""
            path: ""

updateStrategy: {}
  # type: RollingUpdate

extraVolumes: 
  - name: apisix-lib
    hostPath:
      path: /home/zar3bski/Documents/Code/octaave/test_files/apisix-python-plugin-runner/apisix
  - name: apisix-bin
    hostPath:
      path: /home/zar3bski/Documents/Code/octaave/test_files/apisix-python-plugin-runner/bin
  - name: apisix-conf
    hostPath:
      path: /home/zar3bski/Documents/Code/octaave/test_files/apisix-python-plugin-runner/conf

extraVolumeMounts:
  - name: apisix-lib
    mountPath: /usr/lib/python3.9/site-packages/apisix
    readOnly: true
  - name: apisix-bin
    mountPath: /usr/share/extras/py-apisix
    readOnly: true
  - name: apisix-conf
    mountPath: /usr/share/extras/conf
    readOnly: true

discovery:
  enabled: false
  registry:
    # Integration service discovery registry. E.g eureka\dns\nacos\consul_kv
    # reference:
    # https://apisix.apache.org/docs/apisix/discovery#configuration-for-eureka
    # https://apisix.apache.org/docs/apisix/discovery/dns#service-discovery-via-dns
    # https://apisix.apache.org/docs/apisix/discovery/consul_kv#configuration-for-consul-kv
    # https://apisix.apache.org/docs/apisix/discovery/nacos#configuration-for-nacos
    #
    # an eureka example:
    # eureka:
    #   host:
    #     - "http://${username}:${password}@${eureka_host1}:${eureka_port1}"
    #     - "http://${username}:${password}@${eureka_host2}:${eureka_port2}"
    #   prefix: "/eureka/"
    #   fetch_interval: 30
    #   weight: 100
    #   timeout:
    #     connect: 2000
    #     send: 2000
    #     read: 5000

# access log and error log configuration
logs:
  enableAccessLog: true
  accessLog: "/dev/stdout"
  accessLogFormat: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'
  accessLogFormatEscape: default
  errorLog: "/dev/stderr"
  errorLogLevel: "warn"

dns:
  resolvers:
    - 127.0.0.1
    - 172.20.0.10
    - 114.114.114.114
    - 223.5.5.5
    - 1.1.1.1
    - 8.8.8.8
  validity: 30
  timeout: 5

initContainer:
  image: busybox
  tag: 1.28

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80

# Custom configuration snippet.
configurationSnippet:
  main: |
  httpStart: |
  httpEnd: |
  httpSrv: |
  httpAdmin: |
  stream: |
# Observability configuration.
# ref: https://apisix.apache.org/docs/apisix/plugins/prometheus/
serviceMonitor:
  enabled: false
  # namespace where the serviceMonitor is deployed, by default, it is the same as the namespace of the apisix
  namespace: ""
  # name of the serviceMonitor, by default, it is the same as the apisix fullname
  name: ""
  # interval at which metrics should be scraped
  interval: 15sfalse
  metricPrefix: apisix_
  # container port where the metrics are exposed
  containerPort: 9091
  # @param serviceMonitor.labels ServiceMonitor extra labels
  labels: {}
  # @param serviceMonitor.annotations ServiceMonitor annotations
  annotations: {}

# etcd configuration
# use the FQDN address or the IP of the etcd
etcd:
  # install etcd(v3) by default, set false if do not want to install etcd(v3) together
  enabled: true
  host:
    # host or ip e.g. http://172.20.128.89:2379
    - http://etcd.host:2379
  prefix: "/apisix"
  timeout: 30

  # if etcd.enabled is true, set more values of bitnami/etcd helm chart
  auth:
    rbac:
      # No authentication by default
      create: false
      user: ""
      password: ""
    tls:
      enabled: false
      existingSecret: ""
      certFilename: ""
      certKeyFilename: ""
      verify: true
      sni: ""

  service:
    port: 2379

  replicaCount: 3

dashboard:
  enabled: true
  replicaCount: 1

  image:
    repository: apache/apisix-dashboard
    pullPolicy: IfNotPresent
    # Overrides the image tag whose default is the chart appVersion.
    tag: 2.13-alpine

  imagePullSecrets: []
  nameOverride: ""
  fullnameOverride: ""

  serviceAccount:
    # Specifies whether a service account should be created
    create: true
    # Annotations to add to the service account
    annotations: {}
    # The name of the service account to use.
    # If not set and create is true, a name is generated using the fullname template
    name: ""

  podAnnotations: {}

  podSecurityContext: {}
    # fsGroup: 2000

  securityContext: {}
    # capabilities:
    #   drop:
    #   - ALL
    # readOnlyRootFilesystem: true
    # runAsNonRoot: true
    # runAsUser: 1000

  config:
    conf:
      listen:
        host: 0.0.0.0
        port: 9000
      etcd:
        # Supports defining multiple etcd host addresses for an etcd cluster
        endpoints:
          - apisix-etcd:2379
        # apisix configurations prefix
        prefix: "/apisix"
        # Etcd basic auth info
        username: ~
        password: ~
      log:
        errorLog:
          level: warn
          filePath: /dev/stderr
        accessLog:
          filePath: /dev/stdout

    authentication:
      secret: secret
      expireTime: 3600
      users:
        - username: admin
          password: admin

  service:
    type: ClusterIP
    port: 80

    ingress:
      enabled: false
      # Kubernetes 1.18+ support ingressClassName attribute
      className: ""
      annotations: {}
        # kubernetes.io/ingress.class: nginx
        # kubernetes.io/tls-acme: "true"
      # domain access apisix example:
      # hosts:
      # - host: apisix-dashboard.local
      #    paths:
      #     - /*
      hosts:
        - host: apisix-dashboard.local
          paths: []
      tls: []
      #  - secretName: chart-example-tls
      #    hosts:
      #      - chart-example.local

    resources: {}
      # We usually recommend not to specify default resources and to leave this as a conscious
      # choice for the user. This also increases chances charts run on environments with little
      # resources, such as Minikube. If you do want to specify resources, uncomment the following
      # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
      # limits:
      #   cpu: 100m
      #   memory: 128Mi
      # requests:
      #   cpu: 100m
      #   memory: 128Mi

    autoscaling:
      enabled: false
      minReplicas: 1
      maxReplicas: 100
      targetCPUUtilizationPercentage: 80
      # targetMemoryUtilizationPercentage: 80

    nodeSelector: {}

    tolerations: []

    affinity: {}
  
ingress-controller:
  enabled: false

2. create a route

{
  "uri": "/mock/*",
  "name": "mock",
  "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
  ],
  "plugins": {
    "ext-plugin-post-req": {
      "conf": [
        {
          "name": "rewrite",
          "value": "{\"enable\":\"feature\"}"
        }
      ],
      "disable": false
    },
    "ext-plugin-pre-req": {
      "disable": true
    }
  },
  "upstream": {
    "nodes": [
      {
        "host": "172.17.0.1",
        "port": 8081,
        "weight": 1
      }
    ],
    "timeout": {
      "connect": 6,
      "send": 6,
      "read": 6
    },
    "type": "roundrobin",
    "scheme": "http",
    "pass_host": "pass",
    "keepalive_pool": {
      "idle_timeout": 60,
      "requests": 1000,
      "size": 320
    }
  },
  "status": 1
}

3. query the route curl http://localhost:${mapped port of apisix gateway}/mock/

Environment

• APISIX version (run apisix version): 2.13.3
• Operating system (run uname -a): Linux apisix-6878f46cb5-hps57 5.15.0-47-generic apache/apisix#51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 Linux (????? The image is supposed to be Alpine based)
• OpenResty / Nginx version (run openresty -V or nginx -V): 1.21.4.1
• Plugin runner version, for issues related to plugin runners: python plugin v 0.2.0
• LuaRocks version, for installation issues (run luarocks --version):

自定义插件不能使用

2022/07/07 08:03:56 [warn] 43#43: *23 [lua] init.lua:913: 2022-07-07 08:03:56,810 - ERROR - execute plugin stop TypeError, ("'NoneType' object is not callable",)
, context: ngx.timer
2022/07/07 08:05:16 [warn] 43#43: *23 [lua] init.lua:913: 2022-07-07 08:05:16,691 - ERROR - execute plugin stop TypeError, ("'NoneType' object is not callable",)

Report test coverage and make sure it is high enough

Something like https://app.codecov.io/gh/apache/apisix-go-plugin-runner

doc: missing readme

we need to write the first version. this is very important for the community.

If the documentation "readme" is missing, the community will not be able to join.

How can i get response body from this plugin ?

I want to get the content of response, but it doesn't seem to support it at present

request set_body won't change

even call request.set_body, the body won't pass to upstream.

file upload error

Route - Plugin Config -> ext-plugin-pre-req enable

{
"_meta": {
"disable": false
},
"conf": [
{
"name": "rewrite",
"value": "{"body":"hello"}"
}
]
}

when Plugin enabled , file upload show error: 503 Service Temporarily Unavailable

if file size less than 220kb , upload show success, large than 220kb show error

error log

context: ngx.timer
2023/03/17 14:23:05 [warn] 10354#10354: *113 [lua] init.lua:955:
Exception in thread Thread-14:
Traceback (most recent call last):
File "/usr/local/python3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/python3.7/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/server/server.py", line 60, in _threaded
response = handler.dispatch()
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/server/handle.py", line 83, in dispatch
ok = runner_plugin.execute(configs, self.r, req, resp)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/plugin/core.py", line 99, in execute
raise e
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/plugin/core.py", line 91, in execute
p.filter(conf, req, reps)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/plugins/rewrite.py", line 61, in filter
body = request.get_body()
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/http/request.py", line 144, in get_body
val = self.__ask_extra_info(builder, EIInfo.Info.ReqBody, body_req_data)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/http/request.py", line 193, in __ask_extra_info
vector = resp.Result(i)
File "/usr/local/python3.7/lib/python3.7/site-packages/A6/ExtraInfo/Resp.py", line 32, in Result
return self._tab.Get(flatbuffers.number_types.Uint8Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 1))
File "/usr/local/python3.7/lib/python3.7/site-packages/flatbuffers/table.py", line 93, in Get
return flags.py_type(encode.Get(flags.packer_type, self.Bytes, off))
File "/usr/local/python3.7/lib/python3.7/site-packages/flatbuffers/encode.py", line 26, in Get
return packer_type.unpack_from(memoryview_type(buf), head)[0]
struct.error: unpack_from requires a buffer of at least 1 bytes

Exception in thread Thread-15:
Traceback (most recent call last):
File "/usr/local/python3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/python3.7/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/server/server.py", line 60, in _threaded
response = handler.dispatch()
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/server/handle.py", line 83, in dispatch
ok = runner_plugin.execute(configs, self.r, req, resp)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/plugin/core.py", line 99, in execute
raise e
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/plugin/core.py", line 91, in execute
p.filter(conf, req, reps)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/plugins/rewrite.py", line 61, in filter
body = request.get_body()
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/http/request.py", line 144, in get_body
val = self.__ask_extra_info(builder, EIInfo.Info.ReqBody, body_req_data)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/http/request.py", line 193, in __ask_extra_info
vector = resp.Result(i)
File "/usr/local/python3.7/lib/python3.7/site-packages/A6/ExtraInfo/Resp.py", line 32, in Resu, context: ngx.timer
2023/03/17 14:23:05 [warn] 10354#10354: *113 [lua] init.lua:955: lt
return self._tab.Get(flatbuffers.number_types.Uint8Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 1))
File "/usr/local/python3.7/lib/python3.7/site-packages/flatbuffers/table.py", line 93, in Get
return flags.py_type(encode.Get(flags.packer_type, self.Bytes, off))
F, context: ngx.timer
2023/03/17 14:23:05 [warn] 10354#10354: *113 [lua] init.lua:955: ile "/usr/local/python3.7/lib/python3.7/site-packages/flatbuffers/encode.py", line 26, in Get
return packer_type.unpack_from(memoryview_type(buf), head)[0]
struct.error: unpack_from requires a buffer of at least 1 bytes

Exception in thread Thread-16:
Traceback (most recent call last):
File "/usr/local/python3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/python3.7/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/server/server.py", line 60, in _threaded
response = handler.dispatch()
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/server/handle.py", line 83, in dispatch
ok = runner_plugin.execute(configs, self.r, req, resp)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/plugin/core.py", line 99, in execute
raise e
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/plugin/core.py", line 91, in execute
p.filter(conf, req, reps)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/plugins/rewrite.py", line 61, in filter
body = request.get_body()
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/http/request.py", line 144, in get_body
val = self.__ask_extra_info(builder, EIInfo.Info.ReqBody, body_req_data)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/http/request.py", line 193, in __ask_extra_info
vector = resp.Result(i)
File "/usr/local/python3.7/lib/python3.7/site-packages/A6/ExtraInfo/Resp.py", line 32, in Result
return self._tab.Get(flatbuffers.number_types.Uint8Flags, a + flatbuffers.number_types.UOffsetTFlags.py_type(j * 1))
File "/usr/local/python3.7/lib/python3.7/site-packages/flatbuffers/table.py", line 93, in Get
return flags.py_type(encode.Get(flags.packer_type, self.Bytes, off))
File "/usr/local/python3.7/lib/python3.7/site-packages/flatbuffers/encode.py", line 26, in Get
return packer_type.unpack_from(memoryview_type(buf), head)[0]
struct.error: unpack_from requires a buffer of at least 1 bytes

Exception in thread Thread-23:
Traceback (most recent call last):
File "/usr/local/python3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/python3.7/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/server/server.py", line 60, in _threaded
response = handler.dispatch()
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/server/handle.py", line 83, in dispatch
ok = runner_plugin.execute(configs, self.r, req, resp)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/plugin/core.py", line 99, in execute
raise e
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/plugin/core.py", line 91, in execute
p.filter(conf, req, reps)
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/plugins/rewrite.py", line 61, in filter
body = request.get_body()
File "/usr/local/python3.7/lib/python3.7/site-packages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/http/request.py", line 144, in get_body
val = self.__ask_extra_info(builder, EIInfo.Info.ReqBody, body_req_data)
File "/usr/local/python3.7/lib/python3.7/site-p, context: ngx.timer
2023/03/17 14:23:05 [warn] 10354#10354: *113 [lua] init.lua:955: ackages/apache_apisix-0.2.0-py3.7.egg/apisix/runner/http/request.py", line 193, in __ask_extra_info
vector = resp.Result(i)
File "/usr/local/python3.7/lib/python3.7/site-packages/A6/ExtraInfo/Resp.py", line 32, in Result
return self._tab.Get(flatbuffers.number_types.Uint8Flags, a + fla, context: ngx.timer
2023/03/17 14:23:05 [warn] 10354#10354: *113 [lua] init.lua:955: tbuffers.number_types.UOffsetTFlags.py_type(j * 1))
File "/usr/local/python3.7/lib/python3.7/site-packages/flatbuffers/table.py", line 93, in Get
return flags.py_type(encode.Get(flags.packer_type, self.Bytes, off))
File "/usr/local/python3.7/lib/python3.7/site-packages/flatbuffers/encode.py", line 26, in Get
return packer_type.unpack_from(memoryview_type(buf), head)[0]
struct.error: unpack_from requires a buffer of at least 1 bytes
, context: ngx.timer
2023/03/17 14:23:05 [error] 10351#10351: *125273 send() failed (32: Broken pipe), client: 192.168.100.22, server: _, request: "POST /common/oss/upload HTTP/1.0", host: "dsj.com:9443", referrer: "http://dsj.com:9443/"

New release?

Hi, I'm running into issue #53 which should be fixed by #55 but it's not released yet.

Can a new release be made please or an idea of its ETA?

Thanks!

Using Python Plugin Runner in a Kubernetes Deployment - no Python3 in the container

We need to add an external plugin to decode some inbound headers and would like to use the python runner

The plugin requires Python 3.7+

Our intention is to mount the runner and plugin as an external volumes/mounts and point the extPlugin config to those locations

all looks good, except for one small thing

The apisix image is Debian Bullseye which does NOT have Python installed on it at all.

Now, we can pretty much build python via an init container, however, we do not have sudo privileges or su - privileges in hat container so we cannot install the build dependencies which require sudo apt root privileges

We really do not want to have to build and maintain apisix form source ..

Any ideas/suggestions most welcome on how we could get python into the container.

support plugin auto register to runner

Does runner work with ext-plugin-post-resp?

Keep getting error:

[warn] 48#48: *256 [lua] init.lua:913: ERROR - unknown request

So, I want to verify that this runner works with ext-plugin-post-resp?
I don't get this error when using ext-plugin-pre-req or ext-plugin-post-req.

when forwarding tff files, the plugin reports an error

I wrote a custom plugin to control the permission of my application deployed in k8s.
The upstream is a code-server web application.
It can be accessed normally under nginx-ingress.
When I switched from ingress to apisix, most requests can be proxied normally, except for one ttf request like:

http://a1.webide.demo.test/static/out/vs/base/browser/ui/codicons/codicon/codicon.ttf?5d4d76ab2ce5108968ad644d591a16a6
This request returned 503.

Then I looked at the log of apisix, I saw the following error messages：

2022/03/18 13:26:32 [error] 44#44: *295957 [lua] init.lua:703: phase_func(): failed to receive RPC_HTTP_REQ_CALL: timeout, client: 192.168.1.64, server: _, request: "GET /static/out/vs/base/browser/ui/codicons/codicon/codicon.ttf?5d4d76ab2ce5108968ad644d591a16a6 HTTP/1.1", host: "a1.webide.demo.test"
2022/03/18 13:26:32 [warn] 44#44: *295957 [lua] plugin.lua:724: run_plugin(): ext-plugin-pre-req exits with http status code 503, client: 192.168.1.64, server: _, request: "GET /static/out/vs/base/browser/ui/codicons/codicon/codicon.ttf?5d4d76ab2ce5108968ad644d591a16a6 HTTP/1.1", host: "a1.webide.demo.test"
192.168.1.64 - - [18/Mar/2022:13:26:32 +0000] a1.webide.demo.test "GET /static/out/vs/base/browser/ui/codicons/codicon/codicon.ttf?5d4d76ab2ce5108968ad644d591a16a6 HTTP/1.1" 503 596 60.004 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.74 Safari/537.36" - - - "http://a1.webide.demo.test"
192.168.1.64 - - [18/Mar/2022:13:26:33 +0000] a1.webide.demo.test "GET /favicon.ico HTTP/1.1" 304 0 0.308 "http://a1.webide.demo.test/static/out/vs/base/browser/ui/codicons/codicon/codicon.ttf?5d4d76ab2ce5108968ad644d591a16a6" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.74 Safari/537.36" 10.7.230.250:9999 304 0.012 "http://a1.webide.demo.test/favicon.ico"
2022/03/18 13:27:23 [warn] 54#54: *351 [lua] init.lua:775: Exception in thread Thread-44:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
, context: ngx.timer
2022/03/18 13:27:23 [warn] 54#54: *351 [lua] init.lua:775:     self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.9/site-packages/apache_apisix-0.2.0-py3.9.egg/apisix/runner/server/server.py", line 60, in _threaded
    response = handler.dispatch()
  File "/usr/lib/python3.9/site-packages/apache_apisix-0.2.0-py3.9.egg/apisix/runner/server/handle.py", line 41, in dispatch
    req = NewHttpRequest(self.r)
  File "/usr/lib/python3.9/site-packages/apache_apisix-0.2.0-py3.9.egg/apisix/runner/http/request.py", line 63, in __init__
    self.__init()
  File "/usr/lib/python3.9/site-packages/apache_apisix-0.2.0-py3.9.egg/apisix/runner/http/request.py", line 398, in __init
, context: ngx.timer
2022/03/18 13:27:23 [warn] 54#54: *351 [lua] init.lua:775:     arg_dict = runner_utils.parse_dict_vector(req, runner_utils.VECTOR_TYPE_QUERY)
, context: ngx.timer
2022/03/18 13:27:23 [warn] 54#54: *351 [lua] init.lua:775:   File "/usr/lib/python3.9/site-packages/apache_apisix-0.2.0-py3.9.egg/apisix/runner/utils/common.py", line 228, in parse_dict_vector
, context: ngx.timer
2022/03/18 13:27:23 [warn] 54#54: *351 [lua] init.lua:775:     val = getattr(cls, fn)(i).Value().decode()
AttributeError: 'NoneType' object has no attribute 'decode'
, context: ngx.timer

It seems that some part of the plugin that I haven't changed has blocked the TTF request. Maybe bugs？
How to solve this problem？

Attach my apisixroute：

apiVersion: apisix.apache.org/v2beta2
kind: ApisixRoute
metadata:
  name: webide-test
  namespace: dev-3woklfan
spec:
  http:
  - name: rule1
    match:
      hosts:
      - a1.webide.demo.test
      paths:
      - /*
    backends:
    - serviceName: webide-ad5c19-ad5c19-svc
      servicePort: 30000
    websocket: true
    plugins:
    - name: ext-plugin-pre-req
      enable: true
      config:
        conf:
        - name: "authorization"
          value: "{\"test\": \"test\"}"

Is this repo being maintained?

guide doc error

file: apisix-python-plugin-runner/docs/en/latest/getting-started.md
line: 100
content: curl http://127.0.0.1:**9080**/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
the pot is wrong, right port must be 9180.

The CI on master branch is broken

bug: `make setup` error on clean ubuntu server caused by deprecated config in `setup.cfg`.

Current Behavior

Getting requirements to build wheel did not run successfully caused by deprecated config in setup.cfg.

reproduce this error with

$ git clone https://github.com/apache/apisix-python-plugin-runner.git
$ cd apisix-python-plugin-runner
$ make setup
$ make install

Error Logs

Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [62 lines of output]
/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in setup.cfg
!!

          ********************************************************************************
          The license_file parameter is deprecated, use license_files instead.
  
          This deprecation is overdue, please update your project and remove deprecated
          calls to avoid build errors in the future.
  
          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************
  
  !!
    parsed = self.parsers.get(option_name, lambda x: x)(value)
  running egg_info
  writing lib3/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib3/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 325, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 341, in run_setup
      exec(code, locals())
    File "<string>", line 271, in <module>
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 989, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 318, in run
      self.find_sources()
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 326, in find_sources
      mm.run()
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 548, in run
      self.add_defaults()
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 586, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/command/sdist.py", line 113, in add_defaults
      super().add_defaults()
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "<string>", line 201, in get_source_files
    File "/tmp/pip-build-env-1kmna_x7/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
make: *** [Makefile:23: setup] Error 1

Can asyncio improve the performance?

How can i get request body from this plugin ?

When I need to get the body in the request body, whether form or JSON, instead of using the URL param, how can I get it from the request in the plugin

Example failed on mac

I followed https://apisix.apache.org/docs/python-plugin-runner/getting-started/,
apisix version is 2.10.
cloned python-plugin-runner from git.

When I call curl http://127.0.0.1:9080/get -i, the result is 503, and the log of apisix is

2021/10/22 09:41:33 [warn] 44#44: *2 [lua] plugin.lua:192: load_stream(): new plugins: {"limit-conn":true,"ip-restriction":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2021/10/22 09:41:33 [warn] 45#45: *1 [lua] plugin.lua:192: load_stream(): new plugins: {"limit-conn":true,"ip-restriction":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2021/10/22 09:41:33 [warn] 49#49: *6 [lua] plugin.lua:192: load_stream(): new plugins: {"limit-conn":true,"ip-restriction":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2021/10/22 09:41:33 [warn] 46#46: *5 [lua] plugin.lua:192: load_stream(): new plugins: {"limit-conn":true,"ip-restriction":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2021/10/22 09:41:33 [warn] 47#47: *4 [lua] plugin.lua:192: load_stream(): new plugins: {"limit-conn":true,"ip-restriction":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2021/10/22 09:41:33 [warn] 48#48: *3 [lua] plugin.lua:192: load_stream(): new plugins: {"limit-conn":true,"ip-restriction":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2021/10/22 09:41:33 [warn] 52#52: *7 [lua] plugin.lua:192: load_stream(): new plugins: {"limit-conn":true,"ip-restriction":true,"mqtt-proxy":true}, context: init_worker_by_lua*
2021/10/22 10:11:36 [crit] 48#48: *164901 connect() to unix:/tmp/runner.sock failed (2: No such file or directory), client: 172.25.0.1, server: _, request: "GET /get HTTP/1.1", host: "127.0.0.1:9080"
2021/10/22 10:11:36 [error] 48#48: *164901 [lua] init.lua:617: phase_func(): failed to connect to the unix socket unix:/tmp/runner.sock: no such file or directory, client: 172.25.0.1, server: _, request: "GET /get HTTP/1.1", host: "127.0.0.1:9080"
2021/10/22 10:11:36 [warn] 48#48: *164901 [lua] plugin.lua:680: run_plugin(): ext-plugin-pre-req exits with http status code 503, client: 172.25.0.1, server: _, request: "GET /get HTTP/1.1", host: "127.0.0.1:9080"

Add License checker

For example:
https://github.com/apache/apisix-go-plugin-runner/blob/master/.github/workflows/license-checker.yml
https://github.com/apache/apisix-go-plugin-runner/blob/master/.licenserc.yaml