openclarity/apiclarity

An API security tool to capture and analyze API traffic, test API endpoints, reconstruct Open API specification, and identify API security risks. 

APIClarity

APIClarity is a modular tool that addresses several aspects of API Security, focusing specifically on OpenAPI based APIs.

APIClarity approaches API Security in 2 different ways:

• Captures all API traffic in a given environment and performs a set of security analysis to discover all potential security problems with detected APIs
• Actively tests API endpoints to detect security issues in the implementation of such APIs.

OpenAPI automatic reconstruction

Both approaches described above are way more effective when APIClarity is primed with the OpenAPI specifications of the APIs analyzed or tested. However, not all applications have an OpenAPI specification available. For this reason one of the main functionality of APIClarity is the automatic reconstruction of OpenAPI specifications based on observed API traffic. In this case, users have the ability to review and approve the reconstructed specifications.

Security Modules

APIClarity is structured in a modular architecture, which allows to easily add new functionalities.

In the following a brief description of the modules currently implemented:

• Spec Diffs This module compares the API traces with the OAPI specifications provided by the user or previously reconstructed. The result of this comparison provides:
  • List of API endpoints that are observed but not documented in the specs, i.e. Shadow APIs;
  • List of API endpoints that are observed but marked as deprecated in the specs, i.e. Zombie APIs;
  • List of difference between of the APIs observed and their documented specification.
• Trace Analyzer This module analyzes path, headers and body of API requests and responses to discover potential security issues, such as weak authentications, exposure of sensitive information, potential Broken Object Level Authorizations (BOLA) etc.
• BFLA Detector This module detects potential Broken Function Level Authorization. In particular it observes the API interactions and build an authorization model that captures what clients are supposed to be authorized to make the various API calls. Based on such authorization model it then signals violations which may represent potential issues in the API authorization procedures.
• Fuzzer This module actively tests API endpoints based on their specification attempting in discovering security issues in the API server implementation.

High level architecture

Getting started

Supported traffic source integrations

APIClarity supports integrating with the following traffic sources. Install APIClarity and follow the instructions per required integration.

• Istio Service Mesh

  • Make sure that Istio 1.10+ is installed and running in your cluster. See the Official installation instructions for more information.

• Tap via a DaemonSet

  • Integration instructions

• Kong API Gateway

  • Integration instructions

• Tyk API Gateway

  • Integration instructions

• OpenTelemetry Collector (traces only)

  • Integration instructions

The integrations (plugins) for the supported traffic sources above are located in the plugins directory within the codebase and implement the plugins API to export the API events to APIClarity. To enable and configure the supported traffic sources, please check the trafficSource: section in Helm values. Contributions of integrations with additional traffic sources are more than welcome!

Install APIClarity in a K8s cluster using Helm:

1. Add Helm repo

   helm repo add apiclarity https://openclarity.github.io/apiclarity

2. Save APIClarity default chart values

   helm show values apiclarity/apiclarity > values.yaml

3. Update values.yaml with the required traffic source values

4. Deploy APIClarity with Helm for the selected traffic source

   helm install --values values.yaml --create-namespace apiclarity apiclarity/apiclarity -n apiclarity

5. Port forward to APIClarity UI:

   kubectl port-forward -n apiclarity svc/apiclarity-apiclarity 9999:8080

6. Open APIClarity UI in the browser: http://localhost:9999/

7. Generate some traffic in the traced applications and check the APIClarity UI :)

Uninstall APIClarity from a K8s cluster using Helm:

1. Helm uninstall

   helm uninstall apiclarity -n apiclarity

2. Clean resources

   By default, Helm will not remove the PVCs and PVs for the StatefulSets. Run the following command to delete them all:

   kubectl delete pvc -l app.kubernetes.io/instance=apiclarity -n apiclarity

Configurations

The file values.yaml is used to deploy and configure APIClarity on your cluster via Helm. This ConfigMap is used to define the list of headers to ignore when reconstructing the spec.

Testing with a demo application

A good demo application to try APIClarity with is the Sock Shop Demo.

To deploy the Sock Shop Demo, follow these steps:

1. Create the sock-shop namespace and enable Istio injection:

   kubectl create namespace sock-shop
   kubectl label namespaces sock-shop istio-injection=enabled

2. Deploy the Sock Shop Demo to your cluster:

   kubectl apply -f https://raw.githubusercontent.com/microservices-demo/microservices-demo/master/deploy/kubernetes/complete-demo.yaml

3. Deploy APIClarity in the sock-shop namespace (e.g. Istio service-mesh traffic source):

   helm repo add apiclarity https://openclarity.github.io/apiclarity

   helm install --set 'trafficSource.envoyWasm.enabled=true' --set 'trafficSource.envoyWasm.namespaces={sock-shop}' --create-namespace apiclarity apiclarity/apiclarity -n apiclarity

4. Port forward to Sock Shop's front-end service to access the Sock Shop Demo App:

   kubectl port-forward -n sock-shop svc/front-end 7777:80

   Open the Sock Shop Demo App UI in the browser (http://localhost:7777/) and run some transactions to generate data to review on the APIClarity dashboard.

Building

Building from source:

Build and push the image to your repo:

DOCKER_IMAGE=<your docker registry>/apiclarity DOCKER_TAG=<your tag> make push-docker

Update values.yaml accordingly.

Running locally with demo data

1. Build UI & backend locally as described above:

   make ui && make backend

2. Copy the built site:

   cp -r ./ui/build ./site

3. Run backend and frontend locally using demo data:

   Note: You might need to delete the old local state file and local db:

   rm state.gob; rm db.db

   DATABASE_DRIVER=LOCAL K8S_LOCAL=true FAKE_TRACES=true FAKE_TRACES_PATH=./backend/pkg/test/trace_files \
   ENABLE_DB_INFO_LOGS=true ./backend/bin/backend run

   Note: this command requires a proper KUBECONFIG in your environment when K8S_LOCAL=true is used. If you want to run without k8s, use ENABLE_K8S=false instead.

4. Open APIClarity UI in the browser: http://localhost:8080/

Enabling External Trace Sources Support

Enabling external trace sources support, APIClarity can receive the trace sources from the entitites that are external to the K8s cluster. External trace sources such as Gateways, Load balancers, etc. can communicate with APIClarity to report APIs and send the traces.

The following section describes how to deploy APIClarity with the support for external trace sources

1. Add Helm Repo

helm repo add apiclarity https://openclarity.github.io/apiclarity

2. Update values.yaml with

Apiclarity -> tls -> enabled as true
supportExternalTraceSource -> enabled as true

3. Deploy APIClarity with the Helm enabling external traffic sources

helm install --values values.yaml --create-namespace apiclarity apiclarity/apiclarity -n apiclarity

4. Port forward to APIClarity UI:

kubectl port-forward -n apiclarity svc/apiclarity-apiclarity 9999:8080

5. Open APIClarity UI in the browser:

http://localhost:9999 

The following section describes how to register a new external trace source. And this section includes how to access the service, register a new trace source, and how to receive the token and certificate.

1. Port forward for service at 8443

kubectl port-forward -n apiclarity svc/apiclarity-apiclarity 8443:8443

2. Register a new external trace source and receive the token

TRACE_SOURCE_TOKEN=$(curl --http1.1 --insecure -s -H 'Content-Type: application/json' -d '{"name":"apigee_gateway","type":"APIGEE_X"}' https://localhost:8443/api/control/traceSources|jq -r '.token')

3. Receive the certificate To receive the certificate, get the External-IP for the service named as apiclarity-external

kubectl get services -n apiclarity

Then, use the External-IP address with the following command, then extract the certificate with -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- and save to server.crt

openssl s_client -showcerts -connect <External-IP>:10443

Use the above extracted token at the step-2 and certificate at step-3 for configuring subsequent external trace sources such as Apigee X Gateway and BIG-IP LTM Load balancer

Supported Trace Sources

APIClarity can support with the following trace sources and follow the instructions per required integration.

• Apigee X Gateway
  • Integration instructions
• BIG-IP LTM Load balancer
  • Integration instructions
• Kong
  • Integration instructions
• Tyk
  • Integration instructions

Contributing

Pull requests and bug reports are welcome.

For larger changes please create an Issue in GitHub first to discuss your proposed changes and possible implications.

Contributors

https://panoptica.app

License

Apache License, Version 2.0

apiclarity's Issues

Allow Acceptance or Update to Reconstructed Spec When Viewing Diff

Once an API Sepecification is reconstructed and accepted, a Diff may be detected.

When reviewing events in the events tab, diffs are identified. That diff may actually be a valid new use for that API as added by a developer to the API.

Solution: Please add the ability to incorporate that diff / method into the reconstructed API at that time with a button that says something like "Incorporate Diff into API Specification"

Dark Mode Support

Is your feature request related to a problem? Please describe.
I have been using dark mode for all the applications, as result of which I no longer can stick longer to tools which don't support it. It will be great to have APICalrity have it supported right from its initial stage so that it will generate more active users (me and other users who have similar problem) for this project.

Describe the solution you'd like

• Build Dark theme prototypes with micro UI components in mind
• Re design of colour choice of rgb(39, 68, 110) to something that match Light and Dark mode
• Status Code green is too small and light, need it bright and large (if possible same as k8s dashboard indicators)

Describe alternatives you've considered
None

Additional context
None

Gateway not configured for capturing the external traffic.

Trying to configure the APIclarity for capturing the external traffic from one of my sites but not able to get it done using any of the mentioned gateways like tyk, kong etc. Can anyone suggest or assist in doing the same?

API HTTP server does not rewrite UI URLs to work with react-router

What happened:

Display in web browser:

404 page not found

What you expected to happen:

Web browser displays the selected sub-page.

How to reproduce it (as minimally and precisely as possible):

1. kubectl port-forward -n apiclarity svc/apiclarity 9999:8080
2. Browse to http://localhost:9999/inventory

You can reach http://localhost:9999/inventory when going through http://localhost:9999/ first and click on the Inventory button in the sidebar. This problem affects only deeplinks.

Are there any error messages in API Clarity logs?

None relevant to this issue.

Anything else we need to know?:

This issue stems from the way the static UI files are served from the API server in https://github.com/apiclarity/apiclarity/blob/master/api/server/restapi/configure_api_clarity_a_p_is.go#L98

React Router expects all routes to be rewritten onto the index.html so that it can deal with the routing within the React application.

Possibly relevant discussion on Reddit: https://www.reddit.com/r/golang/comments/ffgn46/what_is_the_correct_way_to_serve_react_app_using/

Environment:

• Kubernetes version (use kubectl version --short)

  Client Version: v1.22.1
  Server Version: v1.20.1
  

• Istio version (use istioctl version):

  client version: 1.11.2
  control plane version: 1.11.2
  data plane version: 1.11.2 (16 proxies)
  

• Cloud provider or hardware configuration: Test Cluster on Hetzner Cloud
• Others: None

Issue to see reconstructed data in API clarity UI

Issue to see reconstructed data in API clarity UI

time="2023-12-30T17:17:16Z" level=debug msg="unable to get parameters from body: unable to decode body: invalid character '\\x00' looking for beginning of value" func="github.com/openclarity/apiclarity/backend/pkg/modules/internal/traceanalyzer/nlid.(*NLID).learnIDs" file="/build/backend/pkg/modules/internal/traceanalyzer/nlid/nlid.go:175" time="2023-12-30T17:17:16Z" level=debug msg="unable to get parameters from body: unable to decode body: invalid character '<' looking for beginning of value" func="github.com/openclarity/apiclarity/backend/pkg/modules/internal/traceanalyzer/nlid.(*NLID).learnIDs" file="/build/backend/pkg/modules/internal/traceanalyzer/nlid/nlid.go:175

event_annotations batch insert fails

What happened:

CreateAPIEventAnnotations sometimes fails due to unique constraint violation.

On event_annotations, there's a unique key api_event_ann_idx_model on (module_name, event_id, name). When trying to insert multiple rows this constraint could be violated, and the whole insert operation fails.

What you expected to happen:

Deduplication before trying to batch insert?

How to reproduce it (as minimally and precisely as possible):

Not sure so far. But the problem seems clear.

Are there any error messages in API Clarity logs?

(e.g. kubectl logs -n apiclarity --selector=app=apiclarity)

Thanks in advance!

Add Version Number To UI

Please add the current version number and/or git hash to the UI

no monitored traffic on the UI

APIclarity has been deployed correctly in the sock-shop demo application, why is there no monitored traffic on the UI?

kubectl logs -f apiclarity-apiclarity-7c96cc5c87-kd2md -n apiclarity，and reports warning:
level=warning msg="External trace server not started because TLS is not enabled" func=github.com/openclarity/apiclarity/backend/pkg/backend.Run file="/build/backend/pkg/backend/backend.go:237"

Is it because of this reason?

Building Custom OpenTelemetry Collector fails

What happened:

I'm trying to build a custom OpenTelemetry exporter to use the Trace Analyzer feature.

I installed the latest collector builder:

GO111MODULE=on go install go.opentelemetry.io/collector/cmd/builder@latest

and once trying to run it, it fails with the following error:

builder --config=builder-config.yaml
2023-08-18T13:24:57.204-0400    INFO    internal/command.go:117 OpenTelemetry Collector Builder {"version": "dev", "date": "unknown"}
2023-08-18T13:24:57.205-0400    INFO    internal/command.go:150 Using config file       {"path": "builder-config.yaml"}
2023-08-18T13:24:57.205-0400    INFO    builder/config.go:106   Using go        {"go-executable": "/usr/local/go/bin/go"}
2023-08-18T13:24:57.208-0400    INFO    builder/main.go:69      Sources created {"path": "./otelcol-api"}
Error: failed to update go.mod: exit status 1. Output:
go: github.com/openclarity/apiclarity/plugins/otel-collector/[email protected] (replaced by github.com/openclarity/apiclarity/plugins/otel-collector/[email protected]): version "v0.0.0-20220915093602-8a11adcdb9e1" invalid: missing github.com/openclarity/apiclarity/plugins/otel-collector/apiclarityexporter/go.mod at revision 8a11adcdb9e1

Config:
(copied from the instructions here)

dist:
  name: otelcol-custom
  description: Local OpenTelemetry Collector binary
  output_path: /tmp/dist
exporters:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/alibabacloudlogserviceexporter v0.69.0
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.69.1

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.69.1

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.69.1

What you expected to happen:

A successful build of a custom OTEL Collector.

How to reproduce it (as minimally and precisely as possible):

GO111MODULE=on go install go.opentelemetry.io/collector/cmd/builder@latest
cat > otelcol-builder.yaml <<EOF
dist:
  name: otelcol-api
  description: "OTel Collector distribution with APIClarity support"
  output_path: ./otelcol-api

exporters:
  - gomod: "github.com/openclarity/apiclarity/plugins/otel-collector/apiclarityexporter v0.0.0"
  - gomod:
      "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/jaegerexporter
      v0.53.0"
  - import: go.opentelemetry.io/collector/exporter/loggingexporter
    gomod: go.opentelemetry.io/collector v0.53.0

replaces:
  - github.com/openclarity/apiclarity/plugins/otel-collector/apiclarityexporter v0.0.0 => github.com/openclarity/apiclarity/plugins/otel-collector/apiclarityexporter v0.0.0-20220915093602-8a11adcdb9e1
  - github.com/openclarity/apiclarity/plugins/api v0.0.0 => github.com/openclarity/apiclarity/plugins/api v0.0.0-20220915093602-8a11adcdb9e1

receivers:
  - import: go.opentelemetry.io/collector/receiver/otlpreceiver
    gomod: go.opentelemetry.io/collector v0.53.0

processors:
  - import: go.opentelemetry.io/collector/processor/batchprocessor
    gomod: go.opentelemetry.io/collector v0.53.0
EOF
builder --config=builder-config.yaml

Are there any error messages in API Clarity logs?

(e.g. kubectl logs -n apiclarity --selector=app=apiclarity)

Anything else we need to know?:

• macOS 13.5 Ventura
• go version go1.21.0 darwin/amd64

Environment:

• Kubernetes version (use kubectl version --short):
• Istio version (use istioctl version):
• APIClarity version (use kubectl -n apiclarity exec deploy/apiclarity -- ./backend version)
• Cloud provider or hardware configuration:
• Others:

Support har file parser

Is it possible to import a har file, and the apiclarity process this and create openapi file and show on GUI?

spec validation failed

What happened:

Impossible to reconstruct API

What you expected to happen:

reconstruct API

How to reproduce it (as minimally and precisely as possible):

A sandbox will be provided to reproduce the issue.

Are there any error messages in API Clarity logs?

(e.g. kubectl logs -n apiclarity --selector=app=apiclarity)

time="2023-12-21T11:37:44Z" level=error msg="provided spec is not valid OpenAPI 3.0: {\"components\":{\"schemas\":{\"amfInstanceId_deregCallbackUri_guami_ratType\":{\"properties\":{\"amfInstanceId\":{\"format\":\"uuid\",\"type\":\"string\"},\"deregCallbackUri\":{\"type\":\"string\"},\"guami\":{\"$ref\":\"#/components/schemas/guami\"},\"ratType\":{\"type\":\"string\"}},\"type\":\"object\"},\"dnnInfo\":{\"properties\":{\"defaultDnnIndicator\":{\"type\":\"boolean\"},\"dnn\":{\"type\":\"string\"}},\"type\":\"object\"},\"guami\":{\"properties\":{\"amfId\":{\"type\":\"string\"},\"plmnId\":{\"$ref\":\"#/components/schemas/plmnId\"}},\"type\":\"object\"},\"plmnId\":{\"properties\":{\"mcc\":{\"type\":\"string\"},\"mnc\":{\"type\":\"string\"}},\"type\":\"object\"},\"subscribedSnssaiInfos\":{\"properties\":{\"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\":{\"$ref\":\"#/components/schemas/{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\"}},\"type\":\"object\"},\"subscribedSnssaiInfos_0\":{\"properties\":{\"subscribedSnssaiInfos\":{\"$ref\":\"#/components/schemas/subscribedSnssaiInfos\"}},\"type\":\"object\"},\"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\":{\"properties\":{\"dnnInfos\":{\"items\":{\"$ref\":\"#/components/schemas/dnnInfo\"},\"type\":\"array\"}},\"type\":\"object\"}}},\"info\":{\"contact\":{\"email\":\"[email protected]\"},\"description\":\"This is a generated Open API Spec\",\"license\":{\"name\":\"Apache 2.0\",\"url\":\"https://www.apache.org/licenses/LICENSE-2.0.html\"},\"termsOfService\":\"https://swagger.io/terms/\",\"title\":\"Swagger\",\"version\":\"1.0.0\"},\"openapi\":\"3.0.3\",\"paths\":{\"/nudm-sdm/v2/{param1}/smf-select-data\":{\"get\":{\"parameters\":[{\"in\":\"header\",\"name\":\"3gpp-sbi-sender-timestamp\",\"schema\":{\"type\":\"string\"}},{\"in\":\"header\",\"name\":\"3gpp-sbi-max-rsp-time\",\"schema\":{\"format\":\"int64\",\"type\":\"integer\"}}],\"responses\":{\"200\":{\"content\":{\"application/json\":{\"schema\":{\"$ref\":\"#/components/schemas/subscribedSnssaiInfos_0\"}}},\"description\":\"response\"},\"default\":{\"description\":\"default\"}}},\"parameters\":[{\"in\":\"path\",\"name\":\"param1\",\"required\":true,\"schema\":{\"type\":\"string\"}}]},\"/nudm-uecm/v1/{param1}/registrations/amf-3gpp-access\":{\"parameters\":[{\"in\":\"path\",\"name\":\"param1\",\"required\":true,\"schema\":{\"type\":\"string\"}}],\"put\":{\"parameters\":[{\"in\":\"header\",\"name\":\"3gpp-sbi-sender-timestamp\",\"schema\":{\"type\":\"string\"}},{\"in\":\"header\",\"name\":\"3gpp-sbi-callback\",\"schema\":{\"type\":\"string\"}},{\"in\":\"header\",\"name\":\"3gpp-sbi-max-rsp-time\",\"schema\":{\"format\":\"int64\",\"type\":\"integer\"}}],\"requestBody\":{\"content\":{\"application/json\":{\"schema\":{\"$ref\":\"#/components/schemas/amfInstanceId_deregCallbackUri_guami_ratType\"}}}},\"responses\":{\"201\":{\"content\":{\"application/json\":{\"schema\":{\"$ref\":\"#/components/schemas/amfInstanceId_deregCallbackUri_guami_ratType\"}}},\"description\":\"response\"},\"default\":{\"description\":\"default\"}}}}},\"servers\":[{\"url\":\"http://open5gs-udm-sbi.open5gs:7777\"}]}. spec validation failed. invalid components: schema \"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\": identifier \"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\" is not supported by OpenAPIv3 standard (regexp: \"^[a-zA-Z0-9._-]+$\"). spec validation failed" func=github.com/openclarity/speculator/pkg/spec.LoadAndValidateRawJSONSpec file="/go/pkg/mod/github.com/openclarity/[email protected]/pkg/spec/provided_spec.go:90"
time="2023-12-21T11:37:44Z" level=error msg="Failed to validate the spec. provided spec is not valid OpenAPI 3.0: spec validation failed. invalid components: schema \"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\": identifier \"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\" is not supported by OpenAPIv3 standard (regexp: \"^[a-zA-Z0-9._-]+$\"). spec validation failed\n\nspec: {\"components\":{\"schemas\":{\"amfInstanceId_deregCallbackUri_guami_ratType\":{\"properties\":{\"amfInstanceId\":{\"format\":\"uuid\",\"type\":\"string\"},\"deregCallbackUri\":{\"type\":\"string\"},\"guami\":{\"$ref\":\"#/components/schemas/guami\"},\"ratType\":{\"type\":\"string\"}},\"type\":\"object\"},\"dnnInfo\":{\"properties\":{\"defaultDnnIndicator\":{\"type\":\"boolean\"},\"dnn\":{\"type\":\"string\"}},\"type\":\"object\"},\"guami\":{\"properties\":{\"amfId\":{\"type\":\"string\"},\"plmnId\":{\"$ref\":\"#/components/schemas/plmnId\"}},\"type\":\"object\"},\"plmnId\":{\"properties\":{\"mcc\":{\"type\":\"string\"},\"mnc\":{\"type\":\"string\"}},\"type\":\"object\"},\"subscribedSnssaiInfos\":{\"properties\":{\"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\":{\"$ref\":\"#/components/schemas/{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\"}},\"type\":\"object\"},\"subscribedSnssaiInfos_0\":{\"properties\":{\"subscribedSnssaiInfos\":{\"$ref\":\"#/components/schemas/subscribedSnssaiInfos\"}},\"type\":\"object\"},\"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\":{\"properties\":{\"dnnInfos\":{\"items\":{\"$ref\":\"#/components/schemas/dnnInfo\"},\"type\":\"array\"}},\"type\":\"object\"}}},\"info\":{\"contact\":{\"email\":\"[email protected]\"},\"description\":\"This is a generated Open API Spec\",\"license\":{\"name\":\"Apache 2.0\",\"url\":\"https://www.apache.org/licenses/LICENSE-2.0.html\"},\"termsOfService\":\"https://swagger.io/terms/\",\"title\":\"Swagger\",\"version\":\"1.0.0\"},\"openapi\":\"3.0.3\",\"paths\":{\"/nudm-sdm/v2/{param1}/smf-select-data\":{\"get\":{\"parameters\":[{\"in\":\"header\",\"name\":\"3gpp-sbi-sender-timestamp\",\"schema\":{\"type\":\"string\"}},{\"in\":\"header\",\"name\":\"3gpp-sbi-max-rsp-time\",\"schema\":{\"format\":\"int64\",\"type\":\"integer\"}}],\"responses\":{\"200\":{\"content\":{\"application/json\":{\"schema\":{\"$ref\":\"#/components/schemas/subscribedSnssaiInfos_0\"}}},\"description\":\"response\"},\"default\":{\"description\":\"default\"}}},\"parameters\":[{\"in\":\"path\",\"name\":\"param1\",\"required\":true,\"schema\":{\"type\":\"string\"}}]},\"/nudm-uecm/v1/{param1}/registrations/amf-3gpp-access\":{\"parameters\":[{\"in\":\"path\",\"name\":\"param1\",\"required\":true,\"schema\":{\"type\":\"string\"}}],\"put\":{\"parameters\":[{\"in\":\"header\",\"name\":\"3gpp-sbi-sender-timestamp\",\"schema\":{\"type\":\"string\"}},{\"in\":\"header\",\"name\":\"3gpp-sbi-callback\",\"schema\":{\"type\":\"string\"}},{\"in\":\"header\",\"name\":\"3gpp-sbi-max-rsp-time\",\"schema\":{\"format\":\"int64\",\"type\":\"integer\"}}],\"requestBody\":{\"content\":{\"application/json\":{\"schema\":{\"$ref\":\"#/components/schemas/amfInstanceId_deregCallbackUri_guami_ratType\"}}}},\"responses\":{\"201\":{\"content\":{\"application/json\":{\"schema\":{\"$ref\":\"#/components/schemas/amfInstanceId_deregCallbackUri_guami_ratType\"}}},\"description\":\"response\"},\"default\":{\"description\":\"default\"}}}}},\"servers\":[{\"url\":\"http://open5gs-udm-sbi.open5gs:7777\"}]}" func="github.com/openclarity/speculator/pkg/spec.(*Spec).GenerateOASJson" file="/go/pkg/mod/github.com/openclarity/[email protected]/pkg/spec/spec.go:285"
time="2023-12-21T11:37:44Z" level=error msg="Failed to apply the approved review. failed to apply approved review for spec: open5gs-udm-sbi.open5gs:7777. failed to generate Open API Spec. failed to validate the spec. provided spec is not valid OpenAPI 3.0: spec validation failed. invalid components: schema \"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\": identifier \"{\\n\\t\\\\\\\"sst\\\\\\\":\\t1,\\n\\t\\\\\\\"sd\\\\\\\":\\t\\\\\\\"111111\\\\\\\"\\n}\" is not supported by OpenAPIv3 standard (regexp: \"^[a-zA-Z0-9._-]+$\"). spec validation failed" func="github.com/openclarity/apiclarity/backend/pkg/rest.(*Server).PostAPIInventoryReviewIDApprovedReview" file="/build/backend/pkg/rest/review_controller.go:66"
time="2023-12-21T11:37:45Z" level=error msg="BFLA error: unable to process trace: event 124913 cannot resolve to a spec path" func=github.com/openclarity/apiclarity/backend/pkg/modules/internal/bfla/bfladetector.NewBFLADetector.func1 file="/build/backend/pkg/modules/internal/bfla/bfladetector/learn_and_detect_bfla.go:127"

Anything else we need to know?:

Environment:

• Kubernetes version Client Version: v1.27.3
• Istio version: 1.20.0
• APIClarity version: Version: v0.14.5, Commit: 26129dd
• Cloud provider or hardware configuration: Null
• Others: Null

Support for KrakenD API Gateway

Is your feature request related to a problem? Please describe.
Currently, all of our service API endpoints are served via KrakenD, so without this support, we are unable to use APIClarity.

Describe the solution you'd like
All of the API Clarity features for k8s (isto), but also for KrakenD.

Describe alternatives you've considered
Currently, we are using Swagger and custom-built API monitoring tool, but we still don't have a clear picture of calls between APIs, across all of the different microservices.

Additional context

Add support for OpenAPI Specification V3.0

Currently, goswagger OAS v2.0 structures are used to store spec information - both provided and reconstructed. In order to get full support for OAS v3.0, need to move to V3 structures (e.g. https://github.com/getkin/kin-openapi/tree/master/openapi3). To support older versions, we can use converters such as https://github.com/LucyBot-Inc/api-spec-converter, that can be executed using CLI, for example if the user provided a V2 spec that need to diff against.

Alternatives:
Keep the current goswagger OAS v2.0 structures and use converters such as https://github.com/getkin/kin-openapi/tree/master/openapi2conv to to generate the V3 spec. This is less preferred because if the traffic was according to V3 spec, the reconstruction will be lossy.

Support for AWS API Gateway with Lambda proxy integration

I'm looking to contribute an integration between AWS API Gateway and APIClarity using a Lambda proxy integration as a POC. My initial thoughts is to build a proxy that converts the input and output formats of AWS API Gateway into the schema of the plugin API.

Are you aware of any prior work that uses APIClarity in an AWS environment (outside of k8s)?

OAS3 not importing

Hi there, I'm attempting to pull an OAS3 doc into API Clarity and met with the following error:

json: cannot unmarshal bool i │
│ nto Go struct field SwaggerProps.paths of type string" func="github.com/apiclarity/apiclarity/backend/pkg/rest.(*S │
│ erver).PutAPIInventoryAPIIDSpecsProvidedSpec" file="/build/backend/pkg/rest/apiinventory_upload_spec.go:52"

Here's the actual OAS file, which compiles perfectly in editor.swagger.io

openapi: 3.0.1
info:
  version: '1.0-oas3'
  title: httpbin
  description: An unofficial OpenAPI definition for [httpbin.org](https://httpbin.org) v3

servers:
  - url: https://httpbin.org
  - url: http://httpbin.org
  - url: https://eu.httpbin.org
  - url: http://eu.httpbin.org

tags:
  - name: auth
    description: Operations for testing various authentication types
  - name: HTTP methods
    description: Operations for testing different HTTP methods
  - name: Status codes
    description: Return the specified HTTP status code, or a random status code if more than one are given

# All paths & parameters are described in
# https://github.com/kennethreitz/httpbin/blob/master/httpbin/core.py

externalDocs:
  url: http://httpbin.org/legacy

paths:
  # New operations for parsing time
  /:
    get:
      summary: The current time, in a variety of formats
      tags:
        - time
      servers:
        - url: https://now.httpbin.org
        - url: http://now.httpbin.org
      #externalDocs:
      #  url: /docs # ???
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                type: object
                properties:
                  now:
                    $ref: '#/components/schemas/Timestamp'
                  urls:
                    type: array
                    items:
                      type: string
                    example:
                      - /
                      - /docs
                      - '/when/:human-timestamp'
                      - '/parse/:machine-timestamp'
                required:
                  - now
                  - urls

  /when/{human-timestamp}:
    get:
      tags:
        - time
      servers:
        - url: http://now.httpbin.org
        - url: https://now.httpbin.org
      parameters:
        - in: path
          name: human-timestamp
          required: true
          schema:
            anyOf:
              - type: string # ????
      responses:
        '200':
          $ref: '#/components/responses/TimestampResponse'
        '500':
          description: oops

  /parse/{machine-timestamp}:
    get:
      tags:
        - time
      servers:
        - url: http://now.httpbin.org
        - url: https://now.httpbin.org
      parameters:
        - in: path
          name: machine-timestamp
          required: true
          schema:
            anyOf:
              - type: string # ????
              - type: number
      responses:
        '200':
          $ref: '#/components/responses/TimestampResponse'
        '500':
          description: oops

  /get:
    get:
      tags:
        - HTTP methods
      summary: |
        Returns the GET request's data. Accepts any query parameters and any headers.
      parameters:
        - $ref: '#/components/parameters/freeFormQuery'
      responses:
        '200':  # Change to 'default' ???
          description: OK
          content:
            application/json:
              schema:
                type: object

  /delete:
    delete:
      tags:
        - HTTP methods
      summary: |
        Returns the DELETE request's data. Accepts any query parameters and any headers.
      parameters:
        - $ref: '#/components/parameters/freeFormQuery'
      responses:
        200:
          description: OK

  /post:
    post:
      tags:
        - HTTP methods
      # summary: POSTs a pizza order and returns the POSTed data.
      summary: Returns the POSTed data
      parameters:
        - $ref: '#/components/parameters/freeFormQuery'
      requestBody:
        description: Data provided in the request body will be returned in the response.
        content:
          application/json:
            schema: {}
            example:
              message: Hello, world!
          application/vnd+json:
            schema: {}
            examples:
              pizzaOrder:
                summary: Pizza order data
                description: Longer description ...
                value:
                  custname: Alice
                  custtel: '[phone]'
                  custemail: [email protected]
                  size: medium
                  topping: [cheese, mushroom]
                  delivery: '19:00'
                  comments: Ring the door bell three times
              simpleObject:
                summary: sample object
                value:
                  foo: bar
          application/xml:
            schema:
              type: object
            example:
              message: Hello, world!
          text/plain:
            schema:
              type: string
              example: Hi there
          application/x-www-form-urlencoded:
            schema:
              # anyOf:
              #   - type: object
              #     additionalProperties: true

              #   - description: Pizza order
              type: object
              properties:
                custname:
                  type: string
                  example: Alice
                  description: Customer name
                custtel:
                  type: string
                  example: '[phone]'
                  description: Customer phone number
                custemail:
                  type: string
                  format: email
                  example: [email protected]
                  description: Customer email address
                size:
                  type: string
                  enum:
                    - small
                    - medium
                    - large
                  description: Pizza size
                topping:
                  type: array
                  items:
                    type: string
                    enum:
                      - bacon
                      - cheese
                      - mushroom
                      - onion
                  description: Pizza toppings
                delivery:
                  type: string
                  example: '13:30'
                  description: Delivery time
                comments:
                  type: string
                  example: ASAP
                  description: Comments
            examples:
              pizzaOrder:
                summary: Pizza order data
                description: Longer description ...
                value:
                  custname: Alice
                  custtel: '[phone]'
                  custemail: [email protected]
                  size: medium
                  topping: [cheese, mushroom]
                  delivery: '19:00'
                  comments: Ring the door bell three times
              simpleObject:
                summary: sample object
                value:
                  foo: bar
          # multipart/form-data: {}  # TODO
          '*/*':   # is this valid?
            schema: {}
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/CommonResponse'
              #schema:
              #  type: object

  # /put: {}
  # /patch: {}
  # /delete: {}
  # /response-headers:
  #   parameters:
  #     - in: query
  #       name: headers
  #       # Arbitrary key=value pairs
  #       schema:
  #         type: object
  #         example:
  #           Server: unicorn
  #       style: simple
  #   get: {}
  #   post: {}


  /ip:
    get:
      summary: Returns Origin IP.
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                type: object
                properties:
                  origin:
                    type: string
                    description: >
                      The IP address, or a comma-separated list of IP addresses.
                      For example, "10.100.10.10, 10.100.20.254, 52.91.14.13"'
                    example: 10.100.10.10
                required:
                  - origin
              examples:   # Content examples override schema-level examples
                oneIp:
                  description: Example of a single IP
                  value:
                    origin: 10.100.10.10
                multipleIps:
                  description: Example of multiple IPs
                  value:
                    origin: 10.100.10.10, 10.100.20.254, 52.91.14.13

  /user-agent:
    get:
      summary: Returns the user agent.
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                type: object
                properties:
                  user-agent:
                    type: string
                    example: curl/7.37.0
                required:
                  - user-agent

  /headers:
    get:
      summary: Returns the request headers.
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                type: object
                properties:
                  headers:
                    type: object
                    additionalProperties:
                      type: string
                    # wrong syntax!
                    #example:
                    #  $ref: '#/components/examples/headers'
                required:
                  - headers

  # /anything:
  #   summary: Returns request data, including method used.
  #   get: {}
  #   post: {}
  #   put: {}
  #   patch: {}
  #   delete: {}
  #   trace: {}
  #   options: {}

  # /anything/{anything}:
  #   summary: Returns request data, including method used.

  #   parameters:
  #     - in: path
  #       name: anything
  #       required: true
  #       schema:
  #         type: string
  #       description: An arbitrary subpath

  #   get: {}
  #   post: {}
  #   put: {}
  #   patch: {}
  #   delete: {}
  #   trace: {}
  #   options: {}

  /delay/{n}:
    get:
      summary: Delays responding for min(n, 10) seconds.
      parameters:
        - name: n
          in: path
          required: true
          description: Response delay, in seconds.
          schema:
            type: integer
            minimum: 0
            maximum: 10
      responses:
        '200':
          description: OK
          content:
            application/json: {}
              # schema:
              #   type: object

  /basic-auth/{user}/{password}:
    get:
      summary: Challenges HTTPBasic Auth.
      tags:
        - auth
      security:
        - basicAuth: []
      parameters:
        - $ref: '#/components/parameters/user'
        - $ref: '#/components/parameters/password'
      responses:
        '200':
          $ref: '#/components/responses/200BasicAuth'
        '401':
          description: >-
            Unauthorized (The username and password used for Basic auth do not
            match those in the URL path.)
          headers:
            Www-Authenticate:
              schema:
                type: string
                example: 'Basic realm="Fake Realm"'

  /hidden-basic-auth/{user}/{password}:
    get:
      summary: Hidden Basic authentication
      tags:
        - auth
      description: Returns 404 Not Found unless the request is authenticated.
      security:
        - basicAuth: []
      parameters:
        - $ref: '#/components/parameters/user'
        - $ref: '#/components/parameters/password'
      responses:
        '200':
          $ref: '#/components/responses/200BasicAuth'
        '404':
          description: >-
            Unautorized (the username and password used for Basic auth do not
            match those in the URL path.)

  /bearer:
    get:
      summary: Tests Bearer authentication
      tags:
        - auth
      security:
        - bearerAuth: []
      responses:
        '200':
          description: Authorized
          content:
            application/json:
              schema:
                type: object
                properties:
                  authenticated:
                    type: boolean
                    example: true
                  token:
                    type: string
                    description: Bearer token specified in the request
        '404':
          description: Unauthorized


  /status/{statusCode}:
    summary: Returns the specified HTTP status code, or a random status code if more than one are given
    parameters:
      - name: statusCode
        in: path
        required: true
        description: The status code to return, or a weighted list of statuses to pick from, such as `200:4,500:0.3,418`.
        schema:
          type: array
          items:
            description: HTTP status code. May include optional weight, e.g. `200:0.9`
            oneOf:
              - type: string
              - type: integer
          minItems: 1
          example:
            - '200:5'
            - '500:0.3'
            - 418
        style: simple

    get:
      tags:
        - Status codes
      responses:
        default:
          description: A response with the requested status code.
    post:
      tags:
        - Status codes
      responses:
        default:
          description: A response with the requested status code.
    patch:
      tags:
        - Status codes
      responses:
        default:
          description: A response with the requested status code.
    put:
      tags:
        - Status codes
      responses:
        default:
          description: A response with the requested status code.
    delete:
      tags:
        - Status codes
      responses:
        default:
          description: A response with the requested status code.

  /xml:
    get:
      summary: Returns some XML.
      responses:
        '200':
          description: OK
          content:
            application/xml: {}

  /html:
    get:
      summary: Returns an HTML page
      responses:
        '200':
          description: OK
          content:
            text/html: {}

  /image/{format}:
    get:
      summary: Returns an image with the specified format.
      tags:
        - images
      parameters:
        - in: path
          name: format
          required: true
          schema:
            type: string
            enum:
              - png
              - jpeg
              - webp
              - svg
      responses:
        '200':
          $ref: '#/components/responses/Image'

  /image:
    get:
      summary: Returns an image based on the Accept header.
      tags:
        - images
      responses:
        '200':
          $ref: '#/components/responses/Image'
        '406':
          description: Client did not request a supported media type.

  /cache:
    get:
      summary: >-
        Returns 200 unless an If-Modified-Since or If-None-Match header is
        provided, when it returns a 304.
      parameters:
        - in: header
          name: If-Modified-Since
          required: false
          description: >
            For testing purposes this header accepts any value. (???)
            See also https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since
          schema:
            type: string
            example: 'Wed, 21 Oct 2015 07:28:00 GMT'
        - in: header
          name: If-None-Match
          required: false
          description: >
            For testing purposes this header accepts any value. (???)
            See also https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match
          schema:
            type: string
          examples:
            etag:
              value: '"bfc13a64729c4290ef5b2c2730249c88ca92d82d"' # Quotes are part of the value
            etags:
              value: 'W/"67ab43", "54ed21", "7892dd"'
            asterisk:
              value: '*'
      responses:
        '200':
          description: Neigher If-Modified-Since nor If-None-Match header is provided
        '304':
          description: If-Modified-Since or If-None-Match header is provided

  /cookies:
    get:
      tags:
        - Cookies
      summary: Returns cookie data
      responses:
        200:
          description: OK
  /cookies/set:
    get:
      tags:
        - Cookies
      summary: Sets one or more simple cookies
      parameters:
        - $ref: '#/components/parameters/freeFormQuery'
      responses:
        200:
          description: OK
  /cookies/delete:
    get:
      tags:
        - Cookies
      summary: Delete one or more simple cookies
      parameters:
        - $ref: '#/components/parameters/freeFormQuery'
      responses:
        200:
          description: OK


#################################
# Reusable things
#################################
components:
  schemas:
    CommonResponse:
      description: Response returned by `/get`
      type: object
      properties:
        args:
          type: object
          additionalProperties:
            type: string
          description: Query string parameters specified in the request URL.
        headers:
          type: object
          additionalProperties:
            type: string
          description: >
            Headers (standard or custom) used in the request. Some typical headers are:
              - Accept
              - Accept-Encoding
              - Content-Length
              - Content-Type
              - Host
              - Origin
              - Referred
              - User-Agent
          # not valid syntax
          #example:
          #  $ref: '#/components/examples/headers'
        origin:
          type: string
          description: The origin IP from which the request was made.
          example: '10.100.10.10, 10.100.10.44, 52.91.14.13'
        url:
          type: string
          format: uri
          description: The endpoint URL to which the request was made.
          example: https://httpbin.org/post
      required:
        - args
        - headers
        - origin
        - url

    PostResponse:
      description: Response returned by /post, /put, /patch and /delete
      allOf:
        - $ref: '#/components/schemas/CommonResponse'
        - type: object
          properties:
            data: {}    # Always a plain text string???
              # ???
              # oneOf:
              #   - type: string
              #   - type: object
            files: {}
            #  type: object
            #  description: ???
            form:
              type: object
              additionalProperties:
                type: string
              description: >
                Form parameters specified in "application/x-www-form-urlencoded" and
                `multipart/form-data` requests.
            json:
              description: >
                JSON value sent in the payload.
                Can be object, array, string, number, boolean or `null`.
              nullable: true
              # oneOf:
              #   - type: object
              #   - type: array
              #   - type: string
              #   - type: number
              #   - type: boolean

    TimestampWrapper:
      type: object
      properties:
        timestamp:
          $ref: '#/components/schemas/Timestamp'
      required:
        - timestamp

    Timestamp:
      type: object
      properties:
        epoch:
          type: number
          format: double
          example: 1498229228.0671656
        slang_date:
          type: string
          example: today
        slang_time:
          type: string
          example: now
        iso8601:
          type: string
          # format: ???
          example: '2017-06-23T14:47:08.067166Z'
        rfc2822:
          type: string
          # format: ???
          example: 'Fri, 23 Jun 2017 14:47:08 GMT'
        rfc3339:
          type: string
          # format: ????
          example: '2017-06-23T14:47:08.06Z'
      required:
        - epoch
        - slang_date
        - slang_time
        - iso8601
        - rfc2822
        - rfc3339
      example:
        epoch: 1485183550.84644
        slang_date: Jan 23
        slang_time: 4 months ago
        iso8601: '2017-01-23T14:59:10.846440Z'
        rfc2822: Mon, 23 Jan 2017 14:59:10 GMT
        rfc3339: '2017-01-23T14:59:10.84Z'
      # schema does NOT support plural "examples"
      # examples:
      #   Now:
      #     value:
      #       epoch: 1498229228.0671656
      #       slang_date: today
      #       slang_time: now
      #       iso8601: '2017-06-23T14:47:08.067166Z'
      #       rfc2822: Fri, 23 Jun 2017 14:47:08 GMT
      #       rfc3339: '2017-06-23T14:47:08.06Z'
      #   DateInPast:
      #     description: Example of a date in the past
      #     value:
      #       epoch: 1485183550.84644,
      #       slang_date": Jan 23
      #       slang_time": 4 months ago
      #       iso8601: '2017-01-23T14:59:10.846440Z'
      #       rfc2822: Mon, 23 Jan 2017 14:59:10 GMT
      #       rfc3339: '2017-01-23T14:59:10.84Z'

  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
      description: Use the same username and password as you will provide in path parameters.
    bearerAuth:
      type: http
      scheme: bearer

  parameters:
    user:
      name: user
      in: path
      required: true
      description: Username. Use the same username in the path AND for authorization.
      schema:
        type: string
    password:
      name: password
      in: path
      required: true
      description: Password. Use the same password in the path AND for authorization.
      schema:
        type: string
    freeFormQuery:
      name: freeform
      in: query
      schema:
        type: object
        additionalProperties: true
      # This is the default serialization method, so it can be omitted
      style: form
      explode: true
      description: >
        Enter free-form query parameters in the JSON format
        `{ "param1": "value1", "param2": "value2", ... }`.


        Note that the parameters will be actually sent as
        `?param1=value1&param2=value2&...`


  responses:
    200BasicAuth:
      description: OK
      content:
        application/json:
          schema:
            type: object
            properties:
              authenticated:
                type: boolean
                example: true
              user:
                type: string
                description: The user name specified in the request.
    Image:
      description: OK
      content:
        image/png, image/jpeg, image/webp:
          schema:
            type: string
            format: binary
        image/svg+xml: {}  # string? object?
        image/svg: {}  # string? object?

    TimestampResponse:
      description: OK
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/TimestampWrapper'

  examples:
    headers:
      value:
        Accept: '*/*'
        Connection: close
        Host: httpbin.org
        User-Agent: curl/7.37.0

    Now:
      value:
        epoch: 1498229228.0671656
        slang_date: today
        slang_time: now
        iso8601: '2017-06-23T14:47:08.067166Z'
        rfc2822: Fri, 23 Jun 2017 14:47:08 GMT
        rfc3339: '2017-06-23T14:47:08.06Z'
    DateInPast:
      description: Example of a date in the past
      value:
        epoch: 1485183550.84644,
        slang_date": Jan 23
        slang_time": 4 months ago
        iso8601: '2017-01-23T14:59:10.846440Z'
        rfc2822: Mon, 23 Jan 2017 14:59:10 GMT
        rfc3339: '2017-01-23T14:59:10.84Z'

[Question] Is there any support for Kong API Gateway for external cluster tracing?

Hey all, just wondering if there is any support for Kong on other kubernetes clusters, was only able to find documentation on internal cluster tracing but not external.

BFLA Analyzer is not reporting BFLA violation

What happened:

I enabled trace sampling for APIClarity in the helm chart then I started BFLA learning on one of my app APIs. I ran some app traffic, and then stopped BFLA learning. I went and marked one of the microservice interactions as "illegimate" (Sock Shop front-end service GET request to catalog API), and then generated more app traffic that should have triggered a BFLA violation. Nothing was reported, and no BFLA data is found.

What you expected to happen:

I expected to see a BFLA violation when the front-end service did a GET on the catalog API.

How to reproduce it (as minimally and precisely as possible):

Using Sock Shop, mark the GET catalog from front-end service as "illegimate" then run locust traffic to Sock Shop that will send some of those requests.

Are there any error messages in API Clarity logs?

time="2023-02-16T14:24:49Z" level=info msg="bfla synced for authz model" func="github.com/openclarity/apiclarity/backend/pkg/modules/internal/bfla/bfladetector.(*learnAndDetectBFLA).commandsRunner" file="/build/backend/pkg/modules/internal/bfla/bfladetector/learn_and_detect_bfla.go:523"
time="2023-02-16T14:24:49Z" level=info msg="stop learning applied successfully on api=1" func=github.com/openclarity/apiclarity/backend/pkg/modules/internal/bfla.httpHandler.PutAuthorizationModelApiIDLearningStop file="/build/backend/pkg/modules/internal/bfla/init.go:543"
time="2023-02-16T14:24:49Z" level=debug msg="current state for api 1 is LEARNT" func="github.com/openclarity/apiclarity/backend/pkg/modules/internal/bfla/bfladetector.(*learnAndDetectBFLA).checkBFLAState" file="/build/backend/pkg/modules/internal/bfla/bfladetector/learn_and_detect_bfla.go:336"
time="2023-02-16T14:24:49Z" level=debug msg="current state for api 1 is LEARNT" func="github.com/openclarity/apiclarity/backend/pkg/modules/internal/bfla/bfladetector.(*learnAndDetectBFLA).getState" file="/build/backend/pkg/modules/internal/bfla/bfladetector/learn_and_detect_bfla.go:971"
time="2023-02-16T14:24:51Z" level=debug msg="nothing to update for module=bfla; apiID=1" func="github.com/openclarity/apiclarity/backend/pkg/modules/internal/bfla/recovery.(*persister).persistAPIInfoAnnotations" file="/build/backend/pkg/modules/internal/bfla/recovery/state.go:158"

And then, all I get is this:
time="2023-02-16T14:32:36Z" level=debug msg="nothing to update for module=bfla; apiID=1" func="github.com/openclarity/apiclarity/backend/pkg/modules/internal/bfla/recovery.(*persister).persistAPIInfoAnnotations" file="/build/backend/pkg/modules/internal/bfla/recovery/state.go:158"
time="2023-02-16T14:32:36Z" level=debug msg="Persisted 0 annotations" func="github.com/openclarity/apiclarity/backend/pkg/modules/internal/bfla/recovery.(*persister).persistAPIInfoAnnotations" file="/build/backend/pkg/modules/internal/bfla/recovery/state.go:173"
time="2023-02-16T14:32:36Z" level=debug msg="Acked 0 events; events: []" func="github.com/openclarity/apiclarity/backend/pkg/modules/internal/bfla/recovery.(*persister).persistAPIEventAnnotations" file="/build/backend/pkg/modules/internal/bfla/recovery/state.go:125"

Anything else we need to know?:

Could be related to #340 as API traffic is not being recorded.

Environment:

APIClarity v0.14.2

Simplified Deployment For Envoy/WASM Traffic Sources

Cutting out a ton of backstory / context, a recent deployment of API Clarity to a cluster that was already running Istio resulting in discovering a much simpler method of applying the WASM filter. I think it would be extremely beneficial to replace the extant method with it.

I've written up a gist that explains it here, but the short version is that it eliminates the need for the post-install patching job, the ConfigMap(s) containing the WASM filter binary, and the associated annotations/configuration for the Istio sidecar(s).

If there's interest, I would be very interested in working with a member of the API Clarity team to get it worked into the API Clarity / WASM filter repos.

Only most recent 50 events are saved in API events table

I'm seeing new behavior with APIClarity v0.14.4 - the API events table is only keeping the last 50 events. Prior to 0.14.4, it was keeping many more events. In my test, I ran locust against sock shop APIs and generated over 1000 API calls (see first screenshot for count) but only 50 API events are listed (second screenshot).

If I run more API traffic, 50 new calls are kept, but everything else is gone. This seems like really degraded behavior.

Support for Gravitee.io API Gateway

Hi there,

I'm looking for contributing to provide integration between Gravitee.io API Gateway and API Clarity as a POC.

Do you have any pointer for me to start with? I had a quick look to the plugins you are providing for Kong and Tyk, but I didn't find any relevant information (API ?) where to push instrumentation metrics.

Thanks in advance.

"Deployment" type needs to be configurable

https://github.com/apiclarity/apiclarity/blob/02b3b5d823545ca7f3214e9dab16b9bf18cbc69e/plugins/gateway/tyk/deploy/deploy.sh#L12

this Needs to be configurable as "daemonset" is what the default Tyk Helm Chart is deployed as.

Tyk Integration plugin kills the response

Using httpbin.org upstream with the sample included Tyk integration plugin, the integration works and the telemetry data flows into APIClarity, but the API Response being returned to the client is killed:

$ curl localhost:8080/httpbin/json
curl: (18) transfer closed with 429 bytes remaining to read

Provide better validation error message for uploaded spec

Is your feature request related to a problem? Please describe.
I tried to upload an OpenAPI spec file, but it was missing a version (was an empty string). The Upload Spec UI just says "Error in File", but the log message for APIClarity server is much more detailed: "spec validation failed. invalid info: value of version must be a non-empty string. spec validation failed""

Describe the solution you'd like
I'd like to see the more descriptive error make it's way into the UI to provide feedback about what's wrong with the spec file.

Describe alternatives you've considered
Until I found the log message, I had no idea what was wrong, so there wasn't an alternative.

inject env variables for corporate proxy

If apiclarity needs to run on a cluster behind corporate proxy, wasm post install job fails to complete.

injection of env variables (HTTPS_PROXY, https_proxy, HTTP_PROXY, http_proxy, no_proxy, NO_PROXY) and update git config (git config --global http.proxy, git config --global https.proxy) fixes the issue

Color coding HTTP Verbs

I saw a demo of the solution today and thought that a good idea would be to color code the HTTP verbs similar to what swagger UI supports.

I am talking about this interface:

And using this:

I think that from a UI/UX perspective, it makes it easier to consume.

Support Linkerd and/or api gateway like gloo/ambassador

I am hesitant to adopt istio due to the depth and breadth it requires. Is there any plan to integrate with linkerD or an API Gateway like gloo or ambassador?

Unable to start BFLA learning

What happened:

In APIClarity v0.14.2, I tried to start BFLA learning for an API by hitting the start button, but the UI just refreshed to the screen with the start button again, and a generic error was reported at the top. In the APIClarity log, there was more useful information: "BFLA error: cannot enable traces: trace sampling is not enabled". So I enable trace sampling in the chart, and now it's working. I think either a descriptive error message should be shown in the UI (e.g. "Trace sampling is not enabled") or the BFLA tab shouldn't be shown if trace sampling is not enabled.

What you expected to happen:

Either a better error message should have been displayed, or the BFLA tab should be hidden if trace sampling is not enabled.

How to reproduce it (as minimally and precisely as possible):

Install APIClarity helm chart with trace sampling disabled (default). Try to start BFLA learning.

Are there any error messages in API Clarity logs?

"BFLA error: cannot enable traces: trace sampling is not enabled"

istio proxy do not started because wasm not loaded

Hi
I got these issues:
Invalid path: /var/local/lib/wasm-filters/http-trace-filter.wasm

Do you know why?

Add ADOPTERS.md to APIClarity

Please add ADOPTERS.md file, so we could better track usage of the APIClarity.

Example: bank-vaults, logging-operator

Deployment apiclarity reports error "job failed"

When deploying apiclarity in sock-shop, an error "job failed" is reported.
helm install --set 'trafficSource.envoyWasm.enabled=true' --set 'trafficSource.envoyWasm.namespaces={sock- shop}' --create-namespace apiclarity ./apiclarity-v0.14.5.tgz -n apiclarity --debug

and when helm list -n apiclarity，display status is failed

But, the pods and services are running normally,

and When tested in the Sock Shop demo application, no data was generated

kubectl describe pod apiclarity-apiclarity-7c96cc5c87-jxc48 -n apiclarity reports a error:
Warning Unhealthy 42m kubelet Readiness probe failed: HTTP probe failed with statuscode: 503

Support to Git Sync Reconstructed API spec than only storing in storage drive

Is your feature request related to a problem? Please describe.
As user I would like to save the reconstructed API spec to be stored in Git and version control it and I can reuse in User provided spec (with or without implementation of #89) if and when needed

Describe the solution you'd like
Automatically Git Sync the applied spec using automated Reconstructed Spec to repo (if enabled to do so in settings) to version control what's applied and revert in case we need to than complete rest.

Describe alternatives you've considered
Reset and Redo

Additional context
None

"helm uninstall apiclarity" does not clean pv and pvc resources

When trying to uninstall APIClarity using helm, the PersistentVolume and PersistentVolumeClaim resources are not destroyed.
This causing the next installation to have already an existing data from previous installation.

Need to add to the README the correct way to uninstall APIClarity (meaning delete the pv as well)

Create a helm chart for APIClarity

Currently, APIClarity can be installed only using deployment/apiclarity.yaml.
Need to create a helm chart to manage all the components in the deployment yaml, including volumes with consideration of the cloud platform. In addition, need to create a dependency for the Postgres DB (instead of the static spec that was generated using the Postgres helm template).

Support for local-storage pv class

apiclarity-apiclarity pod can not start

When i installed APIClarity in a K8s cluster using Helm，the apiclarity pod did not start properly.

kubectl describe pod，logs as follow:
`Name: apiclarity-apiclarity-75f75c6db4-7w4q7
Namespace: apiclarity
Priority: 0
Service Account: apiclarity-apiclarity
Node: master/XX.XX.XX.XX
Start Time: Wed, 20 Sep 2023 11:08:05 +0800
Labels: app=apiclarity-apiclarity
pod-template-hash=75f75c6db4
Annotations: sidecar.istio.io/inject: false
Status: Pending
IP: 10.42.0.177
IPs:
IP: 10.42.0.177
Controlled By: ReplicaSet/apiclarity-apiclarity-75f75c6db4
Init Containers:
apiclarity-apiclarity-wait-for-db:
Container ID: docker://7386e858d32f3e92fe18c606811ea03be57c563efbf76e766d2c71512021aa11
Image: docker.io/bitnami/postgresql:14.4.0-debian-11-r4
Image ID: docker-pullable://bitnami/postgresql@sha256:58744ac056007ba278d80090d41722e30c49b20d85c56653f337e7e0a7b2f312
Port:
Host Port:
Command:
sh
-c
until pg_isready -h apiclarity-apiclarity-postgresql -p 5432 -U "postgres" -d "dbname=apiclarity"; do echo waiting for database; sleep 2; done;
State: Running
Started: Wed, 20 Sep 2023 11:08:06 +0800
Ready: False
Restart Count: 0
Limits:
cpu: 200m
memory: 1000Mi
Requests:
cpu: 100m
memory: 200Mi
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qhqkl (ro)
Containers:
apiclarity:
Container ID:
Image: ghcr.io/openclarity/apiclarity:v0.14.5
Image ID:
Port:
Host Port:
Args:
run
--log-level
warning
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 1
memory: 1000Mi
Requests:
cpu: 100m
memory: 200Mi
Liveness: http-get http://:8081/healthz/live delay=10s timeout=10s period=30s #success=1 #failure=5
Readiness: http-get http://:8081/healthz/ready delay=0s timeout=10s period=30s #success=1 #failure=5
Environment:
POD_NAMESPACE: apiclarity (v1:metadata.namespace)
RESPONSE_HEADERS_TO_IGNORE: <set to the key 'response.headers' of config map 'apiclarity-apiclarity-headers-to-ignore-configmap'> Optional: false
REQUEST_HEADERS_TO_IGNORE: <set to the key 'request.headers' of config map 'apiclarity-apiclarity-headers-to-ignore-configmap'> Optional: false
TRACE_SAMPLING_ENABLED: false
BFLA_AUTOMATIC_LEARNING_AND_DETECTION: false
DB_NAME: apiclarity
DB_HOST: apiclarity-apiclarity-postgresql
DB_PORT_NUMBER: 5432
DB_USER: postgres
DB_PASS: <set to the key 'postgres-password' in secret 'apiclarity-postgresql-secret'> Optional: false
STATE_BACKUP_FILE_NAME: /apiclarity/state.gob
FUZZER_JOB_TEMPLATE_CONFIG_MAP_NAME: apiclarity-apiclarity-fuzzer-template
FUZZER_DEPLOYMENT_TYPE: configmap
Mounts:
/apiclarity from apiclarity-apiclarity (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qhqkl (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
apiclarity-apiclarity:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: apiclarity-apiclarity-pvc
ReadOnly: false
kube-api-access-qhqkl:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 18m default-scheduler Successfully assigned apiclarity/apiclarity-apiclarity-75f75c6db4-7w4q7 to master
Normal Pulled 18m kubelet Container image "docker.io/bitnami/postgresql:14.4.0-debian-11-r4" already present on machine
Normal Created 18m kubelet Created container apiclarity-apiclarity-wait-for-db
Normal Started 18m kubelet Started container apiclarity-apiclarity-wait-for-db
`

kubectl logs -f apiclarity-apiclarity-75f75c6db4-7w4q7 -n apiclarity, error reported as follows:

Why is this pod stuck in the initialization state, not to start. Is it because I used the default value file values.yaml and didn't add a traces data source?

OAS reconstruction results in unusual endpoints

I spun up Tyk / APIClarity and used the good ol' httpbin endpoint.

I make the following API requests:

$ curl -s localhost:8080/httpbin/get &
    curl -s localhost:8080/httpbin/post -X POST &
    curl -s localhost:8080/httpbin/xml &
    curl -s localhost:8080/httpbin/json &
    curl -s localhost:8080/httpbin/headers &
    curl -s localhost:8080/httpbin/delete -X DELETE &

Two weird things happen

1. APIClarity never picks up the xml endpoint
2. The reconstructured OAS spec flow doesn't make sense.

Before I continue for the second one, you can recreate this by following this branch.

OK so once I generate load against httpbin, I attempt to reconstruct the OAS spec.

Again (note that xml is missing from list of endpoints.

Number two, I can't edit the documentation at all. And once it's generated, it's basically an empty OAS spec.

Am I misunderstanding how to use the software?

Cannot bind volume on EKS

What happened:

Tried to install apiclarity on EKS version 1.24. Tried it on managed node group and on Fargate.
The ebs-csi controller is provisioned.
Both of the apiclarity pods are stuck on pending and return the following error:
Warning FailedScheduling 9m33s default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

What you expected to happen:

Pods to start normally

How to reproduce it (as minimally and precisely as possible):

EKS cluster version 1.24

Are there any error messages in API Clarity logs?

Warning FailedScheduling 9m33s default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

Anything else we need to know?:

Environment:

• Kubernetes version 1.24
• Istio version (use istioctl version):
  client version: 1.16.1
  control plane version: 1.11.4
  data plane version: 1.11.4 (1 proxies)
• APIClarity version (use kubectl -n apiclarity exec deploy/apiclarity -- ./backend version)
• Cloud provider or hardware configuration:
• AWS + EKS
• Others:

Support User provided spec upload using Git, Postman or Plain Text editor within APIClarity

Is your feature request related to a problem? Please describe.
Having an option to load the User provided spec from Git / Postman and also as plain text within the API clarity helps in easy to use the tool and less context switching.

Describe the solution you'd like
User should be able to upload "user provided spec" by simply writing it on plain text editor inbuilt on APIClarity, and also by simply uploading via mentioning Git URL or Postman API spec.

Describe alternatives you've considered
None

Additional context
None

List shadow and zombie APIs at top-level

Is your feature request related to a problem? Please describe.
RIght now, if a shadow or zombie API is detected, it is reported in two potential places - in the "Latest spec diffs" on the dashboard UI (if it happened recently) and in the API event list. Either way, a specific symbol is shown for shadow or zombie. This is the only indication that a shadow/zombie API has been detected.

Describe the solution you'd like
It would be great if a listing could be added to the trace analysis for the API indicating it is a shadow or zombie. That way, a more permanent record of the detection could be kept. These both weaken API security, so the reporting of them should be lasting and obvious.

Horizontal scaling

Is your feature request related to a problem? Please describe.
One of the requirements of our infrastructure is to have at least two application replicas in k8s or OpenShift cluster. At the moment this is not feasible because the speculator and the traceanalyzer module store data in memory. Also, the speculator saves its state locally to a file.

Describe the solution you'd like
Are there any plans to make the apiclarity horizontally scalable?

Describe alternatives you've considered
What are the alternative options to solve the horizontal scaling problem?

Helm chart versioning

We'll need a proper helm chart version going forward to support APIClarity upgrades: https://github.com/openclarity/apiclarity/blob/master/charts/apiclarity/Chart.yaml#L5

Allow deletion of Reconstructed API Specification

Once an reconstructed API Specification is created, in the API inventory screen there is no way to remove that reconstructed API specification in the event it is no longer valid, or, you want the engine to reconstruct it again.

Solution: Please provide a button to remove the reconstructed API and allow the engine to start over the reconstruction process.

Support docker-compose

Hi there,

I don't use k8s because it's a big platform for me. I want to start apiclarity+kong+kong-plugin in docker-compose.
But I don't know how to do it.

How to use apiclarity+kong+kong-plugin in docker? Is it possible?
Is there an installation instruction in docker-compose?

Thanks in advance.

When trace sampling is enabled, API event not recorded

What happened:

In trying to test out the BFLA analyzer, I enabled trace sampling in the APIClarity helm chart, then ran some application traffic. None of the API traffic was recorded in the API events table. Once I disabled trace sampling, API events started showing up in the table again.

What you expected to happen:

API event should have still been recorded when trace sampling was enabled.

How to reproduce it (as minimally and precisely as possible):

Enable trace sampling, run some application traffic, observe that no new API events are recorded.

Environment:

APIClarity v0.14.2

Cannot unmarshal object into Go struct field IngressSpec.spec.rules of type []v1.IngressRule

What happened:

While configuring kong gateway with apiclarity using steps given at
https://github.com/openclarity/apiclarity/tree/master/plugins/gateway/kong.

I have started sock shop demo on ubuntu 22 machine. And able to see traffic from sock shock
getting reflected in Api clarity. So this setup is done properly. After executing
kubectl apply -f https://bit.ly/kong-ingress-dbless successfully.

While Configure Ingress: I am getting below error,
Cannot unmarshal object into Go struct field IngressSpec.spec.rules of type []v1.IngressRule

What you expected to happen:

It should have applied as per steps given.

How to reproduce it (as minimally and precisely as possible):

Follow steps given at https://github.com/openclarity/apiclarity/tree/master/plugins/gateway/kong

Below yaml setting I used to configure ingress as per compatible apiVersion: networking.k8s.io/v1

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: catalogue
  namespace: sock-shop
spec:
  ingressClassName: kong
  rules:
    http:
      paths:
        - path: /
      pathType: Prefix
      backend:
        service:
          name: catalogue
          port:
            number: 80

Are there any error messages in API Clarity logs?

(e.g. kubectl logs -n apiclarity --selector=app=apiclarity)

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version --short):
  Client Version: v1.25.4
  Kustomize Version: v4.5.7
  Server Version: v1.25.3

• Istio version (use istioctl version):
  none

• APIClarity version (use kubectl -n apiclarity exec deploy/apiclarity -- ./backend version)
  latest from repo

• Cloud provider or hardware configuration:

• Others:

Kong API Gateway plugin:serviceaccount "apiclarity-apiclarity-kong-post-install-job"not found

Error reported in kong-post-install-job：
error looking up service account apiclarity2/apiclarity2-apiclarity-kong-post-install-job: serviceaccount "apiclarity2-apiclarity-kong-post-install-job" not found