curiefense/curiefense

Curiefense is a unified, open source platform protecting cloud native applications.

---

Curiefense is a new application security platform, which protects sites, services, and APIs. It extends Envoy proxy to defend against a variety of threats, including SQL and command injection, cross site scripting (XSS), account takeovers (ATOs), application-layer DDoS, remote file inclusion (RFI), API abuse, and more.

Getting Started

Documentation

• Curiefense Documentation
• Quick Start Guide
• FAQ

Docker

git clone https://github.com/curiefense/curiefense.git
cd curiefense/deploy/compose/
docker-compose up

Video Overview

Community

There are many ways to get involved with Curiefense.

• Twitter
• CNCF Community Group
• Slack

---

This project is named after the famous scientist Marie Salomea Skłodowska Curie. It began in intensive work sessions at Malakoff France, close to her home and laboratory in the outskirts of Paris, and is being released on her birthday (November 7th).

curiefense's Issues

"curieconfctl sync export" should request the server to export to the bucket

curieconfctl sync export tries to export to the bucket locally, rather than asking the server to do so. This breaks when using docker-compose, where by default the bucket is mounted to /bucket, which does not exist on the host.

See https://github.com/curiefense/curiefense/blob/master/curiefense/curieconf/client/curieconf/cli/__init__.py#L346

-> call the export API instead of calling the cloudstorage API

"restrict" parameter has no effect?

"restrict" parameter has no effect?
Default waf profile, active, with constraints as shown on the screenshot
WAF Parameter Constraint: header="name-restrict", value="value", restrict is checked
If I sent an invalid value, the request is not blocked. Reproducer:
requests.get('http://192.168.49.2:30081/', headers={"name-restrict": "invalid"})

Rate limit recommendation next level with charts

Tasker completion deployment

tasker that support two types of tasks.

1. update lists
2. analyze logs

Setup GitHub actions to automatically run curieconf tests

Align checkbox wrapper styling throughout the configuration client

There are many checkboxes throughout the configuration client, some of them use Bulma's wrapper class checkbox while others do not, we need to locate any checkbox input that doesn't have the wrapper and add it.
The wrapper is not intrusive as it attempts to preserve cross-platform compatibility and the user experience by only adding minor manipulations to the wrapper. e.g. "cursor: pointer"

https://bulma.io/documentation/form/checkbox/

Update Entry's attributes / properties

Often there is a need to set a subset of attributes or in some cases a single ine (e.g. "active": true).
Right now, the process is

1. GET X
2. update X localy
3. PUT X

Instead, I propose adding support to submit a JSON that will be merged into the existing entry.
This can be implemented on the same endpoint we have now, or in a new one e.g.

curl /configs/master/d/urlmaps/e/__default__/ \
             --data {"waf_active": true} -H "content-type: application/json"

OR

curl /configs/master/d/urlmaps/e/__default__/attr/ \
             --data {"waf_active": true} -H "content-type: application/json"

Passive challenge

we need to add passive challenge support.

Merge duplicated code for ax and preax into one file

• Move ax and preax functions to a separate file and use them in DocumentEditor, DBEditor, Publish, TagAutoComplete, and VersionControl
• Check for any loose API calls and switch them to use the new ax/preax functions
• Move the notification (toast) system to the same location as well

"curieconfctl conf get master" returns a file that is not a valid input for "curieconfctl conf create"

To Reproduce

curieconfctl conf get master > master.json
curieconfctl conf create -n test master.json

Result:

{
    "errors": {
        "meta.logs": "None is not of type 'array'",
        "delete_documents.limits": "None is not of type 'array'",
        "delete_documents.urlmaps": "None is not of type 'array'",
        "delete_documents.wafsigs": "None is not of type 'array'",
        "delete_documents.wafprofiles": "None is not of type 'array'",
        "delete_documents.aclprofiles": "None is not of type 'array'",
        "delete_documents.profilinglists": "None is not of type 'array'",
        "delete_blobs.geolite2asn": "None is not of type 'boolean'",
        "delete_blobs.geolite2country": "None is not of type 'boolean'"
    },
    "message": "Input payload validation failed"
}

Tested with version cada8ef

built-in automatic lists - uneditable

Assuming tasker is upcoming soon, we will need to set up the convention how to maintain a set of lists that will be visible in the system, yet not editable.

For instance, dynamic banning and releasing of IPs based on violations.

there are two fields which will rule this type of a list:
"id": "..."
and
"source": "..."

similar to the "source": "self-managed" we should have a value that reflects this status, perhaps automation or similar.

enhance rate limit editor

Changes
Left/Right columns
Left column containing: name, description, threshold, ttl
Right column containing: count by, event, action, include, exclude

New entity
blaclist/Whitelist -> Change to better terms blocklist/allowlist

TTL
Add suffix with units (seconds)
Note: needs to be added in flow control as well

DB Interface - from JSON dumps to structued UI

Either schema based UI or fixed for publishinfo, tags and tasks

Bootstrap data has two buckets named "prod"

Bootstrap data -> database master -> key publishinfo -> buckets
There are two buckets called prod. This is not a valid state and causes issues in Publish Configuration page

adding our dashboards to grafana community repository

https://grafana.com/grafana/dashboards

Add SECURITY.md

Add security disclosure policy

lua nil error

requests.get("http://192.168.49.2:30081/?TOTO") yields the following error each time

New bug: requests.get("http://192.168.49.2:30081/?TOTO") yields the following error each time:
[Envoy (Epoch 0)] [2020-12-19 15:28:31.341][29][error][lua] [external/envoy/source/extensions/filters/http/lua/lua_filter.cc:600] script log: ./lua/limit.lua:81: attempt to index local 'rule' (a nil value)
17:30

I have also seen this error several times, can be reproduced with requests.get("http://192.168.49.2:30081/", headers={"Host": "doesnotmatch"})
[Envoy (Epoch 0)] [2020-12-19 15:27:41.109][30][error][lua] [external/envoy/source/extensions/filters/http/lua/lua_filter.cc:600] script log: ./lua/limit.lua:81: attempt to index local 'rule' (a nil value)

Enable API validation

API validation has been disabled on release as it was causing issues. We need to reenable it and fix any issues it may have caused in the past
File 'api.py' Line 482

curiefense/curietasker:latest not found

Describe the bug
Pulling curietasker (curiefense/curietasker:latest)...
ERROR: manifest for curiefense/curietasker:latest not found: manifest unknown: manifest unknown

A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to curiefense/deploy/compose

2. docker-compose up

3. See error
   Run on MacOS

curieconf_cli rename as curiectl

WAF headers filter test are not blocked

I'm sending either an overlong header, or too many headers, but am not getting blocked
(xavier)

Add ROADMAP.md

Add a public roadmap

WAF False positive automation

Secure a CII Best Practices Badge and add to README

https://bestpractices.coreinfrastructure.org/en

Tag Rules - new entry component

When creating a new entry under profiling lists (Tag Rules) we have a generic component with category + entry value, we would like to create a component that changes according to the selected category

Annotation should be in a different input field while still working with [value#annotation] in the value field
Annotation should take inner value annotation [value#annotation] over the annotation field [value] [annotation]
Value field should be split to two if category's values are pairs (e.g. category is args, cookies, headers)
Value should be validated using a regex according to category

our charts in helm hub

DB and Config unification

We started with config, then added DB.
Given both are going to expand with new types and uses, perhaps we can unify them as simply config, where types are defined by a JSON schema, and can be added dynamically by simply adding a new schema.

Test test_non_allowlisted_value_norestrict_wafmatch_excludesig[params-regex-no_ignore_alphanum] fails

Version: 1157774

To reproduce on a minikube deployment:

export IP=$(minikube ip); pytest --log-level INFO --base-protected-url http://$IP:30081 --base-conf-url http://$IP:30000/api/v1/ --base-ui-url http://$IP:30080 -k 'test_non_allowlisted_value_norestrict_wafmatch_excludesig[params-regex-no_ignore_alphanum]' .

Observed error logs for queries associated to this test: none

If I follow the diagram on the documentation correctly, this query should follow this path:

• the regex-norestrict argument has a defined contraint
• the value it receives, htaccess, does NOT match the Matching Value pattern [v]+[a]{1}l?u*e
• WAF Signatures (now "WAF Rules") are evaluated, except rule 100140 which is in "Exclude Sigs"
• The "outcome of the evaluation" should be PASS; it is fail.
• The access log page shows that this request has matched sig_id 100140; it has the wafsig:100140 tag, even though it should not have been evaluated

=> This request is rejected, but should not be

Curielogger support for Elastic

There is a great demand from whomever I speak to add this DB support.

Tests test_ratelimit_scope_tags_* fail

Version: ba04fdd

To reproduce on a minikube deployment:

export IP=$(minikube ip); pytest --log-level INFO --base-protected-url http://$IP:30081 --base-conf-url http://$IP:30000/api/v1/ --base-ui-url http://$IP:30080 -k 'test_ratelimit_scope_tag' .

Observed error logs for queries associated to this test:

[Envoy (Epoch 0)] [2020-12-21 14:02:04.194][33][error][lua] [external/envoy/source/extensions/filters/http/lua/lua_filter.cc:600] script log: ./lua/limit.lua:41: subject has no topointer method

Libinjection integration

Optimize postgres for fast insetion

FROM https://gist.github.com/valyala/ae3cbfa4104f1a022a2af9b8656b1131

Create UNLOGGED table. This reduces the amount of data written to persistent storage by up to 2x.
Set WITH (autovacuum_enabled=false) on the table. This saves CPU time and IO bandwidth on useless vacuuming of the table (since we never DELETE or UPDATE the table).
Insert rows with COPY FROM STDIN. This is the fastest possible approach to insert rows into table.
Minimize the number of indexes in the table, since they slow down inserts. Usually an index on time timestamp with time zone is enough.
Add synchronous_commit = off to postgresql.conf.
Use table inheritance for fast removal of old data:

CREATE TABLE parent ... ; 

CREATE TABLE child_1() INHERITS (parent); 
CREATE TABLE child_2() INHERITS (parent);  

-- always INSERT rows into child_1. 
-- SELECT from parent.  

-- periodically run the follwing sql for rotating child_1 with child_2: 

TRUNCATE TABLE child_2; 
BEGIN; 
ALTER TABLE child_1 RENAME TO child_tmp; 
ALTER TABLE child_2 RENAME TO child_1; 
ALTER TABLE child_tmp RENAME TO child_2; 
COMMIT;

This is much faster comparing to

DELETE FROM parent WHERE time < now() - interval 'given period'

This also avoids table fragmentation, so SELECT queries work faster on the table.

session flow control

new document type named: flowcontrol

structure (draft):

{
	"id": "d45gai67",
	"name": "login",
	"sequence":[
		["GET /login", "HEAD /login", "GET /signup", "GET /index.html"],
		["OPTIONS sub.domain.com/ajax-cors"],
		["GET /login.js"],
		["POST /api/login"]
	]
}

Doc Editor std UI --
Note each entry in the sequence might contain multiple elements.

Add conf client tests to github workflow

We currently run python conf server tests on each new PR, we should add the conf client (non py) tests as well

curieproxy-envoy fails to process when profiling list - header matches

Describe the bug
Version used: docker tag 7a19c72b7752

Using docker-compose up and going through the steps in https://docs.curiefense.io/installation/getting-started-with-curiefense I stumbled upon the issue that when curieproxy matches a header rule in a profiling list, it throws the error:

[error][lua] [source/extensions/filters/http/lua/lua_filter.cc:683] script log: ./lua/tagprofiler.lua:51: bad argument #2 to 're_match' (string or rex_pcre2_regex expected, got nil)

And fails to further process the request (I do get a response, but the request is not logged as there's no meta data). I found that I can even reproduce it in a fresh setup with:

curl http://curie.demo:30081/with/header2 -H content-type:application/json

When running the curielogger in debug mode, I see:

curielogger       | 2020/11/20 14:44:48 [DEBUG] ====>[&{log_entry:<common_properties:<downstream_remote_address:<socket_address:<address:"172.19.0.1" port_value:47538 > > downstream_local_address:<socket_address:<address:"172.19.0.3" port_value:80 > > start_time:<seconds:1605883488 nanos:103722000 > time_to_last_rx_byte:<nanos:51600 > time_to_first_upstream_tx_byte:<nanos:26714300 > time_to_last_upstream_tx_byte:<nanos:26730100 > time_to_first_upstream_rx_byte:<nanos:27768900 > time_to_last_upstream_rx_byte:<nanos:27981600 > time_to_first_downstream_tx_byte:<nanos:27953200 > time_to_last_downstream_tx_byte:<nanos:28014400 > upstream_remote_address:<socket_address:<address:"172.19.0.2" port_value:8080 > > upstream_local_address:<socket_address:<address:"172.19.0.3" port_value:53392 > > upstream_cluster:"target_site" downstream_direct_remote_address:<socket_address:<address:"172.19.0.1" port_value:47538 > > > protocol_version:HTTP11 request:<request_method:GET scheme:"http" authority:"curie.demo:30081" path:"/with/header2" user_agent:"curl/7.64.1" forwarded_for:"172.19.0.1" request_id:"8f961363-0620-4101-8d72-c34755c7d989" request_headers_bytes:272 > response:<response_code:<value:200 > response_headers_bytes:123 response_body_bytes:333 response_code_details:"via_upstream" > > }]
curielogger       | 2020/11/20 14:44:48 [DEBUG] ---> [ 172.19.0.2:8080 172.19.0.3:53392 ] <---
curielogger       | 2020/11/20 14:44:48 [DEBUG] No curiefense metadata => drop log entry

Align conf-client document editor design

In the configuration client, under document editor, we have a few different document types

We need to align their implementation to match each other as much as possible:
Some of them have a "name" label above the "name" input and some do not. They should all have it
Some of them have an "ID" label above the "name" input, others have it below, and some to the side. They should all have it in the same line as the "name" label aligned to the right

Tags autocomplete input new tags issues

• When using the tag autocomplete input with the 'multiple' setting, it does not save unknown tags added when clicking space and only when clicking enter. tags should be submitted with space as well as this is an indicator of a new tag start

• When using the tag autocomplete and inserting new tags, they do not show up in the dropdown until the component is reloaded, the new tags should be available immediately

Redis access optimization

• Adapt strategy we implemented at sophie project
• optimize for single access per request for multiple rules

Make curiefense avaialble as GCP GKE Marketplace deployment

There is an account we are running with the market place already.
Will send all informaiton over slack.

Extending luasockets for IPv6 operations (conversion etc.)

Having no entity breaks the user's flow

When we have no data on a specific entity (for example after the user deletes all flowcontrol through the API) the UI breaks, we should display a user friendly message

Client does not have any tests

The Vue.js client should have unit tests for easier maintenance of code

Add scrollbar (virutal) to tag autocomplete

When there are many tags in autocomplete we currently display all of them, we can add a virutal scrollbar to enhance the user experience

Manually updating profiling list using the 'update now' button does not work properly

Reported by @tzuryby

Change source of list profiling
Clicked 'update now' button (next to last update)
nothing shows up in the UI

Make curiefense avaialble as AWS EKS Marketplace deployment

AWS user sent separatly.
https://aws.amazon.com/marketplace/management/

single script installation `curl | sh` style

we will need to add an option which does not include git clone
assuming downloading an arhcive of label x will be good.
I suggest creating an .sh file that

1. downloads the archive
2. extract it
3. runs docker-compose up from deploy/compose

Header parser does not seems to work properly

First of all, great project!!

I was following the Quick start guide, creating the Header profiling list.

While testing this, in a new install with docker-compose, the requests doesn't get tagged when using the Header validation.

I've seen that the response from the server is always with the Header with the first character capitalized (Foo), so I've tested both with capitalized and not.

Strangely, when following exactly what's in the quick start, no log even appear when this rule is created.

So I've tested with my own IP in the profiling list, and it works fine (the query gets tagged, and further I can block it), so I'm wondering if this is something with the header parsing.

Thank you!

More charts in Grafana

Add `AND` support in profiling lists entries relations

right now we only support "OR" relations between enties.
we must add "AND", and perhaps blocks, e.g.

(
    {}
  OR
    {}
)
AND
(
    {}
  OR
    {}
)

Add TLS support for istio service in Helm deployments