opencspm/opencspm

Open Cloud Security Posture Management Engine

OpenCSPM

Open Cloud Security Posture Management, OpenCSPM, is an open-source platform for gaining deeper insight into your cloud configuration and metadata to help understand and reduce risk over time.

Who is OpenCSPM for?

• Security teams running infrastructure in cloud environments looking to gain visibility into security risk across the entire account and track the resolution of those issues.
• Development and Infrastructure teams looking for a set of security best-practices to follow as they develop new systems and bake security in early and often.
• Red Teams wanting to create targeted scenarios specific to their environment with the highest impact and benefit to their Blue Team.
• Any organization looking to establish a baseline of security issues, prioritize them by risk, and track them to closure over time.

What does OpenCSPM do?

When deployed inside your environment, it can:

1. collect several types of cloud configuration data on a one-time or recurring basis from your cloud account resources (VMs, Clusters, IAM, etc),
2. parse and load that data into a graph database with deep linked relationships between resources to support advanced querying capabilities,
3. run a customizable series of policy checks to determine conformance and record passing/failing resources on a recurring basis,
4. create custom groupings of related policy checks that aid in tracking remediation efforts and reduction in risk over time,
5. and provide notifications when deviations from desired baselines occur to multiple destinations.

High-level Architecture

OpenCSPM has several key components that work together:

1. Structured configuration and metadata is collected from the GCP Cloud Asset Inventory and Kubernetes API to a Storage Bucket or local filesystem.
2. The data is parsed and loaded into a Graph Database with complex relationships intact.
3. One or more customizable sets of policy checks ("control packs") are run against the parsed inventory data, and results are stored.
4. The API/Engine serves up the user interface, handles tracking of campaigns, and notifications.

Getting Started

Ready to get going? Follow the instructions for getting started or watch the following getting started video:

Support and Community

OpenCSPM is supported by community development. If you have found an issue, please file it using a GitHub issue.

Contributing

We invite you to submit requests for new features and bug reports using a Github issue. If you would like to contribute to OpenCSPM, please review our development guide.

Creators

OpenCSPM was made open-source software by Josh Larsen and Brad Geesaman, the co-founders of Darkbit. Darkbit is a cloud-native security assessment consulting firm based in the US that helps organizations understand the risks in their cloud and Kubernetes resource configurations.

opencspm's Issues

docker-compose error on 1.26.2

Describe the bug

$ docker-compose up
ERROR: The Compose file './docker-compose.yml' is invalid because:                                                                                                                      
Invalid top-level property "x-app". Valid top-level sections for this Compose file are: version, services, networks, volumes, and extensions starting with "x-".                        
                                                                                                                                                                                        
You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the
 `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.                                                                 
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/                                                                                         
services.worker.depends_on contains an invalid type, it should be an array                                                                                                              
services.core.depends_on contains an invalid type, it should be an array 

Expected behavior
A clear and concise description of what you expected to happen.

Docker information (please complete the following information):

$ docker --version
Docker version 19.03.13, build 4484c46d9d
$ docker-compose --version
docker-compose version 1.26.2, build eefe0d31

Additional context

Although docker-compose 1.26.2 is not the latest version (and upgrading to 1.27.4 does fix the issue), 1.26.2 is only a few months old so I would expect lots of people are still running it

OpenCSPM UI Not Reflecting Config.yaml Changes Without Restarting

Describe the bug

Having followed instructions at https://github.com/OpenCSPM/opencspm/blob/main/site/data_collection.md#aws-cloud-resources, OpenCSPM does not pick up my custom data.

# ls -l assets/custom
total 316
-rw-r--r-- 1 christophetd christophetd     12 nov.  16 12:35 manifest.txt
-rwxr-xr-x 1 christophetd christophetd 316195 nov.  16 12:35 output.json

$ cat assets/custom/manifest.txt
output.json

$ cat config/config.yaml
---
db:
  host: redis
  port: 6379
buckets:
  # - gs://darkbit-collection-us-cspm
  # - s3://my-other-bucket-here
local_dirs:
  - /app/data/custom
  # - /app/data/test

I did run docker-compose down && docker-compose up after making these modifications.

No visible error in logs

Expected behavior

OpenCSPM should pick up the new resources file

Screenshots

UI:

Docker information (please complete the following information):

$ docker --version
Docker version 19.03.13, build 4484c46d9d

$ docker-compose --version
docker-compose version 1.27.4, build 40524192

Cloud provider (if applicable):

AWS

[jobs] handle background job errors

Background jobs that fail are not indicated in the UI. Also prevents future jobs from running.

[dev] improve worker startup to wait for bundle install to finish

Describe the bug
In the development environment, the worker container fails to start on the initial docker-compose up. This is because bundle hasn't finished installing gems yet.

To Reproduce
Steps to reproduce the behavior:

1. cd docker
2. docker-compose -f docker-compose-development.yml up

Observe repeated errors and restarts from the worker container.

worker_1    | bundler: command not found: sidekiq
worker_1    | Install missing gem executables with `bundle install`
worker_1    | bundler: command not found: sidekiq
worker_1    | Install missing gem executables with `bundle install`
opencspm_worker_1 exited with code 127
worker_1    | bundler: command not found: sidekiq
worker_1    | Install missing gem executables with `bundle install`

Expected behavior
Docker containers should start without error.

Additional context
It doesn't have any adverse effects, but is a poor development experience.

opencspm install question

Hello, I installed opencspm through the introductory guide. The console information is as follows, and the web browser cannot be accessed. What is the problem?

aws_recon: Loader job failed

I am trying to use my own AWS custom data but keep getting erros like

worker_1  | Loading /app/load_dir/combined_for_load.json
worker_1  | Loader job failed - TypeError nil is not a symbol nor a string (/app/app/jobs/lib/loader/asset/loaders/aws_loader/_resource_loader.rb:29:in `respond_to?')
worker_1  | Runner job failed - TypeError nil is not a symbol nor a string (/app/app/jobs/lib/loader/asset/loaders/aws_loader/_resource_loader.rb:29:in `respond_to?')
worker_1  | I, [2021-04-25T17:29:44.756110 #1]  INFO -- : [ActiveJob] [RunnerJob] [d0a242c6-7142-4601-8fcd-3249fcf0893a] Performed RunnerJob (Job ID: d0a242c6-7142-4601-8fcd-3249fcf0893a) from Sidekiq(default) in 180.72ms

I have made sure the manifest is correct, and if i paste the demo data content into the output.json it works. However, when I create my own output.json with aws_recon I keep getting the loader job failed.

I am using custom format and output to newline delimited JSON

actually it seems to be related to specific components only, I was able to get it working by excluding the ones with errors.

the following worked without errors.

aws_recon -x Route53Domains,Lambda,DirectoryService -l -r global,us-east-1 -f custom

Any ideas why these 3 services cause errors like..

for example if i remove Lambda...

aws_recon -x Route53Domains,DirectoryService -l -r global,us-east-1 -f custom

I get

worker_1  | Loader job failed - NoMethodError undefined method `vpc_id' for "{:subnet_ids=>[], :security_group_ids=>[], :vpc_id=>\"\"}":String (/app/app/jobs/lib/loader/asset/loaders/aws_loader/lambda.rb:18:in `function')

or if i removed DirectoryService

aws_recon -x Route53Domains,Lambda -l -r global,us-east-1 -f custom

i get ...

worker_1  | Warning: no SecretsManager loader defined for asset type: secret
worker_1  | 
worker_1  | Warning: no SecretsManager loader defined for asset type: secret
worker_1  | Loader job failed - NameError uninitialized constant AWSLoader::DirectoryService (/app/app/jobs/lib/loader/asset/loaders/aws_loader/_resource_loader.rb:25:in `const_get')
worker_1  | Runner job failed - NameError uninitialized constant AWSLoader::DirectoryService (/app/app/jobs/lib/loader/asset/loaders/aws_loader/_resource_loader.rb:25:in `const_get')
worker_1  | I, [2021-04-25T19:10:46.936245 #1]  INFO -- : [ActiveJob] [RunnerJob] [a805f357-3064-493a-bc9c-12426f846be1] Performed RunnerJob (Job ID: a805f357-3064-493a-bc9c-12426f846be1) from Sidekiq(default) in 4959.08ms

I'v tried both the gem (aws_recon (0.5.7)), and docker (latest) for running aws_recon.

Any help would be appreciated.

[EC2] load network_acl, internet_gateway, and ebs_encryption_settings

worker_1 | No EC2 loader defined for asset type: network_acl
worker_1 | No EC2 loader defined for asset type: network_acl
worker_1 | No EC2 loader defined for asset type: internet_gateway
worker_1 | No EC2 loader defined for asset type: ebs_encryption_settings
worker_1 | No EC2 loader defined for asset type: network_acl
worker_1 | No EC2 loader defined for asset type: internet_gateway

[loader] fail with useful error message when not NDJSON

In asset_router.rb, detect JSON vs NDJSON import and raise an exception with a useful error message instead of NoMethodError undefined method keys' for #<Array:0x000055c06fb0e1d8>

opencspm install question

Hello, I installed opencspm through the introductory guide. The console information is as follows, and the web browser cannot be accessed. What is the problem?

Docker deployment setup of opencspm on non-localhost endpoint

I tried deploying the opencspm application using docker-compose on my test VM (Ubuntu), the application doesn't get deployed on a VM. The same docker-compose works on my local laptop and accessible at localhost:5000 end point.

Spend some time on the source code and it looks like the localhost url is hardcoded in their source code. Tried changing the url in their source code but it didn't work and also tried the configuration of IP Address in the environment configuration as suggested in below error, but the application is still not accessible on the IP Address of a VM.
Any inputs / suggestion will be appreciated
Error:

Potential typo

There is a potential typo in

opencspm/site/img/high-level-arch.excalidraw

Line 3743 in 5795251

"text": "SEIM",

My guess is that it should be SIEM instead of SEIM.

How is data collected from cloud providers?

Currently only GCP is supported?
I expected to see here https://github.com/OpenCSPM/opencspm/tree/main/collection AWS, Azure... or at the moment it is possible to work only with one GCP provider?

Tell me please:

1. how does the provider status survey work?
2. What do you use terraform for?

Thanks!)

Dashboard Status Inactive and Controls Missing

Describe the bug
i have tried to use your tool but when i open the dashboard it gives me inactive and doesn't show data even from the demo you provided. I followed all of the steps in the guide with aws-recon and aws-vault and got the data in a json file and configured the config.yaml file and the cli shows the worker service showing that the job was successfully done.

Expected behavior
The profile section should be active instead of inactive and the controls should be displayed

Screenshots

Cloud provider (if applicable):
i am using aws cloud

[api] remove static index.html from API server

index.html is no longer served from the API server. Remove it.

[ApplicationAutoScaling] graph db loader

Missing loader for ApplicationAutoScaling.

Dear OpenCSPM,

I am rather new to this Docker stuff, and my question may be just my lack of knowledge around the platform. In short, when I create the custom JSON file, extract to the folder specified for OpenCSPM to import. I then attempt to reload the data (via web console / docker-compose down/up) I get the following error.

worker_1 | Warning: no Backup loader defined for asset type: protected_resource
worker_1 | Loader job failed - NameError uninitialized constant AWSLoader::ApplicationAutoScaling (/app/app/jobs/lib/loader/asset/loaders/aws_loader/_resource_loader.rb:25:in const_get') worker_1 | Runner job failed - NameError uninitialized constant AWSLoader::ApplicationAutoScaling (/app/app/jobs/lib/loader/asset/loaders/aws_loader/_resource_loader.rb:25:in const_get')
worker_1 | I, [2021-04-21T14:06:28.283442 #1] INFO -- : [ActiveJob] [RunnerJob] [f82dc9b3-19f5-485b-b664-f60a937113ed] Performed RunnerJob (Job ID: f82dc9b3-19f5-485b-b664-f60a937113ed) from Sidekiq(default) in 44876.54ms
worker_1 | 2021-04-21T14:06:28.295Z pid=1 tid=gp5k0oly1 class=RunnerJob jid=535e242bd81a6098d34bbb62 elapsed=44.947 INFO: done

I hope this is just my lack of knowledge around the dependencies of the tool and you all can provide some insight as to remedy.

On another note, tool is AWESOME and one of the easiest to get up and working from other tools I've used. Thank you all for creating this and hopefully my issue is easily resolve with a few commands on my part.

Thanks in advance!

William

Originally posted by @Nebula-Ninjas in #152

Bug : Profiles showing as "Inactive" and no controls are shown in the dashboard.

I followed the process as mentioned in the video and have been tracking docker-compose logs and all to check if there has been any error.
However , the logs were clear.

Once everything is loaded. I do get the profiles listed in the profiles page with status being inactive.

No controls are showing on the dashboard.

Image below :-

I have loaded the community and enterprise controls both as suggested in the video.

My directory structure below for reference.

├── opencspm
│   ├── assets
│   ├── collection
│   ├── config
│   ├── docker
│   ├── load_dir
│   ├── site
│   └── terraform
└── opencspm-controls
    ├── opencspm-darkbit-community-controls
    └── opencspm-darkbit-enterprise-controls

Error: services.core.depends_on contains an invalid type, it should be an array

Getting the following error when doing a docker-compose up:

ERROR: The Compose file './docker-compose.yml' is invalid because:
services.core.depends_on contains an invalid type, it should be an array
services.worker.depends_on contains an invalid type, it should be an array

Taking a deeper look, it seems related to Version 3 of docker no longer supporting the condition form of depends_on.

From: https://docs.docker.com/compose/compose-file/

There are several things to be aware of when using depends_on:

- depends_on does not wait for db and redis to be “ready” before starting web - only until they have been started. If you need to wait for a service to be ready, see Controlling startup order for more on this problem and strategies for solving it.
- Version 3 no longer supports the condition form of depends_on.
- The depends_on option is ignored when deploying a stack in swarm mode with a version 3 Compose file.

To Reproduce
docker-compose up

Docker information (please complete the following information):
Docker version 19.03.6, build 369ce74a3c
docker-compose version 1.25.5, build 8a1c60f6

Additional context
Not really sure how others are not getting this error since this has been in there for 2 months, so not sure what I'm doing wrong. Removing depends_on does get docker-compose to start, but had to run twice. :) Searched to see if this existed before and didn't see, but forgive me if I'm missing something. Others getting this error?

Fix
Removed condition and changed to arrays:

depends_on:
      - core
      - db
      - redis

depends_on:
      - db
      - redis

I can put an PR for this if you like. Lmk.

Controls are not executed

Describe the bug

I followed all the instructions but when loading the interface I don't see any control failling but there should be some failing (for example darkbit-aws-61).

To Reproduce

No sure what the problem is since the output.json appear to contain all the necessary data and the the UI says that it loaded my custom data set.

Expected behavior

There should be multiple control failing.

Screenshots

Client: Docker Engine - Community
 Cloud integration: 1.0.3
 Version:           20.10.0-rc1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        5cc2396
 Built:             Tue Nov 17 22:49:30 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0-rc1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       131bf7e
  Built:            Tue Nov 17 22:52:57 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.4.1
  GitCommit:        c623d1b36f09f8ef6536a057bd658b3aa8632828
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020

Cloud provider (if applicable):

AWS

Additional context

I'm not sure what info would be relevant to troubleshoot this issue.

FYI: The link to join the slack channel is no longer valid. Maybe you should consider using Discord :)

Docker compose error for complete setup

Describe the bug
While trying to run the full environment locally on Windows, the following error regarding the vim package is occurring.

error:
=> ERROR [ 7/12] RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && DEBIAN_FRONTEND=noninteractive 19.2s

OpenCSPM version
0.1.23

To Reproduce
Steps to reproduce the behavior:

1. Go to 'docker directory
2. run "docker-compose -f docker-compose-development.yml up --build"

Screenshots

Docker information (please complete the following information):

• Output of full docker version (not docker -v)

• Output of full docker-compose version (not docker-compose -v)

D:\OpenCSPM\opencspm-demo\opencspm\docker> docker-compose -f docker-compose-development.yml up
[+] Running 0/4

• core Error 6.0s
• ui Error 6.0s
• shell Error 6.0s
• worker Error 5.0s
  [+] Building 23.8s (12/17)
  => [internal] load build definition from Dockerfile.development 0.0s
  => => transferring dockerfile: 44B 0.0s
  => [internal] load .dockerignore 0.0s
  => => transferring context: 2B 0.0s
  => [internal] load metadata for docker.io/library/ruby:2.7.3-slim-buster 4.5s
  => [auth] library/ruby:pull token for registry-1.docker.io 0.0s
  => [internal] load build context 0.0s
  => => transferring context: 28B 0.0s
  => [ 1/12] FROM docker.io/library/ruby:2.7.3-slim-buster@sha256:360b27772b1a4a4b25395c5e2a8225ab70ea61a29c0fd62cd8705d6ab1417de4 0.0s
  => CACHED [ 2/12] RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends build-esse 0.0s
  => CACHED [ 3/12] RUN curl -sSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - && echo 'deb http://apt.postgresql 0.0s
  => CACHED [ 4/12] RUN curl -sL https://deb.nodesource.com/setup_12.x | bash - 0.0s
  => CACHED [ 5/12] RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && echo 'deb http://dl.yarnpkg.com/debian/ st 0.0s
  => CACHED [ 6/12] COPY Aptfile /tmp/Aptfile 0.0s
  => ERROR [ 7/12] RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && DEBIAN_FRONTEND=noninteractive 19.2s

---

[ 7/12] RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends libpq-dev postgresql-client-13 nodejs yarn=1.22.5-1 $(cat /tmp/Aptfile | xargs) && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && truncate -s 0 /var/log/*log:
#12 6.688 Reading package lists...
#12 7.140 Building dependency tree...
#12 7.243 Reading state information...
#12 7.259 Calculating upgrade...
#12 7.419 The following packages will be upgraded:
#12 7.420 base-files debconf libssl-dev libssl1.1 libsystemd0 libudev1 linux-libc-dev
#12 7.420 openssl tzdata
#12 7.634 9 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
#12 7.634 Need to get 6624 kB of archives.
#12 7.634 After this operation, 26.6 kB of additional disk space will be used.
#12 7.634 Get:1 http://deb.debian.org/debian buster/main amd64 base-files amd64 10.3+deb10u11 [69.9 kB]
#12 7.866 Get:2 http://deb.debian.org/debian buster/main amd64 debconf all 1.5.71+deb10u1 [145 kB]
#12 7.987 Get:3 http://deb.debian.org/debian buster/main amd64 libsystemd0 amd64 241-7deb10u8 [331 kB]
#12 8.152 Get:4 http://deb.debian.org/debian buster/main amd64 libudev1 amd64 241-7deb10u8 [151 kB]
#12 8.254 Get:5 http://deb.debian.org/debian buster-updates/main amd64 tzdata all 2021a-0+deb10u3 [266 kB]
#12 8.369 Get:6 http://deb.debian.org/debian buster/main amd64 libssl-dev amd64 1.1.1d-0+deb10u7 [1795 kB]
#12 9.085 Get:7 http://deb.debian.org/debian buster/main amd64 libssl1.1 amd64 1.1.1d-0+deb10u7 [1539 kB]
#12 9.640 Get:8 http://deb.debian.org/debian buster/main amd64 linux-libc-dev amd64 4.19.208-1 [1482 kB]
#12 10.24 Get:9 http://deb.debian.org/debian buster/main amd64 openssl amd64 1.1.1d-0+deb10u7 [845 kB]
#12 10.67 debconf: delaying package configuration, since apt-utils is not installed
#12 10.69 Fetched 6624 kB in 3s (2129 kB/s)
(Reading database ... 14668 files and directories currently installed.)
#12 10.76 Preparing to unpack .../base-files_10.3+deb10u11_amd64.deb ...
#12 10.78 Unpacking base-files (10.3+deb10u11) over (10.3+deb10u10) ...
#12 10.90 Setting up base-files (10.3+deb10u11) ...
#12 10.91 Installing new version of config file /etc/debian_version ...
(Reading database ... 14668 files and directories currently installed.)
#12 11.06 Preparing to unpack .../debconf_1.5.71+deb10u1_all.deb ...
#12 11.08 Unpacking debconf (1.5.71+deb10u1) over (1.5.71) ...
#12 11.50 Setting up debconf (1.5.71+deb10u1) ...
(Reading database ... 14668 files and directories currently installed.)
#12 11.65 Preparing to unpack .../libsystemd0_241-7deb10u8_amd64.deb ...
#12 11.67 Unpacking libsystemd0:amd64 (241-7deb10u8) over (241-7deb10u7) ...
#12 11.74 Setting up libsystemd0:amd64 (241-7deb10u8) ...
(Reading database ... 14668 files and directories currently installed.)
#12 11.79 Preparing to unpack .../libudev1_241-7deb10u8_amd64.deb ...
#12 11.81 Unpacking libudev1:amd64 (241-7deb10u8) over (241-7deb10u7) ...
#12 11.86 Setting up libudev1:amd64 (241-7deb10u8) ...
(Reading database ... 14668 files and directories currently installed.)
#12 11.92 Preparing to unpack .../tzdata_2021a-0+deb10u3_all.deb ...
#12 11.94 Unpacking tzdata (2021a-0+deb10u3) over (2021a-0+deb10u1) ...
#12 14.48 Preparing to unpack .../libssl-dev_1.1.1d-0+deb10u7_amd64.deb ...
#12 14.49 Unpacking libssl-dev:amd64 (1.1.1d-0+deb10u7) over (1.1.1d-0+deb10u6) ...
#12 15.06 Preparing to unpack .../libssl1.1_1.1.1d-0+deb10u7_amd64.deb ...
#12 15.07 Unpacking libssl1.1:amd64 (1.1.1d-0+deb10u7) over (1.1.1d-0+deb10u6) ...
#12 15.29 Preparing to unpack .../linux-libc-dev_4.19.208-1_amd64.deb ...
#12 15.30 Unpacking linux-libc-dev:amd64 (4.19.208-1) over (4.19.194-2) ...
#12 18.03 Preparing to unpack .../openssl_1.1.1d-0+deb10u7_amd64.deb ...
#12 18.05 Unpacking openssl (1.1.1d-0+deb10u7) over (1.1.1d-0+deb10u6) ...
#12 18.16 Setting up libssl1.1:amd64 (1.1.1d-0+deb10u7) ...
#12 18.23 Setting up linux-libc-dev:amd64 (4.19.208-1) ...
#12 18.25 Setting up tzdata (2021a-0+deb10u3) ...
#12 18.32
#12 18.32 Current default time zone: 'Etc/UTC'
#12 18.32 Local time is now: Thu Dec 30 09:11:53 UTC 2021.
#12 18.32 Universal Time is now: Thu Dec 30 09:11:53 UTC 2021.
#12 18.32 Run 'dpkg-reconfigure tzdata' if you wish to change it.
#12 18.32
#12 18.35 Setting up libssl-dev:amd64 (1.1.1d-0+deb10u7) ...
#12 18.36 Setting up openssl (1.1.1d-0+deb10u7) ...
#12 18.38 Processing triggers for libc-bin (2.28-10) ...
#12 18.48 Reading package lists...
#12 18.97 Building dependency tree...
#12 19.07 Reading state information...
#12 19.13 E: Unable to locate package vim

Cloud provider (if applicable):

• e.g. AWS/GCP/Azure

Additional context
Add any other context about the problem here.

Unable to access UI from remote machine

Hello guys.
I'm trying to launch OpenCSPM in local mode on my Linux virtual machine. It looks like an applications starts correctly, but when I try to access Web UI from web browser on my personal computer I get the following error:

Blocked host:
To allow requests to , add the following to your environment configuration:
config.hosts << ""

Unfortunately I was unable to find instructions on how to fix this. General googling suggested to add line "config.hosts << """ to the file opencspm-demo/opencspm/docker/config/environments/development.rb, but it doesn't seem help.

If you advise on how to fix this I would appreciate it.

[db] upgrade PostgreSQL docker image to 13.1

Upgrade to address https://nvd.nist.gov/vuln/detail/CVE-2020-25695.

[poll] Campaign notifications

This issue is to track notification preferences for campaigns. We are considering adding a number of notification channels/destinations. Help us rank and prioritize notification methods we should focus on first.

React with a 👍🏼 on the comment below for the most important channel to you.

Thanks!

(FATAL: database "opencspm_production" does not exist) when i try do sign in on localhost home page

Describe the bug
After I normally install OpenCSPM with Docker and start it, I accessed the localhost home page, and when I press the "Demo Admin Sign in" button, the following error occurs, apparently related to a missing database::

core_1    | I, [2021-08-05T16:35:44.043332 #9]  INFO -- : [0643ecde-651b-42bc-88c6-70f36762c02b] method=GET path=/api/sessions format=html controller=Api::SessionsController action=index status=401 duration=1.45 view=0.27
db_1      | 2021-08-05 16:35:46.194 UTC [564] FATAL:  database "opencspm_production" does not exist
core_1    | I, [2021-08-05T16:35:46.196979 #9]  INFO -- : [30b83d46-c50b-4d9a-b432-1dfc9fe55ba0] method=POST path=/api/sessions format=html controller=Api::SessionsController action=create status=500 error='ActiveRecord::NoDatabaseError: FATAL:  database "opencspm_production" does not exist
core_1    | ' duration=318.39 view=0.00
core_1    | F, [2021-08-05T16:35:46.198896 #9] FATAL -- : [30b83d46-c50b-4d9a-b432-1dfc9fe55ba0]   
core_1    | [30b83d46-c50b-4d9a-b432-1dfc9fe55ba0] ActiveRecord::NoDatabaseError (FATAL:  database "opencspm_production" does not exist
core_1    | ):
core_1    | [30b83d46-c50b-4d9a-b432-1dfc9fe55ba0]   
core_1    | [30b83d46-c50b-4d9a-b432-1dfc9fe55ba0] app/controllers/api/sessions_controller.rb:16:in `create'

Also, when I click on the Google and GitHub auth buttons on the home page, it takes me to the Heroku website and shows me an Application Error, but I believe the two issues are related.

OpenCSPM version
0.1.23

Screenshots

Docker information (please complete the following information):
20.10.2
docker-compose: 1.26.0

images pull question

Hello，can you push inages to hub.docker.com?
ths.
[root@control-plane docker]# cat docker-compose.yml |grep gcr.io
image: gcr.io/opencspm/engine:0.1.19
image: gcr.io/opencspm/postgres:13.1
image: gcr.io/opencspm/redisgraph:edge-0.0.1
[root@control-plane docker]#

all hosts blocked except localhost on Rails

Describe the bug
I'm trying to use the application inside a GCP Ubuntu VM, applying it to the local IP address and then using IP forwading to redirect external traffic to it

The problem is that, doing the test both in GCP and locally, the problem is always the same, I can't get the application to work with the local IP address instead of localhost because apparently Rails blocks any hosts other than localhost, as shown in the screenshot.

I've tried numerous ways to solve this problem, following some tips on stackoverflow and some tips in this issue, such as:

1. in the config/initializers/custom.rb file, I tried to add the lines config.hosts.clear, config.hosts << "192.168.0.14", config.hosts << "my_hostname_name"

2. in the config/enviroments/development.rb file, I tried to change the ui_host and cookie_domain lines to my hostname and local IP address

3. in the config/application.rb file, I tried to add the Rails.application.config.hosts.clear line to release all hosts

i also asked the firewall to accept all necessary ports like 5000, 8000 and 5432, just in case

and after each change, I reinstalled the entire application using docker-compose down -v and docker-compose up, but none of the alternatives, and in none of the environments did it solve the problem.

is there any way I can solve this? access my application via http://my_local_ip:5000 instead of http://localhost:5000?

OpenCSPM version
0.1.23

Expected behavior
access the application via my local ip address instead of localhost, to establish access for other addresses on the network (ex: http://192.168.0.14:5000)

Screenshots

Docker information
20.10.2
docker-compose: 1.26.0

Cloud provider
GCP

Loader Job Failed - NoMethodError | asset_router.rb

Describe the bug
I am importing a custom JSON file and the Docker worker_1 container is unable to load the custom JSON file. Getting the following error:

Loader job failed - NoMethodError undefined method keys' for #<Array:0x000055c06fb0e1d8> (/app/app/jobs/lib/loader/asset/asset_router.rb:29:in route')

To Reproduce
Steps to reproduce the behavior:

1. Run aws_recon docker image and output the JSON file.
2. Go thru the process of using a custom JSON file.
3. Run OpenCSPM containers
4. Get the following error:

worker_1 | Loader job failed - NoMethodError undefined method keys' for #<Array:0x000055c06fb0e1d8> (/app/app/jobs/lib/loader/asset/asset_router.rb:29:in route')
worker_1 | Runner job failed - NoMethodError undefined method keys' for #<Array:0x000055c06fb0e1d8> (/app/app/jobs/lib/loader/asset/asset_router.rb:29:in route')

Expected behavior
For it to either fail on an unsupported / missing AWS service or load the custom JSON file.

Docker information (please complete the following information):

Server: Docker Engine - Community
Engine:
Version: 20.10.6
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8728dd2
Built: Fri Apr 9 22:44:56 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0

docker-compose version 1.29.1, build c34c88b2
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1h 22 Sep 2020

Cloud provider (if applicable):
AWS

Also ran the docker command to remove orphans, deleted all containers/images and re-pulled to make sure it wasn't something local causing the issue.

Thanks in advance,

William

[feat] CSV export

CSV export from Campaign view.

Discussion: #163

[UI] fix 500 error on modal load

Prior to a full AnalysisJob run, control modals won't load because the server throws a 500 error. The control serializer is trying to map issuess, but there aren't any yet.

Controls are not being executed

Describe the bug
I followed the instructions to collect data using aws_recon and also followed the changes on this issue #78 but I am getting the following errors when i launch opencspm on my local machine

opencspm-worker-1 | Loading /app/load_dir/combined_for_load.json
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 | Warning: no EC2 loader defined for asset type: prefix_list
opencspm-worker-1 |
opencspm-worker-1 |
opencspm-worker-1 | Done loading /app/load_dir/combined_for_load.json. (66 lines)

OpenCSPM version
0.1.23

To Reproduce
Steps to reproduce the behavior:

1. Run aws_recon
2. Copy output.json to opencspm/assets/custom/aws (I tried both this and opencspm/assets/custom/)
3. docker-compose up
4. See error

Expected behavior
My ec2 information should be parsed to the controls and some controls should show as failing for example (darkbit-aws-164)

Screenshots

Docker information (please complete the following information):

• Output of full docker version (not docker -v)
  Client:
  Cloud integration: 1.0.17
  Version: 20.10.8
  API version: 1.41
  Go version: go1.16.6
  Git commit: 3967b7d
  Built: Fri Jul 30 19:55:20 2021
  OS/Arch: darwin/amd64
  Context: default
  Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:31 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0

• Output of full docker-compose version (not docker-compose -v)
  Docker Compose version v2.0.0

Cloud provider (if applicable):
AWS

Additional context
I also downloaded the enterprise controls just to do a complete POC. Any help is appreciated. Thanks

Profile Status inactive

i have tried to use your tool but when i open the dashboard it gives me inactive and doesnt show data even from the demo you provided