surface-security/surface

Security Intelligence Automation Platform

Surface

Asset inventory tracking and security scanners.

Quickstart

AWS

For AWS, check aws-cdk folder

docker

# Clone this repo
git clone https://github.com/surface-security/surface/

# Create a `local.env` for any custom settings
touch surface/local.env

# Launch the docker stack
docker compose -f dev/docker-compose-in-a-box.yml up

# Run the "quick start" script - choose password for `admin` user
dev/box_setup.sh

Open http://localhost:8080 and login as admin.

box_setup.sh created a local Rootbox and added the example, httpx and nmap scanners images (all from here).

You might need to reload nginx and Surface so the migrations and the webserver are put in effect. You can do so with docker container restart dev-nginx-1 dev-surface-1.

Quick check:

• add IPAddress or DNSRecord (and tag it is_external), create a Scanner using example image and choose Run scanner from the actions dropdown
• check scan logs

Documentation

We have in-depth documentation and instructions on this repository's wiki page.

surface's Issues

Move to Uvicorn and ASGI

Related software versions

Create and use ASGI configuration to use uvicorn instead of wsgi.
This would deprecate eventlet, and therefore close #52 .

Description

Upgrade Surface to use ASGI and Uvicorn for asynchronous views and server.

Use cases

Asynchronous views, baby!

References

No response

On dependency version bumps test if integration works in Surface

Related software versions

All

Description

We should create a (reusable) workflow to test if newer versions of the dependencies do not have backward incompatibility in Surface. This is specially relevant as we have different version pins in our dependencies and we need to be sure upgrading them still continues to work as designed.

Proposed solution:

• Workflow should only be triggered when there is a version bump PR in the django-{} app.
• Workflow should "build" Surface as is from the latest tag.
• Install the newer dependency and run manage.py runserver - it should work.
• Run E2E integration test that ensures users are able to log in (to be created) and application loads as expected. There is a sidebar, some entries there, etc.
• Tests should leverage pytest marks and distinguish them between e2e and regression tests, running first regression first then e2e.

Simultaneously, this practice will have to be documented in the wiki to include some details above the pytest marks . Any occasion where something broke, the practice should be to create a regression test when breaking changes are not covered by the tests described above.

Wdyt, @DDuarte @fopina ?

Use cases

As a maintainer, I need to confidently review and merge changes into Surface, having enough confidence that we're not introducing incompatible changes and/or creating backward incompatibilities.

References

#93

Surface should create used labels out of the box

surface/surface/scanners/inputs/ips.py

Line 15 in 456d236

tag_internal_or_external = 'is_external'

Just noticed today that Surface does not create these labels out of the box, meaning that the scanners will face some issues.
Since we provide these input generators out-of-the-box, I believe we should add a migration to create these labels.

Add suport to Python 3.11

Related software versions

Latest

Description

Add support for Python 3.11 in Surface core. Update GitHub actions to try building the project for Python 3.11.

Update document with relevant changes (including to install wheel before any -r requirements*.

Use cases

None.

References

No response

Upgrade project dependencies

Input

Upgrade the project dependencies to their latest when possible.

Output

All open sourced projects have their dependencies upgraded to the latest versions, or the closest possible.

dev (pre)releases

Related software versions

Description

I'll start using this branching/release strategy in my projects and I'd like to recommend it here as well.

• All PRs are done against develop (not main)
• develop should always have the next version (in __version__)
• publish-dev workflow will publish a new pre-release (.devX) on every merge to develop
• On a less frequent basis, a stable version can be released by merging develop to main

This allows for anyone wanting to use latest to install pre-releases while keeping the stable versions "more stable" (meaning that they'd have been tested previously by those using pre-releases).

Steps to setup

• fopina/django-bulk-update-or-create@37ea518 workflow
• Create develop branch from main
• Make develop the default branch
• Add protection rules to main and develop (only pull requests, no force push, etc)
• Bump version in develop to next planned (even if when the release comes, it's a different version due to breaking changes or whatever)

Use cases

Faster releases, easier (actual) "beta" testing, "more stable" stable releases

References

No response

Table 'surface.scanners_rootbox' doesn't exist

I've installed the docker version as detailed in the README.
I get the following errors:
dev-resync-rootbox-1 | django.db.utils.ProgrammingError: (1146, "Table 'surface.scanners_rootbox' doesn't exist")

Please advise.

Full Trace
dev-resync-rootbox-1 | Traceback (most recent call last):
dev-resync-rootbox-1 | File "/surface/surface/./manage.py", line 22, in
dev-resync-rootbox-1 | main()
dev-resync-rootbox-1 | File "/surface/surface/./manage.py", line 18, in main
dev-resync-rootbox-1 | execute_from_command_line(sys.argv)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
dev-resync-rootbox-1 | utility.execute()
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/init.py", line 413, in execute
dev-resync-rootbox-1 | self.fetch_command(subcommand).run_from_argv(self.argv)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
dev-resync-rootbox-1 | self.execute(*args, **cmd_options)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/logbasecommand/base.py", line 64, in execute
dev-resync-rootbox-1 | super().execute(*args, **options)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
dev-resync-rootbox-1 | output = self.handle(*args, **options)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/database_locks/locks.py", line 126, in new_handle
dev-resync-rootbox-1 | return orig_handle(self, *args, **kwargs)
dev-resync-rootbox-1 | File "/surface/surface/scanners/management/commands/resync_rootbox.py", line 181, in handle
dev-resync-rootbox-1 | for rootbox in rootboxes:
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 280, in iter
dev-resync-rootbox-1 | self._fetch_all()
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
dev-resync-rootbox-1 | self._result_cache = list(self._iterable_class(self))
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 51, in iter
dev-resync-rootbox-1 | results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
dev-resync-rootbox-1 | cursor.execute(sql, params)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 98, in execute
dev-resync-rootbox-1 | return super().execute(sql, params)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
dev-resync-rootbox-1 | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
dev-resync-rootbox-1 | return executor(sql, params, many, context)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
dev-resync-rootbox-1 | return self.cursor.execute(sql, params)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in exit
dev-resync-rootbox-1 | raise dj_exc_value.with_traceback(traceback) from exc_value
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
dev-resync-rootbox-1 | return self.cursor.execute(sql, params)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 73, in execute
dev-resync-rootbox-1 | return self.cursor.execute(query, args)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute
dev-resync-rootbox-1 | res = self._query(query)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 319, in _query
dev-resync-rootbox-1 | db.query(q)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 259, in query
dev-resync-rootbox-1 | _mysql.connection.query(self, query)
dev-resync-rootbox-1 | django.db.utils.ProgrammingError: (1146, "Table 'surface.scanners_rootbox' doesn't exist")

invalid chars allowed in scanner name

If scanner name contains space character it will fail execution:

Add dev scripts

Related software versions

All

Description

Add developer scripts to make it easier to develop and contribute to the project.
Some ideas:

1. Database creation and reset (delete, create, create test users, run migrations, for instance);
2. Review docker-compose to facilitate creating a dev environment;
3. Add notes to a CONTRIBUTING.md which should contain notes on how to build the project, run tests, how to contribute and so on.

Use cases

As a developer, I want to easily restart my database schema, as well as speed up some developer tasks (have clear instructions to start the server, connect to the database, etc)

References

No response

Improve documentation on how to build and use for devs

Related software versions

All

Description

Current set of instructions are aiming to the end-user. We're lacking some instructions to build and test Surface locally, for those developers and engineers that will be building on top of the project.
In addition (maybe it's worth a separate issue?), we're missing some shell helper scripts to automate mundane tasks like reset the database, create database from scratch etc.

Use cases

The use case is developers who want to build, contribute and/or build on top of Surface.

References

No response

Deprecate inventory.Finding

Related software versions

v.1.10

Description

Vulnerability management is a complex topic on its own, to the point where its own app makes sense to implement in its own app so all related scripts and helpers are contained in the same app - which is what #125 introduces.

However, now there are two models for the same purpose. Also introduced in the above PR, there is a warning to deprecate invenotry.Finding in favour of vulns.Finding. This issue is to track this deprecation in time.

Use cases

As a developer, I need a single model for Finding, preferably in its own app due to the complexity of the topic.

References

No response

Finish AWS CDK quickstart

Currently the CDK only creates database and the web container.

All the other services should be added:

• dkron agent
• rootbox (either dockerd service or an EC2) - bind to specific NAT gateway to lock in external IP
• slack bot
• notification sender
• everything else in the docker-compose...

squid proxy deploy

run_squid_proxy currently deploys squid to the boxes mentioned as arguments.

update it to also support deploying squid to rootboxes that have (new) deploy proxy field set to True, when no rootboxes are listed as args (auto mode).

new command arg flag --cleanup so it also removes squid from every rootbox that has that flag set to False. adding that as flag as that will increase run time for the command (it will connect to every rootbox) and in most scenarios, removing squid is not really required nor happens often and user can prefer to do it manually

Deprecate DNSNameservers

Related software versions

Latest

Description

The purpose of this model was to create a list of allowed DNS name servers and it's development stopped halfway.
Name servers can be created as NS Records as DNSRecord and it's actually better at correlating information than this model.

Use cases

Cleaner code, better use of existing models.

References

No response

Views should get a baseline of tests to ensure they work when initiated

Related software versions

All

Description

surface-security Django apps (django-{}) should test their views independently so that each change is confidently included in the code base. For that, we should create a test that ensures all filters and all search bars work (by work means no errors are thrown, because the result is ignored).

Whatever solution is planned, propagate across all other relevant apps.

Use cases

As a maintainer, I want to confidently be comfortable changes are not disrupting the code base.

References

No response

Table 'surface.scanners_rootbox' doesn't exis

I've installed the docker version as detailed in the README.
I get the following errors:
dev-resync-rootbox-1 | django.db.utils.ProgrammingError: (1146, "Table 'surface.scanners_rootbox' doesn't exist")

Please advise.

Full Trace
dev-resync-rootbox-1 | Traceback (most recent call last):
dev-resync-rootbox-1 | File "/surface/surface/./manage.py", line 22, in
dev-resync-rootbox-1 | main()
dev-resync-rootbox-1 | File "/surface/surface/./manage.py", line 18, in main
dev-resync-rootbox-1 | execute_from_command_line(sys.argv)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
dev-resync-rootbox-1 | utility.execute()
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/init.py", line 413, in execute
dev-resync-rootbox-1 | self.fetch_command(subcommand).run_from_argv(self.argv)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
dev-resync-rootbox-1 | self.execute(*args, **cmd_options)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/logbasecommand/base.py", line 64, in execute
dev-resync-rootbox-1 | super().execute(*args, **options)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
dev-resync-rootbox-1 | output = self.handle(*args, **options)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/database_locks/locks.py", line 126, in new_handle
dev-resync-rootbox-1 | return orig_handle(self, *args, **kwargs)
dev-resync-rootbox-1 | File "/surface/surface/scanners/management/commands/resync_rootbox.py", line 181, in handle
dev-resync-rootbox-1 | for rootbox in rootboxes:
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 280, in iter
dev-resync-rootbox-1 | self._fetch_all()
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
dev-resync-rootbox-1 | self._result_cache = list(self._iterable_class(self))
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 51, in iter
dev-resync-rootbox-1 | results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
dev-resync-rootbox-1 | cursor.execute(sql, params)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 98, in execute
dev-resync-rootbox-1 | return super().execute(sql, params)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
dev-resync-rootbox-1 | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
dev-resync-rootbox-1 | return executor(sql, params, many, context)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
dev-resync-rootbox-1 | return self.cursor.execute(sql, params)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in exit
dev-resync-rootbox-1 | raise dj_exc_value.with_traceback(traceback) from exc_value
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
dev-resync-rootbox-1 | return self.cursor.execute(sql, params)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 73, in execute
dev-resync-rootbox-1 | return self.cursor.execute(query, args)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute
dev-resync-rootbox-1 | res = self._query(query)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/MySQLdb/cursors.py", line 319, in _query
dev-resync-rootbox-1 | db.query(q)
dev-resync-rootbox-1 | File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 259, in query
dev-resync-rootbox-1 | _mysql.connection.query(self, query)
dev-resync-rootbox-1 | django.db.utils.ProgrammingError: (1146, "Table 'surface.scanners_rootbox' doesn't exist")

cleanup pending

• TLA field in dns_ips
• change to organisation? consider using a swappable model (like auth.User) to avoid migrations within the app?
• import_export mixin: add it globally to every admin model?
• move slack_display method to a model list setting from slackbot or add a register method - otherwise it's impossible to add it to 3rd party models

Surface OSS direction discussion

Since we start splitting in different companies, Surface OSS will definitely take a different shape in each company based on the integrations and types of data the company is working with. A discussion, followed by a documented way to manage a common ground for the future Surface OSS instances is therefore needed. This may include a list of common models, connectors for custom (per company) processing/routines, improvements to be done, etc.

Surface will be with us from now on everywhere we go and working individually will just lower the evolution pace. Instead, we can use this advantage of implementing it in different companies and figure out what are the common needs of these companies. Most of the companies are lacking of Security personnel and efficient ways of working. Maybe we can do something in this aspect, it worths trying at least.