Giter Site home page Giter Site logo

ocsf-server's Introduction

Open Cybersecurity Schema Framework Server

This repository contains the Open Cybersecurity Schema Framework (OCSF) Schema Server source code. The schema server is an HTTP server that provides a convenient way to browse and use the OCSF schema.

You can access the OCSF schema server, which is running the latest released schema, at schema.ocsf.io.

The schema server can be also used locally. To do that clone the ocsf-server and ocsf-schema repositories and follow the instruction below to build and run the schema server.

Clone the OCFS schema (ocsf-schema) repository

git clone https://github.com/ocsf/ocsf-schema.git

Clone the OCFS schema server (ocsf-server) repository

git clone https://github.com/ocsf/ocsf-server.git

Build a server docker image

cd ocsf-server
docker build -t ocsf-server .

Run the server docker image

Change the /path/to to your local OCSF schema directory (use an absolute path). Note, the -p 8443:8443 parameter enables HTTPS with a self-signed SSL certificate.

docker run -it --rm --volume /path/to/ocsf-schema:/app/schema -p 8080:8080 -p 8443:8443 ocsf-server

For example, if the ocsf-schema and ocsf-server repos were cloned to the local directory ~/github-projects, this would be the proper replacement for /path/to:

docker run -it --rm --volume ~/github-projects/ocsf/ocsf-schema:/app/schema -p 8080:8080 -p 8443:8443 ocsf-server

(Note that paths used for volume mounts with docker run cannot be relative.)

To access the schema server, open localhost:8080 or localhost:8443 in your Web browser.

Run the server docker image with a local schema extension:

docker run -it --rm --volume /path/to/ocsf-schema:/app/schema --volume /path/to/ocsf-schema:/app/extension -e SCHEMA_EXTENSION="/app/extension" -p 8080:8080 -p 8443:8443 ocsf-server

Development with docker-compose

The docker-compose environment enables development without needing to install any dependencies (apart from Docker/Podman and docker-compose) on the development machine.

When run, the standard _build and deps folders are created, along with a .mix folder. If the environment needs to be recreated for whatever reason, the _build folder can be removed and docker-compose brought down and up again and the environment will automatically rebuild.

Run the ocsf-server and build the development container

docker-compose up

Then browse to the schema server at http://localhost:8080

Testing the schema with docker-compose

NOTE: it is not necessary to run the server with docker-compose up first in order to test the schema (or run any other commands in the development container).

# docker-compose run ocsf-elixir mix test 
Creating ocsf-server_ocsf-elixir_run ... done
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.


Finished in 0.00 seconds (0.00s async, 0.00s sync)
0 failures

Randomized with seed 933777

Set aliases to avoid docker-compose inflicted RSI

source docker-source.sh

Using aliases to run docker-compose commands

# testschema
Creating ocsf-server_ocsf-elixir_run ... done
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.


Finished in 0.00 seconds (0.00s async, 0.00s sync)
0 failures

Randomized with seed 636407

Using environment variables to change docker-compose defaults

Optional environment variables can be placed in a .env file in the root of the repo to change the default behavior.

An .env.sample is provided, and the following options are available:

SCHEMA_PATH=../ocsf-schema      # Set the local schema path, eg. ../ocsf-schema, defaults to ../ocsf-schema
OCSF_SERVER_PORT=8080           # Set the port for Docker to listen on for forwarding traffic to the Schema server, defaults to 8080
ELIXIR_VERSION=otp-25-alpine    # Set the Elixir container version for development, defaults to otp-25-alpine

Local Usage

This section describes how to build and run the OCSF Schema server.

Required build tools

The Schema server is written in Elixir using the Phoenix Web framework.

The Elixir site maintains a great installation page, see https://elixir-lang.org/install.html for help.

Building the schema server

Elixir uses the mix build tool, which is included in the Elixir installation package.

Install the build tools

mix local.hex --force && mix local.rebar --force

Get the dependencies

Change to the schema directory, fetch and compile the dependencies:

cd ocsf-server
mix do deps.get, deps.compile

Compile the source code

mix compile

Testing local schema changes

You can use mix test command to test the changes made to the schema. For example to ensure the JSON files are correct or the attributes are defined.

Assuming the schema repo has been cloned in ../ocsf-schema directory, then you can test the schema with this command:

SCHEMA_DIR=../ocsf-schema SCHEMA_EXTENSION=extensions mix test

If everything is correct, then you should not see any errors or warnings.

Running the schema server

You can use the Elixir's interactive shell, IEx, to start the schema server use:

SCHEMA_DIR=../ocsf-schema SCHEMA_EXTENSION=extensions iex -S mix phx.server

Now you can access the Schema server at localhost:8080 or localhost:8443.

Reloading the schema

You can use the following command in the iex shell to force reloading the schema with extensions:

Schema.reload(["<extension folder>", "<extension folder>", ...])

Reload the core schema without extensions:

Schema.reload()

Reload the schema only with the linux extension (note the folder is relative to the SCHEMA_DIR folder):

Schema.reload(["extensions/linux"])

Reload the schema with all extensions defined in the extensions folder (note the folder is relative to the SCHEMA_DIR folder):

Schema.reload(["extensions"])

Reload the schema with extensions defined outside the SCHEMA_DIR folder (use an absolute or relative path):

Schema.reload(["/home/schema/cloud", "../dev-ext"])

Runtime configuration

The schema server uses a number of environment variables.

Variable Name Description
HTTP_PORT The server HTTP port number, default: 8080
HTTPS_PORT The server HTTPS port number, default: 8443
SCHEMA_DIR The directory containing the schema, default: ../ocsf-schema
SCHEMA_EXTENSION The directory containing the schema extensions, relative to SCHEMA_DIR or absolute path
RELEASE_NODE The Erlang node name. Set it if you want to run more than one server on the same computer
SCHEMA_DIR=../ocsf-schema SCHEMA_EXTENSION=extensions iex -S mix phx.server

ocsf-server's People

Contributors

rroupski-splunk avatar rroupski avatar floydtree avatar rmouritzen-splunk avatar dfederschmidt avatar mikeradka avatar jp-harvey avatar daveshanley avatar lukas-krecan-s1 avatar

Stargazers

Keyth M Citizen  avatar Douglas Oliveira avatar Kyle Donnelly avatar Angelo T. Aschert avatar  avatar  avatar Robert Kvam avatar  avatar NaphalSec avatar Matěj Ryček avatar Jean-Louis Huynen avatar Niranjan Anandkumar avatar jetlime avatar securecodelabs.com avatar Brad Roether avatar Ali AlCharakh avatar George Pchelkin avatar Ragha Vema avatar Rafael Gumiero avatar Tallie 4NG3L avatar Takanori Suzuki avatar kthecoder avatar Brandon Brown avatar Pratikdhanave-searce avatar Matt Black avatar Massin avatar Dan avatar Zoe Braiterman avatar  avatar Michael avatar Michael Laudenslager avatar  avatar Josh Pavel avatar

Watchers

Joseph Copenhaver avatar  avatar  avatar  avatar  avatar  avatar

ocsf-server's Issues

Swagger.json Validation Issue

When attempting to validate the Swagger.json file using the online Swagger Parser tool at https://apitools.dev/swagger-parser/online/, the validation process fails, indicating the validation log.

Swagger schema validation failed.
  #/info must NOT have additional properties
  #/info must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/schema must be object
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1enrich/post/parameters/2 must have required property 'type'
  #/paths/~1api~1enrich/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1enrich/post/parameters/2 must have required property '$ref'
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1translate/post/parameters/2/schema must be object
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1translate/post/parameters/2 must have required property 'type'
  #/paths/~1api~1translate/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1translate/post/parameters/2 must have required property '$ref'
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1validate/post/parameters/0/schema must be object
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must match exactly one schema in oneOf
  #/paths/~1api~1validate/post/parameters/0 must have required property 'type'
  #/paths/~1api~1validate/post/parameters/0 must match exactly one schema in oneOf
  #/paths/~1api~1validate/post/parameters/0 must have required property '$ref'
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must match exactly one schema in oneOf


Error: Swagger schema validation failed.
  #/info must NOT have additional properties
  #/info must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/schema must be object
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1enrich/post/parameters/2 must have required property 'type'
  #/paths/~1api~1enrich/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1enrich/post/parameters/2 must have required property '$ref'
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1translate/post/parameters/2/schema must be object
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1translate/post/parameters/2 must have required property 'type'
  #/paths/~1api~1translate/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1translate/post/parameters/2 must have required property '$ref'
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1validate/post/parameters/0/schema must be object
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must match exactly one schema in oneOf
  #/paths/~1api~1validate/post/parameters/0 must have required property 'type'
  #/paths/~1api~1validate/post/parameters/0 must match exactly one schema in oneOf
  #/paths/~1api~1validate/post/parameters/0 must have required property '$ref'
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must match exactly one schema in oneOf

    at o (https://apitools.dev/swagger-parser/online/js/bundle.min.js:1:95564)
    at https://apitools.dev/swagger-parser/online/js/bundle.min.js:29:45042

SyntaxError: Swagger schema validation failed.
  #/info must NOT have additional properties
  #/info must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/schema must be object
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1enrich/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1enrich/post/parameters/2 must have required property 'type'
  #/paths/~1api~1enrich/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1enrich/post/parameters/2 must have required property '$ref'
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1enrich/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1translate/post/parameters/2/schema must be object
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2/in must be equal to one of the allowed values
  #/paths/~1api~1translate/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1translate/post/parameters/2 must have required property 'type'
  #/paths/~1api~1translate/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1translate/post/parameters/2 must have required property '$ref'
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must NOT have additional properties
  #/paths/~1api~1translate/post/parameters/2 must match exactly one schema in oneOf
  #/paths/~1api~1validate/post/parameters/0/schema must be object
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0/in must be equal to one of the allowed values
  #/paths/~1api~1validate/post/parameters/0 must match exactly one schema in oneOf
  #/paths/~1api~1validate/post/parameters/0 must have required property 'type'
  #/paths/~1api~1validate/post/parameters/0 must match exactly one schema in oneOf
  #/paths/~1api~1validate/post/parameters/0 must have required property '$ref'
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must NOT have additional properties
  #/paths/~1api~1validate/post/parameters/0 must match exactly one schema in oneOf

    at Function.o [as syntax] (https://apitools.dev/swagger-parser/online/js/bundle.min.js:1:95564)
    at validateSchema (https://apitools.dev/swagger-parser/online/js/bundle.min.js:1:6322)
    at SwaggerParser.validate (https://apitools.dev/swagger-parser/online/js/bundle.min.js:1:3391)

Update Objects page description

The desc should be:
The OCSF objects. An object is a complex data type, which is a collection of other attributes. Some objects represent entities or artifacts, but not all.

Missing requirement property in the JSON schema

When the OCSF server renders a JSON schema for an event class, it does NOT add a requirement property if the underlying json definition of the schema doesn't explicitly add a requirement.

Note that, while defining the schema json files, missing requirement implies it is optional, but that is not translated to the JSON schema that the server consequently renders.

Example object - Metadata object definition

Example json schema -

      "modified_time": {
        "type": "timestamp_t",
        "description": "The time when the event was last modified or enriched.",
        "caption": "Modified Time",
        "_source": "metadata",
        "type_name": "Timestamp"
      }

The "source" of object attributes are not accurate

The "source" of object attributes are not accurate: it always shows the top level class, not where the attribute was added or last updated (modified). This is different from classes, where this information is showing correctly. Internally, this is in the dictionary attribute's _source field.

When this is fixed, we can show this information in the same manner as class attribute on the "Attribute Dictionary" page, as well as the "Objects" and specific object pages.

Related to issue #63.

Extension ID Ordering & Expansion capability for sidebar (extensions & profiles)

The Extension IDs are out of sequence, most likely because of alphabetical sorting by name. Show the extension id on mouse hover.

image

Also, as Extensions and Profiles grow, so will the list on the sidebar. To make a slight improvement to the web ui, add a triangle to expand/collapse extensions and profiles like so:

> Extensions (collapsed)
-------------------------
> Profiles (collapsed)
-------------------------
> Extensions (expanded)
   Linux
   Windows
   Macwhatever
-------------------------
> Profiles (expanded)
  Cloud
  Next Profile
  Yadda Yadda
-------------------------

Set "Access-Control-Allow-Headers" for validate endpoint

Follow-up from #16

In order to use the /api/validate endpoint from browser context, the
Access-Control-Allow-Headers should be set to allow for the Content-Type header to be submitted by the client. The request works without a content-type set but fails if Content-Type: application/json is passed explicitly.

Redesign Profile Implementation

Background: A few things broke in our testing pipelines when working with the Network Activity class schema, it turns out the class now includes 5 profiles (“cloud”, “domain_security”, “host”, “malware”, “user”), which basically add the attributes as defined in the profile definitions to the rendered schema by default. These definitions then define requirements for the attributes within. For example, this is how the malware profile looks like.
With this example in mind, now if you query for a network_activity class schema in the browser, you’ll get a JSON schema that includes all the attributes defined in the profile, which includes additional required fields that you don’t really require to normalize a network activity log. With the malware profile, it adds disposion_id & disposion as required fields (side note - there appears to be a spelling mistake in this attribute, assuming it is disposition_id)
Thus, even if, I as a user do NOT want malware profile for my network_activity class schema, I am forced with the requirement to populate/map disposion_id, else my data will technically be non-compliant with OCSF, which doesn’t make sense.
Additionally, there is a disconnect between browser view and the actually rendered JSON schema, the JSON seems to by default include all profile attributes regardless of what you choose in the browser via the checkboxes. So the checkboxes give false sense of control to the user.

In summary, profiles appear to have become a compulsive factor instead allowing for originally intended freedom.


From my understanding, it boils down to how profiles are defined in the event_class (as includes in the JSON definitions of the event_class)

I think the way it should be implemented is,
You shouldn’t define profiles in the JSON class definitions at all, instead the profile selection should an api parameter where you can explicitly mention which profiles you want for any desired event_class. The same can be translated in the browser in the form of checkboxes.

If no profiles are mentioned the user must be presented with the vanilla definition of the event_class both in the browser & in the actual JSON schema. Profiles should only be added (i.e. the attributes defined in the profiles), when explicitly selected in the api/browser.

Add a new page to show profile's attributes

Currently there is no a simple way to see the attributes defined in the profiles. Add a new page to show the profile's attributes. The page should look similar to the object page.

UI: Behavior of Extension Checkbox

The extensions checkbox seems to only add/remove objects from the server's view. It appears that the checkbox has no impact on the dictionary and other pages.

Also, it seems like there is still some grey area for adopters when it comes to when to use profiles vs extensions. I added an issue to the docs repo to track this here.

Attribute is not shown unless an unrelated profile is selected

The situation is a bit complex so I have created a reproducer here.

  1. I have an extension which has an object "my_object"
  2. The object is used to define an attribute in dictionary.json
  3. The attribute is used in parent_object. The parent object is not used anywhere, but it does not influence the issue.
  4. If I look at the parent_object in OCSF server, I see the my_object attribute as expected
  5. Now I add the my_object attribute to a profile
  6. If I apply the profile on an event like this the OCSF server starts to show the field in parent_object only if I select the Test Profile. This is unexpected the Parent Object should not be influenced by the profile in any way.

Surprisingly enough, the same situation is in OCSF core with "device" that is used in "logger" object and in the "host" profile, but it works as expected there 🤷‍♂️

Add deprecated annotation

As the schema involves, we need to have a formal way to deprecate existing classes, objects, and attributes.

To deprecate an existing class, object, or attribute, you can use the @deprecated annotation to mark a schema entity as deprecated. The @deprecated means that the entity has been superseded and may cease to exist in the future. The annotation emits a warning in the schema server, explaining that its usage is discouraged. In order to deprecate a schema entity, a proposed alternative must already exist.

The '@deprecated' annotation is a JSON object with the following fields:

  • message describes the proposed alternative.
  • since describes when the entity was deprecated.

Notes

  • The deprecated entities can be only removed in major releases. This means deprecated entities in OCSF schema v1.x shall only be removed in OCSF schema v2.x.
  • The deprecation must also be documented in CHANGELOG.

Example

To mark an object as deprecated, you can use the following snippet:

{
  "@deprecated": {
    "message": "Use the new [Bar](https://schema.ocsf.io/objects/bar) object.",
    "since": "v1.0.0"
  },
  "name": "foo",
  "caption: "The Foo Object",
  "description": "...",
  // ...
}

This will generate a warning message when the Foo object is used in the schema:

The foo object has been deprecated. Use the new [Bar](https://schema.ocsf.io/objects/bar) object.

Container profile is incorrectly shown on most class detail pages

Most class detail pages are showing the Container profile when the profile is not, in fact, applicable. With the main ocsf-schema, 62 out of 64 classes include the container profile with only base_event and incident_finding not included. The Container profile associated with no classes (per its profile detail page) and is the only profile that is associated with objects.

What might be happening is that class detail "profiles" section is including classes that reference the objects that in turn use the profile, whereas the profile details page is only showing the direct references in the class and object definitions.

Clicking "JSON Schema" from "Base Event" page leads to 404

Steps to reproduce:

  1. Click on "Base Event" from navigation. This leads to the special /base_event URL.
  2. Click on "JSON Schema".

This results in a 404. Note that navigating from /class/base_event works. This URL can be navigated to from a few location, including from the class list on the "Classes" page.

Cause

This is happening because the JSON Schema JavaScript in priv/static/js/app.js is navigating to /schema/base_event, which doesn't exist. When navigating from a /classes/<class_name> URL, the JavaScript function leads to the working /schema/classes/base_event URL.

Possible fixes

There are (at least) two ways to fix:

  1. Fix the JavaScript function to handle coming from the /base_event URL.
  2. Remove the special /base_event URL entirely from the OCSF Server since it is equivalent to /classes/base_event. This is a larger change, but could result in a overall cleaner solution.

Redirect http traffic on schema.ocf.io

It doesn't look like we have enabled redirection of http traffic to https on schema.ocsf.io

Currently, if you navigate to http://schema.ocsf.io, it doesn't respond (good that it doesn't accept port 80 connections), but we should be redirecting such to 443 instead, for a seamless experience.

Proposal: improve 404 page or redirect -dev versions after release

I've been working with colleagues on mapping detection findings, which was a part of the 1.1.0 release. Before release, this version was listed under 1.1.0-dev, and so our discussions often involved linking to the OCSF server with a version reference like so:

https://schema.ocsf.io/1.1.0-dev/classes/detection_finding?extensions=

As this was the only way to see the new detection finding class. Now that 1.1.0 is released, the link above just goes to a basic 404 page.

image

I wonder if there's a better experience to provide here? Ideas:

  • Simple: improve the basic 404 page so there are at least links back to OCSF schema proper.
  • More complicated: on release of a -dev version, would it make sense if /1.1.0-dev redirected to /1.1.0?

Filter profiles by extension

Currently all profiles are shown, including profiles defined in extensions, regardless whether or not the extensions is check.

Expected behavior, when an extension is checked show its profiles, and if the extension is unchecked then hide its profiles.

Switch schema.ocsf.io SSL certificate to Let's Encrypt by August 1, 2024

The current *.ocsf.io certificate used by the OCSF server at schema.ocsf.io expires on August 1, 2024. We should convert this to a Let's Encrypt certificate, along setting up the ACME protocol to automatically update the certificate.

This work should be started roughly 3 months before the certificate expires to give plenty of time for the conversion, so around May 1, 2024.

JSON Schema export

It would be nice to be able to export the whole JSON schema via the Schema server for internal usage.

Some other nice features would be adding a switch to tell the server how deep you want it to go when exporting some of the circular nested fields

Improve filtering droplist UX

The "n items selected" droplist next to the search box that exists on many screens. This is a filtering mechanism that has confused users in many cases. We want to make these settings more explicit.

Ideas:

  • Move the filtering options to the left hand gutter so it is always visible.
  • Change the HTML so the upper part of the page is always visible.

Keep mobile viewing in mind. The left hand gutter, for instance, doesn't show up at all on mobile browsers (at least not on iPhone). Keeping a portion a page always visible could make mobile viewing more difficult with a large percentage of the visible view locked in place.

Show additional properties of the attributes

The schema can be extended by adding extra properties of the attributes. For example, some of attributes in the dictionary already have a default property, which defines a default value of the attribute if it is missing in the event data. For example:

{
  "count": {
    "caption": "Count",
    "default": 1,
    "description": "The number of times that events in the same logical group occurred during the event <strong>Start Time</strong> to <strong>End Time</strong> period.",
    "type": "integer_t",
    "type_name": "Integer"
  }
}

The schema browser should show those properties.

Support user-defined data types

Using extension to add new data types (already supported) or to extend existing data types (add contains or data examples).

For example:

The schema defines country attribute as a string (the country code, as defined by the ISO 3166-1 standard -- Alpha-2 code). However, some users would like to use a stricter definition, such as a list of country codes. This could be achieved by defining a new type, country_code_t, which is a subtype of string_t, and lists the valid values.

  "country_code_t": {
    "caption": "Country Code",
    "description": "Defined by the ISO 3166-1 standard -- Alpha-2 code. For example: <code>AA</code>.",
    "type": "string_t",
    "values": ["AA", "AB", "AC", "AD", "AE", "AF", "AG", "AH", "AI", "AJ", "AK", "..."]
  },

Then, the data new type can be used to overwrite the dictionary definition of the country attribute:

{
  "country": {
    "type": "country_code_t"
  }
}

Profiles page

Add a new page to show the profiles and the attributes that they defined.

API to convert real data to OCSF schema format

I explored different APIs in OCSF-server and found that it does not have any API for converting existing data to schema format.
To convert a data to this schema, one would require two things:

  • Mapping of their keys to ocsf schema
  • actual data

If it's not in your plan, or not having bandwidth. I can build it.
Or, your plan is to create this automation separately from ocsf-server?

Any thoughts?

Add UI column to specify base-event/inherited attribute vs. net-new attribute

As discussed in the July 18th, 2023 OCSF zoom: This data is already specified when hovering over an attribute in the UI, but I think it warrants its own column for more convenient reading. This would be particularly helpful for consumers who are actively developing with OCSF (i.e. new developers or those who are more comfortable with visual representations). Again, the data is already there -- just need to reformat it/present it in a more stable way.

Because this info may not be necessary for passive readers of the schema, it may be worth having this field as a toggle on-and-off.

Validation of schema using API is incorrect

I've been attempting to validate my "api_activity" event object using the OCSF swagger API and it keeps reporting that "api" is an "undefined attribute name". Removing it reports no error (though API is a required object). I think there is a bug in the validation endpoint. Steps to reproduce:

  1. Go to https://schema.ocsf.io/doc/index.html
  2. Go to "/sample/classes/{name}" and enter "api_activity" in the "Class name" text box. Click "Execute"
  3. Copy result
  4. Go to "/api/validate" and copy the result and click execute

This, for me, resulted in the sample schema failing validation.

Support more ways to define observables

Support more ways to define observables

This issue proposes 2 changes:

  • Add two new ways to define observables.
  • Add a new observable type registry. Instead we are adding duplicate observable type_id detection to the ocsf-server and ocsf-validator.

New ways to define observables

Currently observables can be defined on types in the attribute dictionary and in object definitions. We want to add the capability to also define these observables on attributes in new ways:

  1. Define observable on dictionary attributes.
  2. Define observable on class and object attributes.
  3. Define observables in class definitions by attribute paths. This allows class-specific definitions for attributes that could be nested in a sub-object or sub-array in a new section using a path-based syntax. For definitions that navigate to or through an array, the definition applies to each element of the array.

The class path-based definition will look like this (or something equivalent):

{
    "class": "foo",
    .
    .
    .
    "observables" {
        {
            "device.file.path": 42
        }
    }
}

With these additions, it is possible that observable definitions could overlap. In cases where a specific observable matches more than one type_id, the most general one should be used. The following list ranks from most to least specific:

  1. Observable by attribute's type: a scalar type (defined in types section of attribute dictionary) or object type (defined in the object's definition). Note that attribute scalar types and object types cannot collide.
  2. Observable by object (for attributes who's type is an object).
  3. Observable by dictionary attribute (defined in the attributes section of the attribute dictionary).
  4. Observable by class or object attribute (defined for in a class or object's attributes property for specific attributes).
  5. Observable by class-specific path (defined in class definition).

Observable registry

We also want to add a new observable type registry. A new observable-types.json will be supported to unify these definitions. Extensions can extend these by placing an observable-types.json in the extension directory, as with attributes extensions using an extension-specific dictionary.json. The OCSF Server (and any future tools that read the metaschema) will log problems like missing missing type_id values and colliding type_id values. The OCSF Server (and future tools) may optionally stop these sort of errors, though there should always be a way to continue on in case there metaschema is in a temporarily inconsistent state.

This idea doesn't help avoid observable type_id collisions. As mentioned above, we will instead detect collisions as a validation.

Profiles: show cross references to classes and objects

It would be nice to show at the bottom all of the classes and objects that have registered the profile, like we cross reference other things.

We could do it like objects, where we have the references at the bottom, or we could do it like the dictionary, where we have a column in the profiles page

Schema validation

When validating the schema, profiles == null is equal to no profiles.

Extension leakage from attribute dictionary.

The Compliance event class which is a dev extension and part of the Policy category, also a dev extension, shows up in the cross reference of the CVSS Scores, a core object, even when the Development extension is not selected. This may also occur from the attribute dictionary although I didn't find an instance of it. The browser should not leak extension classes (nor objects) when not selected.

JSON Schema Generator automatically including all profiles

When generating the JSON Schema for an inventory_info event, it appears that profiles are added even if they aren't included as a query parameter.

This causes the cloud profile to be added and makes the cloud object a required field. This ultimately invalidates the schema for an inventory_info event that does not originate in the cloud.

See -> curl -X GET "https://schema.ocsf.io/schema/classes/inventory_info" -H "accept: application/json"

image

Remove object_type from the attribute definitions

Defining a new attribute of an object type requires two statements, type and object_type:

{
  "name": "cat", 
  "caption": "Cat",
  "type": "object_t", 
  "object_type": "cat"
}

The definition could be simplified by removing the object_type and using only the type statement:

{
  "name": "cat", 
  "caption": "Cat",
  "type": "cat"
}

Random Profile Generation

Supplying an empty profile in the Class sample generation request creates random profiles -(url)

"reputation":{"profiles":["tomorrow conclusion lawyers","talks linking views"],"raw_score":71,"score":"Exercise Caution","score_id":6},"subnet_uid":"b548080c-3ebb-11ed-9bb3-0242ac110002","svc_name":"da wonder wyoming","uid":"b5480d16-3ebb-11ed-9311-0242ac110002","vpc_uid":"b548107c-3ebb-11ed-a6f7-0242ac110002"},"duration":12,"end_time":1664321143954142,"message":"school wonderful influence","metadata":{"correlation_uid":"b547b4c4-3ebb-11ed-b3fc-0242ac110002","labels":

https://schema.ocsf.io/sample/classes/network_activity?profiles=

Custom data types not converted to JSON Schema

When defining a custom data type like below it is not converted to its underlying type when generating JSON schema, this renders the schema invalid.

Invalid JSON schema type: managed_state_t. Path '$defs.device.properties.managed_state.type'

image

 "managed_state_t": {
        "caption": "Managed State",
        "description": "Device managed states. One of <code>Unsupported/code>, <code>Secured</code>, <code>Unsecured</code>, or <code>Unknown</code>.",
        "values": [
          "Unsupported",
          "Secured",
          "Unsecured",
          "Unknown"
        ],
        "type": "string_t",
        "type_name": "String"
      }

Run the server docker image

docker run -it --rm --volume /path/to/ocsf-schema:/app/schema -p 8080:8080 -p 8443:8443 ocsf-server

Change the path/to to your local OCSF schema directory (use an absolute path). Note, the -p 8443:8443 parameter enables HTTPS with a self-signed SSL certificate.

The repo of schema doesn't contain folders and files like /path/to/ocsf-schema:/app/schema

Rename event_uid/name as type_uid/name

The problem with event_uid is -- it gives a notion that the identifier is uniquely identifying an event (a unique occurrence of an activity as recorded by a log, think Cloudtrail event), thats going by what an Incident responder would think, at least based on my own experience.

However, the way we currently have event_uid it is simply saying this is a unique type of an event within OCSF which is akin to an event code/type.

timestamp_t not creating datetime counterpart in object

Introducing a custom timestamp_t attribute in extension creates the counterpart in the dictionary, but not in the object.

Steps to reproduce:

  1. Create new timestamp_t attribute in an extension dictionary, add it as field in object.
  2. Run server and navigate to object
  3. Toggle date/time profile, datetime_t attribute does not appear

metadata.profiles = STRING passes validation

If I set metadata.profiles to a string:

"metadata": {
   "profiles": "tanium"
}

The API validation allows it as valid, but no profiles are applied.

As metadata.profiles is an array of strings, it should fail if the value is a single string.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.