oasis-open/cti-taxii-server

OASIS TC Open Repository: TAXII 2 Server Library Written in Python

cti-taxii-server's Issues

Is there a reason to why initialize_mongodb.py, reset_db() is never being called?

the function reset_db() is never being called, is this intended?

discovery endpoint does not list api roots & unit tests are failing

The discovery endpoint does not list api roots using a mongo db backend & 8 unit tests are failing.

Ability to use as a library in an existing Flask application

It would be nice to be able to use Medallion as a library in an existing application. The way it is currently structured (with assumptions of "auth" throughout the code) makes this not possible.

memory_backend.py add_objects() expecting objs to be a dict; it appears to be passed as a string.

It appears that the add_objects() function in memory_backend.py is expecting objs to be a dictionary. It appears to be being passed as a string.

At some point, I believe the json string needs to be converted to a dict. One possibility would be to convert it to a dict in add_objects(); something simple like adding the following seems to work:

objs=json.loads(objs)

BUG: filtering collection objects doesn't actually work for "added_after"

In basic_filter.py ...

Current code has:

        match_version = self.filter_args.get("match[version]")
        if "version" in allowed:
            if not match_version:
                match_version = "last"
            if len(data) > 0 and self.is_manifest_entry(data[0]):  # NEED TO CHECK EMPTY RESULTS!
                results = self.filter_manifest_entries_by_version(results, match_version)  # THIS IS BAD
            else:
                new_results = []
                for bucket in BasicFilter._equivalence_partition_by_id(results):
                    new_results.extend(self.filter_by_version(bucket, match_version))
                results = new_results
        added_after_date = self.filter_args.get("added_after")

First, you need to make sure you're data array actually has data in it, then you need to filter on the actual manifest_info array.

Update README

Provide build status, coverage and version.

Return 422 if object type not supported

As specified here, if the object type can't be processed, the 422 HTTP error code should be returned. However, currently the server returns a 500 if you try adding a single STIX object to a collection instead of adding a bundle.

Figure out why memory_backend tests aren't running on Travis-CI

Issue while running docker on Mac

Following issue occurred while installing cti-taxii-server via docker on Mac
Step 3/5 : COPY ./docker_utils/*.json /opt/taxii
COPY failed: no source files were specified

Implement default values for arguments in backend function calls

An error occurred where a stix2 test was calling a backend function directly and was not giving enough arguments for the function to be called, which led to an error. If the function arguments had defaults, then the error would not have happened.

Similarly, most backend and filter functions are not made to be called directly, though they may be as shown by the example described above. Direct calling should at least be taken into consideration and possibly implemented.

common base module for backend test modules

Memory backend speed

The backend should be restructured to minimize searching and sorting. Currently, data in the filter gets sorted at least 3 times, which at least one is n^3 I think.

Implement limit and next parameter for MongoBackend

We are currently missing this feature.

url

Can I get the url after running the taxii server? the url is for discovery and get the service provided in the server. Thanks

add documentation for Endpoints in the spec

Put references to the Endpoints defined in the spec.

taxii-server ignoring objects past v0.2.1

Enterprise Attack JSON

import json
import socket
from time import sleep
import sys
from stix2 import TAXIICollectionSink
from taxii2client import Collection


def main():
    """
    ....
    """
    attack()


def attack():
    """
    Import attack into TAXII server
    """
    collection = Collection(
        "http://192.168.56.100/tde/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/",
        user="guest",
        password="guest")
    # create TAXIICollectionSINK and push STIX content to it
    taxii_collection_sink = TAXIICollectionSink(collection)
    data = json.load(open('enterprise-attack.json'))
    taxii_collection_sink.add(data)
    print("Attack added to TAXII Server")


if __name__ == "__main__":
    main()

To recreate it grab the latest taxii server and run it. Then compare the the first object and it will be missing a lot of the fields.

Is this intended from version 0.2.1 upwards? If so how can I achieve the same with the latest version?

save_data_to_file() in test_memory_backend fails on Windows

cti-taxii-server/medallion/test/test_memory_backend.py

Lines 396 to 403 in b9441e1

	with tempfile.NamedTemporaryFile() as f:
	self.client.post(
	test.ADD_OBJECTS_EP,
	data=json.dumps(new_bundle),
	headers=post_header,
	)
	self.app.medallion_backend.save_data_to_file(f.name)
	assert os.path.isfile(f.name)

save_data_to_file(f.name) attempts to open the temporary file, which has already been opened by NamedTemporaryFile. That doesn't work (at least on Windows), and causes a permission error.

Python docs say: "Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later)."

https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile

I think save_data_to_file() should probably be changed to accept a file-like object rather than a filename, akin to json.dump().

Found by @chisholm.

Updating and expanding default_data

The default_data file included with the TAXII repo is out of date and pretty basic for data to test against. At the very least, this data must be updated to conform with TAXII 2.1 spec. I also recommend adding some to the data to include some complexity, since this is the data that all tests use to ensure accuracy of changes. An extra step would also include a larger set of data, allowing for the speed and efficiency of the server and future changes to be tested.

I have made some changes to the default_data in my PR #62 but is most likely not entirely accurate as well.

Write OpenAPI specification

Not sure if this is the place to put it, or if it exists and I missed it, but seems as if having an OpenAPI specification in place could help cut integration and on-boarding time due to documention and client/server generation.

Don't have a problem putting one together with the resources available for 2.0, but want to know if it's desired.

I can also ping the mailing list of that's a more appropriate forum for questions like these.

Should the 'max_page_size' config value be checked when no 'limit' parameter is given?

Currently, the mongo and memory backend implementations only check this value when the 'limit' parameter is given, though there has been discussion in the past as to if this value should always be checked.

TAXII Specification compliance for 'timestamp' datatypes on 'added_after' filters

I'm encountering this error when attempting to perform a GET on the objects endpoint, filtering with the ?added_after=2018-06-29T12:58:28-04:00 query parameter.

I did some digging into strptime and the TAXII spec. The above timestamp should be considered valid.

The timestamp field MUST be a valid RFC 3339-formatted timestamp [RFC3339] using the format YYYY-MM-DDTHH:mm:ss.[s+]Z where the “s+” represents 1 or more sub-second values. The brackets denote that sub-second precision is optional, and that if no digits are provided, the decimal place MUST NOT be present.

It appears in utils there is a common.py with a method for handling this type of thing, but it isn't used in a few cases. Not sure if it's feasible to replace all calls to datetime.strptime() with the util method?

Here is the full traceback.

[2018-06-29 13:07:14,392] ERROR in app: Exception on /trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/ [GET]
Traceback (most recent call last):
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/medallion/utils/common.py", line 101, in convert_to_stix_datetime
    return dt.datetime.strptime(timestamp_string, "%Y-%m-%dT%H:%M:%S.%fZ")
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py", line 565, in _strptime_datetime
    tt, fraction = _strptime(data_string, format)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py", line 362, in _strptime
    (data_string, format))
ValueError: time data '2018-06-29T12:58:28-04:00' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/flask_httpauth.py", line 104, in decorated
    return f(*args, **kwargs)
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/medallion/views/objects.py", line 28, in get_or_add_objects
    objects = current_app.medallion_backend.get_objects(api_root, id_, request.args, ("id", "type", "version"))
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/medallion/backends/mongodb_backend.py", line 128, in get_objects
    {"mongodb_collection": api_root_db["manifests"], "_collection_id": id_}
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/medallion/filters/mongodb_filter.py", line 48, in process_filter
    added_after_timestamp = common.convert_to_stix_datetime(added_after_date)
  File "/Users/user/envs/mockweb/lib/python3.6/site-packages/medallion/utils/common.py", line 103, in convert_to_stix_datetime
    return dt.datetime.strptime(timestamp_string, "%Y-%m-%dT%H:%M:%SZ")
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py", line 565, in _strptime_datetime
    tt, fraction = _strptime(data_string, format)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py", line 362, in _strptime
    (data_string, format))
ValueError: time data '2018-06-29T12:58:28-04:00' does not match format '%Y-%m-%dT%H:%M:%SZ'
127.0.0.1 - - [29/Jun/2018 13:07:14] "GET /trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/?added_after=2018-06-29T12:58:28-04:00 HTTP/1.1" 500 -

Improve 2.1 tests

We should try to test only a single concept per test. If several tests need to share the same setup, let's make another class that subclasses TaxiiTest and use the setUp method.

We should also increase the code coverage as much as we can.

Bug: error adding STIX2 bundle via cti-taxii-client #bug

Adding STIX bundle via "cti-taxii-client", the server returns http error code 500.

[17/May/2019 10:10:25] "POST /info/collections/c0616fa4-fa24-4bb7-91d7-4ebb13a176b2/objects/ HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.5/dist-packages/flask_httpauth.py", line 104, in decorated
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/medallion/views/objects.py", line 50, in get_or_add_objects
    status = current_app.medallion_backend.add_objects(api_root, id_, request.get_json(force=True), request_time)
  File "/usr/local/lib/python3.5/dist-packages/medallion/backends/memory_backend.py", line 191, in add_objects
    api_info["status"].append(status)
AttributeError: 'dict' object has no attribute 'append'

Proposed a bugfix as poll request #49
Hope this help.

ImportError: cannot import name create_app

Hi Team,

When I start the medallion server using

python medallion/scripts/run.py example_configs/memory_backend_config_auth_from_file.json OR

python medallion/scripts/run.py example_configs/directory_backend_config_auth_from_file.json

It throws an error

Traceback (most recent call last):
File "medallion/scripts/run.py", line 7, in
from medallion import version, create_app
ImportError: cannot import name create_app

Can you please help? Thanks

spec_version grab latest value

TAXii 2.1 spec says that if a spec_value parameter is not given, each object must be the latest spec_version of the object that is applicable. Currently, the memory backend does not do this and it must be implemented.

BUG: checking new object IDs as existing already always fail

cti-taxii-server/medallion/backends/memory_backend.py

Line 166 in f639892

if new_obj["id"] in collection["objects"]:

This line will always fail, you're checking that the ID of the object belongs in the array of objects instead of checking if the ID of the object exists as the ID of any of the objects

Add ability to easily plug in a backend

Currently if you want to add a new backend, you have to fork and modify the main source code.

It would be nice if you could pass a Python module for the backend instead, this would let people provide their own implementations .

TAXII Server to use HTTPS

Hello !

I have a simple question for the TAXII server. I am wondering, as I could not find it documented in the readthedocs, but if I wanted to have all TAXII communication to be done via HTTPS versus HTTP , how can I enforce that, or where do I enforce it?

An example could also be helpful if possible to be provided!

Thank you!

Improve Mongo backend query efficiency

The current backend stores manifests and objects separately (they are separate collections). Several types of queries require "joining" those two collections together, to match each object to its corresponding manifest. The way this is done is complicated and quite inefficient, owing to the fact that Mongo's "$lookup" pipeline step can only join on one property, and matching an object to its manifest requires checking both ID and version. An additional complication is that an object's version isn't in a fixed property, e.g. it may come from either the "created" or "modified" properties.

Since manifests and objects map 1:1 in TAXII 2.1, one suggestion is to store an object and its manifest within the same Mongo document. That way, no join is required to match them up.

Issue with MongoDBFilter when request involves single version objects and regular versioned objects

The following code needs to be updates:

cti-taxii-server/medallion/filters/mongodb_filter.py

Lines 188 to 205 in b9441e1

	def filter_contains_marking_definition(self, pipeline):
	# If we are matching on id (either match[id]= or /{id}), then check if
	# we are trying to find a marking definition. If so, we don't want do
	# filter by version as marking-definition objects are not versioned.
	if "id" in pipeline[0]["$match"].keys() and pipeline[0]["$match"]["id"].startswith("marking-definition"):
	return True
	if "_type" in pipeline[0]["$match"].keys():
	if ((
	isinstance(pipeline[0]["$match"]["_type"], dict) and
	"$in" in pipeline[0]["$match"]["_type"].keys()
	) and
	("marking-definition" in pipeline[0]["$match"]["_type"]["$in"])):
	return True
	elif pipeline[0]["$match"]["_type"].startswith("marking-definition"):
	return True
	return False

and here...

cti-taxii-server/medallion/filters/mongodb_filter.py

Lines 44 to 62 in b9441e1

	# need to handle marking-definitions differently as they are not versioned like SDO's
	if self.filter_contains_marking_definition(pipeline):
	# If we are finding marking-definitions from the objects collection
	# we need to change the match criteria from "_type" to "type"
	if data.name == "objects" and "_type" in pipeline[0]["$match"].keys():
	pipeline[0]["$match"]["type"] = pipeline[0]["$match"].pop("_type")
	# Calculate total number of matching documents
	if data.name == "objects":
	count = self.get_result_count(pipeline, manifest_info["mongodb_collection"])
	else:
	count = self.get_result_count(pipeline, data)
	self.add_pagination_operations(pipeline)
	cursor = data.aggregate(pipeline)
	results = list(cursor)
	return count, results

Return proper HTTP response codes

There are several situations where the server isn't returning the correct HTTP response code as defined in the spec, and we need to. For example, if an unaccepted content-type header is received.

Re-implement spec_version auto filtering

Work done by #94 needed to be removed during the branch movement.

Reimplement this work and remember to modify the tests affected by this.

Internal server error 500 when getting objects with match filter

I'm running Medallion 0.4.0 server under Ubuntu 16, Python 3.5 and MongoDB 4 as backend.

I want to launch a query with filters, according to the TAXII 2 specification, and this request works:
GET /trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/?match%5Bid%5D=identity--733c5838-34d9-4fbf-949c-62aba761184c
====> 200 OK

But when I add a second object id in the match filter, it returns a 500 error:
GET /trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/?match%5Bid%5D=identity--733c5838-34d9-4fbf-949c-62aba761184c%2Cthreat-actor--dfaa8d77-07e2-4e28-b2c8-92e9f7b04428
====> 500 Error

Is it supported the match query parameter in this way?

BasicFilter.process_filter() can have index error when called with empty object list

This came out during testing for python-stix2.TAXIICollectionSource. Looks to be a bug as process_filter() is called many times from within the taxii-server library where it is possible and appropriate that it would be called and supplied with an empty list of objects to operate on.

Index range error triggered by this line when data=[]:

cti-taxii-server/medallion/filters/basic_filter.py

Line 179 in 6b9a165

if self.is_manifest_entry(data[0]):

Update Docker configuration

The docker_config.json is out-of-date based on the recent merged PR.

1:1 relationship between manifests and objects are not checked when server is created

When a server is instantiated with a data file specified in the config, if an object is missing a manifest, the server will crash when trying to return that object. I suggest implementing a check when the server is created to ensure each manifest has a related object.

Python throws KeyError: 'taxii' error when starting medallion

Hey all,

I'm trying to spin up a test TAXII server for a project. I found this implementation and decided to give it a go instead of rolling my own.

The problem

When I run medallion config.json, I get the following error:

Traceback (most recent call last):
  File "/usr/bin/medallion", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/medallion/scripts/run.py", line 78, in main
    set_config(application_instance, "taxii", configuration)
  File "/usr/lib/python3.6/site-packages/medallion/__init__.py", line 39, in set_config
    flask_application_instance.taxii_config = config[prop_name]
KeyError: 'taxii'

Setup steps performed

I followed the instructions in the Installation and Usage sections to set up medallion:

• I installed python3.6 and pip3.6
• I ran sudo pip3.6 install medallion
• I created my config file based on the Usage instructions (see below)
• I ran medallion config.json, which produced the error reported above.

Config file:

{
     "backend": {
        "module": "medallion.backends.mongodb_backend",
        "module_class": "MongoBackend",
        "uri": "mongodb://localhost:27017/"
     },
    "users": {
       "admin": "Password0",
       "user1": "Password1",
       "user2": "Password2"
    }
}

Notes

• OS: Centos 7.6
• Python version: 3.6.8
• medallion version: 0.4.0

Any ideas how to get around this?

Thanks in advance!

Delete Object: Not implemented for MemoryBackend

This endpoint is yet to be implemented for the Memory Backend

Add support for X-TAXII Custom Headers

Currently, the TAXII server responses do not provide the following headers: X-TAXII-Date-Added-First and X-TAXII-Date-Added-Last.

memory_backend.py update_manifest() is expecting all new_obj STIX objects to have a "modified" element. Not all STIX 2.0 objects do.

It appears that update_manifest() requires all STIX objects to have a "modified" element, that it uses to update the STIX object's version history. If there is not a modified element present, it throws an error.

However, not all STIX 2.0 objects have a "modified" element. For example, a Marking-Definition STIX 2.0 object cannot have a "modified" element. If your STIX bundle contains a Marking-Definition object, update_manifest() will throw an error.

Unable to run cti-taxii-server with uwsgi

Has anyone here have an example of running this using uwsgi?

If so can you please share?

my attempt:

*** Starting uWSGI 2.0.17 (64bit) on [Wed Feb 28 17:38:52 2018] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-16) on 28 February 2018 17:02:10
os: Linux-3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017
nodename: taxii.local
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/vagrant/taxii-server/cti-taxii-server/medallion
detected binary path: /usr/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 4096
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:3031 fd 3
Python version: 3.6.4 (default, Dec 19 2017, 14:48:12)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
Python main interpreter initialized at 0x15d3460
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 416720 bytes (406 KB) for 8 cores
*** Operational MODE: preforking+threaded ***
Traceback (most recent call last):
  File "__init__.py", line 1, in <module>
    from flask import Flask
ModuleNotFoundError: No module named 'flask'
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 30544)
spawned uWSGI worker 1 (pid: 30545, cores: 2)
spawned uWSGI worker 2 (pid: 30546, cores: 2)
spawned uWSGI worker 3 (pid: 30547, cores: 2)
spawned uWSGI worker 4 (pid: 30548, cores: 2)
*** Stats server enabled on 127.0.0.1:9191 fd: 15 ***
^CSIGINT/SIGQUIT received...killing workers...
worker 2 buried after 0 seconds
worker 1 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
goodbye to uWSGI.

MongoDB ISODate limited to millisecond precision

Since we currently store the values for date_added and request_time as an ISODate we cannot store higher precision timestamps as required by these properties.

We should change it to use regular strings but it should not affect any of our current pipeline filter work.

it seems to be a typo in README.

Hi.
It seems to be typo in README.rst where example of config file for MongoBackend.

{
     "backend": {
        "module": "medallion.backends.mongodb_backend",
        "module_class": "MongoBackend",
        "url": <Mongo DB server url>  # e.g., "mongodb://localhost:27017/"
     }
}

It seems to be url -> uri, because of the MongoBackend class initializer's argument name is uri.

class MongoBackend(Backend):

    # access control is handled at the views level

    def __init__(self, uri=None, **kwargs):
        try:
            self.client = MongoClient(uri)
            # The ismaster command is cheap and does not require auth.
            self.client.admin.command("ismaster")
        except ConnectionFailure:
            log.error("Unable to establish a connection to MongoDB server {}".format(uri))

Thanks.

Updates to DirectoryBackend

This backend could be improved because there are some inconsistencies in its design:

• Some resources are defined in the config document (which drastically add up as you add more api roots, collections, server discovery, etc)
• Although the objects are "cached" many other resources are not like the status and the aforementioned resources. The cache mechanism itself could be improved
• The directory structure could be improved as handling them via bundles is impractical

cti-taxii-server/medallion/backends/taxii/directory_backend.py

Lines 1 to 342 in b9441e1

	import datetime
	import json
	import os
	import uuid
	from medallion.backends.taxii.base import Backend
	from medallion.exceptions import ProcessingError
	from medallion.filters.basic_filter import BasicFilter
	from medallion.utils.common import create_bundle, generate_status
	class DirectoryBackend(Backend):
	# access control is handled at the views level
	def __init__(self, path=None, **kwargs):
	self.path = path
	self.discovery_config = self.init_discovery_config(kwargs.get('discovery', None))
	self.api_root_config = self.init_api_root_config(kwargs.get('api-root', None))
	self.collection_config = self.init_collection_config(kwargs.get('collection', None))
	self.cache = {}
	self.statuses = []
	# noinspection PyMethodMayBeStatic
	def init_discovery_config(self, discovery_config):
	if not self.path:
	raise ProcessingError('path was not specified in the config file', 400)
	if not os.path.isdir(self.path):
	raise ProcessingError("directory '{}' was not found".format(self.path), 500)
	return discovery_config
	def update_discovery_config(self):
	dc = self.discovery_config
	collection_dirs = sorted([f for f in os.listdir(self.path) if os.path.isdir(os.path.join(self.path, f))])
	if not collection_dirs:
	raise ProcessingError('at least one api-root directory is required', 500)
	updated_roots = ['{}{}/'.format(dc['host'], f) for f in collection_dirs]
	self.discovery_config['default'] = updated_roots[0]
	self.discovery_config['api_roots'] = updated_roots
	# noinspection PyMethodMayBeStatic
	def init_api_root_config(self, api_root_config):
	if api_root_config:
	return api_root_config
	else:
	raise ProcessingError('api-root was not specified in the config file', 400)
	# noinspection PyMethodMayBeStatic
	def init_collection_config(self, collection_config):
	if collection_config:
	return collection_config
	else:
	raise ProcessingError('collection was not specified in the config file', 400)
	def validate_requested_api_root(self, requested_api_root):
	api_roots = self.discovery_config['api_roots']
	host_port = self.discovery_config['default'].rsplit('/', 2)[0]
	full_api_root = '{}/{}/'.format(host_port, requested_api_root)
	return full_api_root in api_roots
	def server_discovery(self):
	self.update_discovery_config()
	return self.discovery_config
	def get_api_root_information(self, api_root):
	self.update_discovery_config()
	api_roots = self.discovery_config['api_roots']
	for r in api_roots:
	c_dir = r.rsplit('/', 2)[1]
	if api_root == c_dir:
	i_title = "Indicators from directory '{}'".format(c_dir)
	i = {
	"title": i_title,
	"description": "",
	"versions": self.api_root_config['versions'],
	"max-content-length": self.api_root_config['max-content-length']
	}
	return i
	def get_collections(self, api_root, start_index, end_index):
	self.update_discovery_config()
	api_roots = self.discovery_config['api_roots']
	collections = []
	# Generate a collection object for each api_root
	for r in api_roots:
	c_dir = r.rsplit('/', 2)[1]
	if api_root == c_dir:
	c_id = uuid.uuid3(uuid.NAMESPACE_URL, r)
	c_title = "Indicators from directory '{}'".format(c_dir)
	c = {
	"id": str(c_id),
	"title": c_title,
	"description": self.collection_config['description'],
	"can_read": self.collection_config['can_read'],
	"can_write": self.collection_config['can_write'],
	"media_types": self.collection_config['media_types']
	}
	collections.append(c)
	count = len(collections)
	collections = collections if end_index == -1 else collections[start_index:end_index]
	return count, collections
	def get_collection(self, api_root, collection_id):
	count, collections = self.get_collections(api_root, 0, -1)
	for c in collections:
	if 'id' in c and collection_id == c['id']:
	return c
	def set_modified_time_stamp(self, objects, modified):
	for o in objects:
	o['modified'] = modified
	return objects
	def get_modified_time_stamp(self, fp):
	fp_modified = os.path.getmtime(fp)
	dt = datetime.datetime.utcfromtimestamp(fp_modified)
	modified = '{:%Y-%m-%dT%H:%M:%S.%fZ}'.format(dt)
	return modified
	def delete_from_cache(self, api_root):
	p = os.path.join(self.path, api_root)
	files = [f for f in os.listdir(p) if os.path.isfile(os.path.join(p, f)) and f.endswith('.json')]
	for f in self.cache[api_root]['files'].keys():
	if f not in files:
	del self.cache[api_root]['files'][f]
	def add_to_cache(self, api_root, api_root_modified, file_name, file_modified):
	fp = os.path.join(self.path, api_root, file_name)
	u_objects = []
	with open(fp, 'r') as raw_json:
	try:
	stix2 = json.load(raw_json)
	if stix2.get('type', '') == 'bundle' and stix2.get('spec_version', '') == '2.0':
	objects = stix2.get('objects', [])
	u_objects = self.set_modified_time_stamp(objects, file_modified)
	if api_root not in self.cache:
	self.cache[api_root] = {'modified': '', 'files': {}}
	self.cache[api_root]['modified'] = api_root_modified
	self.cache[api_root]['files'][file_name] = {'modified': file_modified, 'objects': u_objects}
	except Exception as e:
	raise ProcessingError('error adding objects to cache', 500, e)
	finally:
	return u_objects
	def with_cache(self, api_root):
	api_root_path = os.path.join(self.path, api_root)
	api_root_modified = self.get_modified_time_stamp(api_root_path)
	if api_root in self.cache:
	if self.cache[api_root]['modified'] == api_root_modified:
	# Return objects from cache
	objects = []
	for k, v in self.cache[api_root]['files'].items():
	objects.extend(v['objects'])
	return objects
	else:
	# Cleanup the cache
	self.delete_from_cache(api_root)
	# Add to the cache and return objects for collection
	dir_list = os.listdir(api_root_path)
	files = [f for f in dir_list if os.path.isfile(os.path.join(api_root_path, f)) and f.endswith('.json')]
	objects = []
	for f in files:
	fp = os.path.join(api_root_path, f)
	file_modified = self.get_modified_time_stamp(fp)
	cached_files = self.cache[api_root]['files']
	if f in cached_files and cached_files[f]['modified'] == file_modified:
	objects.extend(cached_files[f]['objects'])
	else:
	u_objects = self.add_to_cache(api_root, api_root_modified, f, file_modified)
	objects.extend(u_objects)
	return objects
	else:
	# Update the cache and return the objects for the collection
	dir_list = os.listdir(api_root_path)
	files = [f for f in dir_list if os.path.isfile(os.path.join(api_root_path, f)) and f.endswith('.json')]
	objects = []
	for f in files:
	fp = os.path.join(api_root_path, f)
	file_modified = self.get_modified_time_stamp(fp)
	u_objects = self.add_to_cache(api_root, api_root_modified, f, file_modified)
	objects.extend(u_objects)
	return objects
	def get_objects_without_bundle(self, api_root, collection_id, filter_args, allowed_filters):
	self.update_discovery_config()
	if self.validate_requested_api_root(api_root):
	# Get the collection
	collection = None
	num_collections, collections = self.get_collections(api_root, 0, -1)
	for c in collections:
	if 'id' in c and collection_id == c['id']:
	collection = c
	break
	if not collection:
	raise ProcessingError("collection for api-root '{}' was not found".format(api_root), 500)
	# Add the objects to the collection
	collection['objects'] = self.with_cache(api_root)
	# Filter the collection
	filtered_objects = []
	if filter_args:
	full_filter = BasicFilter(filter_args)
	filtered_objects.extend(
	full_filter.process_filter(
	collection.get('objects', []),
	allowed_filters,
	collection.get('manifest', [])
	)
	)
	else:
	filtered_objects.extend(collection.get('objects', []))
	return filtered_objects
	def get_objects(self, api_root, collection_id, filter_args, allowed_filters, start_index, end_index):
	# print('start_index: {}, end_index: {}'.format(start_index, end_index))
	objects = self.get_objects_without_bundle(api_root, collection_id, filter_args, allowed_filters)
	objects.sort(key=lambda x: datetime.datetime.strptime(x['modified'], '%Y-%m-%dT%H:%M:%S.%fZ'))
	count = len(objects)
	objects = objects if end_index == -1 else objects[start_index:end_index]
	return count, create_bundle(objects)
	def get_object(self, api_root, collection_id, object_id, filter_args, allowed_filters):
	objects = self.get_objects_without_bundle(api_root, collection_id, filter_args, allowed_filters)
	req_object = [i for i in objects if i['id'] == object_id]
	if len(req_object) == 1:
	return create_bundle(req_object)
	def get_object_manifest(self, api_root, collection_id, filter_args, allowed_filters, start_index, end_index):
	self.update_discovery_config()
	if self.validate_requested_api_root(api_root):
	count, collections = self.get_collections(api_root, 0, -1)
	for collection in collections:
	if 'id' in collection and collection_id == collection['id']:
	manifest = collection.get('manifest', [])
	if filter_args:
	full_filter = BasicFilter(filter_args)
	manifest = full_filter.process_filter(
	manifest,
	allowed_filters,
	None
	)
	count = len(manifest)
	manifest = manifest if end_index == -1 else manifest[start_index:end_index]
	return count, manifest
	def add_objects(self, api_root, collection_id, objs, request_time):
	failed = 0
	succeeded = 0
	pending = 0
	successes = []
	failures = []
	file_name = '{}--{}.{}'.format(request_time, objs['id'], 'json')
	p = os.path.join(self.path, api_root)
	fp = os.path.join(p, file_name)
	try:
	add_objs = objs['objects']
	num_objs = len(add_objs)
	try:
	# Each add_object request writes the provided bundle to a new file
	with open(fp, 'w') as out_file:
	out_file.write(json.dumps(objs, indent=4, sort_keys=True))
	succeeded += num_objs
	successes = list(map(lambda x: x['id'], add_objs))
	# Update the cache after the file is written
	self.with_cache(api_root)
	except IOError:
	failed += num_objs
	failures = list(map(lambda x: x['id'], add_objs))
	except Exception as e:
	raise ProcessingError('error adding objects', 500, e)
	status = generate_status(request_time, 'complete', succeeded, failed,
	pending, successes_ids=successes, failures=failures)
	self.statuses.append(status)
	return status
	def get_status(self, api_root, status_id):
	for s in self.statuses:
	if status_id == s['id']:
	return s

For example:

<root-path>/
     discovery.json
     <api_root1>/
          <api_root1>.json
          status/
               <status_id>.json
          <collection_id>/
               <collection_id>.json
               objects/
                   <type>/
                        <uuid>/
                             <modified | created | request_time>.json
               manifest/
                       <type>/
                            <type>--<uuid>.json

MissingPropertiesError: No values for required properties for Relationship: (relationship_type, source_ref, target_ref).

hello,

I wanted to push a relationship object to taxii server with below:

from stix2 import TAXIICollectionSink, ThreatActor

#create TAXIICollectionSINK and push STIX content to it
tc_sink = TAXIICollectionSink(collection)

# tc_sink.add(bundle)
tc_sink.add(identity)
tc_sink.add(indicator)
tc_sink.add(relationship)

relationship = Relationship(relationship_type='indicates',
                            source_ref=indicator.id,
                            target_ref=malware.id)

tc_sink.add(relationship)

then, I tried to pull them to apply to our application, but an error occured:

# supply the TAXII2 collection to TAXIICollection
tc_source = TAXIICollectionSource(collection)

#retrieve STIX objects by id
stix_obj = tc_source.get("relationship--3d899276-f4bd-445d-a4d8-9122161a6dcc")

but when I trying to retrieve them, the return object lost the relationship_type, source_ref, target_ref, however I checked mongo db, all properties stored, could you know how to address it?

More header checks

Add a few more header checks for TAXII 2.0

Attach backend to app instance

Rather than calling get_backend() all the time, we can just create it when the app loads and attach it to the Flask object: https://github.com/oasis-open/cti-taxii-server/blob/master/medallion/__init__.py#L6

We don't need to worry about lazily instantiating the variable. If it isn't valid, we can error out quickly when the server is starting.

Object of type ObjectId is not JSON Serializable

Greetings Everyone,

Been playing around with the cti-taxii server and I've come into an issue. It seems that when running Medallion and using MongoDB, I have an issue in trying to retrieve objects from a collection and insert.

I have medallion use a config file that specifies that the backend to be used is my local mongoDB instance.

Attached is an image of Medallion output

Since I was playing around with it in a python3 interpreter, the following is what I did

collection = Collection("http:/127.0.0.1:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/",user="admin",password="Password0")
mw = Malware(name="Test",description="Real bad malware",labels=["ransomware"])
collection.add_objects(json.loads(Bundle(mw).serialize())) #Crashes here and gives the image above
print(collection.get_objects()) #Same issue

Oddly enough, if I use the config.json file that uses the memory as backend, I have no problems with it at all, it works successfully.

Am I just doing a typo? or is this a bug?

MemoryBackend fails to identify duplicate objects correctly

The current logic only holds true if you find the duplicate at the end of the object list, otherwise the boolean value is overwritten.

cti-taxii-server/medallion/backends/taxii/memory_backend.py

Lines 172 to 191 in b9441e1

	for new_obj in objs["objects"]:
	id_and_version_already_present = False
	for obj in collection["objects"]:
	id_and_version_already_present = False
	if new_obj["id"] == obj["id"]:
	if "modified" in new_obj:
	if new_obj["modified"] == obj["modified"]:
	id_and_version_already_present = True
	else:
	# There is no modified field, so this object is immutable
	id_and_version_already_present = True
	if not id_and_version_already_present:
	collection["objects"].append(new_obj)
	self._update_manifest(new_obj, api_root, collection["id"], request_time)
	successes.append(new_obj["id"])
	succeeded += 1
	else:
	failures.append({"id": new_obj["id"], "message": "Unable to process object"})
	failed += 1

Return empty envelope in certain cases

Specifically on the https://github.com/oasis-open/cti-taxii-server/tree/taxii2.1-csd03 branch, in the memory backend, there are a couple cases where endpoints return 404 but should instead return an empty envelope:

• the get-objects endpoint, if a collection is empty and no filters are provided
• the get-manifests endpoint, if a collection is empty or if filters filtered everything out

Thanks to @chisholm for finding this.