misp/misp-stix-converter

A utility repo to assist with converting between MISP and STIX formats

MISP to STIX and back again

Or at least that's the idea

This is the open-sourced version of BAE Systems' internal sync script. It's a bit limited, and it isn't perfect, nor is it bug-free.

But it works™

Installation

If you don't wanna use git,

sudo pip3 install misp_stix_converter

should have you covered. This relies on me actually updating PyPI every time I update the project, so I'd use the git repo wherever possible.

sudo python3 setup.py install

This should install everything it needs!

Useful, huh?

The config file

Copy over the example config to a live version

The default location of this config file is at ~/.misptostix, but this can be overridden with the -c FILE flag when running the scripts.

cp /path/to/config/misp.login.example /path/to/config/misp.login

Then open it and change the variables. This is YAML format, so make sure you don't do a silly and format it wrong!

Usage

For MISP to STIX:

To get all usage information:

misp-to-stix.py -h

So to convert a MISP JSON file to stix json, use

misp-to-stix.py -f INFILE.json --format JSON -o OUTFILE.json (the --format flag is used to specify output format, just for reference)

And to pull a specific event from the MISP instance

misp-to-stix.py -i EVENT_ID...

Alternatively, if you want to pull every event of a certain tag, you can run

misp-to-stix.py -t tlp:white -o out.{}.xml

Which will write all "tlp:white" tagged events to a file formatted by the event's ID, e.g out.29.xml

For STIX to MISP

This only works if you have a live MISP instance to connect the API to.

./stix-to-misp.py INFILE.json

This will convert the file to MISP format and push it. Quite a few bits and bobs get converted, not all of them (I for one blame CyBoX for being weird and layered worse than an onion).

Feel free to add more to the stix-to-misp.py file.

misp-stix-converter's Issues

Error Download as "STIX 1 JSON (metadata + all attributes)"

When I download event from MISP Download as -> STIX 1 JSON (metadata + all attributes) or STIX 2 It doesn't work on a fresh installation of MISP v2.4.150

Data:
MISP v2.4.150
Python 3.6.8
CentOS 7
Guide: https://misp.github.io/MISP/INSTALL.rhel7/

Traceback (most recent call last):
 File "/var/www/MISP/app/files/scripts/stix2/misp2stix2.py", line 34, in <module>
 from misp_stix_converter import MISPtoSTIX20Parser, MISPtoSTIX21Parser
 File "/var/www/MISP/app/files/scripts/misp-stix/misp_stix_converter/__init__.py", line 3, in <module>
 from .misp_stix_converter import misp_attribute_collection_to_stix1, misp_collection_to_stix2_0, misp_collection_to_stix2_1, misp_event_collection_to_stix1, misp_to_stix1, misp_to_stix2_0, misp_to_stix2_1
 File "/var/www/MISP/app/files/scripts/misp-stix/misp_stix_converter/misp_stix_converter.py", line 22, in <module>
 from typing import List, TypedDict, Union
ImportError: cannot import name 'TypedDict'

I solve the error as follows:
References: https://pypi.org/project/typing-extensions/

git diff misp_stix_converter.py
diff --git a/misp_stix_converter/misp_stix_converter.py b/misp_stix_converter/misp_stix_converter.py
index 1ec7fa2..636cbe8 100644
--- a/misp_stix_converter/misp_stix_converter.py
+++ b/misp_stix_converter/misp_stix_converter.py
@@ -19,7 +19,8 @@ from stix.core.ttps import TTPs
 from stix2.base import STIXJSONEncoder
 from stix2.v20 import Bundle as Bundle_v20
 from stix2.v21 import Bundle as Bundle_v21
-from typing import List, TypedDict, Union
+from typing import List, Union
+from typing_extensions import TypedDict
 from uuid import uuid4

 _default_namespace = 'https://github.com/MISP/MISP'

Regards

US-Cert STIX Files

Hi, if you include "import stix.extensions.marking.ais" in your imports before trying to load Stix file from XML it works fine without the need to remove US-Cert markings.

From the documentation of StixProject:

If you are writing code that needs to parse AIS markings, make sure that your
program imports this module before beginning to parse any STIX documents:
.. code-block:: python
import stix.extensions.marking.ais

Convert STIX descriptions to MISP comments

At the moment text in the STIX Indicator:Description field is lost. It would be nice to convert this to an attribute comment in MISP.

The same is true for the stix:Description field which could be converted to Catergory=other Type=comment in MISP.

indicator:Confidence missing after importing STIX1.1.1/2.0 files into MISP

Hello,

I am currently using MISP v2.4.148. I exported STIX files (both 1.1.1 and 2.0 versions) from Anomali TI and then successfully imported those STIX files inside the MISP instance using Import STIX feature. However, there is one issue: every indicator present in the STIX file has a "Confidence" value (indicator:Confidence), but that value is ignored/dropped by MISP during the import process.

Is there any solution to this problem? I really need that confidence value associated with each imported indicator inside the MISP. I am posting one example below for you to look at:

<stix:Indicators>
        <stix:Indicator id="threatstream:indicator-XXXXXX" timestamp="2021-09-12T18:13:40.701650+00:00" xsi:type='indicator:IndicatorType'>
            <indicator:Title>spam_ip: 80.109.155.86</indicator:Title>
            <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">IP Watchlist</indicator:Type>
            <indicator:Description>TS ID: 57531849646; iType: spam_ip; Date First: 2021-09-10T02:26:53.448Z; State: active; Org: Magenta Telekom; Source: NixSpam; MoreDetail: imported by user 668</indicator:Description>
            <indicator:Observable id="threatstream:Observable-XXXXXXX">
                <cybox:Title>spam_ip: 80.109.155.86</cybox:Title>
                <cybox:Keywords>
                    <cybox:Keyword>low</cybox:Keyword>
                </cybox:Keywords>
                <cybox:Object id="threatstream:Address-XXXXXXX">
                    <cybox:Properties xsi:type="AddressObj:AddressObjectType" category="ipv4-addr">
                        <AddressObj:Address_Value>80.109.155.86</AddressObj:Address_Value>
                    </cybox:Properties>
                </cybox:Object>
            </indicator:Observable>
            <indicator:Handling>
                <marking:Marking>
                    <marking:Controlled_Structure>../../../descendant-or-self::node()</marking:Controlled_Structure>
                    <marking:Marking_Structure xsi:type='tlpMarking:TLPMarkingStructureType' color="AMBER"/>
                </marking:Marking>
            </indicator:Handling>
            <indicator:Confidence timestamp="2021-09-12T18:13:40.702217+00:00">
                <stixCommon:Value vocab_reference="https://en.wikipedia.org/wiki/Percentage" vocab_name="Percentage">37</stixCommon:Value>
            </indicator:Confidence>
            <indicator:Producer>
                <stixCommon:Time>
                    <cyboxCommon:Produced_Time>2021-09-10T02:26:53.448000+00:00</cyboxCommon:Produced_Time>
                </stixCommon:Time>
            </indicator:Producer>
        </stix:Indicator>

ImportError: No module named 'misp_stix_converter.servers'

Hello,

Since last update I get the following error when launching stix-to-misp.py and misp-to-stix.py:

ImportError: No module named 'misp_stix_converter.servers'

All dependecies seems to be installed:

#####################

stix-to-misp.py
Traceback (most recent call last):
File "/usr/local/bin/stix-to-misp.py", line 4, in
import('pkg_resources').run_script('Stix-MISP-Converter==0.2', 'stix-to-misp.py')
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 534, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 1445, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python3.4/dist-packages/Stix_MISP_Converter-0.2-py3.4.egg/EGG-INFO/scripts/stix-to-misp.py", line 15, in
ImportError: No module named 'misp_stix_converter.servers'
pip3 list | grep -i stix
stix (1.2.0.1)
Stix-MISP-Converter (0.2)

############################

Thanks,

AIS STIX files that we don't have for testing but a small part of the puzzle

Following past AIS STIX file discussion (we are still looking for sample files for testing), there is an XSD with their marking extension:

• http://www.us-cert.gov/sites/default/files/STIX_Namespace/AIS_Bundle_Marking_1.1.1_v1.0.xsd

Not sure if this would help to complete partially this puzzle.

Automatically include the STIX file as an attachment

Problem with IPv4 Objects

Hello. when converting an MISPEvent to STIX when it finds ip-src or ip-dst it doesn't put the attribute category="ipv4-addr", so when the SIEM QRadar tries to obtain IP STIX Events it doesn't understand it so it skips these type of events. When i manually add the category it can understand them and consumes them.

thank you.

Unable to pull form MISP feed

Search_index() got an unexpected keyword argument 'tag' while using "sudo misp-to-stix.py -t tlp:white -o out.{}.xml"

MISPtoSTIX() uses example XML namespace

I'm preparing a patch for this (and other STIX best practices) but one mayor issue is that currently MISPtoSTIX() returns a STIXPackage and lets the main program do the to_xml() call. Since a custom namespace is defined by using the ns_dict parameter of to_xml() this complicates things quite a lot.

One solution I see is to pass the required format (json/xml) to MISPtoSTIX() along with the custom namespace. Or is there a better way to deal with this?

Add tag when creating event

Hello. How to add a tag when creating an event? Where do I need to change the script?

Bug preventing File and Email Message attributes being built

The conditional in the buildAttribute function of buildMISPAttribute.py checking to see if the current object being processed is a URI, Link URI, or Link object leads to the subsequent conditionals never being processed, as it will always return true.

The bug is on line 560:
elif type_ == uri_object.URI or type_ == link_object.URI or link_object.Link:

and should be changed to:
elif type_ == uri_object.URI or type_ == link_object.URI or type_ == link_object.Link:

TypeError: search_index() got an unexpected keyword argument 'dateto'

I received the following error when trying to convert misp-to-stix:

My command:
misp-to-stix.py -t tlp:white -o out.{}.xml -c /home/misp/misp-t2/bin/MISP-STIX-Converter/misp.login
or
misp-to-stix.py -i 46874 --format xml -o 46874.xml -c /home/misp/misp-t2/bin/MISP-STIX-Converter/misp.login

Error message:

Traceback (most recent call last):
  File "misp-to-stix.py", line 83, in <module>
    package = MISP.pull(args.eid)[0]
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/servers/misp.py", line 67, in pull
TypeError: search_index() got an unexpected keyword argument 'dateto'


MISP version: 2.4.67

MISP to STIX converter using a single indicator for all attributes

Currently (unless I am mistaken), all attributes are added as observables to a single indicator during the conversion (https://github.com/MISP/MISP-STIX-Converter/blob/master/misp_stix_converter/converters/convert.py#L67), however this is most of the time not correct for the actual data. Each attribute should be treated as a separate indicator with 1-2 observables each (For example a filename|sha1 type would translate to an indicator with 2 observables).

When pushing stix package, getting ids of MISP events created

Hey,

When pushing stix package, the corresponding misp event created is not published.
I'd like to publish it right after creation. For that, I need the event id.

Could you return the event id / list of event ids instead of True when using misp.push ?

Thx

Better STIX Parsing

Currently every observable is parsed out of a given STIX input and added as an attribute to a MISP event. In many cases this can give undesired results. A better solution would be as follows:

1. Create an observable map (python dictionary) with the observable id as a key and the observable object as the value, for every observable within the STIX package.
2. Create a TTP map (python dictionary) with the TTP id as a key and the TTP object as the value, for every TTP within the STIX package.
3. If there are indicators within the STIX package, parse the indicator. Use the observable map to retrieve observables part of that indicator (typically they'll be represented only by the ID within the indicator). Use the TTP map to retrieve TTPs part of that indicator (typically they'll be represented only by the ID within the TTP).

• Parsing indicators, rather than straight observables will give you the added value of confidence levels, TTPs, campaigns, actors, etc that can possibly be added as tags to the MISP attributes.
• When parsing indicators, you can ignore observables that are part of an "AND" composition to prevent unwanted results such as a ".exe" file extension attribute. These observables are part of a group and do not make sense on their own. It's best to ignore them until there's functionality within MISP for it.

4. If no indicators exists (some simple feeds will not use them), you can fall back to parsing straight observables. Just loop through the observable map you created before.

Here's some sample code of my implementation (It was a rush job so there is no logging/error checking in most of it) Simply import the stix_to_misp.py file and call the build_event method to create a MISP event. I use my own taxonomy for confidence and TTPs, but those can be changed as needed:

stix_parser.py

from cybox.objects import email_message_object, file_object, address_object, socket_address_object
from cybox.objects import domain_name_object, hostname_object, uri_object


def build_observable_map(pkg):
    map = {}
    if pkg.observables:
        for o in pkg.observables.observables:
            try:
                map[o.id_] = o
            except AttributeError:
                continue
    return map


def build_ttp_map(pkg):
    map = {}
    if pkg.ttps:
        for t in pkg.ttps.ttps:
            try:
                map[t.id_] = t.title
            except AttributeError:
                continue
    return map


def process_indicators(pkg):
    observable_map = build_observable_map(pkg)
    ttp_map = build_ttp_map(pkg)
    indicators = []
    if pkg.indicators:
        for i in pkg.indicators:
            indicator = dict()
            indicator["itypes"] = get_indicator_types(i)
            indicator["confidence"] = get_confidence(i)
            indicator["ttps"] = get_ttps(i, ttp_map)
            indicator["attributes"] = get_indicator_attributes(i, observable_map)

            indicators.append(indicator)

    return indicators


def process_observables(pkg):
    observable_map = build_observable_map(pkg)
    if pkg.observables:
        return get_observable_attributes(pkg.observables.observables, observable_map)
    else:
        return []


def get_indicator_types(indicator):
    itypes = []
    for i in indicator.indicator_types:
        itypes.append(i.value)

    return itypes


def get_confidence(indicator):
    if indicator.confidence:
        return indicator.confidence.value.value
    else:
        return "Unknown"


def get_observable_attributes(observables, observable_map):
    attributes = []
    for o in observables:
        if hasattr(o, "idref") and o.idref:
            try:
                observable = observable_map[o.idref]
            except KeyError:
                continue

            extract_observable(observable, attributes)
        else:
            extract_observable(o, attributes)

    return attributes


def get_indicator_attributes(indicator, observable_map):
    observables = []
    if indicator.observable:
        observables += process_observable(indicator.observable)
    if indicator.observables:
        for o in indicator.observables:
            observables += process_observable(o)

        return get_observable_attributes(observables, observable_map)


def get_ttps(indicator, ttp_map):
    ttps = []
    if indicator.indicated_ttps:
        for ttp in indicator.indicated_ttps:
            if ttp.item.idref:
                try:
                    t = ttp_map[ttp.item.idref]
                except KeyError:
                    continue

                ttps.append(t)
            else:
                ttps.append(ttp.item.title)
    return ttps


def process_observable(observable):
    observables = []

    if observable.observable_composition:
        if observable.observable_composition.operator == "OR":
            for o in observable.observable_composition.observables:
                observables += process_observable(o)
    else:
        observables.append(observable)

    return observables


def add_attribute(obj, attributes):
    if not any(a["value"] == obj["value"] for a in attributes):
        attributes.append(obj)


# Dedicated to File object
def add_file_attr(obj, attributes):

    if obj.file_name:
        add_attribute({"type": "filename", "value": str(obj.file_name)}, attributes)

    if obj.md5:
        if len(obj.md5) == 32:
            add_attribute({"type": "md5", "value": str(obj.md5)}, attributes)

    if obj.sha1:
        if len(obj.sha1) == 40:
            add_attribute({"type": "sha1", "value": str(obj.sha1)}, attributes)

    if obj.sha256:
        if len(obj.sha256) == 64:
            add_attribute({"type": "sha256", "value": str(obj.sha256)}, attributes)

    if obj.sha512:
        if len(obj.sha512) == 128:
            add_attribute({"type": "sha512", "value": str(obj.sha512)}, attributes)


# Dedicated to Address Object (DB)
def add_addr_attr(obj, attributes):
    if obj.is_source:
        add_attribute({"type": "ip-src", "value": str(obj.address_value)}, attributes)

    elif obj.is_destination:
        add_attribute({"type": "ip-dst", "value": str(obj.address_value)}, attributes)

    else:
        # We don't know, first check if it's an IP range
        if hasattr(obj, "condition") and obj.condition:
            if obj.condition == "InclusiveBetween":
                add_attribute({"type": "ip-dst", "value": str(obj.address_value[0])}, attributes)
                add_attribute({"type": "ip-dst", "value": str(obj.address_value[1])}, attributes)
        else:
            add_attribute({"type": "ip-dst", "value": str(obj.address_value)}, attributes)


# Dedicated to EmailMessage (DB)
def add_email_attr(obj, attributes):
    if obj.header:
        # We have a header, can check for to/from etc etc
        if obj.header.from_:
            add_attribute({"type": "email-src", "value": str(obj.header.from_.address_value)}, attributes)
        if obj.header.to:
            for mail in obj.header.to:
                add_attribute({"type": "email-dst", "value": str(mail.address_value)}, attributes)
        if obj.header.subject:
            add_attribute({"type": "email-subject", "value": str(obj.header.subject)}, attributes)


# Dedicated to Domain name (DB)
def add_domain_attr(obj, attributes):
    add_attribute({"type": "domain", "value": str(obj.value)}, attributes)


# Dedicated to Hostname (DB)
def add_hostname_attr(obj, attributes):
    add_attribute({"type": "hostname", "value": str(obj.hostname_value)}, attributes)


# Dedicated to URI (DB)
def add_uri_attr(obj, attributes):
    add_attribute({"type": "url", "value": str(obj.value)}, attributes)


def extract_observable(observable, attributes):

    if hasattr(observable, "object_") and observable.object_:
        prop = observable.object_.properties

        if type(prop) == address_object.Address:
            # Now script uses buildAddressAttribute (DB)
            add_addr_attr(prop, attributes)

        elif type(prop) == domain_name_object.DomainName:
            # Now script uses buildDomainNameAttribute (DB)
            add_domain_attr(prop, attributes)

        elif type(prop) == hostname_object.Hostname:
            # Now script uses buildHostnameAttribute
            add_hostname_attr(prop, attributes)

        elif type(prop) == socket_address_object.SocketAddress:
            if prop.ip_address:
                add_addr_attr(prop.ip_address, attributes)
            if prop.hostname:
                add_hostname_attr(prop.hostname, attributes)

        elif type(prop) == uri_object.URI:
            # Now script uses buildURIAttribute (DB)
            add_uri_attr(prop, attributes)

        elif type(prop) == file_object.File:
            # Now script uses buildFileAttribute (DB)
            add_file_attr(prop, attributes)

        elif type(prop) == email_message_object.EmailMessage:
            # Now script uses buildEmailMessageAttribute (DB)
            add_email_attr(prop, attributes)
        else:
            pass
    else:
        pass

stix_to_misp.py

from stix_parser import process_indicators, process_observables
from stix.core import STIXPackage
from tempfile import SpooledTemporaryFile
import pymisp


def load_stix(stix):
    # Just save the pain and load it if the first character is a <

    if isinstance(stix, STIXPackage):
        # Oh cool we're ok
        # Who tried to load this? Honestly.
        return stix

    elif hasattr(stix, 'read'):
        try:
            stix_package = STIXPackage.from_xml(stix)
        except Exception as ex:
            print("Could not load file")
            return

        return stix_package

    elif isinstance(stix, str):
        # It's text, we'll need to use a temporary file
        f = SpooledTemporaryFile(max_size=10 * 1024)
        f.write(stix.encode("utf-8"))
        f.seek(0)

        return load_stix(f)


def build_event(content, **kwargs):
    pkg = load_stix(content)
    event = pymisp.MISPEvent()

    if pkg.stix_header and pkg.stix_header.title:
        event.info = pkg.stix_header.title
    else:
        event.info = "NO_TITLE"

    event.distribution = kwargs.get("distribution", 0)
    event.threat_level_id = kwargs.get("threat_level_id", 3)
    event.analysis = kwargs.get("analysis", 0)

    if pkg.indicators:
        indicators = process_indicators(pkg)
        for i in indicators:
            build_attribute(i, event)
    elif pkg.observables:
        indicator = dict()
        indicator["attributes"] = process_observables(pkg)
        build_attribute(indicator, event)
    else:
        return None

    unique_attr = []
    for attr in event.attributes:
        if not any(attr.value == u.value for u in unique_attr):
            unique_attr.append(attr)

    event.attributes = unique_attr

    return event


def build_attribute(indicator, event):

    tags = []

    if "confidence" in indicator and indicator["confidence"]:
        tags.append({"name": "Confidence:{}".format(indicator["confidence"])})
    else:
        tags.append({"name": "Confidence:Unknown"})
    if "ttps" in indicator and indicator["ttps"]:
        for ttp in indicator["ttps"]:
            tags.append({"name": "TTP:{}".format(ttp)})
    if "itypes" in indicator and indicator["itypes"]:
        for itype in indicator["itypes"]:
            tags.append({"name": "Detail:{}".format(itype)})
    if "attributes" in indicator and indicator["attributes"]:
        for attr in indicator["attributes"]:
            event.add_attribute(attr["type"], attr["value"], Tag=tags)

Header, Title, or Name added to the MISP event?

This is just terrific and can't believe @FloatingGhost got this to work(comments are worth reading)

At the moment I'm just concerned with getting some type of 'referencing' information with the event back into MISP.

For whatever reason it doesn't pick up the header or title and just uses STIX_File.xml and
event info = STIX Import

but there's definitely a

    <stix:STIX_Header>
        <stix:Package_Intent

and a

 <stixCommon:Identity>
                <stixCommon:Name

and I thought that was what this was doing --

    if isinstance(stix, str) and "\n" not in stix:
        # It's probably just a filename
        filename = stix
    elif hasattr(stix, "name"):
        # Steal this one!
        filename = stix.name
    elif hasattr(stixPackage, "stix_header"):
        # Well it has a header, maybe we can steal it
        if stixPackage.stix_header:
            if stixPackage.stix_header.title not in ["", None]:
                filename = stixPackage.stix_header.title + ".xml"

I swear it used to grab that information and add it to the event.

I don't know if there's any suggestions you can give AND sorry if I haven't given any good 'examples' in here, I'm not sure what I'm allowed to copy and paste.

Thank you for your work on this!

=========================================================
Also, in case anyone else looks for this, I did get it to start tagging by modifying convert.py

    if misp_event.attributes:
        log.debug("Attributes exist. Pushing...")
        if mispAPI:
            response = mispAPI.add_event(
                json.dumps(misp_event, cls=MISPEncode))
            event_id = response['Event']['id']
            event = mispAPI.get_event(event_id)
            uuid = response['Event']['uuid']
            mispAPI.tag(uuid,"TAG_Taxii")
            if response.get('errors'):
                raise Exception("PACKAGE: {}\nERROR: {}".format(
                    json.dumps(misp_event, cls=MISPEncode),
                    response.get('errors')))
            return response
        else:
            return True

MarkingSpecification removed, why?

MISP-STIX-Converter/misp_stix_converter/converters/convert.py

Line 123 in 8ebdb3d

# Remove any "marking" sections because the US-Cert is evil

# Remove any "marking" sections because the US-Cert is evil
log.debug("Removing Marking elements...")
for element in stixXml.findall(".//{http://data-marking.mitre.org/Marking-1}Marking"):
element.getparent().remove(element)

The script remove any marking specification. Why?

I'm working on a "misp_taxii_hook" for the opentaxii's misp implementation that is able to parse the marking structure of the stix package and create the tlp tags as needed.

In my local installation I just tried to edit the file, removing the quoted lines. And it seems to work fine.
The stix parsing works as before, bet I have marking specification to parse.

Can you consider to remove the quoted lines.

UnicodeDecodeError: 'ascii' codec can't decode

When running stix-to-misp.py with any standard STIX file (e.g. https://www.us-cert.gov/sites/default/files/publications/MAR-10219351.r1.v2.stix.xml)
I get errors like

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 15269: ordinal not in range(128)

I've tried a few edits on the .py file to force utf-8 encoding but without any success, This still occurs calling the script with python3 as well (which should encode str as utf-8 afaik)

If i'm to pull down a misp event using misp-to-stix.py the default file output is ascii encoded and this will work perfectly with stix-to-misp.py

I suppose I could convert the .xml to ascii before calling stix-to-misp.py but that's suboptimal, surprising no one else has run into this issue.

Could not load stix file (Probably huge/deep XML tree)

Hi,

I ran into an error while trying to pull data from a TAXII Server in STIX Format. This is the traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 156, in manage_stix
    stix_package = STIXPackage.from_xml(stix)
  File "/usr/local/lib/python3.4/dist-packages/stix/core/stix_package.py", line 249, in from_xml
    return entity_parser.parse_xml(xml_file, encoding=encoding)
  File "/usr/local/lib/python3.4/dist-packages/mixbox/parser.py", line 179, in parse_xml
    xml_etree = get_etree(xml_file, encoding=encoding)
  File "/usr/local/lib/python3.4/dist-packages/mixbox/xml.py", line 55, in get_etree
    return etree.parse(doc, parser=parser)
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81117)
  File "src/lxml/parser.pxi", line 1832, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118116)
  File "src/lxml/parser.pxi", line 1852, in lxml.etree._parseFilelikeDocument (src/lxml/lxml.etree.c:118399)
  File "src/lxml/parser.pxi", line 1747, in lxml.etree._parseDocFromFilelike (src/lxml/lxml.etree.c:117187)
  File "src/lxml/parser.pxi", line 1162, in lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:111914)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105109)
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106817)
  File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105671)
  File "/home/misp/MISP-Taxii-Server/fsisac.xml", line 611
lxml.etree.XMLSyntaxError: AttValue: ' expected, line 611, column 87

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 59, in post_stix
    package = pymisp.tools.stix.load_stix(content_block.content)
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 132, in load_stix
    return manage_stix(f)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 160, in manage_stix
    raise STIXLoadError("Could not load stix file. {0}".format(ex))
misp_stix_converter.errors.STIXLoadError: Could not load stix file. AttValue: ' expected, line 611, column 87 (fsisac.xml, line 611)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 76, in wrapper
    return _process_with_service(service)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 154, in _process_with_service
    response_message = service.process(request.headers, taxii_message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 89, in process
    in_response_to=message.message_id)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/exceptions.py", line 48, in raise_failure
    tb=tb)
  File "/usr/local/lib/python3.4/dist-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 59, in post_stix
    package = pymisp.tools.stix.load_stix(content_block.content)
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 132, in load_stix
    return manage_stix(f)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 160, in manage_stix
    raise STIXLoadError("Could not load stix file. {0}".format(ex))
opentaxii.taxii.exceptions.FailureStatus: Could not load stix file. AttValue: ' expected, line 611, column 87 (fsisac.xml, line 611), event=Status exception}

I checked the XML and it is correct (unfortunately I can't share it)
This is the configuration file

---
domain: "localhost:9000"
support_basic_auth: yes

save_raw_inbox_messages: yes
xml_parser_supports_huge_tree: no

persistence_api:
  class: opentaxii.persistence.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: mysql://taxii:Password1234@localhost/taxiipersist
    create_tables: yes

auth_api:
  class: opentaxii.auth.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: mysql://taxii:Password1234@localhost/taxiiauth
    create_tables: yes
    secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

logging:
  opentaxii: debug
  root: debug

hooks: misp_taxii_hooks.hooks
# Sample configuration for misp_taxii_server

zmq:
    host: 192.168.56.50
    port: 50000

misp:
    url: "https://192.168.56.50"
    api: XXXXXXXXXXXXXXXXXXXXXXXXX

I tried to change the xml_parser_supports_huge_tree option in yes, but nothing changed.
The XML block size that causes this error is 65535 bytes. As you can notice on the stacktrace I modified a bit the source code by writing each XML Block in a fixed file in order to see where is the error.

Can anyone help me?

Thanks in advance

MISPtoSTIX failed to load event

Hi,

I tried to convert all events of the galaxy misp-galaxy:tool=ETERNALBLUE with misp-to-stix.py. But when the code tries to load events (in convert.py MISPtoSTIX(mispJson)), some of them triggers an error.

This is my command line :

misp-to-stix.py -t "misp-galaxy:tool=\"ETERNALBLUE\"" -o ./results/out.{}.xml

And it raises this error :

[] is not of type 'object'
Failed validating 'type' in schema['properties']['Event']['properties']['Galaxy']['items']['properties']['GalaxyCluster']['items']['properties']['meta']:
    {'type': 'object'}
On instance['Event']['Galaxy'][0]['GalaxyCluster'][0]['meta']:
    []

So I looked for galaxy_cluster in schema.json from pymisp and I saw that meta should be a dict** :

"galaxy_cluster": {
"type": "object",
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"uuid": {
"type": "string"
},
"value": {
"type": "string"
},
"description": {
"type": "string"
},
"tag_id": {
"type": "string"
},
"meta": {
"type": "object"
}
}}

But when I searched the event manually (8188 on misppriv) and looked at "Galaxy" key, meta was an empty list**:

u'Galaxy': [{u'GalaxyCluster': [{u'description': u'SMBv2 exploit for Windows 7 SP1 (MS17-010)', u'galaxy_id': u'79', u'value': u'ETERNALBLUE', u'source': u'MISP Project', u'tag_name': u'misp-galaxy:tool="ETERNALBLUE"', u'meta': [], u'type': u'tool''}]}]

So it can't match the json schema and it fails while loading the event.

For now I did an ugly but working trick (still in convert.py MISPtoSTIX(mispJson)):

for galaxy in mispJSON['Event']['Galaxy'] : 
        for galaxy_cluster in galaxy['GalaxyCluster']:
            if galaxy_cluster['meta'] == [] :
                galaxy_cluster["meta"] = {}

So, I think that when there is no value for meta, meta should be {} rather than []. Or all items in galaxies clusters should have meta dict completed.

Thanks for your reading :)

** i cut some of the fields for visibility

STIXPackage has not attribute description

Hi,

After pulling new changes an error occurred

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 76, in wrapper
    return _process_with_service(service)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 154, in _process_with_service
    response_message = service.process(request.headers, taxii_message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 89, in process
    in_response_to=message.message_id)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/exceptions.py", line 48, in raise_failure
    tb=tb)
  File "/usr/local/lib/python3.4/dist-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 53, in post_stix
    package = pymisp.tools.stix.load_stix(content_block.content)
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 19, in load_stix
    threat_level_id=threat_level_id, analysis=analysis)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/buildMISPAttribute.py", line 233, in buildEvent
    if pkg.description:
opentaxii.taxii.exceptions.FailureStatus: 'STIXPackage' object has no attribute 'description', event=Status exception, timestamp=2017-05-22T14:36:21.119956Z}

I can't find any solution, I don't know where is that attribute. Can you help me please?

Thanks,
Danilo

AttributeError: 'module' object has no attribute 'ThreatActor'

Receive the following when converting MISP to STIX with a threat-actor type. Exporting the same event without a threat-actor type is successful. Same behaviour with campaign-name.

misp@misp:/home/MISP-STIX-Converter$ misp-to-stix.py -i 538
/usr/local/lib/python3.4/dist-packages/stix/utils/deprecated.py:48: UserWarning: The use of this field has been deprecated. Received 'str' object.
warnings.warn(msg)
Traceback (most recent call last):
File "/usr/local/bin/misp-to-stix.py", line 5, in
pkg_resources.run_script('misp-stix-converter==0.2.7', 'misp-to-stix.py')
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 528, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 1401, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/EGG-INFO/scripts/misp-to-stix.py", line 80, in
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/servers/misp.py", line 83, in pull
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/servers/misp.py", line 83, in
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/converters/convert.py", line 57, in MISPtoSTIX
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/converters/buildSTIXAttribute.py", line 143, in buildAttribute
AttributeError: 'module' object has no attribute 'ThreatActor'
misp@misp:/home/MISP-STIX-Converter$ pip3 list | grep -i stix
misp-stix-converter (0.2.7)
stix (1.2.0.2)