misp / misp-stix-converter Goto Github PK
View Code? Open in Web Editor NEWA utility repo to assist with converting between MISP and STIX formats
License: GNU Lesser General Public License v3.0
A utility repo to assist with converting between MISP and STIX formats
License: GNU Lesser General Public License v3.0
Hello,
Since last update I get the following error when launching stix-to-misp.py and misp-to-stix.py:
ImportError: No module named 'misp_stix_converter.servers'
All dependecies seems to be installed:
#####################
stix-to-misp.py
Traceback (most recent call last):
File "/usr/local/bin/stix-to-misp.py", line 4, in
import('pkg_resources').run_script('Stix-MISP-Converter==0.2', 'stix-to-misp.py')
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 534, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 1445, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python3.4/dist-packages/Stix_MISP_Converter-0.2-py3.4.egg/EGG-INFO/scripts/stix-to-misp.py", line 15, in
ImportError: No module named 'misp_stix_converter.servers'
pip3 list | grep -i stix
stix (1.2.0.1)
Stix-MISP-Converter (0.2)
############################
Thanks,
Currently every observable is parsed out of a given STIX input and added as an attribute to a MISP event. In many cases this can give undesired results. A better solution would be as follows:
Here's some sample code of my implementation (It was a rush job so there is no logging/error checking in most of it) Simply import the stix_to_misp.py
file and call the build_event
method to create a MISP event. I use my own taxonomy for confidence and TTPs, but those can be changed as needed:
stix_parser.py
from cybox.objects import email_message_object, file_object, address_object, socket_address_object
from cybox.objects import domain_name_object, hostname_object, uri_object
def build_observable_map(pkg):
map = {}
if pkg.observables:
for o in pkg.observables.observables:
try:
map[o.id_] = o
except AttributeError:
continue
return map
def build_ttp_map(pkg):
map = {}
if pkg.ttps:
for t in pkg.ttps.ttps:
try:
map[t.id_] = t.title
except AttributeError:
continue
return map
def process_indicators(pkg):
observable_map = build_observable_map(pkg)
ttp_map = build_ttp_map(pkg)
indicators = []
if pkg.indicators:
for i in pkg.indicators:
indicator = dict()
indicator["itypes"] = get_indicator_types(i)
indicator["confidence"] = get_confidence(i)
indicator["ttps"] = get_ttps(i, ttp_map)
indicator["attributes"] = get_indicator_attributes(i, observable_map)
indicators.append(indicator)
return indicators
def process_observables(pkg):
observable_map = build_observable_map(pkg)
if pkg.observables:
return get_observable_attributes(pkg.observables.observables, observable_map)
else:
return []
def get_indicator_types(indicator):
itypes = []
for i in indicator.indicator_types:
itypes.append(i.value)
return itypes
def get_confidence(indicator):
if indicator.confidence:
return indicator.confidence.value.value
else:
return "Unknown"
def get_observable_attributes(observables, observable_map):
attributes = []
for o in observables:
if hasattr(o, "idref") and o.idref:
try:
observable = observable_map[o.idref]
except KeyError:
continue
extract_observable(observable, attributes)
else:
extract_observable(o, attributes)
return attributes
def get_indicator_attributes(indicator, observable_map):
observables = []
if indicator.observable:
observables += process_observable(indicator.observable)
if indicator.observables:
for o in indicator.observables:
observables += process_observable(o)
return get_observable_attributes(observables, observable_map)
def get_ttps(indicator, ttp_map):
ttps = []
if indicator.indicated_ttps:
for ttp in indicator.indicated_ttps:
if ttp.item.idref:
try:
t = ttp_map[ttp.item.idref]
except KeyError:
continue
ttps.append(t)
else:
ttps.append(ttp.item.title)
return ttps
def process_observable(observable):
observables = []
if observable.observable_composition:
if observable.observable_composition.operator == "OR":
for o in observable.observable_composition.observables:
observables += process_observable(o)
else:
observables.append(observable)
return observables
def add_attribute(obj, attributes):
if not any(a["value"] == obj["value"] for a in attributes):
attributes.append(obj)
# Dedicated to File object
def add_file_attr(obj, attributes):
if obj.file_name:
add_attribute({"type": "filename", "value": str(obj.file_name)}, attributes)
if obj.md5:
if len(obj.md5) == 32:
add_attribute({"type": "md5", "value": str(obj.md5)}, attributes)
if obj.sha1:
if len(obj.sha1) == 40:
add_attribute({"type": "sha1", "value": str(obj.sha1)}, attributes)
if obj.sha256:
if len(obj.sha256) == 64:
add_attribute({"type": "sha256", "value": str(obj.sha256)}, attributes)
if obj.sha512:
if len(obj.sha512) == 128:
add_attribute({"type": "sha512", "value": str(obj.sha512)}, attributes)
# Dedicated to Address Object (DB)
def add_addr_attr(obj, attributes):
if obj.is_source:
add_attribute({"type": "ip-src", "value": str(obj.address_value)}, attributes)
elif obj.is_destination:
add_attribute({"type": "ip-dst", "value": str(obj.address_value)}, attributes)
else:
# We don't know, first check if it's an IP range
if hasattr(obj, "condition") and obj.condition:
if obj.condition == "InclusiveBetween":
add_attribute({"type": "ip-dst", "value": str(obj.address_value[0])}, attributes)
add_attribute({"type": "ip-dst", "value": str(obj.address_value[1])}, attributes)
else:
add_attribute({"type": "ip-dst", "value": str(obj.address_value)}, attributes)
# Dedicated to EmailMessage (DB)
def add_email_attr(obj, attributes):
if obj.header:
# We have a header, can check for to/from etc etc
if obj.header.from_:
add_attribute({"type": "email-src", "value": str(obj.header.from_.address_value)}, attributes)
if obj.header.to:
for mail in obj.header.to:
add_attribute({"type": "email-dst", "value": str(mail.address_value)}, attributes)
if obj.header.subject:
add_attribute({"type": "email-subject", "value": str(obj.header.subject)}, attributes)
# Dedicated to Domain name (DB)
def add_domain_attr(obj, attributes):
add_attribute({"type": "domain", "value": str(obj.value)}, attributes)
# Dedicated to Hostname (DB)
def add_hostname_attr(obj, attributes):
add_attribute({"type": "hostname", "value": str(obj.hostname_value)}, attributes)
# Dedicated to URI (DB)
def add_uri_attr(obj, attributes):
add_attribute({"type": "url", "value": str(obj.value)}, attributes)
def extract_observable(observable, attributes):
if hasattr(observable, "object_") and observable.object_:
prop = observable.object_.properties
if type(prop) == address_object.Address:
# Now script uses buildAddressAttribute (DB)
add_addr_attr(prop, attributes)
elif type(prop) == domain_name_object.DomainName:
# Now script uses buildDomainNameAttribute (DB)
add_domain_attr(prop, attributes)
elif type(prop) == hostname_object.Hostname:
# Now script uses buildHostnameAttribute
add_hostname_attr(prop, attributes)
elif type(prop) == socket_address_object.SocketAddress:
if prop.ip_address:
add_addr_attr(prop.ip_address, attributes)
if prop.hostname:
add_hostname_attr(prop.hostname, attributes)
elif type(prop) == uri_object.URI:
# Now script uses buildURIAttribute (DB)
add_uri_attr(prop, attributes)
elif type(prop) == file_object.File:
# Now script uses buildFileAttribute (DB)
add_file_attr(prop, attributes)
elif type(prop) == email_message_object.EmailMessage:
# Now script uses buildEmailMessageAttribute (DB)
add_email_attr(prop, attributes)
else:
pass
else:
pass
stix_to_misp.py
from stix_parser import process_indicators, process_observables
from stix.core import STIXPackage
from tempfile import SpooledTemporaryFile
import pymisp
def load_stix(stix):
# Just save the pain and load it if the first character is a <
if isinstance(stix, STIXPackage):
# Oh cool we're ok
# Who tried to load this? Honestly.
return stix
elif hasattr(stix, 'read'):
try:
stix_package = STIXPackage.from_xml(stix)
except Exception as ex:
print("Could not load file")
return
return stix_package
elif isinstance(stix, str):
# It's text, we'll need to use a temporary file
f = SpooledTemporaryFile(max_size=10 * 1024)
f.write(stix.encode("utf-8"))
f.seek(0)
return load_stix(f)
def build_event(content, **kwargs):
pkg = load_stix(content)
event = pymisp.MISPEvent()
if pkg.stix_header and pkg.stix_header.title:
event.info = pkg.stix_header.title
else:
event.info = "NO_TITLE"
event.distribution = kwargs.get("distribution", 0)
event.threat_level_id = kwargs.get("threat_level_id", 3)
event.analysis = kwargs.get("analysis", 0)
if pkg.indicators:
indicators = process_indicators(pkg)
for i in indicators:
build_attribute(i, event)
elif pkg.observables:
indicator = dict()
indicator["attributes"] = process_observables(pkg)
build_attribute(indicator, event)
else:
return None
unique_attr = []
for attr in event.attributes:
if not any(attr.value == u.value for u in unique_attr):
unique_attr.append(attr)
event.attributes = unique_attr
return event
def build_attribute(indicator, event):
tags = []
if "confidence" in indicator and indicator["confidence"]:
tags.append({"name": "Confidence:{}".format(indicator["confidence"])})
else:
tags.append({"name": "Confidence:Unknown"})
if "ttps" in indicator and indicator["ttps"]:
for ttp in indicator["ttps"]:
tags.append({"name": "TTP:{}".format(ttp)})
if "itypes" in indicator and indicator["itypes"]:
for itype in indicator["itypes"]:
tags.append({"name": "Detail:{}".format(itype)})
if "attributes" in indicator and indicator["attributes"]:
for attr in indicator["attributes"]:
event.add_attribute(attr["type"], attr["value"], Tag=tags)
Hello. How to add a tag when creating an event? Where do I need to change the script?
When running stix-to-misp.py with any standard STIX file (e.g. https://www.us-cert.gov/sites/default/files/publications/MAR-10219351.r1.v2.stix.xml)
I get errors like
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 15269: ordinal not in range(128)
I've tried a few edits on the .py file to force utf-8 encoding but without any success, This still occurs calling the script with python3 as well (which should encode str as utf-8 afaik)
If i'm to pull down a misp event using misp-to-stix.py the default file output is ascii encoded and this will work perfectly with stix-to-misp.py
I suppose I could convert the .xml to ascii before calling stix-to-misp.py but that's suboptimal, surprising no one else has run into this issue.
Currently (unless I am mistaken), all attributes are added as observables to a single indicator during the conversion (https://github.com/MISP/MISP-STIX-Converter/blob/master/misp_stix_converter/converters/convert.py#L67), however this is most of the time not correct for the actual data. Each attribute should be treated as a separate indicator with 1-2 observables each (For example a filename|sha1 type would translate to an indicator with 2 observables).
This is just terrific and can't believe @FloatingGhost got this to work(comments are worth reading)
At the moment I'm just concerned with getting some type of 'referencing' information with the event back into MISP.
For whatever reason it doesn't pick up the header or title and just uses STIX_File.xml and
event info = STIX Import
but there's definitely a
<stix:STIX_Header>
<stix:Package_Intent
and a
<stixCommon:Identity>
<stixCommon:Name
and I thought that was what this was doing --
if isinstance(stix, str) and "\n" not in stix:
# It's probably just a filename
filename = stix
elif hasattr(stix, "name"):
# Steal this one!
filename = stix.name
elif hasattr(stixPackage, "stix_header"):
# Well it has a header, maybe we can steal it
if stixPackage.stix_header:
if stixPackage.stix_header.title not in ["", None]:
filename = stixPackage.stix_header.title + ".xml"
I swear it used to grab that information and add it to the event.
I don't know if there's any suggestions you can give AND sorry if I haven't given any good 'examples' in here, I'm not sure what I'm allowed to copy and paste.
Thank you for your work on this!
=========================================================
Also, in case anyone else looks for this, I did get it to start tagging by modifying convert.py
if misp_event.attributes:
log.debug("Attributes exist. Pushing...")
if mispAPI:
response = mispAPI.add_event(
json.dumps(misp_event, cls=MISPEncode))
event_id = response['Event']['id']
event = mispAPI.get_event(event_id)
uuid = response['Event']['uuid']
mispAPI.tag(uuid,"TAG_Taxii")
if response.get('errors'):
raise Exception("PACKAGE: {}\nERROR: {}".format(
json.dumps(misp_event, cls=MISPEncode),
response.get('errors')))
return response
else:
return True
Hi,
After pulling new changes an error occurred
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 76, in wrapper
return _process_with_service(service)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 154, in _process_with_service
response_message = service.process(request.headers, taxii_message)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 89, in process
in_response_to=message.message_id)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/exceptions.py", line 48, in raise_failure
tb=tb)
File "/usr/local/lib/python3.4/dist-packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
response_message = handler.handle_message(self, message)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
return InboxMessage11Handler.handle_message(service, request)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
inbox_message_id=inbox_message.id if inbox_message else None)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
collection_ids=collection_ids, service_id=service_id)
File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
for receiver in self.receivers_for(sender)]
File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
for receiver in self.receivers_for(sender)]
File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 53, in post_stix
package = pymisp.tools.stix.load_stix(content_block.content)
File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 19, in load_stix
threat_level_id=threat_level_id, analysis=analysis)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/buildMISPAttribute.py", line 233, in buildEvent
if pkg.description:
opentaxii.taxii.exceptions.FailureStatus: 'STIXPackage' object has no attribute 'description', event=Status exception, timestamp=2017-05-22T14:36:21.119956Z}
I can't find any solution, I don't know where is that attribute. Can you help me please?
Thanks,
Danilo
Receive the following when converting MISP to STIX with a threat-actor type. Exporting the same event without a threat-actor type is successful. Same behaviour with campaign-name.
misp@misp:/home/MISP-STIX-Converter$ misp-to-stix.py -i 538
/usr/local/lib/python3.4/dist-packages/stix/utils/deprecated.py:48: UserWarning: The use of this field has been deprecated. Received 'str' object.
warnings.warn(msg)
Traceback (most recent call last):
File "/usr/local/bin/misp-to-stix.py", line 5, in
pkg_resources.run_script('misp-stix-converter==0.2.7', 'misp-to-stix.py')
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 528, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 1401, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/EGG-INFO/scripts/misp-to-stix.py", line 80, in
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/servers/misp.py", line 83, in pull
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/servers/misp.py", line 83, in
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/converters/convert.py", line 57, in MISPtoSTIX
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/converters/buildSTIXAttribute.py", line 143, in buildAttribute
AttributeError: 'module' object has no attribute 'ThreatActor'
misp@misp:/home/MISP-STIX-Converter$ pip3 list | grep -i stix
misp-stix-converter (0.2.7)
stix (1.2.0.2)
I received the following error when trying to convert misp-to-stix:
My command:
misp-to-stix.py -t tlp:white -o out.{}.xml -c /home/misp/misp-t2/bin/MISP-STIX-Converter/misp.login
or
misp-to-stix.py -i 46874 --format xml -o 46874.xml -c /home/misp/misp-t2/bin/MISP-STIX-Converter/misp.login
Error message:
Traceback (most recent call last):
File "misp-to-stix.py", line 83, in <module>
package = MISP.pull(args.eid)[0]
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.7-py3.4.egg/misp_stix_converter/servers/misp.py", line 67, in pull
TypeError: search_index() got an unexpected keyword argument 'dateto'
MISP version: 2.4.67
Hello. when converting an MISPEvent to STIX when it finds ip-src
or ip-dst
it doesn't put the attribute category="ipv4-addr"
, so when the SIEM QRadar tries to obtain IP STIX Events it doesn't understand it so it skips these type of events. When i manually add the category
it can understand them and consumes them.
thank you.
Following past AIS STIX file discussion (we are still looking for sample files for testing), there is an XSD with their marking extension:
Not sure if this would help to complete partially this puzzle.
Hey,
When pushing stix package, the corresponding misp event created is not published.
I'd like to publish it right after creation. For that, I need the event id.
Could you return the event id / list of event ids instead of True
when using misp.push
?
Thx
I'm preparing a patch for this (and other STIX best practices) but one mayor issue is that currently MISPtoSTIX() returns a STIXPackage and lets the main program do the to_xml() call. Since a custom namespace is defined by using the ns_dict parameter of to_xml() this complicates things quite a lot.
One solution I see is to pass the required format (json/xml) to MISPtoSTIX() along with the custom namespace. Or is there a better way to deal with this?
Hello,
I am currently using MISP v2.4.148. I exported STIX files (both 1.1.1 and 2.0 versions) from Anomali TI and then successfully imported those STIX files inside the MISP instance using Import STIX feature. However, there is one issue: every indicator present in the STIX file has a "Confidence" value (indicator:Confidence
), but that value is ignored/dropped by MISP during the import process.
Is there any solution to this problem? I really need that confidence value associated with each imported indicator inside the MISP. I am posting one example below for you to look at:
<stix:Indicators>
<stix:Indicator id="threatstream:indicator-XXXXXX" timestamp="2021-09-12T18:13:40.701650+00:00" xsi:type='indicator:IndicatorType'>
<indicator:Title>spam_ip: 80.109.155.86</indicator:Title>
<indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">IP Watchlist</indicator:Type>
<indicator:Description>TS ID: 57531849646; iType: spam_ip; Date First: 2021-09-10T02:26:53.448Z; State: active; Org: Magenta Telekom; Source: NixSpam; MoreDetail: imported by user 668</indicator:Description>
<indicator:Observable id="threatstream:Observable-XXXXXXX">
<cybox:Title>spam_ip: 80.109.155.86</cybox:Title>
<cybox:Keywords>
<cybox:Keyword>low</cybox:Keyword>
</cybox:Keywords>
<cybox:Object id="threatstream:Address-XXXXXXX">
<cybox:Properties xsi:type="AddressObj:AddressObjectType" category="ipv4-addr">
<AddressObj:Address_Value>80.109.155.86</AddressObj:Address_Value>
</cybox:Properties>
</cybox:Object>
</indicator:Observable>
<indicator:Handling>
<marking:Marking>
<marking:Controlled_Structure>../../../descendant-or-self::node()</marking:Controlled_Structure>
<marking:Marking_Structure xsi:type='tlpMarking:TLPMarkingStructureType' color="AMBER"/>
</marking:Marking>
</indicator:Handling>
<indicator:Confidence timestamp="2021-09-12T18:13:40.702217+00:00">
<stixCommon:Value vocab_reference="https://en.wikipedia.org/wiki/Percentage" vocab_name="Percentage">37</stixCommon:Value>
</indicator:Confidence>
<indicator:Producer>
<stixCommon:Time>
<cyboxCommon:Produced_Time>2021-09-10T02:26:53.448000+00:00</cyboxCommon:Produced_Time>
</stixCommon:Time>
</indicator:Producer>
</stix:Indicator>
The conditional in the buildAttribute function of buildMISPAttribute.py checking to see if the current object being processed is a URI, Link URI, or Link object leads to the subsequent conditionals never being processed, as it will always return true.
The bug is on line 560:
elif type_ == uri_object.URI or type_ == link_object.URI or link_object.Link:
and should be changed to:
elif type_ == uri_object.URI or type_ == link_object.URI or type_ == link_object.Link:
# Remove any "marking" sections because the US-Cert is evil
log.debug("Removing Marking elements...")
for element in stixXml.findall(".//{http://data-marking.mitre.org/Marking-1}Marking"):
element.getparent().remove(element)
The script remove any marking specification. Why?
I'm working on a "misp_taxii_hook" for the opentaxii's misp implementation that is able to parse the marking structure of the stix package and create the tlp tags as needed.
In my local installation I just tried to edit the file, removing the quoted lines. And it seems to work fine.
The stix parsing works as before, bet I have marking specification to parse.
Can you consider to remove the quoted lines.
Hi,
I tried to convert all events of the galaxy misp-galaxy:tool=ETERNALBLUE with misp-to-stix.py. But when the code tries to load events (in convert.py MISPtoSTIX(mispJson)), some of them triggers an error.
This is my command line :
misp-to-stix.py -t "misp-galaxy:tool=\"ETERNALBLUE\"" -o ./results/out.{}.xml
And it raises this error :
[] is not of type 'object'
Failed validating 'type' in schema['properties']['Event']['properties']['Galaxy']['items']['properties']['GalaxyCluster']['items']['properties']['meta']:
{'type': 'object'}
On instance['Event']['Galaxy'][0]['GalaxyCluster'][0]['meta']:
[]
So I looked for galaxy_cluster in schema.json from pymisp and I saw that meta should be a dict** :
"galaxy_cluster": {
"type": "object",
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"uuid": {
"type": "string"
},
"value": {
"type": "string"
},
"description": {
"type": "string"
},
"tag_id": {
"type": "string"
},
"meta": {
"type": "object"
}
}}
But when I searched the event manually (8188 on misppriv) and looked at "Galaxy" key, meta was an empty list**:
u'Galaxy': [{u'GalaxyCluster': [{u'description': u'SMBv2 exploit for Windows 7 SP1 (MS17-010)', u'galaxy_id': u'79', u'value': u'ETERNALBLUE', u'source': u'MISP Project', u'tag_name': u'misp-galaxy:tool="ETERNALBLUE"', u'meta': [], u'type': u'tool''}]}]
So it can't match the json schema and it fails while loading the event.
For now I did an ugly but working trick (still in convert.py MISPtoSTIX(mispJson)):
for galaxy in mispJSON['Event']['Galaxy'] :
for galaxy_cluster in galaxy['GalaxyCluster']:
if galaxy_cluster['meta'] == [] :
galaxy_cluster["meta"] = {}
So, I think that when there is no value for meta, meta should be {} rather than []. Or all items in galaxies clusters should have meta dict completed.
Thanks for your reading :)
** i cut some of the fields for visibility
At the moment text in the STIX Indicator:Description field is lost. It would be nice to convert this to an attribute comment in MISP.
The same is true for the stix:Description field which could be converted to Catergory=other Type=comment in MISP.
Search_index() got an unexpected keyword argument 'tag' while using "sudo misp-to-stix.py -t tlp:white -o out.{}.xml"
Hi,
I ran into an error while trying to pull data from a TAXII Server in STIX Format. This is the traceback
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 156, in manage_stix
stix_package = STIXPackage.from_xml(stix)
File "/usr/local/lib/python3.4/dist-packages/stix/core/stix_package.py", line 249, in from_xml
return entity_parser.parse_xml(xml_file, encoding=encoding)
File "/usr/local/lib/python3.4/dist-packages/mixbox/parser.py", line 179, in parse_xml
xml_etree = get_etree(xml_file, encoding=encoding)
File "/usr/local/lib/python3.4/dist-packages/mixbox/xml.py", line 55, in get_etree
return etree.parse(doc, parser=parser)
File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81117)
File "src/lxml/parser.pxi", line 1832, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118116)
File "src/lxml/parser.pxi", line 1852, in lxml.etree._parseFilelikeDocument (src/lxml/lxml.etree.c:118399)
File "src/lxml/parser.pxi", line 1747, in lxml.etree._parseDocFromFilelike (src/lxml/lxml.etree.c:117187)
File "src/lxml/parser.pxi", line 1162, in lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:111914)
File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105109)
File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106817)
File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105671)
File "/home/misp/MISP-Taxii-Server/fsisac.xml", line 611
lxml.etree.XMLSyntaxError: AttValue: ' expected, line 611, column 87
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
response_message = handler.handle_message(self, message)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
return InboxMessage11Handler.handle_message(service, request)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
inbox_message_id=inbox_message.id if inbox_message else None)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
collection_ids=collection_ids, service_id=service_id)
File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
for receiver in self.receivers_for(sender)]
File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
for receiver in self.receivers_for(sender)]
File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 59, in post_stix
package = pymisp.tools.stix.load_stix(content_block.content)
File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
stix = convert.load_stix(stix)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 132, in load_stix
return manage_stix(f)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 160, in manage_stix
raise STIXLoadError("Could not load stix file. {0}".format(ex))
misp_stix_converter.errors.STIXLoadError: Could not load stix file. AttValue: ' expected, line 611, column 87 (fsisac.xml, line 611)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 76, in wrapper
return _process_with_service(service)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 154, in _process_with_service
response_message = service.process(request.headers, taxii_message)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 89, in process
in_response_to=message.message_id)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/exceptions.py", line 48, in raise_failure
tb=tb)
File "/usr/local/lib/python3.4/dist-packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
response_message = handler.handle_message(self, message)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
return InboxMessage11Handler.handle_message(service, request)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
inbox_message_id=inbox_message.id if inbox_message else None)
File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
collection_ids=collection_ids, service_id=service_id)
File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
for receiver in self.receivers_for(sender)]
File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
for receiver in self.receivers_for(sender)]
File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 59, in post_stix
package = pymisp.tools.stix.load_stix(content_block.content)
File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
stix = convert.load_stix(stix)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 132, in load_stix
return manage_stix(f)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 160, in manage_stix
raise STIXLoadError("Could not load stix file. {0}".format(ex))
opentaxii.taxii.exceptions.FailureStatus: Could not load stix file. AttValue: ' expected, line 611, column 87 (fsisac.xml, line 611), event=Status exception}
I checked the XML and it is correct (unfortunately I can't share it)
This is the configuration file
---
domain: "localhost:9000"
support_basic_auth: yes
save_raw_inbox_messages: yes
xml_parser_supports_huge_tree: no
persistence_api:
class: opentaxii.persistence.sqldb.SQLDatabaseAPI
parameters:
db_connection: mysql://taxii:Password1234@localhost/taxiipersist
create_tables: yes
auth_api:
class: opentaxii.auth.sqldb.SQLDatabaseAPI
parameters:
db_connection: mysql://taxii:Password1234@localhost/taxiiauth
create_tables: yes
secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
logging:
opentaxii: debug
root: debug
hooks: misp_taxii_hooks.hooks
# Sample configuration for misp_taxii_server
zmq:
host: 192.168.56.50
port: 50000
misp:
url: "https://192.168.56.50"
api: XXXXXXXXXXXXXXXXXXXXXXXXX
I tried to change the xml_parser_supports_huge_tree option in yes, but nothing changed.
The XML block size that causes this error is 65535 bytes. As you can notice on the stacktrace I modified a bit the source code by writing each XML Block in a fixed file in order to see where is the error.
Can anyone help me?
Thanks in advance
When I download event from MISP Download as -> STIX 1 JSON (metadata + all attributes) or STIX 2 It doesn't work on a fresh installation of MISP v2.4.150
Data:
MISP v2.4.150
Python 3.6.8
CentOS 7
Guide: https://misp.github.io/MISP/INSTALL.rhel7/
Traceback (most recent call last):
File "/var/www/MISP/app/files/scripts/stix2/misp2stix2.py", line 34, in <module>
from misp_stix_converter import MISPtoSTIX20Parser, MISPtoSTIX21Parser
File "/var/www/MISP/app/files/scripts/misp-stix/misp_stix_converter/__init__.py", line 3, in <module>
from .misp_stix_converter import misp_attribute_collection_to_stix1, misp_collection_to_stix2_0, misp_collection_to_stix2_1, misp_event_collection_to_stix1, misp_to_stix1, misp_to_stix2_0, misp_to_stix2_1
File "/var/www/MISP/app/files/scripts/misp-stix/misp_stix_converter/misp_stix_converter.py", line 22, in <module>
from typing import List, TypedDict, Union
ImportError: cannot import name 'TypedDict'
I solve the error as follows:
References: https://pypi.org/project/typing-extensions/
git diff misp_stix_converter.py
diff --git a/misp_stix_converter/misp_stix_converter.py b/misp_stix_converter/misp_stix_converter.py
index 1ec7fa2..636cbe8 100644
--- a/misp_stix_converter/misp_stix_converter.py
+++ b/misp_stix_converter/misp_stix_converter.py
@@ -19,7 +19,8 @@ from stix.core.ttps import TTPs
from stix2.base import STIXJSONEncoder
from stix2.v20 import Bundle as Bundle_v20
from stix2.v21 import Bundle as Bundle_v21
-from typing import List, TypedDict, Union
+from typing import List, Union
+from typing_extensions import TypedDict
from uuid import uuid4
_default_namespace = 'https://github.com/MISP/MISP'
Regards
Hi, if you include "import stix.extensions.marking.ais" in your imports before trying to load Stix file from XML it works fine without the need to remove US-Cert markings.
From the documentation of StixProject:
If you are writing code that needs to parse AIS markings, make sure that your
program imports this module before beginning to parse any STIX documents:
.. code-block:: python
import stix.extensions.marking.ais
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.