misp/misp-modules

Modules for expansion services, enrichment, import and export in MISP and other tools.

MISP modules

MISP modules are autonomous modules that can be used to extend MISP for new services such as expansion, import, export and workflow action.

MISP modules can be also installed and used without MISP as a standalone tool accessible via a convenient web interface.

The modules are written in Python 3 following a simple API interface. The objective is to ease the extensions of MISP functionalities without modifying core components. The API is available via a simple REST API which is independent from MISP installation or configuration and can be used with other tools.

For more information: Extending MISP with Python modules slides from MISP training.

Installation

Installation instructions can be found in the installation documentation.

How to add your own MISP modules?

Developing a MISP module yourself is fairly easy. Start with a template or existing module and continue from there.
More information can be found in the contribute section of the documentation.

Documentation

In order to provide documentation about some modules that require specific input / output / configuration, the documentation contains detailed information about the general purpose, requirements, features, input and ouput of each of these modules:

• *description - quick description of the general purpose of the module, as the one given by the moduleinfo
• requirements - special libraries needed to make the module work
• features - description of the way to use the module, with the required MISP features to make the module give the intended result
• references - link(s) giving additional information about the format concerned in the module
• input - description of the format of data used in input
• output - description of the format given as the result of the module execution

Licenses

For further Information see the license file.

List of MISP modules

Expansion Modules

• Abuse IPDB - AbuseIPDB MISP expansion module
• OSINT DigitalSide - On demand query API for OSINT.digitalside.it project.
• APIVoid - Module to query APIVoid with some domain attributes.
• AssemblyLine Query - A module tu query the AssemblyLine API with a submission ID to get the submission report and parse it.
• AssemblyLine Submit - A module to submit samples and URLs to AssemblyLine for advanced analysis, and return the link of the submission.
• Backscatter.io - Backscatter.io module to bring mass-scanning observations into MISP.
• BTC Scam Check - An expansion hover module to query a special dns blacklist to check if a bitcoin address has been abused.
• BTC Steroids - An expansion hover module to get a blockchain balance from a BTC address in MISP.
• Censys Enrich - An expansion module to enrich attributes in MISP by quering the censys.io API
• CIRCL Passive DNS - Module to access CIRCL Passive DNS.
• CIRCL Passive SSL - Modules to access CIRCL Passive SSL.
• ClaamAV - Submit file to ClamAV
• Cluster25 Expand - Module to query Cluster25 CTI.
• Country Code - Module to expand country codes.
• CPE Lookup - An expansion module to query the CVE search API with a cpe code to get its related vulnerabilities.
• CrowdSec CTI - Hover module to lookup an IP in CrowdSec's CTI
• CrowdStrike Falcon - Module to query CrowdStrike Falcon.
• Cuckoo Submit - Submit files and URLs to Cuckoo Sandbox
• CVE Lookup - An expansion hover module to expand information about CVE id.
• CVE Advanced Lookup - An expansion module to query the CIRCL CVE search API for more information about a vulnerability (CVE).
• Cytomic Orion Lookup - An expansion module to enrich attributes in MISP by quering the Cytomic Orion API
• DBL Spamhaus Lookup - Checks Spamhaus DBL for a domain name.
• DNS Resolver - jj
• DOCX Enrich - Module to extract freetext from a .docx document.
• DomainTools Lookup - DomainTools MISP expansion module.
• EQL Query Generator - EQL query generation for a MISP attribute.
• EUPI Lookup - A module to query the Phishing Initiative service (https://phishing-initiative.lu).
• URL Components Extractor - Extract URL components
• Farsight DNSDB Lookup - Module to access Farsight DNSDB Passive DNS.
• GeoIP ASN Lookup - Query a local copy of the Maxmind Geolite ASN database (MMDB format)
• GeoIP City Lookup - An expansion module to query a local copy of Maxmind's Geolite database with an IP address, in order to get information about the city where it is located.
• GeoIP Country Lookup - Query a local copy of Maxminds Geolite database, updated for MMDB format
• Google Safe Browsing Lookup - Google safe browsing expansion module
• Google Search - An expansion hover module to expand google search information about an URL
• Google Threat Intelligence Lookup - An expansion module to have the observable's threat score assessed by Google Threat Intelligence.
• GreyNoise Lookup - Module to query IP and CVE information from GreyNoise
• Hashdd Lookup - A hover module to check hashes against hashdd.com including NSLR dataset.
• CIRCL Hashlookup Lookup - An expansion module to query the CIRCL hashlookup services to find it if a hash is part of a known set such as NSRL.
• Have I Been Pwned Lookup - Module to access haveibeenpwned.com API.
• HTML to Markdown - Expansion module to fetch the html content from an url and convert it into markdown.
• HYAS Insight Lookup - HYAS Insight integration to MISP provides direct, high volume access to HYAS Insight data. It enables investigators and analysts to understand and defend against cyber adversaries and their infrastructure.
• Intel471 Lookup - Module to access Intel 471
• IP2Location.io Lookup - An expansion module to query IP2Location.io to gather more information on a given IP address.
• IPASN-History Lookup - Module to query an IP ASN history service (https://github.com/D4-project/IPASN-History).
• IPInfo.io Lookup - An expansion module to query ipinfo.io to gather more information on a given IP address.
• IPQualityScore Lookup - IPQualityScore MISP Expansion Module for IP reputation, Email Validation, Phone Number Validation, Malicious Domain and Malicious URL Scanner.
• IPRep Lookup - Module to query IPRep data for IP addresses.
• Ninja Template Rendering - Render the template with the data passed
• Joe Sandbox Import - Query Joe Sandbox API with a submission url to get the json report and extract its data that is parsed and converted into MISP attributes and objects.
• Joe Sandbox Submit - A module to submit files or URLs to Joe Sandbox for an advanced analysis, and return the link of the submission.
• Lastline Lookup - Deprecation notice: this module will be deprecated by December 2021, please use vmware_nsx module. Query Lastline with an analysis link and parse the report into MISP attributes and objects.
• Lastline Submit - Deprecation notice: this module will be deprecated by December 2021, please use vmware_nsx module. Module to submit a file or URL to Lastline.
• Macaddress.io Lookup - MISP hover module for macaddress.io
• Macvendors Lookup - Module to access Macvendors API.
• Malware Bazaar Lookup - Query Malware Bazaar to get additional information about the input hash.
• McAfee MVISION Insights Lookup - Lookup McAfee MVISION Insights Details
• GeoIP Enrichment - A hover and expansion module to enrich an ip with geolocation and ASN information from an mmdb server instance, such as CIRCL's ip.circl.lu.
• MWDB Submit - Module to push malware samples to a MWDB instance
• OCR Enrich - Module to process some optical character recognition on pictures.
• ODS Enrich - Module to extract freetext from a .ods document.
• ODT Enrich - Module to extract freetext from a .odt document.
• Onyphe Lookup - Module to process a query on Onyphe.
• Onyphe Full Lookup - Module to process a full query on Onyphe.
• AlienVault OTX Lookup - Module to get information from AlienVault OTX.
• Passive SSH Enrichment - An expansion module to enrich, SSH key fingerprints and IP addresses with information collected by passive-ssh
• PassiveTotal Lookup - The PassiveTotal MISP expansion module brings the datasets derived from Internet scanning directly into your MISP instance. This module supports passive DNS, historic SSL, WHOIS, and host attributes. In order to use the module, you must have a valid PassiveTotal account username and API key. Registration is free and can be done by visiting https://www.passivetotal.org/register
• PDF Enrich - Module to extract freetext from a PDF document.
• PPTX Enrich - Module to extract freetext from a .pptx document.
• Qintel QSentry Lookup - A hover and expansion module which queries Qintel QSentry for ip reputation data
• QR Code Decode - Module to decode QR codes.
• RandomcoinDB Lookup - Module to access the ransomcoinDB (see https://ransomcoindb.concinnity-risks.com)
• Real-time Blackhost Lists Lookup - Module to check an IPv4 address against known RBLs.
• Recorded Future Enrich - Module to enrich attributes with threat intelligence from Recorded Future.
• Reverse DNS - Simple Reverse DNS expansion service to resolve reverse DNS from MISP attributes.
• SecurityTrails Lookup - An expansion modules for SecurityTrails.
• Shodan Lookup - Module to query on Shodan.
• Sigma Rule Converter - An expansion hover module to display the result of sigma queries.
• Sigma Syntax Validator - An expansion hover module to perform a syntax check on sigma rules.
• SigMF Expansion - Expands a SigMF Recording object into a SigMF Expanded Recording object, extracts a SigMF archive into a SigMF Recording object.
• Socialscan Lookup - A hover module to get information on the availability of an email address or username on some online platforms.
• SophosLabs Intelix Lookup - An expansion module to query the Sophoslabs intelix API to get additional information about an ip address, url, domain or sha256 attribute.
• URL Archiver - Module to cache web pages of analysis reports, OSINT sources. The module returns a link of the cached page.
• Stairwell Lookup - Module to query the Stairwell API to get additional information about the input hash attribute
• STIX2 Pattern Syntax Validator - An expansion hover module to perform a syntax check on stix2 patterns.
• ThreatCrowd Lookup - Module to get information from ThreatCrowd.
• ThreadFox Lookup - Module to search for an IOC on ThreatFox by abuse.ch.
• ThreatMiner Lookup - Module to get information from ThreatMiner.
• TruSTAR Enrich - Module to get enrich indicators with TruSTAR.
• URLhaus Lookup - Query of the URLhaus API to get additional information about the input attribute.
• URLScan Lookup - An expansion module to query urlscan.io.
• VARIoT db Lookup - An expansion module to query the VARIoT db API for more information about a vulnerability.
• VirusTotal v3 Lookup - Enrich observables with the VirusTotal v3 API
• VirusTotal Public API Lookup - Enrich observables with the VirusTotal v3 public API
• VMRay Submit - Module to submit a sample to VMRay.
• VMware NSX Defender Enrich - Module to enrich a file or URL with VMware NSX Defender.
• VulnDB Lookup - Module to query VulnDB (RiskBasedSecurity.com).
• Vulnerability Lookup - An expansion module to query Vulnerability Lookup
• Vulners Lookup - An expansion hover module to expand information about CVE id using Vulners API.
• Vysion Enrich - Module to enrich the information by making use of the Vysion API.
• WhoisFreaks Lookup - An expansion module for https://whoisfreaks.com/ that will provide an enriched analysis of the provided domain, including WHOIS and DNS information.
• Wikidata Lookup - An expansion hover module to extract information from Wikidata to have additional information about particular term for analysis.
• IBM X-Force Exchange Lookup - An expansion module for IBM X-Force Exchange.
• XLXS Enrich - Module to extract freetext from a .xlsx document.
• YARA Rule Generator - jj
• YARA Syntax Validator - An expansion hover module to perform a syntax check on if yara rules are valid or not.
• Yeti Lookup - Module to process a query on Yeti.

Export Modules

• CEF Export - Module to export a MISP event in CEF format.
• Cisco fireSIGHT blockrule Export - Module to export malicious network activity attributes to Cisco fireSIGHT manager block rules.
• Microsoft Defender for Endpoint KQL Export - Defender for Endpoint KQL hunting query export module
• GoAML Export - This module is used to export MISP events containing transaction objects into GoAML format.
• Lite Export - Lite export of a MISP event.
• EQL Query Export - Export MISP event in Event Query Language
• Nexthink NXQL Export - Nexthink NXQL query export module
• OSQuery Export - OSQuery export of a MISP event.
• Event to PDF Export - Simple export of a MISP event to PDF.
• ThreatStream Export - Module to export a structured CSV file for uploading to threatStream.
• ThreadConnect Export - Module to export a structured CSV file for uploading to ThreatConnect.
• VirusTotal Collections Export - Creates a VT Collection from an event iocs.
• VirusTotal Graph Export - This module is used to create a VirusTotal Graph from a MISP event.
• YARA Rule Export - This module is used to export MISP events to YARA.

Import Modules

• PDNS COF Importer - Passive DNS Common Output Format (COF) MISP importer
• CSV Import - Module to import MISP attributes from a csv file.
• Cuckoo Sandbox Import - Module to import Cuckoo JSON.
• Email Import - Email import module for MISP
• GoAML Import - Module to import MISP objects about financial transactions from GoAML files.
• Import Blueprint - Generic blueprint to be copy-pasted to quickly boostrap creation of import module.
• Joe Sandbox Import - A module to import data from a Joe Sandbox analysis json report.
• Lastline Import - Deprecation notice: this module will be deprecated by December 2021, please use vmware_nsx module. Module to import and parse reports from Lastline analysis links.
• MISP JSON Import - Module to import MISP JSON format for merging MISP events.
• OCR Import - Optical Character Recognition (OCR) module for MISP.
• OpenIOC Import - Module to import OpenIOC packages.
• TAXII 2.1 Import - Import content from a TAXII 2.1 server
• ThreadAnalyzer Sandbox Import - Module to import ThreatAnalyzer archive.zip / analysis.json files.
• URL Import - Simple URL import tool with Faup
• VMRay API Import - Module to import VMRay (VTI) results.
• VMRay Summary JSON Import - Import a VMRay Summary JSON report.

Action Modules

• Mattermost - Simplistic module to send message to a Mattermost channel.
• Slack - Simplistic module to send messages to a Slack channel.
• Test action - This module is merely a test, always returning true. Triggers on event publishing.

misp-modules's Issues

Import Module: stiximport fails to import 'category="ipv4-addr"'

I'm working on my test MISP environment.
The misp-module stiximport seems to ignore IPV4 indicators (not yet tested with IPV6).

I'm testing the import using Grizzly Steppe STIX xml indicators, released by US-CERT: https://www.us-cert.gov/security-publications/GRIZZLY-STEPPE-Russian-Malicious-Cyber-Activity.
The import give no error, but in the results form available there's not IPv4 indicators listed.

Any solution? Thanks in advance for your support.

config field from admin interface don't be return

In a misp module when you add new configuration field :

moduleconfig = ["myField"]

A new ligne will be create in the server admin interface.

But when I add a value, my module can't read the value because in request the key does not exist.

request = json.loads(q)
print(request.get("config")) # always False

The debug mod for misp-module don't give this information too.

Fresh install fails to start

sudo -u www-data misp-modules -s

Traceback (most recent call last):
File "/usr/local/bin/misp-modules", line 9, in
load_entry_point('misp-modules==1.0', 'console_scripts', 'misp-modules')()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 542, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2568, in load_entry_point
raise ImportError("Entry point %r not found" % ((group, name),))
ImportError: Entry point ('console_scripts', 'misp-modules') not found

Any idea?

Easing deployment: implement as installable application

Will the project be open to implementing MISP modules as an installable Python package and publishing releases on PyPI? It would be fantastic to simply pip install misp-modules or similar, and for the script at bin/misp-modules.py to be migrated to a setuptools entry-point.

This would significantly ease deployment and aid use cases like installing into a virtualenv, etc.

Proposal to add .msg import/parsing capabilities

Proposal to add .msg import capabilities into import_email.py (or another appropriate module). At this point, Outlook .msg files (in their binary format) cannot be parsed for relevant attributes - however, .eml files can be parsed effectively.

A solution currently exists for parsing .msg files (https://github.com/mattgwwalker/msg-extractor) but it is not able to parse embedded .msg files. I propose creating a new module for dealing with this issue, and then converging with import_email.py when a reliable solution exists.

uwhois module not installed.

I'm getting an error

uwhois module not installed.

I've installed uwhoisd, but I cannot find a uwhois module ...

pip3 list | grep -i whois
cymruwhois (1.0)
uwhoisd (0.0.7)
whois (0.7)

MISP module that export one attribute by row

For your consideration

I have modified de export CEF module in that way:

My module by each attribute, read its type and writes the attribute value in a specific column. It's interesting for us to create ArcSight Active List.

My python isnt good so maybe something isnt the best way to do it
exp_col_export.txt

Connecting to internet services (passivetotal, virustotal, ...) through a proxy

Our MISP is placed in an internal environment, internet services are only reachable by going through a squid proxy (without any auth, as our MISP IP is whitelisted).
While syncing with other MISP instances in the internet works after configuring the proxy in the "Settings-->Proxy" tab, misp-modules like virustotal, countrycode etc. fail.
Is this not implemented yet, or did I just miss the required step while working through the readme?

www-data@misp:/usr/local/src/misp-modules$ /usr/local/bin/misp-modules
HTTPConnectionPool(host='www.geognos.com', port=80): Max retries exceeded with url: /api/en/countries/info/all.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f35addd2438>: Failed to establish a new connection: [Errno -2] Name or service not known',))
2016-10-05 14:47:25,682 - misp-modules - INFO - Launch MISP modules server from current directory.
2016-10-05 14:47:25,682 - misp-modules - INFO - Helpers loaded cache.py
2016-10-05 14:47:25,686 - misp-modules - INFO - MISP modules ocr imported
2016-10-05 14:47:25,686 - misp-modules - INFO - MISP modules testimport imported
2016-10-05 14:47:25,793 - misp-modules - INFO - MISP modules stiximport imported
2016-10-05 14:47:25,795 - misp-modules - WARNING - MISP modules countrycode failed due to HTTPConnectionPool(host='www.geognos.com', port=80): Max retries exceeded with url: /api/en/countries/info/all.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f35aaf8eb38>: Failed to establish a new connection: [Errno -2] Name or service not known',))

OpenIOC importer: hostname recongnized as ip-dst

The OpenIOC importer does not properly parse RouteEntryItem of type String. It tries to add them as ip-dst instead of hostname.

      <IndicatorItem id="xyz" condition="is">
        <Context document="RouteEntryItem" search="RouteEntryItem/Destination" type="mir" />
        <Content type="string">example.com</Content>
      </IndicatorItem>

virustotal KeyError: 'submission_names

I get following error when trying to test virustotal module.

2016-09-08 14:01:50,272 - misp-modules - ERROR - Something went wrong:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 195, in post
response = yield tornado.gen.with_timeout(timeout, self.run_request(jsonpayload))
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
value = future.result()
File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "", line 3, in raise_exc_info
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
result = self.fn(_self.args, *_self.kwargs)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 183, in run_request
response = mhandlers[x['module']].handler(q=jsonpayload)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/modules/expansion/virustotal.py", line 39, in handler
r["results"] += getDomain(q["domain"], key)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/modules/expansion/virustotal.py", line 85, in getDomain
toReturn += getIP(res["ip_address"], key, True)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/modules/expansion/virustotal.py", line 67, in getIP
toReturn += getMoreInfo(req, key)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/modules/expansion/virustotal.py", line 141, in getMoreInfo
"values":data["submission_names"],
KeyError: 'submission_names'
ERROR:misp-modules:Something went wrong:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 195, in post
response = yield tornado.gen.with_timeout(timeout, self.run_request(jsonpayload))
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
value = future.result()
File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "", line 3, in raise_exc_info
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
result = self.fn(_self.args, *_self.kwargs)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 183, in run_request
response = mhandlers[x['module']].handler(q=jsonpayload)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/modules/expansion/virustotal.py", line 39, in handler
r["results"] += getDomain(q["domain"], key)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/modules/expansion/virustotal.py", line 85, in getDomain
toReturn += getIP(res["ip_address"], key, True)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/modules/expansion/virustotal.py", line 67, in getIP
toReturn += getMoreInfo(req, key)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/modules/expansion/virustotal.py", line 141, in getMoreInfo
"values":data["submission_names"],
KeyError: 'submission_names'
{"error": "Something went wrong, look in the server logs for details"}[14:01:50]

To test module I run following command. curl -s http://127.0.0.1:6666/query -H "Content-Type: application/json" --data @body_virustotal.json -X POST

body_virustotal.json

{
"domain": "aijazeera.org",
"module": "virustotal",
"config":
{
"apikey": "myAPIkey",
"event_limit": "5"
}
}

misp-modules not returning any modules

running on debian 8 (64bit) ...

starts up fine,
2016-08-03 14:28:05,601 - misp-modules - INFO - MISP modules server started on localhost port 6666

however
$ curl -s http://127.0.0.1:6666/modules
returns an empty set
[]

no error messages, nothing I could divine in strace... how can I troubleshoot?

thx

Payload Security + MISP

Just found that I can download OpenIoC report from payload security. Tried to import this into MISP thinking it would populate all the IoCs but it only attaches the report.

We're using MISP version 2.4.55

Found this in settings:
Plugin.Import_services_enable false

Travis is failing

The tests are failing in all python versions since this merge: a510098

Email Import Module Error if certain email fields don't exist

It looks like there may be some missing logic if certain email fields don't exist when trying to import attributes from an email using the email import module. Below is the full error code, but I'm guessing it's due to "In-Reply-To" not existing in the email i was trying to import.

Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/misp_modules-1.0-py3.4.egg/misp_modules/init.py", line 197, in post
response = yield tornado.gen.with_timeout(timeout, self.run_request(jsonpayload))
File "/usr/lib64/python3.4/site-packages/tornado/gen.py", line 1015, in run
value = future.result()
File "/usr/lib64/python3.4/site-packages/tornado/concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "", line 3, in raise_exc_info
File "/usr/lib64/python3.4/concurrent/futures/thread.py", line 54, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3.4/site-packages/misp_modules-1.0-py3.4.egg/misp_modules/init.py", line 185, in run_request
response = mhandlers[x['module']].handler(q=jsonpayload)
File "/usr/lib/python3.4/site-packages/misp_modules-1.0-py3.4.egg/misp_modules/modules/import_mod/email_import.py", line 55, in handler
results.append({"values": message.get('In-Reply-To').strip(),
AttributeError: 'NoneType' object has no attribute 'strip'

UTF encoding causing errors

After a casual mention by @Rafiot that UTF encoding caused errors when importing emails in the e-mail import module I created some unit-tests so I could figure out how to fix it. Yet, the unit-tests will not fail under UTF 8, 16, or 32 in the body or the headers. So, I am adding this issue so @Rafiot can attach a failing e-mail and I can reproduce the aforementioned UTF issues.

misp-modules and /usr/bin/python

misp-modules runs as #!/usr/bin/python

Many systems will default to python 2.x

The module should run under /usr/bin/python3 when this exists

New module type - lookup modules

Would simply tie into a search system in MISP

• do grep queries against a directory of files (or a subset of them)
  • for example, do a search for IP addresses in all PCAP files visilbe to the user
  • find all attributes that are PCAP files visible to the user
  • pass to search module
  • search module parses the files / indeces from the attachment file store
  • returns valid IDs with the values contained

Proposal for misp-modules

A list of modules to be done:

• Country code expansion (hover and expansion)
• CIRCL Passive SSL (hover and expansion)
• CIRCL Passive DNS (hover and expansion)
• DMA and Cuckoo sandbox submission (hover and expansion)
• SWIFT Bank lookup (hover)
• Export module: Bro IDS
• asn.shadowserver.org / whois
• RIPEstat module / https://stat.ripe.net/docs/data_api
• OpenDNS module via https://investigate.api.opendns.com/
• Malware Hash Registry / whois / http://www.team-cymru.org/MHR.html
• Import/Export module: Facebook ThreatExchange
• Export module: bpf filter for tcpdump or Google stenographer
• Export module: markdown pdf export including natural language description of a MISP event
• Export module: markdown (table of indicators - to ease the production of reports)

Feel free to add your modules proposal or pick a module to do ;-)

Module config values not passed to import module

Values set via Plugin settings (moduleconfig) are not passed to import modules.
(test: see stiximport.py with max_size)

Email Import Module and Attachment Attribute Mapping Suggestion

This is just a suggestion. When using the Email Import Module, attachments are mapped as attributes with Category : External analysis and Type: attachment

This looks due to the values choice in email.import.py
185 for attch_item in attachment_files:
186 attch_item["types"] = ['attachment']
187 results.append(attch_item)

Perhaps a better choice for mapping would be to use ['email-attachment'] which is within Payload delivery?

By using attachment, MISP thinks the attribute object is actually a saved file and will offer the ability to download it (which of course it isn't actually saved or downloadable)

Import module to support the EXIF sanitization

Import module to support the EXIF sanitization

Extension errors seem to be ignored in the web UI

It appears that extension errors are ignored by the web interface. Logging of my extension showed that the issue was flagged and returned, but instead of seeing the error, it only showed a blank enrichment page.

Installation and management instructions for extensions

Is there a published set of documents I can reference to test my extension? It's not entirely clear where they should be installed and how I can manage them within my MISP instance.

Testing / Troubleshooting - misp-modules

I can see via curl that modules are installed and running. However, I do not see any traffic from MISP itself. How do I trigger MISP to connect to misp-modules?

I do have it configured under 'Plugin Settings' with the following values

• Plugin.Enrichment_services_enable true
• Plugin.Enrichment_hover_enable true
• Plugin.Enrichment_timeout 5
• Plugin.Enrichment_hover_timeout 2
• Plugin.Enrichment_services_url http://127.0.0.1
• Plugin.Enrichment_services_port 6666

Enhancement: Import Modules accessible via API

It seems the expansion modules can be used via API calls. Is there any way to do the same with import modules?

Set "ids" flag on some attributes during import / enrichment

Hi,

How do you set the IDS flag for some (not all) attributes during import or enrichment?

In the import / enrichment modules I'd like set IDS flag default to True for certain attribute types.

kr,

koen

Set timeout for one module

Hi,

Is there a way to set specific timeout for one module? There's a global "Plugin.Import_timeout" but for some modules a higher timeout is necessary.
Ideally you can supply a default value from the module (also see #66)

kr,

koen

Option to remove "paste" field

Hi,

An option in the import modules to remove the "paste data" textbox.
In some cases it doesn't make sense (for example import module that takes an external ID to fetch data but no data to paste).

kr,

koen

IBM X-Force services (to add)

crits/crits_services@6e49662

Set IDS flag for all attributes added via Email Import module

I'm not sure if this is module specific or a MISP bug, but when using Email Import Module to populate attributes, checking the IDS button at the top of the attribute screen will not auto-check it for all attributes. You must individually place a check in each IDS box next to each attribute. Not an overly big deal but the functionality seems to work for other areas of MISP and not the email import area.

Exception in passivetotal.py

I'm not sure what caused this, just noticed this in the terminal I used the start misp-modules:

2017-04-05 15:26:23,920 - misp-modules - ERROR - Something went wrong:
Traceback (most recent call last):
  File "/opt/rh/rh-python34/root/usr/lib/python3.4/site-packages/misp_modules/__init__.py", line 197, in post
    response = yield tornado.gen.with_timeout(timeout, self.run_request(jsonpayload))
  File "/opt/rh/rh-python34/root/usr/lib64/python3.4/site-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/opt/rh/rh-python34/root/usr/lib64/python3.4/site-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/opt/rh/rh-python34/root/usr/lib64/python3.4/concurrent/futures/thread.py", line 54, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/rh/rh-python34/root/usr/lib/python3.4/site-packages/misp_modules/__init__.py", line 185, in run_request
    response = mhandlers[x['module']].handler(q=jsonpayload)
  File "/opt/rh/rh-python34/root/usr/lib/python3.4/site-packages/misp_modules/modules/expansion/passivetotal.py", line 305, in handler
    for service in profile['playbook']['services']:
TypeError: 'NoneType' object is not subscriptable

Passing parameter to module

Hi,

A nice feature would be to pass an extra parameter to the external module, when requesting the enrichment. Ideally you get offered a selection of available parameters/values.

For example a default enrichment returns basic query information but with an extra parameter you can ask it to do something extra.

The only way that I see it now doable is by having two modules.
A parameter in the administrator setting is not useful because it's something you decide when you want to do the enrichment and not in general.

kr,

koen

OpenIOC importer: default comment is NoneNone

When using the OpenIOC importer the default comment for md5 / ip-dst attributes is 'NoneNone'. This should either be a single 'None' or empty (preferred).

Seg Fault while Installing misp-modules

pip3 install -I .

Processing /usr/local/src/misp-modules
Collecting tornado (from misp-modules===1.0)
Collecting dnspython3 (from misp-modules===1.0)
Collecting requests (from misp-modules===1.0)
Using cached requests-2.13.0-py2.py3-none-any.whl
Collecting urlarchiver (from misp-modules===1.0)
Collecting passivetotal (from misp-modules===1.0)
Collecting PyPDNS (from misp-modules===1.0)
Collecting pypssl (from misp-modules===1.0)
Collecting redis (from misp-modules===1.0)
Using cached redis-2.10.5-py2.py3-none-any.whl
Collecting pyeupi (from misp-modules===1.0)
Collecting ipasn-redis (from misp-modules===1.0)
Collecting asnhistory (from misp-modules===1.0)
Collecting stix (from misp-modules===1.0)
Using cached stix-1.2.0.3-py2.py3-none-any.whl
Collecting cybox (from misp-modules===1.0)
Collecting pillow (from misp-modules===1.0)
Using cached Pillow-4.0.0-cp35-cp35m-manylinux1_x86_64.whl
Collecting pytesseract (from misp-modules===1.0)
Collecting shodan (from misp-modules===1.0)
Collecting dnspython==1.15.0 (from dnspython3->misp-modules===1.0)
Using cached dnspython-1.15.0-py2.py3-none-any.whl
Collecting url-normalize (from urlarchiver->misp-modules===1.0)
Using cached url_normalize-1.3.1-py3-none-any.whl
Collecting future (from passivetotal->misp-modules===1.0)
Collecting python-dateutil (from passivetotal->misp-modules===1.0)
Using cached python_dateutil-2.6.0-py2.py3-none-any.whl
Collecting ez-setup (from passivetotal->misp-modules===1.0)
Collecting requests-cache (from PyPDNS->misp-modules===1.0)
Using cached requests_cache-0.4.13-py2.py3-none-any.whl
Collecting dateutils (from asnhistory->misp-modules===1.0)
Collecting mixbox>=1.0.1 (from stix->misp-modules===1.0)
Using cached mixbox-1.0.1-py2.py3-none-any.whl
Collecting lxml>=2.3 (from stix->misp-modules===1.0)
Using cached lxml-3.7.3-cp35-cp35m-manylinux1_x86_64.whl
Collecting olefile (from pillow->misp-modules===1.0)
Collecting colorama (from shodan->misp-modules===1.0)
Using cached colorama-0.3.7-py2.py3-none-any.whl
Collecting simplejson (from shodan->misp-modules===1.0)
Collecting click (from shodan->misp-modules===1.0)
Using cached click-6.7-py2.py3-none-any.whl
Collecting click-plugins (from shodan->misp-modules===1.0)
Collecting six>=1.5 (from python-dateutil->passivetotal->misp-modules===1.0)
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting argparse (from dateutils->asnhistory->misp-modules===1.0)
Using cached argparse-1.4.0-py2.py3-none-any.whl
Collecting pytz (from dateutils->asnhistory->misp-modules===1.0)
Using cached pytz-2016.10-py2.py3-none-any.whl
Collecting ordered-set (from mixbox>=1.0.1->stix->misp-modules===1.0)
Building wheels for collected packages: misp-modules
Running setup.py bdist_wheel for misp-modules ... done

Successfully built misp-modules

Installing collected packages: tornado, dnspython, dnspython3, requests, url-normalize, urlarchiver, future, six, python-dateutil, ez-setup, passivetotal, requests-cache, PyPDNS, pypssl, redis, pyeupi, ipasn-redis, argparse, pytz, dateutils, asnhistory, ordered-set, lxml, mixbox, cybox, stix, olefile, pillow, pytesseract, colorama, simplejson, click, click-plugins, shodan, misp-modules
Successfully installed PyPDNS-1.3 argparse-1.4.0 asnhistory-2.0.4 click-6.7 click-plugins-1.0.3 colorama-0.3.7 cybox-2.1.0.13 dateutils-0.6.6 dnspython-1.15.0 dnspython3-1.15.0 ez-setup-0.9 future-0.16.0 ipasn-redis-2.0 lxml-3.7.3 misp-modules-1.0 mixbox-1.0.1 olefile-0.44 ordered-set-2.0.1 passivetotal-1.0.30 pillow-4.0.0 pyeupi-1.0 pypssl-2.1 pytesseract-0.1.6 python-dateutil-2.6.0 pytz-2016.10 redis-2.10.5 requests-2.13.0 requests-cache-0.4.13 shodan-1.6.4 simplejson-3.10.0 six-1.10.0 stix-1.2.0.3 tornado-4.4.2 url-normalize-1.3.1 urlarchiver-0.2

You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Segmentation fault (core dumped)

Submit attachment data to enrichment modules

Hi,

When an event has an attachment and you try to use enrichment on that attribute then only the name of the attachment is submitted to the module, not the actual content.

What would be the best way to get the attachment data submitted to the module?

thanks,

koen

using export modules

Hi,
I have installed misp_modules and enabled the cef export plugin under Enrichments.
How do I export in this format now?

Under Event Actions/Export I don't see any new options there.
Am I looking in the wrong place?

Thanks,
Jeff

3rd party anonymous importer

Flask web server that receives a json blob looking like that:

{'type': "email", 'data': <base64 encoded blob>}

and send the b64 encoded blob to the right importer directly (without going through MISP first)

misp-modules rc.local startup

Alright, I have no idea what is going on with this now. We got the misp-modules to be executed from any user (from a previous issue), but now when I try and start it at boot from rc.local it doesn't stay up...

So when I execute it after I start up the misp application sudo -u www-data bash /var/www/MISP/app/Console/worker/start.sh then it won't show up in my log at all by the way this is my rc.local file (has extra stuff for debugging)

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
exec 2> /tmp/rc.local.log  # send stderr from rc.local to a log file
exec 1>&2                      # send stdout to the same log file
set -x
# nohup sudo -u www-data /usr/local/bin/misp-modules &
# echo $!
# sleep 5
sudo -u www-data bash /var/www/MISP/app/Console/worker/start.sh
sudo -u www-data /usr/local/bin/misp-modules &

#misp-modules &
exit 0

this is the way the /tmp/rc.local.log file looks

+ sudo -u www-data bash /var/www/MISP/app/Console/worker/start.sh
Stopping workers
   There is no workers to stop ...

Creating workers
Starting worker ... Done

Creating workers
Starting worker ... Done

Creating workers
Starting worker ... Done

Creating workers
Starting worker ... Done

Creating the scheduler workers
The scheduler worker is already running

but if I swap around the times that it is executed and my rc.local file looks like this

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
exec 2> /tmp/rc.local.log  # send stderr from rc.local to a log file
exec 1>&2                      # send stdout to the same log file
set -x
nohup sudo -u www-data /usr/local/bin/misp-modules &
echo $!
sleep 5
sudo -u www-data bash /var/www/MISP/app/Console/worker/start.sh
# sudo -u www-data /usr/local/bin/misp-modules &

#misp-modules &
exit 0

then my debug data from /tmp/rc.local.log look like this... (took out timestamps)

cat /tmp/rc.local.log
+ sleep 5
+ sudo -u www-data /usr/bin/python3 /usr/local/bin/misp-modules
misp-modules - INFO - Launch MISP modules server from current directory.
misp-modules - INFO - Helpers loaded cache.py
misp-modules - INFO - MISP modules openiocimport imported
misp-modules - INFO - MISP modules stiximport imported
misp-modules - INFO - MISP modules vmray_import imported
misp-modules - INFO - MISP modules testimport imported
misp-modules - INFO - MISP modules cuckooimport imported
misp-modules - INFO - MISP modules email_import imported
misp-modules - INFO - MISP modules ocr imported
misp-modules - INFO - MISP modules mispjson imported
misp-modules - INFO - MISP modules passivetotal imported
misp-modules - INFO - MISP modules asn_history imported
misp-modules - INFO - MISP modules cve imported
misp-modules - INFO - MISP modules reversedns imported
misp-modules - INFO - MISP modules countrycode imported
misp-modules - INFO - MISP modules wiki imported
misp-modules - INFO - MISP modules shodan imported
misp-modules - INFO - MISP modules circl_passivedns imported
misp-modules - INFO - MISP modules eupi imported
misp-modules - INFO - MISP modules whois imported
misp-modules - INFO - MISP modules xforceexchange imported
misp-modules - INFO - MISP modules threatminer imported
misp-modules - INFO - MISP modules sourcecache imported
misp-modules - INFO - MISP modules iprep imported
misp-modules - INFO - MISP modules circl_passivessl imported
misp-modules - INFO - MISP modules vmray_submit imported
misp-modules - INFO - MISP modules threatcrowd imported
misp-modules - INFO - MISP modules ipasn imported
misp-modules - INFO - MISP modules dns imported
misp-modules - INFO - MISP modules otx imported
misp-modules - INFO - MISP modules domaintools imported
misp-modules - INFO - MISP modules geoip_country imported
misp-modules - INFO - MISP modules virustotal imported
misp-modules - INFO - MISP modules liteexport imported
misp-modules - INFO - MISP modules cef_export imported
misp-modules - INFO - MISP modules testexport imported
misp-modules - INFO - MISP modules server started on 127.0.0.1 port 6666
+ sudo -u www-data bash /var/www/MISP/app/Console/worker/start.sh
Stopping workers
   There is no workers to stop ...

Creating workers
Starting worker ... Done

Creating workers
Starting worker ... Done

Creating workers
Starting worker ... Done

Creating workers
Starting worker ... Done

Creating the scheduler workers
The scheduler worker is already running

so I would think that it is executed, but when I check netstat (and on the misp application) it isn't running. The process is dead and gone...so any ideas what could be possibly happening?

Email Import Module Not Redirecting to Event Page After Attribute Submission

A majority of the time when parsing attributes from the Email Import module, after choosing to submit them to the MISP event, the page will not refresh to the Event page (it will stay stuck on the attribute submission page) so you have to click View Event on the left to get back to the Event. There is no error and the attributes are added to the event, but it will not automatically redirect to the Event with updated attributes after submission.

The issue occurs in multiple browsers (IE, FF, Chrome) so may be something with content of specific emails rather than a browser issue. Has anyone else experienced this?

HTTP error authentication incorrect?

I just upgraded my MISP docker with the latest git version of misp-modules.
It reports the following exception:

ERROR:misp-modules:Something went wrong:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 195, in post
response = yield tornado.gen.with_timeout(timeout, self.run_request(jsonpayload))
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
value = future.result()
File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "", line 3, in raise_exc_info
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
result = self.fn(_self.args, *_self.kwargs)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 183, in run_request
response = mhandlers[x['module']].handler(q=jsonpayload)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/modules/expansion/circl_passivedns.py", line 32, in handler
res = x.query(toquery)
File "/usr/local/lib/python3.4/dist-packages/pypdns/api.py", line 47, in query
raise Exception('HTTP error authentication incorrect?')
Exception: HTTP error authentication incorrect?

Any idea?
Some passivedns lookups are ok, others aren't?

/x

Email Import returns Invalid file upload. -- due to issues parsing URLs that are base64 or quopri encoded

In trying to upload an EML for parsing which had a base64 encoded body, MISP returned "Invalid file upload." The cause is that we had URL parsing enabled and it only looks for HTML parts to parse not base64 or quopri. Please see about adjusting the code to look for these conditions as well.

Example:

Current code: if (extract_urls is True and part.get_content_type() == 'text/html'):

Future code:

for part in message.walk():
	decoded_part = part.get_payload()
	if part.__getitem__("Content-Transfer-Encoding") == "quoted-printable":
		decoded_part = quote_printable_decode(part)	
	elif part.__getitem__("Content-Transfer-Encoding") == "base64":
		decoded_part = base64_decode(part.get_payload())	
	if part.get_content_subtype() == "plain":
		all_urls.extend(get_urls_from_plain(decoded_part))	
	elif part.get_content_subtype() == "html":
		all_urls.extend(get_urls_from_html(decoded_part))

stiximport does not work

It seems there's a problem with "stiximport" module. I tried both on a custom installation (Debian 8) and downloading the virtual image. Latest MISP version available.

The error is:

##################
2017-02-14 12:17:41,473 - misp-modules - ERROR - Something went wrong:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 75, in load_stix
stix_package = STIXPackage.from_json(stix)
File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 468, in from_json
d = json.load(json_doc)
File "/usr/lib/python3.4/json/init.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.4/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 197, in post
response = yield tornado.gen.with_timeout(timeout, self.run_request(jsonpayload))
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
value = future.result()
File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "", line 3, in raise_exc_info
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 185, in run_request
response = mhandlers[x['module']].handler(q=jsonpayload)
File "modules/import_mod/stiximport.py", line 35, in handler
pkg = stix.load_stix(package)
File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
stix = convert.load_stix(stix)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 106, in load_stix
return load_stix(f)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 76, in load_stix
except (TypeError, json.JSONDecodeError):
AttributeError: 'module' object has no attribute 'JSONDecodeError'
ERROR:misp-modules:Something went wrong:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 75, in load_stix
stix_package = STIXPackage.from_json(stix)
File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 468, in from_json
d = json.load(json_doc)
File "/usr/lib/python3.4/json/init.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.4/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 197, in post
response = yield tornado.gen.with_timeout(timeout, self.run_request(jsonpayload))
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
value = future.result()
File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "", line 3, in raise_exc_info
File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.4/dist-packages/misp_modules/init.py", line 185, in run_request
response = mhandlers[x['module']].handler(q=jsonpayload)
File "modules/import_mod/stiximport.py", line 35, in handler
pkg = stix.load_stix(package)
File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
stix = convert.load_stix(stix)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 106, in load_stix
return load_stix(f)
File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 76, in load_stix
except (TypeError, json.JSONDecodeError):
AttributeError: 'module' object has no attribute 'JSONDecodeError'
##################

I received the same error in both installations.

Any suggestion?

Set default value for moduleconfig value

Hi,

What is the best way to set a default value for a module config parameter?
You can list the parameters to be displayed in the "plugin settings" via moduleconfig. I'd like them to be filled with preset values (f.e. value1 = True, value2 = False, value3=123).
You can set the value in code but for user-friendlyness having it displayed immediately in the interface would be nice.

Currently only needed for 'expansion' but might be useful for import/export also.

kr,

koen

Email Import Module Errors out when it encounters unrecognized chars in subject

When emails contain characters that the native encoding cannot map to, the module will return a generic error “Something went wrong, look in the server logs for details”

This appears due particularly to email subjects like"🚀 john smith ? You are welcome" and there are two errors I found:

1. all_headers += "{0}: {1}\n".format(k.strip(), v.strip())
AttributeError: 'Header' object has no attribute 'strip'

A string cannot be returned because the unrecognized character cannot be encoded, so a ‘Header object’ gets returned, but as Header objects are not strings, the .strip() method will not work.

Possible solution is to apply the strip method in the results appending and also always encode the all_headers value so a string is returned regardless of unrecognized characters:
all_headers += "{0}: {1}\n".format(k, v)
results.append({"values": all_headers.strip().encode('utf-8'), "type": 'email-header'})

2. results.append({"values": message.get('Subject'), "type": 'email-subject'}) grabs the subject, but it would get returned as an ‘Header object’ and not the actual subject.

Example: 'type': 'email-subject', 'values': <email.header.Header object at 0x02BED890>

Possible solution is the same as item one, always encoding the subject.
results.append({"values": message.get('Subject').encode('utf-8'), "type": 'email-subject'})

Newly created import module is not showing up

My idea is to create a new import modules for misp.

I did a cp

    cp /usr/local/src/misp-modules/misp_modules/modules/import_modtestimport.py /usr/local/src/misp-modules/misp_modules/modules/import_mod/testimport2.py 

Changed

vi /usr/local/src/misp-modules/misp_modules/modules/import_mod/__init__.py

to:

    __all__ = ['testimport2', 'ocr', 'stiximport']

But:

    /usr/local/bin/misp-modules -p 6666 -l 127.0.0.1 -t

Gives me:

uwhois module not installed.
2016-12-01 14:22:29,125 - misp-modules - INFO - Launch MISP modules server from current directory.
2016-12-01 14:22:29,126 - misp-modules - INFO - Helpers loaded cache.py 
2016-12-01 14:22:29,127 - misp-modules - INFO - MISP modules eupi imported
2016-12-01 14:22:29,319 - misp-modules - INFO - MISP modules countrycode imported
2016-12-01 14:22:29,320 - misp-modules - INFO - MISP modules vmray_submit imported
2016-12-01 14:22:29,320 - misp-modules - INFO - MISP modules cve imported
2016-12-01 14:22:29,321 - misp-modules - INFO - MISP modules circl_passivessl imported
2016-12-01 14:22:29,321 - misp-modules - INFO - MISP modules wiki imported
2016-12-01 14:22:29,322 - misp-modules - INFO - MISP modules virustotal imported
2016-12-01 14:22:29,322 - misp-modules - INFO - MISP modules asn_history imported
2016-12-01 14:22:29,322 - misp-modules - INFO - MISP modules shodan imported
2016-12-01 14:22:29,323 - misp-modules - INFO - MISP modules ipasn imported
2016-12-01 14:22:29,323 - misp-modules - INFO - MISP modules dns imported
uwhois module not installed.
2016-12-01 14:22:29,324 - misp-modules - INFO - MISP modules whois imported
2016-12-01 14:22:29,325 - misp-modules - INFO - MISP modules sourcecache imported
2016-12-01 14:22:29,325 - misp-modules - INFO - MISP modules circl_passivedns imported
2016-12-01 14:22:29,326 - misp-modules - INFO - MISP modules passivetotal imported
2016-12-01 14:22:29,326 - misp-modules - INFO - MISP modules reversedns imported
2016-12-01 14:22:29,327 - misp-modules - INFO - MISP modules testimport imported
2016-12-01 14:22:29,328 - misp-modules - INFO - MISP modules stiximport imported
2016-12-01 14:22:29,328 - misp-modules - INFO - MISP modules ocr imported
2016-12-01 14:22:29,329 - misp-modules - INFO - MISP modules vmray_import imported
2016-12-01 14:22:29,330 - misp-modules - INFO - MISP modules testexport imported
2016-12-01 14:22:29,330 - misp-modules - INFO - MISP modules cef_export imported
2016-12-01 14:22:29,334 - misp-modules - INFO - MISP modules server started on 127.0.0.1 port 6666
2016-12-01 14:22:29,335 - misp-modules - INFO - MISP modules started in test-mode, quitting immediately.

So I am missing testimport2 showing up in loaded modules.
Any ideas?

Unable to load MISP modules from package.

After updating to the newest MISP-modules package today. I ran the following commands

cd misp-modules
sudo pip3 install -I -r REQUIREMENTS
sudo pip3 install -I .
sudo vi /etc/rc.local, add this line: sudo -u www-data misp-modules -s &

after this I get the following error

2017-06-06 10:36:17,125 - misp-modules - INFO - Unable to load MISP modules from package.

before I did this update the modules were working just fine.

Weird networking and python permissions

Hi,

I first wanted to share some information with you all in case you hadn't heard about it yet/if another user runs into the same problem.

INFORMATION

So I just wanted to let all know about this, and it makes sense but I ran into it on another persons system while working on it. If you are trying to run misp-modules on Ubuntu (or at least on NAME="Ubuntu" VERSION="16.04.2 LTS (Xenial Xerus)" kernel version 4.4.0-83-generic) and you disable ipv6 but the device is still capable of ipv6 (shown from the netstat below for ssh). Then python will still try to default to ipv6, and since there is no ipv6 address you aren't able to start any of the modules...

tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1511/mysqld
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      1327/redis-server 1
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1484/apache2
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1214/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1784/master
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1484/apache2
tcp6       0      0 :::22                   :::*                    LISTEN      1214/sshd

It shows here that weird things still happen in the kernel for ubuntu to allow this, but since there is no ipv6 address for any interface (including lo) then when trying to run misp-modules it will show the following

2017-07-18 12:05:15,782 - misp-modules - INFO - Launch MISP modules server from current directory.
2017-07-18 12:05:15,783 - misp-modules - INFO - Helpers loaded cache.py
2017-07-18 12:05:15,784 - misp-modules - INFO - MISP modules openiocimport imported
2017-07-18 12:05:15,784 - misp-modules - INFO - MISP modules stiximport imported
2017-07-18 12:05:15,785 - misp-modules - INFO - MISP modules vmray_import imported
2017-07-18 12:05:15,785 - misp-modules - INFO - MISP modules testimport imported
2017-07-18 12:05:15,785 - misp-modules - INFO - MISP modules cuckooimport imported
2017-07-18 12:05:15,786 - misp-modules - INFO - MISP modules email_import imported
2017-07-18 12:05:15,786 - misp-modules - INFO - MISP modules ocr imported
2017-07-18 12:05:15,786 - misp-modules - INFO - MISP modules mispjson imported
2017-07-18 12:05:15,787 - misp-modules - INFO - MISP modules passivetotal imported
2017-07-18 12:05:15,788 - misp-modules - INFO - MISP modules asn_history imported
2017-07-18 12:05:15,788 - misp-modules - INFO - MISP modules cve imported
2017-07-18 12:05:15,788 - misp-modules - INFO - MISP modules reversedns imported
2017-07-18 12:05:16,005 - misp-modules - INFO - MISP modules countrycode imported
2017-07-18 12:05:16,006 - misp-modules - INFO - MISP modules wiki imported
2017-07-18 12:05:16,006 - misp-modules - INFO - MISP modules shodan imported
2017-07-18 12:05:16,006 - misp-modules - INFO - MISP modules circl_passivedns imported
2017-07-18 12:05:16,006 - misp-modules - INFO - MISP modules eupi imported
2017-07-18 12:05:16,007 - misp-modules - INFO - MISP modules whois imported
2017-07-18 12:05:16,007 - misp-modules - INFO - MISP modules xforceexchange imported
2017-07-18 12:05:16,007 - misp-modules - INFO - MISP modules threatminer imported
2017-07-18 12:05:16,008 - misp-modules - INFO - MISP modules sourcecache imported
2017-07-18 12:05:16,008 - misp-modules - INFO - MISP modules iprep imported
2017-07-18 12:05:16,008 - misp-modules - INFO - MISP modules circl_passivessl imported
2017-07-18 12:05:16,009 - misp-modules - INFO - MISP modules vmray_submit imported
2017-07-18 12:05:16,009 - misp-modules - INFO - MISP modules threatcrowd imported
2017-07-18 12:05:16,009 - misp-modules - INFO - MISP modules ipasn imported
2017-07-18 12:05:16,010 - misp-modules - INFO - MISP modules dns imported
2017-07-18 12:05:16,010 - misp-modules - INFO - MISP modules otx imported
2017-07-18 12:05:16,010 - misp-modules - INFO - MISP modules domaintools imported
2017-07-18 12:05:16,011 - misp-modules - INFO - MISP modules geoip_country imported
2017-07-18 12:05:16,011 - misp-modules - INFO - MISP modules virustotal imported
2017-07-18 12:05:16,012 - misp-modules - INFO - MISP modules liteexport imported
2017-07-18 12:05:16,012 - misp-modules - INFO - MISP modules cef_export imported
2017-07-18 12:05:16,012 - misp-modules - INFO - MISP modules testexport imported
Traceback (most recent call last):
  File "/usr/local/bin/misp-modules", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/misp_modules/__init__.py", line 238, in main
    application.listen(port, address=listen)
  File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1943, in listen
    server.listen(port, address)
  File "/usr/local/lib/python3.5/dist-packages/tornado/tcpserver.py", line 142, in listen
    sockets = bind_sockets(port, address=address)
  File "/usr/local/lib/python3.5/dist-packages/tornado/netutil.py", line 197, in bind_sockets
    sock.bind(sockaddr)
OSError: [Errno 99] Cannot assign requested address

The reason why (at least I believe from understanding what is going on and looking at the code while changing some things and then it working) is because of course it doesn't have an ipv6 address and when trying to bind to localhost it tries to bind to the localhost ipv6 address (non existent). So the way I fixed it was by going into the /usr/local/lib/python3.5/dist-packages/misp_modules/__init__.py and changing localhost to 127.0.0.1. after doing that everything worked properly from that standpoint.

QUESTION

I was curious if you have ever ran into that when trying to run misp-modules command, that no user was able to run it except for root?
There is something wrong in being able to read the library or something for misp-modules and I don't know what is going on exactly...
So I know it is a permission issue because when I try and run the command you recommend when running ubuntu to start the misp-modules sudo -u www-data misp-modules & or sudo -u www-data misp-modules -s & I get the following error

Traceback (most recent call last):
  File "/usr/local/bin/misp-modules", line 7, in <module>
    from misp_modules import main
ImportError: cannot import name 'main'

so I did some investigating and when I run python3 for an interactive shell (as an unprivileged user) and then do the following

import misp_modules
help(misp_modules)

I get the following

Help on package misp_modules:
NAME
    misp_modules
PACKAGE CONTENTS
FILE
    (built-in)
(END)

but when I do the same as root or sudoing this is what I get

Help on package misp_modules:

NAME
    misp_modules

DESCRIPTION
    # -*- coding: utf-8 -*-
    #
    # Core MISP expansion modules loader and web service
    #
    # Copyright (C) 2016 Alexandre Dulaunoy
    # Copyright (C) 2016 CIRCL - Computer Incident Response Center Luxembourg
    #
    # This program is free software: you can redistribute it and/or modify
    # it under the terms of the GNU Affero General Public License as published by
    # the Free Software Foundation, either version 3 of the License, or
    # (at your option) any later version.
    #
    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    # GNU Affero General Public License for more details.
    #
    # You should have received a copy of the GNU Affero General Public License
    # along with this program.  If not, see <http://www.gnu.org/licenses/>.
...

which shows that it imports correctly, so do you all possibly know what it could be? I did the install properly and when I run sudo misp-modules everything works properly, but if I don't do a sudo or run as root then it won't be able to import everything ( as I showed you from the error above).
Any ideas? Because I really don't want to run this as root but as the www-data user like how you all say.

thank you for this awesome product I am loving it so far!

Module settings / module for proxy settings

Have a way in tornado to set per module the proxy settings instead o just having a global setting.

Return values of modules as three-tuple

Hi,

is it possible to implement the return values of modules as three-tuple?
E.g.: (VALUE, CATEGORY, ATTRIBUTE_TYPE)
The actual implementation needs the user to assign categories and types which takes, when importing lots of information, time. From my point of view, it is easier for users, when the assigning-step is optional and the module pre-assigns all the stuff. The user just checks if the assignments are correct, at the end.

Kind regards
Nils

BGP Ranking module

With that: https://github.com/CIRCL/bgpranking-redis-api/blob/master/example/api_web/client/bgpranking_web/api.py