This repo contains the code for the engine and the API of MineMeld, an extensible Threat Intelligence processing framework.
For details check the MineMeld Wiki
Engine of MineMeld
License: Apache License 2.0
This repo contains the code for the engine and the API of MineMeld, an extensible Threat Intelligence processing framework.
For details check the MineMeld Wiki
Prototypes should be added in a library in the local directory.
$ find . -name '*.py' -exec grep -s recev x {} ;
./minemeld-core/minemeld/ft/base.py: LOG.error("update recevied from checkpointed source")
./minemeld-core/minemeld/ft/base.py: raise AssertionError("update recevied from checkpointed source")
./minemeld-core/minemeld/ft/base.py: LOG.error("withdraw recevied from checkpointed source")
./minemeld-core/minemeld/ft/base.py: raise AssertionError("withdraw recevied from checkpointed source")
At a first look the counters of dagpusher don't sum up. Double check.
The AAA API to change password and admin users should double check the current user password if the authentication comes from the session cookie.
A new miner should be added to support ingesting indicators from CIF (http://csirtgadgets.org).
Current implementation of TAXII miner uses MITRE python library and loads the full response in memory. This doesn't scale too well and should be changed into an event-based STIX parser.
2016-09-25T22:19:30 (6395)amqp._callback ERROR: Exception in handling update on topic taxii_test with params {u'source': u'taxii_test', u'indicator': u'::ffff:1.1.1.1', u'value': {u'confidence': 40, u'sources': [u'taxii_test'], [...], u'type': u'IPv4', [...]}}
Nodes should be identified by an id independent from the name. This would permit node renaming and avoid name collision.
There is a JSON file listing all the RedHat and Akami IPs that can be used as an additional check for RedHat updates. We should create a Miner to parse the JSON file.
Refs:
https://access.redhat.com/solutions/65300
https://access.redhat.com/articles/1525183
https://access.redhat.com/sites/default/files/attachments/cdn-ranges-2015-07-14.zip
unverified, but should be checked.
Miner for Autofocus export lists and tags
To use blueprints and better code separation.
An output & miner nodes should be added to push and retrieve indicators to MISP (http://www.misp-project.org/). Suggested by @sn8doc.
Currently Syslog processor doesn't export full details of matching session. The processor should be enhanced to export full details to an external logstash instance for history logging.
Like IPv4 aggregator but for IPv6 :-)
Starting with support for operator OR.
Add support of specifying a jmespath query inside a parameter to the feed to filter indicators.
Following exception is thrown when indicator contains non ASCII characters:
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 358, in _callback
m(**params)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 121, in _counter
f(self, *args, **kwargs)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 494, in update
value=fltvalue
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 121, in _counter
f(self, *args, **kwargs)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 920, in filtered_update
self._add_indicator(now, indicator, value)
File "/opt/minemeld/engine/0.9.26/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 842, in _add_indicator
indicator
UnicodeEncodeError: 'ascii' codec can't encode characters in position 77-80: ordinal not in range(128)
Version: 0.9.18
How to reproduce:
Exception is thrown:
2016-07-22T11:14:03 (5015)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/run/launcher.py", line 45, in _run_chassis
c.configure(fts)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/chassis.py", line 90, in configure
config=ftconfig.get('config', {})
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/__init__.py", line 25, in factory
config=config
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 371, in __init__
super(SyslogMiner, self).__init__(name, chassis, config)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 194, in __init__
self.configure()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 410, in configure
self._load_side_config()
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 491, in _load_side_config
cf = self._compile_rule(fname, f)
File "/opt/minemeld/engine/0.9.18/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 466, in _compile_rule
result['fields'] = [fld for fld in fields if type(fld) == str]
TypeError: 'NoneType' object is not iterable
Create a miner for the URLs http://www.malwaredomainlist.com/mdlcsv.php instead of using only the IP addresses.
Background
Users would like to share indicators with platforms supporting STIX/TAXII. This is currently not possible.
Requirements
Currently DagPusher supports only non-persistent registered-ips. We should support also persistent registered-ips:
TAXII Discovery and collection management services use the X-Server header to build the URL, this should be fixed.
Hello,
When node of type "Output" is enabled, in "Config" Tab its position is shown as "Processor". However, when the node is disabled, its position correctly matches the type - "Output"
Currently the User-Agent header generated by the base poller Miners doesn't contain any reference to MineMeld. Adding a MineMeld specific string with version would help the feed sources to have a sense of the number of accesses performed via MineMeld.
Right now URLs are handled using a naive aggregator, where aggregation and whitelisting are based on string matching.
A URL specific aggregator should:
Error messages in TAXII endpoints are too generic, they should be more specific to facilitate troubleshooting.
Exception handling domain indicators in taxii.DataFeed. Trace:
2016-09-12T17:09:19 (3329)amqp._callback ERROR: Exception in handling update on topic domainAggregator with params {u'source': u'domainAggregator', u'indicator': u'bestinghana.com', u'value': {u'confidence': 70, u'share_level': u'red', u'sources': [u'autofocusMMDemoEL'], u'autofocus_label': u'mmdemo', u'first_seen': 1473370831665L, u'type': u'domain', u'last_seen': 1473370831665L}}
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/comm/amqp.py", line 344, in _callback
m(**params)
File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 122, in _counter
f(self, *args, **kwargs)
File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 453, in update
value=fltvalue
File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/base.py", line 122, in _counter
f(self, *args, **kwargs)
File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 838, in filtered_update
self._add_indicator(now, id_, indicator, value)
File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 777, in _add_indicator
type_mapper['mapper'](oid, indicator, value)
File "/opt/minemeld/engine/0.9.20/local/lib/python2.7/site-packages/minemeld/ft/taxii.py", line 608, in _stix_domain_observable
type_="FQDN"
TypeError: __init__() got an unexpected keyword argument 'type_'
Currently there is no way to modify field contents inside CSV Miners, like it is possible with the plain text Miner. That would be useful.
Could be mined using DNS TXT queries. Ref: https://cloud.google.com/compute/docs/faq#networking
It should be possible to upload CSV files via the API and translate them into the format processed by the local Miners.
Currently syslogMatcher supports IPs and domains indicators, this to track support for matching URLs.
This to avoid issues when some product are capitalised. It happens from time to time.
Currently the historic period polled during the first request of the TAXII Miner is fixed to 1 hour. This should be made configurable.
Miner for AWS IP ranges, available here:
https://ip-ranges.amazonaws.com/ip-ranges.json
Hi everyone,
Currently we have an syslog to local logstash prototype available as an output. We however utilize ArcSight and would love to be able to receive either the 'logstash' syslog or a CEF formatted one. CEF syntax can be found here.
Creating such an prototype would support the ArcSight community out there!
Regards,
Forseti
This to improve compatibility with TAXII clients not supporting IP ranges.
Currently JSON miner retrieves the whole JSON feed before parsing it. This is too memory intensive. We should try switching to an streaming JSON parser like this one: http://lloyd.github.io/yajl/
Could be mined using DNS TXT queries. Ref: https://support.google.com/a/answer/60764?hl=en
For persistent feeds.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.