crits/crits_services

CRITs Services Collection

Welcome to CRITs

What Is CRITs?

CRITs is a web-based tool which combines an analytic engine with a cyber threat database that not only serves as a repository for attack data and malware, but also provides analysts with a powerful platform for conducting malware analyses, correlating malware, and for targeting data. These analyses and correlations can also be saved and exploited within CRITs. CRITs employs a simple but very useful hierarchy to structure cyber threat information. This structure gives analysts the power to 'pivot' on metadata to discover previously unknown related content.

Visit our website for more information, documentation, and links to community content such as our mailing lists and IRC channel.

Installation

CRITs is designed to work on a 64-bit architecture of Ubuntu or RHEL6 using Python 2.7. Installation has beta support for OSX using Homebrew. It is also possible to install CRITs on CentOS.

If you require the use of a 32-bit OS, you will need to download 32-bit versions of the pre-compiled dependencies.

The following instructions assume you are running Ubuntu or RHEL6 64-bit with Python 2.7. If you are on RHEL which does not come with Python 2.7, you will need to install it. If you do, ensure all python library dependencies are installed using Python 2.7. Also, make sure you install mod_wsgi against the Python 2.7 install if you are looking to use Apache. More information on this can be found in the Github wiki at https://github.com/crits/crits/wiki/Common-Questions.

Quick install using bootstrap

CRITs comes with a bootstrap script which will help you:

• Install all of the dependencies.
• Configure CRITs for database connectivity and your first admin user.
• Get MongoDB running with default settings.
• Use Django's runserver to quickly get you up and running with the CRITs interface.

Just run the following:

    sh script/bootstrap

Once you've run bootstrap once, do not use it again to get the runserver going, you'll be going through the install process again. Instead use the server script:

    sh script/server

Production CRITs install

If you are looking for a more permanent and performant CRITs installation or just interested in tweaking things, read more about setting up CRITs for production.

What's next?

We recommend adding services to your CRITs install. Services extend the features and functionality of the core project allowing you to enhance CRITs based on your needs. You can find more information about how to do this here.

Thanks for using CRITs!

crits_services's Issues

Passivetotal lookups always fail

I have my Passivetotal API key in the Passivetotal service settings, but attempting passivetotal lookups fails.

Lookup attempts generate the following entry in crits.log:

INFO 2014-07-29 22:07:20,240 crits.services.core Running passivetotal_lookup on IP 53d7d17dece7a56ba1c720b4, force=True, execute=process
ERROR 2014-07-29 22:07:20,794 crits.services.core Error running service passivetotal_lookup
Traceback (most recent call last):
  File "/data/crits/crits/services/core.py", line 651, in execute
    self._scan(self.current_task.context)
  File "/data/crits_services/passivetotal_service/__init__.py", line 63, in _scan
    loaded = json.loads(response.content) # handling a valid response
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

How do I fix this?

[Email] Investigate phishtank integration

It seems this site is becoming popular with verifying phishing attempts. I can't speak to the quality of the data, etc. but might be worth looking into.

[Relationships Service] Centralize Main Node

Since we’re using an automatic layout, it can be difficult to see what you were looking at when you invoke the Relationships Service (i.e., it’s not necessarily the central node, which is the one with the largest degree). It would be nice to highlight that element in some way (bolded?).

[TAXII Service] Consider parsing Status Message responses

A Poll Service, if behaving correctly, can return one of two messages: a Poll Response or a Status Message. Sometimes the Status Message can contain useful debug / error information (e.g., Data Collection Not Found). Instead of treating all non-Poll Response responses the same, would it make sense to, in the case of a Status Message response, parse out some of the information in the Status Message and use that to populate the ret['reason']?

The code I'm looking at is here: https://github.com/crits/crits_services/blob/master/taxii_service/handlers.py#L122

Packetmail.net service

It would be good to provide a service for packetmail.net reputation system (pull intel on a specific IP address). It would be good to contact Nathan Fowler for more information on this.

[PCAP] Reconstruct webpages

Ability to reconstruct web pages visited that are found in network traffic. It should return a Screenshot of the page instead of the actual HTML for safety.

[Yara] Support Yara 3.0

Not sure if there's anything we need to do but we should check and support the latest release!

[TAXII_Service] Problem configuring Taxii Service

Hello,

We're running latest source and having trouble configuring the Taxii Service. Trying to set it to point to the public YETI at http://taxiitest.mitre.org/.

We got over the hump of making it look "available" in the services control panel.

These are our config params:

{
hostname:   http://taxiitest.mitre.org/
https   : False
keyfile:    /data/certs/taxii.cert.key
certfile:   /data/certs/taxii.cert.cert
data_feed:  default
create_events:  False
certfiles:  MySource,default,/data/certs/taxii.cert.cert
}

But when browsing to "Taxii Agent" menu item it still blows up with python errors.

URLError at /services/taxii_service/taxii_agent/ <urlopen error [Errno -2] Name or service not known>

and then a long page with debug info.

In addition, when working on detail screen and then with then Bulding and Sending TAXII message, also unable to send. always getting Cannot contact TAXII Server at: http://taxiitest.mitre.org/

Is there a good basic set of config parameters that we can use on the TAXII Service config screen?

Thank you.

[Email] Reconstructor

Utilizing the contents of the email in CRITs, reconstruct the email. Instead of rendering the reconstructed email in the UI (potentially dangerous with HTML), a screenshot of what it looks like rendered should be generated and the user should be able to see that (could leverage the new Screenshots feature coming out soon?).

Popen zombie issues

I've noticed that when running CRITs (4.0-master) on Apache, the services using subprocess.Popen() (e.g. upx_service) leave behind zombie processes:
19052 ? Ss 0:00 /usr/sbin/httpd
19054 ? Sl 0:03 _ /usr/sbin/httpd
19055 ? S 0:08 _ /usr/sbin/httpd
19056 ? S 0:00 _ /usr/sbin/httpd
19057 ? S 0:05 _ /usr/sbin/httpd
19058 ? S 0:09 _ /usr/sbin/httpd
19201 ? Z 0:00 | _ [httpd]
19059 ? S 0:00 _ /usr/sbin/httpd
19060 ? S 0:07 _ /usr/sbin/httpd
19061 ? S 0:05 _ /usr/sbin/httpd
19062 ? S 0:06 _ /usr/sbin/httpd
19172 ? Z 0:00 | _ [httpd]
19087 ? S 0:06 _ /usr/sbin/httpd
19161 ? Z 0:00 | _ [httpd]

consuming Intel Feeds

It would be nice to be able to consume public / private intel feeds into CRITs. After this is gathered CRITs would be the point of reference for blocklists, context for IPs / domains / hashes.

Predictive Intelligence Service

A service which takes into account information in the system and makes predictions about potential new attacks.

For example, if you have Events with ties to Samples, Emails, and Indicators it might look at the frequency in which the start of the attack was an Email, what the attachments/URLs were, what types of Indicators were involved, and who was Targeted.

From that info it can "best guess" at the characteristics of the next attack based on patterns from previous ones.

Graphs would be nice to include showing the data it used for determining the results making it easier for analysts to follow.

IBM X-Force Integration

Another service could be IBM's X-Force integration(https://exchange.xforce.ibmcloud.com). IBM supports api to query for Domain, Malware, IP. these things can be integrated to the crits with crits service.

Triage and distribution problems

If a user uploads a sample and a service is configured to run on triage, it may refuse to run due to a lack of run-time configuration options. A good example of this is the yara service (which I am fixing). Here's the details:

The yara service is configured to know about sigfiles 'foo' and 'bar' and is enabled for triage.
A user uploads a sample expecting Yara to run on triage.
The service code calls validate_runtime_config() which raises an exception because there are no specified signature files.

The fix for this particular case is easy enough, just fall back to all signature files specified in the service config. I'm working on a patch for that right now.

However, if the service is configured to be run in a distributed manner then the problem can move from "which signature files" to "which API keys" and it is not clear which to fallback on.

Just off the top of my head, I see two things we can do (I'm sure there are more):

1. We can apply a preference value to each API key, and pick the one with the highest value as the default in case one is not specified.
2. Require that the API key you want to use for distributed purposes via automated methods have a specific name, which we can search on.

I'm not sure I like either of these problems, but I wanted to see what others think.

Issue : usr/lib/python2.7/urllib2.py in do_open, line 1184

hi, i executed all dependencie...
under services i can see taxi agent as well.

when i click on that i get below error.

pls help with appropriate solution as i am in need of implementing this functionality...

URLError at /services/taxii_service/taxii_agent/

<urlopen error [Errno -2] Name or service not known>

Request Method: GET
Request URL: https://192.168.139.138/services/taxii_service/taxii_agent/
Django Version: 1.6.5
Exception Type: URLError
Exception Value:

<urlopen error [Errno -2] Name or service not known>

Exception Location: /usr/lib/python2.7/urllib2.py in do_open, line 1184
Python Executable: /usr/bin/python
Python Version: 2.7.6
Python Path:

[u'/data/crits_services-master',
'/data/crits/crits',
'/data/crits',
'/usr/local/lib/python2.7/dist-packages/python_dateutil-2.2-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg',
'/usr/local/lib/python2.7/dist-packages/django_tastypie-0.11.0-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/python_mimeparse-0.1.4-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/django_tastypie_mongoengine-0.4.5-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/biplist-0.8-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/mongoengine-0.8.7-py2.7.egg'

data enrichment from services

Some of the services provide other pivoting points (data miner pulls out IPs and FQDNs, VirusTotal pulls IPs and domains associated with some data points). It would be nice to automatically add the IOCs and the relationships based on the service query

[TAXII service] Can't set up TAXII service for servers that don't require authentication

The TAXII service configuration requires certfiles and keyfiles for authenticating to a TAXII server. I'd like to connect to Hail a Taxii. Since it is open-source intelligence, it doesn't require authentication to collect intelligence. Without authentication, I can't use the taxii_service to connect to it.

peinfo broken on pe32+ (64 bit) binaries

Running the peinfo service on 64bit binaries causes an error. The error is:

Error running service: Bitstrings must have the same length for ^ operator.

I've tracked it down to the first XOR operation. I'm curious if bumping the first slice to be [0:8] is the right fix? I don't have time right now to track it down, but if someone does I'd be happy.

        #image characteristics
        img_chars = bitstring.BitArray(hex(exe.FILE_HEADER.Characteristics))
        #pad to 16 bits
        img_chars = bitstring.BitArray(bytes=img_chars.tobytes())
        img_chars_xor = img_chars[0:7] ^ img_chars[8:15]

[TAXII] Custom Control Panel for configuring multiple servers

The TAXII service should support:

• Configuring multiple TAXII servers with their own configuration options and authentication.
• Each server should also be able to be configured with a list of feeds to pull from.
  • These feeds should be able to be entered manually or through polling the TAXII server for what available feeds there are to subscribe to (whether publicly available or requiring authentication to see).
• Bonus if each TAXII server can be configured to use the appropriate version of TAXII so we can stop this "try 1.1 fail back to 1.0" hack that we do.
• The TAXII Agent which currently just executes when you click on the Nav menu item should instead bring you to a page where you can configure which servers and feeds you wish to poll for new content.

Since this configuration is quite epic in scope and complexity, it makes more sense for the TAXII service to come with its own section in the Control Panel.

• Instead of going to Services -> taxii_service to configure it, there would be a new core-driven section in the Control Panel which should be able to look at all registered services and include whatever appropriate template to add the service's own Control Panel section.
• This section should be completely customizable by the service so it can present its own views/handlers/templates to render configuration as it sees fit.
• Not all services should be required to do this, only those whose configuration is complex enough where the current configuration page is not sufficient.
• When distributed_services is accepted, there will be a new way for services to do configuration, which will allow them to link the Admin to the appropriate Control Panel section.

The TAXII service should define its own MongoEngine class for describing the document in the database for each server that gets configured. It can use that class to define its own new collection to house this information. This will also be useful so GridFS can store things like certificate files for authentication and public/private keys for encryption/decryption, etc.

[Yara Service] upgrade and use compiled rules file for performance

Yara 2.0 supports precompiled rules files to cut CPU cycles. The yara service should use precompiled rules with change detection to avoid recompiling the rules for every scan.

Two new services: Malware Tracker service integration into CRITS

We rely on the two services provided by Malware Tracker in our day to day processing. Currenly, reports from this are manually integrated into our CRITS malware records. I thought it might be a good idea to integrate these two services into crits_services, much like VirusTotal, perhaps with the ability to add indicators, as well.

PDFExaminer (https://www.malwaretracker.com/pdf.php)
Cryptam (https://www.malwaretracker.com/doc.php)

They have a published API: https://www.malwaretracker.com/tools.php

I believe it would provide value, as we use these services API for bulk submission, but then face the issue of linking up results sets to their records in CRITS.

Configurable web clients?

I wanted to bring this up, so that people could perhaps have a configurable way to get through their proxy servers.

There might be a few different use-cases such as:

• Some people don't use proxies
• Some people use transparent proxies
• Some people use NTLM for authenticating to their proxies
• Some people may need to ensure that the User-Agent string is configured in a certain way
• Some people may use a different proxy for internal access than for external.
• Some people may need to force https for everything
• Some people need to go through proxies for accessing external services such as VT

Now... every service uses a different framework to connect to where it needs to connect to and grab data. Great (I'm all for the diversity), but service xyz fails to connect to xyz servers.

meta_checker service crashes

On master branch of CRITs:

Traceback:
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  112.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py" in _wrapped_view
  22.                 return view_func(request, *args, **kwargs)
File "/vagrant/crits/services/views.py" in get_form
  189.         return service_run(request, name, crits_type, identifier)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py" in _wrapped_view
  22.                 return view_func(request, *args, **kwargs)
File "/vagrant/crits/services/views.py" in service_run
  258.                          custom_config=custom_config)
File "/vagrant/crits/services/handlers.py" in run_service
  178.         service_class.valid_for(obj)
File "/vagrant/crits_services/meta_checker/__init__.py" in valid_for
  18.         if len(obj.analysis) == 0:

Exception Type: AttributeError at /services/form/meta_checker/Sample/5474a773e490aa49fe678c5f/
Exception Value: 'Sample' object has no attribute 'analysis'

I assume related to analysis being moved to its own section

Cuckoo for URI - URL indicators

Hi all,

How about extending the cuckoo sandbox service for URI - URL indicators?

Thanks in advance,
Andreas

taxii agent service issue

after clickint taxii agent got below message.can u pls suggest what went wrong..adedd key and certificate, data feed: default, host: taxii.mitre.org

Results:
Status: Failure
Reason: FAILURE: Date: Wed, 29 Oct 2014 17:13:47 GMT Server: Apache Accept-Ranges: bytes Connection: close Transfer-Encoding: chunked Content-Type: text/html <style type="text/css"> #script { visibility:collapse; visibility:hidden; font-size:0px; height:0px; width:0px } #noscript { visibility:visible; font-size:inherit; height:inherit; width:inherit} </style> <script src="/includes/browserheight.js" language="JavaScript" type="text/javascript"></script> <title>TAXII - 404 Error </title>

Trusted Automated eXchange of Indicator Information Enabling Cyber Threat Information Exchange

TAXII Specifications — Version 1.1

Home > 404 Error 

---

About Documents FAQs Training TAXII Specifications Current Release Documentation Getting Started Developer Documentation Community Discussion List Discussion Archives GitHub Repositories Test Server Contact Us News & Events Calendar Free Newsletter Search the Site <script type="text/javascript">browserheight();</script>		Section Contents About TAXII Documents Frequently Asked Questions (FAQ) Training Other Items of Interest Release Archive Privacy Policy Terms of Use
	404 Error — Not Found The page you were looking for has not been found. You will be redirected to the home page in 10 seconds…

Page Last Updated: December 10, 2012 

TAXII is sponsored by the office of Cybersecurity and Communications at the U.S. Department of Homeland Security. Copyright © 2012 - 2014, The MITRE Corporation. TAXII and the TAXII logo are trademarks of The MITRE Corporation. Contact [email protected] for more information.

Privacy policy Terms of use Contact us

<script type="text/javascript"> var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-35443076-1']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); </script>
Successes: 0
Type Count Results
Failures 0

Events 0

Certificates 0

Domains 0

Emails 0

Indicators 0

IPs 0

PCAPs 0

Raw Data 0

Samples 0

[Relationships Service] Cross Browser Issues

In FF for example, the SVG viewport appears to be small in height and the nodes are centering outside the visible region, not sure yet what the right fix is for these issues, maybe someone else has an idea..

geolocation

It would be nice to have a geolocation service to add context for IPs or domains (might need to talk to the maxmind folks)

Visualizing Documents

I was at a conference a couple of weeks ago and saw a very simple idea that I think might be useful to people using CRITs. The basic premise was to take a sample and generate an image from it where each pixel represents one byte of the file. If the byte is printable then color it red. If the byte is not-printable color it white. When applied to things that should have relative small amounts of unprintable characters it's pretty easy to spot large chunks of unprintable characters together.

The service could work only on PDFs and office documents to start and could save the resulting image back to the database in much the same manner as the metacap viewer service does. The example I saw was using PIL to generate the image.

This could be done fairly trivially with D3 since we have that in core already :)

A couple examples that might help get the gears turning:

• http://bost.ocks.org/mike/miserables/ (visibility of small squares in a grid)
• http://bl.ocks.org/tjdecke/5558084 (a more preferred format where we can basically design it like a hex dump)
• https://github.com/mbostock/d3/wiki/Gallery (more examples)

I think if we used something like the 2nd example, but with smaller squares like the first example, this would actually be pretty awesome. I've always loved the idea of doing heatmaps for binary content!

[Yara service] Can't enable yara service without rules, can't create rules without service

I'm trying to install the Yara rule service so that I can create Yara rules to run against samples I collect.

I installed the Yara service to my services directory and created a yara folder to store rules that I create. When I go into the service status in the control panel, it reports that the service is misconfigured because no signature files are specified. Apparently I can't enable the yara service without signature files present, but I thought we used the Yara service and its Yara rule tester to create the rules to use.

It seems strange that I would have to create at least one rule offline before I can enable the Yara service that would handle subsequent signatures.

Create bootstrap scripts for all services who need one

Since people will install CRITs and want to run services, we should make bootstraps that make it easy for them to install the service dependencies.

CIF quering service

Hi there,

It would be good to have the capability to check against a CIF database when analysing a sample,domain, IP (just like VT, PassiveTotal, ThreatRecon etc..).

Any thoughts?

bootstrap scripts for services

I tried installing the services after bootstraping CRITs install, and I ran into missing python modules (off the top of myhead the missing were: python-yara, libtaxii, m4crypto, and perhaps one more that I can't remember at this time).

Perhaps, it would be a good idea to add a bootstrap script for the services. But in this case, maybe the script could go through the service folders and run a bootstrap for each one of them.

OPSWAT service is using proxy even if it's set not to

When the HTTP_PROXY/HTTPS_PROXY are set in the environment, the code needs to set the empty handler or the global vars will be used.

    if config.get('use_proxy'):
        self._debug("OPSWAT: proxy handler set to: %s" % settings.HTTP_PROXY)
        proxy_handler = urllib2.ProxyHandler({'http': settings.HTTP_PROXY})
    else:
        self._debug("OPSWAT: proxy handler unset")
        proxy_handler = urllib2.ProxyHandler({})
    opener = urllib2.build_opener(proxy_handler)
    urllib2.install_opener(opener)

no module named libtaxii

hi got this erro

ImportError at /

No module named libtaxii

Request Method: GET
Request URL: https://192.168.139.138/
Django Version: 1.6.5
Exception Type: ImportError
Exception Value:

No module named libtaxii

Exception Location: /data/crits_services-master/taxii_service/handlers.py in , line 10
Python Executable: /usr/bin/python
Python Version: 2.7.6
Python Path:

[u'/data/crits_services-master',
'/data/crits/crits',
'/data/crits',
'/usr/local/lib/python2.7/dist-packages/python_dateutil-2.2-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg',
'/usr/local/lib/python2.7/dist-packages/django_tastypie-0.11.0-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/python_mimeparse-0.1.4-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/django_tastypie_mongoengine-0.4.5-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/biplist-0.8-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/mongoengine-0.8.7-py2.7.egg',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-x86_64-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',

virustotal_service Dependencies (RHEL)

The VirusTotal service dependencies document states "The Virustotal service has no dependencies outside of those that are required for CRITs to run." This appears to be accurate for Ubuntu, but the CRITs "install_dependencies.sh" script on RHEL does not include simplejson, which is required by the VirusTotal service.

continuation to erro in #62

after installing i go this error again:

URLError at /services/taxii_service/taxii_agent/

Request Method: GET
Request URL: https://192.168.139.138/services/taxii_service/taxii_agent/
Django Version: 1.6.5
Exception Type: URLError
Exception Value:

Exception Location: /usr/lib/python2.7/urllib2.py in do_open, line 1184
Python Executable: /usr/bin/python
Python Version: 2.7.6
Python Path:

[u'/data/crits_services-master',
'/data/crits/crits',
'/data/crits',
'/usr/local/lib/python2.7/dist-packages/python_dateutil-2.2-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg',
'/usr/local/lib/python2.7/dist-packages/django_tastypie-0.11.0-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/python_mimeparse-0.1.4-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/django_tastypie_mongoengine-0.4.5-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/biplist-0.8-py2.7.egg',
'/usr/local/lib/python2.7/dist-packages/mongoengine-0.8.7-py2.7.egg',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-x86_64-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PILcompat',
'/usr/lib/python2.7/dist-packages/gtk-2.0',
'/usr/lib/pymodules/python2.7',
'/usr/lib/python2.7/dist-packages/ubuntu-sso-client']

chopshop_service __init__.py does not initialize jsonclass correctly

Within crits_services/chopshop_service/init.py, the run function contains the following code on line 137:

chopui.jsonout = jsonhandler

This appears to be a mistake since jsonhandler should be assigned to chopui.jsonclass. As a result, an error is thrown at line 147 when this call is made:

chopui.jsonclass.set_service(self)

Python throws an error stating the uninitialized jsonclass variable does not have a set_service method. To correct this, I removed line 137 where jsonout is initialized to the jsonhandler and replaced it with the following two lines:

chopui.jsonout = True
chopui.jsonclass = jsonhandler

This appeared to fix the issue within my environment.

[Email] Enable Yara Service for Email

Make the Yara service support emails.

r

[Email] Header Field Counts

A service which can be run on an email. It will look at every other email in the system and compare them by header fields. It will then create counts for how many of those emails matched (or was close to matching?) the current email broken down by header field.

The results would have a link to a global search for email on that field with the value. This will allow analysts to quickly determine which header fields for this email are potentially useful for pivoting on without having to go field-by-field manually.

For example, we might see something like:

From Address: [email protected]
Matches: 528 (this would be a link to /search/?q=type%3Aemail+field%3Afrom+foo%40bar.com&search_type=global or whatever the appropriate field should be)

[TAXII Service] Make URLs configurable

Currently, URLs (well, the path portion at least) are hard coded in the TAXII Service. I found two instances (for Inbox and Poll) here:

• https://github.com/crits/crits_services/blob/master/taxii_service/__init__.py#L192
• https://github.com/crits/crits_services/blob/master/taxii_service/handlers.py#L92

I think that since the path portion of the URL can be different on each TAXII Server, these should be configurable.

-Mark

Conversation Tracker Service

A service that would consolidate comments from related objects into one view for context. This would allow an analyst to see all related comments for an artifact. We could optionally show related objects, which are equally difficult to put into context.

For example, imagine an analyst viewing a malware sample. Any comments made on child samples would not be visible and the user would have to navigate to each related item to view comments. It's possible that an analyst may have made relevant comments to a related email, child malware sample, pcap, etc.

stix_doc_converter.py can't be run successfully

When attempting to invoke stix_doc_converter.py, using the syntax:

python manage.py runscript crits_scripts stix_doc_converter -e -- -i //stix.xml -o / -s

I get the following error:

Traceback (most recent call last):
File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/usr/CRITsVirtualEnv/lib/python2.7/site-packages/django/core/management/init.py", line 399, in execute_from_command_line
utility.execute()
File "/usr/CRITsVirtualEnv/lib/python2.7/site-packages/django/core/management/init.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/CRITsVirtualEnv/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(_args, *_options.dict)
File "/usr/CRITsVirtualEnv/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(_args, *_options)
File "/data/crits/crits/core/management/commands/runscript.py", line 89, in handle
script = script_class(username=username)
TypeError: init() got an unexpected keyword argument 'username'

I've tried it using other authentication options, but get the same behavior. I'm suspecting the script API may have changed since this was written... Also, it's the only one of the scripts that uses argparse instead of optparse. I don't really speak python, but is this a quick and easy fix? I've got some data I'd like to import that is in STIX 1.0, and uses the STIX_Packages root node, so nothing else I've been able to find will convert it.

Thanks
John

yarGen service

I recently built a service that runs yarGen against all samples related to a given TLO. It outputs both "simple" and "super" rules that can then be tested with the Yara Rule Tester service.

Unfortunately, the code needs a bit of cleanup, and some tweaks need to be made in order to customize the rules it generates (via the addition of a forms.py, I would assume). It's a little too messy at this point for me to want to submit a PR, and I've run out of time to work on it for the time being. If anyone wants to pick it up and run with it, feel free to fork it from https://github.com/TheDr1ver/crits_services/tree/master/yargen_service

yarGen - https://github.com/Neo23x0/yarGen

Timeline service needs to escape HTML

If comments contain malicious JS like <script>alert("hi");</script> the normal Comments section on the Details page will properly escape, but the Timeline service doesn't causing the JS to get executed.

MalShare

Another service could be the integration with http://malshare.com . The MD5 could be queried in MalShare database. The actual malware (in case one has one the MD5) could potentially be downloaded.

[TAXII Service] Is CB_STIX_XML_10 the right Binding ID?

This issue is based on the assumption that CRITs now supports STIX 1.1.1 (as indicated by the information on https://github.com/crits/crits/wiki/Structured-Data-Exchange-Format-Implementations, where python-stix minimum version is 1.1.1.0).

There are a few places in the taxii_service code where the constant for STIX 1.0 is used, where it seems that the constant for STIX 1.1.1 would be the correct constant to use:

• https://github.com/crits/crits_services/blob/master/taxii_service/handlers.py#L169

  elif content_block.content_binding == t.CB_STIX_XML_10: should probably be elif content_block.content_binding == t.CB_STIX_XML_111:

• https://github.com/crits/crits_services/blob/master/taxii_service/handlers.py#L274

  content_binding = t.CB_STIX_XML_10, should probably be content_binding = t.CB_STIX_XML_111,

Is this correct?

Thank you.
-Mark

EDIT: Grammar

[Email] Indicator Discovery

Scan the Indicators in the system for any that match any of the header fields or body of the email, and present them as results to the analyst. Will save them time pivoting on header fields looking for existing Indicators.

[TAXII Service] tm.validate_xml is incorrect

In handlers.py # 136 (https://github.com/crits/crits_services/blob/master/taxii_service/handlers.py#L136). tm.validate_xml ... is called.

The problem is that this attempts to validate what is probably a TAXII 1.1 message against the TAXII 1.0 schema, which will almost always fail, as TAXII 1.0 and TAXII 1.1 XML Messages use different namespaces.

I think the fix is to change tm to tm11 in that line of code.

-Mark

Decapsulation of AV quarantine file formats

AVs qarantine suspicious files, to do so they wrap the original files in some clever ways. Once quarantined, the files are safe for transport, and can be safely moved around the environment.
If you are paying attention to your AV, you not only would like to have the original files, but also associated metadata. Decapsulator like this might turn out to be pretty useful in cases where AV's generic/heuristic signatures caught something interesting. Otherwise, it's nice for quickly getting the details to whitelist the file used by some buiness-app that AV just ate for breakfast.