mitre/heimdall-mongo

A Mongo-based version of Heimdall (Deprecated)

Heimdall

Heimdall is a centralized aggregation tool for InSpec evaluations

Description

Heimdall supports viewing of InSpec profiles and evaluations in a convenient interface. Data uploads can be automated through usage of curl, and added as a step after an InSpec pipeline stage.

Versioning and State of Development

This project uses the Semantic Versioning Policy.

Branches

The master branch contains the latest version of the software leading up to a new release.

Other branches contain feature-specific updates.

Tags

Tags indicate official releases of the project.

Please note 0.x releases are works in progress (WIP) and may change at any time.

Heimdall vs Heimdall-Lite

There two versions of the MITRE Heimdall Viewer - the full Heimdall and the Heimdall-Lite version. We produced each to meet different needs and use-cases.

Features

	Heimdall-Lite	Heimdall
Installation Requirements	any web server	rails 5.x Server MongoDB instance
Overview Dashboard & Counts	x	x
800-53 Partition and TreeMap View	x	x
Data Table / Control Summary	x	x
InSpec Code / Control Viewer	x	x
SSP Content Generator	x	x
PDF Report and Print View	x	x
Users & Roles & multi-team support		x
Authentication & Authorization	Hosting Webserver	Hosting Webserver LDAP GitHub OAUTH & SAML GitLab OAUTH & SAML
Advanced Data / Filters for Reports and Viewing		x
Multiple Report Output (DISA Checklist XML, CAT, XCCDF-Results, and more)		x
Authenticated REST API		x
InSpec Run 'Delta' View		x
Multi-Report Tagging, Filtering and Compairison		x

Use Cases

Heimdall-Lite	Heimdall
Ship the App & Data via simple Email	Multiple Teams Support
Minimal Footprint & Deployment Time	Timeline and Report History
Local or disconnected Use	Centralized Deployment Model
One-Time Quick Reviews	Need to view the delta between one or more runs
Decentralized Deployment	Need to view subsets of the 800-53 control alignment
Minimal A&A Time	Need to produce more complex reports in multiple formats

Installation

curl -o /etc/yum.repos.d/heimdall.repo https://dl.packager.io/srv/mitre/heimdall/master/installer/el/7.repo

yum install -y heimdall

systemctl start mongod systemctl enable mongod

heimdall scale web=1

Dependencies

You can setup a deployment/development environment through bundler or docker.

If you wish to use docker, then the dependencies are:

• Docker
• docker-compose (installable with pip)

If you wish to use ruby and are on Ubuntu 16, then the dependencies are:

• Ruby 2.4.4
• build-essentials (your distribution's gcc package)
• Bundler
• libpq-dev
• nodejs
• mongodb

Run directly with Ruby (Instead of Docker)

This mode is primarily for developers, shared heimdall instances should be deployed in production mode. Since this is a Ruby application it is suggested to use Rbenv or RVM for ruby version management.

1. Install rbenv or RVM
2. Install dependencies
   • apt-get install build-essential libpq-dev nodejs libxml2-dev libmagick++-dev mongodb-server -y
3. Install ruby by running rbenv install or rvm install $(cat .ruby-version) from the root directory of this project
4. Run the following in a terminal
   • bundle install
   • bundle exec rake db:create
   • bundle exec rake db:migrate
   • bundle exec rails s (Start the server on localhost)

Run With Docker

Login Configuration

If you would like to use your organization's internal User authentication service, when deploying the dockerized Heimdall instance, you'll need to edit config/ldap.yml to point to your organization's LDAP server. You do not have to use your internal LDAP. However, people will have to create an account in Heimdall to perform most actions You may view ldap.example.yml for how authentication of people's internal email addresses works with a LDAP server which allows anonymous access.

Setup Docker Container

These steps need to be performed once per machine in order to prepare your machine to run heimdall in Docker.

1. Install Docker
2. Download heimdall by running git clone https://github.com/mitre/heimdall.git.
3. Navigate to the base folder where docker-compose.yml is located
4. Run the following commands in a terminal window from the heimdall source directory:
   • ./setup-docker-secrets.sh
   • docker-compose up -d

Managing Docker Container

The following commands are useful for managing the data in your docker container: * docker-compose run web rake db:reset This destroys and rebuilds the db * docker-compose run web rake db:migrate This updates the db

Running Docker Container

Make sure you have run the setup steps at least once before following these steps!

1. Run the following command in a terminal window:
   • docker-compose up -d
2. Go to 127.0.0.1:3000/heimdall in a web browser

Updating Docker Container

A new version of the docker container can be retrieved by running

docker-compose pull
docker-compose up -d
docker-compose run web bundle exec rake db:migrate

This will fetch the latest version of the container, redeploy if a newer version exists, and then apply any database migrations if applicable. No data should be lost by this operation.

Stopping the Container

docker-compose down # From the source directory you started from

Usage

You can access a Demo instance if you have access to the company's intranet at https://inspec-dev.mitre.org

You can login via the company LDAP server, or by creating a new account.

Once you have an account you can upload jsons for evaluations and profiles then view them by clicking on the evaluations and profiles tab at the top of the page.

When uploading data you may go to the circles tab, and select public. This will allow all visitors to view the profile/evaluation you uploaded.

To upload through curl you'll need an API key. This is located on your profile page which can be reached by clicking on your user name in the top right corner, then on profile.

The upload API takes three parameters: the file, your email address, and your API key.

curl -F "file=@FILE_PATH" -F email=EMAIL -F api_key=API_KEY http://localhost:3000/evaluation_upload_api

Configuration

See docker-compose.yml for container configuration

Build container from behind an Intercepting proxy

Contact us for advice, we'll be able to send most people our setup.

Host container off relative url

Edit RAILS_RELATIVE_URL_ROOT line from docker-compose.yml

Switch container to dev mode

Set RAILS_ENV = to development in docker-compose.yml

Development

Clone, edit, then please submit a PR with an issue number associated.

Contributing, Issues and Support

Contributing

Please feel free to look through our issues, make a fork and submit PRs and improvements. We love hearing from our end-users and the community and will be happy to engage with you on suggestions, updates, fixes or new capabilities.

Issues and Support

Please feel free to contact us by opening an issue on the issue board, or, at [email protected] should you have any suggestions, questions or issues. If you have more general questions about the use of our software or other concerns, please contact us at [email protected].

Licensing and Authors

Authors

• Robert Thew
• Aaron Lippold
• Matthew Dromazos
• Luke Malinowski

NOTICE

© 2018 The MITRE Corporation.

Approved for Public Release; Distribution Unlimited. Case Number 18-3678.

NOTICE

MITRE hereby grants express written permission to use, reproduce, distribute, modify, and otherwise leverage this software to the extent permitted by the licensed terms provided in the LICENSE.md file included with this project.

NOTICE

This software was produced for the U. S. Government under Contract Number HHSM-500-2012-00008I, and is subject to Federal Acquisition Regulation Clause 52.227-14, Rights in Data-General.

No other use other than that granted to the U. S. Government, or to those acting on behalf of the U. S. Government under that Clause is authorized without the express written permission of The MITRE Corporation.

For further information, please contact The MITRE Corporation, Contracts Management Office, 7515 Colshire Drive, McLean, VA 22102-7539, [phone].

heimdall-mongo's Issues

Error uploading json with descriptions field

The InSpec output that our tests are generating has this additional "descriptions": field that is not included in the control.rb model.

"controls": [
        {
          "id": "V-13613",
          "title": "The Web site software ...",
          "desc": "The IAVM process ...",
          "descriptions": [
            {
              "label": "default",
              "data": "The IAVM process ..."
            }
          ],
          "impact": 0.5,
          "etc": "etc ..."
       }]

When uploading an evaluation that includes this field, the following error is thrown:

Attempted to set a value for 'descriptions' which is not allowed on the model Control.
summary:
  Without including Mongoid::Attributes::Dynamic in your model and the attribute does not already 
  exist in the attributes hash, attempting to call Control#descriptions= for it is not allowed. This is also
 triggered by passing the attribute to any method that accepts an attributes hash, and is raised instead 
  of getting a NoMethodError.
resolution:
  You can include Mongoid::Attributes::Dynamic if you expect to be writing values for undefined fields often.

Is this a problem with our InSpec output? How can I correct that? If not, I have a fix ready to go that adds the following to control.rb and lets me upload my evaluation without a problem.

  field :descriptions, type: Array, default: []

Make the 'DISA Checklist' a choice between 'raw' and 'download'

I think for the most part folks will want to 'download' a copy of the checklist by default rather than a raw file display. We should have the option for both as the 'raw' output would be used by systems pulling the data but for humans, a 'save as' options seems to make sense.

pdf report: please add the enhancement titles as well

SC-5: MY TITLE
AC-3: TITLE

Print the control code with new lines

Redirect to login page with a model "error" rather than a rails error page

When you are currently not logged in and you git a page that requires you to be authenticated we are going to the standard rails error page.

We should update this to redirect to the login page with a red error notification saying that you need to log in to use this feature, section or whatnot.

User does not show as logged in

After logging in, the button "Log In" link in the navbar is still shown and none of the additional options are available in the navbar.

I was using the "remember me" function.

Here is what happens when I try to log in again...

default profile is missing from the asset pipeline in production

heimdall_web.1.zba4fbducy4b@default    | F, [2018-08-01T12:33:10.319933 #1] FATAL -- : [ab75f27c-aab7-4364-9479-dc81000486e9]   
heimdall_web.1.zba4fbducy4b@default    | F, [2018-08-01T12:33:10.320064 #1] FATAL -- : [ab75f27c-aab7-4364-9479-dc81000486e9] ActionView::Template::Error (The asset "default_profile_pic.jpeg" is not present in the asset pipeline.):
heimdall_web.1.zba4fbducy4b@default    | F, [2018-08-01T12:33:10.320321 #1] FATAL -- : [ab75f27c-aab7-4364-9479-dc81000486e9]     20:               <% if @user.image_url.present? %>
heimdall_web.1.zba4fbducy4b@default    | [ab75f27c-aab7-4364-9479-dc81000486e9]     21:                 <%= image_tag @user.image_url.to_s, class: "rounded-circle img-thumbnail"  %>
heimdall_web.1.zba4fbducy4b@default    | [ab75f27c-aab7-4364-9479-dc81000486e9]     22:               <% else %>
heimdall_web.1.zba4fbducy4b@default    | [ab75f27c-aab7-4364-9479-dc81000486e9]     23:                 <%= image_tag "default_profile_pic.jpeg", class: "rounded-circle img-thumbnail" %>
heimdall_web.1.zba4fbducy4b@default    | [ab75f27c-aab7-4364-9479-dc81000486e9]     24:               <% end %>
heimdall_web.1.zba4fbducy4b@default    | [ab75f27c-aab7-4364-9479-dc81000486e9]     25:                 <!-- <img src="" class="rounded-circle img-thumbnail" alt="profile-image"> -->
heimdall_web.1.zba4fbducy4b@default    | [ab75f27c-aab7-4364-9479-dc81000486e9]     26:             </div>
heimdall_web.1.zba4fbducy4b@default    | F, [2018-08-01T12:33:10.320448 #1] FATAL -- : [ab75f27c-aab7-4364-9479-dc81000486e9]   
heimdall_web.1.zba4fbducy4b@default    | F, [2018-08-01T12:33:10.320589 #1] FATAL -- : [ab75f27c-aab7-4364-9479-dc81000486e9] app/views/users/show.html.erb:23:in `_app_views_users_show_html_erb__1777844105659209954_47082317246760'

Likely just a config change is needed

Control Family Filters

Ability to create a 'grouped filter' of controls families to view the data in sub-sets with respect to the selected family of controls.

Clean up old / merged branches

warning: constant OpenSSL::Cipher::Cipher is deprecated

The fix is in a PR opened in February, but it looks like gibberish isn't very actively maintained so it has not been merged yet: mdp/gibberish#27

Additionally, the deprecation itself is purely an API level change per the following source. The deprecated class is identical to the new class with exception of the deprecation message being inserted:
https://github.com/ruby/openssl/blob/master/lib/openssl/cipher.rb

In all, Gibberish seems to be more of an issue than the usage of the Cipher::Cipher openssl class.

Shrink docker image

Work to shrink docker image to 300MB uncompressed.
Don't know if it's doable, but its a target to shoot for.

Support of new InSpec 3.0 data features

InSpec 3.x introduced two breaking change to our apps - multi-section describe blocks

1. Multi-Section describe blocks
   see: inspec/inspec#3424

2. Text based impacts
   see: inspec/inspec#3359

We will need to update all our apps to support both pre and post InSpec.

Serve heimdall application at baseurl

Please consider serving Heimdall application at the baseurl so that application can be accessed at
http://localhost:3000 rather than http://localhost:3000/heimdall

Add a REST API to heimdall2 for pushing data and such...

Clean up or close resolved issues

`rake assets:precompile --trace` fails if the db does not exist yet

This is due to config/initializers/z_circle.rb.
rails/rails#25246 has a simple solution for ignoring the db during precompile.
Then when the server actually starts it will add the public circle.

However, I'd like to know if the DB really should exist during precompilation, and the build environment is what actually needs to be fixed.

Improve gitlab-runner deploy

Currently gitlab-runner CI cannot handle deploying nginx-passenger builds as it is running in a container, and it needs to modify rvm installed gems for passenger to work. Additionally launching containers from within a container only mostly works, it cannot setup networks.

add the `inspec_tools` results parsing and `min accepted results` summary etc. in UX and API

Given the work @rx294 is doing with inspec_tools for further parsing of results JSON for the totals of the results. This also adds the ability to define a set of acceptable number of high, med, low, none, skipped, NA etc. and the 80% of compliance etc.

It would be good to be able to add another indicator that we link to the both all results or the circle the results are a part of. We can also add a setting for alerting when a scan result fails to meet the minimum result.

For example:

1. All results should be at least 80% compliant with no high or critical findings
2. All results should have only 0 critical, 3 high, 5 med, any low or none or skipped
3. We will need a model for setting this attached to the circle or generally

This would be a good thing we could add to the default or summary screen and we could also add this as a general set of data for all the results or of your circles you are a part of.

We should also add this data to:

1. The results of the api upload data
2. Add an api call to ask for the summary results of a result file directly

NIST Tags, CIS Tags and CCI tags as well and data

• https://github.com/usnistgov/sctools/tree/master/bt-angular
• https://nvd.nist.gov/800-53/ ( Display format and labels )
• http://iasecontent.disa.mil/stigs/zip/u_cci_list.zip

cci_tags: {...}
reason:
nist_tags: {...}
reason:
cis_tags: {...}
reason:
note: should be very similar to a nist_tags structure.

Increase the size of the code block in the editor of a control

dynamically add Profile Error tile

In the (hopefully rare) instance where a profile control returns "no result" either through poor coding or not running with appropriate credentials/authorization, dynamically add a stand-alone tile to the right of the findings block of four tiles. Label this "Profile Error", and below it in parenthesis:
(no result from test - check profile run privileges or author of profile)

Alpine 3.8.x and ImageMagick

see: http://wordsandmagic.com/2017/09/01/RMagick-2-16-0-Error-MagickWand/

docker-compose.yml should imply secret generation for secrets.yml

Dockerfile and docker-compose.yml have the ability to manage secrets through docker which limits leakage by avoiding storing them un-encrypted on the main FS. They are mounted to /run/secrets/ in the container.

Show Details on profiles page does not work

This is actually an error with baseurls in show_details.

Timezone fix

Currently Heimdall app follows UTC time... it should preferably pull timezone and time from the OS

Please consider a sorting feature for the evaluation view

Currently the evaluations are populated on the table with oldest entry on the top, which would make it difficult to examine newer entries especially as the list grows large

Please consider a table sort feature and possibly make Descending by timestamp the default option so that latest entry is on top.

Gitlab CI should deploy development builds to the Dev Server

Add docker-compose up to .gitlab-ci.yml with the assumption that gitlab CI tool has some form of dind. Either through explicit docker in docker or volume export of docker socket to CI container.

Allow Curl access on heimdall

We want to be able to run curl commands from other systems to upload an evaluation.

Select and display multiple 'results' in one view. ( proactive layers )

We would like the ability to select more than one results sets and display them in the gui and 'turn on / turn off' each 'layer' as we do review of the 'results stack'.

The redirect after login is to '/' as opposed to '/baseurl'

May be fixable with https://github.com/plataformatec/devise/wiki/How-To:-redirect-to-a-specific-page-on-successful-sign-in.

compute severity off impact value

Given that we are now using the impact element inside the control to assign severity if and when a severity tag doesn't exist we need to add this to heimdall.

6f3d41b
4785de8

Some of the new profiles and results that follow the current best practice of not using the severity tag are not able to be imported.

Unable to see test results in heimdall

I am having an issue where test results are unable to load for a specific test, below are the results as showing in heimdall-lite:

And the results in heimdall:

The JSON output in question can be found at http://employeeshare.mitre.org/r/rbclark/transfer/broken-test.json

Ability to load multiple jsons and display at one time

Load multiple results files at once - we need this to show a 'system level' compliance view.

Recommended compliance level

Change "Compliance Level" to "Compliance Level [ (Not a Finding) / (Not a Finding + Open + Not Reviewed) * 100% ]"

Also, change Compliance Level calculation to match. (to match Heimdall lite version)

Evaluations export formats broken

The export functions for XCCDF, CSV, and CKL don't work anymore. An exception is thrown. It is probably an issue with the refactoring of inspec_tools.

Add support for docker swarm deployment

Related to issue #36, fix here should also add code for docker secret usage.

Compliance level change

For Compliance Level, change formula description and calculation to:
Compliance Level [Not A Finding / (Not A Finding + Open + Not Reviewed + Profile Error) * 100]

(Profile Error is what we know as InSpec outputting "no result" due to poor coding or not running as root, etc.)

Clear Filter Button Hidden

The "Clear Filter" button on the evaluations /heimdall/evaluations/<uuid> gets hidden behind some of the other UI elements as you scroll down the page. See the below images:

Add support for private gem repos

docker builds need some support for private gem repos due to inspec tools, easiest fix is ENV vars, they're easy to change and inherently difficult to commit by accident.

Tag Evaluations During Upload

Add ability to tag an evaluation into a Circle before the upload takes place. Currently you can only add an evaluation to a Circle after it's uploaded.

Graphs on evaluations do not display when using a base url

The issue stems from loadTreemapData() in the evaluations view using an absolute path to route to a given evaluation.json.

Inserting <%= home_path %> to the api_path variable in loadTreemapData() in the show view's erb is the first fix that comes to mind.

loadTreemapData is also in the profiles show view, but I do not know if it is bugged or does anything at all.

Use Dockerhub to distribute pre-built images

Requiring end users to always build their own image is less than ideal. Most of the time it is possible and more convenient to provide a prebuilt image through dockerhub. Additionally, it is a lot easier for users who are behind a proxy.

Profiles from evaluation

In the current verision, Inspec Profile is extracted from every evaluation uploaded and listed in the Profiles even if they are duplicates.

Only unique profiles entries needs to be populated.

Uniqueness could be evaluated using profile shasum entry.

Findings relabel

For the "Not Applicable" findings tile, change the text in parenthesis to:
(zero impact: exception for this system and/or absent component)

For "Not Reviewed", change the text to
(can only be tested manually or disabled test)

Link to app root in header doesn't respect base url

When linking to the app's root we get '/' instead of the appropriate
'/base_url' which derives from RAILS_RELATIVE_URL_ROOT

NIST SP 800-53 Security Control labeling

Any time a label for NIST 800 53, should read "NIST SP 800-53"

Update LDAP configs/mv ldap.yml to ldap.example.yml

Mitre's internal LDAP is changing so we should update our example ldap.yml.
https://comm.mitre.org/ldap/

Production builds are not able to be served from a baseurl

fontawesome and ionicons fail to load when served to a production environment, with a baseurl set. Additionally, the app itself does not respect RAILS_RELATIVE_URL_ROOT when loading.

Add the ability to push a JSON file via a secure PUTS request to a running instance.

We need the ability to upload results to a running heimdall without humans clicking on buttons. So perhaps what we really need is an API?

Pages slow to load due to number of queries

Every time I load an evaluation I notice it takes about 6-10 seconds for the results to show up. It seems that this is due to about 450 repeated calls to

MONGODB | localhost:27017 | dashboard_development.find | STARTED | {"find"=>"results", "filter"=>{"control_id"=>BSON::ObjectId('5c4b5b65c60ac1581a738697')}}
MONGODB | localhost:27017 | dashboard_development.find | SUCCEEDED | 0.019s```

Rails should make it possible to load all of these results at once instead of repeated calls to the database, which should significantly speed up the load times of the page.