rapid7/metasploit_data_models

MSF database code, gemified

#MetasploitDataModels

Part of Metasploit's database layer

Purpose

MetasploitDataModels exists to do several key things:

1. Allow code sharing between Metasploit Framework (MSF) and the commercial versions of Metasploit (Community, Express, Pro -- usually referred to collectively as "Pro")

2. Give developers a lightweight entry point to MSF's backend for use in developing tools that gather data intended for later use with Metasploit (e.g. specialized scanners).

3. Make it easy to keep commercial stuff private while increasing the functionality of the open-source tools we provide to the community.

Usage

Rails

In a Rails application, MetasploitDataModels acts a Rails Engine and the models are available to application just as if they were defined under app/models. If your Rails appliation needs to modify the models, this can be done using ActiveSupport.on_load hooks in initializers. The block passed to on_load hook is evaluated in the context of the model class, so defining method and including modules will work just like reopeninng the class, but ActiveSupport.on_load ensures that the monkey patches will work after reloading in development mode. Each class has a different on_load name, which is just the class name converted to an underscored symbol, so Mdm::ApiKey runs the :mdm_api_key load hooks, etc.

# Gemfile
gem :metasploiit_data_models, :git => git://github.com/rapid7/metasploit_data_models.git, :tag => 'v0.3.0'

# config/initializers/metasploit_data_models.rb
ActiveSupport.on_load(:mdm_api_key) do
  # Returns the String obfuscated token for display. Meant to avoid CSRF
  # api-key stealing attackes.
  def obfuscated_token
    token[0..3] + "****************************"
  end
end

This gem's Rails::Engine is not required automatically. You'll need to also add the following to your config/application.rb:

require 'metasploit_data_models/engine'

ArelHelper classes must also be included in your applications app/models/application_record.rb class:

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true
  include ArelHelpers::ArelTable
  include ArelHelpers::JoinAssociation
end

Metasploit Framework

In Metasploit Framework, MetasploitDataModels::Engine is loaded, but the data models are only if the user wants to use the database.

Elsewhere

In Metasploit Pro, MDM is loaded via the metasploit_data_models gem: https://rubygems.org/gems/metasploit_data_models

Developer Info

Console

The gem includes a console based on Pry

Give it a path to a working MSF database.yml file for full ActiveRecord-based access to your data.

Note: "development" mode is hardcoded into the console currently.

Contributing

See CONTRIBUTING.md

metasploit_data_models's Issues

Update to Recog 2.0+

The recog gem dependency should be updated to 2.0 in order to account for a major version change. Recog 2.0 requires Ruby 2.1

Models are not loaded in proper order

Having a small problem with the latest version:

kinetik@epitome ~/metasploit-framework $ bundle install
Fetching gem metadata from https://rubygems.org/.........
Fetching additional metadata from https://rubygems.org/..
Resolving dependencies...
Using rake 10.3.2
Using i18n 0.6.9
Using multi_json 1.0.4
Using activesupport 3.2.18
Using builder 3.0.4
Using activemodel 3.2.18
Using arel 3.0.3
Using tzinfo 0.3.39
Using activerecord 3.2.18
Using bcrypt 3.1.7
Using database_cleaner 1.3.0
Using diff-lcs 1.2.5
Using factory_girl 4.4.0
Using fivemat 1.2.1
Using json 1.8.1
Using metasploit-concern 0.1.0
Using metasploit-model 0.24.1
Using pg 0.17.1
Installing metasploit_data_models 0.17.5
Using mini_portile 0.6.0
Using msgpack 0.5.8
Using network_interface 0.0.1
Using nokogiri 1.6.2.1
Using packetfu 1.1.9
Using pcaprub 0.11.3
Using redcarpet 3.1.2
Using rkelly-remix 0.0.6
Using robots 0.10.1
Using rspec-support 3.0.2
Using rspec-core 3.0.2
Using rspec-expectations 3.0.2
Using rspec-mocks 3.0.2
Using rspec 3.0.0
Using shoulda-matchers 2.6.1
Using simplecov-html 0.5.3
Using simplecov 0.5.4
Using sqlite3 1.3.9
Using timecop 0.7.1
Using yard 0.8.7.4
Using bundler 1.6.3
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
kinetik@epitome ~/metasploit-framework $ bundle exec ./msfconsole 
/home/kinetik/.gem/ruby/1.9.1/gems/metasploit_data_models-0.17.5/app/model/mdm/service.rb:196:in `<class:Service>': uninitialized constant MetasploitDataModels::Search::Operator (NameError)
        from /home/kinetik/.gem/ruby/1.9.1/gems/metasploit_data_models-0.17.5/app/models/mdm/service.rb:2:in `<top (required)>'
        from /home/kinetik/metasploit/metasploit-framework/lib/fastlib.rb:374:in `require'
        from /home/kinetik/metasploit/metasploit-framework/lib/fastlib.rb:374:in `require'
        from /home/kinetik/.gem/ruby/1.9.1/gems/metasploit_data_models-0.17.5/lib/metasploit_data_models/models.rb:17:in `block in require_models'
        from /home/kinetik/.gem/ruby/1.9.1/gems/metasploit_data_models-0.17.5/lib/metasploit_data_models/models.rb:16:in `glob'
        from /home/kinetik/.gem/ruby/1.9.1/gems/metasploit_data_models-0.17.5/lib/metasploit_data_models/models.rb:16:in `require_models'
        from /home/kinetik/metasploit/metasploit-framework/lib/msf/core/db_manager.rb:106:in `initialize_database_support'
        from /home/kinetik/metasploit/metasploit-framework/lib/msf/core/db_manager.rb:78:in `initialize'
        from /home/kinetik/metasploit/metasploit-framework/lib/msf/core/framework.rb:88:in `new'
        from /home/kinetik/metasploit/metasploit-framework/lib/msf/core/framework.rb:88:in `initialize'
        from /home/kinetik/metasploit/metasploit-framework/lib/msf/base/simple/framework.rb:73:in `new'
        from /home/kinetik/metasploit/metasploit-framework/lib/msf/base/simple/framework.rb:73:in `create'
        from /home/kinetik/metasploit/metasploit-framework/lib/msf/ui/console/driver.rb:82:in `initialize'
        from ./msfconsole:148:in `new'
        from ./msfconsole:148:in `<main>'

It looks like the issue is in lib/metasploit_data_models/models.rb, lines 14-18. It pulls in files in the wrong order, requiring files whose dependencies haven't been loaded yet.

Recog service fingerprints will never be matched

Due to an improper use of Hash.merge in https://github.com/rapid7/metasploit_data_models/blob/10fce8e90f20688ef78de054d54c4031425662a4/lib/mdm/host/operating_system_normalization.rb#L207-228.