likkey/ceitinspector

The source code of CeitInspector for paper "Challenges and Opportunities: An In-depth Empirical Study on Configuration Error Injection Testing"

CeitInspector

CeitInspector is a framework aimed at providing a comprehensive misconfiguration testing for all the software systems.

Motivation

Similar to software bugs, configuration errors have become a major cause for system failures. Misconfigurations are not only harmful, but also very hard to discover without proper warning notifications by the software.

How does it work?

CeitInspector will automatically inject configuration errors into different software systems, carry out the tests for the system and generate a report by analyzing the test results. The report will help developers spot configurations that are prone to be misconfigured, give them insights to fix these "bugs".

The configuration parameters to be tested and all the system integrated testcases should be defined beforehand.

Supported Software Systems

• Httpd
• MariaDb
• Nginx
• PostgreSQL
• Redis
• Squid
• VsFtpd
• HDFS
• Alluxio
• To be added..

Sample Report

CeitInspector will list all the configuration parameters that are tested, as well as the type of misconfigurations injected.

The report will include Testcase Results which indicates if the system successfully finished the tests after the misconfiguration was injected.

The report also includes Analyzer Results which is generated by analyzing the results of the system integrated tests. Normally, we consider it bad if the system finished all the tests without and error messages or if the system failed but didn't provide enough error message to help the user find out what went wrong. Here, we used Whoosh to provide a very simple version of analyzer by the indexing and searching target workds in test outputs. The more capable version of result analyzer is still to be developed.

How To Run CeitInspector (A Simple Example of Squid)

Redis

You are supposed to run the redis-server while testing. Redis server is used to store general results for testcases.

$wget http://download.redis.io/releases/redis-3.0.6.tar.gz
$tar xzf redis-3.0.6.tar.gz
$cd redis-3.0.6
$make && make install
$redis-server&

Python Package Dependencies

$pip install whoosh
$pip install wordsegmentation
$pip install wordsegment
$pip install redis

How to use

code can be found in examples/Squid

import sys
sys.path.append( "../.." )
from ceitinspector import MainEngine

# Run the whole tests online
me = MainEngine() #start up the main engine
me.print_options() #print all the parameters that can be tested (the parameters should be pre-defined in option_list.json)
me.self_check() #See if the tests can be finished properly with the original configuration file
me.run() #run all the tests, and all the result outputs will be saved in the Results directory
me.failures_analyzing() #use oracles and test result outputs to generate the report
me.dump_overall_results(file_path="/CeitInspector_squid_conferr.csv") #output the final report

Things worth knowing

option_list.json should be manually written before the tests to define which configuration parameters are to be tested. You can either walk through the configuration file to find out different parameters or read the documentations of the system related to configuration.

test_scripts.json is a file that defines testcases. Is is usually bash scripts that will run in order.

test_oracles.json is a file that defines successful test outputs and some settings.

"4": {
    "oracle": "Accepting HTTP Socket connections",
    "running": true,
    "timeout": 2,
    "ignored": false,
    "log2annotate": [
      "Processing"
    ],
    "log2purge": [
      "d{4}/d{2}/d{2} d{2}:d{2}:d{2}|"
    ]
  }

oracle is used to compare with test outputs to define a successful run. timeout defines the max running time for this case in seconds. log2annotate and log2purge are used when analyzing the results.

Directory Structure

CeitInspector 
    ├── core
    │   ├── analysis.py                # Result Analysis Engine, analyze the output log messages
    │   ├── test.py                # Test Engine, control the testing process
    │   ├── parseconf.py                # Conf Parser, read and modify the conf
    │   ├── misconf.py                # Misconf Gneration Engine, generate the misconfiguration according to the requirements
    │   ├── main.py                # Main function
    │   ├── config.py                # Configuration for CeitInspector
    │   ├── database.py                # Database Engine, record the test results 
    │   ├── global_variables.py                # Global Viarables
    │   └── log.py                # Log Engine for CeitInspector
    ├── modules
    │   ├── conf_parser                # modules to help parse the configuration
    │   │   ├── augeas.py
    │   │   ├── elektra.py
    │   │   ├── nginx_parser.py
    │   │   └── plain_text.py
    │   ├── data_recorder                # modules to record the data during the tests
    │   ├── misconf_generator                # modules to generate the misconfigruations
    │   │   ├── ConfErr
    │   │   ├── ConfTest
    │   │   └── Fuzzing
    │   ├── result_analyzer                # modules to help analyze the results
    │   ├── supporter                # modules to help CeitInspector adjust the unique features from SUT
    │   └── system_tester                # modules to help execute the test cases and record the results from oracles
    └── utils                # utils such like help calculate the statistics, generate uniformed json files etc.

CeitInspector is well organized to make adding and modification of modules easier. The modules are defined and separated by their functionalities. Files in Core will call on a uniform interface and different modules will do works differently with the same interface. The selection of different modules can be configured by setting.json

Settings

{
  "software_name": "Squid",
  "conf_path": "/etc/squid/squid.conf",
  "conf_parse_mode": "PlainText",
  "misconf_mode": "ConfErr",
  "test_mode": "Default",
  "add_new_options": true,
  "interval": 0.25,
  "log_file_path": "/var/log/squid/cache.log",
  "char2cut": 0
}

conf_path

The path to the system's configuration file

conf_parse_mode

• PlainText Used in most cases. Parse config files as plain text.
• Augeas DEPRECATED
• Elektra A way to universally parse all config files. Not completed yet.
• Nginx Specifically used for Nginx

misconf_mode

• ConfErr Add misconfigurations in methods like omission, misspelling, deletion, change of delimiter and change letter case.
• Fuzzing Generate a random string to replace the original value
• ConfTest Add misconfigurations according to the type of parameters

test_mode

• Default
• Httpd
• Nginx
• etc.

add_new_options

Specify if you want to test with the parameter defined in option_list.json but not in the configuration file.

interval

The time interval between test cases. Value unit: second.

char2cut

Deprecated.

What else can be done to improve this project?

• Add supports to more software systems.
• Implement a universal parser for configuration files so that minimum additional development works should be needed when adding new systems.
• Improve the result analyzer so that more useful feedbacks can be given.
• Add more ways to inject misconfigurations and also make the injected misconfigurations better connected with the original ones.