mozillasecurity/grizzly

A cross-platform browser fuzzing framework

Grizzly

Grizzly is a modular general purpose browser fuzzing framework. The goal is to create a platform that can be extended via the creation of plug-ins to support multiple combinations of browsers and fuzzers. An Adapter is used to add support for a fuzzer and a Target to add support for a browser. Generated test cases are intended to be standalone and not require Grizzly.

Cross platform compatibility is available for Windows, Linux and MacOS. However not all features may be available.

For additional information please check out the wiki or the announcement.

Quick Start

Install the latest version from PyPI. For more details see getting started on the wiki.

python3 -m pip install grizzly-framework --upgrade

Fuzzing - Run the no-op test adapter to check everything is working.

python3 -m grizzly <browser-binary> no-op

Reduce - Grizzly Reduce can reduce a test case.

python3 -m grizzly.reduce <browser-binary> <testcase>

Replay - Grizzly Replay can replay a test case with different builds and debuggers.

python3 -m grizzly.replay <browser-binary> <testcase>

Bugzilla is also supported by Grizzly Replay. Bugs can be replayed via a bug ID:

python3 -m grizzly.replay.bugzilla <browser-binary> <bug_id>

Questions

Common questions can be found on the Q&A page. Questions can also be asked in the #fuzzing channel.

Please be sure you are using the latest version Grizzly before reporting issues.

grizzly's Issues

[reducer] Reset Q4s when pool is cycled

Resetting Q4 to Q5 (or Q6 depending on system) will prevent orphaned Q4 crashes

[replay] Add ability to map include paths

[reducer] Crashes found during reduction are not packaged properly

See https://fuzzmanager.fuzzing.mozilla.org/crashmanager/crashes/3485273/

[tests] intermittent test_reduce_status_reporter_04 failure

________________________ test_reduce_status_reporter_04 ________________________
tmp_path = PosixPath('/tmp/pytest-of-travis/pytest-0/test_reduce_status_reporter_040')
    def test_reduce_status_reporter_04(tmp_path):
        """test ReduceStatusReporter._summary() ignore inactive"""
        ReduceStatusReporter.CPU_POLL_INTERVAL = 0.01
        test_db = tmp_path / "test.db"
        Status.DB_FILE = str(test_db)
        status = ReduceStatus.start()
        try:
            status.reduce_error = 1
            status.reduce_fail = 2
            status.reduce_pass = 10
            status.report(force=True)
        finally:
            status.close()
        rptr = ReduceStatusReporter.load(str(test_db))
        assert rptr.reports is not None
        assert len(rptr.reports) == 1
        output = rptr._summary(sysinfo=True, timestamp=True)
        assert "Processed" in output
        assert "Reduced" in output
        assert "No Repro" in output
        assert "Error" in output
        assert "Timestamp" in output
>       assert "Active" not in output
E       AssertionError: assert 'Active' not in '---- Processed (13)\n ...9/08/02 17:04:09 +0000'
E         'Active' is contained here:
E           ---- Processed (13)
E              Reduced : 10
E             No Repro : 2
E               Errors : 1
E           ------- Active
E         ?         ++++++
E           Iterations : 0
E                 Rate : 1 @ 0.00
E              Runtime : 0:00:01
E             Mismatch : 0
E           CPU & Load : 2 @ 0.0% (0.67, 0.22, 0.08)
E               Memory : 6.7GB of 7.3GB free
E                 Disk : 56.5GB of 67.8GB free
E            Timestamp : 2019/08/02 17:04:09 +0000
grizzly/common/test_status_reporter.py:397: AssertionError

Don't log on every iteration.

We currently log every iteration made. This verbosity makes long logs harder to navigate.
We should log iterations with exponential back-off, like libFuzzer does.

[sapphire-test] intermittent test test_sapphire_09

Saw this test fail, and when I re-ran it passed.

_________________________________________________________________________________________________ test_sapphire_09 __________________________________________________________________________________________________

client = <sapphire.conftest._SimpleClient object at 0x7f1d2d825ad0>, tmp_path = PosixPath('/tmp/pytest-of-truber/pytest-13/test_sapphire_090')

    def test_sapphire_09(client, tmp_path):
        """test serving a large (100MB) file"""
        serv = Sapphire(timeout=10)
        try:
            t_file = _TestFile("test_case.html")
            data_hash = hashlib.md5()
            with (tmp_path / t_file.url).open("wb") as test_fp:
                # write 100MB of 'A'
                data = b"A" * (100 * 1024)  # 100KB of 'A'
                for _ in range(1024):
                    test_fp.write(data)
                    data_hash.update(data)
            t_file.md5_org = data_hash.hexdigest()
            client.launch("127.0.0.1", serv.get_port(), [t_file])
            assert serv.serve_path(str(tmp_path))[0] == SERVED_ALL
        finally:
            serv.close()
>       assert t_file.code == 200
E       assert None == 200
E        +  where None = <sapphire.test_sapphire._TestFile object at 0x7f1d2da0be50>.code
sapphire/test_sapphire.py:251: AssertionError
------------------------------------------------------------------------------------------------- Captured log call -------------------------------------------------------------------------------------------------
core.py                    484 DEBUG    serve_path: /tmp/pytest-of-truber/pytest-13/test_sapphire_090
core.py                     92 DEBUG    required: 'test_case.html'
core.py                    103 DEBUG    sapphire has 1 files required to serve
core.py                    400 DEBUG    starting client_listener
core.py                    314 DEBUG    check_request(u'test_case.html')
core.py                    326 DEBUG    expecting to finish
core.py                    333 DEBUG    target u'/tmp/pytest-of-truber/pytest-13/test_sapphire_090/test_case.html'
core.py                    370 DEBUG    sending file: 104,857,600 bytes
core.py                    377 DEBUG    200 u'/tmp/pytest-of-truber/pytest-13/test_sapphire_090/test_case.html' (0 to go)
core.py                    457 DEBUG    shutting down and cleaning up workers
======================================================================================= 1 failed, 254 passed in 15.23 seconds =======================================================================================

Fallback to GDB if no ASAN information is found.

Stack overflows often times have no ASAN information associated with them. In these cases we probably should fall back to gdb and submit to fuzzmanager using the gdb output.

[ci] Add Grizzly to codecov.io

Better reporting of browser launch crashes

Reports from crashes that happen on launch should have more information provided. This will help make it clear that it was in fact a browser launch crash and test cases are not actually missing.

Served include files should be added to the TestCase

[reducer] Dynamically adjust retries

Add the ability for the reducer to adjust the number of retries needed to reproduce an issue.
This functionality is already in place with --timeout

Ability to display debug print from only certain modules

If DEBUG=1 is set we get the output from all modules and some are very verbose (avalanche). Having more control over which modules output debug info would be helpful.

Include full minidump_stackwalk output in testcase

Currently we have a lot of crashes on the server that only have partial symbols. It seems possible that there is something wrong with the symbolizing process, like missing symbols, corrupt dump or anything like that. In order to diagnose the problem, having the full minidump_stackwalk output in the testcase archive would be very helpful. This output should also include the stderr output, because that might have vital error messages.

Add ability to add abort strings from CorpusManagers

This will allow CorpusManagers to detect issues other than crashes.

RuntimeError: Failed to create FM signature

[2019-07-22 22:17:00] Starting Grizzly
[2019-07-22 22:17:00] Running with Xvfb
[2019-07-22 22:17:00] Found 0 input files(s)
[2019-07-22 22:17:00] Running in FUZZING mode
[2019-07-22 22:17:00] Using prefs '/home/firefox/code/browsers/prefs.js'
[2019-07-22 22:17:01] Results will be reported via FuzzManager
[2019-07-22 22:17:01] Launching target
[2019-07-22 22:17:05] I0001-R00
[2019-07-22 22:17:36] I0002-R00
[2019-07-22 22:18:42] I0004-R00
[2019-07-22 22:21:16] Terminating browser...
[2019-07-22 22:21:16] Result detected
[2019-07-22 22:21:16] Reporting results...
[2019-07-22 22:21:16] Report is unsupported by FM, saved to '/tmp/grz_logs_h_wj9p'
[2019-07-22 22:21:16] Shutting down...
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/firefox/code/grizzly/grizzly/main.py", line 13, in
sys.exit(console_main())
File "grizzly/main.py", line 51, in console_main
return main(GrizzlyArgs().parse_args())
File "grizzly/main.py", line 161, in main
session.run()
File "grizzly/session.py", line 196, in run
self.check_results(not files_served, server_status == sapphire.SERVED_TIMEOUT)
File "grizzly/session.py", line 56, in check_results
self.report_result()
File "grizzly/session.py", line 155, in report_result
self.reporter.submit(result_logs, self.iomanager.tests)
File "grizzly/common/reporter.py", line 254, in submit
self._submit(report, test_cases)
File "grizzly/common/reporter.py", line 450, in _submit
raise RuntimeError("Failed to create FM signature")
RuntimeError: Failed to create FM signature

This happens every few minutes

Here are the contents of /tmp/grz_logs_h_wj9p/:
at log_metadata.json
{
"log_stderr.txt": {
"st_atime": 1563833821.0280266,
"st_blksize": 4096,
"st_blocks": 8,
"st_ctime": 1563834076.2376258,
"st_dev": 116,
"st_gid": 999,
"st_ino": 2097289,
"st_mode": 33152,
"st_mtime": 1563834076.2376258,
"st_nlink": 1,
"st_rdev": 0,
"st_size": 2137,
"st_uid": 999
},
"log_stdout.txt": {
"st_atime": 1563833821.0280266,
"st_blksize": 4096,
"st_blocks": 8,
"st_ctime": 1563834015.2532432,
"st_dev": 116,
"st_gid": 999,
"st_ino": 2097288,
"st_mode": 33152,
"st_mtime": 1563834015.2532432,
"st_nlink": 1,
"st_rdev": 0,
"st_size": 1679,
"st_uid": 999
}
}
[ffpuppet] Launch command: /home/firefox/code/browsers/firefox/firefox -no-remote -profile /tmp/ffprof_SPOzga http://127.0.0.1:32156

JavaScript error: resource://gre/modules/Sidebar.jsm, line 13: TypeError: window.docShell is null
JavaScript error: http://127.0.0.1:16023/test_0001.html, line 2677: TypeError: The expression cannot be converted to return the specified type.
JavaScript error: http://127.0.0.1:16023/test_0002.html, line 4454: TypeError: The expression cannot be converted to return the specified type.
JavaScript error: http://127.0.0.1:16023/test_0002.html, line 4454: TypeError: The expression cannot be converted to return the specified type.
JavaScript error: , line 0: TypeError: The expression cannot be converted to return the specified type.
JavaScript error: , line 0: NotSupportedError: The media resource indicated by the src attribute or assigned media provider object was not suitable.
JavaScript error: http://127.0.0.1:16023/test_0002.html, line 3033: TypeError: The expression cannot be converted to return the specified type.
JavaScript error: http://127.0.0.1:16023/test_0003.html, line 1580: SyntaxError: An invalid or illegal string was specified
JavaScript error: http://127.0.0.1:16023/test_0004.html, line 3619: SyntaxError: An invalid or illegal string was specified
JavaScript error: http://127.0.0.1:16023/test_0004.html, line 3619: SyntaxError: An invalid or illegal string was specified
JavaScript error: http://127.0.0.1:16023/test_0005.html, line 2951: SyntaxError: An invalid or illegal string was specified
JavaScript error: http://127.0.0.1:16023/test_0005.html, line 2951: SyntaxError: An invalid or illegal string was specified
JavaScript error: , line 0: AbortError: The fetching process for the media resource was aborted by the user agent at the user's request.
JavaScript error: http://127.0.0.1:16023/test_0006.html, line 1388: TypeError: The expression cannot be converted to return the specified type.
JavaScript error: http://127.0.0.1:16023/test_0006.html, line 1601: TypeError: The expression cannot be converted to return the specified type.
[ffpuppet] Reason code: CLOSED

cat log_stdout.txt
1563833822083 [email protected] WARN Loading extension '[email protected]': Reading manifest: Invalid extension permission: mozillaAddons
1563833822083 [email protected] WARN Loading extension '[email protected]': Reading manifest: Invalid extension permission: telemetry
1563833822084 [email protected] WARN Loading extension '[email protected]': Reading manifest: Invalid extension permission: resource://pdf.js/
1563833822084 [email protected] WARN Loading extension '[email protected]': Reading manifest: Invalid extension permission: about:reader*
[grz harness][Mon, 22 Jul 2019 22:17:05 GMT] Using test case time limit of 30000
1563833844682 addons.productaddons WARN Failed downloading XML, status: 0, reason: error
[grz harness][Mon, 22 Jul 2019 22:17:35 GMT] Test case time limit exceeded
[grz harness][Mon, 22 Jul 2019 22:17:35 GMT] Closing test case
[grz harness][Mon, 22 Jul 2019 22:18:08 GMT] Test case time limit exceeded
[grz harness][Mon, 22 Jul 2019 22:18:08 GMT] Closing test case
[grz harness][Mon, 22 Jul 2019 22:18:41 GMT] Test case time limit exceeded
[grz harness][Mon, 22 Jul 2019 22:18:41 GMT] Closing test case
[grz harness][Mon, 22 Jul 2019 22:19:12 GMT] Test case time limit exceeded
[grz harness][Mon, 22 Jul 2019 22:19:12 GMT] Closing test case
[grz harness][Mon, 22 Jul 2019 22:19:44 GMT] Test case time limit exceeded
[grz harness][Mon, 22 Jul 2019 22:19:44 GMT] Closing test case
[grz harness][Mon, 22 Jul 2019 22:20:15 GMT] Test case time limit exceeded
[grz harness][Mon, 22 Jul 2019 22:20:15 GMT] Closing test case

[reducer] Bucket iterator should also provide ability to filter by quality

[sapphire] "thread.error: release unlocked lock" following Ctrl-C on macos

Seen after Ctrl-C:

Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/truber/grizzly-auto-run/grizzly/grizzly/__main__.py", line 13, in <module>
    sys.exit(console_main())
  File "/Users/truber/grizzly-auto-run/grizzly/grizzly/main.py", line 53, in console_main
    return main(GrizzlyArgs().parse_args())
  File "/Users/truber/grizzly-auto-run/grizzly/grizzly/main.py", line 152, in main
    session.run()
  File "/Users/truber/grizzly-auto-run/grizzly/grizzly/session.py", line 180, in run
    optional_files=current_test.get_optional())
  File "/Users/truber/grizzly-auto-run/grizzly/sapphire/core.py", line 530, in serve_path
    while not job.is_complete(wait=0.5):
  File "/Users/truber/grizzly-auto-run/grizzly/sapphire/core.py", line 163, in is_complete
    return self._complete.wait(wait)
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 615, in wait
    return self.__flag
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 289, in __exit__
    return self.__lock.__exit__(*args)
thread.error: release unlocked lock

Python 3 support

[reducer] add crash ID to metadata when reporting crashes to fuzzmanager from grizzly.reduce.crash

Currently metadata contains this:

grizzly_input=[['grizzly.reduce', 'tmpwgQmAw.zip']]

This isn't very useful. We could put the crash ID in this field.

Support installing multiple extensions from cmdline

[reducer] Report partially reduced test cases

This should be done if a test case becomes less reliable and the initial test case is then marked as QUAL_REDUCER_BROKE (Q8)

Add or display ASAN_OPTIONS in crash report

Example: https://fuzzmanager.fuzzing.mozilla.org/crashmanager/crashes/3637015 (see Environment). Observation is based on the rr reports in which this is not included or displayed.

Make `RuntimeError: Failed to create FM signature` more descriptive

As @kkuehl saw in issue #71 the error message reported when FM signature creations fails could be a bit better.

[reducer] Relaunch match retries

Add the ability to have relaunch match the current value for retries.

[reducer] TestFileExists: '.lithium-garbage.bin' exists in test

I've seen this a few times now running the reducer.

[2019-08-16 11:14:23] Testcase was interesting 0.0% of 11 attempts using harness for iteration.
[2019-08-16 11:14:23] Running for 11 iterations to assess reliability without harness.
[2019-08-16 11:14:23] Exception during reduce
Traceback (most recent call last):
  File "grizzly/reduce/reduce.py", line 452, in run
    result = reducer.run()
  File "/home/user/code/lithium/src/lithium/reducer.py", line 1312, in run
    result = self.strategy.main(self.testcase, self.interesting, self.testcaseTempFilename)
  File "grizzly/reduce/strategies.py", line 118, in main
    result = interesting(testcase, writeIt=False)  # pylint: disable=invalid-name
  File "/home/user/code/lithium/src/lithium/reducer.py", line 1469, in interesting
    inter = self.conditionScript.interesting(self.conditionArgs, tempPrefix)
  File "grizzly/reduce/interesting.py", line 251, in interesting
    testcase.add_from_data("", ".lithium-garbage.bin", required=True)
  File "grizzly/common/storage.py", line 173, in add_from_data
    self.add_file(tfile, required=required)
  File "grizzly/common/storage.py", line 155, in add_file
    self._add(self._files.required, test_file)
  File "grizzly/common/storage.py", line 117, in _add
    raise TestFileExists("%r exists in test" % (test_file.file_name,))
TestFileExists: '.lithium-garbage.bin' exists in test

RuntimeError: Fatal error parsing ASan trace (Index mismatch, got index 14 but expected 12)

[2017-08-03 05:59:08] Potential issue detected
[2017-08-03 05:59:10] Collecting logs...
[2017-08-03 05:59:10] Shutting down...
Traceback (most recent call last):
File "grizzly.py", line 368, in
main(parse_args())
File "grizzly.py", line 314, in main
result_reporter.report(reversed(test_cases), args.binary, log_limit=0x20000)
File "/home/ubuntu/grizzly/reporter.py", line 81, in report
self._report(*args, **kwargs)
File "/home/ubuntu/grizzly/reporter.py", line 164, in _report
ProgramConfiguration.fromBinary(target_binary))
File "/usr/local/lib/python2.7/dist-packages/FTB/Signatures/CrashInfo.py", line 195, in fromRawCrashData
return ASanCrashInfo(stdout, stderr, configuration, auxCrashData)
File "/usr/local/lib/python2.7/dist-packages/FTB/Signatures/CrashInfo.py", line 537, in init
raise RuntimeError("Fatal error parsing ASan trace (Index mismatch, got index %s but expected %s)" % (index, expectedIndex))
RuntimeError: Fatal error parsing ASan trace (Index mismatch, got index 14 but expected 12)

Grizzly status should use JSON

Grizzly status is currently written as key: value lines. This should be changed to output JSON to eliminate parsing errors.

[reducer] Perform analysis pass between reduction modes

This should reduce the number of Q8 we see reported.

Add GCOV support

GCOV support should include the ability to specify a total amount of iterations to perform, as well as signaling the process after each iteration to ensure that coverage is not lost once a crash is hit.

Both can be controlled by a command line option --gcov-iterations.

Permission denied when adding test file with leading path separator

Grizzly raises a permission denied error when attempting to add a test file with a leading path separator.

WARNING Tracebacks detected!
Log: '/home/ubuntu/grizzly/screenlog.1'
[2017-09-13 17:46:46] Now fuzzing: ownerdiscard.html
[2017-09-13 17:46:46] I0034-R09 
[2017-09-13 17:46:51] Potential issue detected
[2017-09-13 17:47:00] Collecting logs...
[2017-09-13 17:47:00] Shutting down...
Traceback (most recent call last):
  File "grizzly.py", line 429, in <module>
    main(parse_args())
  File "grizzly.py", line 368, in main
    result_reporter.report(reversed(test_cases), args.binary, log_limit=0x40000)
  File "/home/ubuntu/grizzly/reporter.py", line 81, in report
    self._report(*args, **kwargs)
  File "/home/ubuntu/grizzly/reporter.py", line 202, in _report
    test_case.dump(dump_path, include_details=True)
  File "/home/ubuntu/grizzly/corpman/corpman.py", line 95, in dump
    with open(os.path.join(log_dir, test_file.file_name), "wb") as out_fp:
IOError: [Errno 13] Permission denied: u'/ie.png'

[reporter] rr trace is sometimes included in FM report

[replay] Add FuzzManagerReporter support

[replay] Add ability run multiple test cases in sequence

[sapphire] Dynamic request should expose socket

[reducer] Failed reduction attempt due to low target probability

The test case was not 100% reliable however we settle on --min-crashes=2 --repeat=2 for reduction.

...
[2019-10-24 14:05:31] Running for 11 iterations to assess reliability using harness.
...
[2019-10-24 14:12:00] Testcase was interesting 90.9% of 11 attempts using harness for iteration.
[2019-10-24 14:12:00] Running for 11 iterations to assess reliability without harness.
...
[2019-10-24 14:20:15] Testcase was interesting 54.5% of 11 attempts without harness.
[2019-10-24 14:20:15] Analysis results:
[2019-10-24 14:20:15] * testcase was more reliable with the harness
[2019-10-24 14:20:15] * adjusted parameters: --min-crashes=2 --repeat=2 --relaunch=2
[2019-10-24 14:20:15]   Tests performed: 22
[2019-10-24 14:20:15]   Test total: 44 lines
[2019-10-24 14:20:15] AnalyzeTestcase succeeded
[2019-10-24 14:20:15] Reducing /tmp/grzreduceJ0R0mK/tc/harness_XT1yKR.html with minimize on lines
[2019-10-24 14:20:15] The original testcase has 2 lines.
[2019-10-24 14:20:15] Checking that the original testcase is 'interesting'...
[2019-10-24 14:21:28] Uninteresting: no failure detected
[2019-10-24 14:21:28] Lithium result: the original testcase is not 'interesting'!
[2019-10-24 14:21:28]   Tests performed: 1
[2019-10-24 14:21:28]   Test total: 2 lines
[2019-10-24 14:21:28] Could not reduce: The testcase was not reproducible
[2019-10-24 14:21:28] Reduction failed: QUAL_NOT_REPRODUCIBLE
[2019-10-24 14:21:28] Shutting down...

Add used environment variables to report for FuzzManager

[tests] intermittent test_sapphire_17 failure

test_sapphire_17 _______________________________
client_factory = <function client_factory.<locals>._get_client at 0x000002273241BB88>
tmp_path = WindowsPath('C:/Users/travis/AppData/Local/Temp/pytest-of-travis/pytest-0/test_sapphire_170')
    def test_sapphire_17(client_factory, tmp_path):
        """test include directories"""
        inc1_path = tmp_path / "inc1"
        inc2_path = tmp_path / "inc2"
        root_path = tmp_path / "root"
        inc1_path.mkdir()
        inc2_path.mkdir()
        root_path.mkdir()
        files_to_serve = list()
        serv = Sapphire(timeout=10)
        try:
            # add files to inc dirs
            inc1 = _create_test("included_file1.html", inc1_path, data=b"blah....1")
            files_to_serve.append(inc1)
    
            # add a nested dir
            nest_path = inc1_path / "nested"
            nest_path.mkdir()
            # add file in a nested dir in inc1
            nest = _create_test("nested_file.html", nest_path, data=b"blah... .nested", url_prefix="nested/")
            assert nest_path / "nested_file.html"
            files_to_serve.append(nest)
    
            # test 404 in nested dir in inc1
            nest_404 = _TestFile("nested/nested_file_404.html")
            files_to_serve.append(nest_404)
    
            # test path mounted somewhere other than /
            inc2 = _create_test("included_file2.html", inc2_path, data=b"blah....2", url_prefix="inc_test/")
            files_to_serve.append(inc2)
    
            # test 404 in include dir
            inc404 = _TestFile("inc_test/included_file_404.html")
            assert not (nest_path / "included_file_404.html").is_file()
            files_to_serve.append(inc404)
    
            # test 403
            inc403 = _create_test("no_access.html", tmp_path, data=b"no_access", url_prefix="inc_test/../")
            assert (tmp_path / "no_access.html").is_file()
            files_to_serve.append(inc403)
    
            # test file
            test = _create_test("test_case.html", root_path)
            files_to_serve.append(test)
    
            serv.add_include("/", str(inc1_path))  # mount at '/'
            serv.add_include("inc_test", str(inc2_path))  # mount at '/inc_test'
    
            client_incs = client_factory()
            client_reqs = client_factory()
            # client that requests the include files
            # TODO: find out why test fails without in_order=True and fix or make a note
            client_incs.launch("127.0.0.1", serv.get_port(), files_to_serve, in_order=True)
            client_reqs.launch("127.0.0.1", serv.get_port(), [test], delay=0.1)
            # delayed client that requests the required files (once others are requested)
            status, files_served = serv.serve_path(str(root_path))
        finally:
            serv.close()
        assert status == SERVED_ALL
        assert len(files_served) == 4
        assert client_incs.wait(timeout=10)
        client_incs.close()
        assert client_reqs.wait(timeout=10)
        client_reqs.close()
        assert inc1.code == 200
        assert inc2.code == 200
        assert nest.code == 200
        assert test.code == 200
        assert nest_404.code == 404
        assert inc404.code == 404
>       assert inc403.code == 403
E       assert None == 403
E         -None
E         +403
sapphire\test_sapphire.py:463: AssertionError
------------------------------ Captured log call ------------------------------
DEBUG    sapphire:core.py:618 mapping include '' -> 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc1'
DEBUG    sapphire:core.py:618 mapping include 'inc_test' -> 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc2'
DEBUG    sapphire:core.py:489 serve_path: C:\Users\travis\AppData\Local\Temp\pytest-of-travis\pytest-0\test_sapphire_170\root
DEBUG    sapphire:core.py:95 required: 'test_case.html'
DEBUG    sapphire:core.py:106 sapphire has 1 files required to serve
DEBUG    sapphire:core.py:405 starting client_listener
DEBUG    sapphire:core.py:319 check_request('included_file1.html')
DEBUG    sapphire:core.py:136 looking up 'included_file1.html' in include map
DEBUG    sapphire:core.py:144 include map does not contain 'included_file1.html'
DEBUG    sapphire:core.py:148 checking include map at '/'
DEBUG    sapphire:core.py:338 target 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc1\\included_file1.html'
DEBUG    sapphire:core.py:375 sending file: 9 bytes
DEBUG    sapphire:core.py:382 200 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc1\\included_file1.html' (1 to go)
DEBUG    sapphire:core.py:319 check_request('nested/nested_file.html')
DEBUG    sapphire:core.py:136 looking up 'nested/nested_file.html' in include map
DEBUG    sapphire:core.py:144 include map does not contain 'nested/nested_file.html'
DEBUG    sapphire:core.py:136 looking up 'nested' in include map
DEBUG    sapphire:core.py:144 include map does not contain 'nested'
DEBUG    sapphire:core.py:148 checking include map at '/'
DEBUG    sapphire:core.py:338 target 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc1\\nested\\nested_file.html'
DEBUG    sapphire:core.py:375 sending file: 15 bytes
DEBUG    sapphire:core.py:382 200 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc1\\nested\\nested_file.html' (1 to go)
DEBUG    sapphire:core.py:319 check_request('nested/nested_file_404.html')
DEBUG    sapphire:core.py:136 looking up 'nested/nested_file_404.html' in include map
DEBUG    sapphire:core.py:144 include map does not contain 'nested/nested_file_404.html'
DEBUG    sapphire:core.py:136 looking up 'nested' in include map
DEBUG    sapphire:core.py:144 include map does not contain 'nested'
DEBUG    sapphire:core.py:148 checking include map at '/'
DEBUG    sapphire:core.py:338 target 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc1\\nested\\nested_file_404.html'
DEBUG    sapphire:core.py:341 404 'nested/nested_file_404.html' (1 to go)
DEBUG    sapphire:core.py:319 check_request('inc_test/included_file2.html')
DEBUG    sapphire:core.py:136 looking up 'inc_test/included_file2.html' in include map
DEBUG    sapphire:core.py:144 include map does not contain 'inc_test/included_file2.html'
DEBUG    sapphire:core.py:136 looking up 'inc_test' in include map
DEBUG    sapphire:core.py:338 target 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc2\\included_file2.html'
DEBUG    sapphire:core.py:375 sending file: 9 bytes
DEBUG    sapphire:core.py:382 200 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc2\\included_file2.html' (1 to go)
DEBUG    sapphire:core.py:319 check_request('inc_test/included_file_404.html')
DEBUG    sapphire:core.py:136 looking up 'inc_test/included_file_404.html' in include map
DEBUG    sapphire:core.py:144 include map does not contain 'inc_test/included_file_404.html'
DEBUG    sapphire:core.py:136 looking up 'inc_test' in include map
DEBUG    sapphire:core.py:338 target 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\inc2\\included_file_404.html'
DEBUG    sapphire:core.py:341 404 'inc_test/included_file_404.html' (1 to go)
DEBUG    sapphire:core.py:319 check_request('test_case.html')
DEBUG    sapphire:core.py:331 expecting to finish
DEBUG    sapphire:core.py:338 target 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\root\\test_case.html'
DEBUG    sapphire:core.py:375 sending file: 5 bytes
DEBUG    sapphire:core.py:382 200 'C:\\Users\\travis\\AppData\\Local\\Temp\\pytest-of-travis\\pytest-0\\test_sapphire_170\\root\\test_case.html' (0 to go)
DEBUG    sapphire:core.py:462 shutting down and cleaning up workers
DEBUG    sphr_test:conftest.py:139 ConnectionResetError - [WinError 10054] An existing connection was forcibly closed by the remote host - line 82 (processing: inc_test/../no_access.html)

[ci] Add Grizzly to TravisCI

[sapphire] Remove 'working_path' from serve_testcase()

This is unused so let's remove it. Maybe we could solve the problem another way? The idea was to allow the use of a ramdisk to store 'wwwroot'.

[reducer] Remove file from test case as soon as it's no longer needed

We currently remove all unnecessary files at the end of reduction, Doing this as soon as possible will be helpful when a reduction fails part way and the initial test case contains many files.

KeyError: 'shortDescription' when hitting cached frequent signature

Once grizzly hits a frequent (locally cached) signature, it will throw an exception and terminate. This currently affects all instances that we are running.

The problem is in this part of the code:

cache_file, cache_signature = collector.search(crash_info)
if cache_signature is not None:
    if cache_signature['frequent']:
        log.info("Crash matched existing signature: %s", cache_signature["shortDescription"])
        return

The cache_signature variable actually does not hold the signature but the metadata for that particular signature. However, even if the signature was used, that wouldn't work because there is no shortDescription property anywhere. The closest thing we can use is crash_info.createShortDescription() which will show the short description for the particular crash that we are processing right now.

[sapphire] Handle query separator '?'

At the moment requests containing '?' are not properly handled

[sapphire] Error when calling bind on Windows

Traceback (most recent call last):
  File "grizzly\reduce\reduce.py", line 452, in run
    result = reducer.run()
  File "c:\users\user\code\fuzzmanager\lithium\src\lithium\reducer.py", line 1312, in run
    result = self.strategy.main(self.testcase, self.interesting, self.testcaseTempFilename)
  File "c:\users\user\code\fuzzmanager\lithium\src\lithium\reducer.py", line 378, in main
    result, anySingle, testcase = self.run(testcase, interesting, tempFilename)  # pylint: disable=invalid-name
  File "c:\users\user\code\fuzzmanager\lithium\src\lithium\reducer.py", line 454, in run
    if interesting(test_to_try):
  File "c:\users\user\code\fuzzmanager\lithium\src\lithium\reducer.py", line 1469, in interesting
    inter = self.conditionScript.interesting(self.conditionArgs, tempPrefix)
  File "grizzly\reduce\interesting.py", line 254, in interesting
    if self._run(testcase, run_prefix):
  File "grizzly\reduce\interesting.py", line 305, in _run
    self.server = sapphire.Sapphire()
  File "sapphire\core.py", line 221, in __init__
    self._socket = Sapphire._create_listening_socket(allow_remote, port)
  File "sapphire\core.py", line 270, in _create_listening_socket
    sock.bind(("0.0.0.0" if allow_remote else "127.0.0.1", port))
  File "C:\Python27\lib\socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions

[tests] intermittent test_sapphire_25 failure

This appears to be due to running on a slow/busy machine.

______________________________ test_sapphire_25 _______________________________

client_factory = <function _get_client at 0x0000000008554DD8>
tmp_path = WindowsPath('c:/users/travis/appdata/local/temp/pytest-of-travis/pytest-0/test_sapphire_250')

    def test_sapphire_25(client_factory, tmp_path):
        """test all request types via multiple connections"""
        def _dyn_test_cb():
            return b"A" if random.getrandbits(1) else b"AA"
    
        serv = Sapphire(timeout=10)
        default_pool_limit = Sapphire.WORKER_POOL_LIMIT
        try:
            Sapphire.WORKER_POOL_LIMIT = 10
            to_serve = list()
            for i in range(50):
                # add required files
                to_serve.append(_create_test("test_%03d.html" % i, tmp_path, data=b"A" * ((i % 2) + 1)))
                # add a missing files
                to_serve.append(_TestFile("missing_%03d.html" % i))
                # add optional files
                opt_path = tmp_path / ("opt_%03d.html" % i)
                opt_path.write_bytes(b"A" * ((i % 2) + 1))
                to_serve.append(_TestFile(opt_path.name))
                # add redirects
                redir_target = _create_test("redir_%03d.html" % i, tmp_path, data=b"AA")
                to_serve.append(_TestFile("redir_%03d" % i))
                serv.set_redirect(to_serve[-1].url, redir_target.url, required=random.getrandbits(1) > 0)
                # add dynamic responses
                to_serve.append(_TestFile("dynm_%03d" % i))
                serv.add_dynamic_response(to_serve[-1].url, _dyn_test_cb, mime_type="text/plain")
            clients = list()
            for _ in range(100):  # number of clients to spawn
                clients.append(client_factory(rx_size=1))
                throttle = 0.05 if random.getrandbits(1) else 0
                clients[-1].launch("127.0.0.1", serv.get_port(), to_serve, throttle=throttle)
>           assert serv.serve_path(str(tmp_path))[0] == SERVED_ALL
E           assert 3 == 0
E             -3
E             +0

sapphire\test_sapphire.py:616: AssertionError

[status] sqlite3.OperationalError is hit when running lots of instances

This was hit multiple times while running 24 instances of grizzly in parallel.

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/worker/grizzly-auto-run/grizzly/grizzly/__main__.py", line 13, in <module>
    sys.exit(console_main())
  File "/home/worker/grizzly-auto-run/grizzly/grizzly/main.py", line 53, in console_main
    return main(GrizzlyArgs().parse_args())
  File "/home/worker/grizzly-auto-run/grizzly/grizzly/main.py", line 158, in main
    display_mode=display_mode)
  File "/home/worker/grizzly-auto-run/grizzly/grizzly/session.py", line 42, in __init__
    self.status = Status.start()
  File "/home/worker/grizzly-auto-run/grizzly/grizzly/common/status.py", line 193, in star
t
    time_stamp INTEGER DEFAULT 0);""")
sqlite3.OperationalError: database is locked

[reducer] Relaunch should reset when less than or same as retries

Whenever we have a new call to interesting (next chunk removal), we should reset the relaunch counter if <= retries. Otherwise a relaunch could overlap a retry period, which can be problematic.

If relaunch > retries, we do overlap by choice but this should only be for issues where we're not worried about browser state being a problem between removals. (this is the default).

[tests][reducer] intermittent test_idle_timeout failure

This was seen in CI on a Windows machine.

______________________________ test_idle_timeout ______________________________

monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x00000000077C2D48>
tmp_path = WindowsPath('c:/users/travis/appdata/local/temp/pytest-of-travis/pytest-0/test_idle_timeout0')

    def test_idle_timeout(monkeypatch, tmp_path):
        "idle is not polled until idle_timeout has passed, and idle_poll results in timeout"
        now = [0]
        first_poll = []
    
        def mytime():
            now[0] += 1
            return now[0]
    
        class MyTarget(FakeTarget):
    
            def poll_for_idle(self, *args, **kwds):
                if not first_poll:
                    first_poll.append(now[0])
                FakeTarget.poll_for_idle(self, *args, **kwds)
                return time.time() >= 45
    
        class MyServer(FakeServer):
    
            def serve_testcase(self, *args, **kwds):
                continue_cb = kwds["continue_cb"]
                while continue_cb():
                    pass
                return sapphire.SERVED_ALL, []
    
        monkeypatch.setattr(time, "time", mytime)
        monkeypatch.setattr(sapphire, "Sapphire", MyServer)
    
        obj = Interesting([], MyTarget(), 30, False, False, 0, 1, 1, 10, 25, 30, FakeReduceStatus(), False)
        create_target_binary(obj.target, tmp_path)
        (tmp_path / "test.html").touch()
        obj.reduce_file = str(tmp_path / "test.html")
        obj.init(None)
        (tmp_path / "lithium0").mkdir()
        assert obj.interesting(None, tmp_path / "lithium0")
>       assert abs(first_poll[0] - 30) <= 5, "polling started at %d" % (first_poll[0],)
E       IndexError: list index out of range

grizzly\reduce\test_interesting.py:470: IndexError

[reducer] Only update quality when running with --fuzzmanager

[tests] intermittent test_sapphire_18 failure

This was seen in CI on a Windows machine running Python 2.7

______________________________ test_sapphire_18 _______________________________

    def test_sapphire_18():
        """test mapping with bad urls"""
>       serv = Sapphire(timeout=1)

sapphire\test_sapphire.py:476: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
sapphire\core.py:220: in __init__
    self._socket = Sapphire._create_listening_socket(allow_remote, port)
sapphire\core.py:269: in _create_listening_socket
    sock.bind(("0.0.0.0" if allow_remote else "127.0.0.1", port))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = 'bind', self = <socket._socketobject object at 0x000000000834FEE8>
args = (('127.0.0.1', 49675),)

    def meth(name,self,*args):
>       return getattr(self._sock,name)(*args)
E       error: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions

C:\Python27\lib\socket.py:228: error

Add data_size to TestCase

This should return the sum of the size of all the files in the testcase.