mozillasecurity/fuzzfetch

Downloader for Firefox/jsshell builds for fuzzing.

fuzzfetch's Issues

Traceback due to incomplete download

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/user/code/fuzzfetch/src/fuzzfetch/__main__.py", line 9, in <module>
    Fetcher.main()
  File "/home/user/code/fuzzfetch/src/fuzzfetch/fetch.py", line 680, in main
    obj.extract_build(out_tmp, tests=extract_args['tests'], full_symbols=extract_args['full_symbols'])
  File "/home/user/code/fuzzfetch/src/fuzzfetch/fetch.py", line 377, in extract_build
    self.extract_tar(path)
  File "/home/user/code/fuzzfetch/src/fuzzfetch/fetch.py", line 513, in extract_tar
    for member in tar.getmembers():
  File "/usr/lib/python2.7/tarfile.py", line 1836, in getmembers
    self._load()        # all members, we first have to
  File "/usr/lib/python2.7/tarfile.py", line 2417, in _load
    tarinfo = self.next()
  File "/usr/lib/python2.7/tarfile.py", line 2348, in next
    self.fileobj.seek(self.offset - 1)
EOFError: compressed file ended before the logical end-of-stream was detected

Support all platforms.

Need to add support for win32/64 & linux32 builds.

DOMFuzz may be useful as a reference.

This should also include overriding which platform you want (eg. downloading 32-bit on a 64-bit platform).

Add buildflags field to fuzzmanager metadata

Add a buildflags field to the fuzzmanager metadata and populate it with the actual build flag.

I.e. debug == --enable-debug

test_nearest_retrieval is flaky

Updating mock fixtures for test_nearest_retrieval is too difficult. I added a skip for now.

target.common.tetss.zip not found

This used to work correctly:

fuzzfetch -a --fuzzing -n ~/build --tests common reftests gtest
[2018-08-14 07:56:49] Identified task: https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-fuzzing-asan-opt
[2018-08-14 07:56:49] > Task ID: Y8DJ8m1FTMC2SZz8cntzlA
[2018-08-14 07:56:49] > Rank: 1534239069
[2018-08-14 07:56:49] > Changeset: 914b3b370ad059a04ad751642b74e013f8e3ad08
[2018-08-14 07:56:49] > Build ID: 20180814093109
[2018-08-14 07:56:49] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/Y8DJ8m1FTMC2SZz8cntzlA/artifacts/public/build/target.tar.bz2 ..
[2018-08-14 07:57:49] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/Y8DJ8m1FTMC2SZz8cntzlA/artifacts/public/build/target.common.tests.zip ..
Traceback (most recent call last):
File "/usr/local/bin/fuzzfetch", line 11, in
load_entry_point('fuzzfetch==0.6.0', 'console_scripts', 'fuzzfetch')()
File "build/bdist.linux-x86_64/egg/fuzzfetch/fetch.py", line 755, in main
File "build/bdist.linux-x86_64/egg/fuzzfetch/fetch.py", line 448, in extract_build
File "build/bdist.linux-x86_64/egg/fuzzfetch/fetch.py", line 553, in extract_zip
File "build/bdist.linux-x86_64/egg/fuzzfetch/fetch.py", line 74, in _get_url
fuzzfetch.fetch.FetcherException: 404 Client Error: Not Found for url: https://queue.taskcluster.net/v1/task/Y8DJ8m1FTMC2SZz8cntzlA/artifacts/public/build/target.common.tests.zip

Removing common:
fuzzfetch -a --fuzzing -n ~/build --tests reftests gtest
[2018-08-14 08:46:52] Identified task: https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-fuzzing-asan-opt
[2018-08-14 08:46:52] > Task ID: Y8DJ8m1FTMC2SZz8cntzlA
[2018-08-14 08:46:52] > Rank: 1534239069
[2018-08-14 08:46:52] > Changeset: 914b3b370ad059a04ad751642b74e013f8e3ad08
[2018-08-14 08:46:52] > Build ID: 20180814093109
[2018-08-14 08:46:52] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/Y8DJ8m1FTMC2SZz8cntzlA/artifacts/public/build/target.tar.bz2 ..
[2018-08-14 08:47:52] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/Y8DJ8m1FTMC2SZz8cntzlA/artifacts/public/build/target.reftest.tests.zip ..
Traceback (most recent call last):
File "/usr/local/bin/fuzzfetch", line 11, in
load_entry_point('fuzzfetch==0.6.0', 'console_scripts', 'fuzzfetch')()
File "build/bdist.linux-x86_64/egg/fuzzfetch/fetch.py", line 755, in main
File "build/bdist.linux-x86_64/egg/fuzzfetch/fetch.py", line 450, in extract_build
File "build/bdist.linux-x86_64/egg/fuzzfetch/fetch.py", line 553, in extract_zip
File "build/bdist.linux-x86_64/egg/fuzzfetch/fetch.py", line 74, in _get_url
fuzzfetch.fetch.FetcherException: 404 Client Error: Not Found for url: https://queue.taskcluster.net/v1/task/Y8DJ8m1FTMC2SZz8cntzlA/artifacts/public/build/target.reftest.tests.zip

Removing reftests and common works properly:
fuzzfetch -a --fuzzing -n ~/build --tests gtest
[2018-08-14 08:48:43] Identified task: https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-fuzzing-asan-opt
[2018-08-14 08:48:43] > Task ID: Y8DJ8m1FTMC2SZz8cntzlA
[2018-08-14 08:48:43] > Rank: 1534239069
[2018-08-14 08:48:43] > Changeset: 914b3b370ad059a04ad751642b74e013f8e3ad08
[2018-08-14 08:48:43] > Build ID: 20180814093109
[2018-08-14 08:48:43] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/Y8DJ8m1FTMC2SZz8cntzlA/artifacts/public/build/target.tar.bz2 ..
[2018-08-14 08:49:43] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/Y8DJ8m1FTMC2SZz8cntzlA/artifacts/public/build/target.gtest.tests.zip ..

can't download opt coverage build on Windows

fuzzfetch fails to pull down the coverage build when trying from windows. If you specify debug using -d then it will pull down a debug build.

Add support for spidermonkey-only builds

eg. https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.latest.firefox/sm-fuzzing-linux64

These are packaged very differently.

Use 7z for decompression if available.

Using 7z instead of libbz2 is about twice as fast due to multi-threading. We should use 7z in a sub-process if it is installed.

No such file or directory: u'gtest/dependentlibs.list.gtest'

fuzzfetch -d -a -n /build --tests common reftests gtest

[2017-09-06 23:05:22] Identified task: https://index.taskcluster.net/v1//task/gecko.v2.mozilla-central.latest.firefox.linux64-asan-debug
[2017-09-06 23:05:22] > Task ID: JWUeT0H2SyO5Jm7N_-BOug
[2017-09-06 23:05:22] > Rank: 1504719482
[2017-09-06 23:05:22] > Changeset: 93dd2e456c0ecca00fb4d28744e88078a77deaf7
[2017-09-06 23:05:22] > Build ID: 20170906173802
[2017-09-06 23:05:22] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/JWUeT0H2SyO5Jm7N_-BOug/artifacts/public/build/target.tar.bz2 ..
[2017-09-06 23:07:07] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/JWUeT0H2SyO5Jm7N_-BOug/artifacts/public/build/target.common.tests.zip ..
[2017-09-06 23:07:16] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/JWUeT0H2SyO5Jm7N_-BOug/artifacts/public/build/target.reftest.tests.zip ..
[2017-09-06 23:07:24] > Downloading and extracting archive: https://queue.taskcluster.net/v1/task/JWUeT0H2SyO5Jm7N_-BOug/artifacts/public/build/target.gtest.tests.zip ..
Traceback (most recent call last):
File "/usr/local/bin/fuzzfetch", line 9, in
load_entry_point('fuzzfetch==0.5.3', 'console_scripts', 'fuzzfetch')()
File "/usr/local/lib/python2.7/dist-packages/fuzzfetch/fetch.py", line 629, in main
obj.extract_build(out_tmp, tests=extract_args['tests'], full_symbols=extract_args['full_symbols'])
File "/usr/local/lib/python2.7/dist-packages/fuzzfetch/fetch.py", line 368, in extract_build
os.path.join(path, 'dependentlibs.list.gtest'))
File "/usr/lib/python2.7/shutil.py", line 119, in copy
copyfile(src, dst)
File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'gtest/dependentlibs.list.gtest'

Support passing a BuildID to `--build`

eg. 20210429122950

This should be easy to add since they are already listed this way under pushdate (https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.pushdate.2021.04.29.20210429122950.firefox)

Declare fuzzfetch to have types

py.typed should be in an empty file in src/fuzzfetch since @jschwartzentruber added type information in rev 48d367c.

setup.cfg should also have py.typed mentioned in package_data under [options].

See https://www.python.org/dev/peps/pep-0561/#packaging-type-information

fuzzfetch-0.5.4.tar.gz does not contain requirements.txt

pip install fuzzfetch
Collecting fuzzfetch
Downloading fuzzfetch-0.5.4.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-33YAft/fuzzfetch/setup.py", line 28, in
install_requires=open('requirements.txt').read().strip().splitlines(),
IOError: [Errno 2] No such file or directory: 'requirements.txt'

tar zxvf fuzzfetch-0.5.4.tar.gz
fuzzfetch-0.5.4/
fuzzfetch-0.5.4/src/
fuzzfetch-0.5.4/src/fuzzfetch/
fuzzfetch-0.5.4/src/fuzzfetch/init.py
fuzzfetch-0.5.4/src/fuzzfetch/main.py
fuzzfetch-0.5.4/src/fuzzfetch/fetch.py
fuzzfetch-0.5.4/src/fuzzfetch.egg-info/
fuzzfetch-0.5.4/src/fuzzfetch.egg-info/PKG-INFO
fuzzfetch-0.5.4/src/fuzzfetch.egg-info/SOURCES.txt
fuzzfetch-0.5.4/src/fuzzfetch.egg-info/dependency_links.txt
fuzzfetch-0.5.4/src/fuzzfetch.egg-info/entry_points.txt
fuzzfetch-0.5.4/src/fuzzfetch.egg-info/requires.txt
fuzzfetch-0.5.4/src/fuzzfetch.egg-info/top_level.txt
fuzzfetch-0.5.4/README.md
fuzzfetch-0.5.4/setup.cfg
fuzzfetch-0.5.4/setup.py
fuzzfetch-0.5.4/PKG-INFO

Automatically download minidump_stackwalk

When downloading a build with fuzzfetch it would be nice if the appropriate minidump_stackwalk executable was also downloaded for that build

Validate builds using target.checksums

Validate builds after download via TC's target.checksum.

Add support for --disable-optimize builds.

bug 1681209 is adding support for -noopt builds in TC. We should support downloading these.

Example namespaces:

• gecko.v2.try.revision.02baaa68985d57dfc1aec5488fa2507080a135eb.firefox.linux64-fuzzing-asan-noopt
• gecko.v2.try.revision.02baaa68985d57dfc1aec5488fa2507080a135eb.firefox.linux64-fuzzing-noopt-debug

Passing the full namespace to --build already works, but the folder name for the asan build is wrong:

try-20210513204114-fuzzing-asan-opt
try-20210513204114-fuzzing-noopt-debug

We need to add --noopt (or --no-opt?) as a command-line option and a BuildFlags entry.

Warn if 7z is not installed

Using 7z is much faster, we should notify users when it is not installed.

os.symlink not available on windows/py2

On py3 os.symlink is available, but may require administrator.

Symlinks are used to support domfuzz's historical layout. Domfuzz should support the newer layout and then this can be dropped.

Gtest also uses a symlink for the dependentlibs.list.gtest. We can probably copy this instead.

This is causing some pylint warnings on windows only.

Emit version information

Add argument to emit version information.

Windows ASAN opt builds are old

Both fuzzfetch -a and fuzzfetch --inbound -a return old builds on win64.

https://ci.appveyor.com/project/MozillaSecurity/fuzzfetch/build/1.0.5/job/kjs7wslmyyn5894k

Modify --target to accept a list of arguments

Currently, fuzzfetch can only be used to download a single target at a time. As suggested by @jschwartzentruber, it would be convenient to accept a list of args (i.e. firefox, js, gtest, common, etc).

Support searchfox data

It would be good to have the ability to download searchfox data, for example this task:

https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.pushdate.2023.06.01.20230601042516.firefox/linux64-searchfox-debug

What we need from this task:

• target.mozsearch-index.zip: Contains the actual searchfox data files
• target.mozsearch-distinclude.map: A mapping for headers
• target-generated-files.tar.gz: All of the generated source files. These are not specific to searchfox, but it's very handy to get them here because when working with the searchfox data, it is mostly about source analysis and you quickly run into the situation where you have the regular source tree present but lack the generated sources for completeness.

The actual build from that isn't needed as far as I can tell.

This data exists on multiple OSes and should be supported as such, but afaik these are all debug configurations.

Link to "bin" points to parent on MacOS

lrwxr-xr-x 1 posidron staff 2 Jun 1 23:03 bin -> ..

Nightly.app/Contents/MacOS contains the stuff but 'bin' in 'dist' is linked to the parent.

fuzzfetch.FetcherException: Unknown machine for Darwin: arm64

Encountered this on an M1 Max when trying to follow the command-line instructions at https://bugzilla.mozilla.org/show_bug.cgi?id=1798769#c0. Specfically, the command

python -m fuzzfetch --build 2db9822e6dd3 --debug --fuzzing -n firefox

results in this error.

Add build string to fuzzmanagerconf

Add the build string to firefox.fuzzmanagerconf so that we can identify the build later.

No argument to specify 32-bit download on 64-bit platform.

We need to add an argument to Fetcher & command-line flags to specify downloading win32 on win64 or linux on linux64.

Fetcher.build_datetime should return a tz aware datetime object

datetime.strptime() returns a naive datetime object by default. TaskCluster uses UTC, we should make that explicit.

Fuzzing debug builds no longer include crashreporter-symbols.zip

With the recent change[1] to fuzzing-debug builds crashreporter-symbols.zip is no longer built. This breaks the downloads.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1649062

MacOS Build download broken

$ python -m fuzzfetch --fuzzing -a -o ~/workspace/browsers/
[2019-08-06 19:38:20] Identified task: https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.firefox.macosx64-fuzzing-asan-opt
[2019-08-06 19:38:20] > Task ID: SXe6uN7XRKaS2y6UUNFLLg
[2019-08-06 19:38:20] > Rank: 1565127812
[2019-08-06 19:38:20] > Changeset: fb699b3c084c8d35e52f2b282de90ecb7b0992cd
[2019-08-06 19:38:20] > Build ID: 20190806214332
[2019-08-06 19:38:20] > Downloading: https://queue.taskcluster.net/v1/task/SXe6uN7XRKaS2y6UUNFLLg/artifacts/public/build/target.dmg (143.12MB total)
[2019-08-06 19:38:27] .. downloaded (18.66MB/s)
[2019-08-06 19:38:27] .. extracting
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/tsmith2/code/fuzzfetch/src/fuzzfetch/__main__.py", line 10, in <module>
    Fetcher.main()
  File "/Users/tsmith2/code/fuzzfetch/src/fuzzfetch/fetch.py", line 862, in main
    obj.extract_build(out, tests=extract_args['tests'], full_symbols=extract_args['full_symbols'])
  File "/Users/tsmith2/code/fuzzfetch/src/fuzzfetch/fetch.py", line 568, in extract_build
    self._write_fuzzmanagerconf(path)
  File "/Users/tsmith2/code/fuzzfetch/src/fuzzfetch/fetch.py", line 604, in _write_fuzzmanagerconf
    assert len(ff_loc) == 1
AssertionError

TaskCluster should be mocked for unittests.

Currently we try fetching metadata for all supported builds from production TaskCluster.

We should mirror the requests made and commit them in so we can mock them with the built-in HTTP server implementation. We should also add stripped down data files (zip/tar.bz2/dmg) with 0-length contents to test extraction.

Python exception "ImportError: cannot import name ConverterMapping"

Not a big deal but following the tutorial https://github.com/MozillaSecurity/grizzly/wiki/Getting-Started

I get:

% python2 -m fuzzfetch -a -n firefox --fuzzing -o $CODE/browsers/
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 163, in _run_module_as_main
    mod_name, _Error)
  File "/usr/lib/python2.7/runpy.py", line 111, in _get_module_details
    __import__(mod_name)  # Do not catch exceptions initializing package
  File "/home/sylvestre/dev/mozilla/fuzzfetch/src/fuzzfetch/__init__.py", line 7, in <module>
    from .fetch import *  # noqa pylint: disable=wildcard-import
  File "/home/sylvestre/dev/mozilla/fuzzfetch/src/fuzzfetch/fetch.py", line 28, in <module>
    import configparser  # pylint: disable=wrong-import-order
  File "/home/sylvestre/.local/lib/python2.7/site-packages/configparser.py", line 11, in <module>
    from backports.configparser import (
ImportError: cannot import name ConverterMapping

Removing the Debian package was enough to fix the issue

sudo apt remove python-configparser                            

(but remove flake8 too)

xpcshell binary is extracted in the wrong location.

xpcshell is contained in common.tests.zip under the common folder, but it needs to be next to the firefox binary for it to find some locale data settings. We should move it after extraction.

It also needs xpcshell.fuzzmanagerconf created for FuzzManager.

Use taskcluster client for API access.

Instead of using requests to navigate Taskcluster, we should use the actual Python API for it:

https://github.com/taskcluster/taskcluster/tree/master/clients/client-py#readme

Better error message for network failure

Exception for network errors and invalid builds is currently the same.

fuzzfetch.FetcherException: Unable to find usable archive...

This makes it unclear if invalid build is being requested or something else is wrong.

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

Fixate linter versions

We'll need to fixate linter versions to prevent new linter checks from failing our CI tests if we are not really ready for them.

`symbols` directory is not placed next to Firefox binary on MacOS

In automation we expect the symbols directory to be unpacked in to the same directory as the Firefox binary