shellphish/shellphish-afl

A pip wrapper around AFL.

Shellphish AFL package

AFL packaged for easy pip installation.

Installation

First install the prerequisite packages:

  sudo apt-get install build-essential gcc-multilib debootstrap debian-archive-keyring
  sudo apt-get build-dep qemu

Then install shellphish-afl:

 pip install git+https://github.com/shellphish/shellphish-afl

shellphish-afl's Issues

Dot

Upgrade to latest AFL

Is there any plan to use the latest version of AFL?

Running setup.py install for shellphish-afl ... -Killed

Collecting git+https://github.com/shellphish/shellphish-afl.git
Cloning https://github.com/shellphish/shellphish-afl.git to /tmp/pip-h4g2IR-build
Installing collected packages: shellphish-afl
Running setup.py install for shellphish-afl ... -Killed

I wanna install rex on my computer. I follow the link in angr/rex#5, in the last step, it still report this error. I do the installation in docker angr. I repeat many times. the results are the same. right now , I don't how to do...., call for help

patches/afl-patch.diff not distributed in shellphish-afl pip

setup.py does not package patches/afl-patch.diff. This results in the following error when attempting to pip install shellphish-afl:

Downloading/unpacking shellphish-afl
  Downloading shellphish-afl-1.0.tar.gz
  Running setup.py (path:/home/singlethink/.virtualenvs/driller/build/shellphish-afl/setup.py) egg_info for package shellphish-afl

Installing collected packages: shellphish-afl
  Running setup.py install for shellphish-afl
    Setting up AFL-other-arch
    Cloning into 'bin/afl-unix'...
    error: [Errno 2] No such file or directory: 'patches/afl-patch.diff'
    Complete output from command /home/singlethink/.virtualenvs/driller/bin/python -c "import setuptools, tokenize;__file__='/home/singlethink/.virtualenvs/driller/build/shellphish-afl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-WJdyJC-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/singlethink/.virtualenvs/driller/include/site/python2.7:
    running install

running build

Setting up AFL-other-arch

Cloning into 'bin/afl-unix'...

error: [Errno 2] No such file or directory: 'patches/afl-patch.diff'

----------------------------------------
Cleaning up...
Command /home/singlethink/.virtualenvs/driller/bin/python -c "import setuptools, tokenize;__file__='/home/singlethink/.virtualenvs/driller/build/shellphish-afl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-WJdyJC-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/singlethink/.virtualenvs/driller/include/site/python2.7 failed with error code 1 in /home/singlethink/.virtualenvs/driller/build/shellphish-afl
Storing debug log for failure in /home/singlethink/.pip/pip.log

Temporary workaround

python setup.py install from a clone of the shellphish-afl repo

memfd patch not applied during pip installation from git

I'm trying to setup the shellphuzz but it appears as the pip installation does not apply the memfd patch necessary for afl qemu mode.
My Dockerfile:

FROM ubuntu:18.04
  
RUN cp /etc/apt/sources.list /etc/apt/sources.list~
RUN sed -Ei 's/^# deb-src /deb-src /' /etc/apt/sources.list
RUN apt-get update && apt-get install build-essential gcc-multilib debootstrap debian-archive-keyring libtool-bin bison python3 virtualenv git python3-dev automake -y
RUN apt-get build-dep qemu -y
  
RUN virtualenv -ppython3 /opt/venv
RUN /opt/venv/bin/pip install git+https://github.com/shellphish/driller

RUN /opt/venv/bin/pip install git+https://github.com/shellphish/shellphish-afl
RUN /opt/venv/bin/pip install git+https://github.com/shellphish/fuzzer

It runs without problems until it tries to execute afl's build_qemu_support.sh script:

  util/memfd.c:40:12: error: static declaration of 'memfd_create' follows non-static declaration
   static int memfd_create(const char *name, unsigned int flags)
              ^~~~~~~~~~~~
  In file included from /usr/include/x86_64-linux-gnu/bits/mman-linux.h:115:0,
                   from /usr/include/x86_64-linux-gnu/bits/mman.h:45,
                   from /usr/include/x86_64-linux-gnu/sys/mman.h:41,
                   from /tmp/pip-req-build-rwi9pkjk/bin/afl-unix/qemu_mode/qemu-2.10.0/include/sysemu/os-posix.h:29,
                   from /tmp/pip-req-build-rwi9pkjk/bin/afl-unix/qemu_mode/qemu-2.10.0/include/qemu/osdep.h:104,
                   from util/memfd.c:28:
  /usr/include/x86_64-linux-gnu/bits/mman-shared.h:46:5: note: previous declaration of 'memfd_create' was here
   int memfd_create (const char *__name, unsigned int __flags) __THROW;
       ^~~~~~~~~~~~
  /tmp/pip-req-build-rwi9pkjk/bin/afl-unix/qemu_mode/qemu-2.10.0/rules.mak:66: recipe for target 'util/memfd.o' failed
  make: *** [util/memfd.o] Error 1
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-rwi9pkjk/setup.py", line 123, in <module>
      scripts=['fetchlibs.sh'],
    File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/venv/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-rwi9pkjk/setup.py", line 102, in run
      self.execute(_setup_other_arch, (), msg="Setting up AFL-other-arch")
    File "/usr/lib/python3.6/distutils/cmd.py", line 335, in execute
      util.execute(func, args, msg, dry_run=self.dry_run)
    File "/usr/lib/python3.6/distutils/util.py", line 301, in execute
      func(*args)
    File "/tmp/pip-req-build-rwi9pkjk/setup.py", line 42, in _setup_other_arch
      if subprocess.check_call(['./build.sh'] + SUPPORTED_ARCHES, cwd=AFL_UNIX_INSTALL_PATH) != 0:
    File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['./build.sh', 'aarch64', 'x86_64', 'i386', 'arm', 'ppc', 'ppc64', 'mips', 'mipsel', 'mips64']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for shellphish-afl

The error code originates from the missing memfd patch. Even though the memfd.diff is included in this repo, it looks like its not patched:

  [+] Cryptographic signature on qemu-2.10.0.tar.xz checks out.
  [*] Uncompressing archive (this will take a while)...
  [+] Unpacking successful.
  [*] Configuring QEMU for ...
  [*] Applying patches...
  patching file linux-user/elfload.c
  patching file accel/tcg/cpu-exec.c
  patching file linux-user/syscall.c
  [+] Patching done.

EDIT:

when i normally build qemu support the patching looks more like this:

[*] Applying patches...
patching file linux-user/elfload.c
patching file accel/tcg/cpu-exec.c
patching file linux-user/syscall.c
patching file configure
Hunk #1 succeeded at 3855 (offset -68 lines).
patching file util/memfd.c
[+] Patching done.

an error when pip install git+https://github.com/shellphish/shellphish-afl

fakeforksrv.c:2:23: fatal error: sys/prctl.h: No such file or directory
run_via_fakeforksrv.c:3:23: fatal error: sys/prctl.h: No such file or directory
compilation terminated.
compilation terminated.
: recipe for target 'fakeforksrv' failed
make[1]: *** [fakeforksrv] Error 1
make[1]: *** Waiting for unfinished jobs....
: recipe for target 'run_via_fakeforksrv' failed
make[1]: *** [run_via_fakeforksrv] Error 1
make[1]: Leaving directory '/tmp/pip-build-GCn4rm/shellphish-afl/bin/afl-multi-cgc/fakeforksrv'
Makefile:2: recipe for target 'all' failed
make: *** [all] Error 2
error: Unable to make afl-multi-cgc

Error:python not found

ERROR: Python not found. Use --python=/path/to/python

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-req-build-elpyupcw/setup.py", line 118, in <module>
    setup(
  File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/zzr/Envs/angr/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/usr/lib/python3.8/distutils/command/install.py", line 589, in run
    self.run_command('build')
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/tmp/pip-req-build-elpyupcw/setup.py", line 102, in run
    self.execute(_setup_other_arch, (), msg="Setting up AFL-other-arch")
  File "/usr/lib/python3.8/distutils/cmd.py", line 335, in execute
    util.execute(func, args, msg, dry_run=self.dry_run)
  File "/usr/lib/python3.8/distutils/util.py", line 303, in execute
    func(*args)
  File "/tmp/pip-req-build-elpyupcw/setup.py", line 42, in _setup_other_arch
    if subprocess.check_call(['./build.sh'] + SUPPORTED_ARCHES, cwd=AFL_UNIX_INSTALL_PATH) != 0:
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./build.sh', 'aarch64', 'x86_64', 'i386', 'arm', 'ppc', 'ppc64', 'mips', 'mipsel', 'mips64']' returned non-zero exit status 1.
----------------------------------------

ERROR: Command errored out with exit status 1: /home/zzr/Envs/angr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-elpyupcw/setup.py'"'"'; file='"'"'/tmp/pip-req-build-elpyupcw/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-meyx0lsj/install-record.txt --single-version-externally-managed --compile --install-headers /home/zzr/Envs/angr/include/site/python3.8/shellphish-afl Check the logs for full command output.

another issue with shellphish-afl

$ pip install shellphish-afl
Collecting shellphish-afl
  Downloading shellphish-afl-1.0.tar.gz
Building wheels for collected packages: shellphish-afl
  Running setup.py bdist_wheel for shellphish-afl ... error
  Complete output from command /home/hexfuzz/.virtualenvs/driller+/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-raFEWP/shellphish-afl/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmps7bSp5pip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  Setting up AFL-other-arch
  Cloning into 'bin/afl-unix'...
  
error: [Errno 2] No such file or directory: 'patches/afl-patch.diff'
  
  ----------------------------------------
  Failed building wheel for shellphish-afl
  Running setup.py clean for shellphish-afl

error in installing shellphish-afl

I am trying to install this package on ubuntu 17.04

it seems that there is some issue with the keys when downloading
packages for armel. To be more specific, when executing

fetcharch armel ubuntu precise

fetchlibs.sh

It emits the following error message"

I: Retrieving InRelease 
I: Failed to retrieve InRelease
I: Retrieving Release 
I: Retrieving Release.gpg 
I: Checking Release signature
gpgv: Signature made Wed 25 Apr 2012 06:49:42 PM EDT
gpgv:                using DSA key 
gpgv: Can't check signature: No public key
E: Release signed by unknown key (key id 40976EAF437D05B5)

Doutb

How to add the repository

downlaod Error In Shellphish

It says Download ngrok then it says Download error . pls help me

Setup: Endless loop when calling 'as'

Looks like there may be an issue when installing afl with the following error:

    [-] PROGRAM ABORT : ^[[1;37mEndless loop when calling 'as' (remove '.' from your PATH)^[[1;31m
             Location : ^[[0mmain(), afl-as.c:509

    Makefile:70: recipe for target 'afl-fuzz' failed

Technically this is coming from afl, so it's certainly something they need to look into as well. The problem here is that, at least on Ubuntu, the "as" command is provided by the common package binutils. This means that compiling won't work until you remove the binutils as instruction from your path. Not sure the best way to handle this programmatically, but it's causing build failures.

Error while pip install

Tried pip install this library on ubuntu 16.04

But I got this error

  I: Retrieving InRelease
  I: Failed to retrieve InRelease
  I: Retrieving Release
  I: Retrieving Release.gpg
  I: Checking Release signature
  gpgv: WARNING: multiple signatures detected.  Only the first will be checked.
  gpgv: Signature made Sat Jul  6 09:36:55 2019 UTC using RSA key ID 2B90D010
  gpgv: Good signature from "Debian Archive Automatic Signing Key (8/jessie) <[email protected]>"
  I: Valid Release signature (key id 126C0D24BD8A2942CC7DF8AC7638D0442B90D010)
  E: Invalid Release file, no entry for main/binary-mips/Packages
  error: Unable to fetch libraries

Error while running 32 bit binary in 64 bit machine

Hi,
I experience something very wired when running my 32 binary on 64 bit machine with shellphish/fuzzer.

If I execute the command below:

/~/PycharmProjects/driller-afl/venv/bin/python /~/PycharmProjects/driller-afl/venv/bin/shellphuzz --memory none -c 1 -d 1 --length-extension 20 --driller-timeout 80 /~/PycharmProjects/driller-afl/venv/test_cases/test_new_null_pointer_32

I get the error:

�[0;36mafl-fuzz �[1;37m1.85b�[0m by <[email protected]>
�[1;32m[+] �[0mLooks like we're not running on a tty, so I'll be a bit less verbose.�[0m
�[1;32m[+] �[0mYou have 3 CPU cores and 2 runnable tasks (utilization: 67%).�[0m
�[1;32m[+] �[0mTry parallel jobs - see /usr/local/share/doc/afl/parallel_fuzzing.txt.�[0m
�[1;34m[*] �[0mChecking core_pattern...�[0m
�[1;34m[*] �[0mSetting up output directories...�[0m
�[1;34m[*] �[0mScanning '/dev/shm/work/test_new_null_pointer_32/input'...�[0m
�[1;32m[+] �[0mNo auto-generated dictionary tokens to reuse.�[0m
�[1;34m[*] �[0mCreating hard links for all input files...�[0m
�[1;34m[*] �[0mLoading extra dictionary from '/dev/shm/work/test_new_null_pointer_32/test_new_null_pointer_32.dict' (level 0)...�[0m
�[1;32m[+] �[0mLoaded 1 extra tokens, size range 19 B to 19 B.�[0m
�[1;34m[*] �[0mValidating target binary...�[0m
�[1;34m[*] �[0mAttempting dry run with 'id:000000,orig:seed-0'...�[0m
�[1;34m[*] �[0mSpinning up the fork server...�[0m

�[1;31m[-] �[0mHmm, looks like the target binary terminated before we could complete a
    handshake with the injected code. Perhaps there is a horrible bug in the
    fuzzer. Poke <[email protected]> for troubleshooting tips.
��)B�[?25h�[1;31m
[-] PROGRAM ABORT : �[1;37mFork server handshake failed�[1;31m
         Location : �[0minit_forkserver(), afl-fuzz.c:2082

However, if I run AFL through cmd with the same afl-fuzz and afl-qemu-tracer files like below:

guest@vm:~/Documents/Fuzzing$ export AFL_PATH='/~/PycharmProjects/driller-afl/venv/bin/afl-unix/tracers/i386'
guest@vm:~/Documents/Fuzzing$ /~/PycharmProjects/driller-afl/venv/bin/afl-unix/afl-fuzz  -i 'afl-master32/qemu-32-test/input' -o 'afl-master32/qemu-32-test/output' -m none -Q ./afl-master32/qemu-32-test/test_new_null_pointer_32 

everything works as excepted.

issue_driller_afl.zip

please your help.

shellphish-afl on PyPI out of sync with afl-other-arch

The version of shellphish-afl on PyPI is out of date relative to the code in afl-other-arch, so when shellphish-afl is installed with pip the patch does not apply cleanly. Updating the version of shellphish-afl on PyPI should fix the problem.

$ virtualenv shellphish-afl-venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/ben/shellphish-afl-venv/bin/python2
Also creating executable in /home/ben/shellphish-afl-venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
$ source shellphish-afl-venv/bin/activate
(shellphish-afl-venv) $ pip install shellphish-afl
Collecting shellphish-afl
  Using cached https://files.pythonhosted.org/packages/37/5b/069984ef05eed213e1c66814e364a23afd2c538965dd9b4fb88c8a858aad/shellphish-afl-1.1.tar.gz
Building wheels for collected packages: shellphish-afl
  Running setup.py bdist_wheel for shellphish-afl ... error
  Complete output from command /home/ben/shellphish-afl-venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-gXI13f/shellphish-afl/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-XZLLRq --python-tag cp27:
  running bdist_wheel
  running build
  Setting up AFL-other-arch
  Cloning into 'bin/afl-unix'...
  patching file qemu_mode/patches/elfload.diff
  Hunk #1 FAILED at 1.
  Hunk #2 FAILED at 9.
  2 out of 2 hunks FAILED -- saving rejects to file qemu_mode/patches/elfload.diff.rej
  error: Unable to apply AFL patch
  
  ----------------------------------------
  Failed building wheel for shellphish-afl
  Running setup.py clean for shellphish-afl
Failed to build shellphish-afl
...