dasec/fishy

Toolkit for Filesystem based Data Hiding Techniques.

fishy's Issues

APFS: Parse non-perfect images

Parse APFS images that aren't perfectly prepared as the Wiki entry explains. Possible solution: find first NXSB instance and count the used blocks.

Doc:Metadata Add missing documentation for metadata encryption

The new metadata encryption feature needs to be documented in:

• README
• Metadata module and class
• Documentation

APFS Timestamp Hiding Utilities

Several improvements can/have to be made to the APFS Timestamp technique:

1. The choice of which timestamps should be written to needs to be simplified and accessible from the console (right now, the code would have to be changed).

2. Instead of writing to 4 bytes, the technique should only write to 30 bits.

FAT:FileSlack: Traverse directories for info output

The fileslack subcommand traverses the directory structure and uses all found files, if a directory is given as the destination. The info switch does not respect this, but treats a directory as a file, which results in unusable and false output.

NTFS FileSlack: cant write into slackspace of file 'another'

I tried to write something to file slack of an NTFS filesystem, but got this:

$ echo "Simple test" | fishy -d utils/testfs-ntfs.dd fileslack -w -d another -m /tmp/meta.json                 
filesize:12
Traceback (most recent call last):
  File "/usr/bin/fishy", line 11, in <module>
    load_entry_point('fishy==0.1', 'console_scripts', 'fishy')()
  File "/usr/lib/python3.6/site-packages/fishy-0.1-py3.6.egg/fishy/cli.py", line 127, in main
    do_fileslack(args, device)
  File "/usr/lib/python3.6/site-packages/fishy-0.1-py3.6.egg/fishy/cli.py", line 46, in do_fileslack
    slacker.write(sys.stdin.buffer, args.destination)
  File "/usr/lib/python3.6/site-packages/fishy-0.1-py3.6.egg/fishy/file_slack.py", line 75, in write
    slack_metadata = self.fs.write(instream, filepaths)
  File "/usr/lib/python3.6/site-packages/fishy-0.1-py3.6.egg/fishy/ntfs/ntfsSlackSpace.py", line 23, in write
    m = self.slackTool.write(instream, filepaths)
  File "/usr/lib/python3.6/site-packages/fishy-0.1-py3.6.egg/fishy/ntfs/ntfsSlack.py", line 121, in write
    raise Exception("Not enough slack space")
Exception: Not enough slack space

Don't know if it matters, but the actual filesize is 4 Bytes on my system...

$ ls -l utils/fs-files/another 
-rwxr-xr-x 1 xxxxx users 4 Oct 26 20:48 utils/fs-files/another

I used the ntfs-image created by running ./create_testfs.sh

Writing to other files, like 'long_file.txt', work fine.

Info-Switch Bug

Some Info-Switches that incorporate both non-metadata and metadata variants throw exceptions after printing the information when no metadata is used.

E.g. ext4 Superblock Slack throws a "not implemented" exception.

FAT,NTFS: construct incompatibility

The construct library changed some things in their recent 2.9.X release so that our code is currently incompatible with their current version.

Maybe someone has the time to fix those incompatibilities. Meanwhile I will fix our requirements to construct in its pre 2.9 version.

NTFS:FileSlack: Make info switch consistent to FAT:FileSlack

The current info method of the ntfs.fileslack module lacks some information, that the user would need to thoughfully handle fileslack. My suggestion would be to unify the output to match the output of the fat.fileslack module.

For instance:

$ fishy -d testfs-fat12.dd fileslack -d another -m /tmp/foo -i
File: another
  Occupied in last cluster: 4
  Ram Slack: 508
  File Slack: 1536

ext4: BlockGroup class

Implement Block Group class for Data / Inode Bitmaps and Data Blocks

Info-switch requires Metadata file no matter what

ext4: Bitmap class for Inode / Data Bitmaps

APFS Superblock Slack Read bug

When reading larger amounts of hidden data, the technique reads parts of the filesystem structure instead of the hidden data (e.g. seems to happen when reading a volume superblock for the first time).

APFS: Timestamp Hiding Improvement

As of right now, Timestamp Hiding in APFS hides data in 4 bytes of the nanosecond part of the timestamps. This potentially also affects the seconds of the timestamp.

To fix: Either write to only 3 bytes of the timestamp or determine the exact amount of bits (likely 30) and use them.

Metadata:Test: write metadata encryption tests

The metadata encryption feature needs to be unit tested.

ext4:reseved_gdt_blocks use all backup-copies from first copy on | unexpected order used

FileSlack: possible overwrites in directory autoexpand feature

If a user supplies a directory plus a file in this directory as destinations for fileslack exploitation, the autoexpansion of directories could lead to multiple writes into the slack space of the same file. For instance:

$ fishy -d testfs_fat12.dd fileslack -w -m "meta.json" -d adir/afile.txt -d adir longfile.txt

would first write into adir/afile.txt, then expand adir to adir/afile.txt and then write again into the slack space of adir/afile.txt.

This is an issue in FAT fileslack implementation, but I'm not sure if the NTFS implementation is affected.

APFS hiding techniques are not working

InodeTable.getAllInodes always returns empty list. Seems like it cannot parse APFS structures correctly.

Metadata:encryption --password expects password in plaintext

Use getpass.getpass()

ext4: Data Hiding in Data Block Slack Space

ext4: Create GDT class

Implement GDT class for parsing GDT.

Metadata: Fix reading metadata

Reading metadata files via the cli tool failes since the introduction of the metadata encryption feature.

Without encryption of the metadata file:

$ echo nonsense | fishy -d testfs-fat12.dd fileslack -d onedirectory -m /tmp/foo -w
$ fishy metadata -m /tmp/foo                                                       
Traceback (most recent call last):
  File "/usr/bin/fishy", line 11, in <module>
    load_entry_point('fishy', 'console_scripts', 'fishy')()
  File "/home/matti/Seafile/Dokumente/Studium/5.Semester/PSE_-_Projekt_Systementwicklung/fishy/fishy/cli.py", line 204, in main
    do_metadata(args)
  File "/home/matti/Seafile/Dokumente/Studium/5.Semester/PSE_-_Projekt_Systementwicklung/fishy/fishy/cli.py", line 24, in do_metadata
    meta.read(args.metadata)
  File "/home/matti/Seafile/Dokumente/Studium/5.Semester/PSE_-_Projekt_Systementwicklung/fishy/fishy/metadata.py", line 211, in read
    self.metadata = json.loads(instream.read().decode("utf8"))
AttributeError: 'str' object has no attribute 'decode'

With encryption of the metadata file:

$ echo nonsense | fishy -d testfs-fat12.dd -p lol fileslack -d onedirectory -m /tmp/foo -w
$ fishy metadata -m /tmp/foo                                                              
Traceback (most recent call last):
  File "/usr/bin/fishy", line 11, in <module>
    load_entry_point('fishy', 'console_scripts', 'fishy')()
  File "/home/matti/Seafile/Dokumente/Studium/5.Semester/PSE_-_Projekt_Systementwicklung/fishy/fishy/cli.py", line 204, in main
    do_metadata(args)
  File "/home/matti/Seafile/Dokumente/Studium/5.Semester/PSE_-_Projekt_Systementwicklung/fishy/fishy/cli.py", line 24, in do_metadata
    meta.read(args.metadata)
  File "/home/matti/Seafile/Dokumente/Studium/5.Semester/PSE_-_Projekt_Systementwicklung/fishy/fishy/metadata.py", line 211, in read
    self.metadata = json.loads(instream.read().decode("utf8"))
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa6 in position 9: invalid start byte

As the unit tests for reading and writing metadata pass, there might some inconsistencies between the tests and the usage via the cli.

Licensing

We still need to decide under which license this project will be published...

Has anyone preferences?

NTFS:mftslack --seek in combination with --domirr results in wrong offset/mft record in MFT_Mirr

MFT record 1 used size: 408
MFT record 2 used size: 303
-> writes Mirror of hidden data 2 (hidden in record 2) to mirror of record 1 at position 303

ext4: Inode class

Document responsibilities

Don't know which form/which place are the best to document the responsibilities, but we must start somehow, so here some notes about what I remember. Please feel free to add stuff you or others have done.

• Framework:
  • Architecture: me
  • Metadata encryption: @D-Kern
• FAT:
  • FAT filesystem parser: me
  • Hiding Technique FileSlack : me
  • Hiding Technique Bad Cluster: me
  • Hiding Technique Additional Cluster Allocation: me
• NTFS:
• EXT4:
• Documentation:
  • Introduction Stuff: me
  • FAT Filesystem explanation: me
  • Module Overview: me

FAT: FileSlack & Additional Cluster Allocation: clear already written data if write fails

If an error occures during the write process, already written data should be cleared before the program terminates. As we don't write a metadata file in this case, currently the user is not able to wipe those fragments from disk.

NTFS: Fileslack, mftslack, bad cluster should not open filesystem streams itself

All three named hiding techniques open their filesystem streams by themself. This violates the principles, we defined earlier.
The cli already passes an open filesytem stream to the do_ function. So, they should use it.

ext4: Finish superblock class structure

Add remaining superblock fields to structure.

Failing tests for FAT and NTFS

Some of us discovered a few problems while running the unittests. Under Debian and arch linux all FAT tests pass, but the NTFS test of MFT File Records failes with:

____________________________________________________________________ TestGetRecordOfFile.test_get_record_of_file ____________________________________________________________________

self = <test_ntfs.TestGetRecordOfFile object at 0x7f78e89e5fd0>, testfs_ntfs_stable1 = ['/tmp/tmpjej3b9pj/testfs-ntfs-stable1.dd']

    def test_get_record_of_file(self, testfs_ntfs_stable1):
        """
            Tests if the correct record is returned for
            the supplied name
            """
        with open(testfs_ntfs_stable1[0], 'rb') as fs:
            ntfs = NTFS(fs)
            assert ntfs.get_record_of_file('$MFT') == 0
            assert ntfs.get_record_of_file('$MFTMirr') == 1
            assert ntfs.get_record_of_file('$BadClus') == 8
            assert ntfs.get_record_of_file('another') == 64
>           assert ntfs.get_record_of_file('onedirectory/nested_directory') == 69
E           AssertionError: assert 71 == 69
E            +  where 71 = <bound method NTFS.get_record_of_file of <fishy.ntfs.ntfs_filesystem.ntfs.NTFS object at 0x7f78e89e5748>>('onedirectory/nested_directory')
E            +    where <bound method NTFS.get_record_of_file of <fishy.ntfs.ntfs_filesystem.ntfs.NTFS object at 0x7f78e89e5748>> = <fishy.ntfs.ntfs_filesystem.ntfs.NTFS object at 0x7f78e89e5748>.get_record_of_file

tests/test_ntfs.py:69: AssertionError

Under ubuntu there are issues with the FAT tests but NTFS tests pass.

I was not able to reproduce the FAT test failures under the latest ubuntu version.

Put wrapper modules into their own directory

As suggested by the authority(tm) we should put our wrapper modules into their own directory, so the root package looks not so messy.

ext4: Parser class

ext4: Data Hiding in Reserved GDT Blocks

Implement hiding technique for Reserved GDT Blocks

Refine cli interface

The are some inconsistencies in the cli interface.

• The metadata subcommand does not require a device (-d), but all other commands do.
• The info option of fileslack subcommand should not require a metadata file

Also it might be nicer to move the -d option behind the subcommand.

The argparse configuration should require all options which are actually required and should not require options, which are not required...

Edit:

• Maybe we should seperate informational functionality (fattools, metadata) from hiding techniques via additional subgroups
• Read/write/clear/info options of a hiding technique should only be used once at a time. We could propably implement this via add_mutually_exclusive_group
• for fileslack subcommand, the -d option must be required, when writing to fileslack

Some other things are wrong or need extension in the help output:

• as the mftslack options were copied from fileslack, this keyword occures in the help output, but is wrong there.
• subcommand help output should be more descriptive

Documentation: Extend and add multiple sections

We need to add some sections to the documentation:

• Why we wrote this tool, why should our program be used, analysis of existing tools...

• Use case of our tool

  • Maybe just describing a brief scenario where one could use this tool
  • Mention that no encryption is applied to the hidden data, so the user is responsible for that.

• Document create_testfs.sh including testfs filestructure (the reason/test case for each file/directory)

• Limitations of our program (e.g. linux only)

• Evaluation of our program and hiding techniques

  • Gained Capacity for each hiding technique
  • Detection Rate with fsck and forensic analysis tools
  • stability of the hiding technique

• Extended usage, e.g. use pipes to add checksumming and encryption

• Metadata: explicitly mention that it is serialized as json

• Filesystem data structures for ntfs and ext4

• Hiding Technique documentation (nearly all)

• List all implemented hiding techiques in README.md AND doc/source/01_getting_started.rst

• Api Reference for nearly all important modules

• Development: move notes from readme to documentation and extend it

• Future work (-> turn following bullet points into continuous text!)

  • Use a more secure clearing technique for hiding techniques -c option
  • Add possibility to hide multiple files
  • Implement a fuse filesystem, which uses multiple hiding techniques to store data
  • Maybe add encryption feature
  • Hide Metadata files itself
  • more reliable filesystem autodetection (FAT and NTFS detection only check for an ASCII string in bootsector)

• Extend "abstract"

• include documentation structure (01_overview.rst)

• Turn most bullet points into text

• Make sphinx size included images to a format that fits on page for the pdf version.

• For all:

  • Please check all written stuff for consistency. Many eyes see more than just one two.
  • Add your name to the author variable in fishy/doc/source/conf.py (if you want), so that it will appear on the documentation.
  • Check if all of your work is mentioned in responsibilities (doc/source/07.2_responsibilities.rst)

APFS: Optimize Inode-centric techniques for larger amounts of inodes

e.g. calculate the amount of inodes needed and only extract that amount instead of all inodes (might break info switches - need to be changed as well)