nyxxxie/serenity

Universal guided editor and reverse engineering tool for binary data.

serenity's Issues

class Project should take care of adding, opening, closing and removing files

Support Kaitai Struct?

I've had my eye on Kaitai Struct for awhile now. Basically it attempts to achieve what serenity's template engine does. I like the idea of relying on a project that someone else maintains, but it is unclear how they will integrate with serenity. I'd like to explore it and see if they're worth supporting or even replacing the existing template system.

Modify editorwidget to cycle applicable data views via spacebar press

editorwidget should be modified to switch the editor currently being used to view the file with the spacebar key. User should be able to go backwards too with shift+spacebar.

Please review current DB schema draft and decide on a way for File and FileView to access "changes" and "files" tables in the database

Current DB schema draft for "files" and "changes" might not have things you will agree on so please review it and express your opinion on it. Other than that, at the moment we access sqlite connection using Project.db member variable, is this going to stay this way or will it be hidden behind some methods?

test issue

Import

This is likely a far ways off from being ready to work on, but I want to somehow allow importing the reversing info for one file into the project of another. Say we're working on reversing a library or something separately, and we find out that library is imported by another target we're reversing. It'd be nice to have that information about the library handy.

Again, this is probably a feature that'll have to be started later when bae is integrated in the project and the binary reversing workflow is solid, just wanted to make a note now.

Create makefile

Create a makefile for easier interfacing with spade by package managers and users. Define the following targets:

• all (AKA first target) - Builds native executable
• unix - builds frozen executable for spade
• win32 - builds frozen executable for win32
• osx - builds frozen executable for osx
• install - installs spade
• virtualenv - creates and activates virtualenv
• venv - same thing as virtualenv
• docs - builds sphinx docs
• help - displays help

See if there's a system or standard that already exists for this? Also, the packaging/install method might be subject to change between this issue and when the feature is going to be implemented, so check how install is currently done (as of 5/20/2017, dependent on #3) before implementing.

what is the harm?

Create addon system

Create addon system that will mirror the main install file structure. This will allow users to package templates, types, and scripts for other users to use. If you want to implement this, you should probably contact me (otherwise I'll do it later and make issues for specific features in the addon system)

Add pylint rules

Pylint is pretty cool and will help to automate the process of people checking to make sure their code style conforms with this project's. We should add rules to make this happen and tell people they have to run it prior to submitting code.

changes_comments should have compound foreign key (file_id, hash)

There can exist same change hash for two or more different files. Example:

0. you open two files with exactly same content
1. their base changes are both 0
2. erase first 4 bytes in both
3. change hash is sha256(base + file_pos + change_type + data)
4. get collision

Create basic UI

Create basic framework for a ui. Basically mimic what mycroft has right now where there's a menubar, dock widget area, and main editor area.

Add menu bar to ui

The menubar is that little bit on the top of a window that contains [ File | Edit | Windows | ... ]. We should add one that contains the following.

• File
  • New - creates new file in current project
  • Open - opens existing file and adds it to current project
  • vertical line
  • Save - Saves currently focused file
  • Save As... - Saves currently focused file as
  • Save All - Save all opened files
  • vertical line
  • Quit - Exits program
• Edit
  • Undo - Undoes last change
  • Redo - Redoes last change
  • vertical line
  • Cut - Redoes last change
  • Copy - Redoes last change
  • Paste - Redoes last change
• Windows
  • Somewhere in here should be a submenu that is populated dynamically somehow by all dockwindows currently added to mainwindow. Clicking on a window should hide it

Implementation for specific menu actions is optional, the implementer can just use actions with empty bodies if they wish

PRAGMA foreign_key = ON; doesn't make cascade deletes work for whatever reason

Create a template view widget

In order to visualize and edit #20 in the gui, create a gui to view their tree structure. Feature #20 must be complete before work on this feature can begin.

Implement File history

It is desired that File (see #26) should be able to store a history of its changes so that a user may undo/redo changes they may have made. This is being implemented at the library level since the spade library is primarily designed to be used with another client editor application (means less code for client to write and allows for the client to be remote). Simple linear history model may be implemented, but a tree-based model is preferred.

Create typing system

We need a way to define types to be used with the template system. This feature should be implemented in two parts:

• Type: Provides to_string/from_string transforms and indication of size in memory. Uniquly identified by a string (EG: "int32_t").
• Typesystem: Manages and stores types. Ensures there are no duplicates and allows for easy access via a for loop or type name lookup.

In addition, we should provide certain builtin types off the bat (byte, char, etc). These should be placed in their own directory in the project called types.

Create markable/content-based hex editor scroll bar

Create a scroll bar that allows for marks to be added to and removed from it. For example, this scrollbar would allow us to mark the position on the scrollbar where a search has found results. It would also be neat if we made a scroll bar that showed the colors of struct-defined areas in a file.

• existing impl: https://freddie.witherden.org/tools/libqmarkedscrollbar/doxygen/class_marked_scroll_bar.html

Implement File api

Because we want to add in the ability to store file history, file-specific metadata, etc, we'll need to make a custom file wrapper. This feature is currently being worked on by @fst3a, this issue is meant as a way to formalize it and provide a forum for discussion.

Based on @fst3a and I's private conversations, I propose we implement file as follows:
class File (or SFile or _File):

• read(), size(), etc (traditional file methods) implemented in a backwards compatible manner.
• insert(), replace(), and erase() added in addition to write() (write() will just call one of the previous 3 methods). This will provide a better interface to an eventual change history feature.
• Handle concurrency issues with something similar to a shared-exclusive lock. This thread might be relevant to this feature.
• Provide ability to link with project. This should be enforced for now (file must specify project in initializer), but it would be neat if this was optional in the future.

This will serve as a base implementation of file, with other features (change history, FileView, caching, mmaping) being implemented in other features.

Replace flags system in client with gflags

Google's flag library system is amazing, let's use it. https://github.com/google/python-gflags

Create templating system

Template files are C struct-like definitions of a target file's internal structure. Templates should be parsed and made accessable to users in such a way that they can be interacted with like a tree. An example of a template file would look similar to this:

struct struct1 {
    int32_t attribute1;
    int16_t attribute2; // this is a comment
};

struct struct2 {
    char data[32];
};

/* Entry point */
struct FILE {
    int blah;
    struct1 s1;
    struct2 s2;
};

The bare minimum implementation would parse a template file and allow it to be interacted with by the following means:

• Return Field associated with an offset into the file. Fields should also be capable of representing arrays in template files.
• Return Struct or Field associated with a string identifer (EG: template.get("FILE.header.signature") = Field object that marks the signature field in the header of the target file)
• Return entry point (FILE struct in template) of template and allow user to explore file in tree format.

The Field and Struct objects are defined as follows:

• Struct: Object that represents a struct in a template file. Should be able to return parent struct, fields/structs that are members of this struct, offset in file, and size.
• Field: Object that represents a variable (or data) in the target file. Should be able to return parent struct, type (this should be linked to typesystem, but can just return a string for now), and size. Additionally, it should provide some way to access info about the template array if it is defined as one.

Some features that'd be nice to have but aren't required for this issue (we can make them into seperate issues later):

• Ability to edit template (insert new struct instances of a given definition, define new structs, etc)
• Ability to store template in project database
• Ability to define array size as a member of a struct (EG: char member[size] or char member[FILE.header.size])
• Ability to "decompile" template back to a template file.

Change project data storage to something non-binary

Storing project data in a non-binary format (e.g. xml, json, etc) would allow for it to be friendly to version control software like git, enabling users to collaborate with other reverse engineers in their work.

Currently the project class stores all data in a sqlite database, which is great for performance and ease of query but is stored in a binary format that isn't friendly to git. I think an ideal solution would be to invent a project format that can be distributed as one compressed archive file and then decompressed into a collection of multiple files containing various pieces of data. Some of those files could be binary but serve as a caching mechanism and could be regenerated from info committed to version control.

The exact way this feature will be manifested will become apparent post-v1 when spade is usable and we have an idea how quickly the sqlite database expands.

Implement FileView api

It is desired that File (see #26) should be able to allow a user to work with a file at a particular point in time while. This should be implemented with File history (see #27), and should offer a read-only view to a file (depending on what history model is used)

Add unit testing capability

Unit tests are a great way to both develop and check a codebase for potential defects that may unexpectedly arise due to changes in other files. It would be nice if we adopted a unit testing methodoloy similar to other python projects (have a directory in project root called "tests", run tests from setup.py).

class Project should provide getter for db connection instead of direct access

Implement projectview widget

This widget should display a tree showing project(s) and their currently open files, key regions for files, templates associated with files, etc. Make sure that this can be easily modified so that we can display new items on any tree level.

Create config system

It would be cool to have a config library that allows us to get user defined values to use in program features. Implement a class in core that provides an API to do this, and saves/loads values from a yaml settings file. Default config file values should be stored in code somehow, preferably defined inside of modules that make use of them (for example, register config variables inside of hexeditor.py if any of them relate directly to it). Saved yaml files should override these default values.

Create install method

Lib should be able to install itself and its default directories + default contents. Create the means for this to happen. Probably will just amount to 30 minutes of fiddling with setup.py

Improve contributing docs

Make the current contributing.md into an overview of the development lifecycle for a feature, and point detailed stuff to docs in docs folder or more detailed sections. Also, determine if we want to actually use the issue tracker for non-bugs. Maybe, just use it for requests and use pull requests for features in development (check out how other people do it)

Life of a Feature

• Issue raised (person proposes an idea they want to see, possibly also implement)
• WIP Pull Request Opened, referencing issue
• Feature Developed, Tests Created, Pylinted, etc
• Feature Reviewed (tests run, pylint verified, merge-readiness confirmed)
• Feature Integrated, Issue Closed!

Implement project api

Project should be able to save and load its state, create a database for storing information on itself, open files, templates, and anything else, and also deliver a list of files and templates it is tracking.

Create CONTRIBUTING.md document

This document should outline what kinds of submissions are welcome, development practices and expectations, coding styles, etc. Move any complex guides to their own files in docs folder (for example, the process of creating and running tests?)

Enable VACUUM

VACUUM must be enabled for project db or you will get fragging and other bad stuff

Improve core testing

The amount and quality of testing we do for SFile and Project is pretty pathetic. We have pretty low coverage and mix the two modules' tests together. This situation should be fixed.

Add disassembly view

We want to be able to display a series of bytes as disassembly (preferably with support for metadata tables, xrefs, and other built in data tables). There should be a view to do this. View should support selecting code, displaying address, bytes, and instructions (with comments), and highlighting. It's reccomended that a custom widget be created similar to the hex editor so that things like flowgraphs can be added later. It's also reccomended that the output of capstone be tokenized for the editor somehow.

For now, just use capstone on an ordinary file containing x86 assembly to test the widget, when #20 and #21 are done we can modify it to use the spade file api.

Integrate hex editor with file api

Hex editor was developed using the normal python file api, modify it to use the spade file api. Make a qt slot that sets the file to display.

Rename to Serenity

I was on the train today and thought that'd make a much better name than spade. This issue will involve changing the repository name, url, and anything internal that relies in the name "spade". If you're looking at this and interested in following the project, be aware this change will take place in the next few weeks after posting this.

Add Python REPL widget

Add a Python REPL to the QT ui that we can use to interact with the project and program live with. Would be nice if many of the main objects are already imported and ready for our use.

Modify typesystem to support dynamic-sized types.

Some types might determine their size at runtime based on actual data. We should support those types. This depends on #21 being completed.