codilime/veles

Binary data analysis and visualization tool

veles's Issues

"Find" doesn't find overlapping matches

Hex view paint event is slow

We should improve hex view performance, which isn't perfect even on Release builds.

An idea for optimization: instead of drawing background for every byte separately we can do one call to fillRect for contiguous bytes with the same background.

Also, we should do some profiling to see what's really slow.

Hex Edit font on macOS is not fixed-width

Rename class 'NodeWidget' to something more appropriate

CPack fails if build directory is not inside source dir (macOS only?)

$ cpack -D CPACK_PACKAGE_FILE_NAME=veles-osx -G "DragNDrop" -C RelWithDebInfo
CPack: Create package using DragNDrop
CPack: Install projects
CPack: - Run preinstall target for: VELES
CPack: - Install project: VELES
CPack: - Install component: Unspecified
Scanning dependencies of target server
[ 33%] Copying server script
[ 33%] Creating veles python virtual environment
[ 66%] Installing veles python lib

Installed /Users/test/veles-build-RelWithDebInfo/python/.eggs/pbr-3.1.1-py3.6.egg
ERROR:root:Error parsing
Traceback (most recent call last):
File "/Users/test/veles-build-RelWithDebInfo/python/.eggs/pbr-3.1.1-py3.6.egg/pbr/core.py", line 111, in pbr
attrs = util.cfg_to_args(path, dist.script_args)
File "/Users/test/veles-build-RelWithDebInfo/python/.eggs/pbr-3.1.1-py3.6.egg/pbr/util.py", line 249, in cfg_to_args
pbr.hooks.setup_hook(config)
File "/Users/test/veles-build-RelWithDebInfo/python/.eggs/pbr-3.1.1-py3.6.egg/pbr/hooks/init.py", line 25, in setup_hook
metadata_config.run()
File "/Users/test/veles-build-RelWithDebInfo/python/.eggs/pbr-3.1.1-py3.6.egg/pbr/hooks/base.py", line 27, in run
self.hook()
File "/Users/test/veles-build-RelWithDebInfo/python/.eggs/pbr-3.1.1-py3.6.egg/pbr/hooks/metadata.py", line 26, in hook
self.config['name'], self.config.get('version', None))
File "/Users/test/veles-build-RelWithDebInfo/python/.eggs/pbr-3.1.1-py3.6.egg/pbr/packaging.py", line 755, in get_version
name=package_name))
Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name veles was given, but was not able to be found.
error in setup command: Error parsing /Users/test/veles-build-RelWithDebInfo/python/setup.cfg: Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name veles was given, but was not able to be found.
make[3]: *** [veles-server/veleslib] Error 1
make[2]: *** [CMakeFiles/server.dir/all] Error 2
make[1]: *** [CMakeFiles/server.dir/rule] Error 2
make: *** [server] Error 2
CPack: - Install component: application
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: /Users/test/Qt/5.7/clang_64/lib found in: /Users/test/veles-build-RelWithDebInfo/_CPack_Packages/Darwin/DragNDrop/veles-osx/ALL_IN_ONE/./veles.app/Contents/MacOS/veles (for architecture x86_64), required for specified option "-delete_rpath /Users/test/Qt/5.7/clang_64/lib"
CPack: - Install component: server
CPack: Create package
CPack: - package: /Users/test/veles-build-RelWithDebInfo/veles-osx.dmg generated.

Element under mouse is not highlighted in dark theme in Database tab

To see the difference you can compare behavior of Database tab in white and dark themes.

Dark theme should be the default one

IMO it looks much better, especially when considering the hex view.

Items in "Database" tab should be sorted

We should sort them according to insertion time or last change time.

Bitmap reader for Hex Edit

It would be nice to implement a hide-able sidebar which allows a user to read bytes under the cursor as pixels and display them as an image.

Requirements:

• Has to be very fast (moving the cursor should not lag when this sidebar is open).
• Bits per channel should be selectable.
• Channels + their order should be selectable (we should support RGBA, ARGB, RGB, BGR, BW etc.).
• The feature from the previous point should have a heuristic for default value (useful property: for n channels with m bytes per channel data tend to repeat every n*m bytes).
• Number of columns (x-size of the bitmap) should be selectable and easy to iterate over to find the good one.
• Number of columns should have a good heuristic for the default value (example idea: iterate over all possible x-sizes and pick the one with the smallest vertical pixel value delta - this is ofc O(bytes_cnt^2), so we should find something faster).

For an example implementation of this feature one can check out Hexplorer's Pixel View (which is a nightmare to use, so don't copy ideas from it ;)
https://a.fsdn.com/con/app/proj/hexplorer/screenshots/56698.jpg/1

"Find" should start searching at cursor

Fix Windows installer

Few things needs to be fixed:

• MSVC Redistributable should be installed with /quiet switch
• "Create shortcut on desktop" should be checked by default

Handle two open instances of Veles gracefully.

macOS build sometimes fails to set RPATH

For some reason (race cond somewhere?) macOS builds in Jenkins generate broken .dmg files from time to time.

Broken sample (RelWithDebInfo build of d95cbb1):
veles-broken.zip
Working sample (exactly the same code built again):
veles-working.zip

Log from macOS error reporting:

Process: veles [8379]
Path: /Applications/veles.app/Contents/MacOS/veles
Identifier: veles
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: ??? [1]
Responsible: veles [8379]
User ID: 502

Date/Time: 2017-09-18 13:50:32.174 +0200
OS Version: Mac OS X 10.12.6 (16G29)
Report Version: 12

Time Awake Since Boot: 760000 seconds
Time Since Wake: 3500 seconds

System Integrity Protection: disabled

Crashed Thread: 0

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Termination Reason: DYLD, [0x1] Library missing

Application Specific Information:
dyld: launch, loading dependent libraries

Dyld Error Message:
Library not loaded: @rpath/QtNetwork.framework/Versions/5/QtNetwork
Referenced from: /Applications/veles.app/Contents/MacOS/veles
Reason: image not found

Binary Images:
0x101009000 - 0x1011bbfff + (??? - ???) <1E9D333A-E1DC-3816-9D0F-A9B96BA59F52> /Applications/veles.app/Contents/MacOS/veles
0x10c59e000 - 0x10c5dbdc7 dyld (433.5) <322C06B7-8878-311D-888C-C8FD2CA96FF3> /usr/lib/dyld
0x7fffa5728000 - 0x7fffa5739ff3 libz.1.dylib (67) <46E3FFA2-4328-327A-8D34-A03E20BFFB8E> /usr/lib/libz.1.dylib

Searching backwards in "Find/Replace" dialog should be a separate button

We should have "Find previous" and optionally also "Replace previous" buttons instead of a checkbox.

Slot mismatch in QObject::connect in VisualizationPanel

When starting 3d visualization on 7d973b6 on x86_64-linux:

QObject::connect: No such slot veles::visualization::VisualizationPanel::setSampleSize(int)
QObject::connect: (sender name: 'SamplingMethodDialog')
QObject::connect: Incompatible sender/receiver arguments
QCheckBox::stateChanged(int) --> veles::visualization::TrigramWidget::setUseBrightnessHeuristic(Qt::CheckState)

Add Ctrl+PgUp and Ctrl+PgDown for tab navigation

Reason: Ctrl+Tab and Ctrl+Shift+Tab aren't too convenient.

Specific usage CMAKE_OSX_DEPLOYMENT_TARGET

Info

Current Veles version for macOS requires at least macOS Sierra: 10.12.4 (16E195)
See CMakeLists.txt

Details

• CMake seems to ignore CMAKE_OSX_DEPLOYMENT_TARGET
• Confused about CMake's cached variable setting priority

set(CMAKE_OSX_DEPLOYMENT_TARGET "10.10" CACHE STRING "Minimum OS X deployment version")
If not set explicitly as a cache variable the CMAKE_OSX_DEPLOYMENT_TARGET is initialised by the MACOSX_DEPLOYMENT_TARGET environment variable.

CMake complains about cyclic dependencies

I did a git checkout 2017.06.0.SWOND
cmake -D CMAKE_BUILD_TYPE=Release ..
gives

CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "parser" of type STATIC_LIBRARY
    depends on "veles_db" (weak)
    depends on "parser_autogen" (strong)
  "veles_db" of type STATIC_LIBRARY
    depends on "parser" (weak)
    depends on "veles_db_autogen" (strong)
  "parser_autogen" of type UTILITY
    depends on "veles_db" (strong)
  "veles_db_autogen" of type UTILITY
    depends on "parser" (strong)
At least one of these targets is not a STATIC_LIBRARY.  Cyclic dependencies are allowed only among static libraries.

Better application icon

The current one scales terribly and disappears on a white background. It also pixelates on non-default DPIs on Windows.

Console should be visible in debug builds on Windows

"Connection->Connect" doesn't work when server is already running

Log:

Trying to start a new server...
    working directory: [...]/build-veles-Desktop_Qt_5_8_0_MSVC2015_64bit-Release/python
    python script name: srv.py
    python interpreter executable: py.exe
    arguments:
        srv.py
        127.0.0.1:3135
        [...key...]
        [...]/veles.vdb

Process of locally created server started.
Waiting for a new server to start...
Process of locally created server finished. Exit code: 62097.
NetworkClient: New connection status: Not Connected.
NetworkClient: Socket error - The remote host closed the connection
Process of locally created server finished. Exit code: 62097.
NetworkClient: TCP socket disconnected.
NetworkClient::connect
    client interface: 127.0.0.1
    server host: 127.0.0.1
    server port: 3135
NetworkClient: bind successful.
NetworkClient: New connection status: Connecting.
NetworkClient: New connection status: Not Connected.
NetworkClient: Socket error - Connection refused

Windows configuration should be saved and restored

Currently you have to reopen all windows on Veles restart and change their sizes/placement to the preferred ones. It would be nice to restore that settings on Veles restart.

This is not trivial, because open files belong to a database, and can't be open when user switches between two different DBs. We probably need to discuss how exactly should this behave.

Debian pls!?

Hi, is possible to get veles to run on debian testing(or)stretch ? the issue is that in debian, python2 is default by nature & py3 req.txt are all based on py3.

thx in adv.

Missing repository information

No description or website provided.

on top of the main view, above file listing.

[hex view] Moving cursor doesn't work properly

[hex view]
Scrolling doesn't work properly when cursor is out of current view and is moved using keyboard.

Fix performance of entropy mode in minimap

Change ASCII area behavior for non-printables

• Null bytes should be displayed as an empty space, not a dot.
• There should be an option to display all bytes (using some extended ASCII characters) instead of replacing them with a dot. This is a nice feature when working with files which contain text+data, but a terrible one when working with plain binary data (it hides some patterns in the data when looking at ASCII area).

What happened to the blue/orange colors of the point cloud?

The point butt used to be blue/orange to represent location. Is this a setting I'm missing, was this removed, or is this a bug? All the points are just white for me.

Version: 2017.03.0.Świtezianka

Linux alura 4.4.0-71-generic #92-Ubuntu SMP Fri Mar 24 12:59:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
client glx vendor string: NVIDIA Corporation
client glx version string: 1.4
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce GTX 1070/PCIe/SSE2
OpenGL core profile version string: 4.5.0 NVIDIA 375.39
OpenGL core profile shading language version string: 4.50 NVIDIA
OpenGL version string: 4.5.0 NVIDIA 375.39
OpenGL shading language version string: 4.50 NVIDIA
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 375.39
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20

We should enable support for high-DPIs on macOS

Current rendering on Retina (2560x1600):

[hex view] Blinking cursor causes constant high CPU usage

At every timer tick the whole hex view is redrawn, which causes high CPU usage even when there's no user interaction.
IMO we should remove that blinking and then mark focus with solid border and lack of it with dashed border (or something similar).

OSX client cannot connect on fresh install

Appears to be an issue with SSL (which is default) for connecting to localhost server:

Trying to start a new server...
working directory: /Applications/veles.app/Contents/Resources/veles-server
python script name: srv.py
python interpreter executable: /Applications/veles.app/Contents/MacOS/../Resources/veles-server/venv/bin/python3
arguments:
srv.py
--cert-dir
/Users/user/Library/Application Support/Codisec/Veles
veles+ssl://@127.0.0.1:3135
/Users/user/Library/Application Support/Codisec/Veles/veles.vdb

Waiting for a new server to start...
Process of locally created server started.
INFO:root:Świtezianka server is starting up...
INFO:root:Opening database...
INFO:root:Loading plugins...
INFO:root:Starting SSL server...
NetworkClient::connect
client interface: 127.0.0.1
server host: 127.0.0.1
server port: 3135
NetworkClient: bind successful.
NetworkClient: New connection status: Connecting.
INFO:veles.server.proto:Client url: VELES+SSL://:476C9D1E89AD56029E382904F33CAA72C8A4FC7F3F9CF53228618D5B892B0F26@127.0.0.1:3135
INFO:root:Ready.
NetworkClient: unexpected error: The root CA certificate is not trusted for this purpose
NetworkClient: New connection status: Not Connected.
NetworkClient: Socket error - The host name did not match any of the valid hosts for this certificate
NetworkClient: TCP socket disconnected.

Disabling SSL in the Connect menu, and reconnecting twice seems to work.

Builds on Windows should be multithreaded

ImportError: No module named 'veles'

Installed on a fresh upgrade to Sierra 10.12.6 macbook pro ~2013

---- snip ----

Trying to start a new server...
working directory: /Applications/veles.app/Contents/Resources/veles-server
python script name: srv.py
python interpreter executable: /Applications/veles.app/Contents/MacOS/../Resources/veles-server/venv/bin/python3
arguments:
srv.py
--cert-dir
/Users/halon/Library/Application Support/Codisec/Veles
veles+ssl://@127.0.0.1:3135
/Users/halon/Library/Application Support/Codisec/Veles/veles.vdb

Waiting for a new server to start...
Process of locally created server started.
Traceback (most recent call last):
File "srv.py", line 22, in
from veles.server.conn import AsyncLocalConnection
ImportError: No module named 'veles'
Process of locally created server finished. Exit code: 1.

Insert in Hex View fills a second nibble with old data / garbage

[Feature request] Hotkeys

It would be nice to have hotkeys for the RBM menu in hex editor (probably also for other stuff).

E.g.:

• CTRL+G -> Go to address
• CTRL+S -> Save to file

Also the hotkeys should be displayed in the menu items, so it would be easy to familiarize with them.

Add +/- buttons which change number of columns in hex view

Add orthogonal/perspective switch to trigram view

only works with osx 10.12

would be good if this could work with at-least the previous version of osx.

Add IPython console to the UI

Some requirements/ideas:

• Should have a handy and intuitive shortcut (Ctrl+` ? this one is quite standard, but not too easy to reach on the keyboard). Second press should hide the console.
• Should be located in the lower pane, like in Sublime Text
• Should have features from IPython (or just be a wrapper on it) like syntax highlighting and completion
• Should allow multiline editing, like in Python Tools for VS

Hex view color scale could be more informative

Currently hex view colors looks nice, but IMO they are not too useful. We can improve this by varying color intensity (instead of hue) according to byte value, this way it's much easier to spot patterns in the data. For an example of this coloring scheme see Hexplorer.

Also, another idea: (could be customizable in options)
Colorize hexes in various "buckets": e.g. a-zA-Z range in one color, 0-9 in another, 0x00-0x20 in another etc.

Changing workspace or minimizing hides NodeTreeWidget on Ubuntu

QDockWidget::visibilityChanged is emitted while changing workspace or minimizing (Ubuntu 16.04.2 LTS, Qt 5.9.1).
It toggles 'Node tree' button and hides NodeTreeWidget.

It's a side effect of toggling 'Node tree' button when user closes Node tree with cross icon.

make install target should create python virtualenv

Currently, after make install Veles still needs to be run using prepared virtualenv.

install target needs to be extended to prepare Veles venv and default python interpreter should point to this environment.

"Go to address" doesn't move cursor

Point scaling dependent on viewport size in 3D.

Add includes grouping to .clang-format

Included headers should be grouped together according to Google C++ Style Guide rules: https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes

This can be added to .clang-format. See "IncludeCategories" at https://clang.llvm.org/docs/ClangFormatStyleOptions.html.

Code should typedef address type and byte value type

Currently we use inconsistent types for file addresses and byte values. We should refactor code to use typedefs/using for them.

CMake 3.9 compatibility - parser/veles_db cyclic dependencies

CMake 3.9 currently complains about cyclic dependencies:

"parser" of type STATIC_LIBRARY
     depends on "veles_db" (weak)
     depends on "parser_autogen" (strong)
"veles_db" of type STATIC_LIBRARY
     depends on "parser" (weak)
     depends on "veles_db_autogen" (strong)
"parser_autogen" of type UTILITY
     depends on "veles_db" (strong)
"veles_db_autogen" of type UTILITY
     depends on "parser" (strong)

opening hex then selecting "elf(ksy)" causes hard crash on osx

uname -a
Darwin Macpro-2.local 15.6.0 Darwin Kernel Version 15.6.0: Fri Feb 17 10:21:18 PST 2017; root:xnu-3248.60.11.4.1~1/RELEASE_X86_64 x86_64

sw_vers -productVersion
10.11.6

crash.txt.zip

Synchronizing Minimap's visibility with its button

In the same way as in NodeTree.
Warning: Use 'SIGNAL(triggered(bool))' not 'SIGNAL(toggled(bool))'.

Hidden 3D Views still generate high CPU usage

3D Views generate high CPU usage even when their tabs are not shown. This sums up to quite heavy consumption when more visualizations are open in separate tabs (even if only one is displayed at once).