This repository contains a number of parsers for different provenance capture systems. Please go to their respective directories for detailed documentation.
⚠️ cadets
andcdm
parsers are not up-to-date. Use with caution.
License: GNU General Public License v2.0
What is the difference between a stream graph and a base graph? Are they all provenance graph?
Traceback (most recent call last):
File "D:\modeler-master\model.py", line 256, in
precision, recall, accuracy, f_measure, printout = test_graphs(test_files, submodels, tm, ns)
File "D:\modeler-master\model.py", line 135, in test_graphs
test_info)
File "D:\modeler-master\helper\profile.py", line 257, in test_single_graph
max_abnormal_point = max(abnormal_point)
ValueError: max() arg is an empty sequence
Hi! I have encountered the following error when trying to run provparser
on a clean installation of Ubuntu 18.04 (run in a virtual machine).
[WARNING] Use '-t fivedirections' only for ta1-fivedirections-e3-official data. Exit now otherwise...
[+] initiating parser...
[i] multiprocessing support is on for processing but not scanning
[+] processing regular JSON files in directory data/...
[i] multiprocesses processing regular JSON files
[+] setting up database python1.log.db in current directory...
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/anny/rocksdb/build/parsers/cdm/ProvParser/provparser/provparser", line 96, in process
with open(os.path.join(args.input, fn), 'r') as fileobj:
IOError: [Errno 13] Permission denied: 'data/python1.log'
AttributeError: 'list' object has no attribute 'clear'
Exception AttributeError: "'list' object has no attribute 'clear'" in 'rocksdb._rocksdb.DB.__dealloc__' ignored
[+] node parsing is done
[+] parsing files again to output final results.
[i] multiprocessing support is on...
[+] processing regular JSON files in directory data/...
[i] opening output file camflow-out.txt for writing...
Traceback (most recent call last):
File "/home/anny/rocksdb/build/parsers/cdm/ProvParser/testenv/bin/provparser", line 7, in <module>
exec(compile(f.read(), __file__, 'exec'))
File "/home/anny/rocksdb/build/parsers/cdm/ProvParser/provparser/provparser", line 371, in <module>
fileobj = open(os.path.join(args.input, sortedfilenames[0]), 'r')
IOError: [Errno 13] Permission denied: 'data/python1.log'
AttributeError: 'list' object has no attribute 'clear'
Exception AttributeError: "'list' object has no attribute 'clear'" in 'rocksdb._rocksdb.DB.__dealloc__' ignored
Do you happen to have any suggestions on how to fix this, please?
It seems like somewhere in rocksdb
they use a python3 function which is not compatible with the python2. In another VM I have also tried to make python3 the default version, and installed everything, but when I would try to run provparser
I would get an error along the lines of prepare
module is not found (I don't have the exact error text but I can redo the installation and paste the error code here)
In Makefile:
number=0 ; while [ $$number -le 8 ] ; do
cd ../../../data/cadets-e3/benign && mkdir cadets-e3-benign-$$number && tar zxvf cadets-e3-benign-$$number.gz.tar -C cadets-e3-benign-$$number && mv cadets-e3-benign-$$number/cadets-e3-benign.txt.* ../edgelists_benign ;
cd ../../../data/cadets-e3/benign && rm -f cadets-e3-benign-$$number.gz.tar && rm -rf cadets-e3-benign-$$number ;
number=expr $$number + 1
;
done
How to get benign or attack files of the cadets(e.g. cadets-e3-benign-0.gz.tar)?
I'm trying to process DARPA TC dataset now, but the text doesn't mention how the initial processing is done, can you give an idea?
There are so many errors during installation
Error compiling Cython file:
------------------------------------------------------------
...
def get_backup_info(self):
cdef vector[backup.BackupInfo] backup_info
with nogil:
self.engine.GetBackupInfo(cython.address(backup_info))
^
------------------------------------------------------------
rocksdb/_rocksdb.pyx:2413:44: Converting to Python object not allowed without gil
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-Z9fjnD/python-rocksdb/setup.py", line 46, in <module>
zip_safe=False,
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/setuptools/__init__.py", line 162, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/usr/lib/python2.7/distutils/command/install.py", line 601, in run
self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/usr/lib/python2.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/usr/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
self.build_extension(ext)
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 205, in build_extension
_build_ext.build_extension(self, ext)
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/Cython/Distutils/build_ext.py", line 131, in build_extension
ext,force=self.force, quiet=self.verbose == 0, **options
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
cythonize_one(*args)
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: rocksdb/_rocksdb.pyx
----------------------------------------
Command "/home/yan/Unicorn/parsers/cdm/ProvParser/base/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-73Osqm/python-rocksdb/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-OZufrw/install-record.txt --single-version-externally-managed --compile --install-headers /home/yan/Unicorn/parsers/cdm/ProvParser/base/include/site/python2.7/python-rocksdb" failed with error code 1 in /tmp/pip-install-73Osqm/python-rocksdb/
I tried to install python-rocksdb (I successfully installed rocksdb before this)
However, the following error occurs
rocksdb/_rocksdb.pyx:2413:37: Constructing Python tuple not allowed without gil
Error compiling Cython file:
------------------------------------------------------------
...
def get_backup_info(self):
cdef vector[backup.BackupInfo] backup_info
with nogil:
self.engine.GetBackupInfo(cython.address(backup_info))
^
------------------------------------------------------------
rocksdb/_rocksdb.pyx:2413:44: Converting to Python object not allowed without gil
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-YYx60b/python-rocksdb/setup.py", line 46, in <module>
zip_safe=False,
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/setuptools/__init__.py", line 162, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/usr/lib/python2.7/distutils/command/install.py", line 601, in run
self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/usr/lib/python2.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/usr/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
self.build_extension(ext)
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 205, in build_extension
_build_ext.build_extension(self, ext)
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/Cython/Distutils/build_ext.py", line 131, in build_extension
ext,force=self.force, quiet=self.verbose == 0, **options
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
cythonize_one(*args)
File "/home/yan/Unicorn/parsers/cdm/ProvParser/base/lib/python2.7/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: rocksdb/_rocksdb.pyx
----------------------------------------
Command "/home/yan/Unicorn/parsers/cdm/ProvParser/base/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-YYx60b/python-rocksdb/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-cxlFXz/install-record.txt --single-version-externally-managed --compile --install-headers /home/yan/Unicorn/parsers/cdm/ProvParser/base/include/site/python2.7/python-rocksdb" failed with error code 1 in /tmp/pip-install-YYx60b/python-rocksdb/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.