Comments (10)
Thanks @wsnipex for finding the issue. Yes, our current version of the training dataset requires around 11GB of memory. We can certainly generate a smaller version of the training dataset for memory-constrained devices. We will add this to our current list of features. The detection results, however, would vary with smaller dataset.
from control-flag.
Same result on Ubuntu 20.04, with gcc 9.3.0.
from control-flag.
I got it to run by closing all memory intensive applications and adjusting oom_killer:
echo -15 > /proc/$(pidof cf_file_scanner)/oom_adj
it's been running for 5h now ;)
slightly off topic: I noticed it only uses [num physical cores] parallel threads by default, so hyperthreading/logical cores are unused.
from control-flag.
@AlexisTM Feel free to use smaller training datasets that we have released a couple of days ago.
from control-flag.
Thanks @raniervf and @AlexisTM - we're looking into this.
In addition, if you have the bandwidth and can locate the cause for the crash for it, we'd love to have you send us a pull request.
Best,
The ControlFlag Team
from control-flag.
Thanks for trying out ControlFlag. Would you mind filling out the template below for the bug report? That will help us triage and fix faster.
--
Describe the bug
A clear and concise description of what the bug is.
Exact command to reproduce
Please specify ControlFlag command that you used
Callstack (if it is a crash bug) or error info
Please provide any info about the error - error message, code, etc.
Expected behavior
A clear and concise description of what you expected to happen.
Environment (please complete the following information):
- OS:
- Compiler:
- 32-bit or 64-bit?
- Build command:
ControlFlag commit
Additional context
Add any other context about the problem here.
from control-flag.
looks like out of memory, this is what happens for me, when I try to scan a large C++ repo:
$ journalctl -xe |grep Killed
Out of memory: Killed process 197753 (cf_file_scanner) total-vm:10291624kB, anon-rss:10266540kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:20152kB oom_score_adj:0
for reference: the box has 16GB Ram and 2GB swap.
from control-flag.
Whooohooo! Let us know how your results look -- how big is your code base? Apologies in advance for any false positives you see. We're still working on bringing that number down.
Regarding hyperthreading, we've found that using the hardware threads tends to work more efficiently than using all the logical / software threads that can be enabled using hyperthreading. However, perhaps that was just environment to our execution environment. Perhaps we can look at adding a flag to enable hyperthreading if a user would like to use them?
Thoughts, @nhasabni ?
from control-flag.
That is probably the reason. I have 16GB RAM on that machine.
from control-flag.
I will try to use it with the least amount of apps first, as I do not have much time to retrain it.
from control-flag.
Related Issues (20)
- Docker image with control-flag already built
- [BUG] Is it training instead of evaluating? Is it working? HOT 5
- Segmentation fault while scan_for_anomalies.sh HOT 7
- I've tried it with ClickHouse and it did not find anything meaningful. HOT 11
- Is it possible to mine java pattern? HOT 1
- [FEATURE]Support for the Cpp programming language HOT 2
- [FEATURE]Support for Non-Control structures HOT 2
- [BUG] Authentication Error, Not Handled Correctly HOT 4
- Is it possible to mine c# pattern HOT 2
- [FEATURE] HOT 2
- [BUG] Limited to 16 threads? Missing logfiles? HOT 3
- [FEATURE] Create `requirements.txt` File
- [FEATURE] Give Warning About Possible Antivirus Activity Upon Downloading Top 100 Repos HOT 1
- [BUG] Anomaly report: first "Did you mean" is same as found expression
- [BUG] Line numbers of potential anomaly are incorrect
- [BUG] cf_file_scanner segfaults while scanning files
- [FEATURE]Can I support Golang?
- [BUG] dead link, C++ "Large" date set cannot be downloaded (404) HOT 1
- Support for the Kotlin programming language? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from control-flag.