kolkir/cppminer

cppminer produces a code2seq compatible datasets from C++ code bases.

Hi there 👋

My name is Kirill Kolodiaznyi and I'm a software developer.

🔬 Research & Intersest

The research I’m working on now is the implementation of a robotic platform based on the NVIDIA Jetson Nano 2Gb device with monocular SLAM navigation. The main aim of this work is to improve feature matching algorithms used for camera positioning in a SLAM pipeline. The recent experiments are about using deep learning for SLAM, especially using the Local Feature Matching with Transformers for camera positioning.

Another topic of my interest is multi-threaded algorithms and using them for performance improvements in projects I’m working on. The result of this work is the library that represents higher-level, task-based parallelism. It’s based on the idea of the Intel TBB work-stealing threads management approach.

Also, I’m interested in learning and applying the C++ language for solving Machine Learning tasks. I wrote several articles and a book about the application of C++ to ML. Additionally, I made C++ implementations of popular object detection methods like Faster R-CNN and Mask R-CNN.

✍ Publications

• Hands-On Machine Learning with C++: Build, train, and deploy end-to-end machine learning and deep learning pipelines

• Local Feature Matching with Transformers for low-end devices Feb 1, 2022, arXiv:2202.00770 [cs.CV]

🔧 Technologies & Tools

cppminer's Issues

miner.py

Hey

When running miner.py and giving path and output. I am getting the following error.

if len(self.samples) > 0:
AttributeError: 'AstParser' object has no attribute 'samples'

How to fix this?

LIBCLANG TOOLING ERROR

LIBCLANG TOOLING ERROR: fixed-compilation-database: Error while opening fixed database: no such file or directory
json-compilation-database: Error while opening JSON database: no such file or directory

LIBCLANG TOOLING ERROR

When I try python3 miner.py -c 1000 ../data path/to on my ubuntu16.04,I got this error. it looks like this:
Parallel processes num: 4
Max contexts num: 1000
Max path length: 0
Max sub-tokens num: 5
Max AST depth: 0
Input path: /home/cleowang/code2seq/cppminer/data
Output path: /home/cleowang/code2seq/cppminer/src/path/to
Parsing files ...
LIBCLANG TOOLING ERROR: fixed-compilation-database: Error while opening fixed database: No such file or directory
json-compilation-database: Error while opening JSON database: No such file or directory

LIBCLANG TOOLING ERROR: fixed-compilation-database: Error while opening fixed database: No such file or directory
json-compilation-database: Error while opening JSON database: No such file or directory

How can i fix it? thank you!

UnpicklingError

File "D:\Masters\industrialproject\C++attempt1\code2vec\code2vec\vocabularies.py", line 224, in _load_word_freq_dict
token_to_count = pickle.load(file)
_pickle.UnpicklingError: invalid load key, '\x1f'.

I am getting this error when I am trying to use the Chromium preprocessed data with Code2vec.

ModuleNotFoundError

Hi,

I am trying to train the model and I am getting this error message.

error:
No module named 'cppminer.cpp_parser'

I am not able to resolve the issue .

Thanks

target label and context content

@Kolkir
Hi. I hope you doing good. I have confusion with the target label and context path. After running miner.py. How can I differentiate between the target label and the context path? Also If I have my own label. How can I use my own label?

Thanks

ValueError: ctypes objects containing pointers cannot be pickled

Can anyone find a solution for this?

Traceback (most recent call last):
  File "/Users/lsiddiqsunny/Documents/cppminer/src/miner.py", line 144, in <module>
    main()
  File "/Users/lsiddiqsunny/Documents/cppminer/src/miner.py", line 119, in main
    p.start()
  File "/Users/lsiddiqsunny/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/lsiddiqsunny/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/lsiddiqsunny/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/lsiddiqsunny/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/lsiddiqsunny/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/lsiddiqsunny/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/lsiddiqsunny/.pyenv/versions/3.9.4/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
ValueError: ctypes objects containing pointers cannot be pickled