se-sic/cppstats

toolsuite for analyzing cpp-preprocessor-based software product lines

cppstats's Issues

cppstats --version is written to stderr.

This seems to be a bug.
Actual

localhost ~/projects/codeface $ cppstats --version > test
cppstats v3.8.4
localhost ~/projects/codeface $ cat test
localhost ~/projects/codeface $ cppstats --version 2> test
localhost ~/projects/codeface $ cat test
cppstats v3.8.4

Expected

localhost ~/projects/codeface $ cppstats --version > test
localhost ~/projects/codeface $ cat test
cppstats v3.8.4
localhost ~/projects/codeface $ cppstats --version 2> test
cppstats v3.8.4
localhost ~/projects/codeface $ cat test

Warning when running without X

When started from command line, cppstats always triggers a warning from gtk:

$ ./cppstats.py --version                                                                                                                                                 
/usr/lib64/python2.7/site-packages/gtk-2.0/gtk/__init__.py:57: GtkWarning: could not open display
  warnings.warn(str(e), _gtk.Warning)
cppstats v0.8.5

Cppstats doesn't run properly in travis environment

When running in travis I get:
$ /usr/bin/env cppstats --kind featurelocations --file /tmp/tmp4GFmR1.c /tmp/tmpgqa8Ef.csv

# starting 'featurelocations' preparations: tmp4GFmR1.c

# finished 'featurelocations' preparations: tmp4GFmR1.c

# starting 'featurelocations' analysis: tmppBq45v.xml

ERROR: cannot parse (/tmp/tmp72c_rI_cppstats_featurelocations/_cppstats_featurelocations/tmppBq45v.xml). Skipping this file.

# finished 'featurelocations' analysis: tmppBq45v.xml

The test-file I used was:

#if Test
// example
#else
// more
#endif

Any idea why that is happening? The already existing error handling code (https://github.com/clhunsen/cppstats/blob/master/analyses/featurelocations.py#L687) seems to be handling that situation wrong as I would expected a crash instead of creating an invalid .csv file. However I need the root cause so cppstats runs properly on travis.

You can see the error in https://travis-ci.org/matthid/codeface/builds/49739940#L4149
This makes the following unit test in codeface fail: https://github.com/matthid/codeface/blob/integration/codeface/test/unit/test_cppstats_works.py#L62

Any help/hint is appreciated.

--version switch

Add a switch --version to the command-line interface, so that anybody can check if cppstats exists and is runnable.

prefererred output:
cppstats <version number>

Command-line completion for scripts

A working auto-completion for command-line parameters would be great.

A working solution is presented here:
http://furius.ca/optcomplete/

Escape file names in CSV files

According to RFC 4180 2.5 to 2.7, field names should be escpaed with double quotes, because they can contain symbols equal to the separator (, or ;).

featurelocation contains code if not all brackets are closed

I'm using cppstats to get the feature location in c-code. The resulting csv-file is parsed by an python script to save the data into a database. When the script failed to run, i was looking into the csv-file to determine the source of the error.

The c-file processed was from the openldap repo and contained a ifdef statement with an opening bracket and no closing bracket. Resulting in an output of the code following the statement (including line-breaks what lead to an invalid csv format). The line in question was https://git.openldap.org/openldap/openldap/-/blob/86c216d10cad596eb8ca37bd9263f56b8dc1042a/libraries/libldap/os-ip.c#L617

I think it would be a good idea to not only end an entry on the right closing bracket, but also when the end of the line is reached.

Delete tmp files after a run has completed.

For example when running

cppstats.py --kind featurelocations --file git-repos/openssl/crypto/thr_id.c out.csv

After the program terminates there are temporary files left, like:

folder: /tmp/tmpeVwJ9f_cppstats_featurelocations (2 of those after a single run)
file: /tmp/tmp11LQtX.xml (one of this)

After a lot of cppstats runs (for example within codeface) your drive will run out of space eventually.

Make collection of expressions per file configurable

Right now, during the execution of the 'general' analysis, for each file all #ifdef expressions are collected, but each distinct expression is only collected once per file (comparison via string equality).
see: analyses/general.py, function _wrapFeatureUp()

This should be configurable to affect the construction of these local (per-file) pools.
Like: --localpool [once_per_file | once_per_project | all]

• once_per_file
  • collects each expression once per file
  • per-file-redundant expressions are removed
  • (same behavior as before)
• once_per_project
  • collects each expression once per project
  • all redundant expressions are removed
• all
  • collects all expressions per file
  • no expressions are removed

Rewriting of #ifdefs

Right now, during the execution of the 'general' analysis, the #ifdef expressions get rewritten with following pattern. Make this configurable.
see: analyses/general.py, function _getFeatureSignature()

Example

#ifdef A && B
  #ifdef C
  #endif
#elif D
#else
#endif

gets

#ifdef A && B
  #if (A && B) && C
  #endif
#elif !(A && B) && D
#**elif !(A && B)** && !(D)
#endif

Enchancement Idea

This behavior should be configurable to influence the following scattering and tangling analyses to the user's requirements.

Like: --rewriteifdefs [yes | no]

• yes
  • The #ifdef expressions get rewritten as described before
  • This should be default, as it is the bahvior from previous studies.
• no
  • No rewriting is performed.
  • This yields no expression for an #else branch, for example!

Test issue

This is just an test issue to test the connection to PyCharm.

Adjust file names to original paths

Right now, most of the cppstats output files contain paths like this: /home/tester/Documents/cppstats/test/_cppstats/asd/io_unix.C.xml. This is inconvienient, as the user must map the srcML files to the original source files.

Idea

There are two points that need to be considered here:

1. Use the path within the source folder.
2. Shorten the path to a relative path, starting from the parent folder of the cppstats.csv file. (for brevity and hiding sensitive data)

Result

/home/tester/Documents/cppstats/test/_cppstats/asd/io_unix.C.xml

would get

test/source/asd/io_unix.C

License?

Hi: what would be the license for this code?
Thanks!

Push pyparsing to newest version

As pip/setuptools/wheel/... imports the newest version of pyparsing (2.1.10), we need to use that version ourselves within cppstats. The goal is to remove the explicit version for pyparsing in the filesetup.py.

In a fresh virtual environment, the following package is listed:

drwxr-xr-x  2 hunsen splstaff 4.0K Feb 20 16:32 pyparsing-2.1.10.dist-info

cppstats as Python module

Right now, cppstats is only callable via command line. However, since it's mostly written in Python, it would be favorable if it did not only provide a CLI but also an API for direct integration as a Python library into other projects.

This would require some abstraction between CLI parameters captured by ArgumentParser and the actual runtime configuration used in each pass.

This should primarily allow for better error handling when embedding cppstats into automated processes.

Better error-handling for file-based analysis

Regarding the file-based analysis, we can exit cppstats with an error, if any part of the analysis fails. See also issue #11:

The already existing error handling code (https://github.com/clhunsen/cppstats/blob/master/analyses/featurelocations.py#L687) seems to be handling that situation wrong as I would expected a crash instead of creating an invalid .csv file.

The same holds for writing of the srcML files. We should also incorporate the return value from executed commands into this (cf. https://github.com/clhunsen/cppstats/blob/master/preparation.py#L107)

Re-introduce proper logging

With the rewrite in Python, the ability of logging got lost. This need to be re-introduced.

Requirements

• logging for each project in input file independently

• insert a header with the configuration of cppstats

  =================================================================
  cppstats <version>
  project: <absolute path of project folder>
  [preparation: <kind of preparation>; <absolute path>]
  analysis: <kind of analysis>; <result file(s)>
  =================================================================
  
  [<preparation log>]
  
  =================================================================
  
  <analysis log>
  
  =================================================================
  End Time: <time>
  =================================================================
  

  The second row should depend on the CLI argument --filenamesRelative.

• adjustable log-directory

• log files should be named by <timestamp> <project's basename>.log.