paulross/cpip

CPIP - a C/C++ preprocessor implemented in Python.

CPIP

CPIP is a C/C++ Preprocessor implemented in Python. It faithfully records all aspects of preprocessing and can produce visualisations that make debugging preprocessing far easier.

Features

• Conformant C/C++ preprocessor.
• Gives programatic access to every preprocessing token and the state of the preprocessor at any point during preprocessing.
• Top level tools such as CPIPMain.py can generate preprocessor visualisations from the command line.
• Requires only Python 3.6+
• Fully documented: https://cpip.readthedocs.io.
• Free software: GNU General Public License v2

Installation

Install cpip from PyPi:

(CPIP) $ pip install cpip

This will give you access to the command line entry point cpipmain, to check this:

(CPIP) $ cpipmain --help
...

You can check CPIP on some demonstration C code, for example, from the CPIP directory there is a demo directory. You can process this thus:

(CPIP) $ cpipmain -l20 -j1 -o demo/output -J demo/sys/ -I demo/usr/ demo/src/main.cpp
2017-11-17 10:29:16,370 INFO     preprocessFileToOutput(): demo/src/main.cpp
2017-11-17 10:29:16,371 INFO     TU in HTML:
2017-11-17 10:29:16,372 INFO       tmp/output_normal_02/main.cpp.html
2017-11-17 10:29:16,391 INFO     preprocessFileToOutput(): Processing TU done.
2017-11-17 10:29:16,391 INFO     Macro history to:
2017-11-17 10:29:16,391 INFO       tmp/output_normal_02
2017-11-17 10:29:16,400 INFO     Include graph (SVG) to:
2017-11-17 10:29:16,401 INFO       tmp/output_normal_02/main.cpp.include.svg
2017-11-17 10:29:16,423 INFO     Writing include graph (TEXT) to:
2017-11-17 10:29:16,423 INFO       tmp/output_normal_02/main.cpp.include.svg
2017-11-17 10:29:16,424 INFO     Conditional compilation graph in HTML:
2017-11-17 10:29:16,424 INFO       tmp/output_normal_02/main.cpp.ccg.html
2017-11-17 10:29:16,431 INFO     Done: demo/src/main.cpp
2017-11-17 10:29:16,432 INFO     ITU in HTML: .../main.cpp
2017-11-17 10:29:16,448 INFO     ITU in HTML: .../system.h
2017-11-17 10:29:16,453 INFO     ITU in HTML: .../user.h
2017-11-17 10:29:16,462 INFO     preprocessFileToOutput(): demo/src/main.cpp DONE
CPU time =    0.087 (S)
Bye, bye!

The result will be in demo/output/index.html.

There are other installation methods including directly from source.

Visualising Preprocessing

The top level entry point cpipmain (the script CPIPMain.py) acts like a preprocessor that generates HTML and SVG output for a source code file or directory. This output makes it easy to understand what the preprocessor is doing to your source.

Here is some of that output when preprocessing a single Linux kernel file cpu.c (complete output). The index.html page shows how CPIPMain.py was invoked [1], this has a link to to preprocessing pages for that file:

This page has a single link that takes you to the landing page for the file cpu.c, at the top this links to other pages that visualise source code, #include dependencies, conditional compilation and macros:

Lower down this page is a table of files that were involved in preprocessing:

Visualising the Source Code

From the cpu.c landing page the link "Original Source" takes you to a syntax highlighted page of the original source of cpu.c.

The cpu.c landing page link "Translation Unit" takes you to a page that shows the complete translation unit of cpu.c (i.e. incorporating all the #include files). This page is annotated so that you can understand what part of the translation unit comes from which file.

Visualising the #include Dependencies

The cpu.c landing page link "Normal [SVG]" takes you to a page that shows the dependencies created by #include directives. This is a very rich page that represents a tree with the root at center left. #include's are in order from top to bottom. Each block represents a file, the size is proportional to the number of preprocessing tokens.

Zooming in with the controls at the top gives more detail. If the box is coloured cyan it is because the file does not add any content to the translation unit, usually because of conditional compilation:

The page is dynamic and hovering over various areas provides more information:

How and Why the File was Included

Hovering just to the left of the file box produces a popup that explains how the file inclusion process worked for this file, it has the following fields:

• Inc: The filename and line number of the #include directive.
• As: The conditional compilation state at the point of the #include directive.
• How: The text of the #include directive followed by the directory that this file was found in, this directory is prefixed by sys= for a system include and usr= for a user include.

Hovering over the filename above the file box shows the file stack (children are below parents).

This plot can also tell you what types of preprocessor tokens were processed for each file. The coloured bars on the left of the file box indicate the proportion of preprocessing token types, the left is the file on its own, the right is the file and its child files. To understand the legend hover over those bars:

To see the actual count of preprocessing tokens hover over the file box:

Visualising Conditional Compilation

The preprocessor is also responsible for handling conditional compilation which becomes very complicated for large projects. CPIPMain.py produces a succinct representation showing only the conditional directives. The links in each comment takes you to the syntax highlighted page for that file.

Understanding Macros

CPIP tracks every macro definition and usage and CPIPMain.py produces a page that describes all the macros encountered:

Each link on the page takes you to a description of the macro containing:

• The macro name, how many times it was referenced and whether it is still defined at the end of preprocessing.
• The verbatim macro definition (rewritten over several lines for long macros).
• File name and line number of definition, linked.
• Places that the macro was used, directly or indirectly. This is a table of file paths with links to the use point.

• Dependencies, two way:

  • Macros that this macro invokes.
  • Macros that invoke this macro.

Status

Licence

CPIP is a C/C++ Preprocessor implemented in Python. Copyright (C) 2008-2017 Paul Ross

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Also many thanks to SourceForge that hosted this project for many years.

Footnotes

[1]	This was invoked by:

$ python3 CPIPMain.py -kp -l20 -o ../../output/linux/cpu -S __STDC__=1 -D __KERNEL__ -D __EXPORTED_HEADERS__ -D BITS_PER_LONG=64 -D CONFIG_HZ=100 -D __x86_64__ -D __GNUC__=4 -D __has_feature(x)=0 -D __has_extension=__has_feature -D __has_attribute=__has_feature -D __has_include=__has_feature -P ~/dev/linux/linux-3.13/include/linux/kconfig.h -J /usr/include/ -J /usr/include/c++/4.2.1/ -J /usr/include/c++/4.2.1/tr1/ -J /Users/paulross/dev/linux/linux-3.13/include/ -J /Users/paulross/dev/linux/linux-3.13/include/uapi/ -J ~/dev/linux/linux-3.13/arch/x86/include/uapi/ -J ~/dev/linux/linux-3.13/arch/x86/include/ -J ~/dev/linux/linux-3.13/arch/x86/include/generated/ ~/dev/linux/linux-3.13/kernel/cpu.c

cpip's Issues

Make same referenced files common

This issue is only significant when processing directories of C code which produces and links to large amounts of near duplicate data.

When processing included file the contents is linked to the macro page. These links, and the macro page are prefixed with the ITU filename. This makes every version of, say Python.h different even if it is processed with the same macro environment. This makes it impossible to replace common files with a single one that all link to which would reduce duplication hugely.

Change the name of the macro file to just macros.html and all hrefs to that.

passing Defines to PpLexer

For some reason I am not able to pass Defines to the PpLexer using the preDefines.
Like in the Demo, how do i pass AUSTRALIAN & LANG_SUPPORT for example ?

myLex = PpLexer.PpLexer(cpp_file, handlerObj)

File line/column mapping is wrong

There are some xfailing tests in tests/unit/test_core/test_FileLocation.py that illustrate the problem. The remedy is being worked on the fileLineColumn branch.

PpLexer crashes when using annotateLineFile=True

PpLexer has an option annotateLineFile which, when set True, provides GCC style annotations. PpLexer crashes with this option as it makes a call to self._fis.currentFileIsSystemFile and the currentFileIsSystemFile attribute does not exist.

There is no test in tests.unit.test_core.test_PpLexer that uses annotateLineFile=True.

cpip-0.9.8rc0 setup (install) appears to be incomplete

I was trying to install the latest version using 'python setup.py install' but the package appears to be incomplete (i.e. cpip.core is missing).

cpipmain --help

Traceback (most recent call last):
File "/usr/local/bin/cpipmain", line 11, in
load_entry_point('cpip==0.9.8rc0', 'console_scripts', 'cpipmain')()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 490, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2854, in load_entry_point
return ep.load()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2445, in load
return self.resolve()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 2451, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/local/lib/python3.8/dist-packages/cpip-0.9.8rc0-py3.8.egg/cpip/CPIPMain.py", line 135, in
from cpip import CppCondGraphToHtml
File "/usr/local/lib/python3.8/dist-packages/cpip-0.9.8rc0-py3.8.egg/cpip/CppCondGraphToHtml.py", line 30, in
from cpip.core import CppCond
ModuleNotFoundError: No module named 'cpip.core'

Remove duplicate HTML when processing directories

When processing a directory of code each include is rendered in HTML for each translation unit. This is quite expensive in space. The idea is to post process the output of cpipmain by finding identical files and copying them into a common area and deleting the duplicates. The links need to be updated as well, both ways.

Make a tool cpipduperelink that accomplishes this.

SVG include graphs do not resize <svg> page when zooming

The root <svg> element has attributes height and width that have values appropriate to 100% scaling. This means when zoomed out (<100%) the browser displays a bigger page than necessary and when zoomed in (>100%) elements outside the page are not displayed.

The function scaleGraphic() needs to adjust those attribute values on the <svg> element. Probably the easiest way to do this is to pass them in to scaleGraphic() as arguments and supply them specifically when writing the <g id="scaleGroup"> group of elements.

Support wide character-literal variants and comparison

We currently support the L e.g. L'A' prefix for character-literal (ISO/IEC 14882:1998(E) 2.13.2 Character literals [lex.ccon]). However we do not support comparison between wide and narrow characters. For example this produces FAIL:

#if L'A' == 'A'
PASS
#else
FAIL
#endif

Also later versions of the standard have u, U as equivalents (ISO/IEC 14882 / N3242 :2011(E) 2.14.2 Character literals [lex.ccon], ISO/IEC 9899:2011 6.4.4.4 etc.).

This requires some simple changes to PpToken.evalConstExpr() and PpTokeniser._sliceCharacterLiteral() plus tests.

Discovered when processing Python-3.6.2/Python/sysmodule.c

Some ITU files are not colourised with conditional True/False

Some ITU HTML files are not colourised according to their conditional compilation state, instead they are marked as ambiguous.

Some investigation shows that if a file has no conditional compilation directives then that file does not appear in CppCondGraphVisitorConditionalLines and the call to isCompiled() raises so ItuToHtml _incAndWriteLine() treats this as ambiguous.

The solution is that if the path is not in CppCondGraphVisitorConditionalLines then isCompiled() should return 1.

Problems with Python 3.7

When I try to use cpip with Python 3.7 I run into some problems:

the verbose flag on namedtuple has been removed. this was easy to patch into your code. Once I passed this hurdle I get this callstack, which I can't understand:

1> File "C:\src\honey\tools\cpip\core\PpLexer.py", line 480, in ppTokens
1> for aTok in self._genPreIncludeTokens():
1> File "C:\src\honey\tools\cpip\core\PpLexer.py", line 375, in _genPreIncludeTokens
1> for aTok in self._genPpTokensRecursive(myGen):
1>RuntimeError: generator raised StopIteration

This is for the latest commit (hash: 10e6959). I had to use the master as the last official release 0.9.7 does not contain the bugfix for file/line related to the missing currentFileIsSystemFile property of FileIncludeStack.

Accidental token pasting

The preprocessor can be coerced into accidentally pasting tokens to create significant other tokens. For example:

#define PLUS +
+PLUS

This should produce + + not ++.

There are some xfailing tests in tests/unit/test_core/test_PpLexer.py. Search for @pytest.mark.xfail(reason='Need to fix accidental token pasting.')

Some links:

• GCC documentation https://gcc.gnu.org/onlinedocs/cppinternals/Token-Spacing.html
• CPP Internals: http://wwwcdf.pd.infn.it/localdoc/cppinternals.ps.gz