bmwcarit / emma Goto Github PK
View Code? Open in Web Editor NEWEmma Memory and Mapfile Analyser
License: GNU General Public License v3.0
Emma Memory and Mapfile Analyser
License: GNU General Public License v3.0
This version is needed because this is the first one to have the Logger class.
This is relatively dangerous (file Emma/emma_libs/categorisation.py
):
def __evalCategoryOfAnElement(nameString, categories, categoriesKeywords, keywordCategorisedElements):
"""
Function to find the category of an element. First the categorisation will be tried with the categories file,
and if that fails with the categoriesKeywords file. If this still fails a default value will be set for the category.
If the element was categorised by a keyword then the element will be added to the keywordCategorisedElements list.
...
"""
Also, the *Keywords.json
files should only be loaded/used when the command line flag is set.
Don't use the keywords fallback for a normal analysis an instead print a info or weak warning to std out that a category ist not in the configuration. If the categorisation file is empty skip the print to std out and display a general weak warning that categorisation is not configured.
html
documentation anymore -> Less errors when not regeneratedUse -r
to reset the config:
-r
will now reset the config meaning the config file will be deleted-r
option and let the user delete the file by hand if neccesarySince GitHub does not understand code blocks with indentation and tripple colons like
:::bash
echo "blah"
::bash
echo "blah"
it is desirable to change all code blocks in the documentation to fenced markdown code:
echo "blah"
We have to check if the html generation is affected by this change.
I reviewed the logging messages and created an Excel sheet (see attached) containing the desired behaviour (+ some comments).
Should be fixed together with issue #13 .
When a section is fully filled by objects/sections only a <Section_Entry> will exist. Is is confusing that it has a start address and a length (of 0 bytes) but no end address (because it would lead to a size of at least 1 byte).
No content shown at all for start and end address + length.
Following scenario:
# Case 0: The object is completely overlapping the section
# S |--------------| or: |-----|
# O |--------------| |--------|
# <= --^ ^-- >=
# FIXME: Is the right case valid? (MSc) --------^^^^^^^^^^^^
The FIXME
already indicates the first issue:
Is is a valid case that an object could start before and/or end after the section start/end?
Second issue:
S ------||---------------|
O |---------|
Can an object span over two different sections?
For the ease of use we should integrate emma into the pip database. Second it could extend the willingness to use emma.
Tasks for this would be:
setup.py
, ...)After the recent changes Emma can only be invoked via module calls (->like python -m Emma.emma --project doc/test_project --mapfiles doc/test_project/mapfiles --dir doc/test_project/results
from the top level folder).
This is lengthy. Plus, an overall script for all modules would be convenient and will provide a better overview about the modules (a la git add
or svn checkout
, ...).
Emma.py
The genDoc
folder is ignored currently by git at the moment. This contains some pictures files that are missing from the .md
documents becauase of this. They can be found in the .html
docs because of the picture embedding. These need to be not ignored anymore.
.md
documents are presented rightWith the memory manager redesign (issue #5 ) we smoothed the ways for implementing a parser (memory manager) for GCC.
See also this issue: https://cc-github.bmwgroup.net/marcelschmalzlpartner/MAdZ/issues/74
==
Infineon XMC 2Go blinky project licence:
| test_project_gcc (v1.3+) | Map files based on XMC 2Go Initial Start project | No | BSD-3-Clause | [Infineon -> Documents -> XMC 2Go Initial Start](https://www.infineon.com/cms/en/product/evaluation-boards/kit_xmc_2go_xmc1100_v1/#!documents) |
The address before and after the translation should be printed (possibly when debug output is active) in order to make debugging easier.
Currently not all pypiscout calls (esp. Emma-vis) are used with the SCout Logger implementation.
Originally posted by @holzkohlengrill in #8 (comment)
Meaning sc.error()
should be sc().error()
etc.
TODOs:
sc.*
calls with logging class callsIf --append
is active --overview
will (silently) not be executed.
Both should be executed.
Incorrect input data causes deltas to crash. For example:
(info) : Started processing at 14:52:04
Enter project root path >C:\0-repos\Emma\MAdZ-configs\Project
0: Section_Summary
1: Object_Summary
2: Objects_in_Sections
Choose File type >
0
Select two indices seperated by one space >0 0
Traceback (most recent call last):
File "C:/0-repos/Emma/Emma/emma_deltas.py", line 153, in <module>
runEmmaDeltas()
File "C:/0-repos/Emma/Emma/emma_deltas.py", line 149, in runEmmaDeltas
main(parsedArguments)
File "C:/0-repos/Emma/Emma/emma_deltas.py", line 127, in main
candidates = filePresenter.chooseCandidates()
File "C:\0-repos\Emma\Emma\emma_delta_libs\FilePresenter.py", line 55, in chooseCandidates
indices: typing.List[int] = [int(i) for i in indices]
File "C:\0-repos\Emma\Emma\emma_delta_libs\FilePresenter.py", line 55, in <listcomp>
indices: typing.List[int] = [int(i) for i in indices]
ValueError: invalid literal for int() with base 10: ''
Process finished with exit code 1
If an invalid root path is entered deltas raises a FileNotFound
error.
Consider also the expected behaviour in the attachment from #14 .
The handling of the --dir parameter is quite strange in the Emma-vis:
If you got many objects (~ >100 000) the runtime starts to get significantly high.
Acceptable runtime (within minutes).
GitHub pages can be created relatively easy. Possibly it makes sense to provide the documentation also in this format.
It has to be evaluated
List of potential tools:
After recent discussions it makes to state different map file directories per configID.
mapfiles
As a user I want to output image(s) which visualise objets and sections that I can verify and better understand the results (especially regarding overlap resolution).
0x
consider in hex otherwise in dec)Some picture links has an ending width="XY%" /></div>
where the /
after the %
symbol seems to be extra. This will lead to a generated html where the <p>
and </p>
tags are not matched according to PyCharm.
PyCharm detects the error with the following pictures:
/
symbols and rerun PyCharm inspectionThe verbose option does not show any effect.
Reason: The command line argument is not passed to the SCout constructor.
Control inverse verbosity via -v
No change.
In order to enable seamless integration to quality tools (TeamScale will be the first step towards this direction) Emma should generate some JSON output.
path
tag) (non-code metric) in JSON formatPython will throw an exception when a JSON input file has no valid syntax. This is not pretty.
=> emma + emma_vis
A SCout error message should appear combined with a proper exit of Emma.
Python JSONDecodeError appears.
Use invalid JSON config file.
===
=> Same issue with FileNotFoundError
=> emma + emma_vis + emma_deltas
At least a few sanity check checking for mandatory flags are missing (e.g. "end"
in addressSpaces*.json
.
Sanity checks for all mandatory json flags.
This should be like a "dry run". We need to run Emma in order to get all sections/objects from the map files. Then we auto-generate the categories config file based on the keyword file(s).
Currently .csv
files are saved for such a run (which is the default behaviour for a normal run) which is not desired.
For a "keywords-run" no results should be stored.
TODO in: https://github.com/bmwcarit/Emma/blob/master/emma.py#L136 (invalid link: check commit from Jul 2019)
Due to pypi, pylint and other dependencies it is necessary to move those files one level deeper.
Exceptions:
The regex'es for the GHS parser match Global Symbols which they should not.
It should not match for example:
00000000+000000 D GetWriteBlock..libs.some.global..symbol.blah.blub.
but currently it does.
I figured this out since the address translation failed warning appeared. From the FQN it seems that only the section regex might be affected.
Possibly we can stop if a line starts with Global Symbols
minus leading spaces. However we should check if we always can rely on this order: Sections Summary -> Object Summary -> Global Symbols
Emma visualiser should not prompt if there is only one report present.
The --noprompt
command line argument and it´s documentation needs to be checked.
In the readme it was documented the following way: "exit and fail on user prompts".
The code might always be executing the first one.
Test this in the code and make sure it was documented accordingly.
SECTIONS_TO_EXCLUDE
(find it here) contains kind of specific data. It would be better placed in the globalConfig file.
Define a default behaviour (= current behaviour with one exception (see below)) and add a weak warning notifying that this setting is not overwritten and the default SECTIONS_TO_EXCLUDE
is currently active.
Exception: remove:
".unused_ram",
".mr_rw_NandFlashDataBuffer"
but keep the variable SECTIONS_TO_EXCLUDE
.
The build process could be speed-up splitting installations to the stages where those packages were only needed.
This could be handles with yaml aliases similar to the linux distributions already existing.
We want a stable and comfortable user interface which can compare quickly and easily Emma analyses.
The current user interface is quite rudamentary.
See also this ancient issue for more details (note: not everything is still up-to-date what is stated there): https://cc-github.bmwgroup.net/marcelschmalzlpartner/MAdZ/issues/45
Consider also the thoughts in: https://cc-github.bmwgroup.net/marcelschmalzlpartner/MAdZ/issues/44
ISSUE_#5_MemoryManagerRedesign
branchConfiguration
MapfileProcessor
MemoryMap
Since we switched to GitHub pages there is no need for html documentation anymore.
To be discussed if we should keep the html generation part for docs. However it might likely fail at some point since it won't be executed regularly anymore.
In order to give the user a better understanding about overlaps, containments and duplicates a graphical visualisation would help. Currently this is only possible by viewing the .csv
file (-> textual representation).
In small projects it is still easy to keep an overview however when projects grow this can happen hundreds of times.
Another alternative or complementary solution would be to reshape or add some information that tools like Gephi could import the .csv
and/or .dot
file.
The great thing here about the .csv
import is that you can switch from graphical to "data laboratory" representation (and back) and get more information about this specific element in this view.
Note about Gephi DOT support:
Gephi currently doesn’t provide a complete support of the DOT format. Subgraphs are not supported, nor custom attributes or size. Only labels and colors are imported if present. Directed and undirected graphs are supported.
From: https://gephi.org/users/supported-graph-formats/graphviz-dot-format/
emma_libs/memoryManager.py:156
is a good place to get started with the implementation.
graphviz
(MIT) for dot files); decide whether we need an external libraryThe following topics need to be done:
ISSUE_#3_Testing
branchIf no supplement folder is detected a weak warning should inform the user that this folder does not exist in project path and that no supplement files will be attached to the report.
An error message is shown if the folder is not present.
Emma crashes when a VAS is not found in virtualSections but stated in the patterns config.
Emma should throw an error if this happens.
When entering a negative offset in the configuration (addressSpaces.json) check if this is handled correctly.
{
"offset": "<NEGATIVE_ADDRESS>",
"memory": {
...
Failure
Traceback (most recent call last):
File "C:\0-repos\Emma\tests\functional_tests\test__cmd_line.py", line 212, in test_normalRun
Emma.emma_vis.main(argsEmmaVis)
File "C:\0-repos\Emma\tests\functional_tests\..\..\Emma\emma_vis.py", line 110, in main
consumptionModule.appendModuleConsumptionToMarkdownOverview(markdownFilePath)
File "C:\0-repos\Emma\tests\functional_tests\..\..\Emma\emma_vis_libs\dataVisualiserObjects.py", line 151, in appendModuleConsumptionToMarkdownOverview
self.plotByCategorisedModules(plotShow=False) # Re-write .png to ensure up-to-date overview
File "C:\0-repos\Emma\tests\functional_tests\..\..\Emma\emma_vis_libs\dataVisualiserObjects.py", line 52, in plotByCategorisedModules
Emma.shared_libs.emma_helper.saveMatplotlibPicture(figure, os.path.join(self.resultsPath, filename), MEMORY_ESTIMATION_PICTURE_FILE_EXTENSION, MEMORY_ESTIMATION_PICTURE_DPI, False)
File "C:\0-repos\Emma\tests\functional_tests\..\..\Emma\shared_libs\emma_helper.py", line 299, in saveMatplotlibPicture
pictureData.savefig(fileObject, format=savefigFormat, dpi=savefigDpi, transparent=savefigTransparent)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\matplotlib\figure.py", line 2180, in savefig
self.canvas.print_figure(fname, **kwargs)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\matplotlib\backend_bases.py", line 2082, in print_figure
**kwargs)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\matplotlib\backends\backend_agg.py", line 527, in print_png
FigureCanvasAgg.draw(self)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\matplotlib\backends\backend_agg.py", line 386, in draw
self.renderer = self.get_renderer(cleared=True)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\matplotlib\backends\backend_agg.py", line 399, in get_renderer
self.renderer = RendererAgg(w, h, self.figure.dpi)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\matplotlib\backends\backend_agg.py", line 86, in __init__
self._renderer = _RendererAgg(int(width), int(height), dpi)
MemoryError: In RendererAgg: Out of memory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python37-32\lib\unittest\case.py", line 59, in testPartExecutor
yield
File "C:\Program Files (x86)\Python37-32\lib\unittest\case.py", line 615, in run
testMethod()
File "C:\0-repos\Emma\tests\functional_tests\test__cmd_line.py", line 215, in test_normalRun
self.fail("Unexpected exception: " + str(e))
File "C:\Program Files (x86)\Python37-32\lib\unittest\case.py", line 680, in fail
raise self.failureException(msg)
AssertionError: Unexpected exception: In RendererAgg: Out of memory
Interestingly that does not always happen. I could not reproduce a scenario that always produces this error - even on the same machine.
Mostly this happens during a unit test run.
Python version: 3.7.0
OS: Win10
Installed packages:
pip3 list
Package Version
-------------------- ----------
-yqt5-sip 4.19.13
Arpeggio 1.9.0
astroid 2.1.0
atomicwrites 1.3.0
attrs 19.1.0
backcall 0.1.0
bleach 3.0.2
certifi 2018.10.15
chardet 3.0.4
Click 7.0
codacy-coverage 1.3.11
colorama 0.4.0
coverage 4.5.3
coveralls 1.8.1
csv2md 1.0.1
csvtomd 0.3.0
cycler 0.10.0
decorator 4.3.0
defusedxml 0.5.0
docopt 0.6.2
docutils 0.14
entrypoints 0.2.3
gprof2dot 2017.9.19
graphviz 0.11.1
idna 2.7
ipykernel 5.1.0
ipython 7.1.1
ipython-genutils 0.2.0
isort 4.3.17
jedi 0.13.1
Jinja2 2.10
jsonschema 2.6.0
jupyter-client 5.2.3
jupyter-core 4.4.0
jupyterlab 0.35.4
jupyterlab-server 0.2.0
kiwisolver 1.0.1
lazy-object-proxy 1.3.1
lxml 4.3.4
Markdown 3.1.1
Markups 3.0.0
MarkupSafe 1.1.0
matplotlib 3.1.0
mccabe 0.6.1
mistune 0.8.4
more-itertools 7.0.0
nbconvert 5.4.0
nbformat 4.4.0
nose 1.3.7
notebook 5.7.0
numpy 1.15.2
ordered-set 3.1.1
pandas 0.24.2
pandocfilters 1.4.2
parso 0.3.1
pathspec 0.5.9
pickleshare 0.7.5
pip 19.2.1
pluggy 0.9.0
ply 3.11
prometheus-client 0.4.2
prompt-toolkit 2.0.7
py 1.8.0
pydeps 1.7.1
pyecore 0.10.2
pyenchant 2.0.0
Pygments 2.3.1
pylint 2.2.2
pyparsing 2.2.2
pypiscout 2.0
pypiwin32 223
PyQt5 5.12.1
PyQt5-sip 4.19.15
PySide2 5.11.2
pytest 4.4.1
pytest-cov 2.7.1
python-dateutil 2.7.3
python-gitlab 1.6.0
python-markdown-math 0.6
pytz 2018.5
pywin32 224
pywinpty 0.5.4
PyYAML 5.1.1
pyzmq 17.1.2
requests 2.21.0
RestrictedPython 4.0
scikit-learn 0.20.1
scipy 1.1.0
seaborn 0.9.0
Send2Trash 1.5.0
setuptools 39.0.1
six 1.11.0
sklearn 0.0
stdlib-list 0.5.0
terminado 0.8.1
testpath 0.4.2
textX 1.8.0
tornado 5.1.1
traitlets 4.3.2
typed-ast 1.3.4
urllib3 1.24.1
wcwidth 0.1.7
webencodings 0.5.1
wrapt 1.11.1
yamllint 1.16.0
WARNING: You are using pip version 19.2.1, however version 19.2.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
Legends are not placed ideally in figures. Place legend below figure.
During the creation of the .png files from the unfiltered call graph dot files a segmentation fault happens. It seems that the current DPI value of 300 causes the problem. With a 200 DPI, the conversion succeeds.
GHS Integrate files (.int
) used by GHS Intex state VASes and additional information. This is a better alternative than using .ael
files.
Add this to the documentation.
Code coverage is not displayed any more on Codacy. Our Travis build job still says it sends reports during the deployment stage.
The new pypiscout
has the feature to disable log printing based on the log levels.
With this functionality it is possible to completely turn off logging.
It was detected, that even when the logging turned off, there are still messages printed. These are messages that are made with the print()
python function.
pypiscout
calls.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.