vitsalis/pycg

Static Python call graph generator

pycg's Issues

Relative/partial module path of class due to relative import

Hi @vitsalis @gdrosos happy to see the project has seen some cleanup. My team has a unique use-case where our classes are called as if they're functions (with methods called from __new__). That said, we don't think this should impact the way PyCG interpret the call path.

In this case we import a class like:

from library.parent import child

Where the ls of library/parent/ looks like [ child_classes.py __init__.py ] and the child_classes.py contains the class class child():. The __init__.py file has the statement from .child_classes import child.

What doesn't work

When I attempt to import this class child and use it in my other class other_class like:

# called_file.py
from library.parent import child

class other_class():
    process():
        stuff = child()
# ...

PyCG finds that other_class.process depends on child_classes.child. Where the call graph is...

library.other_class.process: [..., child_classes.child, ...]
````.
Here we would instead **expect the full module path** 

library.other_class.process: [..., library.parent.child_classes.child, ...]

## What does work

However when I change the import to use the full path.

```py
from library.parent.child_classes import child

PyCG does create a tree that reflects the full module path library.parent.child_classes.child for library.other_class.process.

---

Question: Wondering if this is expected or if there is something we should keep in mind in our file structure or init files?

PyCG can't support the analysis of function pointers in a call graph.

Hi,

It's a great project. But I've encountered an issue where PyCG cannot handle function pointers in the code.

Here's an example:

# pycg_test.py
def test1():
    print("test1")

class Foo():
    def __init__(self, name=None, fp=None):
        self.name = name
        self._handler = fp or test1

    def test2(self):
        self._handler()

c = Foo()
c.test2()

The minimum expected result for test2 should be: "pycg_test.Foo.test2": ["pycg_test.Foo._handler"]. If higher requirements are needed, it should extend to the following fp and test1, i.e., "pycg_test.Foo.test2": ["pycg_test.Foo._handler", "pycg_test.test1"].

However, the current output from PyCG is insufficient:

$ pycg pycg_test.py
{"pycg_test": ["pycg_test.Foo.__init__", "pycg_test.Foo.test2"], "pycg_test.test1": ["<builtin>.print"], "<builtin>.print": [], "pycg_test.Foo.__init__": [], "pycg_test.Foo.test2": []}

I want to ask if there's any solution available. I'm also willing to provide a PR. Thank you!

<Performace Bug>: It costs too much time to analyze DL packages(e.g., numpy)

Hello, it costs me tooooo much time to analyze deep learning packages(numpy, tensorflow, etc.), which is unacceptable. Do you have any idea about optimizing PyCG to reduce its running time?

As you mentioned here, it's likely to improve the complexity. Can you give me some suggestions on how to optimize DefinitionManager.complete_definitions()? (I don’t mind sacrificing a certain precision in exchange for significantly less execution time.)

I'm looking forward to receiving your constructive suggestions. Thanks!

Can pycg analyze a function call chain?

Thank you for making the interesting tool public! I am reading the ICSE paper and trying to use this tool.

I have one question about the behavior of pycg.
I have the following code example.py:

class C:
        def __init__(self):
                pass

        def m(self):
                return "x({0})"

obj = C()
print(obj.m().format(1))

I executed pycg example.py.

I expected that pycg reports that the example file calls str.format (called in the bottom line) because the function body is included in the analysis.

However, pycg resulted in:

{"example": ["example.C.m", "example.C.__init__", "<builtin>.print"], "example.C.__init__": [], "example.C.m": [], "<builtin>.print": []}

The str.format function is not included in the result.

Do I miss any important options?

Can it be used to generate static backward slice?

Can this tool be extended for inter-procedural backward slicing?

Can PyCG optimize to flow sensitive?

Sometimes the PyCG can not get right callgraph because it is flow in-sensitive, there is a example

def nested_func():
    pass


def nested_func2():
    pass


def param_func(b):
    b()


def func(a):
    a(nested_func)


def func2(a):
    a(nested_func2)


var_b = func
var_c = func2

var_b = var_c
var_a = var_b

var_a(param_func)

will get the wrong callgraph because the

{
  "micro-benchmark.snippets.test": [
    "micro-benchmark.snippets.test.func",
    "micro-benchmark.snippets.test.func2"
  ],
  "micro-benchmark.snippets.test.nested_func": [],
  "micro-benchmark.snippets.test.nested_func2": [],
  "micro-benchmark.snippets.test.param_func": [
    "micro-benchmark.snippets.test.nested_func2",
    "micro-benchmark.snippets.test.nested_func"
  ],
  "micro-benchmark.snippets.test.func": [
    "micro-benchmark.snippets.test.param_func"
  ],
  "micro-benchmark.snippets.test.func2": [
    "micro-benchmark.snippets.test.param_func"
  ]
}

I think it is not easy to optimizate, I want to know if it can be optimizated like these cases?

Cannot generate call graph

I tried running the command pycg --package scipy $(find scipy -type f -name "*.py") -o scipy.json

Where i downloaded pycg using sudo pip install pycg
And i create a folder name pycg in my desktop, after that i clone scipy source code into that folder using git clone
and in the folder i run command pycg --package scipy $(find scipy -type f -name "*.py") -o scipy.json,
But it keep giving this error, and i don't know how to deal with it, seem like it is not the problem with scipy folder.

Traceback (most recent call last):
File "/usr/local/bin/pycg", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/pycg/main.py", line 79, in main
cg.analyze()
File "/usr/local/lib/python3.8/dist-packages/pycg/pycg.py", line 155, in analyze
self.do_pass(PreProcessor, True,
File "/usr/local/lib/python3.8/dist-packages/pycg/pycg.py", line 146, in do_pass
processor = cls(input_file, input_mod,
File "/usr/local/lib/python3.8/dist-packages/pycg/processing/preprocessor.py", line 33, in init
super().init(filename, modname, modules_analyzed)
File "/usr/local/lib/python3.8/dist-packages/pycg/processing/base.py", line 36, in init
with open(filename, "rt") as f:
File "", line 991, in _find_and_load
File "", line 971, in _find_and_load_unlocked
File "", line 914, in _find_spec
File "", line 1407, in find_spec
File "", line 1379, in _get_spec
File "", line 1540, in find_spec
File "", line 1494, in _get_spec
File "/usr/local/lib/python3.8/dist-packages/pycg/machinery/imports.py", line 39, in init
ig_obj.create_edge(self.fullname)
File "/usr/local/lib/python3.8/dist-packages/pycg/machinery/imports.py", line 87, in create_edge
raise ImportManagerError("Can't add edge to a non existing node")
pycg.machinery.imports.ImportManagerError: Can't add edge to a non existing node
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 55, in apport_excepthook
import apt_pkg
File "", line 991, in _find_and_load
File "", line 971, in _find_and_load_unlocked
File "", line 914, in _find_spec
File "", line 1407, in find_spec
File "", line 1379, in _get_spec
File "", line 1540, in find_spec
File "", line 1494, in _get_spec
File "/usr/local/lib/python3.8/dist-packages/pycg/machinery/imports.py", line 39, in init
ig_obj.create_edge(self.fullname)
File "/usr/local/lib/python3.8/dist-packages/pycg/machinery/imports.py", line 87, in create_edge
raise ImportManagerError("Can't add edge to a non existing node")
pycg.machinery.imports.ImportManagerError: Can't add edge to a non existing node

Original exception was:
Traceback (most recent call last):
File "/usr/local/bin/pycg", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/pycg/main.py", line 79, in main
cg.analyze()
File "/usr/local/lib/python3.8/dist-packages/pycg/pycg.py", line 155, in analyze
self.do_pass(PreProcessor, True,
File "/usr/local/lib/python3.8/dist-packages/pycg/pycg.py", line 146, in do_pass
processor = cls(input_file, input_mod,
File "/usr/local/lib/python3.8/dist-packages/pycg/processing/preprocessor.py", line 33, in init
super().init(filename, modname, modules_analyzed)
File "/usr/local/lib/python3.8/dist-packages/pycg/processing/base.py", line 36, in init
with open(filename, "rt") as f:
File "", line 991, in _find_and_load
File "", line 971, in _find_and_load_unlocked
File "", line 914, in _find_spec
File "", line 1407, in find_spec
File "", line 1379, in _get_spec
File "", line 1540, in find_spec
File "", line 1494, in _get_spec
File "/usr/local/lib/python3.8/dist-packages/pycg/machinery/imports.py", line 39, in init
ig_obj.create_edge(self.fullname)
File "/usr/local/lib/python3.8/dist-packages/pycg/machinery/imports.py", line 87, in create_edge
raise ImportManagerError("Can't add edge to a non existing node")
pycg.machinery.imports.ImportManagerError: Can't add edge to a non existing node

a unexpected error about scanning binary file, (bug report)

Hello, I have read your work "PyCG: Practical Call Graph Generation in Python", and think it is very nice and insightful. But when I try to generate a py file's graph, which contain 'import numpy', your provided script will scan some unexpected binary files following:

/home/xnli/.conda/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so,

I think maybe scanning binary files is not your expected behaviour of your provided script?

What tool have you used to generate the figures from JSON/FASTEN?

The paper contains some figures with arrows and colors, very useful for human (rather than machine) understanding.

However it looks like this project generates output in either JSON or FASTEN format which are machine-friendly but not human friendly and certainly not graphics. How have you generated the figures in the paper and/or what would you recommend people to use to go from JSON (or FASTEN) to an output more suitable for human consumption?

Possible to have option to exclude builtin functions?

Is it possible to have an option, e.g. --exclude-builtin to not include any <builtin> functions?

I use this mainly to interpret complicated repos not initiated by me, and reducing built-in functions will reduce the clutter in the graph.

AttributeError: module 'importlib' has no attribute 'abc'. Did you mean: '_abc'?

After installing pycg using pip, I get the following traceback when trying to run on a sample code.

Traceback (most recent call last):
  File "/home/kyle/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/kyle/.local/lib/python3.10/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/kyle/.local/lib/python3.10/site-packages/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/home/kyle/.local/lib/python3.10/site-packages/pycg/pycg.py", line 144, in do_pass
    self.import_manager.install_hooks()
  File "/home/kyle/.local/lib/python3.10/site-packages/pycg/machinery/imports.py", line 203, in install_hooks
    loader = get_custom_loader(self)
  File "/home/kyle/.local/lib/python3.10/site-packages/pycg/machinery/imports.py", line 34, in get_custom_loader
    class CustomLoader(importlib.abc.SourceLoader):
AttributeError: module 'importlib' has no attribute 'abc'. Did you mean: '_abc'?

It seems like the the use of importlib.abs.SourceLoader does not work since the subpackage importlib.abc is not imported directly.

For my installation I have temporally fixed the problem by including the following import:

import importlib.abs as iabs

I then adjusted the class definition as follows:

class CustomLoader(iabc.SourceLoader):

I think there is a better way to do this, but I thought you should at least know of the error. I'm running Python 3.10.x on Arch Linux.

Can't add edge to a non existing node

When I try to run a simple example on PyCG, it always raises exception pycg.machinery.imports.ImportManagerError.

My setting is as follow:
python version: 3.6.9
pycg commit: 99c991

I setup test directory like

mypkg/
  __init__.py
  main.py

and main.py contains

def foo():
    bar()
def bar():
    print('bar')

Running pycg --package mypkg mypkg/main.py -o cg.json raises an exception as follow:

Traceback (most recent call last):
  File "/home/jwhur/.pyenv/versions/3.6.9/bin/pycg", line 11, in <module>
    sys.exit(main())
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/pycg.py", line 157, in analyze
    self.class_manager, self.module_manager)
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/pycg.py", line 147, in do_pass
    modules_analyzed=modules_analyzed, *args, **kwargs)
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/processing/preprocessor.py", line 33, in __init__
    super().__init__(filename, modname, modules_analyzed)
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/processing/base.py", line 36, in __init__
    with open(filename, "rt") as f:
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 894, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1129, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1273, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1229, in _get_spec
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/machinery/imports.py", line 39, in __init__
    ig_obj.create_edge(self.fullname)
  File "/home/jwhur/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pycg/machinery/imports.py", line 87, in create_edge
    raise ImportManagerError("Can't add edge to a non existing node")
pycg.machinery.imports.ImportManagerError: Can't add edge to a non existing node

Not being able to run PyCG from source due to circular import(?)

Hi, I would like to run PyCG from the source, however on both windows/ubuntu(wsl) with Python 3.10/3.9, trying python ./pycg/__main__.py -h would throw an ImportError:

/home/*/.virtualenvs/PyCG/bin/python /home/*/PyCG/pycg/__main__.py 
Traceback (most recent call last):
  File "/home/*/PyCG/pycg/__main__.py", line 4, in <module>
    from pycg import formats
  File "/home/*/PyCG/pycg/pycg.py", line 23, in <module>
    from pycg import utils
ImportError: cannot import name 'utils' from partially initialized module 'pycg' (most likely due to a circular import) (/home/*/PyCG/pycg/pycg.py)

I am relatively not familiar with this problem and would like to learn how to properly config the project so that it can be successfully launched. Also, if it is possible, it would be more beginner-friendly if we could add a how to set up/contribute from source section etc. to README. Sorry for bothering you with this :(

what is the meaning of NEXT_MEHOD and ITER_METHOD?

you seem to use them as getters in the definition manager but never actually set them :)
where did you mean to add the setters?

`pycg` uses a lot of memory to create a Call Graph for specific packages

Hey @gdrosos and @vitsalis ,

running the pypi-plugin which uses pycg internally, for the packages click:7.0 and py:1.11.0, the process of creating a Call Graph uses all my memory (16 GB) and all my swap space (30 GB) without creating a Call Graph, while for other packages it only consumes a tiny fraction of that.

Not sure how to explain that better, so let me know which further information you need.

Thx for your help!

Additional packages:
pyrsistent:0.18.1
Django:1.11.29
rjsmin:1.2.0
pyinotify:0.9.6
pytest:3.2.5
tqdm:4.64.0

Bug Report: <TypeError: object of type 'Definition' has no len()>

Hello, there is a bug in your code, I'm looking forward that you can fix it as soon as possible. The detailed information is listed as below:

1. Bug Description:

<TypeError: object of type 'Definition' has no len()> occurred when I was using PyCG to resolve tensorflow library code.

Other important information in the bug report may include:

File "/home/allen/anaconda3/envs/python3.9/lib/python3.9/site-packages/pycg/processing/base.py", line 187, in do_assign
if pos < len(decoded):

2. Root Cause:

I installed pycg-0.0.4 after running command "pip install pycg" as your README.md suggested. However, in pycg/processing/base.py, line 184~185, your code written in github is if not isinstance(decoded, Definition) and pos < len(decoded): while the code in my local host after installation is if pos < len(decoded): , which triggered the bug.

3. Possible Solution:

Release the latest version to python.

Thanks!

Does pycg work in colab?

functions call stack with random order?

Hello,

very nice tool with plentry of potential IMHO!

I was just wondering whether the order of the function call could be preserved?

For instance for a function I would get the following:

   "flow.runner.run":[
      "<builtin>.len",
      "<builtin>.print",
      "flow.display.plot",
      "pandas.read_csv",
      "numpy.arange",
      "pandas.DataFrame"
   ],

the order of the functions called in the array is not the same order as the original python file. Also if I run pycg several times in a row, I would get a different order each time.

Would it be possible to keep the actual call order? That would definitely be valuable to ensure the sequence of functions being called.
Thanks for your insight.

`pycg` throws `pycg.machinery.imports.ImportManagerError: Can't add edge to a non existing node Command exited with non-zero status 1`

Hey @vitsalis and @gdrosos ,

I am trying to run the pypi-plugin on branch cg_producer-integration upon the pipup package, which results in the error messages written down in the attached output.txt file.

Strange thing is, that one Call Graph can be created (the one for zipp, line 89 in output.txt) while all the others can not.
I have the feeling that maybe the tool is not able to "go deeper" into the folder structure:
For zipp, the structure is /pycg-data/untar/zipp.py while for the others its e.g. pycg-data/untar/more_itertools.
To find python files, it has to dive in one deeper (this is just a guess though).

output_verbose.txt is almost the same output, except that I added some print() statements here to see where the process failed.

I also added a print("Package Path to generate Call Graph") and print(package_path) here, which gives me the feeling, that the problem lays in this line as it has to go one folder deeper to find the python files.

All just guesses, let me know what you think and thx in advance!

output.txt
output_verbose.txt

analysis does not follow explicit super()-calls

I have noticed that pycg does not follow methods called on super() (A).
pycg does follow method-calls on other objects (B), and it follows implicit calls to super (C) (when calling a method on self that is only defined on the superclass)

example of (A), (B), (C) in a file called mod.py

class Sup:

    def __init__(self):
        self.init_me()
    
    def init_me(self):
        pass
    
    def super_func(self):
        pass

class D:
    def __init__(self):
        pass
    
    def d_method(self):
        pass


class C(Sup):

    def __init__(self, a):
        super().__init__() # (A)
        self.init_C()
        D().d_method() # (B)
        self.super_func() # (C)

    def init_C(self):
        pass

    def unused(self):
        pass

The output of pycg mod.py is:
{"mod": [], "mod.Sup.__init__": ["mod.Sup.init_me"], "mod.Sup.init_me": [], "mod.Sup.super_func": [], "mod.D.__init__": [], "mod.D.d_method": [], "mod.C.__init__": ["mod.D.d_method", "mod.D.__init__", "mod.C.init_C", "<builtin>.super", "mod.Sup.super_func"], "<builtin>.super": [], "mod.C.init_C": [], "mod.C.unused": []}

So the tool correctly identifies C.__init__ as calling d_method from class D and super_func from the super-class, but not Sup.__init__, which is surprising to me.

async methods of a class are ignored

pycg traverses all methods of class except async def method(...):

Add class def in addition to MRO

# pycg/processing/cgprocessor.py
...
if pointer_def.get_type() == utils.constants.CLS_DEF:

    # Class def
    self.call_graph.add_edge(self.current_method, pointer)
...

For our purposes we would also want the class itself. Would it be possible to add the option to include it as an edge in the config?

AttributeError: 'NoneType' object has no attribute 'reset_counters'

Hi,

I encountered the following error when using PyCG to parse tensorflow models:

Original Error Description

• AttributeError: 'NoneType' object has no attribute 'reset_counters'
• The problem occurs in /pycg/processing/base.py/self.scope_manager.get_scope(self.current_ns).reset_counters()

Traceback

• Error in sys.excepthook, exactly, in /usr/lib/python3/dist-packages/apport_python_hook.py/from apport.fileutils import likely_packaged, get_recent_crashes)
• ImportError: cannot import name 'likely_packaged'

So, I'm confused about the following two questions:

1. What is the cause of the above error?
2. Can it be fixed or avoided?

PyCG fails when parsing files whose names are used in init file declarations

Description

When PyCG processes a python package which includes a function/class declaration in the __init__.py, it fails to process the classes of a file which has the same name with the declaration, and is stored in the same directory.

Steps to Reproduce

1. Declare in an __init__py file a function test()
2. Create in the same directory a python file named test.py and declare a class within.
3. Run: pycg __init__.py test.py

PyCG will yield the following error:

   ...
    File "/opt/homebrew/lib/python3.10/site-packages/pycg-0.0.6-py3.10.egg/pycg/processing/preprocessor.py", line 65, in analyze_submodule
    super().analyze_submodule(PreProcessor, modname,
  File "/opt/homebrew/lib/python3.10/site-packages/pycg-0.0.6-py3.10.egg/pycg/processing/base.py", line 485, in analyze_submodule
    visitor.analyze()
  File "/opt/homebrew/lib/python3.10/site-packages/pycg-0.0.6-py3.10.egg/pycg/processing/preprocessor.py", line 375, in analyze
    self.visit(ast.parse(self.contents, self.filename))
  File "/opt/homebrew/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/opt/homebrew/lib/python3.10/site-packages/pycg-0.0.6-py3.10.egg/pycg/processing/preprocessor.py", line 114, in visit_Module
    super().visit_Module(node)
  File "/opt/homebrew/lib/python3.10/site-packages/pycg-0.0.6-py3.10.egg/pycg/processing/base.py", line 61, in visit_Module
    self.generic_visit(node)
  File "/opt/homebrew/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py", line 418, in generic_visit
    self.visit(item)
  File "/opt/homebrew/Cellar/[email protected]/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ast.py", line 410, in visit
    return visitor(node)
  File "/opt/homebrew/lib/python3.10/site-packages/pycg-0.0.6-py3.10.egg/pycg/processing/preprocessor.py", line 368, in visit_ClassDef
    super().visit_ClassDef(node)
  File "/opt/homebrew/lib/python3.10/site-packages/pycg-0.0.6-py3.10.egg/pycg/processing/base.py", line 126, in visit_ClassDef
    self.scope_manager.get_scope(self.current_ns).reset_counters()
AttributeError: 'NoneType' object has no attribute 'reset_counters'

Root Cause

When PyCG visits a new module/class/function declaration it creates a new scope for the corresponding namespace.
Therefore, when pycg visits the test()method in the init file it creates a scope with the namespace modulepath.test

During the preprocessing phase, when PyCG visits the module test defined from test.py, it checks if the scope of the corresponding module namespace already exists in the scope manager:

PyCG/pycg/processing/preprocessor.py

Lines 91 to 97 in 4448238

	root_sc = self.scope_manager.get_scope(self.modname)
	if not root_sc:
	# initialize module scopes
	items = self.scope_manager.handle_module(self.modname,
	self.filename, self.contents)
	root_sc = self.scope_manager.get_scope(self.modname)

Since the namespace modulepath.test exists already due to the _init_.py file declaration, PyCG mistakenly percieves the test modules as a root module and does not initialize the scopes declared in the module correctly.

As a result, in the base processing stage occuring later, when PyCG visits a new class definition in module test,

PyCG/pycg/processing/base.py

Line 126 in 4448238

self.scope_manager.get_scope(self.current_ns).reset_counters()

it resets the counters of the corresponding namespace (e.g. modulepath.test.Class1) declared in the scope manager, but since the scopes of the module test have not been initialized, the specific namespace and consequently the scope does not exists, resulting in an AttributeError, since NoneType object has no attribute reset_counters. The error occurs only in Class declarations and not in function declarations because currently PyCG checks if the specific function scope exists before resetting the counters:

PyCG/pycg/processing/base.py

Lines 68 to 69 in 4448238

	if self.scope_manager.get_scope(self.current_ns):
	self.scope_manager.get_scope(self.current_ns).reset_counters()

Propably this check throws the issue under the carpet.

Proposed Fix

In order to tacke this issue, ideally we should make PyCG able to distinguish the difference between a scope defined through the a declaration in an init file and a module declaration with the same name.
In order to implement this, we could implement a dictionary mapping each scope namespace to its type (e.g. module/function/class scope) and we could modify PyCG to match two scopes only when their type is the same.

Related to #50

Detect Pandas errors

shell:
pycg --package pandas-1.2.3 $(find pandas-1.2.3 -type f -name "*.py")
Got the following error：
Traceback (most recent call last): File "/usr/local/bin/pycg", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/pycg/__main__.py", line 56, in main cg.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/pycg.py", line 99, in analyze self.class_manager, self.module_manager) File "/usr/local/lib/python3.7/site-packages/pycg/pycg.py", line 89, in do_pass processor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 202, in visit_Import self.analyze_submodule(modname) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 202, in visit_Import self.analyze_submodule(modname) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 292, in visit_FunctionDef super().visit_FunctionDef(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 70, in visit_FunctionDef self.visit(stmt) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 206, in visit_ImportFrom self.visit_Import(node, prefix=node.module, level=node.level) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 188, in visit_Import self.analyze_submodule(imported_name) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 66, in analyze_submodule self.module_manager, modules_analyzed=self.get_modules_analyzed()) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 463, in analyze_submodule visitor.analyze() File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 485, in analyze self.visit(ast.parse(self.contents, self.filename)) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 111, in visit_Module super().visit_Module(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 61, in visit_Module self.generic_visit(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 292, in visit_FunctionDef super().visit_FunctionDef(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 70, in visit_FunctionDef self.visit(stmt) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/ast.py", line 279, in generic_visit self.visit(item) File "/usr/local/lib/python3.7/ast.py", line 271, in visit return visitor(node) File "/usr/local/lib/python3.7/site-packages/pycg/processing/preprocessor.py", line 308, in visit_Assign self._visit_assign(node.value, node.targets) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 192, in _visit_assign do_assign(decoded, target) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 181, in do_assign do_assign(decoded[pos], elt) File "/usr/local/lib/python3.7/site-packages/pycg/processing/base.py", line 180, in do_assign if pos < len(decoded): TypeError: object of type 'Definition' has no len()

How to generate image of call graph

Hi Vitalis,

Great work! How can a PNG or JPG or SVG of the call graph be created? This would be very helpful in your README.

Thanks a lot for sharing your tool.

make test occur a error

I try in Python3.7 and Python3.10, It occurs the same error.
Is it because the mock version? Can you provide a requirements.txt?

root@9b6f438e0ef0:~/PyCG# make test
python3 -m unittest discover -s pycg/tests -p "*_test.py"
..............F.............E
======================================================================
ERROR: test_handle_module (scopes_test.ScopeManagerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/PyCG/pycg/tests/scopes_test.py", line 55, in test_handle_module
    items = sm.handle_module("root", "", "")
  File "/root/PyCG/pycg/machinery/scopes.py", line 55, in handle_module
    process(modulename, None, symtable.symtable(contents, filename, compile_type="exec"))
  File "/root/PyCG/pycg/machinery/scopes.py", line 34, in process
    if table.get_name() == 'top' and table.get_lineno() == 0:
AttributeError: 'MockTable' object has no attribute 'get_lineno'

======================================================================
FAIL: test_handle_import (imports_test.ImportsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/PyCG/pycg/tests/imports_test.py", line 182, in test_handle_import
    mock_import.assert_called_once_with(".mod2", package="mod1")
  File "/usr/local/lib/python3.7/site-packages/mock/mock.py", line 958, in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/mock/mock.py", line 945, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: expected call not found.
Expected: import_module('.mod2', package='mod1')
Actual: import_module('', package='mod1')

----------------------------------------------------------------------
Ran 29 tests in 0.006s

FAILED (failures=1, errors=1)
make: *** [Makefile:2: test] Error 1

Broken link

The link to the paper on the homepage is broken. If you don't have any other ways, you may want to link https://web.archive.org/web/20220331023325/https://vitsalis.com/papers/pycg.pdf instead

PermissionError : Permission denied

This is a simple problem but I cant seem to solve it. Using the command "pycg folder" gives me a PermissionError in all cases.

I am using pycg on anaconda, windows 10, and tried to launch anaconda on administrator mode but it still results in this error, even for an empty folder.

Did anyone stumble across the same problem ?

Help interpreting the outputs

Are there some examples that help explain the output produced by PyCG? My question is specifically about the two possible outputs from PyCG:
(1) --as-graph-output AS_GRAPH_OUTPUT
Output for the assignment graph
(2) -o OUTPUT, --output OUTPUT
Output path

What is the difference between these two? It would be great if there an example input with these two outputs along with an explanation on interpreting them.

[need help] Is this uncomplete call graph within expectation?

Hello, I have read your work "PyCG: Practical Call Graph Generation in Python", and think it is valuable, helpful and insightful. However, when I try to scan this file as following:

class tmp_C:
    def tmp_f_in_c(self):
        print('tmp_f_in_c called')
        return None

def tmp_f():
    a = tmp_C
    returned = a()
    return returned


def f_sample():

    tmp_c = tmp_f()
    tmp_c.tmp_f_in_c()

the extracted callgraph is
{"small": ["small.f_sample"], "small.tmp_C.tmp_f_in_c": ["<builtin>.print"], "<builtin>.print": [], "small.tmp_f": [], "small.f_sample": ["small.tmp_f"]}
there is not 'small.tmp_C.tmp_f_in_c' in 'small.f_sample' 's calling function while 'small.f_sample' has called 'small.tmp_C.tmp_f_in_c'. So I'd ask whether your callgraph generator could extract this complete callgraph within expectation? It will be a great and wonderful help if you response me. Thank you!

SyntaxError -> can't load unicodedata module

When I was using PyCG to resolve tensorflow, the following error occurred:

SyntaxError: (unicode error) \N escapes not supported (can't load unicodedata module)

More precisely, it looks like:

Traceback (most recent call last):
File "/home/allen/anaconda3/envs/python3.9/bin/pycg", line 8, in
sys.exit(main())
File "/home/allen/anaconda3/envs/python3.9/lib/python3.9/site-packages/pycg/main.py", line 79, in main
cg.analyze()
File "/home/allen/anaconda3/envs/python3.9/lib/python3.9/site-packages/pycg/pycg.py", line 155, in analyze
self.do_pass(PreProcessor, True,
File "/home/allen/anaconda3/envs/python3.9/lib/python3.9/site-packages/pycg/pycg.py", line 148, in do_pass
processor.analyze()
File "/home/allen/anaconda3/envs/python3.9/lib/python3.9/site-packages/pycg/processing/preprocessor.py", line 375, in analyze
self.visit(ast.parse(self.contents, self.filename))
File "/home/allen/anaconda3/envs/python3.9/lib/python3.9/ast.py", line 50, in parse
return compile(source, filename, mode, flags,
File "/home/allen/DL_API/Static_Analysis/3rd_lib/tensorflow-master/tensorflow/tools/docs/generate2.py", line 121
has_gradient = "\N{HEAVY CHECK MARK}\N{VARIATION SELECTOR-16}"
^
SyntaxError: (unicode error) \N escapes not supported (can't load unicodedata module)

I'm curious about the cause of the above error. Do you have any idea?

Error when trying to scan source code

I apologize for this and have a lot of confusion running PyCG.

usage: pycg [-h] [--package PACKAGE] [--fasten] [--product PRODUCT] [--forge FORGE] [--version VERSION]
            [--timestamp TIMESTAMP] [--max-iter MAX_ITER] [--operation {call-graph,key-error}]
            [--as-graph-output AS_GRAPH_OUTPUT] [-o OUTPUT]
            [entry_point ...]

When I try to use it to analyze tensorflow source code, it returns the error pycg: error: unrecognized arguments: -type f -name *.py). Am I doing something wrong or running it in the wrong place?

how to analyse cg of __init__.py

It seems that PyCG will skip this files

Detected non-deterministic results under various configurations

Hi, I have recently been using PyCG for an empirical study to detect non-deterministic behaviors in static analyzers. The experiments resulted in discovering some nondeterministic analysis results across multiple runs under various configurations of PyCG.

The details of the experimental setup are as below:

• The experiments were conducted on the PyCG micro and macro benchmarks.

• The experiments were conducted under 3 sample configurations which were generated using a 2-way covering array from the configuration space.

• The timeout set for PyCG running on both micro-benchmark and macro-benchmark was 5 minutes/program.

• We ran PyCG on each program-configuration combination 10 times and compared the results across 10 runs for detecting non-deterministic behaviors.

• All experiments were conducted in docker containers. The hardware environment is a server with 128GB of RAM and 24 Intel Xeon Silver 4116 [email protected] running Ubuntu 16.04.

In the end, the experiments detected non-deterministic results on 3 programs, which were all from micro-benchmarks. These results were observed under 2 out of 3 sampled configurations.

The attached data is the detected nondeterministic results from micro-benchmark and macro-benchmark and configuration files
(note1: the configurations are hash-coded in the detected results, but the actual configuration options and values that each hash code stands for can be found in the attached configuration files.)

Source code analysis with PyCG

Hi, is it possible to use PyCG in a Python script? I would like to analyze Tensorflow's python scripts which consist of 2800 python files. I would like to iterate through the folder and print out a JSON file for each python script.

Thank you for sharing your tool!

resolve *args, **kwargs

Does pyCG resolve higher-order functions passed in *args or **kwargs? from the snippets, it seems like it works amazingly for normal dicts and lists, so I'm assuming it is possible, but I can't find a snippet for that.
For clarity, I mean usage like:

def foo(f):
    f()

def bar():
    pass

foo(**{"f": bar})

To get a cg like: main->foo->bar

can't get complete call graph

I do a Test and find pycg will generate uncomplete call graph.
.
First , the code of /test1/init.py is as following:

class People():

    def __init__(self) -> None:
        pass

    def returnPeople(self):
        p = People()
        return p

    def talk(self):
        print("hello")

the code of /test2/main.py is as following:

import sys
sys.path.append(r"/mnt/c/Users/79102/Desktop/project/pyPJ/pycgTest")
from thirdLibrary.test1 import People
p1 = People()
p2 = p1.returnPeople()
p2.talk()

The abode can code can run , and print("hello")
Then i run pycg , throw all python files into it , get the following result

{
    "test1": [],
    "test1.People.__init__": [],
    "test1.People.returnPeople": [
        "test1.People.__init__"
    ],
    "test1.People.talk": [
        "<builtin>.print"
    ],
    "<builtin>.print": [],
    "test2.main": [
        "thirdLibrary.test1.People",
        "sys.path.append"
    ],
    "sys.path.append": [],
    "thirdLibrary.test1.People": [],
    "test2": []
}

Pycg don't process p1.returnPeople() ,and get incompletecall graph.
When i modify the code of /test2/main.py

class People():
    def __init__(self) -> None:
        pass

    def returnPeople(self):
        p = People()
        return p

    def talk(self):
        print("hello")
p1 = People()
p2 = p1.returnPeople()
p2.talk()

I want to get the same result , but the result is as following :

{
    "test2.main": [
        "test2.main.People.returnPeople",
        "test2.main.People.__init__",
        "test2.main.People.talk"
    ],
    "test2.main.People.__init__": [],
    "test2.main.People.returnPeople": [
        "test2.main.People.__init__"
    ],
    "test2.main.People.talk": [
        "<builtin>.print"
    ],
    "<builtin>.print": []
}

The latter is the correct result . When i debug pycg and find the the Intermediate definition of both is the ame , so i think maybe there is a matter with processing definition . I think it would be critical when parsing multiple files.

How to Restore Call Chain?

Hi,

Given the following code:

 import tensorflow as tf
 ds = tf.data.TFRecordDataset('train.record')
 ds = ds.repeat(10000)
 ds = ds.batch(1)
 counter = ds.as_numpy_iterator()

PyCG can only extract tensorflow.data.TFRecordDataset while what I expected are:

 tensorflow.data.TFRecordDataset
 tensorflow.data.Dataset.repeat
 tensorflow.data.Dataset.batch
 tensorflow.data.as_numpy_iterator

Other import information may contain:

1. I've set --max-iter as 2. (So, being not converged may cause the error? Which value of --max-iter would be reasonable?)
2. Tensorflow library code and the application code to analyze are both given.

How can I solve the above problem? Can you give me some suggestions?

Thanks! @vitsalis

Install of `0.0.7` failing in GitHub actions

Overview

I don't know which of my dependencies also depends on PyCG, I'm not installing it directly. But yesterday, suddenly, an action which runs every 20 minutes and has been running for months started to fail because it could not install PyCG==0.0.7.

I tried Mac, Ubuntu, various versions of Python 3.9 and 3.10, and I can provide more logs if needed, but they all fail showing the same series of errors.

Errors

Example: log

Notable highlights:

• Getting requirements to build wheel: finished with status 'error'
• KeyError: 'text'
• Several deprecation warnings

Log from when it starts to install PyCG

Collecting PyCG==0.0.7
Downloading PyCG-0.0.7.tar.gz (34 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'error'
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [127 lines of output]
/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/dist.py:472: SetuptoolsDeprecationWarning: Invalid dash-separated options
!!

          ********************************************************************************
          Usage of dash-separated 'description-file' will not be supported in future
          versions. Please use the underscore name 'description_file' instead.
  
          By 2024-Sep-26, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.
  
          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************
  
  !!
    opt = self.warn_dash_deprecation(opt, section)
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/config/_apply_pyprojecttoml.py:75: _MissingDynamic: `license` defined outside of `pyproject.toml` is ignored.
  !!
  
          ********************************************************************************
          The following seems to be defined outside of `pyproject.toml`:
  
          `license = 'Apache Software License'`
  
          According to the spec (see the link below), however, setuptools CANNOT
          consider this value unless `license` is listed as `dynamic`.
  
          https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
  
          To prevent this problem, you can list `license` under `dynamic` or alternatively
          remove the `[project]` table from your file and rely entirely on other means of
          configuration.
          ********************************************************************************
  
  !!
    _handle_missing_dynamic(dist, project_table)
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/config/_apply_pyprojecttoml.py:75: _MissingDynamic: `authors` defined outside of `pyproject.toml` is ignored.
  !!
  
          ********************************************************************************
          The following seems to be defined outside of `pyproject.toml`:
  
          `authors = 'Vitalis Salis'`
  
          According to the spec (see the link below), however, setuptools CANNOT
          consider this value unless `authors` is listed as `dynamic`.
  
          https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
  
          To prevent this problem, you can list `authors` under `dynamic` or alternatively
          remove the `[project]` table from your file and rely entirely on other means of
          configuration.
          ********************************************************************************
  
  !!
    _handle_missing_dynamic(dist, project_table)
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/config/_apply_pyprojecttoml.py:75: _MissingDynamic: `classifiers` defined outside of `pyproject.toml` is ignored.
  !!
  
          ********************************************************************************
          The following seems to be defined outside of `pyproject.toml`:
  
          `classifiers = ['License :: OSI Approved :: Apache Software License', 'Programming Language :: Python :: 3']`
  
          According to the spec (see the link below), however, setuptools CANNOT
          consider this value unless `classifiers` is listed as `dynamic`.
  
          https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
  
          To prevent this problem, you can list `classifiers` under `dynamic` or alternatively
          remove the `[project]` table from your file and rely entirely on other means of
          configuration.
          ********************************************************************************
  
  !!
    _handle_missing_dynamic(dist, project_table)
  /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/config/_apply_pyprojecttoml.py:75: _MissingDynamic: `scripts` defined outside of `pyproject.toml` is ignored.
  !!
  
          ********************************************************************************
          The following seems to be defined outside of `pyproject.toml`:
  
          `scripts = ['pycg=pycg.__main__:main']`
  
          According to the spec (see the link below), however, setuptools CANNOT
          consider this value unless `scripts` is listed as `dynamic`.
  
          https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
  
          To prevent this problem, you can list `scripts` under `dynamic` or alternatively
          remove the `[project]` table from your file and rely entirely on other means of
          configuration.
          ********************************************************************************
  
  !!
    _handle_missing_dynamic(dist, project_table)
  Traceback (most recent call last):
    File "/Users/runner/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/runner/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/runner/hostedtoolcache/Python/3.9.7/x64/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 480, in run_setup
      super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 61, in <module>
    File "<string>", line 35, in setup_package
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 159, in setup
      dist.parse_config_files()
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 627, in parse_config_files
      pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 67, in apply_configuration
      return _apply(dist, config, filepath)
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/config/_apply_pyprojecttoml.py", line 56, in apply
      _apply_project_table(dist, config, root_dir)
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/config/_apply_pyprojecttoml.py", line 82, in _apply_project_table
      corresp(dist, value, root_dir)
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-ct_b_jty/overlay/lib/python3.9/site-packages/setuptools/config/_apply_pyprojecttoml.py", line 183, in _license
      _set_config(dist, "license", val["text"])
  KeyError: 'text'
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Error: Process completed with exit code 1.

Additional information

My actions failed on PyCG==0.0.7 with the following setups:

1. Ubuntu 22.04.3, Python 3.9.18
2. MacOS 12.7.1, Python 3.9.7
3. MacOS 12.7.1, Python 3.10.13

I believe all above setups were using the following versions:

• pip==23.3.1
• wheel==0.41.3
• setuptools==69.0.2

My action succeeded after installing PyCG==0.0.6 with MacOS 12.7.1, Python 3.9.7 setup.

I had no problems installing on PyCG==0.0.7 on my local Macbook running MacOS 12.6.5, using Python 3.10.4.

Original issue in my repository:

• jhu-bids/TermHub#627

Panda or Pytorch cannot be detected

I cannot use PYCG to detect Pandas or PyTorch, it seems that it takes so long that I quit the program

Error running pycg.

Hi there,

writing the Fasten-PyPI-plugin, we would need to run pycg locally to analyze missing call graphs on the FASTEN server.

Trying out the tool, I got with several packages this error.

michelescarlato@splinter:~/call-graphs-creation/scipy-1.8.1$ pycg --fasten  --package scipy $(find scipy -type f -name "*.py") --product "scipy" --version "1.8.1" --forge "PyPI" --max-iter -1 -o cg.json
Traceback (most recent call last):
  File "/home/michelescarlato/.local/bin/pycg", line 8, in <module>
    sys.exit(main())
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/__main__.py", line 79, in main
    cg.analyze()
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/pycg.py", line 155, in analyze
    self.do_pass(PreProcessor, True,
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/pycg.py", line 144, in do_pass
    self.import_manager.install_hooks()
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/machinery/imports.py", line 203, in install_hooks
    loader = get_custom_loader(self)
  File "/home/michelescarlato/.local/lib/python3.10/site-packages/pycg/machinery/imports.py", line 34, in get_custom_loader
    class CustomLoader(importlib.abc.SourceLoader):
AttributeError: module 'importlib' has no attribute 'abc'. Did you mean: '_abc'?

Am I doing something wrong?

Macro-benchmark

Hey @vitsalis ,

can you provide the Macro-benchmark and their call graphs?

Ignore built-in functions, numpy, torch...

I have a pretty complex code and I am only interested in how the functions defined in that code relate to each other. I am not interested in seeing all calls to numpy and pytorch, for example. Excluding these would likely speed up the execution as well. Is there a way to exclude these?

Handling posonlyargs

I am working on a complex python project and encountered an issue. As also mentioned here: https://stackoverflow.com/questions/59564990/typeerror-required-field-posonlyargs-missing-from-arguments-error-running-a-p , ast reads posonlyargs after python 3.8 and it is not handled here:

PyCG/pycg/processing/preprocessor.py

Line 51 in 014d21c

def _get_fun_defaults(self, node):

While I am not sure, adding another condition that checks node.args.args seems to make it work since it will not fail at

PyCG/pycg/processing/preprocessor.py

Line 59 in 014d21c

defaults[node.args.args[cnt].arg] = self.decode_node(d)

Otherwise, node.args.args becomes 0 and for loop does not stop with:

PyCG/pycg/processing/preprocessor.py

Lines 55 to 56 in 014d21c

	if not d:
	continue

Instead,

if not d or not node.args.args:
     continue

can resolve it.

I am using Python 3.11 on Windows 11

PyCG on file text

Is there a simple way to call PyCG on a file contents instead passing files?

pycg can‘t find the real function

repo：https://github.com/PaddlePaddle/PaddleClas
cmd :pycg --package ppclas $(find ppclas -type f -name "*.py") -o ppclas.json
error: funciton:ppcls/arch/backbone/base/theseus_layer.py is subclass paddle.nn.Layer
the function train is from nn.Layer
but the ppclas.json not show this message
e.g.:

How to get the return value of a class or method?

Hi, @vitsalis

I wanna exploit PyCG to analyze the following code:

 def foo():
    pass

 class TensorSliceDataset:
    def __init__(self):
       pass
    def __call__(self, tensors, name):
       pass
    def repeat(self):
       pass
    def shuffle(self, buffer_size):
       pass

 class MyDataset:
    def __init__(self):
       pass
    def __bool__(self):
       return True
    @staticmethod
    def from_tensor_slices(tensors, name=None):
       foo()
       return TensorSliceDataset(tensors, name=name)

 tensor_data = [1., 2., 3.]
 dataset = MyDataset.from_tensor_slices(tensor_data)
 dataset = dataset.repeat().shuffle(buffer_size=100)

The corresponding result of PyCG is as follows:

{"dataset": ["dataset.TensorSliceDataset.repeat", "dataset.MyDataset.from_tensor_slices"], "dataset.foo": [], "dataset.TensorSliceDataset.__init__": [], "dataset.TensorSliceDataset.__call__": [], "dataset.TensorSliceDataset.repeat": [], "dataset.TensorSliceDataset.shuffle": [], "dataset.MyDataset.__init__": [], "dataset.MyDataset.__bool__": [], "dataset.MyDataset.from_tensor_slices": ["dataset.foo", "dataset.TensorSliceDataset.__init__"]}

As shown in the above result, two methods(foo and TensorSliceDataset.__init__) are called by MyDataset.from_tensor_slices. I want to distinguish TensorSliceDataset.__init__ which is returned by the caller from foo which is a normal method, how can I make it?

---

I've tried to check the value of return_ns and unfortunately found that it's not what I want, but node.value seems to be the returned method which is what I want. Can you give me some suggestions on distinguishing the returned method from other methods?

list definition creation causes an infinite loop in the post processor

i will give and example since it happens vary rarely but maybe you will be able to see where the issue is:

example.py:
list1 = [123]

def func1():
for func in list1:
x = func([], [])
list2 = [456]
for func in list2:
x = func([], [])