Not working

[ooreilly]

Not working

[ooreilly]

Glad to see that you have started digging into the code :) I agree that we should avoid using extract.py and instead build a tree traversal tool for this purpose. I think with the tool as written, you sort of get to decide what methods, and in what order to include them in the output file. Unfortunately, the output will become kind of "flat" in the sense that will you will not get different sizes for the header sections, as you mentioned. However, if you have docstrings organized in a tree data structure, then it should be fairly straightforward to set an appropriate header based on the nesting. Perhaps you want to provide another template for describing the overall structure with everything you want on the page. I think it will be good to make it a template to leave room for customizations.

[ooreilly]

I'm afraid the tool is not capable of extracting all docstrings automatically from a file. You need to tell it what function, or method to extract. Since you only have methods, I have modified your example to extract one of your methods. That said, I like the idea of using . to get it to extract all docstrings in a file. If you want to give the implementation a shot I would be happy to accept a pull request. Otherwise, you can make a new issue having . as a feature request, and I will get to it when I have some time.

Also, to make your example work, you need to either remove "Returns:", or provide some description of what the function is returning, or it will fail with the rather cryptic message: missing indent after ''.

class Hook:
    def before_build_model(self, estimator):
        """
        This function is called before calling the `model_fn` function

        Args:
            estimator: A reference to the estimator

        Returns:
            None

        """
        pass  

$ mydocstring hook.py Hook.before_build_model --markdown

Outputs:

Hook.before_build_model

def before_build_model(self, estimator):

---

This function is called before calling the model_fn function

Args

• estimator : A reference to the estimator

Returns

None

[FrancescoSaverioZuppichini]

Hi @ooreilly so happy to read you fast response. I think your tool is the only one that actually works. I will take a look and see If I can add the way to extract all the docstring from a file :)

[FrancescoSaverioZuppichini]

The code is little hard to change since it was written specifically to do one thing at the time so changing to handle multiple matches is not so trivial.

I have written the code to parse all the classes and function in one method and it works but there is no clear way to render the methods of each class. If I treat them like function then they will be rendered after all the classes signature.

I am sorry, the code is very good written but there is no clear way to treat tree-structured data or multiple data at the same time so I will give up, I hope I will find a better tool since it seems impossible to find something that generates markdown doc from google docstring.

Also the method extractor seems not to return the args when I call it idk

[ooreilly]

Hi @FrancescoSaverioZuppichini, thanks for your kind words.
You are right that the code is sort of build for extracting one function at a time. I wanted something that didn't require traversing the object tree because I wanted to parse non-Python code as well. That said, I think if you just want to generate docstrings for Python source then the simplest solution is probably to put together an object tree traversal tool. I am not sure I understand what you mean by "they will be rendered after classes signature". Are you trying to use extract.py for your purpose? My suggestion would be to not use it for this. I wrote some example code to get you started. You probably want to use recursion to support deep tree structures.

#test.py
class Test(object):
    """
    docstring 0
    """

    def method1(self, arg1, arg2):
        """
        docstring 1
        """
        pass
    
    def method2(self, arg1, arg2):
        """
        docstring 2
        """
        pass

def function1(arg1, arg2):
    """
    docstring 3
    """
    pass

def function2(arg1, arg2):
    """
    docstring 4
    """
    pass

#extract.py
import test
import inspect
"""
 https://docs.python.org/3/library/inspect.html
"""

classdoc = {}
funcdoc = {}
for objs in test.__dict__.items():
    # Get Class and methods
    if callable(objs[1]) and inspect.isclass(objs[1]):
        if hasattr(objs[1], '__doc__'):
            classdoc[objs[0]] = objs[1].__doc__
        pass
        #todo .. 

    # Get functions
    if callable(objs[1]) and not inspect.isclass(objs[1]):
        if hasattr(objs[1], '__doc__'):
            funcdoc[objs[0]] = objs[1].__doc__

print(classdoc)
print(funcdoc)

$ python extract.py

Returns

{'Test': '\n    docstring 0\n    '}
{'function1': '\n    docstring 3\n    ', 'function2': '\n    docstring 4\n    '}

Once you have all the docstrings extracted, you can pass them on to parser.py. The parser will split the different sections of the docstring up and put the data in a dict. See tests/test_parse.py for some examples. To output markdown, I use a predefined template, and you can design your own. The code I use to populate the template is for some reason placed in command.py. Here is the relevant part.

from mako.template import Template
# self.template defaults to templates/google_docstring.md
 template = Template(filename=self.template)
hd1 = '#'
hd2 = '##'
hd3 = '###'
headers, data = self.parser.markdown()
print(template.render(header=self.docstring, sections=data,
        headers=headers, h1=hd1, h2=hd2, h3=hd3)) 

Let me know if you have any questions.

[FrancescoSaverioZuppichini]

I have already dug into the code :) Yes basically I was writing the code to extract everything from a model. Don't judge me, it is a little mess but I was just playing:

So in extract I check the query and if I found '.'

def extract(filestr, query):
    """
    Extracts a docstring from source.

    Arguments:
        filestr: A string that specifies filename of the source code to extract
            from.
        query: A string that specifies what type of docstring to extract.

    """
    import os

    filename = os.path.splitext(filestr)
    ext = filename[1]

    options = {'.py' : PyExtract}

    if ext in options:
        extractor = options[ext](filestr)

    if query == '.':
        return extractor.extract_all()

    return extractor.extract(query)

I call this function

    def extract_all(self):
        spec = importlib.util.spec_from_file_location("module.name", self.filename)
        source = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(source)
        attrs = source.__dict__

        queries = []

        for  key, value in attrs.items():
            if inspect.isclass(value):
                match = self.extract(key)
                match['methods'] = []
                print(key)
                for k, v in value.__dict__.items():
                    if inspect.isfunction(v):
                        match['methods'].append(self.extract(key + '.' + k))

                queries.append(match)
            if inspect.isfunction(value):
                queries.append(self.extract(key))




        return queries

And this more or less works but the code should be recursive. But this is more a workaround that a correct implementation. Also, I had to change the Command markdown function since I need to store all the parsed stuff (classes, functions and methods) and then iteratevely render.

  def markdown(self):
        """
        Output docstring as markdown using a template.
        """
        from mako.template import Template
        template = Template(filename=self.template)
        hd1 = '#'
        hd2 = '##'
        hd3 = '###'

        for parser,docstring in zip(self.parsers, self.docstring):

            headers, data = parser.markdown()
            print(template.render(header=docstring, sections=data,
                                  headers=headers, h1=hd1, h2=hd2, h3=hd3))
            if 'methods' in docstring:
                for method in docstring['methods']:
                    parser  = parse.parser(method['docstring'], 'Google')

                    headers, data = parser.markdown()

                    print(template.render(header=method, sections=data,
                                          headers=headers, h1=hd1, h2=hd2, h3=hd3))

Btw, this code is shit so really I was just playing to become familiar. I think what we should change is

• Always, always load everything as a module and recursively find if they are class of function so we create a sort of a tree. It is way much easier to just import the file and then check if the function name is the one provided by the user.
• in the markdown render each node in the tree also according to the depth, so the class methods will be rendered maybe with a smaller header than the class name

I don't understand why you need two separate parsers for function and methods. Thank you for everything. Btw, the code is great you have really good skill with python!

[FrancescoSaverioZuppichini]

What I meant was that with your code, the classes, function, and methods are rendered sequentially its means that If I have a class with two methods and another class with one method, all the classes all rendered first and the methods after them in the file.

[FrancescoSaverioZuppichini]

Hey :) I can play around later or tomorrow morning to merge my idea of the tree with the current code. The template part is a black box to me, basically, you just provide the title, the args etc and it will render it in markdown?

In order to create the 'tree' I think I will need to modify almost everything. I mean, the abstraction that you made, the Command, Parser etc are really cool but I think I will modify the source code. It makes sense to always assume everything and filter based on the user preferences than the other way around.

Could you explain to me why you need one parser for function and one for method? Let me show If I understood. Basically, based on the type, class, function..., you create a dictionary that describes the code { args: [....], docstring : [...]} and that is your abstraction level? So basically our node in the tree.

Also man, I think that this is the only tool to convert google docstring to markdown. It will be amazing if we can make it works in every scenario

[ooreilly]

Sounds good :) The template part relies on another package: http://www.makotemplates.org/ Best place to learn how it works would be to check that resource. Basically, I have another file templates/google_docstring.md that contains a mixture of regular markdown syntax and mako template syntax to grab the data from the dict built by calling parse.py.

Yep, I think what you just want is to use parse.py for extracting the docstring data, and then build a tree-based parser and some renderer for handling all the docstrings for the different parts of the tree . Again, my suggestion would be to use the template so that you can build some type of theme and make it easier for others to customize your work.

Do you mean the regex functions that parse function and method separately in extract.py? I forgot that you also need to fetch the arguments from the function signature. Instead of using my regex functions, you can use the python module inspect. See here: https://docs.python.org/3/library/inspect.html#inspect.getargspec
I think that will make it easier for you, because then you can call that function as you are traversing the tree. I think it make sense to have the attributes args, docstring for a Node in the tree.

[FrancescoSaverioZuppichini]

Sorry for the late response, I am pretty busy today. I will start write some code tomorrow and reply to you here :)

[ooreilly]

Thanks for letting me know and I look forward to seeing the update. :)

[ooreilly]

Closed due to inactivity.