An alternative to the actual graph plot in Jupyter Notebook

[scikit-tda]

An alternative to the actual graph plot in Jupyter Notebook

[MLWave]

Beautiful! Definitely interested in implementing this as an option. Thanks!

[empet]

@sauln Plotly displays a toolbar in the upper-right part of each plot (just hover the mouse over that region to see it), that allows to zoom in, zoom out, select a box, and more.
At this moment one works on ipyplotly that will allow node dragging. It will
be available no sooner than two months.

In visuals.py there are a few lines to format the tooltips, meta data ,via html tags (i.e. the original meta_data), but when path_html is None, the text is formated as in Plotly. These are the only duplicates.

Example:

if custom_tooltips is not None:
        if  path_html is None:
            tooltip += "<br>Members: "
        else:
            tooltip += "<h3>Members</h3>"
        for custom_tooltip in custom_tooltips[member_ids]:
            tooltip += "%s " % (custom_tooltip)

[sauln]

Hi @empet!

1. I wish we could use f-strings, but for the time being, we should continue to support older Python versions. An aesthetically pleasing replacement that I've been using is

text = "Foo is {age} {units} old".format(**locals())

If you have to do lots of string formatting, I would try to offload it to Jinja2 as much as possible.

2. I assume the histogram values originally assumed the color function was normalized between 0 and 1. The normalization you suggest in the function look great and I'd readily merge a PR with those changes!

3. Both look amazing, but I think the first on is a little easier to look at.

Thank you for putting all your hard work into this! I look forward to incorporating and using these great improvements!

[sauln]

These look beautiful! Thank you for working on this. I'm excited to use this for my own projects 💃

Please do open a PR and I will work over the next few weeks to review and demo everything.

[empet]

@sauln Finally here are the saved pdfs from plotly images of the kmapper graphs:

mapper-cat.pdf

breast-cancer1.pdf

breast-cancer-h.pdf

[sauln]

This is really cool! Thanks for putting all of this together.

I would love to see what the changes are to the main kmapper.py file, but didn't see them on your fork. Please feel free to create a pull request so we can do a more formal code review.

I agree that this sort of setup could be more natural for many Python programmers, and I have been frustrated myself working with km in a notebook. For these reasons, I think this could be a great alternative.

My main worries are:

• Is there interactivity? I've relied on being able to manipulate and draw the nodes around, and I'm not sure if plotly will support this sort of manipulation.
• How fractured will the visualization tooling become? It would be preferable to not have to duplicate features in both systems.

[sauln]

@empet Sorry for the delay, I completed the code review. Overall, the changes look great. We'll definitely merge as soon as a few things in the PR are cleaned up. See my code review.

[empet]

@sauln I performed the changes listed here. Let me know if more are needed.

[sauln]

Hi @Empt, I hope all has been going well. I implemented a basic adapter to convert the graph output to networkx representation, and started scratching some plotting methods using matplotlib. These can be found here and here respectively.

I'm thinking that the networkx representation will make it considerably easier to leverage existing visualization tools. Do you think the plotly methods will be able to build on this?

I know the original implementation followed the D3 visualization closely. Do you think it would be possible to have everything self contained in drawing.py?

This code is all in draft, so any thoughts you have on how to architecture this would be great.

Thanks,

[empet]

Hi @sauln,

Today I started working on the new version of Plotly viz of kmapper graphs.

It's not difficult to use networkx, but unfortunately it provides less graph layouts compared with igraph. Moreover, the kamada-kawai layout that is very useful for kmapper graphs does not work for disconnected networks: http://nbviewer.jupyter.org/github/empet/Networks/blob/master/kk-layout-in-3-packages-netsci.ipynb

Using Plotly 3.+ I can save a svgor png image of the graph. svg can be converted to pdf via Inkscape https://inkscape.org/en/ and then printed or included in a LaTeX file for publication.

[empet]

@sauln To be more convincing I paste here three images for comparison:

1. igraph kk layout for digits data set:

2. networkx kk layout for the same data:

3. networkx spring layout:

[sauln]

oh wow, the igraph layout is so much better! I'll have to familiarize myself with the library.

[empet]

@sauln I have a few questions:

1. May I use f-strings that can be interpreted only by Python 3.6+? They have a much more simple syntax compared with older string formatting methods.

2. I have a perplexity about the function build_histogram().
   Why did you choose hmin=0, hmax=1?

For the breast cancer data set, for example, I chose

color_function=lens[:,0]-lens[:,0].min()

that has values between:

color_function.min(), color_function.max()=0.0, 0.30720700761143532

In this case
min(node_averages), max(node_averages) are 0.044648797686642538, 0.29316562035235677
and the corresponding histogram is this one:

It looks odd because in the function
build_histogram, h_min, h_max=0, 1
but in this case they should be min(node_averages), max(node_averages)

I normalized the data=node_averages inside the body of the build_histogram()
and got the right histogram (see the next two images).

More precisely the modified function is:

def build_histogram(data):
    # Build histogram of data based on values of color_function

    h_min, h_max = min(data), max(data)#HERE 
    hist, bin_edges = np.histogram(data, range=(h_min, h_max), bins=10)

    bin_mids = np.mean(np.array(list(zip(bin_edges, bin_edges[1:]))), axis=1)

    histogram = []
    max_bucket_value = max(hist)
    sum_bucket_value = sum(hist)
    for bar, mid in zip(hist, bin_mids):
        height = int(((bar / max_bucket_value) * 100) + 1)
        perc = round((bar / sum_bucket_value) * 100., 1)
        print(_color_idx(mid))
        color = palette[_color_idx((mid-h_min)/(h_max-h_min))]# AND HERE is the normalization

        histogram.append({
            'height': height,
            'perc': perc,
            'color': color
        })
    return histogram

3. For Color function distribution I could display information only in tooltips or both in tooltips and in printed text on each bar. Which one do you suggest to implement?

Here are the two versions:

[empet]

Hi @sauln,

Working further I realized that the function build_histogram() is OK with h_min, h_max=0,1,
because the color_function is normalized inside the function init_color_function().
It was my fault since I didn't integrate completely the plotlyviz in kepler-mapper, and I forced it to work by pasting a few functions from visual.py into the notebook. So my color_function got non-normalized.
Hence no change is needed. I'm sorry for misleading you!

[empet]

Hi @sauln,

Here are two images that illustrate how I decided to display the mapper graph+the mapper summary:

respectively the cluster stats:

The mapper summary is splitted into two parts:
the first one consists in data about graph, displayed under its plot, and the second, Algorithms and Functions + Graph Node Distribution as a separate dashboard.

I couldn't imitate your html layout, because the notebook page is a portrait page, not a landscape one, to include text on the two lateral sides.

The new version of plotly-kmapper requires plotly 3.1+, and ipywidgets. A plotly figure can be defined as a FigureWidget. Moreover the Ipywidgets allow to update the Cluster Member Distribution when hovering over the graph nodes (see the gif file), and the textbox displaying the cluster stats.

In a few days will be released the plotly 3.2 version that integrates a module for saving plotly plots as raster or vector images (svg, pdf). Then I'll add a function to save the mapper graph in one of these image formats.
Could you,please, suggest how to proceed with this new version? Should I open a new PR?

[sauln]

These all look great! Let’s start merging!