ahitrin/siebenapp

Experimental dependency-aware goal manager

siebenapp's Issues

Cancel edit

When we add, insert or rename a goal there should be a possibility to deny any changes and keep goal tree unchanged. But currently there is no possibility to cancel edit as it's started.

We could add a new "Cancel" button at the right of edit field. Also it would be nice to add "Save" button too.

Save actions history

Currently, there is impossible to undo changes made to the goaltree (even by mistake). In order to fix this, we need to save changes history in the project file.

Rendering: tweak algorithms for better placement of edges

Current state of rendering of large goal trees is far from perfection. Let's see an example fragment:

Here, we have a dependency between goals 25 and 28 (selected with a purple color):

Looks like a green path should look better. Alas, the rendering engine cannot find it. I guess, we could try to use force-based algorithms to tune edge positions after the initial rendering.

Do not call dot directly

Current rendering mechanism has serious disadvantages:

• It's not secure (because of blind relying on correctness of an external dot program)
• It doesn't allow much customization of goals look
• It doesn't allow to use mouse for goal selection
• It creates a bunch of garbage files (work.dot and work.png)

We should try to use libgraphviz to calculate goal positions and draw goals using Qt instead.

Rendering: do not redraw all goals when it's possible

There a frequently used case when we could avoid redrawing of all goals on GUI. It's a Select event. It affects neither goal positions, nor goal sizes, nor edges between goals (except when a deselected goal disappears).

If we track this situation, overall performance should be significantly improved, especially on large graphs. And it doesn't look like a challenging issue. We could just track user actions and cache previous rendering results.

Multi-document mode

It would be useful to have a possibility to open several goal trees inside one SiebenApp window.

Asynchronously re-read updated documents

Motivation: current DB model is quite performant. All the data is read once on load, and then is only updated. Nevertheless, DB files could be updated from external sources (e.g. by external synchronization processes like Dropbox/Git/etc). Since changes in a DB file are not currently monitored, such scenario may lead to data loss.

Solution: we should implement a mechanism of checking "freshness" of DB files and provide some kind of "refreshing" when changed occur.

Main window redesign

Goals:

• Add menu
• Make more tasty L&F
• Move UI design details from code to QML
• Publish new screenshots

Extract common interface for Goal class

We need to finish work that was started in the domain.py. It seems better to implement classic OOP inheritance than dynamic dispatch using "overriden" attributes. The code should became less murky and more supportable by IDEs when we're done

Simple link categories

It would be useful to distinguish two types of links:

• Subgoals: one big goal can be split into several smaller ones. For example, "to publish version 0.5 I must implement following features".
• Blockers: a goal may be "blocked" by another one which is not a subgoal of it. For example, "before posting news on project I must implement nicer user interface".

Ideas how to work with different kind of links:

• Use different hotkeys. For example, L to create subgoal link, and l to create dependency link.
• Show only subgoals (not blockers) when zooming. Blocked subgoals cannot be closed even in zoomed view
• Delete only subgoals (not blockers) when deleting a goal
• Maybe each goal should have at most 1 "parent" goal, but any amount of blocked ones. When a user creates new "subgoal" link, the previous one is replaced with "blocker" link

Zoom: inconsistent behavior when close zoom root

How to reproduce:

1. Create a goal with few subgoals
2. Zoom onto this goal
3. Try to close it without closing cubgoals

Expected behavior: a message "This goal can't be closed because it have open subgoals". And nothing more.
Actual behavior: unzoom

We don't need an unzoom here!

Save multiple zoom levels

Currently we have only 1 goal to zoom into. It's not useful when working with large goal trees. You constantly switch back to the global view when unzoom, and have to visually search for the next goal.

A -> B -> C -> D
  (zoom on B)
B -> C -> D
  (zoom on C)
C -> D
  (unzoom on C)
A -> B -> C -> D

The simplest solution is to have stacked zoom levels. Like this:

A -> B -> C -> D
  (zoom on B)
B -> C -> D
  (zoom on C)
C -> D
  (unzoom on C)
B -> C -> D
  (unzoom on B)
A -> B -> C -> D

When unzooming, you should return back to the previous zoom root, not to the global tree root.

Application packaging

Application should be installable as binary package on all standard platforms: Windows, Linux, MacOS X.

Sieben-manage: merge database files

Sometimes it would be useful to join two or more goaltrees into a single one. This should be made in sieben-manage script.

Planned usage:

sieben-manage merge first.db second.db [third.db ...]

This command should take all goals and relations from second.db (and other DBs when listed) and put them as subgoals of the root goal in first.db. Only the first.db file will be changed. When command is executed several times it adds several subtrees to the first.db (no check on "already merged" is done).

Create adaptive goal tree enumeration

See https://github.com/ahitrin/SiebenApp/blob/master/docs/architecture/decisions/0002-create-adaptive-goal-tree-enumeration.md

Clieben: sort goals topologically

Current goal ordering in clieben is too primitive. It would be better to use topological sorting instead. This would make all open goals grouping together so they become easier to manage.

Show goal progress

It's often useful to quickly understand how much progress has been made for the given subtree. Currently, this value could only be estimated manually, and requires switching into "Open & Closed goals" view.

In order to track progress more easily, we should show 'goal progress' counter on the goal widget. The rules are following:

1.Goal progress is a ratio of closed/all subgoals (including the goal itself).
2. For a single goal, progress ratio could be either 0% (open) or 100% (closed)
3. For a complex goal with subgoals, the ratio lies between 0 (all subgoals are open) and 100 (all subgoals and the goal itself are closed).
4. Blockers do not affect progress ratio

Currently, it's not clear how the ratio should be displayed. Maybe, percents would be enough. Or maybe it's better to include both percents and open/total counters.

Toggle views independently of each other

Toggle view switching exists here from the very first versions of the applictation. But it seems not much useful for now. Instead of switch loop, we should try to implement independent view switches. This should bring us more mainstreamly UX.

Restore export into .dot format

Sometimes I'd like to overview a whole goaltree at once (or even print it). The easiest way to do it is restoring "export to dot" functionality that existed in the system before #5.

This should be done in a separate application.

Filter layer

Problem: in large goaltrees, it's hard to find subgoals even when we know their name. We need a mechanism that provides at least simple search facilities

Idea: Add a new layer for filter. When pressing f hotkey, the user is asked for the search string (regex?). When it's entered, this layer shows only goals matching this string/regex (together with selected goals and, possibly, edges). When it's cleared, this layer filters nothing.

Warning messages do not appear on Qt UI

How to reproduce:

1. Select a goal
2. Press "hold selection"
3. Press "link"

Expected behavior: a message should appear - "Goal can't be linked to itself"

Actual behavior: nothing happens

Property tests sometimes hang

After some update of Hypothesis dependency property test may run for too long. Looks like a bug in Hypothesis library. It may be related to HypothesisWorks/hypothesis#2340 or HypothesisWorks/hypothesis#2308.

This causes CI builds to fail. When I've caught this bug locally, it was possible to take a profile. Seems that it hangs somewhere inside shrinking function.

Find/Filter goals

When working with large goal trees it would be nice to have an ability of quick search for goals.

Current vision:

1. Implement it as a new layer between Zoom and Enumeration
2. Use new hotkey f to toggle enable/disable filter
3. Show pseudo-goal that contains filter string (when filter is enabled)
4. Show currently selected goal even when it doesn't match filter string

Show unique id of the each goal

Several planned features require to know unique id of each subgoal. Currently, it's not available to the user. Let's simply show it with a small font under the goal "selection number".

Now:

With ID:

(of course, it shouldn't be green)

Renew README

Goal: split app tutorial and meta-info (current status, etc).

• Extract tutorial from README and move it into /doc
• Provide more meta links in new README

Previous selection may get lost on unzoom

https://travis-ci.org/ahitrin/SiebenApp/jobs/636474599

=================================== FAILURES ===================================
________________________ TestGoalTreeRandomWalk.runTest ________________________
self = <hypothesis.stateful.GoaltreeRandomWalk.TestCase testMethod=runTest>
    def runTest(self):
>       run_state_machine_as_test(state_machine_class)
../../../virtualenv/python3.8.0/lib/python3.8/site-packages/hypothesis/stateful.py:283: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../virtualenv/python3.8.0/lib/python3.8/site-packages/hypothesis/stateful.py:178: in run_state_machine_as_test
    run_state_machine(state_machine_factory)
../../../virtualenv/python3.8.0/lib/python3.8/site-packages/hypothesis/stateful.py:91: in run_state_machine
    @given(st.data())
siebenapp/tests/test_properties.py:106: in goaltree_is_always_valid
    assert self.goaltree.verify()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <siebenapp.zoom.Zoom object at 0x7fb9eaf729d0>
    def verify(self) -> bool:
        ok = self.goaltree.verify()
        if len(self.zoom_root) == 1:
            return ok
        visible_goals = self._build_visible_goals()
        assert (
            self.settings["selection"] in visible_goals
        ), "Selected goal must be within visible area"
>       assert (
            self.settings["previous_selection"] in visible_goals
        ), "Prev-selected goal must be within visible area"
E       AssertionError: Prev-selected goal must be within visible area
siebenapp/zoom.py:113: AssertionError
---------------------------------- Hypothesis ----------------------------------
Falsifying example:
state = GoaltreeRandomWalk()
state.open_db_connection()
Goals: [(1, 'Root', 1)], Edges: [], Selection: []
state.add_goal(b=False)
Goals: [(1, 'Root', 1), (2, 'a', 1)], Edges: [(1, 2, 1)], Selection: []
Draw 1: 2
state.select_random_goal(d=data(...))
Goals: [(1, 'Root', 1), (2, 'a', 1)], Edges: [(1, 2, 1)], Selection: [('selection', 2)]
state.zoom()
Goals: [(1, 'Root', 1), (2, 'a', 1)], Edges: [(1, 2, 1)], Selection: [('selection', 2), ('previous_selection', 2)]
state.add_goal(b=False)
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1)], Selection: [('selection', 2), ('previous_selection', 2)]
state.add_goal(b=False)
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1), (4, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1), (2, 4, 1)], Selection: [('selection', 2), ('previous_selection', 2)]
state.add_goal(b=False)
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1), (4, 'a', 1), (5, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1), (2, 4, 1), (2, 5, 1)], Selection: [('selection', 2), ('previous_selection', 2)]
Draw 2: 5
state.select_random_goal(d=data(...))
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1), (4, 'a', 1), (5, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1), (2, 4, 1), (2, 5, 1)], Selection: [('previous_selection', 2), ('selection', 5)]
state.zoom()
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1), (4, 'a', 1), (5, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1), (2, 4, 1), (2, 5, 1)], Selection: [('selection', 5), ('previous_selection', 5)]
state.add_goal(b=False)
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1), (4, 'a', 1), (5, 'a', 1), (6, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1), (2, 4, 1), (2, 5, 1), (5, 6, 1)], Selection: [('selection', 5), ('previous_selection', 5)]
Draw 3: 6
state.select_random_goal(d=data(...))
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1), (4, 'a', 1), (5, 'a', 1), (6, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1), (2, 4, 1), (2, 5, 1), (5, 6, 1)], Selection: [('previous_selection', 5), ('selection', 6)]
state.zoom()
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1), (4, 'a', 1), (5, 'a', 1), (6, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1), (2, 4, 1), (2, 5, 1), (5, 6, 1)], Selection: [('selection', 6), ('previous_selection', 6)]
state.zoom()
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1), (4, 'a', 1), (5, 'a', 1), (6, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1), (2, 4, 1), (2, 5, 1), (5, 6, 1)], Selection: [('selection', 6), ('previous_selection', 6)]
Draw 4: 5
state.select_random_goal(d=data(...))
Goals: [(1, 'Root', 1), (2, 'a', 1), (3, 'a', 1), (4, 'a', 1), (5, 'a', 1), (6, 'a', 1)], Edges: [(1, 2, 1), (2, 3, 1), (2, 4, 1), (2, 5, 1), (5, 6, 1)], Selection: [('previous_selection', 6), ('selection', 5)]
state.zoom()
state.teardown()
================== 1 failed, 123 passed in 166.95s (0:02:46) ===================

Use DSL to build and display goal trees

Motivation: more readable tests. We could use declarative DSL to build trees instead of constructing it manually.

DSL example:

1so Root -> 2, 3, 4
2-c Top 1 ->
3-o Middle -> 4
4po Top 2 ->

There is one row per goal and all its subgoals. A format is simple:

1. Goal number (numeric)
2. Selection mark: s for current selection, p for previous selection, - for others
3. Open mark: o for open goals, c for closed ones
4. A single whitespace (delimiter)
5. Goal name
6. An arrow -> (delimiter)
7. A list of all subgoals (zero or more numbers separated with ', ')

Usage example

Currently goaltrees are constructed in tests in non-visual way:

def test_closed_leaf_goal_could_not_be_reopened(self):
    self.goals.add('A')
    self.goals.select(2)
    self.goals.add('B')
    self.goals.select(3)
    self.goals.toggle_close()
    self.goals.select(2)
    self.goals.toggle_close()
    assert self.goals.all(keys='open,top') == {
        1: {'open': True, 'top': True},
        2: {'open': False, 'top': False},
        3: {'open': False, 'top': False}}
    ...

With declarative DSL, goaltree construction should become much more concise and readable:

def test_closed_leaf_goal_could_not_be_reopened(self):
    self.goals = build_goaltree('''
    1so Root -> 2
    2-c A -> 3
    3-c B ->
    ''')
    assert self.goals.all(keys='open,top') == {
        1: {'open': True, 'top': True},
        2: {'open': False, 'top': False},
        3: {'open': False, 'top': False}}
    ...

Add confirmation on delete

It's sometimes too easy to accidentally hit d button in GUI and unintentionally remove a goal. This action cannot be reverted. That's pity.

We should allow delete goals only with a confirmation. It could be done in the same way as renaming, via the prompt.

Remove experimental DearPyGui app

I've completely lost interest on it. It's difficult to make it less ugly, it's difficult to beat performance issues. Also, DPG library itself (for 1.x version) was put into maintenance-only mode recently.

That's why it doesn't seem reasonable to continue work on DPG version anymore. #255 was an interesting experiment that brought improvements into rendering process. But now we should put a comma.

No symbol escaping in goal names

With using 'plain text' for dot output, not all symbols are allowed in node names. For example, names containing quote marks could not be rendered.

Possible solution: we should try to use html.escape to cleanup such symbols before rendering work.dot.

Show error messages

Motivation: not all goal actions are possible in some situations. For example, the last link between the leaf goal and its parent could not be removed. User should be notified about reasons why some action could not be performed.

Of course, this notification must not be intrusive. Currently, some message in status line at the bottom of the screen could be enough. More changes in interface are to come in #5, and notification mechanism could also be modified after this change.

RenderResult: add 'select' field

Currently, information about selected subgoals is distributed between several places:

• RenderRow.select field
• Graph.selections() method
• Graph.settings("selection") method

This seems overcomplicated. It would be better to keep this information in a single place. Mostly probably, the better place for it should be a new field in RenderResult class. It should be filled from arguments in RenderResult.__init__. All clients should use it instead of old methods. Old ways to obtain selections should be removed.

Consider layers in 'sieben-manage dot'

Currently, sieben-manage dot exports a whole goal tree from the given file. It's not useful in many situations.

To make its output closer to what user may see in Sieben UI, we should add an opportunity to consider all existing layers and optionally enable/disable existing views.

E.g.:

# by default, shows only open goals (or selected)
sieben-manage dot my.db
# show both open and closed goals (or selected)
sieben-manage dot -n my.db
# show both open and closed goals, but only switchable (or selected)
sieben-manage dot -nt my.db
# show only open (or selected) goals with progress counters
sieben-manage dot -p my.db

Feature drop: this improvement will make sieben-manage unable to mitigate zoom roots. That's OK for now. There is a workaround: to export a full goal tree we could unzoom it in Sieben UI, perform an export, and zoom it back.

Get rid of legacy dict in Graph.q()

A weak-typed Graph.q(keys: str)->Dict[int, Any] lives in code from the very first versions of the application. At the beginning, it provided great flexibility and allowed to write simpler tests. But now, it mostly brakes the development process. We should replace it with a modern pythonic data structure.

A 'keys' argument of 'q' method should also go away during this cleanup. In production code, we always use the same keys list.

What do we expect from this change:

• Improvements of code readability (fields instead of string keys)
• A better support of IDE and mypy (completion, type checks)
• Slightly better performance
• Reduction of cyclomatic complexity (fewer ifs)
• An ability to return richer structure (e.g., containing some metadata)

Use QML to draw user interface

With QML, user interface should become more attractive and nice comparing to native widgets.

RenderRow: add 'attr' dict field for optional attributes

There are several attributes that are added into each goal on some stages and seems not applicable to be RenderRow fields. Examples are x, y, row, col that are added in Renderer.

In addition, there are potential new properties that could be added dynamically to some RenderRow instances: filter properties, autolink properties, zoom root attribute, progress status, and so on. We could use these attributes instead of current "fake goals". This will reduce total amount of nodes and edges shown, increasing understandability of rendered graph. Also, we could finally get rid of RenderResult.graph() method, collecting everything within rows.

Remove support for Python 3.8

Python goes forward, so we should do the same. Set Python 3.9 as the minimal supported version. Improve code style using new syntax.

Edit field may lose focus too early

Edit field may lose focus on many events including language switching (using Super+Space). This makes it impossible to create goals in another language.

Markdown export in sieben-manage

Sometimes it would be helpful to export a goaltree from Sieben database to an external system that support nothing more than formatted text (a kind of wiki or GitHub). The most popular structured plain text format nowadays is Markdown.

So, let's implement a markdown export in sieben-manage. It should work like existing sieben-manage dot, with new enhancements planned to implement in #310.

Auto-link new subgoals to the given goal

Problem: sometimes it would be useful to auto-assign another parent goal when adding a new goal.

An example: I work on system that contains of two modules, foo and bar. While working on foo, I understand that it requires a new feature in bar. To reflect this, I add a new subgoal: "bar: implement feature". But I'm currently zoomed into the "foo" subtree, so the new subgoal belongs to it. To make this goal visible in the "bar" subtree I have to unzoom and manually add a new parent link between "bar" root and my new subgoal. This workflow is far from perfect.

Idea: allow to assign special regexes for any goal. When a new goal is being added, it's name is checked against all regexes we have. For all matching regexes a new blocker link should be created, and one of the regexed goals become a new parent of the given subgoal.

With the given example, I would add "bar.*" regex to the "bar" root goal, and it would become the parent of any new (or renamed) goal which name starts with "bar.."

Replace Graph methods with commands

We try to split different pieces of functionality into separate classes (Goals, Zoom, Enumeration, yet more to come in #67). But the main challenge here grows from the dependencies between different methods. The less each layer knows about others the better.

A good way to achieve this is usually replacing direct method calls with event/command handling. Let's do it

RenderResult: add 'roots' field

While working on #329, I've discovered that there's the same pattern in many places:

1. We obtain a RenderResult instance via q() method
2. We iterate over its 'rows' field or use unreliable tricks in order to find "root" goals - e.g., goals that have no parent ones.

To make things simple, we should try to add roots: Set[GoalId] field into RenderResult and calculate it right in RenderResult.__init__. Then, when we use this instance obtained from q(), start iterating right from its values. Calls to settings("root") should also be replaced with calls to render_result.roots.

There could be several root goals, for example, when we switch on "Only switchable goals" toggle.

CLI mode

I'd like to use SiebenApp in GUI-less environment (e.g., smartphone + Termux). In order to do this, we need to finish and merge changes from the cli branch.

Drop Python 3.5 support

The planned EOL of Python 3.5 should happen at Sep 2020. New versions are wide available. So, it seems reasonable to drop support for the old version of the language.

In addition, this removal will unblock possibility to use Python 3.6 features which could improve readability and/or performance of the application.

Try alternatives to Qt

Motivation: currently, SiebenApp depends on two external libraries that must be installed beforehand: Python and Qt 5. This makes installation and running a bit difficult. Modern Python UI is not restricted by Qt only. It also includes few interesting alternatives. We should take a look at them, so they could make installation and running simpler.

We're looking for the following benefits:

• Rich and performant drawing tools (since we have a lot of custom-driven boxes and lines)
• The drawing engine should be included into python package

Currently, we have the following list of possible alternatives:

• skia-python
• DearPyGui

Zoom/Unzoom

In order to reduce amount of visible goals it would be useful to zoom view to the selected goal. This goal should be shown as new root. All goals that do not depend on the selected one should be hidden.

Unzooming currently should just reset view back to the very root goal.

Allow multiple root goals

Currently, only one root goal is allowed. It's a legacy of my previous experiments (e.g., Clokado) which were aimed to solve a single big goal a time. SiebenApp allow to work with more complex goals, and there is no actual need to stick all of them to the single root goal.

Let's try to allow addition of new root goals. This could simplify overall goaltree height and reduce its complexity.

Bug: focus may be lost after unlink in zoomed mode

Initial structure - like this:

open_(1, Root, [2, 3]),
open_(2, Zoom root, [4])
open_(3, New parent)
open_(4, Top)

Actions:

1. Link 3->4 (parent)
2. Zoom on 2
3. Unlink 2->4

Expected result: both selections belongs to the selected scope (only 2)
Actual result: goal 4 still remains selected (out of scope)

Use embedded sqlite DB for persistence

Current state of goal tree is kept in file sieben.db. Despite of its extension, it's nothing more than raw dump of app's inner state made with pickle module. Therefore it's impossible to keep several goal trees (like several projects and so on) in one file. Also, migrations are difficult to implement.

We should add support for embedded DB instead. I guess, Sqlite would be a good choice.

Hotkeys help

Currently all actions are available only via keyboard. And it's impossible to see full list of actions from the application. We should add a simple dialog window containing such list.