interacto/interacto-ts

Front-end framework for processing user interface events in TypeScript

Interacto

The TypeScript implementation of Interacto.

Interacto is a front-end framework for processing user interface events.
With Interacto developers handle user interactions (DnD, drag-lock, double-click, button click, pan, multi-touch, etc.) instead of low-level UI events.
Developers configure how to turn a selected user interaction into a (undoable) UI command using a fluent API.
Interacto also provides a native support for undo/redo operations.

See https://interacto.github.io for usages and documentation.

How to install

For Angular 15 and 16:

npm install interacto@latest

For Angular 13 and 14 or rxjs 7:

npm install [email protected]

For Angular 12 or rxjs 6:

npm install [email protected]

For Angular, you may have to install the dedicated interacto-angular library as well.

interacto-ts's Issues

Interacto binding must have unique ID

Interacto binding must have unique ID to be identified in logging messages.
Those ID must be inferable (eg interaction name + command name) and not randomly generated at run time (which would prevent the clear identification of the bindings at run time).

Make routines cumulative

• first
• then
• end
• cancel
• endOrCancel
• ifHadEffects
• ifHadNoEffect
• ifCannotExecute
• catch

Remove single instances

Single instances are global variables/constants.
A correct design should rely on dependency injection.

To investigate:

• For Angular, we should create an Angular library that relies on the Angular DI. Create an Angular module.
• Create an Angular lib to demonstrate the ability to use the Angular DI within Interacto.
• For VueJS, look at their DI system.
• For React, look at their DI system.

The single instances / global variables to remove:

• CommandsRegistry
• UndoCollector
• observer in Bindings.ts

This may lead to the encapsulation all the routines located in Bindings.ts in a class injected in, for example, Angular components (similarly to other Angular Modules).

Add methods in MultiTouchData

We can add various methods in MultiTouchData.

• isVertical(pxTolerance) that returns true whether the line of each touch are relatively vertical in the same direction. pxTolerance stands for the pixel tolerance for considering the line vertical. Will use the related methods to add in SrcTgtPointsData
• isHorizontal(pxTolerance) that returns true whether the line of each touch is relatively horizontal in the same direction.

The with routine uses key code only: this causes issue with different layouts

For example: with('KeyA') works when the q key is pressed with a French keyboard.
A workaround is to change the prototype of with to allow both code and key.

with(...keys: Array<string>, codes?: Array<string>)

Adding various methods in SrcTgtPointsData

We can add various methods in SrcTgtPointsData.

• isVertical(pxTolerance) that returns true whether the line between the two points is relatively vertical. pxTolerance stands for the pixel tolerance for considering the line vertical. The methods isStable in Pan.ts will help.
• isHorizontal(pxTolerance) that returns true whether the line between the two points is relatively horizontal
• duration that computes the duration of the move (in milliseconds)
• velocity that computes the velocity of the move distance / duration
• write tests

These methods will be useful to refactor the pan, swipe, etc. interactions.

Support of selective undo history

Principle

This allows users to cherry-pick on command to undo in the history. Undoing such a picked command does not undo all the command performed after, i.e. it is not an n-time undo operation.

Selective undo history can be linear or tree-based.

Note that the behavior of the undo algorithm may depend on the semantics of the command.
Some commands of the same type, if executed successively, are cumulative. Example: a move command (drawing app) based on a translation vector (relative position).

Similarly, some commands replace. Examples: a move command based on a absolute position; changing the color of a shape.
With replacing commands, eg two change color commands on the same object (so regional history too), if the user undoes the first change color command, what to do with the next ones? And how to redo this command (and reinsert of related commands)?

Challenges

Conflicts/dependencies between commands

Example 1: drawing app. The user moves two times a shape. Then the user cherry-pick the first move command and undoes it. What happens to the second move command since it depends on the first one?

Support transient touch interactions

See https://inria.hal.science/hal-01714269/file/pn1142-averyA_noCR.pdf

feat(routine): Add an onError binding routine

The goal of such a routine is to capture errors in bindings definition for specific processing.
Currently, bindings log errors (exceptions).

swipe should cancel when more touches used than expected

If the swipe is defined to use one touch, and the user swipe with two touches, the interaction should be cancelled.

New Logging System

Implement a new logging system that fits Interacto needs:

• log interaction, binding, command messages, errors
• compute fitts' law on demande
• compute/provide UI usage information (paths, etc.)
• send data to a back-end (so support privacy issues)
• support the bridge pattern to ease testability and customisation of loggers

Will remove typescript-logging dependency

Implement the selective undo history for painting app

https://research.yyoon.net/papers/chi15-bam-Aquamarine.pdf

• check the potential different with the code editor history (#40)
• implement the painting algorithm to be use on angular-example

Unit testing interaction data implementation classes

• KeyDataImpl
• KeysDataImpl
• PointDataImpl
• PointsDataImpl
• TouchDataImpl
• TapDataImpl
• MultiTouchDataImpl
• ScrollDataImpl
• SrcTgtPointsDataImpl
• SrcTgtTouchDataImpl
• WidgetDataImpl

Add a class `UndoableCommand`

The goal of this class is just to implement by default several methods from Undoable.
This abstract class should extend CommandBase and implements Undoable and implements by default:

• getVisualSnapshot see #7 with undefined
• getUndoName with the name of the class (introspection)

Add Touch DragLock

Touch Draglock is missing

Support of regional undo history

Get the history related to a given object, instead of having a unique global history

Use cases:

• drawing app: select an object. Its history is extracted from the global one to be then display
• text editor: from a given caret location, the history related to the position (more or less) is extracted from the global one to be then display

This is a specific case of selective undo as undoing the latest command of a given object may mean cherry-picking this command from the global one.

They are various corner scenarios / challenges:

Composite commands

Illustrative example with a drawing app: the user moves 3 shapes. One command that moves those 3 shapes is produced. The user selects one of those three shapes. Its history is displayed. The user undoes the move command => how to undoes only part of the initial command? (part of the solution: composite pattern)

Producing regional histories

How to extract from a global history a history specific to a given object? Part of the solution: an undoable command has a method that returns the associated object. By default returns undefined or [].

First example with the drawing app. The user draws several shapes and then selects one of them. The regional history asks the global one commands which associated object is the data object corresponding to the model of the selected shape.

Second example with a text editor. The user types some text: three successive commands "Supporting" "the" "regional undo".
Then the user selects the "the" text and put it in red. Finally, the user put the caret on the "the" word. The commands related to the given caret position form the regional undo.

Managing regional conflicts

Using the previous example of the text editor, the user undoes the command that produced the "the" word. What to do with the command that put this word in red? Part of the solution: identify command that creates objects, so that cancelling such commands removes the related command from the global history.

Second example using the text example. The user selects the whole text at put it in green. Then the user undoes the creation of the word "the". How to manage the green color command (composite command?)?

Redoing cherry-picked creation command

Using the previous example of the text editor. The user now wants to redo the creation of the word "the" (so that it includes the re-integration of the color command). How to access regional history of removed objects?

DnD-1: the spring does not update after a move

If I pause the DnD to let the spring appear, and then a re-move again, the spring is still here. After another pause the spring does not update.

The when` routine may be run once (at start) in some cases

In some cases we do not what to execute the when all the time. But just at the beginning to check some preconditions.
We may support that by adding an optional parameter to the when routine:

when(fn, once?: boolean)

Since we can cumulate the when routines, we can write several when routines to cover different cases.

Investigate how to add animations to user interactions

Examples:

• how to add a timeout circle around the cursor for longtouch interactions
• how to add press/tap effects

Should effects be added to user interactions or to binders?

Be able to compare commands to identify equivalent redo

After an undo, a user can perform an new command. If this new command is equivalent to the redo one, this is not necessary to:

• create a history branch (tree-based history)
• clean the previous timeline (linear history)

Example: sudoku game. Undo a set value command and then perform the same than the undone one.

Shortcut for producing AnonCmd

Writing:

binder
  .toProduce(() => new AnonCmd(() => {
    ...
  }))
  .bind();

is verbose.
Maybe we can shortcut that, for examples :

binder
  .toProduce(() => {
    ...
  }))
  .bind();

where the arrow function is of type () => void so that is detects it is an AnonCmd to produce. Have to investigate ambiguities.

Or:

binder
  .toProduceAnon(() => {
    ...
  }))
  .bind();

Investigate the use of the painting history for reciprocal DnD

Currently the reciprocal DnD can cancel the ongoing interaction and command.
Using the painting history (#41), we may extend the reciprocal DnD to support prior move commands done on the selected shape

Visual/textual snapshot of undoable commands

Investigate the possibility for a command to produce a visual or textual snapshot of the undo/redo states.

The idea is to add a novel method in Undoable:

getVisualSnapshot(): string | SVGElement | undefined;

The default implementation of this method should return undefined.

Developers can override this method in their command to, for example, get some information about the impact of the undoable command (more than the mere method getUndoName that just returns the name of the undoable).

Scenario with example-angular: one mouse-over undo buttons (in the undo history panel) to have a thumbnail depicting the impact of the command. Example: a text diff for SetText.

So:

• Add this new method getVisualSnapshot in Undoable (with documentation)
• Implements it by default in `UndoableCommand

For image (to see later):
For canvas, see toDataURL:
https://stackoverflow.com/questions/6887183/how-to-take-screenshot-of-a-div-with-javascript

For other elements:
https://www.npmjs.com/package/html2canvas

Investigate how to have touch or mouse binders

Currently a mouse binding does not work using a touch screen.
We should investigate how one can write binders like that:

longTouchOrPressBinder(100)...
touchOrPressBinder()...
tapOrClicksBinder(3)...

Implement pinch binder

Use the multi-touch interaction to create a predefined pinch binder

Logger: have a usage logging level

We need a specific logging level for logging in usage information.
We have to investigate how it can work with the standard logging messages.

Usage focuses on bindings usage. So just have to log binding information:

{
  "id": "...",
  "start": 0,
  "duration": 1,
  "cancel": false
}

POST /api/interacto/stats

Implement the reciprocal Drag-and-Drop interaction/binder

From research paper:
https://hal.archives-ouvertes.fr/hal-01185805/document

The goal is to implement this user interaction either as a novel user interaction class or a specific redefined binder.

New properties in `SrcTgtPointsData`

In interaction data SrcTgtPointsData we want to provide six new computed properties:

• diffClientX and diffClientY
• diffPageX and diffPageY
• diffScreenX and diffScreenY

For example diffClientX computes tgt.clientX - src.clientX. ie the translation vector.

These new properties must be documented and tested.

Implement the selective undo history for code editor

https://kilthub.cmu.edu/articles/Supporting_Selective_Undo_in_a_Code_Editor/6470474/files/11899031.pdf

See his github repo for more details on the implementation. Exemple:
https://github.com/yyoon/azurite-eclipse/blob/develop/edu.cmu.scs.azurite/src/edu/cmu/scs/azurite/model/undo/SelectiveUndoEngine.java

Add pinchFactor in MultiTouchData

MultiTouchData should contain a method pinchFactor(pxTolerance: number): number | undefined

This method will compute the pinch ratio made by a two-touches interaction.
The principle of a pinch:

• the two touches must get closer
• the two touches must follow the same line but in opposite direction (to get closer)
• the ratio is computed as the distance between the end position of the touches / the distance between the starting position of the touches
• if the two touches do not get closer or do not follow the same line, the method will return undefined.

Move to TypeScript 4.3

TypeScript 4.3 just released:
https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-3.html

• update TypeDoc npm install typedoc@latest (have to wait for an update of it)
• update eslint typescript npm install @typescript-eslint/eslint-plugin @typescript-eslint/parser
• npm install typescript
• Let mandatory the use of the new override keyword: "noImplicitOverride": true, in tsconfig.json
• Fix all the compiler errors by adding override to methods :s

Touch DnD continues to run with several touches

Even if the touch DnD must stops on several touches, in a specific case it goes on:
The first touch is on a shape, and the second on the containing canvas.

Add a routine `name` to name bindings

Developers should be able to name their Interacto bindings.
To do so, we can add a dedicated routine name.
To be used to identify bindings in the logging system

Can we compute Fitts and Finger-Fitts' laws at run time?

Investigate the possible computation of Fitts' law value during the execution of a binding.
Also see Finger-Fitts' law.

May be integrated within the logging system.

Add the commands `HistoryGoBack` and `HistoryGoForward` to Interacto

The commands HistoryGoBack and HistoryGoForward are defined in the example-angular app.
We should move it in Interacto with the name UndoNTimes and RedoNTimes.

Implement pan binder

Use the multi-touch interaction to create a predefined pan binder