Settings ui extension side

Needing a voice command for every setting is inconvenient and even more so the more settings that are added

Wrong hint position when an iframe pushes the page down in quirks mode

For example, when the google translate banner appears in a page all of the hints are move down

Dismiss only closes the last hovered element

Hovering one element should also close the last hovered element

Add ability to copy links to clipboard

This would be a really great addition and is pretty much one of the last main features I used from Vimium.

It would be nice to have a subtle signal/notification, maybe by the alphabet characters you just used, that the action was successful. Kinda like this but aligned with the size and style of rango.

Timing issues with updating tab title

I am not sure what could be going on here but when I visit this website: https://registry.terraform.io/providers/hashicorp/aws/latest/docs

The title ends up being - https://registry.terraform.io/providers/hashicorp/aws/latest/docs

Which smells like a race condition with updating the window title, maybe some client side code by the website VS rango.

In a different browser I can see how the tab does have an original title, which is then updated after some time presumably by some client side code.

It starts out as:

Terraform Registry
and then is updated to
Docs overview | hashicorp/aws | Terraform Registry

But in Firefox with rango it just ends up being: - https://registry.terraform.io/providers/hashicorp/aws/latest/docs

Can't submit to the Chrome Web Store due to using Manifest v2

No more v2 extensions are allowed in the Chrome Web Store

Add settings to remove hints when scrolling

On many pages, slack for example, the scrolling behavior with hints are distracting.

Missing hints from iframe

I have only just seen this now and haven't dug into how when it is happening, highlighted blue area does not have hints and it is an iframe.

Support for clicking on multiple targets

Both for direct and explicit clicking. Probably the easiest is for talon side to always have multiple targets and then sort that out extension side.

Fix bug with duplicated hints

In gmail for example I occasionally have a duplicate hint on the right hand side panel

Create command to focus on an input field and clear its content

Use talon action to set clipboard text

Makes it so that clipboard managers in talon get updated

https://github.com/david-tejada/rango-talon/blob/ea32cd017701691e9d7cf6a0a794c47a6b3af347/browser.py#L140

actions.clip.set_text(response["action"]["textToCopy"])

The clipboard updates for the communication should still use clip directly to not spam clipboard managers with rango commands.

Support targets where click event was added with addEventListener

Example of this is view replies for 9gag comments

Possible solutions
https://stackoverflow.com/questions/9046741/get-event-listeners-attached-to-node-using-addeventlistener

Don't show hens for disabled elements

Being able to click hints for disabled buttons is not useful

Open multiple links in new tabs in one utterance which don't steal focus

Related to #24 but it seems like a large enough issue to have tracked separately.

Once we can open links which don't take the focus of the window it would be cool to be able to open N number of links in one Talon utterance.

With rango direct clicking enabled, and hints a,b,c,d present on the page.

Saying *stash air and bat and cat would open the three links identified by those hints in new tabs but the original tab remains the focussed one.

*stash to be replaced with whatever the outcome of #24 is.

Better support for video players

Youtube for example. Not all controls are clickable and it would be nice if we could figure out a better way to do scrubbing the timeline.

Hints capture should use rago_hint instead of letters

Right now the hints capture supports more than two letters. Better would be to just reuse hint capture.

@mod.capture(rule="<user.rango_hint> (and <user.rango_hint>)*")

https://github.com/david-tejada/rango-talon/blob/ea32cd017701691e9d7cf6a0a794c47a6b3af347/browser.py#L65

Hints rendering in the wrong positions on some websites after scrolling

Setting to use constant hint colors

I would like a setting to enable user specific colors for the hint text, border and background

Hints look a little jittery/twitchy since 0.2.0

In the process of debugging #28 I checked out version v0.1.9 at 57f216a. In unrelated to that issue I did notice that the v0.2.0 version does result in the hints being a little more jittery than version version v0.1.9 when scrolling.

I used this page for testing: https://en.wikipedia.org/wiki/Main_Page

I had noticed this when v0.2.0 was released but didn't bring it up, going back to the older version it is pretty clear though.

I am unsure how the rendering/positioning/throttling code has changed between versions and this may be a conscious decision compromising between smoothness and performance. But thought I would bring this us up if not.

Here is a quick screen recording showing how it looks for me: https://www.youtube.com/watch?v=AEfzh33mrjY

Hidden radio button but visible label not showing hint

For example, here in sort packages

Ability to define whether or not links opened in new tabs become focussed

I think this means being able to set the selected value https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/tabs/create when creating the tab here

rango/src/background/tabs-messaging.ts

Line 90 in b90edaa

await browser.tabs.create({

as it defaults to true.

I am unsure what to suggest for the Talon command, and whether or not it should share the existing blank ... command with an extra qualifier or if it should get a new one. I think I'm leaning towards a new word, some for consideration:

stash air
load air
set air
store air
pack air

Trying to think of words which invoke imagery of stashing something away for later use as is common with opening links in new tabs

Fix hint positioning when clickable element spans multiple lines

Don't show hints on disabled elements

Being able to click on disabled buttons is not useful

Direct clicking when focussed on input field inputs `Å` - doesn't click requested element

Having seen the recent change where you demonstrated clicking a link while focussed on an input field I tried it out. Though when I do, it doesn't quite work, when focussed on an input and saying the letter for a hint the request is sent to rango, the letter Å is typed into the input field, and then Talon reports a timeout waiting for response.

I'm on macos 12.3.1, using up-to-date Firefox.

Extensión doesn't work on Chrome due to a clipboard api bug

Due to this bug I'm not able to read the command send from talon through the clipboard from a background script. Once we have talon rpc that won't be necessary but in the meantime I'll have to move the code that reads the command to a content script.

Ability to hide rango hints per tab

#20 means that I frequently have to hide/show rango hints when jumping around tabs as I spend a lot of time in Confluence and Jira where this is a problem. It would be great if I could toggle the hints per tab to not have to deal with that.

Also a sidenote, I find when I toggle the rango hints it seems to cause some significant work for my browser and it lags for a bit. I wonder if this is due to having multiple tabs open and rango having to do a lot of work to show all the hints after them being toggled off? Being able to toggle per tab would also help with that.

Support mouse up and mouse down click events

Today these hints are not clickable

Implement missing vimium features

I would like to completely replace vimium with rango, but there is some features I still use that are vimium explicit.

Links which copy things to clipboard don't always work due to lack of user activation

I found this random page you can see the issue from https://github.github.io/clipboard-copy-element/examples/

have something in your clipboard - abc for example
find a button/link which adds something to your clipboard - eg. xyz
click the button/link with rango
paste clipboard contents to see what is in it.

Result will be abc, when the desired contents is xyz.

When rango receives a request and performs its action, before it writes the response message it could inspect the clipboard again and if it isn't still the original request then send the current contents back to talon in the response object hopefully allowing Talon to set this as clipboard contents and not the original content before the action.

Copy arbitrary text from webpage

This has been mentioned a few times in Slack I think but thought it wise to get an issue open for discussion and ideas!!

Usecase

Being able to fluently copy portions of a webpage to the clipboard.

Possible Issues

Given the problems described in #18 do you know if there will be issues copying portions of the webpage to the clipboard?
Performance, marking up even more of the page could be quite the performance hit.
How to know what characters/words/tokens are visible in the viewport - this seems difficult to do performatively, or at all.

Ideas

Here are some ideas about how to reference areas of a webpage:

Cusorless is the obvious example here. This does seem like a significant amount of work though and in the interest of thinking of other interactions lets continue...
Referencing words in the current viewport literally, for text The quick brown fox jumps over the lazy dog, take quick past fox to select/copy quick brown fox
Copying/selecting lines of text - each line could be marked up with number/letter/color hints
Use containing elements like div or other HTML elements (see image below where I added an outline to all div elements)
Leveraging the existing vscode draft capability, where you could copy a chunk and then refine more in vscode

Inspiration

A list of some other tools for inspiration:

Dense Grid reference screen with number and letter combinations
Cusorless 🎩

UX

Perhaps there should be two commands:

Selecting text only - possibly allowing for further refinement
Copy the requested text to clipboard directly.

Given most of the time spent using a web browser would be reading and navigating it seems that any mode that would allow you to copy text would be something that you had to switch into to allow for more markup on the page and actions to become available. Though this would impact the fluency aspect.

Support iframes

Currently we don't have support for iframes. They are complicated. Vimium has support for them so they has to be a way to solve this issue

Editing a rich text field which contains a rango hint submits the hint contained when submitting the form

I have only witnessed this in Jira but have not gone looking to replicate anywhere else yet.

If you are editing a rich text input area which has a link in it, it is marked up with a rango hint bq

If you then submit the form the hint letters end up in the saved content - bq

Fix hovering timeout when hovering another element before the first one times out

If you hover over one element and, before the timeout removes the hover, you hover over another element, this other element only stays hovered for they remaining time that the first element had remaining.

open link in new tab

it would be nice, if you could open links in a new tab
like control+click

Ability to copy clickable element text to clipboard

Rango has the ability to copy link urls to clipboard.
I think it would be useful to be able to copy the clickable elements text content to clipboard as well.
For example:

https://docs.aws.amazon.com/rds/index.html

copy text pit => Performance Insights in the AWS CLI Reference would be put in the clipboard.

This is getting a bit into cursorless style territory but maybe not too much to implement.

Not able to click element in google mail - other UI elements aren't markedup with hint

I haven't noticed this anywhere else yet but I have problems selecting the checkbox in the top left of google mail to select all unread mail. Saying the letter hint performs the animation of the hint but the checkbox is not selected. This does work with Vimium but I haven't looked at what could be the difference yet.

Another thing I saw was that in the google mail UI some clickable elements with Vimium aren't marked up with hints in rango

Rango:

Vimium:

Url in title doesn't update when clicking on a hash link

For example, if you go https://github.com/david-tejada/rango/blob/main/readme.md and click on the header "installation", the url in the title doesn't update to https://github.com/david-tejada/rango/blob/main/readme.md#installation

Setting to use upper case hint letters

I think that upper case letters would be easier to distinguish

Response schema does not always include action type - Talon errors in logs

This line expects a certain response schema that rango isn't consistently using for all action types: https://github.com/david-tejada/rango-talon/blob/adc4905d84538a5544b90fba25548b0f0634af31/browser.py#L67

This results in the Talon logs having errors:

2022-05-01 10:44:51    IO {'type': 'response'}
2022-05-01 10:44:51 ERROR     7: talon/scripting/talon_script.py:592| 
    6: talon/scripting/talon_script.py:297| 
    5: talon/scripting/talon_script.py:559| 
    4:      talon/scripting/actions.py:85 | 
    3:        talon/scripting/types.py:411| 
    2:     user/rango-talon/browser.py:115| execute_command(command)
    1:     user/rango-talon/browser.py:69 | if response["action"]["type"] == "copyLink":
KeyError: 'action'

[The below error was raised while handling the above exception(s)]
2022-05-01 10:44:51 ERROR cb error topic="phrase" cb=<bound method SpeechSystem.engine_event of <talon.scripting.speech_system.SpeechSystem object at 0x110d91c20>>
   30:       lib/python3.9/threading.py:930* # cron thread
   29:       lib/python3.9/threading.py:973* 
   28:       lib/python3.9/threading.py:910* 
   27:                    talon/cron.py:155| 
   26:                    talon/cron.py:106| 
   25:                     talon/vad.py:21 | 
   24: talon/scripting/speech_system.py:355| 
   23:             talon/engines/w2l.py:706| 
   22:      talon/scripting/dispatch.py:105| 
   21:      talon/scripting/dispatch.py:144| 
   20:      talon/scripting/dispatch.py:133| 
   19:          talon/scripting/rctx.py:233| # 'phrase' main:_redispatch()
   18: talon/scripting/speech_system.py:62 | 
   17:      talon/scripting/dispatch.py:105| 
   16:      talon/scripting/dispatch.py:144| 
   15:      talon/scripting/dispatch.py:133| 
   14: ------------------------------------# cron thread
   13:          talon/scripting/rctx.py:233| # 'phrase' main:engine_event()
   12: talon/scripting/speech_system.py:407| 
   11:       talon/scripting/actions.py:85 | 
   10:         talon/scripting/types.py:411| 
    9:     talon/scripting/core/core.py:100| 
    8:       talon/scripting/actions.py:85 | 
    7:         talon/scripting/types.py:411| 
    6:     talon/scripting/core/core.py:129| 
    5:       talon/scripting/actions.py:85 | 
    4:         talon/scripting/types.py:411| 
    3:     talon/scripting/core/core.py:134| 
    2:  talon/scripting/talon_script.py:693| 
    1:  talon/scripting/talon_script.py:596| 
talon.scripting.talon_script.TalonScriptError: 
 in script at /Users/rickymcmillen/.talon/user/rango-talon/direct_clicking.talon:4:
 > user.browser_click_hint("{letter_1}{letter_2 or ''}", 0)
KeyError: 'action'
2

Could probably work around this in the Talon browser.py but I figured it would be better get rango to use a consistent response schema so logging it here.

Thanks again for your contributions!!

Add keyboard shortcut to `hints toggle` command

This will be a lot helpful if we don't have talon enabled and want to quickly hide all the hints

User feedback on non visual commands

Commands like copyLocationProperty have no visual feedback that they completed. It would be nice to get something.

Clicking on an input field should leave the cursor at the end

Change the color of the hovered hint while it is hovered

Move to multiple hints container inside the first positioned ancestor so that hints always stay with their element

Related to #38.

I have been giving some thought to this and I think it's possible to have hints always be next to the element they're hinting. Instead of having just one hints container every hint would be inside a hints container within the first positioned ancestor (position other than "static"). This would require a considerate rework of the way hints are place in the page but I think it's worth it. The advantages are:

Hints don't jump around when scrolling.
No need to calculate if the element is obscured as the hint would be in the same stacking context.
Hints wouldn't appear over other elements as in #10.
Hints would display even if there is an invisible overlay in the page.
Performance gain as there is no need to recalculate position with every scroll/intersection.

Position hints would work something like this:

Find the first positioned ancestor of the element. Store this element in intersector for future need.
Store the dimensions of this element to reposition hints in case the element resizes.
Find out if there is a hints container within that element and if not create it and place the hint inside. The hint will be position absolute and top and left can be calculated using getBoundingClientRect on the container and the element hinted.
On scroll check for new fixed or sticky elements added with javascript and reposition the hints of elements within.
On scroll/intersection/input event check if the containers have been resized and in that case reposition hints.

A command for scrolling an specific element: We will need to find the closest ancestor with overflow.
Add commands for scrolling an element to a certain position (top, center , bottom)

Add title attribute to show action

Would be nice if we could get the title attribute as well when we do the show action

Support for exclude singles grammar

When exclude singles is enabled the grammar still triggers on single letters. Maybe a talon setting would be sufficient?

david-tejada / rango Goto Github PK

rango's Issues

Usecase

Possible Issues

Ideas

Inspiration

UX

Recommend Projects

Recommend Topics

Recommend Org