josephsl/wintenapps

Windows App Essentials NVDA add-on

About me

Originally from Seoul, South Korea, I am a graduate student at University of Colorado Boulder, speech and debate coach, volunteer software engineer, and an accessibility advocate. My interests are operating systems, Python, assistive technologies, screen readers, software source code as rhetoric, and organizational communication.

What I do

• Nonvisual Desktop Access: I have been a part of this screen reader project since 2012, starting out as a translator then moving onto publishing add-ons and pull requests. Some of the highlights of my work include wxPython Phoenix (version 4) research, Python 2 to 3 transition, ongoing work on Windows 10 and 11 support, and launching NVDA Users and Developers Conference (NVDACon) in 2014.
• College teaching: currently I serve as a teaching assistant for an introductory communication studies course at CU Boulder. Prior to this, I coached competitive speech and debate to college students.
• Windows Insider: I have been part of Windows Insider Program since 2014 and have been testing preview releases of Windows operating systems, with an emphasis on providing feedback that centers accessibility of various features for people with disabilities.

Keep in touch

You can contact me via email, LinkedIn, GitHub, and social media channels such as Twitter/X.

wintenapps's Issues

Support for Developer Tools in Internet Explorer

Hi,
I'm an italian blind developer.
I use eclipse and notepad++ for more time.
In the front end application, I have the problem to debug or simple explore javascript file, add breackpoint and debug it.
After a lot of tools testing, browser testing and addons testing, for me, the best solution is to use the internet Explorer developer tools integrated in this browser.
There are a lot of hotkey at this page https://msdn.microsoft.com/en-us/library/dd565630(v=vs.85).aspx
The only problems are in the integrated editor, sometimes nvda lost the focus, and when go up or down in the editor, nvda read a not editor or source code info, but for example menus info or popup info.
For a developer, the js debug is a very important feature.

Is possible to fix it in the future version of App Essentials addon?

Thanks a lot.

Auto-suggest: announce suggestions count when it changes

Hi,

Currently NVDA announces suggestions count when controller for event is fired, whereas Narrator says "suggestions" when suggestions list changes.

Technical: this was traced to UIA layout invalidated event - NVDA does not recognize this event at the core level, therefore add a base implementation in the add-on. Note that this means increased verbosity, especially when people are searching for things in Microsoft Store.

Thanks.

Allow speaking of the street side name in maps

Assign this to me if you'd like.

1. open maps and go click show my location, and then street side.
2. objnav to the map graphic.
3. drill in once.
4. If there's no street announced, tab until "Use keyboard to move to a different Streetside location" is announced.
5. Press up arrow.
6. objnav to one below the map.
7. keep trying until you see a street name.

I propose reading street names out as they change with cursor keys.

Add support for dragging events

Hi,

Noted by several users: when moving tiles in Start menu, NVDA won't announce new tile position.

Technical: this is because drag events are not supported by NVDA. Thankfully, when tiles do move, drag start and drag complete events are fired. For now, handle the completion event.

Thanks.

Synchronize wintenApps simplified Chinese translation

Dear NVDA Translation Community
After a period of internal community testing, I am ready to synchronize the simplified Chinese translation of wintenApps to the NVDA translation community.
Finally, I thank every friend who gave me advice during the translation process.
zh_cn.zip

WinTenObjs: do not initialize the global plugin class if running on older Windows 10 releases

Hi,

Although this won't affect user experience for newer Windows 10 builds, it is still a major problem:

Imagine a user inserting a USB flash drive on an older Windows 10 PC and runs NVDA stored on the flash drive. If a recent Windows 10 App Essentials add-on release is installed, NVDA may appear to do nothing or keep playing error tones.

Cause: numerous COM errors, ultimately caused by attempting to access certain UIA values only available in newer Windows 10 releases.
Solution: check the build as well, and if it is less than minimum version tuple (major, minor, build), just quit.

Thanks.

Easily movement in actual forecast in microsoft_msn_weather

Hello. Would it be possible to add easily movement in actual forecast? I have to use object navigation for reading forecast. It could be for example weather forecast listing in edit field, when I press some shortcut key. Thanks, Michal.

Introduce add-on update facility

Hi,

Because this add-on is released frequently, it would be helpful to let people update this add-on straight from the add-on config interface. The channel value will be stored as part of the add-on manifest in order to not disturb NVDA Core config.

Thanks.

Event tracking: record if a UIA property event is dropped for this object by the app module

Hi,

Follow-up to nvaccess/nvda:10508:

When recording events for UIA objects, if appModule.shouldProcessUIAPropertyChangedEvent exists, record if the chosen property change event is dropped for this object (do nothing otherwise). This will help developers figure out if an event isn't working due to this being the case.

Thanks.

Alarms and Clock: announce selected value for looping selector list

Hi,

Values for looping selector in Alarms and Clock are not announced.

STR:

1. From Alarms and Clock, press Control+N for new alarm.
2. Go through time pickers.

Expected: Values are announced.
Actual: Just the list label is announced.
Ideal: Announce both the label and the currently selected value.

Details: try not using obj.value function for this, as it could introduce double speaking. instead, modify obj.reportFocus.

Thanks.

Security: prevent add-on from checking for updates and adding UIA events while in secure screens

Hi,

A user reports:

In secure screens, the add-on should not check for updates.

Confirmed - the add-on should not add UIA event handlers for additional events in secure screens as well for consistency.

Thanks.

Skype app/message history: shortened message entry must stay on a braille display

Hi,

Reported by @jcsteh:

In Skype app's message history list, when reportFocus is invoked on message items, braille times out.

STR:

1. Run Skype UWP and go to a conversation.
2. Tab to message history.
3. Press Shift+TAB to go to Add People button, then press TAB.
4. Scroll through the message history with a braille display connected.

Expected: shortened message item should remain on the braille display no matter what.
Actual: message item disappears from the display after a timeout.
Cause: use of ui.message.
Short-term solution: do something about obj.name.
Long-term solution: UIA tree must be modified to expose all parts of a message item such as sender's name even if the message was sent by the user (currently it isn't).

Thanks.

Mail: Suppress "edit" announcement in message body

Hi,

Reported by many users: In Mail app, NVDA says "edit" each time one presses up or down arrows.

Technical: This is same as NVDA Core ticket 6271. A temporary workaround (changing role) will be included until Edge RS1 branch shows up in NVDA Core.

Thanks.

Alarms and Clock: double announcement when selecting time in February 2017 update

Hi,

A new Alarms and Clock app update rolled out in February 2017. While it resolves UIA selection event, this results in double announcement when selecting date and time.

Technical: Proper UIA events are fired, and UI structure has changed .For backward compatibility purposes, the custom looping selector overlay selection has been modified to handle old and new interfaces.

Thanks.

in skype uwp pressing end doesn'still reads skype message

1. press end in the messages list

expected: shorter text

Actual: longer text, I.E. with "skype message"

Calender app: Announce name of day in month overview

Hi there.

I was hoping it could be implemented, so NVDA would announce the day of week when going through the month overview in the calender.

As an example:
Right now NVDA says:

• 2. Maj 2021, no events.
• 3. Maj 2021, 1 events.

And I was hoping it could say something like.

• Sunday, 2. Maj 2021, no events.
• Monday, 3. Maj 2021, 1 events.

ps: I don't know if the english examples above is what it says, word for word, but I hope you get the idea.

If this is already possible, please do tell me. I've searched high and low, but don't think this is possible.

Take care

Install tasks: provide different warnings on different builds

Hi,

Although it will not impact current add-on releases, it will become noticeable in 2023 and beyond:

Windows 10 (and also, Windows 11) falls under modern lifecycle policy from Microsoft. This means features can come and go without notice, and major releases are made at least once a year. Historiclaly, Windows App Essentials supports feature updates on active support (as of time of this writing, the oldest actively supported feature update is Version 1507, although for consumers, it is 2004)..

Windows App Essentials works best with client releases. Even then, it might be possible that the add-on might get installed on server systems where many universal apps are not present. In the past, client and server releases had the same build number, but Server 2022 uses build 20348, which might not be the one Windows 10 21H2 clients may get.

Adding to this picture is Windows 11 and feature update support duration. Feature updates for Windows 11 will be supported for at least two years (two for consumers, three years for enterprises). This means a "whole" may exist where Wiondows 10 is supported while an old Windows 11 release may not. The first instance of this will happen for consumers in 2023.

Specifically:

• 2021: Windows builds 19042, 19043, 20348, 22000
• 2022: 19043, 20348, 22000, next Windows 11 build
• 2023: a Windows 10 feature update build, 22000, next Windows 11 update, next+next Windows 11 update
• Late 2023 to 2024: a Windows 10 feature update, next and next+next Windows 11 update

This means, by 2024, Windows App Essentials cannot simply check for oldest feature update build.

Therefore enhance install tasks by:

1. Present different install messages: one on older Windows feature updates, one on server support, one on Windows 11 feature updates support.
2. allow installation on server systems provided that users are warned about lack of features.
3. If trying to install on unsupported Windows feature updates across versions (10/11), present the usual error message.

In case of feature update checks, both Windows 10 and 11 releases must be checked from global plugin portion, too.

Because the eventual implementation relies on WinVersion constants introduced in NVDA 2021.1, it won't be released to stable version users until fall.

Thanks.

Skype Preview: typing indicator text isn't announced in recent app releases

Hi,

Discoverd by @derekriemer and confirmed: in recent Skype Preview versions, due to UI placement and other changes, typing indicator announcement does not work properly. One solution is to provide an overlay class just for this object, but the ultimate solution should come from Microsoft.

Technical: no UIA automation ID for typing indicator, and no useful event (apart from offscreen flag) is fired. Ideally, automation ID should be there to let Narrator, NVDA and other screen readers take a look at this text and announce changes, along with a suitable event to let screen reader users know change in typing indicator text.

Thanks.

IUIA5: NVDA 2017.4 does not come with IUIAutomation5 interface in UIA handler

Hi,

Received multiple reports regarding this: if recent add-on releases are run from NVDA 2017.4, it won't work because of missing IUIA5.

As this is a critical issue, an immediate fix will be deployed. Thanks.

NVDA 2017.4 does not support IUIAutomation5

Hi,

Reported by at least two users: WintenApps 18.03 causes NVDA to say "task switching" and does not let it announce update check messages.

STR:

1. Install WintenApps 18.03 on NVDA 2017.4.
2. start using it.

Expected: no issues.
Actual: UIA support breaks.
Cause: Due to perhaps older version of the SDK in use, IUIAutomation5::IUIAutomationNotificationEventHandler isn't included in NVDA 2017.4. This attribute is part of 2018.1 (thanks to newer SDK in use), so make a note of this.
Issue type: regression (immediate patch required).

Thanks.

Remove additional dialog classes check

Hi,

In older Windows and app releases, dialogs were not recognized as such. To get around this, Windows App Essentials defined a list of dialog classes for elements that should be recognized as a dialog. Starting in Windows 10 1809 (October 2018 Update), UIA6 (specifically, IUIAutomationElement9) includes a boolean to specify if a window is a dialog, allowing NVDA and other screen readers to read dialog content automatically.

When first introduced, not many apps took advantage of the boolean property. In recent years, more apps (and Windows itslef) are taking advantage of this property, making NVDA read more dialogs. Therefore the list of dialog classes has shrunk to a point where there is none as of August 2021. Therefore deprecate and remove additional diaog classes check from the global plugin portion.

The implementation will be divided into two parts:

1. Deprecate (nickel 1): comment out dialog check code (at the cost of tagging Dialog import with Flake8 F401 NOQA flag for now).
2. Remove (nickel 2): remove dialog checks altogether.

In case additional dialog classes must be added again, it should be done from apps themselves, or if it spans multiple apps (same class name is used), overlay class chooser is preferred. This assumes future diaog classes are not listed in classes defined in UIA handler module.

Thanks.

Modern keyboard: on some systems, touch keyboard keys keeps firing name change events

Hi,

Reported by a user: on some systems, when focus moves to a new location, touch keyboard keys fire name change event. This is inconsistent.

Technical: name change event is fired by touch keyboard keys (class name: CRootKey), so suppress this if possible.

Thanks.

Announce UWP tooltips

Hi,

Reported by a user: when tooltips appear, Narrator can announce them. This is true for UWP-based tooltips.

Technical: we need to catch more tooltip objects.

Thanks.

Update channel: warn before switching from stable to development snapshots

Hi,

Caused by a code mix-up: when 17.08.1 was released, update channel was set to development. In addition to fixing this through a dedicated release branch, a dialog should be added to warn users about potential issues when running development snapshots such as loss of functionality.

Thanks.

skype uwp alt+1 and friends break

after I have selected a user for the first time and am in one of those edit fields, alt+1 stops working. I can't get str beyond that.

suggestion: please provide change log for each new release for the users

Hi there. I'm the regular user of your add-on, and really appreciate you all for such a nice work for win10 users out there. But here is a small suggestion from my side, that please provide the change log for each of the new release made to the add-on. It can be in ReadMe.ME section, or in add-on about/help pages. It would be much more helpful for your clients, to know your work and efforts in details about every release of the add-on.
Thanks for reading,

Search suggestions: prevent NVDA from announcing description if it is the same as the item label

Hi,

Perhaps due to an update to Cortana: in some cases, the item label and its description will be the same when performing searches. If this happens and if NVDA is told to announce object descriptions, NVDA will announce label twice. Thus catch this and prevent this from happening in the future.

Thanks.

Update check: What if other add-on is updating

Updates aren't allowed while NVDA itself is updating. What if other add-on is updating at the same time?

NVDA keeps announcing Skype Preview message from other apps

Hi,

In the latest NVDA next snapshot (13831), NVDA will be told to accept events from all UWP controls (NVDA Core issue 6713). This means background messages, such as Skype preview notices will be spoken while using other apps (a particular issue is when a message comes in and then a toast from Skype Preview appears). Thus for event handlers, check for foreground window before doing anything.

Thanks.

Mail: Suppress read-only state announcement

Hi,

Reported by users using Mail app: when reading message content, NVDA keeps announcing "read-only".

Technical: NVDA Core's Edge RS1 branch fixes this. a temporary workaround will be provided for a while, to be removed once Edge RS1 makes its way to NVDA master branch. Thanks.

Version 1809/Sets: announce suggestions

Hi,

Sets is used to group apps into Edge tabs. This is done by pressing Control+Windows=T to open a new app tab, search for an item from an embedded Cortana search field, then launch the resulting item.

Unfortunately, controller for event says 0 items when in fact results are present. Thankfully, item selected event is fired by results to indicate appearance of suggestions. Thus work around this by detecting Sets-based suggestions list and announce selected item.

Thanks.

Update check and preferences dialog: disable them if running on Windows Store version of NVDA

Hi,

As Windows Store version of NVDA 9or for that matter, UWP) runs in a restricted environment (namely no way to check for updates and no support for add-ons), disable update check and preferences dialog if this is the case. Prefreences dialog is included here because all it does now is configure add-on update cadence and channel.

Thanks.

Introduce support for debug logging

Hi,

In recent NVDA snapshots, it became possible to restart NVDA with debug logging enabled for a single session. This opens up possibilities for this ad-on, including logging UIA properties for troubleshooting event handling and other goodies.

Thanks.

Bing Maps: issues with "IAccessible AttributeError"

Hi,

Several issues were observed in Maps app, all related to the following regression:

AttributeError: IAccessible object has no attribute "UIAElement"

Cause: over-zealous overlay class chooser that only checked for UIA elements, not knowing that an ancestor might be an IAccessible object.

This causes following issues:

• AttributeError exception logged.
• Cannot press TAB to move between various buttons (to reproduce, open Maps, go to Location button and press TAB).

Thanks.

Quick link menu: NVDA says "unknown" when opening and closing this menu

Hi,

Reported by several people and subsequently confirmed: when opening and closing quick link menu, NVDA says "unknown".

STR:

1. Press Windows+X to open quick link menu.
2. Select an item.

Expected: NVDA just announces menu items.
Actual: NVDA says "unknown".
Cause: focus event on an unknown app launcher window.

As this is annoying, a hotfix for 18.04.2 will be created to work around this. Thanks.

Windows 10 version check: use build range instead of winVersion.isWin10

Hi,

See NvDA issue 11795: starting with October 2020 Update, Microsoft is using YYHx for release ID's, breaking winVersion.isWin10 as currently designed (takes integer argument for version). Because of this, use build range instead when checking Windows 10 versions.

Thanks.

Allow installed copies of NVDA to download add-on updates on its own

Hi,

Until now, when a user said "yes" to add-on updates, a browser opens. For installed copies of NVDA, it would make more sense to let the add-on itself download updates on its own, using a variation of NVDA Core's update downloader (tests shows it is working for another add-on). Thus adopt this behavior in spring.

Thanks.

Calculator: restructure Windows 10 and 11 app modules

Hi,

Work in progress:

At the moment there are two Calculator modules, one for Windows 10 and another for Windows 11. So far the biggest difference is UIA tree - in Windows 11 (calculatorapp.exe), one must descend one more level when fetching results element. Combined with massive restructuring of UIA notification event handler, it makes sense to reorganize these app modules as follows:

• Windows 10 Calculator will serve as the base of Windows 11 Calculator as before with an addition of version check.
• Windows 11 Calculator will be a stub app module, essentially an alias of Windows 10 Calculator.

The biggest benefit is that only the base Calculator app module (Windows 10) needs to be edited to support Windows 11 unless substantial features are added to Windows 11 version that makes the current approach unworkable (such as notification activity Id changes).

Thanks.

Context menu items expose position information

Hi,

It was observed that certain context menu items expose position information (all have the UIA class name of MenuFlyoutItem), and hearing position info for them is quite annoying. So find a way to prevent NVDA from announcing position information.

Thanks.

SearchUI: separate context menu items from suggestions

Hi,
Found a design flaw while debugging context menu item announcement for Sets:

Regardless of where SearchUI is found, context menu items should be announced. Thankfully, the automation ID for context menu itself is stable, allowing items to be represented as an overlay class.

Thanks.

NVDA announces new incomming mails until I press the Control key

I'm using Windows 10 Creators ubdate, NVDA version 2017.1 and the last stable version of Winten App Essentials addon. Since the last addon ubdate, NVDA announces every new incomming mail until I press the Control key. Before downloading the last addon ubdate, everything worked well for me. I tried to restart NVDA and reload all addons again, but it didn't help.

Event tracking and logging: remove unused drag/drop events

Hi,

Partially reverts #43:

Even though the add-on handles six UIA drag/drop events, only two (drag complete, drop target dropped) are actively handled (passive handling means only event info is logged and nothing else). Therefore drop other four events (drag start, drag cancel, drop target drag enter, drop target drag leave) from being tracked and logged - this involves removing these events from NVDA Core's UIA handler events map. These events may come back if they are found to be useful in various apps.

Thanks.

Introduce ability to trace only select apps and/or events

Hi,

Currently when one wishes to trace events, one would restart NVDA with debug logging enabled. At the moment all sorts of UIA events from all apps will be tracked, but there might be situations where one may wish to troubleshoot event tracing with just one or two events and/or inside one or two apps.

Thus introduce a series of command-line switches unique to this add-on that will allow users and developers to limit event tracing to an app (given its executable) and events. Also, if any of these new switches are given, enable event tracing in log levels other than debug mode so people can report event tracing issues without being bogged down by all sorts of debug output.

Thanks.

Suggestions close doesn't play when exiting settings.

1. open settings.
2. type "account"
3. wait for the zoop sound.
4. alt+f4.
   What should happen? Suggestions closed (reverse xoop sound). No sound is played.

Mail app: let table navigation commands be used to review message headers

Hi,

Similar to Outlook, NVDA should be able to use table nav commands in Mail app to review message headers. Thankfully, this is quite simple, as all the conditions to treat mail items as fake table row are there (children are columns). For now, a prototype will be included in WinTenApps, but it should become part of NVDA Core in 2017.2.

Thanks.

UIA events: backport UIA event changes from NVDA 2018.4-dev to 2018.3.x

Hi,

As a form of experiment and to gather feedback from those using old Windows 10 releases, backport WinEvent modifications made in NVDA 2018.4-dev to NVDA 2018.3.x via WinTenApps. This will mean a custom UIA handler object must be active.

Thanks.

Skype Preview: do not announce chat channel type such as "Skype Message" when announcing conversation content

Hi,

Because Skype is now used for SMS as well as Skype messages, Microsoft has added channel type to message items. This means NVDA would announce "Skype Message" every time message is received. A regular expression-based solution is now employed to prevent NVDA from announcing chat channel.

Thanks.

Edge: announce aria-alert text

Hi,

NVDA supports aria-alert on Firefox (and possibly Chrome and IE if coded correctly), but it isn't supported in Edge. Would it be possible to add support for aria-alert role in Edge?

Technical: yes: live region changed event is fired, coming from objects with varying automation ID's. Thankfully, its children contains alert text, so grab them in one sweep (string joining operation).
This came out of a Twitter conversation with an accessibility tester.

Thanks.

Word 2019 last version duplicates NVDA attribute change notification when formatting text through keystrokes

See nvaccess/nvda#10950.
Could this issue be fixed by winTenApps, as with "delete back word"?

Dialog detection: some dialogs are still recognized as windows

Hi,

Regression introduced in recent dev builds (does not exist in stable releases):

It appears the dialog detection removal was too optimistic: despite being recognized as such, some dialogs (notably restart to install updates screen in Windows 11 Insider Preview) are recognized as windows, not dialogs. Therefore revert dialog detection removal commits from main branch.

Thanks.