x-slayer / flutter_accessibility_service Goto Github PK

View Code? Open in Web Editor NEW

28.0 3.0 21.0 157 KB

Flutter plugin for interacting with Accessibility Service in Android.

Home Page: https://pub.dev/packages/flutter_accessibility_service

License: MIT License

Java 46.43% Dart 53.57%

flutter accessibility accessibilityservice pub-dev x-slayer

flutter_accessibility_service's Introduction

👋 Hello! I'm Iheb Briki known as X-SLAYER

I'm a full-stack developer and mainly use Flutter,React & nextJs

Remember, open source is not commercial software development. The projects I share here are just tools I've built for myself to make my life better. Be mindful and respect the time that I (and every other contributor) provide to the community free of charge. 💜

flutter_accessibility_service's People

Contributors

Stargazers

Watchers

flutter_accessibility_service's Issues

Is non blocking overlay possible with the plugin?

I have tried to implement an overlay that doesn't cover the whole screen but failed to do so. For example, I want to show a small UI card on home screen. User should be able to see and interact with the home screen of device normally. Currently, the overlay covers all screen, and we are not able to see any portion of, or interact with the home screen.

Can you please let me know how we can achieve it with this plugin?

[Query] Fix for StackOverflow in `getSubNodes`

There have been recent changes in the package to make a fix for this issue.

Can you please explain how it is fixed?

If not fixed, I would love to discuss on how we can do it.

A team member has added a set to handle duplicates but It feels unreliable because of generateNodeId
(E.g. In a chat application, parent node can be the same user name for multiple entries)

PR for changes by @avisihag #15

Stream stops when app is closed

Hello,

Thank you for this. Is it normal that the stream stops when the app is closed ? I have used Accessibility Service on Android app before and even if the app is closed, the streams continue. Is there any way to achieve the same with this plugin ? Like using an isolate or something like that?

Thanks

I am using the FlutterAccessibilityService.accessStream.firstWhere function and after some time my app has started to hang alot. Can you explain what is going wrong. May be the stream is not getting destroyed. But i did not find any option to destroy it

Questions.

Nice project bro, Just gonna ask if this library is in Production ready? Do you personally use this for production-level apps or just for pet projects? Planning to use this for our company's product. I'll wait for your reply before coding haha, Thanks.

Accessibility permission is disabled if I close the application or turn off my cell phone

After having granted the accessibility permission to my application, if I close it the permission stops working and when I reopen it when I activate the permission in settings it tells me that the permission stopped working, how can I do so that when I grant the permission with the permission package is maintained even if I close the application or restart my cell phone.

How can I detect if accessibility mode is active?

Initially I would only need to detect
Does this package help?

Att

can we perform clicks?

can we perform clicks on specific part of screen using this plugin outside of our app?.

Invalid Manifest

There are some questions before use

I was looking for information about flutter accessibility service and found this project, it's really an interesting project

However, I am not an android developer, can this project help me to control other APPs?

For example, can I use this project to control android to automatically click on an application? Do something automatically after opening the app?

Looking forward to your help, thank you

Action Type is always 0

@X-SLAYER When using this plugin to receive accessibility events the action type that is received of an event, It is always 0. Why is it so?

stream doesn't always return

Hello, could you answer how to work with your package? For example, I want to get a specific element on the screen, but your stream doesn't always return all the elements. I go to the chrome page and only the text from the timer is returned. How can I return all the elements? I want to automate my work. For example, how to return not only changes, but also simply return the current elements?

Also, for example you don't have a click event as a percentage of the screen. You will have to create all this yourself

Can I return the className in AccessibilityEvent? Need to use, or each state of the node

W/FlutterJNI(23350): Tried to send a platform message to Flutter, but FlutterJNI was detached from native C++. Could not send. Channel: x-slayer/accessibility_event.

Issue Description

Problem:
I am attempting to identify the currently active application using the Flutter Accessibility Service. After following the instructions provided on the package’s pub.dev page, I utilized event.packageName to achieve this. However, upon implementing the code in my main.dart, I encountered the following error:

W/FlutterJNI(23350): Tried to send a platform message to Flutter, but FlutterJNI was detached from native C++. Could not send. Channel: x-slayer/accessibility_event. Response ID: 131

This error occurs after my application is closed, and persists even when the application is reopened.

Steps to Reproduce:

Install the flutter_accessibility_service package and follow the instructions in the README.

I'm just adding the following damn code to the MyApp class in main.dart:

StreamSubscription<AccessibilityEvent>? _subscription;
List<AccessibilityEvent?> events = [];
String currentApp = '';

String getCurrentApp() {
  if (_subscription?.isPaused ?? false) {
    _subscription?.resume();
    return '';
  }
  _subscription = FlutterAccessibilityService.accessStream.listen((event) async {
    setState(() {
      currentApp = event.packageName!;
    });
    log("current event : $currentApp");
  });
  return currentApp;
}

Run the application on an emulator and enable accessibility permissions.
Close the app.

Expected Behavior: The getCurrentApp() function should log the package name in the terminal, but it does not execute as expected, and no package name is logged.

Actual Behavior: The error message is displayed, and the expected behavior does not occur.

any idea how to fix it?
your help is wanted! @X-SLAYER @fahimfoysal2 @avisihag
Thank You

Have to request permission on each run

Whenever we are using any feature from the plugin such as showOverlayWindow, performGlobalAction, etc. we have to request permission again, on each run.

Expected Output: We should be able to request the permission once, and it should not be required for future runs, unless user manually revokes the permission, or the app gets deleted.

Also, if this gets implemented, it would be great to have revokeAccessibilityPermission as well.

By the way, huge thanks for building this, as this takes care of all the heavy lifting we'll have to do to implement Accessability service in Flutter.

Can you add the dispatchGesture() call?

I noticed you didn't implement the dispatchGesture() AccessibilityService() method. Could you add that if it is not too much trouble? I need that for my application but I don't understand how the plugin works to add it myself.

Its doesn't working on Android 12

Its doesn't working on Android 12:
I have did the installation of the plugin, and tried to reproduce the list on Start Stream example, and doesn't wok.

My setup:

pubspec.yml

environment:
  sdk: '>=2.18.1 <3.0.0'

dependencies:
  flutter:
    sdk: flutter
  flutter_accessibility_service: ^0.2.2
  flutter_local_notifications: ^12.0.3
  http: ^0.13.5

I have some questions.

          Hello, is there any plan to support this function in the future? I hope to realize software automation control function with the help of this plugin.

Originally posted by @zhangyusheng211 in #2 (comment)

The environment for this example does not work properly

Could not open settings generic class cache for settings file 'C:\Users\xxx\Downloads\flutter_accessibility_service-main\flutter_accessibility_service-main\example\android\settings.gradle' (C:\Users\xxx.gradle\caches\6.7\scripts\f0emg6u6oecmxqzgk5g9nn4ui).

BUG! exception in phase 'semantic analysis' in source unit 'BuildScript' Unsupported class file major version 61

ANR issues due to `getSubNodes` recursion depth

When iterating through the child nodes of a parent node, the method onAccessibilityEvent invokes getSubNodes, a recursive function. This recursion, influenced by the depth of the tree (treeDepth), has the potential to cause a stack overflow, leading to Application Not Responding (ANR) errors.

There are discussions on StackOverflow, of ANRs occurring when the accessibility service is used for navigating through AccessibilityNodeInfo.

To mitigate this issue, I propose to add a new parameter, maximumTreeDepth, which would serve as a threshold for the recursion depth. Once this threshold is reached, further traversal operations would be halted, and the method getNextTexts can return.

This approach allows users to balance the trade-off between performance aspects (such as minimizing ANRs and reducing battery consumption) and usability, by adjusting the maximumTreeDepth parameter according to the needs of their specific application.

Let me know what do you think about it, I can then raise a PR for the same!

Feature: Allow app to receive `KeyEvent` from Accessibility Service

Feature Request

As Accessibility Service also provides key press events listener, can we please have this feature?
Do you have any plan for it?
If not, can you give me a guideline how can I implement it?

Thanks

How to access same on iOS side?

I can see that in android this works fine but how we can use the same for iOS? Is there any support?

@X-SLAYER

[Android 14] Required RECEIVER_NOT_EXPORTED/RECEIVER_EXPORTED

Required param for support on android 14.

Discussion: https://stackoverflow.com/questions/77235063/one-of-receiver-exported-or-receiver-not-exported-should-be-specified-when-a-rec

Usage from a foreground service

I am experimenting with using the flutter accessiblity service from a flutter foreground task. When I try to listen to the events in the foreground task then I get the following exception if the app is closed:

PlatformException(error, Attempt to invoke virtual method 'void android.app.Activity.startActivityForResult(android.content.Intent, int)' on a null object reference, null, java.lang.NullPointerException: Attempt to invoke virtual method 'void android.app.Activity.startActivityForResult(android.content.Intent, int)' on a null object reference
      	at slayer.accessibility.service.flutter_accessibility_service.FlutterAccessibilityServicePlugin.onMethodCall(FlutterAccessibilityServicePlugin.java:56)
      	at io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler.onMessage(MethodChannel.java:258)

It is probably because the package is trying to use an Activity context to call startActivityForResult. But this context may not be available in a foreground service.

Is there a way to use it from a foreground service?

Constant crashing used with foreground service

Start fine, but there's some sort of lifecycle issues which inevitably cause it to loop-crash.

I've tried a few different approaches to handle the lifecycles, initialisation, and isolate management, but I have been unable to find an incantation that prevents these errors:

These errors have been preventing us from launching our app for many months now and I simply cannot find a way round them. Any help would be gratefully appreciated! Absolutely mission critical.

FYI: our foreground service runner is https://pub.dev/packages/flutter_foreground_task.

Error 1

Fatal Exception: java.lang.NullPointerException: Attempt to invoke virtual method 'io.flutter.embedding.engine.renderer.FlutterRenderer io.flutter.embedding.engine.a.r()' on a null object reference
       at io.flutter.embedding.android.FlutterView.attachToFlutterEngine(FlutterView.java:48)
       at slayer.accessibility.service.flutter_accessibility_service.AccessibilityListener.onServiceConnected(AccessibilityListener.java:40)
       at android.accessibilityservice.AccessibilityService.dispatchServiceConnected(AccessibilityService.java:858)
       at android.accessibilityservice.AccessibilityService.-$$Nest$mdispatchServiceConnected()
       at android.accessibilityservice.AccessibilityService$2.onServiceConnected(AccessibilityService.java:2734)
       at android.accessibilityservice.AccessibilityService$IAccessibilityServiceClientWrapper.lambda$init$0(AccessibilityService.java:2900)
       at android.accessibilityservice.AccessibilityService$IAccessibilityServiceClientWrapper.$r8$lambda$ufMCaNdIS-PlK1AEBiWmKpGSCoc()
       at android.accessibilityservice.AccessibilityService$IAccessibilityServiceClientWrapper$$ExternalSyntheticLambda16.run(:8)
       at android.os.Handler.handleCallback(Handler.java:958)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loopOnce(Looper.java:205)
       at android.os.Looper.loop(Looper.java:294)
       at android.app.ActivityThread.main(ActivityThread.java:8248)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)

Error 2

Fatal Exception: java.lang.RuntimeException: Unable to start service slayer.accessibility.service.flutter_accessibility_service.AccessibilityListener@585db68 with null: java.lang.NullPointerException: Attempt to invoke virtual method 'boolean android.content.Intent.getBooleanExtra(java.lang.String, boolean)' on a null object reference
       at android.app.ActivityThread.handleServiceArgs(ActivityThread.java:4880)
       at android.app.ActivityThread.-$$Nest$mhandleServiceArgs()
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2314)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loopOnce(Looper.java:205)
       at android.os.Looper.loop(Looper.java:294)
       at android.app.ActivityThread.main(ActivityThread.java:8248)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)
Caused by java.lang.NullPointerException: Attempt to invoke virtual method 'boolean android.content.Intent.getBooleanExtra(java.lang.String, boolean)' on a null object reference
       at slayer.accessibility.service.flutter_accessibility_service.AccessibilityListener.onStartCommand(AccessibilityListener.java:3)
       at android.app.ActivityThread.handleServiceArgs(ActivityThread.java:4862)
       at android.app.ActivityThread.-$$Nest$mhandleServiceArgs()
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2314)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loopOnce(Looper.java:205)
       at android.os.Looper.loop(Looper.java:294)
       at android.app.ActivityThread.main(ActivityThread.java:8248)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)

Error 3

Fatal Exception: java.lang.RuntimeException: Unable to destroy activity {ai.mycopilot.beta/ai.mycopilot.beta.MainActivity}: java.lang.IllegalArgumentException: Receiver not registered: f9.e$a@b92a94a
       at android.app.ActivityThread.performDestroyActivity(ActivityThread.java:5654)
       at android.app.ActivityThread.handleDestroyActivity(ActivityThread.java:5686)
       at android.app.servertransaction.DestroyActivityItem.execute(DestroyActivityItem.java:47)
       at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45)
       at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:180)
       at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:98)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2468)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loopOnce(Looper.java:205)
       at android.os.Looper.loop(Looper.java:294)
       at android.app.ActivityThread.main(ActivityThread.java:8248)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)
Caused by java.lang.IllegalArgumentException: Receiver not registered: f9.e$a@b92a94a
       at android.app.LoadedApk.forgetReceiverDispatcher(LoadedApk.java:1671)
       at android.app.ContextImpl.unregisterReceiver(ContextImpl.java:1872)
       at android.content.ContextWrapper.unregisterReceiver(ContextWrapper.java:820)
       at slayer.accessibility.service.flutter_accessibility_service.FlutterAccessibilityServicePlugin.onDetachedFromEngine(FlutterAccessibilityServicePlugin.java:15)
       at io.flutter.embedding.engine.FlutterEngineConnectionRegistry.remove(FlutterEngineConnectionRegistry.java:121)
       at io.flutter.embedding.engine.FlutterEngineConnectionRegistry.remove(FlutterEngineConnectionRegistry.java:16)
       at io.flutter.embedding.engine.FlutterEngineConnectionRegistry.removeAll(FlutterEngineConnectionRegistry.java:11)
       at io.flutter.embedding.engine.FlutterEngineConnectionRegistry.destroy(FlutterEngineConnectionRegistry.java:10)
       at io.flutter.embedding.engine.FlutterEngine.destroy(FlutterEngine.java:31)
       at io.flutter.embedding.android.FlutterActivityAndFragmentDelegate.onDetach(FlutterActivityAndFragmentDelegate.java:100)
       at io.flutter.embedding.android.FlutterActivity.onDestroy(FlutterActivity.java:18)
       at android.app.Activity.performDestroy(Activity.java:8902)
       at android.app.Instrumentation.callActivityOnDestroy(Instrumentation.java:1491)
       at android.app.ActivityThread.performDestroyActivity(ActivityThread.java:5641)
       at android.app.ActivityThread.handleDestroyActivity(ActivityThread.java:5686)
       at android.app.servertransaction.DestroyActivityItem.execute(DestroyActivityItem.java:47)
       at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45)
       at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:180)
       at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:98)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2468)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loopOnce(Looper.java:205)
       at android.os.Looper.loop(Looper.java:294)
       at android.app.ActivityThread.main(ActivityThread.java:8248)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)