x-slayer / flutter_accessibility_service Goto Github PK
View Code? Open in Web Editor NEWFlutter plugin for interacting with Accessibility Service in Android.
Home Page: https://pub.dev/packages/flutter_accessibility_service
License: MIT License
Flutter plugin for interacting with Accessibility Service in Android.
Home Page: https://pub.dev/packages/flutter_accessibility_service
License: MIT License
Initially I would only need to detect
Does this package help?
Att
Could not open settings generic class cache for settings file 'C:\Users\xxx\Downloads\flutter_accessibility_service-main\flutter_accessibility_service-main\example\android\settings.gradle' (C:\Users\xxx.gradle\caches\6.7\scripts\f0emg6u6oecmxqzgk5g9nn4ui).
BUG! exception in phase 'semantic analysis' in source unit 'BuildScript' Unsupported class file major version 61
There have been recent changes in the package to make a fix for this issue.
Can you please explain how it is fixed?
If not fixed, I would love to discuss on how we can do it.
A team member has added a set to handle duplicates but It feels unreliable because of generateNodeId
(E.g. In a chat application, parent node can be the same user name for multiple entries)
I am experimenting with using the flutter accessiblity service from a flutter foreground task. When I try to listen to the events in the foreground task then I get the following exception if the app is closed:
PlatformException(error, Attempt to invoke virtual method 'void android.app.Activity.startActivityForResult(android.content.Intent, int)' on a null object reference, null, java.lang.NullPointerException: Attempt to invoke virtual method 'void android.app.Activity.startActivityForResult(android.content.Intent, int)' on a null object reference
at slayer.accessibility.service.flutter_accessibility_service.FlutterAccessibilityServicePlugin.onMethodCall(FlutterAccessibilityServicePlugin.java:56)
at io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler.onMessage(MethodChannel.java:258)
It is probably because the package is trying to use an Activity context to call startActivityForResult. But this context may not be available in a foreground service.
Is there a way to use it from a foreground service?
Nice project bro, Just gonna ask if this library is in Production ready? Do you personally use this for production-level apps or just for pet projects? Planning to use this for our company's product. I'll wait for your reply before coding haha, Thanks.
Hello, could you answer how to work with your package? For example, I want to get a specific element on the screen, but your stream doesn't always return all the elements. I go to the chrome page and only the text from the timer is returned. How can I return all the elements? I want to automate my work. For example, how to return not only changes, but also simply return the current elements?
Also, for example you don't have a click event as a percentage of the screen. You will have to create all this yourself
Start fine, but there's some sort of lifecycle issues which inevitably cause it to loop-crash.
I've tried a few different approaches to handle the lifecycles, initialisation, and isolate management, but I have been unable to find an incantation that prevents these errors:
These errors have been preventing us from launching our app for many months now and I simply cannot find a way round them. Any help would be gratefully appreciated! Absolutely mission critical.
FYI: our foreground service runner is https://pub.dev/packages/flutter_foreground_task.
Error 1
Fatal Exception: java.lang.NullPointerException: Attempt to invoke virtual method 'io.flutter.embedding.engine.renderer.FlutterRenderer io.flutter.embedding.engine.a.r()' on a null object reference
at io.flutter.embedding.android.FlutterView.attachToFlutterEngine(FlutterView.java:48)
at slayer.accessibility.service.flutter_accessibility_service.AccessibilityListener.onServiceConnected(AccessibilityListener.java:40)
at android.accessibilityservice.AccessibilityService.dispatchServiceConnected(AccessibilityService.java:858)
at android.accessibilityservice.AccessibilityService.-$$Nest$mdispatchServiceConnected()
at android.accessibilityservice.AccessibilityService$2.onServiceConnected(AccessibilityService.java:2734)
at android.accessibilityservice.AccessibilityService$IAccessibilityServiceClientWrapper.lambda$init$0(AccessibilityService.java:2900)
at android.accessibilityservice.AccessibilityService$IAccessibilityServiceClientWrapper.$r8$lambda$ufMCaNdIS-PlK1AEBiWmKpGSCoc()
at android.accessibilityservice.AccessibilityService$IAccessibilityServiceClientWrapper$$ExternalSyntheticLambda16.run(:8)
at android.os.Handler.handleCallback(Handler.java:958)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:294)
at android.app.ActivityThread.main(ActivityThread.java:8248)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)
Error 2
Fatal Exception: java.lang.RuntimeException: Unable to start service slayer.accessibility.service.flutter_accessibility_service.AccessibilityListener@585db68 with null: java.lang.NullPointerException: Attempt to invoke virtual method 'boolean android.content.Intent.getBooleanExtra(java.lang.String, boolean)' on a null object reference
at android.app.ActivityThread.handleServiceArgs(ActivityThread.java:4880)
at android.app.ActivityThread.-$$Nest$mhandleServiceArgs()
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2314)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:294)
at android.app.ActivityThread.main(ActivityThread.java:8248)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)
Caused by java.lang.NullPointerException: Attempt to invoke virtual method 'boolean android.content.Intent.getBooleanExtra(java.lang.String, boolean)' on a null object reference
at slayer.accessibility.service.flutter_accessibility_service.AccessibilityListener.onStartCommand(AccessibilityListener.java:3)
at android.app.ActivityThread.handleServiceArgs(ActivityThread.java:4862)
at android.app.ActivityThread.-$$Nest$mhandleServiceArgs()
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2314)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:294)
at android.app.ActivityThread.main(ActivityThread.java:8248)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)
Error 3
Fatal Exception: java.lang.RuntimeException: Unable to destroy activity {ai.mycopilot.beta/ai.mycopilot.beta.MainActivity}: java.lang.IllegalArgumentException: Receiver not registered: f9.e$a@b92a94a
at android.app.ActivityThread.performDestroyActivity(ActivityThread.java:5654)
at android.app.ActivityThread.handleDestroyActivity(ActivityThread.java:5686)
at android.app.servertransaction.DestroyActivityItem.execute(DestroyActivityItem.java:47)
at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45)
at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:180)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:98)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2468)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:294)
at android.app.ActivityThread.main(ActivityThread.java:8248)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)
Caused by java.lang.IllegalArgumentException: Receiver not registered: f9.e$a@b92a94a
at android.app.LoadedApk.forgetReceiverDispatcher(LoadedApk.java:1671)
at android.app.ContextImpl.unregisterReceiver(ContextImpl.java:1872)
at android.content.ContextWrapper.unregisterReceiver(ContextWrapper.java:820)
at slayer.accessibility.service.flutter_accessibility_service.FlutterAccessibilityServicePlugin.onDetachedFromEngine(FlutterAccessibilityServicePlugin.java:15)
at io.flutter.embedding.engine.FlutterEngineConnectionRegistry.remove(FlutterEngineConnectionRegistry.java:121)
at io.flutter.embedding.engine.FlutterEngineConnectionRegistry.remove(FlutterEngineConnectionRegistry.java:16)
at io.flutter.embedding.engine.FlutterEngineConnectionRegistry.removeAll(FlutterEngineConnectionRegistry.java:11)
at io.flutter.embedding.engine.FlutterEngineConnectionRegistry.destroy(FlutterEngineConnectionRegistry.java:10)
at io.flutter.embedding.engine.FlutterEngine.destroy(FlutterEngine.java:31)
at io.flutter.embedding.android.FlutterActivityAndFragmentDelegate.onDetach(FlutterActivityAndFragmentDelegate.java:100)
at io.flutter.embedding.android.FlutterActivity.onDestroy(FlutterActivity.java:18)
at android.app.Activity.performDestroy(Activity.java:8902)
at android.app.Instrumentation.callActivityOnDestroy(Instrumentation.java:1491)
at android.app.ActivityThread.performDestroyActivity(ActivityThread.java:5641)
at android.app.ActivityThread.handleDestroyActivity(ActivityThread.java:5686)
at android.app.servertransaction.DestroyActivityItem.execute(DestroyActivityItem.java:47)
at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45)
at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:180)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:98)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2468)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loopOnce(Looper.java:205)
at android.os.Looper.loop(Looper.java:294)
at android.app.ActivityThread.main(ActivityThread.java:8248)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)
Error 3
Fatal Exception: java.lang.NoSuchMethodError: No interface method anyMatch(Lj$/util/function/Predicate;)Z in class Lj$/util/stream/Stream; or its super classes (declaration of 'j$.util.stream.Stream' appears in /data/app/~~ZLFWe7X3Fq5Nv8kTOjKieQ==/androidx.test.tools.crawler-MQkMpzA8jfAAOwsP0Op8kw==/base.apk!classes2.dex)
at io.flutter.embedding.android.FlutterView.sendUserSettingsToFlutter(FlutterView.java:51)
at io.flutter.embedding.android.FlutterView.attachToFlutterEngine(FlutterView.java:263)
at io.flutter.embedding.android.FlutterActivityAndFragmentDelegate.onCreateView(FlutterActivityAndFragmentDelegate.java:130)
at io.flutter.embedding.android.FlutterActivity.createFlutterView(FlutterActivity.java:18)
at io.flutter.embedding.android.FlutterActivity.onCreate(FlutterActivity.java:31)
at android.app.Activity.performCreate(Activity.java:8074)
at android.app.Activity.performCreate(Activity.java:8054)
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1341)
at androidx.test.runner.MonitoringInstrumentation.callActivityOnCreate(MonitoringInstrumentation.java:2)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3688)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3864)
at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:103)
at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2253)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loopOnce(Looper.java:201)
at android.os.Looper.loop(Looper.java:288)
at android.app.ActivityThread.main(ActivityThread.java:7870)
at java.lang.reflect.Method.invoke(Method.java)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003)
Hello, is there any plan to support this function in the future? I hope to realize software automation control function with the help of this plugin.
Originally posted by @zhangyusheng211 in #2 (comment)
Required param for support on android 14.
Problem:
I am attempting to identify the currently active application using the Flutter Accessibility Service. After following the instructions provided on the package’s pub.dev page, I utilized event.packageName
to achieve this. However, upon implementing the code in my main.dart,
I encountered the following error:
W/FlutterJNI(23350): Tried to send a platform message to Flutter, but FlutterJNI was detached from native C++. Could not send. Channel: x-slayer/accessibility_event. Response ID: 131
This error occurs after my application is closed, and persists even when the application is reopened.
Steps to Reproduce:
flutter_accessibility_service
package and follow the instructions in the README.
StreamSubscription<AccessibilityEvent>? _subscription;
List<AccessibilityEvent?> events = [];
String currentApp = '';
String getCurrentApp() {
if (_subscription?.isPaused ?? false) {
_subscription?.resume();
return '';
}
_subscription = FlutterAccessibilityService.accessStream.listen((event) async {
setState(() {
currentApp = event.packageName!;
});
log("current event : $currentApp");
});
return currentApp;
}
Expected Behavior: The getCurrentApp()
function should log the package name in the terminal, but it does not execute as expected, and no package name is logged.
Actual Behavior: The error message is displayed, and the expected behavior does not occur.
any idea how to fix it?
your help is wanted! @X-SLAYER @fahimfoysal2 @avisihag
Thank You
Hello,
Thank you for this. Is it normal that the stream stops when the app is closed ? I have used Accessibility Service on Android app before and even if the app is closed, the streams continue. Is there any way to achieve the same with this plugin ? Like using an isolate or something like that?
Thanks
I noticed you didn't implement the dispatchGesture() AccessibilityService() method. Could you add that if it is not too much trouble? I need that for my application but I don't understand how the plugin works to add it myself.
Can I return the className in AccessibilityEvent? Need to use, or each state of the node
I have tried to implement an overlay that doesn't cover the whole screen but failed to do so. For example, I want to show a small UI card on home screen. User should be able to see and interact with the home screen of device normally. Currently, the overlay covers all screen, and we are not able to see any portion of, or interact with the home screen.
Can you please let me know how we can achieve it with this plugin?
I can see that in android this works fine but how we can use the same for iOS? Is there any support?
When iterating through the child nodes of a parent node, the method onAccessibilityEvent
invokes getSubNodes
, a recursive function. This recursion, influenced by the depth of the tree (treeDepth
), has the potential to cause a stack overflow, leading to Application Not Responding (ANR) errors.
There are discussions on StackOverflow, of ANRs occurring when the accessibility service is used for navigating through AccessibilityNodeInfo.
To mitigate this issue, I propose to add a new parameter, maximumTreeDepth
, which would serve as a threshold for the recursion depth. Once this threshold is reached, further traversal operations would be halted, and the method getNextTexts
can return.
This approach allows users to balance the trade-off between performance aspects (such as minimizing ANRs and reducing battery consumption) and usability, by adjusting the maximumTreeDepth parameter according to the needs of their specific application.
Let me know what do you think about it, I can then raise a PR for the same!
hi
I was looking for information about flutter accessibility service and found this project, it's really an interesting project
However, I am not an android developer, can this project help me to control other APPs?
For example, can I use this project to control android to automatically click on an application? Do something automatically after opening the app?
Looking forward to your help, thank you
Feature Request
Thanks
Its doesn't working on Android 12:
I have did the installation of the plugin, and tried to reproduce the list on Start Stream example, and doesn't wok.
My setup:
pubspec.yml
environment:
sdk: '>=2.18.1 <3.0.0'
dependencies:
flutter:
sdk: flutter
flutter_accessibility_service: ^0.2.2
flutter_local_notifications: ^12.0.3
http: ^0.13.5
Whenever we are using any feature from the plugin such as showOverlayWindow, performGlobalAction, etc. we have to request permission again, on each run.
Expected Output: We should be able to request the permission once, and it should not be required for future runs, unless user manually revokes the permission, or the app gets deleted.
Also, if this gets implemented, it would be great to have revokeAccessibilityPermission as well.
By the way, huge thanks for building this, as this takes care of all the heavy lifting we'll have to do to implement Accessability service in Flutter.
Its a common problem that accessibility service often leads to ANRs because of timeouts in onListen/locks(ANR triggered by main thread waiting for too long
).
Do you think it would be better to offload the processing inside onListen to a separate thread with a timeout? If the Thread doesn't finish processing within a set time, timeout it and exit the onListen.
What does the author think?
Something like this
@Override
public void onAccessibilityEvent(AccessibilityEvent event) {
Callable<Void> task = () -> {
handler()
};
Future<Void> future = executor.submit(task);
try {
future.get(5, TimeUnit.SECONDS);
} catch (TimeoutException e) {
future.cancel(true);
handleTimeout(event);
} catch (Exception e) {
e.printStackTrace();
}
}
After having granted the accessibility permission to my application, if I close it the permission stops working and when I reopen it when I activate the permission in settings it tells me that the permission stopped working, how can I do so that when I grant the permission with the permission package is maintained even if I close the application or restart my cell phone.
can we perform clicks on specific part of screen using this plugin outside of our app?.
@X-SLAYER When using this plugin to receive accessibility events the action type that is received of an event, It is always 0. Why is it so?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.