Explore RWKV-LM

[openadaptai]

Explore RWKV-LM

[dianzrong]

RWKV-LM can be used throughout the project to:

• summarize texts associated with the screenshots
• better suggest prompts (like google's autofill)
• better understand prompts from the user
• in the future: handle translations

Implementation:

• can get "infinite" context lengths through iterative refinement
  • states would be the recording up until that point and the token would be the next screenshot

Limitation:

• this process can be computationally expensive

[angelala3252]

As Dian mentioned above, RWKV-LM could be helpful for us since it could analyze large amounts of past user actions to better predict future actions. It could also learn how to do more complex tasks that the user themselves don't know how to do, as it could learn from longer texts like instruction manuals. These benefits would also help with efficiency and speed up the automation.

As for implementation, we could consider combining RWKV-LM with another tool that could help with the graphical side of things. For example, we could have RWKV_LM understand the task we wish to execute and lay out the steps needed, then using something like OCR or MiniGPT-4, we could handle how to execute the steps on the GUI.