Explore TaskMatrix (previously Visual ChatGPT) / LowCodeLLM

[abrichr]

Explore TaskMatrix (previously Visual ChatGPT) / LowCodeLLM

[0dm]

Since TaskMatrix recently updated for GroundingDINO and segment-anything support (2 weeks ago), it could definitely be of use for generating segmentation results.

Some ideas I have:

• Use TaskMatrix's templates to define execution flows for complex tasks (or common ones).
• Compatibility: we can modify the resolution/scale of screenshots to make recordings more compatible with other devices.
• Can implement a form of pop-up detection, since sometimes unexpected events can occur (like a pop-up from another application / OS), which would usually break automation processes until a human intervenes.

[ahmetutku]

To add to @0dm, we could also implement:

• Regression Testing: We can utilize TaskMatrix to automate testing after every change (or every set of changes) to ensure that the changed code is bug-free & functional
• Automating User Input: We could use TaskMatrix to simulate user input (e.g. typing commands). In addition, we can use verify the response. Saving time for the coder!

[abrichr]

Use TaskMatrix's templates to define execution flows for complex tasks (or common ones).

@0dm what would this look like?

[0dm]

forgive me, it's almost 6am and my brain is going to implode (i haven't slept). here's a rough idea with one of their example templates.

they define a template:

class ObjectSegmenting:
    template_model = True # Add this line to show this is a template model.
    def __init__(self,  Text2Box:Text2Box, Segmenting:Segmenting):
        # self.llm = OpenAI(temperature=0)
        self.grounding = Text2Box
        self.sam = Segmenting
        
        # methods below...

now, this template can be applied for a variety of situations, such as the prompt, "segment the given object:" or even "segment the cat" (🐈‍⬛ 😨).

then, they collect all the templates for a list of possible functions:

 # Load Template Foundation Models
      for class_name, module in globals().items():
          if getattr(module, 'template_model', False):
              template_required_names = {k for k in inspect.signature(module.__init__).parameters.keys() if k!='self'}
              # ...

In puterbot, we could define templates for "open application", "send email", etc. and have these functions activate on certain phrases like "open <application> => open chrome" . This would make the codebase very organized and make combining tasks easier, since replays could invoke numerous functions in sequence (open chrome => goto url gmail => compose new email => ...)

[abrichr]

Thank you @0dm ! Let's talk through this on a call. Please grab some time on my calendar here:

https://www.getclockwise.com/c/richard-abrich/quick-meeting (today after 3pm or tomorrow, otherwise next week).

[0dm]

@abrichr , scheduled for 4:15 tomorrow!

[abrichr]

https://aclanthology.org/2024.naacl-demo.2/

https://github.com/chenfei-wu/TaskMatrix/tree/main/LowCodeLLM

https://www.youtube.com/watch?v=jb2C1vaeO3E