oneirocom/magick

Magick is a cutting-edge toolkit for a new kind of AI builder. Make Magick with us!

magick's Issues

Typing "Shift+F" will create a comment box on the graph even if you don't have it focused. Same with Shift+C

For example, naming a node "Agent Facts" will create a comment box as I'm typing

Delete does not get registered

The delete/backspace does not remove the nodes

Avatar window is missing from available windows to open

Avatar window is open by default, but not available from the menu

Fix padding / margin on inspector

The inspector padding and margin are bad

If an input has a default value of 0 it's rendered empty

 
builder(node: MagickNode) {
    const out = new Rete.Output('output', 'output', numSocket)
    const dataInput = new Rete.Input('trigger', 'Trigger', triggerSocket, true)

    const _Min = new NumberControl({
      dataKey: '_Min',
      name: 'Value min',
      icon: 'moon',
    })

    const _Max = new NumberControl({
      dataKey: '_Max',
      name: 'Value max',
      icon: 'moon',
    })

    const name = new InputControl({
      dataKey: 'name',
      name: 'Name',
      icon: 'moon',
    })

   
    node.data._Max = 100
    node.data._Min = 0
    
    node.inspector.add(name).add(_Min).add(_Max)   
    return node.addOutput(out).addInput(dataInput)
  }

Move event manager to drawer

Event manager should be moved from the composer window to its own drawer/screen for easy management

Weaviate Neural Search Demo

We use Weaviate for neural search, and have a search running. Demonstrate long term memory using Weaviate by recalling stored events from database, creating vector embeddings and storing them into a weaviate store, then recalling facts and relevant conversation based on input. This should all be done in a Magick spell or set of spells so that agents can automatically add to their long term memory and recall from it.

https://app.dework.xyz/m3/ai-43567/view/board-lblry0sg?taskId=551855dd-ae87-4b66-8f86-826ae12bff84

Requirements:

• new weaviate nodes for storage and retrieval
• Weaviate schema that mirrors events
• Weaviate endpoints on server
• retrieval nodes could be divided into different types of vector retrieval. And minimum a semantic search, and question answer search.

Automatic sockets for inspector controls

Data controls should be able to create a socket on the component which it is registered. This will mean that all inspector components also have sockets that can be used to override values. This creates convenience and ease of use.

Plugin system for connectors

Connectors currently are all different and loaded all together in the entity. We need a proposal and implementation of a module system for the connectors. A clean API that can be generalized to all our connectors. We should start with migrating over a single connector as a trial. Let's start with discord.

Public variables

Public variable nodes. Should have different types. Could possibly just make our current variables exposable as public. A public variable can be inject into the spell. So either as data from outside or as sockets into a spell. Used to create sets of data for different entities to share the same brain/spell.

Need to consider whether public variables can be populated from the top level spell all the way down to subspells or not.

Move avatar window into plugin-avatar

We want the avatar and all avatar core code out.

• Make a new plugin, duplicating the Fine Tune plugin and copying the routes everywhere
• Move all avatar code into plugin window

Move search corpus to side window

Alphabetize node menu

Currently, the node menu categories are in a state of disarray. It would be good to alphabetize it so it is easier to navigate.

Current dropdown:

Persistent layout

Layouts right now reset after reloading. These could be stored in redux for persistence between sessions. Bonus for saving multiple layouts for recall later.

Connector simulator

Playtesting with the conversational VRM or with the playtest window for agents is a pain, as we have to add a bunch of raw config to the "data" dropdown, which is just a temporary hack to pass more complicated data out.

It would be awesome if we had a dropdown with preset data for different connectors (discord, twitter, etc) which would make it really easy to test each of them as the data comes through the pipe.

Improve UI of inspector components

Currently all the components in the inspector are poorly spaced and clunky. Here is a rough design I did up that I think would improve it a lot. Eeach one of the data Control components could probably be improved and polished more. Ignore the coloring in this design and just use the global primary color for the borders, etc.

Move ethereum to plugin abstraction

Move all ethereum nodes into @magickml/plugins-ethereum

Move values for ethereum key to a component that is injected into the agent manager

Variable nodes are easily corrupted

If I clone a string or prompt node it can break the whole app, destroy all my strings, etc

Add import to main menu / move to top level menu

Right now you have to go to "open" to import, import should just be in the file menu

Test contribution add

@all-contributors please add @michaelsharpe for code and platform

Add Contributors

Add alex as a contributor for past code contributions

Move all search to plugin abstraction

Speaker / sender not stored properly in db

If a component has a name, show it

Some components can be named - for example variables

We removed this functionality because it was changing the name of the actual component and breaking things.

What we want is to show the node name on the node, if it exists

This is the current

We want more like this:

import/export project

A project for now will be considered all the spells in a person's workspace, since everything currently is local and personal. A project will be all the spells in the DB, along with a file spellbook.json or similar which is at the root. The spellbook will list the name of all the spells. All spells, plus this spellbook.json, will be zipped and downloaded.

For import, someone selects the spellbook.json and all the spells, and uploads them. We read the spellbook and then load all the spells in.

note: spellbook may not be needed for this. But there may be metadata we want to associate anyways and it may be good to set the pattern now.

Move node.display into socket plugin

moon: hmm weird bug we're noticing, where node.display is not a function, might be related to worker fixes
parzvial: Oh yeah. We would need to remove those. On server when worker runs node.display isn’t a function since it is running on the server.
I could move the display functionality into the socket plugin when the node received the socket message from the servers worker.

Fix autosave

Autosave is currently disabled. There is a tendency, though hard to reproduce, when the autosave will case a whole spell to be wiped. It is likely on a spell reload event when the spell is empty for a brief moment, and that is registered as a diff.

Save spell isn't working

When I go to save a spell, I am unable to, because it's not doing or saying anything

Add "rename" option to edit menu

We should be able to rename a spell without having to "save as" and delete the other, kind of annoying

Migrate to Supabase, make a "demo" database, add auth

• Replace all sequelize and pg calls with prisma
• Replace postgres locally with prisma and sqlite
• Add instructions and a community cluster for developing supabase

Change comment from "undefined" to "right click to edit"

Right now the comment plugin makes a comment that is undefined. Annoying. Its should should "right click to edit"

Broken Link.

In readme.md file there is a broken link below screenshots. It is an all contributors badge which is not shown up there.

Screenshots:

Multimodal Graph

We want to be able to receive on input via REST API and then write into a Discord channel. This will require multi modality. If we have this as a standard interface then we can add plugins and connectors to it easily.

Entity server crashes

Whenever an error occurs in a node on a graph, it crashes the whole entity server. We need much better error reporting and handling on the entity server so this doesn't happen.

Add self to contributors

Jupyter Notebook Integration

Allow users to call a running Notebook from their Jupyter server by name (ideally validated via dropdown menu)

Bounty here
https://app.dework.xyz/m3/ai-43567?taskId=7b7d8230-9e9f-4add-a090-06781768cf30

Move agent window to drawer

The side drawer should be populated with functionality that pertains to a whole project, or whose UX would be really bad inside of a single window in the spell composer.

Replace Google STT with Whisper

We tried this with C++, was hard.

A better solution is to run a server on banana.dev

We currently use Google Speech to Text to figure out what people are saying, but Whisper is better quality and can be run locally. A C++ version is here, which can be ported to WASM as well:https://github.com/ggerganov/whisper.cpp

https://app.dework.xyz/m3/ai-43567/view/board-lblry0sg?taskId=0e1e9ec5-a406-47bc-ac26-0b169dc48e79

Clean http/https handling

Currently, users are getting stuck starting up because of invalid self-signed certificates.

We should either default to http and unsigned (which may have ramifications for some of our services and pipelines) or solve our certificate stuff and just have everything clean and working so https works everywhere locally and http works when users are behind tls

(feathers) Ensure Discord bot closes and starts back up

Currently, Discord bot can hold its connection and receive messages twice. We need to make sure that agents die completely when reset, and that Discord connection is properly handled.

Agent should never respond twice, basically. If the root spell is saved, or the front end is reset, or anything happens that could trigger a reset (hot reload), we want to make sure there are no issues.

** UPDATE:

I have added some logic to isolate the double-agents. However, if the discord bot crashes, we need to make sure it dies and restarts successfully.

Async responses

Right now the whole graph has to complete before returning.

We need to be able to output and trigger responses asynchronously to the graph completing.

For one, this will enable multiple outputs.

Also, this will enable us to run memory tasks etc after responding, which should dramatically improve response time.

This is a prerequisite to multimodal graphs with inputs and outputs from other sources.

(feathers) make sure plugins only register once, even if hot reloaded client

It seems like we might have some double reloading issues from hotreloading. Please review logs and ensure that when saving a file in client or server, or when reloading the client, plugins are not hot reloaded twice.

the goal is to make sure that our hotreloading is robust to any strangeness that may have arison from recent changes.

Add blog contributors

Electron app

Package app into stand alone electron for easy download and experimentation. Will internally use sqlLite. Implement with Capacitor.

Pressing Delete/Backspace while writting in input or textarea deletes the selected component

Replace JSON with stringified JSON in database, make default database local sqlite

We want to remove the docker dependency so that users can quickstart without it, using a local sqlite instance. However, our current models use JSON. We can dynamically set which we startup with, but only if the models use the same supported features.

• Replace JSON with stringified JSON in models
• Make sure calls are creating JSON from the returned text
• Add default to sqlite in model and .env
• Update README

Add projects

Provide channel type to Agent from Discord

We want to present the channel ID and channel type (msg vs DM), this is mostly hooked up but not provided throught Discord adapter so it is undefined in the agent type.

Create a test suite of Node tests to make Magick robust.

We should really have a handful of core tests that ensure essential functionality is working. Subspells. Spell runner. Core components.

This might be a single robust spell with subspells that runs on PR that tests almost everything. Though AI services are mocked. Interface makes this easy to do.

We should be able to ensure that all basic functionality for now is secure once main branch is stable. We can expand to more robust and granular tests after.

Python Integration

Note, this has been claimed by volit on Discord, who has until Jan 10 to report results. Anyone else may discuss with him to speed it up.
https://app.dework.xyz/m3/ai-43567/view/board-lblry0sg?taskId=02418aba-715b-4255-9524-87b9c2e7c1c2