jjhw/adas

Automated Design of Agentic Systems

Automated Design of Agentic Systems

In this work, we describe a newly forming research area Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways.

We present a simple yet effective ADAS algorithm named Meta Agent Search to demonstrate that agents can invent novel and powerful agent designs. In Meta Agent Search, a "meta" agent iteratively programs interesting new agents in code based on previous discoveries.

Setup

conda create -n adas python=3.11
conda activate adas
pip install -r requirements.txt

# provide your OpenAI API key
export OPENAI_API_KEY="YOUR KEY HERE"

Running Instructions

Running Meta Agent Search

To run experiments for each domain, navigate to its respective folder. The code in each folder is self-contained. Launch experiments using the search.py script located in each domain's folder.

python {DOMAIN}/search.py

Replace {DOMAIN} with the specific domain folder name {_arc, _drop, _mgsm, ...} to run the experiment for.

Customizing Meta Agent Search for New Domains

You can easily adapt the code to search for new domains. To do so, follow these steps:

1. Modify the evaluate_forward_fn() function and adjust any necessary formatting prompts (e.g. this line) in the search.py file.

2. Consider adding additional basic functions for the meta agent to utilize during the design process (similar to this line).

3. Update the domain-specific information within the prompts to match the requirements of your new domain (e.g. this line).

4. Run the search and evaluation on your new domain.

Safety Consideration

Warning

The code in this repository involves executing untrusted model-generated code. We strongly advise users to be aware of this safety concern. While it is highly unlikely that model-generated code will perform overtly malicious actions in our current settings and with the models we use, such code may still act destructively due to limitations in model capability or alignment. By using this repository, you acknowledge and accept these risks.

Citing

If you find this project useful, please consider citing:

@article{hu2024ADAS,
title={Automated Design of Agentic Systems},
author={Hu, Shengran and Lu, Cong and Clune, Jeff},
journal={arXiv preprint arXiv:2408.08435},
year={2024}
}