woooodyy/agentgym

Code and implementations for the paper "AgentGym: Evolving Large Language Model-based Agents across Diverse Environments" by Zhiheng Xi et al.

agentgym's Issues

运行conda create -n webshop -f environment.yml 遇到问题

你好，想问一下运行conda create -n webshop -f environment.yml的过程中遇到了安装问题：
Retrieving notices: ...working... done
Channels:

• defaults
  Platform: linux-64
  Collecting package metadata (repodata.json): done
  Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

• environment.yml

Current channels:

• defaults

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

请问一下应该如何解决

pip install doesn't work

Facing the following error when I do pip install agentenv:

Defaulting to user installation because normal site-packages is not writeable
ERROR: Ignored the following versions that require a different python version: 0.0.1 Requires-Python >=3.10
ERROR: Could not find a version that satisfies the requirement agentenv (from versions: none)
ERROR: No matching distribution found for agentenv

可以在readme指明那些环境需要docker/外网吗

如题，我感觉是不是就是webarea和tool系列需要

requests.exceptions.RequestException: Failed to create environment:

I try to follow the directions to set up an evaluation for WebShop (I assume the way to run other tasks is similar but you just activate those servers and then change the task name in the script)

I followed the instructions here and launched the server: https://github.com/WooooDyy/AgentGym/blob/main/docs/tutorials/en/01-evaluation.md

But when I try to run the code provided to get the actual evaluation, I get this error: requests.exceptions.RequestException: Failed to create environment: <Response [500]>

how do I fix this? I followed all the instructions.

python theirEvalCode.py 
/u/isha/miniconda3/envs/agentGym/lib/python3.12/site-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
Downloading shards: 100%|█████████████████| 3/3 [00:00<00:00, 448.76it/s]
Loading checkpoint shards: 100%|███████████| 3/3 [00:24<00:00,  8.16s/it]
Traceback (most recent call last):
  File "/u/isha/agents_new/AgentGym/outputs/theirEvalCode.py", line 18, in <module>
    WebshopTask(
  File "/u/isha/agents_new/AgentGym/agentenv/agentenv/envs/webshop.py", line 108, in __init__
    super().__init__(client_args, n_clients, *args, **kwargs)
  File "/u/isha/agents_new/AgentGym/agentenv/agentenv/controller/task.py", line 54, in __init__
    self.clients = [self.env_client_cls(**client_args) for _ in range(n_clients)]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/isha/agents_new/AgentGym/agentenv/agentenv/envs/webshop.py", line 34, in __init__
    raise RequestException(f"Failed to create environment: {ok}")
requests.exceptions.RequestException: Failed to create environment: <Response [500]>

The file "AgentEnvironments/goals.json" is missing.

I encountered the following error while running the eval_agentlm_webshop.py file according to the process in "Evaluation: 01-evaluation":

with open("/mnt/petrelfs/chenwenxiang/AgentEnvironments/goals.json","w") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/petrelfs/chenwenxiang/AgentEnvironments/goals.json'

How can I resolve this issue? Is it possible that the goals.json file has not been released yet?

Inconsistent number of instructions for sciworld_test.json on HF dataset

Dear authors,

Thanks for your great work!

I'm trying to reproduce the evaluation results as shown in the paper. However, I just noticed a difference in the number of instructions between the paper and the code.

Table 2 of the paper says there are 200 evaluation instructions for the Sciworld environment, but there are 1042 samples in the sciworld_test.json on AgentEval HF dataset. Also, the conversation contents should be [], rather than all the trajectories.

Could you please update the sciworld_test.json file on HF datasets to the correct version, which should contain 200 samples and is without any conversation content?

Thanks in advance.

loss字段在训练中没有用到吗？

 {
        "conversations": [
            {
                "from": "human",
                "loss": null,
                "value": "You are web shopping.\nI will give you instructions about what to do.\nYou have to follow the instructions.\nEvery round I will give you an observation and a list of available actions, you have to respond an action based on the state and instruction.\nYou can use search action if search is available.\nYou can click one of the buttons in clickables.\nAn action should be of the following structure:\nsearch[keywords]\nclick[value]\nIf the action is not valid, perform nothing.\nKeywords in search are up to you, but the value in click must be a value in the list of available actions.\nRemember that your keywords in search should be carefully designed.\nYour response should use the following format:\n\nThought:\nI think ... \n\nAction: \nclick[something]"
            },
            {
                "from": "gpt",
                "loss": false,
                "value": "Ok."
            },
            {
                "from": "human",
                "loss": null,
                "value": "WebShop [SEP] Instruction: [SEP] Find me men's shorts with elastic waist, classic fit, short sleeve for gym workout with color: navy, and size: x-large, and price lower than 50.00 dollars [SEP] Search"
            },
            {
                "from": "gpt",
                "loss": true,
                "value": "Thought:\nI think I should use the search bar to look for the shorts that match the given requirements.\n\nAction:\nsearch[men's shorts elastic waist classic fit short sleeve gym workout navy x-large price < 50.00]"
            }
]
 },

如题,bc中loss有true和false，但貌似训练的时候并没有检查loss而将所有from等于gpt的loss都改成了true？

agentenv-sqlgym环境配置错误

您好，当我配置agentenv-sqlgym环境时遇到错误，无法解决：
pkg_info.replace_header("Metadata-Version", "2.1")
KeyError：'metadata-version'

详细信息如下：
Building wheels for collected packages: agentenv_sqlgym
Building wheel for agentenv_sqlgym (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for agentenv_sqlgym (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [54 lines of output]
running bdist_wheel
running build
running build_py
running egg_info
writing agentenv_sqlgym.egg-info/PKG-INFO
writing dependency_links to agentenv_sqlgym.egg-info/dependency_links.txt
writing entry points to agentenv_sqlgym.egg-info/entry_points.txt
writing requirements to agentenv_sqlgym.egg-info/requires.txt
writing top-level names to agentenv_sqlgym.egg-info/top_level.txt
reading manifest file 'agentenv_sqlgym.egg-info/SOURCES.txt'
writing manifest file 'agentenv_sqlgym.egg-info/SOURCES.txt'
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
running install_egg_info
Copying agentenv_sqlgym.egg-info to build/bdist.linux-x86_64/wheel/agentenv_sqlgym-0.0.1-py3.8.egg-info
running install_scripts
Traceback (most recent call last):
File "/root/miniforge3/envs/agentenv-sqlgym/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/root/miniforge3/envs/agentenv-sqlgym/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/root/miniforge3/envs/agentenv-sqlgym/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 415, in build_wheel
return self._build_with_temp_dir(
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
self.run_setup()
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 497, in run_setup
super().run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 313, in run_setup
exec(code, locals())
File "", line 1, in
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/init.py", line 106, in setup
return distutils.core.setup(**attrs)
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 970, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 974, in run_command
super().run_command(command)
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/command/bdist_wheel.py", line 425, in run
self.egg2dist(self.egginfo_dir, distinfo_dir)
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/command/bdist_wheel.py", line 563, in egg2dist
pkg_info = pkginfo_to_metadata(egginfo_path, pkginfo_path)
File "/tmp/pip-build-env-p6enm782/overlay/lib/python3.8/site-packages/setuptools/_vendor/wheel/metadata.py", line 150, in pkginfo_to_metadata
pkg_info.replace_header("Metadata-Version", "2.1")
File "/root/miniforge3/envs/agentenv-sqlgym/lib/python3.8/email/message.py", line 558, in replace_header
raise KeyError(_name)
KeyError: 'metadata-version'

A few errors when running setup.sh on webshop

I followed the readme.md and run:

conda env create -n agentenv-webshop -f environment.yml
conda activate agentenv-webshop
bash ./setup.sh

Here are the errors when running ./setup.sh:

The following packages will be UPDATED:

  openjdk                                11.0.21-h4260e57_0 --> 11.0.23-h24d6bf4_0 

The following packages will be DOWNGRADED:

  alsa-lib                                1.2.12-h4ab18f5_0 --> 1.2.11-hd590300_1 


Proceed ([y]/n)? 


Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Traceback (most recent call last):
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/spacy/__init__.py", line 14, in <module>
    from . import pipeline  # noqa: F401
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/spacy/pipeline/__init__.py", line 1, in <module>
    from .attributeruler import AttributeRuler
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/spacy/pipeline/attributeruler.py", line 6, in <module>
    from .pipe import Pipe
  File "spacy/pipeline/pipe.pyx", line 8, in init spacy.pipeline.pipe
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/spacy/training/__init__.py", line 11, in <module>
    from .callbacks import create_copy_from_base_model  # noqa: F401
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/spacy/training/callbacks.py", line 3, in <module>
    from ..language import Language
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/spacy/language.py", line 25, in <module>
    from .training.initialize import init_vocab, init_tok2vec
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/spacy/training/initialize.py", line 14, in <module>
    from .pretrain import get_tok2vec_ref
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/spacy/training/pretrain.py", line 16, in <module>
    from ..schemas import ConfigSchemaPretrain
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/spacy/schemas.py", line 216, in <module>
    class TokenPattern(BaseModel):
  File "pydantic/main.py", line 299, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 411, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 342, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 451, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 545, in pydantic.fields.ModelField._type_analysis
  File "pydantic/fields.py", line 550, in pydantic.fields.ModelField._type_analysis
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/typing.py", line 774, in __subclasscheck__
    return issubclass(cls, self.__origin__)
TypeError: issubclass() arg 1 must be a class
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
Traceback (most recent call last):
  File "convert_product_file_format.py", line 7, in <module>
    from web_agent_site.engine.engine import load_products
  File "/home/joaoabrantis/qd/AgentGym/agentenv-webshop/webshop/search_engine/../web_agent_site/engine/engine.py", line 16, in <module>
    from pyserini.search.lucene import LuceneSearcher
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/pyserini/search/__init__.py", line 19, in <module>
    from .lucene import JLuceneSearcherResult, LuceneSimilarities, LuceneFusionSearcher, LuceneSearcher
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/pyserini/search/lucene/__init__.py", line 18, in <module>
    from ._impact_searcher import JImpactSearcherResult, LuceneImpactSearcher
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/pyserini/search/lucene/_impact_searcher.py", line 28, in <module>
    from pyserini.encode import QueryEncoder, TokFreqQueryEncoder, UniCoilQueryEncoder, \
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/pyserini/encode/__init__.py", line 17, in <module>
    from ._base import DocumentEncoder, QueryEncoder, JsonlCollectionIterator,\
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/pyserini/encode/_base.py", line 19, in <module>
    import faiss
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/faiss/__init__.py", line 16, in <module>
    from .loader import *
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/faiss/loader.py", line 65, in <module>
    from .swigfaiss import *
  File "/home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages/faiss/swigfaiss.py", line 13, in <module>
    from . import _swigfaiss
ImportError: numpy.core.multiarray failed to import
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2024-06-19 10:54:55,992 INFO  [main] index.IndexCollection (IndexCollection.java:645) - Setting log level to INFO
2024-06-19 10:54:55,993 INFO  [main] index.IndexCollection (IndexCollection.java:648) - Starting indexer...
2024-06-19 10:54:55,993 INFO  [main] index.IndexCollection (IndexCollection.java:649) - ============ Loading Parameters ============
2024-06-19 10:54:55,993 INFO  [main] index.IndexCollection (IndexCollection.java:650) - DocumentCollection path: resources_100
2024-06-19 10:54:55,993 INFO  [main] index.IndexCollection (IndexCollection.java:651) - CollectionClass: JsonCollection
2024-06-19 10:54:55,994 INFO  [main] index.IndexCollection (IndexCollection.java:652) - Generator: DefaultLuceneDocumentGenerator
2024-06-19 10:54:55,994 INFO  [main] index.IndexCollection (IndexCollection.java:653) - Threads: 1
2024-06-19 10:54:55,994 INFO  [main] index.IndexCollection (IndexCollection.java:654) - Language: en
2024-06-19 10:54:55,994 INFO  [main] index.IndexCollection (IndexCollection.java:655) - Stemmer: porter
2024-06-19 10:54:55,994 INFO  [main] index.IndexCollection (IndexCollection.java:656) - Keep stopwords? false
2024-06-19 10:54:55,994 INFO  [main] index.IndexCollection (IndexCollection.java:657) - Stopwords: null
2024-06-19 10:54:55,994 INFO  [main] index.IndexCollection (IndexCollection.java:658) - Store positions? true
2024-06-19 10:54:55,994 INFO  [main] index.IndexCollection (IndexCollection.java:659) - Store docvectors? true
2024-06-19 10:54:55,994 INFO  [main] index.IndexCollection (IndexCollection.java:660) - Store document "contents" field? false
2024-06-19 10:54:55,995 INFO  [main] index.IndexCollection (IndexCollection.java:661) - Store document "raw" field? true
2024-06-19 10:54:55,995 INFO  [main] index.IndexCollection (IndexCollection.java:662) - Additional fields to index: []
2024-06-19 10:54:55,995 INFO  [main] index.IndexCollection (IndexCollection.java:663) - Optimize (merge segments)? false
2024-06-19 10:54:55,995 INFO  [main] index.IndexCollection (IndexCollection.java:664) - Whitelist: null
2024-06-19 10:54:55,995 INFO  [main] index.IndexCollection (IndexCollection.java:665) - Pretokenized?: false
2024-06-19 10:54:55,995 INFO  [main] index.IndexCollection (IndexCollection.java:685) - Directly building Lucene indexes...
2024-06-19 10:54:55,995 INFO  [main] index.IndexCollection (IndexCollection.java:686) - Index path: indexes_100
2024-06-19 10:54:55,996 INFO  [main] index.IndexCollection (IndexCollection.java:741) - ============ Indexing Collection ============
2024-06-19 10:54:56,109 INFO  [main] index.IndexCollection (IndexCollection.java:845) - Thread pool with 1 threads initialized.
2024-06-19 10:54:56,110 INFO  [main] index.IndexCollection (IndexCollection.java:847) - Initializing collection in resources_100
2024-06-19 10:54:56,111 INFO  [main] index.IndexCollection (IndexCollection.java:856) - 0 files found
2024-06-19 10:54:56,111 INFO  [main] index.IndexCollection (IndexCollection.java:857) - Starting to index...
2024-06-19 10:54:56,120 INFO  [main] index.IndexCollection (IndexCollection.java:941) - Indexing Complete! 0 documents indexed
2024-06-19 10:54:56,120 INFO  [main] index.IndexCollection (IndexCollection.java:942) - ============ Final Counter Values ============
2024-06-19 10:54:56,121 INFO  [main] index.IndexCollection (IndexCollection.java:943) - indexed:                0
2024-06-19 10:54:56,121 INFO  [main] index.IndexCollection (IndexCollection.java:944) - unindexable:            0
2024-06-19 10:54:56,121 INFO  [main] index.IndexCollection (IndexCollection.java:945) - empty:                  0
2024-06-19 10:54:56,121 INFO  [main] index.IndexCollection (IndexCollection.java:946) - skipped:                0
2024-06-19 10:54:56,121 INFO  [main] index.IndexCollection (IndexCollection.java:947) - errors:                 0
2024-06-19 10:54:56,124 INFO  [main] index.IndexCollection (IndexCollection.java:950) - Total 0 documents indexed in 00:00:00
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2024-06-19 10:54:56,972 INFO  [main] index.IndexCollection (IndexCollection.java:645) - Setting log level to INFO
2024-06-19 10:54:56,973 INFO  [main] index.IndexCollection (IndexCollection.java:648) - Starting indexer...
2024-06-19 10:54:56,973 INFO  [main] index.IndexCollection (IndexCollection.java:649) - ============ Loading Parameters ============
2024-06-19 10:54:56,974 INFO  [main] index.IndexCollection (IndexCollection.java:650) - DocumentCollection path: resources
2024-06-19 10:54:56,974 INFO  [main] index.IndexCollection (IndexCollection.java:651) - CollectionClass: JsonCollection
2024-06-19 10:54:56,974 INFO  [main] index.IndexCollection (IndexCollection.java:652) - Generator: DefaultLuceneDocumentGenerator
2024-06-19 10:54:56,974 INFO  [main] index.IndexCollection (IndexCollection.java:653) - Threads: 1
2024-06-19 10:54:56,974 INFO  [main] index.IndexCollection (IndexCollection.java:654) - Language: en
2024-06-19 10:54:56,974 INFO  [main] index.IndexCollection (IndexCollection.java:655) - Stemmer: porter
2024-06-19 10:54:56,974 INFO  [main] index.IndexCollection (IndexCollection.java:656) - Keep stopwords? false
2024-06-19 10:54:56,974 INFO  [main] index.IndexCollection (IndexCollection.java:657) - Stopwords: null
2024-06-19 10:54:56,974 INFO  [main] index.IndexCollection (IndexCollection.java:658) - Store positions? true
2024-06-19 10:54:56,975 INFO  [main] index.IndexCollection (IndexCollection.java:659) - Store docvectors? true
2024-06-19 10:54:56,975 INFO  [main] index.IndexCollection (IndexCollection.java:660) - Store document "contents" field? false
2024-06-19 10:54:56,975 INFO  [main] index.IndexCollection (IndexCollection.java:661) - Store document "raw" field? true
2024-06-19 10:54:56,975 INFO  [main] index.IndexCollection (IndexCollection.java:662) - Additional fields to index: []
2024-06-19 10:54:56,975 INFO  [main] index.IndexCollection (IndexCollection.java:663) - Optimize (merge segments)? false
2024-06-19 10:54:56,975 INFO  [main] index.IndexCollection (IndexCollection.java:664) - Whitelist: null
2024-06-19 10:54:56,975 INFO  [main] index.IndexCollection (IndexCollection.java:665) - Pretokenized?: false
2024-06-19 10:54:56,975 INFO  [main] index.IndexCollection (IndexCollection.java:685) - Directly building Lucene indexes...
2024-06-19 10:54:56,975 INFO  [main] index.IndexCollection (IndexCollection.java:686) - Index path: indexes
2024-06-19 10:54:56,976 INFO  [main] index.IndexCollection (IndexCollection.java:741) - ============ Indexing Collection ============
2024-06-19 10:54:57,086 INFO  [main] index.IndexCollection (IndexCollection.java:845) - Thread pool with 1 threads initialized.
2024-06-19 10:54:57,086 INFO  [main] index.IndexCollection (IndexCollection.java:847) - Initializing collection in resources
2024-06-19 10:54:57,087 INFO  [main] index.IndexCollection (IndexCollection.java:856) - 0 files found
2024-06-19 10:54:57,087 INFO  [main] index.IndexCollection (IndexCollection.java:857) - Starting to index...
2024-06-19 10:54:57,096 INFO  [main] index.IndexCollection (IndexCollection.java:941) - Indexing Complete! 0 documents indexed
2024-06-19 10:54:57,096 INFO  [main] index.IndexCollection (IndexCollection.java:942) - ============ Final Counter Values ============
2024-06-19 10:54:57,096 INFO  [main] index.IndexCollection (IndexCollection.java:943) - indexed:                0
2024-06-19 10:54:57,096 INFO  [main] index.IndexCollection (IndexCollection.java:944) - unindexable:            0
2024-06-19 10:54:57,096 INFO  [main] index.IndexCollection (IndexCollection.java:945) - empty:                  0
2024-06-19 10:54:57,096 INFO  [main] index.IndexCollection (IndexCollection.java:946) - skipped:                0
2024-06-19 10:54:57,096 INFO  [main] index.IndexCollection (IndexCollection.java:947) - errors:                 0
2024-06-19 10:54:57,099 INFO  [main] index.IndexCollection (IndexCollection.java:950) - Total 0 documents indexed in 00:00:00
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2024-06-19 10:54:57,892 INFO  [main] index.IndexCollection (IndexCollection.java:645) - Setting log level to INFO
2024-06-19 10:54:57,893 INFO  [main] index.IndexCollection (IndexCollection.java:648) - Starting indexer...
2024-06-19 10:54:57,893 INFO  [main] index.IndexCollection (IndexCollection.java:649) - ============ Loading Parameters ============
2024-06-19 10:54:57,893 INFO  [main] index.IndexCollection (IndexCollection.java:650) - DocumentCollection path: resources_1k
2024-06-19 10:54:57,893 INFO  [main] index.IndexCollection (IndexCollection.java:651) - CollectionClass: JsonCollection
2024-06-19 10:54:57,893 INFO  [main] index.IndexCollection (IndexCollection.java:652) - Generator: DefaultLuceneDocumentGenerator
2024-06-19 10:54:57,893 INFO  [main] index.IndexCollection (IndexCollection.java:653) - Threads: 1
2024-06-19 10:54:57,894 INFO  [main] index.IndexCollection (IndexCollection.java:654) - Language: en
2024-06-19 10:54:57,894 INFO  [main] index.IndexCollection (IndexCollection.java:655) - Stemmer: porter
2024-06-19 10:54:57,894 INFO  [main] index.IndexCollection (IndexCollection.java:656) - Keep stopwords? false
2024-06-19 10:54:57,894 INFO  [main] index.IndexCollection (IndexCollection.java:657) - Stopwords: null
2024-06-19 10:54:57,894 INFO  [main] index.IndexCollection (IndexCollection.java:658) - Store positions? true
2024-06-19 10:54:57,894 INFO  [main] index.IndexCollection (IndexCollection.java:659) - Store docvectors? true
2024-06-19 10:54:57,894 INFO  [main] index.IndexCollection (IndexCollection.java:660) - Store document "contents" field? false
2024-06-19 10:54:57,894 INFO  [main] index.IndexCollection (IndexCollection.java:661) - Store document "raw" field? true
2024-06-19 10:54:57,894 INFO  [main] index.IndexCollection (IndexCollection.java:662) - Additional fields to index: []
2024-06-19 10:54:57,895 INFO  [main] index.IndexCollection (IndexCollection.java:663) - Optimize (merge segments)? false
2024-06-19 10:54:57,895 INFO  [main] index.IndexCollection (IndexCollection.java:664) - Whitelist: null
2024-06-19 10:54:57,895 INFO  [main] index.IndexCollection (IndexCollection.java:665) - Pretokenized?: false
2024-06-19 10:54:57,895 INFO  [main] index.IndexCollection (IndexCollection.java:685) - Directly building Lucene indexes...
2024-06-19 10:54:57,895 INFO  [main] index.IndexCollection (IndexCollection.java:686) - Index path: indexes_1k
2024-06-19 10:54:57,896 INFO  [main] index.IndexCollection (IndexCollection.java:741) - ============ Indexing Collection ============
2024-06-19 10:54:58,007 INFO  [main] index.IndexCollection (IndexCollection.java:845) - Thread pool with 1 threads initialized.
2024-06-19 10:54:58,007 INFO  [main] index.IndexCollection (IndexCollection.java:847) - Initializing collection in resources_1k
2024-06-19 10:54:58,008 INFO  [main] index.IndexCollection (IndexCollection.java:856) - 0 files found
2024-06-19 10:54:58,008 INFO  [main] index.IndexCollection (IndexCollection.java:857) - Starting to index...
2024-06-19 10:54:58,017 INFO  [main] index.IndexCollection (IndexCollection.java:941) - Indexing Complete! 0 documents indexed
2024-06-19 10:54:58,017 INFO  [main] index.IndexCollection (IndexCollection.java:942) - ============ Final Counter Values ============
2024-06-19 10:54:58,017 INFO  [main] index.IndexCollection (IndexCollection.java:943) - indexed:                0
2024-06-19 10:54:58,017 INFO  [main] index.IndexCollection (IndexCollection.java:944) - unindexable:            0
2024-06-19 10:54:58,017 INFO  [main] index.IndexCollection (IndexCollection.java:945) - empty:                  0
2024-06-19 10:54:58,017 INFO  [main] index.IndexCollection (IndexCollection.java:946) - skipped:                0
2024-06-19 10:54:58,017 INFO  [main] index.IndexCollection (IndexCollection.java:947) - errors:                 0
2024-06-19 10:54:58,020 INFO  [main] index.IndexCollection (IndexCollection.java:950) - Total 0 documents indexed in 00:00:00
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2024-06-19 10:54:58,872 INFO  [main] index.IndexCollection (IndexCollection.java:645) - Setting log level to INFO
2024-06-19 10:54:58,873 INFO  [main] index.IndexCollection (IndexCollection.java:648) - Starting indexer...
2024-06-19 10:54:58,873 INFO  [main] index.IndexCollection (IndexCollection.java:649) - ============ Loading Parameters ============
2024-06-19 10:54:58,873 INFO  [main] index.IndexCollection (IndexCollection.java:650) - DocumentCollection path: resources_100k
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:651) - CollectionClass: JsonCollection
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:652) - Generator: DefaultLuceneDocumentGenerator
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:653) - Threads: 1
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:654) - Language: en
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:655) - Stemmer: porter
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:656) - Keep stopwords? false
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:657) - Stopwords: null
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:658) - Store positions? true
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:659) - Store docvectors? true
2024-06-19 10:54:58,874 INFO  [main] index.IndexCollection (IndexCollection.java:660) - Store document "contents" field? false
2024-06-19 10:54:58,875 INFO  [main] index.IndexCollection (IndexCollection.java:661) - Store document "raw" field? true
2024-06-19 10:54:58,875 INFO  [main] index.IndexCollection (IndexCollection.java:662) - Additional fields to index: []
2024-06-19 10:54:58,875 INFO  [main] index.IndexCollection (IndexCollection.java:663) - Optimize (merge segments)? false
2024-06-19 10:54:58,875 INFO  [main] index.IndexCollection (IndexCollection.java:664) - Whitelist: null
2024-06-19 10:54:58,875 INFO  [main] index.IndexCollection (IndexCollection.java:665) - Pretokenized?: false
2024-06-19 10:54:58,875 INFO  [main] index.IndexCollection (IndexCollection.java:685) - Directly building Lucene indexes...
2024-06-19 10:54:58,875 INFO  [main] index.IndexCollection (IndexCollection.java:686) - Index path: indexes_100k
2024-06-19 10:54:58,876 INFO  [main] index.IndexCollection (IndexCollection.java:741) - ============ Indexing Collection ============
2024-06-19 10:54:58,986 INFO  [main] index.IndexCollection (IndexCollection.java:845) - Thread pool with 1 threads initialized.
2024-06-19 10:54:58,986 INFO  [main] index.IndexCollection (IndexCollection.java:847) - Initializing collection in resources_100k
2024-06-19 10:54:58,987 INFO  [main] index.IndexCollection (IndexCollection.java:856) - 0 files found
2024-06-19 10:54:58,987 INFO  [main] index.IndexCollection (IndexCollection.java:857) - Starting to index...
2024-06-19 10:54:58,996 INFO  [main] index.IndexCollection (IndexCollection.java:941) - Indexing Complete! 0 documents indexed
2024-06-19 10:54:58,996 INFO  [main] index.IndexCollection (IndexCollection.java:942) - ============ Final Counter Values ============
2024-06-19 10:54:58,996 INFO  [main] index.IndexCollection (IndexCollection.java:943) - indexed:                0
2024-06-19 10:54:58,996 INFO  [main] index.IndexCollection (IndexCollection.java:944) - unindexable:            0
2024-06-19 10:54:58,996 INFO  [main] index.IndexCollection (IndexCollection.java:945) - empty:                  0
2024-06-19 10:54:58,996 INFO  [main] index.IndexCollection (IndexCollection.java:946) - skipped:                0
2024-06-19 10:54:58,997 INFO  [main] index.IndexCollection (IndexCollection.java:947) - errors:                 0
2024-06-19 10:54:58,999 INFO  [main] index.IndexCollection (IndexCollection.java:950) - Total 0 documents indexed in 00:00:00
Downloading 50 example human trajectories...
Downloading example trajectories complete
Collecting typing_extensions<4.6.0
  Using cached typing_extensions-4.5.0-py3-none-any.whl.metadata (8.5 kB)
Collecting gym==0.23.1
  Using cached gym-0.23.1-py3-none-any.whl
Requirement already satisfied: numpy>=1.18.0 in /home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages (from gym==0.23.1) (1.22.4)
Requirement already satisfied: cloudpickle>=1.2.0 in /home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages (from gym==0.23.1) (3.0.0)
Requirement already satisfied: gym-notices>=0.0.4 in /home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages (from gym==0.23.1) (0.0.8)
Requirement already satisfied: importlib-metadata>=4.10.0 in /home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages (from gym==0.23.1) (7.1.0)
Requirement already satisfied: zipp>=0.5 in /home/joaoabrantis/miniconda3/envs/agentenv-webshop/lib/python3.8/site-packages (from importlib-metadata>=4.10.0->gym==0.23.1) (3.19.2)
Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Installing collected packages: typing_extensions, gym
  Attempting uninstall: typing_extensions
    Found existing installation: typing_extensions 4.12.2
    Uninstalling typing_extensions-4.12.2:
      Successfully uninstalled typing_extensions-4.12.2
  Attempting uninstall: gym
    Found existing installation: gym 0.24.0
    Uninstalling gym-0.24.0:
      Successfully uninstalled gym-0.24.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
fastapi 0.110.3 requires typing-extensions>=4.8.0, but you have typing-extensions 4.5.0 which is incompatible.
Successfully installed gym-0.23.1 typing_extensions-4.5.0
DEPRECATION: https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.3.0/en_core_web_lg-3.3.0-py3-none-any.whl#egg=en_core_web_lg==3.3.0 contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at https://github.com/pypa/pip/issues/11617

Questions about example training scripts

Hi,
I have some questions about parameters in the agentevol training script:

1. Where can we get the training data? iter_data_path="./iter_data/iter_data_all_task"
2. What's the difference between the test step and the inference step?
3. Where can we get the inference data?
   inference_file_list=("webshop.json" "alfworld.json" "textcraft.json" "sciworld.json" "sqlgym.json" "wordle.json" "maze.json" "babyai.json" "weather.json" "movie.json" "todo.json" "academia.json" "sheet.json" "webarena.json")
cur_inference_file=./small_exploration_data/${inference_file_list[$index]}
4. What do I need to change to train a model that does NOT use the Llama 2 Chat template?

Thanks,

Question about evolution

Hi, thanks for releasing such an interesting work!
I have a question about the evolution procedure. According to the objective, if the reward r(e,u,r) is binary, is it just equal to selecting successful trajectories and using them as training data?
I don't know if I understand it correctly. Looking for the reply. :)