microsoft/llf-bench

A benchmark for evaluating learning agents based on just language feedback

llf-bench's Issues

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Environment name error

Following the README file and running the following code with the latest version would yield error: Environment llf-Gridworld-v0 doesn't exist.

import llfbench as gym
env = gym.make('llf-Gridworld-v0')

I looked into the __init__ file, line 5 is causing the issue env = gym.make(env_name), where the "gym" is the gymnasium rather than llf-bench.

Action required: migrate or opt-out of migration to GitHub inside Microsoft

Migrate non-Open Source or non-External Collaboration repositories to GitHub inside Microsoft

In order to protect and secure Microsoft, private or internal repositories in GitHub for Open Source which are not related to open source projects or require collaboration with 3rd parties (customer, partners, etc.) must be migrated to GitHub inside Microsoft a.k.a GitHub Enterprise Cloud with Enterprise Managed User (GHEC EMU).

Action

✍️ Please RSVP to opt-in or opt-out of the migration to GitHub inside Microsoft.

❗Only users with admin permission in the repository are allowed to respond. Failure to provide a response will result to your repository getting automatically archived.🔒

Instructions

Reply with a comment on this issue containing one of the following optin or optout command options below.

✅ Opt-in to migrate

@gimsvc optin --date <target_migration_date in mm-dd-yyyy format>

Example: @gimsvc optin --date 03-15-2023

OR

❌ Opt-out of migration

@gimsvc optout --reason <staging|collaboration|delete|other>

Example: @gimsvc optout --reason staging

Options:

• staging : This repository will ship as Open Source or go public
• collaboration : Used for external or 3rd party collaboration with customers, partners, suppliers, etc.
• delete : This repository will be deleted because it is no longer needed.
• other : Other reasons not specified

Need more help? 🖐️

• Email [email protected]. ✉️
• Post your questions in GitHub inside Microsoft Team in Microsoft Teams. 🗨️

Why the metaworld env is truncated to 30 steps

Why the meta world env is truncated to 30 steps, can agents complete all games within 30 steps?

poem doesn't support copy.deepcopy()

maybe other environments also have this problem, i haven't tried.

(the random generator state needs to be copied in a custom __deepcopy__ hook ...)

Missing code for alfworld and meta world

Thanks for the interesting work. I try to use this repo, but for alfworld and meta world, there seems many packages are missing. For example, in the screenshot, there's no alfworld.agents in the file system but the code try to import it. I think you used the code from alfworld directly but somehow deleted some directories. Could you please fix that? Thank [you!]