batsresearch/alfred

A system for prompted weak supervision.

alfred's Issues

How would Alfred work with Transformers' Agents?

Is your feature request related to a problem? Please describe.
Nope.

Describe the solution you'd like
Since transformers' agents are new, it would be cool to see the integration of transformer agents into Alfred for a wider audience reach. Transformers agents are similar, if I understand Alfred correctly, to the initialization of multimodal agents in Alfred, but Alfred offers other capabilities such as calibrating. Since there will be more and more people using Transformers Agents, it would be great to think about its integration for the next version of Alfred.

Describe alternatives you've considered
N/A

Additional context
N/A

Feature Request: Adding Methods to Get Embeddings

Tentative Usage:
Client.embed(Union[instance, dataset]) -> embedding(s)

Integration with LM-Eval-Harness

Is your feature request related to a problem? Please describe.
Since lm-eval-harness is going to be one of the de facto eval benchmark repo, I believe documenting the way to integrate alfred with it will be great. Since if a person wants to evaluate with Alfred, he/she has to set up the eval scripts/prompt templates by themselves, I believe this can be ameliorated if there's an integration with lm-eval-harness.

Describe the solution you'd like
An extra piece of code that connects both.

Additional context
I will work on this. Welcome any suggestions on how to make this work.