kordinglab/llm4teach-freetext-server

freetext

A LLM-guided feedback tool that guides freeform essay responses, for educational use.

Installation

poetry install

Before you can use the server, you must either (1) change the assignment and response stores in server.py to run locally (e.g., JSON stores), or (2) update the AWS credentials in the config.py file. Note that no config.py is provided; you must create a new file. But we do provide a config.example.py file that you can use as a template. See that file for itemwise documentation of the configuration options.

Usage

For development server usage (with live reloads), run:

uv run uvicorn freetext.server:app --reload --port 9900

For production server usage, run:

uv run uvicorn freetext.server:app --port 9900

Examples

See the manuscript for examples.

Our Paper

https://arxiv.org/abs/2308.02439

Full-resolution versions of all images and tables from this publication are available at https://llm4edu.experiments.kordinglab.com/paper.

The FreeText server will be hosted temporarily for public use at https://llm4edu.experiments.kordinglab.com/app, with an interactive example assignment available at https://llm4edu.experiments.kordinglab.com/app/assignments/1393754a-d80f-474d-bff7-b1fec36cdbb7. Educators may contact us at the correspondence email of this preprint for a token, which is required to create new questions on our public instance.

Our Jupyter Notebook Widget is available on GitHub at https://github.com/KordingLab/freetext-jupyter, and is powered by the FreeText Server, which can be found at https://github.com/KordingLab/llm4teach-freetext-server.

If this work is useful to your research, please cite the following:

@misc{matelsky2023large,
    title={A large language model-assisted education tool to provide feedback on open-ended responses},
    author={Jordan K. Matelsky and Felipe Parodi and Tony Liu and Richard D. Lange and Konrad P. Kording},
    year={2023},
    eprint={2308.02439},
    archivePrefix={arXiv},
    primaryClass={cs.CY}
}

---

Made with 💚 at the Kording Lab

llm4teach-freetext-server's Issues

Remove assignment private data when responding to API requests

Interactively determine criteria

• Instructor writes question.
• LLM responds and is automatically graded. If it fails the criteria, flag to instructor to be more explicit in question.

Also:

• Instructor writes question.
• LLM automatically generates set of criteria for approval.

Supporting multiple users/courses in a single server

It would be nice if we supported scaling to multiple users/courses on a single server. That being said, permissions are a pain and many of these ideas could be offloaded to a wrapper service like a fully-featured CMS. Just brainstorming some "would be nice" features for the purposes of scaling demo v0.2 to multiple users (like a department or university) on a single server:

• User– and Course–level permissions for reading and writing assignments
• Interface for editing prompts
• Select among different feedback styles, store prompt ID inside the assignment
• UI for browsing list of assignments

Add basic secret-auth for question creation

...to temporarily scope instructor userbase until we have a ROM on cost per question.

Set tone and audience configurably

Rewrite OpenAI feedback provider to use `guidance`'s chat mode

This will enable us to use the (much cheaper) GPT-3.5 Turbo model instead of the current davinci.