dngros/lmwrapper

An object-oriented wrapper around language models (like openai endpoints or huggingface)

lmwrapper's Issues

Accelerate Support

Accelerate allows training/inference of large models by automagically splitting the layers across CUDA devices. Initially, we had some issues with logprobs due to model forward patching. Now that we generally use transition scores, it may work with accelerate.

CodeLLaMa stop tokens sometimes breaking

There seems to be some issues with CodeLLaMa stop tokens that breaks it sometimes. I will try to get a working reproduction and understand what's going on.

CC: @claudiosv

Organization id settings via environoment variable

Add api for sampling more than once

Add some builtin support for generating multiple generations. This leaves room for more efficient generation on backends like HF (you only need to encode once). Also can potentially cache multiple generations (ideally in clever way that lets you grow the number of generations and reuse prior caches).

Let api keys be project-specific file

Get a Encoder-decoder model running in CI

Weird whitespace on first token of Mistral output

The whitespace on the first token of Mistral-7B-v0.1 seems to differ from the huggingface output. This can be fixed upstream (like when synthegrator snaps whitespce), but we need to identify the root cause.

Ability to specify differing token limits for inputs and outputs

In ba839a4 we add support for GPT-4-Turbo. However, GPT-4-Turbo behaves differently than other models. There is a large input limit (128,000 tokens), but a smaller output limit of 4096 tokens. We don't have any way of representing this currently, so features like checking if a prompt will go over and prompt trimming will not behave as expected.

Pytest cleverness to only load huggingface models once in test_huggingface

Some of the huggingface models take several minutes just to load. This makes running the slow tests even more painful. There is probably some pytest cleverness here on loading once and running each. Not sure though.

Top Tokens

Expose the probabilities of the top tokens in a prediction result

TODO:

• OpenAI
• HF Models

vLLM Support

vLLM offers an OpenAI compatible HTTP server and faster inference. We'd like to offer it as an option for lmwrapper.

Test regular NL encoder-decoder

#12 adds support for codet5+. We will want to also eventually verify that a more mainstream encoder-decoder model (like normal t5 or one of its more state-of-the-art successors) works too.