batsresearch/bonito

A lightweight library for generating synthetic instruction tuning datasets for your data without GPT.

Bonito

Bonito is an open-source model for conditional task generation: the task of converting unannotated text into task-specific training datasets for instruction tuning. This repo is a lightweight library for Bonito to easily create synthetic datasets built on top of the Hugging Face transformers and vllm libraries.

• Paper: Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation
• Model: bonito-v1
• Dataset: ctga-v1
• Code: To reproduce experiments in our paper, see nayak-aclfindings24-code.

News

• 🐟 June 2024: Bonito is accepted to ACL Findings 2024.

Installation

Create an environment and install the package using the following commands:

conda create -n bonito python=3.9
conda activate bonito
pip install -e .

Basic Usage

To generate synthetic instruction tuning dataset using Bonito, you can use the following code:

from bonito import Bonito
from vllm import SamplingParams
from datasets import load_dataset

# Initialize the Bonito model
bonito = Bonito("BatsResearch/bonito-v1")

# load dataset with unannotated text
unannotated_text = load_dataset(
    "BatsResearch/bonito-experiment",
    "unannotated_contract_nli"
)["train"].select(range(10))

# Generate synthetic instruction tuning dataset
sampling_params = SamplingParams(max_tokens=256, top_p=0.95, temperature=0.5, n=1)
synthetic_dataset = bonito.generate_tasks(
    unannotated_text,
    context_col="input",
    task_type="nli",
    sampling_params=sampling_params
)

Supported Task Types

Here we include the supported task types [full name (short form)]: extractive question answering (exqa), multiple-choice question answering (mcqa), question generation (qg), question answering without choices (qa), yes-no question answering (ynqa), coreference resolution (coref), paraphrase generation (paraphrase), paraphrase identification (paraphrase_id), sentence completion (sent_comp), sentiment (sentiment), summarization (summarization), text generation (text_gen), topic classification (topic_class), word sense disambiguation (wsd), textual entailment (te), natural language inference (nli)

You can use either the full name or the short form to specify the task_type in generate_tasks.

Tutorial

We have created a tutorial here for how to use a quantized version of the model in a Google Colab T4 instance. The quantized version was graciously contributed by user alexandreteles. We have an additional tutorial to try out the Bonito model on A100 GPU on Google Colab here.

Citation

If you use Bonito in your research, please cite the following paper:

@inproceedings{bonito:aclfindings24,
  title = {Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation},
  author = {Nayak, Nihal V. and Nan, Yiyang and Trost, Avi and Bach, Stephen H.},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024}}

bonito's Issues

Quantized versions

Hello, I've been playing around with the model and made quantized versions available at:

• alexandreteles/bonito-v1-awq (can be used directly with vllm)
• alexandreteles/bonito-v1-gguf (for llama.cpp inference)

As soon as you have decided on a license, I will update the repositories accordingly. On a different topic, are there prospects to add more conversational tasks to the dataset in the future?

Thank you for this, it is an amazing project remarkably close to something I wanted to do myself. Good luck with publishing!

Getting diverse outputs when sampling n>1

Do you have any suggestions for generating diverse outputs for a context? I've tried different sampling parameters but the it seems that the generated sequences are always the same. E.g.

context_dataset= datasets.Dataset.from_list([{"context":"""The Space Shuttle program was the fourth human spaceflight program carried out by the U.S. National Aeronautics and Space Administration (NASA)"""}])
synthetic_dataset = bonito_model.generate_tasks(
    context_dataset,
    context_col="context",
    task_type="exqa",
    sampling_params=SamplingParams(max_tokens=256, top_p=0.5, temperature=1.1, n=3),
)
for r in synthetic_dataset:
    print(r)

will generate

{'input': 'Refer to the passage below and answer the following question:\n\nPassage: The Space Shuttle program was the fourth human spaceflight program carried out by the U.S. National Aeronautics and Space Administration (NASA)\n\nQuestion: What does NASA stand for?', 'output': 'National Aeronautics and Space Administration'}
{'input': 'Refer to the passage below and answer the following question:\n\nPassage: The Space Shuttle program was the fourth human spaceflight program carried out by the U.S. National Aeronautics and Space Administration (NASA)\n\nQuestion: What does NASA stand for?', 'output': 'National Aeronautics and Space Administration'}
{'input': 'Refer to the passage below and answer the following question:\n\nPassage: The Space Shuttle program was the fourth human spaceflight program carried out by the U.S. National Aeronautics and Space Administration (NASA)\n\nQuestion: What does NASA stand for?', 'output': 'National Aeronautics and Space Administration'}

CUDA cannot be found during installation

Hello,

I have tried installing the library for this model today but failed due to the following error:

(bonito) C:\Users\kritt\Desktop\Bonito\vllm>pip install -e .
Obtaining file:///C:/Users/kritt/Desktop/Bonito/vllm
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... error
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> [21 lines of output]
C:\Users\kritt\AppData\Local\Temp\pip-build-env-v3xcni1y\overlay\Lib\site-packages\torch\nn\modules\transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_numpy.cpp:84.)
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1'
Traceback (most recent call last):
File "C:\Users\kritt.conda\envs\bonito\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in
main()
File "C:\Users\kritt.conda\envs\bonito\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "C:\Users\kritt.conda\envs\bonito\lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 132, in get_requires_for_build_editable
return hook(config_settings)
File "C:\Users\kritt\AppData\Local\Temp\pip-build-env-v3xcni1y\overlay\Lib\site-packages\setuptools\build_meta.py", line 448, in get_requires_for_build_editable
return self.get_requires_for_build_wheel(config_settings)
File "C:\Users\kritt\AppData\Local\Temp\pip-build-env-v3xcni1y\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "C:\Users\kritt\AppData\Local\Temp\pip-build-env-v3xcni1y\overlay\Lib\site-packages\setuptools\build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "C:\Users\kritt\AppData\Local\Temp\pip-build-env-v3xcni1y\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
exec(code, locals())
File "", line 446, in
File "", line 406, in get_vllm_version
NameError: name 'nvcc_cuda_version' is not defined
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I already checked/tried the followings:

• CUDA_HOME variable in the system, looked for nvcc.exe and found it in the bin folder. I believe it is already correct.

• Numpy was installed as below:

(bonito) C:\Users\kritt\Desktop\Bonito\vllm>pip install numpy
Requirement already satisfied: numpy in c:\users\kritt.conda\envs\bonito\lib\site-packages (1.26.4)

• pip install -e . always ended up with this error.
• pip install vllm also ended with the same error.

Can anyone provide guidance on how to resolve this issue?

Tutorial not running

Hello,
when trying to make bonito work for my application, I ran into some issues. To try and find the problems I wanted to run the google colab example and slowly make changes to see where my mistake was. However the tutorial colab doesn't seem to work in its original form. I have been considering trying an older version of bonito but I am not sure how to do this when cloning from git. Any help would be greatly appreciated.
This is the output from the installation where a pip Error occurs:

Cloning into 'bonito'...
remote: Enumerating objects: 91, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 91 (delta 36), reused 26 (delta 26), pack-reused 45
Receiving objects: 100% (91/91), 784.34 KiB | 34.10 MiB/s, done.
Resolving deltas: 100% (38/38), done.
Processing ./bonito
  Preparing metadata (setup.py) ... done
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (from bonito==0.0.2) (4.41.0)
Collecting datasets (from bonito==0.0.2)
  Downloading datasets-2.19.1-py3-none-any.whl (542 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 542.0/542.0 kB 7.9 MB/s eta 0:00:00
Collecting vllm (from bonito==0.0.2)
  Downloading vllm-0.4.2-cp310-cp310-manylinux1_x86_64.whl (67.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.7/67.7 MB 11.6 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (3.14.0)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (1.25.2)
Requirement already satisfied: pyarrow>=12.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (14.0.2)
Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (0.6)
Collecting dill<0.3.9,>=0.3.0 (from datasets->bonito==0.0.2)
  Downloading dill-0.3.8-py3-none-any.whl (116 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.3/116.3 kB 17.9 MB/s eta 0:00:00
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (2.0.3)
Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (2.31.0)
Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (4.66.4)
Collecting xxhash (from datasets->bonito==0.0.2)
  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.1/194.1 kB 25.0 MB/s eta 0:00:00
Collecting multiprocess (from datasets->bonito==0.0.2)
  Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 14.7 MB/s eta 0:00:00
Requirement already satisfied: fsspec[http]<=2024.3.1,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (2023.6.0)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (3.9.5)
Requirement already satisfied: huggingface-hub>=0.21.2 in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (0.23.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (24.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets->bonito==0.0.2) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers->bonito==0.0.2) (2023.12.25)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers->bonito==0.0.2) (0.19.1)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers->bonito==0.0.2) (0.4.3)
Requirement already satisfied: cmake>=3.21 in /usr/local/lib/python3.10/dist-packages (from vllm->bonito==0.0.2) (3.27.9)
Collecting ninja (from vllm->bonito==0.0.2)
  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.2/307.2 kB 18.8 MB/s eta 0:00:00
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from vllm->bonito==0.0.2) (5.9.5)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from vllm->bonito==0.0.2) (0.1.99)
Requirement already satisfied: py-cpuinfo in /usr/local/lib/python3.10/dist-packages (from vllm->bonito==0.0.2) (9.0.0)
Collecting fastapi (from vllm->bonito==0.0.2)
  Downloading fastapi-0.111.0-py3-none-any.whl (91 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.0/92.0 kB 10.3 MB/s eta 0:00:00
Collecting openai (from vllm->bonito==0.0.2)
  Downloading openai-1.30.3-py3-none-any.whl (320 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 320.6/320.6 kB 23.8 MB/s eta 0:00:00
Collecting uvicorn[standard] (from vllm->bonito==0.0.2)
  Downloading uvicorn-0.29.0-py3-none-any.whl (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 4.6 MB/s eta 0:00:00
Requirement already satisfied: pydantic>=2.0 in /usr/local/lib/python3.10/dist-packages (from vllm->bonito==0.0.2) (2.7.1)
Requirement already satisfied: prometheus-client>=0.18.0 in /usr/local/lib/python3.10/dist-packages (from vllm->bonito==0.0.2) (0.20.0)
Collecting prometheus-fastapi-instrumentator>=7.0.0 (from vllm->bonito==0.0.2)
  Downloading prometheus_fastapi_instrumentator-7.0.0-py3-none-any.whl (19 kB)
Collecting tiktoken==0.6.0 (from vllm->bonito==0.0.2)
  Downloading tiktoken-0.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 60.5 MB/s eta 0:00:00
Collecting lm-format-enforcer==0.9.8 (from vllm->bonito==0.0.2)
  Downloading lm_format_enforcer-0.9.8-py3-none-any.whl (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.9/40.9 kB 5.6 MB/s eta 0:00:00
Collecting outlines==0.0.34 (from vllm->bonito==0.0.2)
  Downloading outlines-0.0.34-py3-none-any.whl (76 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.5/76.5 kB 3.5 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from vllm->bonito==0.0.2) (4.11.0)
Collecting ray>=2.9 (from vllm->bonito==0.0.2)
  Downloading ray-2.23.0-cp310-cp310-manylinux2014_x86_64.whl (65.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.7/65.7 MB 7.4 MB/s eta 0:00:00
Collecting nvidia-ml-py (from vllm->bonito==0.0.2)
  Downloading nvidia_ml_py-12.550.52-py3-none-any.whl (39 kB)
Collecting vllm-nccl-cu12<2.19,>=2.18 (from vllm->bonito==0.0.2)
  Downloading vllm_nccl_cu12-2.18.1.0.4.0.tar.gz (6.2 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: torch==2.3.0 in /usr/local/lib/python3.10/dist-packages (from vllm->bonito==0.0.2) (2.3.0+cu121)
Collecting xformers==0.0.26.post1 (from vllm->bonito==0.0.2)
  Downloading xformers-0.0.26.post1-cp310-cp310-manylinux2014_x86_64.whl (222.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.7/222.7 MB 6.0 MB/s eta 0:00:00
Collecting interegular>=0.3.2 (from lm-format-enforcer==0.9.8->vllm->bonito==0.0.2)
  Downloading interegular-0.3.3-py37-none-any.whl (23 kB)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm->bonito==0.0.2) (3.1.4)
Collecting lark (from outlines==0.0.34->vllm->bonito==0.0.2)
  Downloading lark-1.1.9-py3-none-any.whl (111 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.7/111.7 kB 18.4 MB/s eta 0:00:00
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm->bonito==0.0.2) (1.6.0)
Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm->bonito==0.0.2) (2.2.1)
Collecting diskcache (from outlines==0.0.34->vllm->bonito==0.0.2)
  Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 7.5 MB/s eta 0:00:00
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm->bonito==0.0.2) (1.11.4)
Requirement already satisfied: numba in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm->bonito==0.0.2) (0.58.1)
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm->bonito==0.0.2) (1.4.2)
Requirement already satisfied: referencing in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm->bonito==0.0.2) (0.35.1)
Requirement already satisfied: jsonschema in /usr/local/lib/python3.10/dist-packages (from outlines==0.0.34->vllm->bonito==0.0.2) (4.19.2)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->vllm->bonito==0.0.2) (1.12)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->vllm->bonito==0.0.2) (3.3)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
Collecting nvidia-curand-cu12==10.3.2.106 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
Collecting nvidia-nccl-cu12==2.20.5 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch==2.3.0->vllm->bonito==0.0.2)
  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
Requirement already satisfied: triton==2.3.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.0->vllm->bonito==0.0.2) (2.3.0)
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch==2.3.0->vllm->bonito==0.0.2)
  Downloading nvidia_nvjitlink_cu12-12.5.40-py3-none-manylinux2014_x86_64.whl (21.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.3/21.3 MB 71.3 MB/s eta 0:00:00
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->bonito==0.0.2) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->bonito==0.0.2) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->bonito==0.0.2) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->bonito==0.0.2) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->bonito==0.0.2) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->bonito==0.0.2) (4.0.3)
Collecting starlette<1.0.0,>=0.30.0 (from prometheus-fastapi-instrumentator>=7.0.0->vllm->bonito==0.0.2)
  Downloading starlette-0.37.2-py3-none-any.whl (71 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.9/71.9 kB 10.9 MB/s eta 0:00:00
Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.0->vllm->bonito==0.0.2) (0.7.0)
Requirement already satisfied: pydantic-core==2.18.2 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.0->vllm->bonito==0.0.2) (2.18.2)
Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm->bonito==0.0.2) (8.1.7)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm->bonito==0.0.2) (1.0.8)
Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm->bonito==0.0.2) (3.20.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets->bonito==0.0.2) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets->bonito==0.0.2) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets->bonito==0.0.2) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets->bonito==0.0.2) (2024.2.2)
Collecting fastapi-cli>=0.0.2 (from fastapi->vllm->bonito==0.0.2)
  Downloading fastapi_cli-0.0.4-py3-none-any.whl (9.5 kB)
Collecting httpx>=0.23.0 (from fastapi->vllm->bonito==0.0.2)
  Downloading httpx-0.27.0-py3-none-any.whl (75 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.6/75.6 kB 12.7 MB/s eta 0:00:00
Collecting python-multipart>=0.0.7 (from fastapi->vllm->bonito==0.0.2)
  Downloading python_multipart-0.0.9-py3-none-any.whl (22 kB)
Collecting ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1 (from fastapi->vllm->bonito==0.0.2)
  Downloading ujson-5.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.6/53.6 kB 8.7 MB/s eta 0:00:00
Collecting orjson>=3.2.1 (from fastapi->vllm->bonito==0.0.2)
  Downloading orjson-3.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.5/142.5 kB 21.9 MB/s eta 0:00:00
Collecting email_validator>=2.0.0 (from fastapi->vllm->bonito==0.0.2)
  Downloading email_validator-2.1.1-py3-none-any.whl (30 kB)
Collecting h11>=0.8 (from uvicorn[standard]->vllm->bonito==0.0.2)
  Downloading h11-0.14.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 9.7 MB/s eta 0:00:00
Collecting httptools>=0.5.0 (from uvicorn[standard]->vllm->bonito==0.0.2)
  Downloading httptools-0.6.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (341 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.4/341.4 kB 41.8 MB/s eta 0:00:00
Collecting python-dotenv>=0.13 (from uvicorn[standard]->vllm->bonito==0.0.2)
  Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]->vllm->bonito==0.0.2)
  Downloading uvloop-0.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 81.5 MB/s eta 0:00:00
Collecting watchfiles>=0.13 (from uvicorn[standard]->vllm->bonito==0.0.2)
  Downloading watchfiles-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 86.3 MB/s eta 0:00:00
Collecting websockets>=10.4 (from uvicorn[standard]->vllm->bonito==0.0.2)
  Downloading websockets-12.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (130 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.2/130.2 kB 17.4 MB/s eta 0:00:00
Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai->vllm->bonito==0.0.2) (3.7.1)
Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai->vllm->bonito==0.0.2) (1.7.0)
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai->vllm->bonito==0.0.2) (1.3.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->bonito==0.0.2) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->bonito==0.0.2) (2023.4)
Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->bonito==0.0.2) (2024.1)
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai->vllm->bonito==0.0.2) (1.2.1)
Collecting dnspython>=2.0.0 (from email_validator>=2.0.0->fastapi->vllm->bonito==0.0.2)
  Downloading dnspython-2.6.1-py3-none-any.whl (307 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.7/307.7 kB 42.7 MB/s eta 0:00:00
Collecting typer>=0.12.3 (from fastapi-cli>=0.0.2->fastapi->vllm->bonito==0.0.2)
  Downloading typer-0.12.3-py3-none-any.whl (47 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.2/47.2 kB 7.4 MB/s eta 0:00:00
Collecting httpcore==1.* (from httpx>=0.23.0->fastapi->vllm->bonito==0.0.2)
  Downloading httpcore-1.0.5-py3-none-any.whl (77 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.9/77.9 kB 12.5 MB/s eta 0:00:00
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->outlines==0.0.34->vllm->bonito==0.0.2) (2.1.5)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets->bonito==0.0.2) (1.16.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema->outlines==0.0.34->vllm->bonito==0.0.2) (2023.12.1)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema->outlines==0.0.34->vllm->bonito==0.0.2) (0.18.1)
Requirement already satisfied: llvmlite<0.42,>=0.41.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba->outlines==0.0.34->vllm->bonito==0.0.2) (0.41.1)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.3.0->vllm->bonito==0.0.2) (1.3.0)
Collecting shellingham>=1.3.0 (from typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm->bonito==0.0.2)
  Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm->bonito==0.0.2) (13.7.1)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm->bonito==0.0.2) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm->bonito==0.0.2) (2.16.1)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm->bonito==0.0.2) (0.1.2)
Building wheels for collected packages: bonito, vllm-nccl-cu12
  Building wheel for bonito (setup.py) ... done
  Created wheel for bonito: filename=bonito-0.0.2-py3-none-any.whl size=4585 sha256=ad2c9ec579a44fbef608e6babff389ec446e2e1946495a5197a0cef1900431c6
  Stored in directory: /tmp/pip-ephem-wheel-cache-a1qhngbb/wheels/c5/fe/de/e0c4849775dee927ba7352098bc3e060482a9ac937dde7f9a3
  Building wheel for vllm-nccl-cu12 (setup.py) ... done
  Created wheel for vllm-nccl-cu12: filename=vllm_nccl_cu12-2.18.1.0.4.0-py3-none-any.whl size=5418 sha256=a60cfd55ede786f0554b93235479600d23a44f1dd8b8b5fcc70ad018e23c692c
  Stored in directory: /root/.cache/pip/wheels/d1/28/b5/e99e6ea84b08c0bf19a218d408316e55e02ff725d3616fb79d
Successfully built bonito vllm-nccl-cu12
Installing collected packages: vllm-nccl-cu12, nvidia-ml-py, ninja, xxhash, websockets, uvloop, ujson, shellingham, python-multipart, python-dotenv, orjson, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, lark, interegular, httptools, h11, dnspython, diskcache, dill, watchfiles, uvicorn, tiktoken, starlette, nvidia-cusparse-cu12, nvidia-cudnn-cu12, multiprocess, httpcore, email_validator, typer, prometheus-fastapi-instrumentator, nvidia-cusolver-cu12, lm-format-enforcer, httpx, ray, openai, fastapi-cli, datasets, xformers, outlines, fastapi, vllm, bonito
  Attempting uninstall: typer
    Found existing installation: typer 0.9.4
    Uninstalling typer-0.9.4:
      Successfully uninstalled typer-0.9.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
spacy 3.7.4 requires typer<0.10.0,>=0.3.0, but you have typer 0.12.3 which is incompatible.
weasel 0.3.4 requires typer<0.10.0,>=0.3.0, but you have typer 0.12.3 which is incompatible.
Successfully installed bonito-0.0.2 datasets-2.19.1 dill-0.3.8 diskcache-5.6.3 dnspython-2.6.1 email_validator-2.1.1 fastapi-0.111.0 fastapi-cli-0.0.4 h11-0.14.0 httpcore-1.0.5 httptools-0.6.1 httpx-0.27.0 interegular-0.3.3 lark-1.1.9 lm-format-enforcer-0.9.8 multiprocess-0.70.16 ninja-1.11.1.1 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-ml-py-12.550.52 nvidia-nccl-cu12-2.20.5 nvidia-nvjitlink-cu12-12.5.40 nvidia-nvtx-cu12-12.1.105 openai-1.30.3 orjson-3.10.3 outlines-0.0.34 prometheus-fastapi-instrumentator-7.0.0 python-dotenv-1.0.1 python-multipart-0.0.9 ray-2.23.0 shellingham-1.5.4 starlette-0.37.2 tiktoken-0.6.0 typer-0.12.3 ujson-5.10.0 uvicorn-0.29.0 uvloop-0.19.0 vllm-0.4.2 vllm-nccl-cu12-2.18.1.0.4.0 watchfiles-0.21.0 websockets-12.0 xformers-0.0.26.post1 xxhash-3.4.1

The code keeps running without further problems until the "Generate the synthetic instructions"-section. Here again the output:

The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

---------------------------------------------------------------------------

AssertionError                            Traceback (most recent call last)

[<ipython-input-6-0ad9bfde7270>](https://localhost:8080/#) in <cell line: 7>()
      5 # Generate synthetic instruction tuning dataset
      6 sampling_params = {'max_new_tokens':256, 'top_p':0.95, 'temperature':0.7, 'num_return_sequences':1}
----> 7 synthetic_dataset = bonito.generate_task(
      8     unannotated_paragraph,
      9     task_type="nli",

18 frames

[/usr/local/lib/python3.10/dist-packages/awq/modules/fused/norm.py](https://localhost:8080/#) in forward(self, x)
     17 
     18     def forward(self, x):
---> 19         assert AWQ_INSTALLED, (
     20             "AWQ kernels could not be loaded. "
     21             "Please install them from https://github.com/casper-hansen/AutoAWQ_kernels"

AssertionError: AWQ kernels could not be loaded. Please install them from https://github.com/casper-hansen/AutoAWQ_kernels

It did say it successfully installed the autoawq-kernels-0.0.6+cu122 so I am not sure how I should try to solve this problem.

What are the bare-bone template of the Bonito?

I would like to use this model with Ollama or llama.cpp but I would like to know the bare-bone explanation of bonito's template. Would you mind giving a short explanation?

What's the quality of synthetic data?

@nihalnayak and team: Thanks for a really interesting paper!
How do you think about the quality of synthetic data produced by Bonito?

For example, if I use the quantized model in Colab, are the outputs any worse than in the original? How much worse?

What if I skip the Bonito pipeline and just ask some LLM "Create a question and answer pair from <some unannotated text>" -- will I get output that is just as good as what Bonito produces?

Thanks!
Alon

Does bonito support multiple graphics cards?

Does bonito support multiple graphics cards? For example, if I have two 4090 graphics cards, can I run bonito-v1?

AWQ kernels could not be loaded. Please install them from https://github.com/casper-hansen/AutoAWQ_kernels

In Quantized Bonito Tutorial .The following error occurred in the last step
Fetching 10 files: 100%
 10/10 [00:00<00:00, 415.21it/s]
Replacing layers...: 100%|██████████| 32/32 [00:10<00:00, 3.06it/s]
Fusing layers...: 100%|██████████| 32/32 [00:00<00:00, 92.50it/s]
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's attention_mask to obtain reliable results.
Setting pad_token_id to eos_token_id:2 for open-end generation.

AssertionError Traceback (most recent call last)
in <cell line: 14>()
12 # Generate synthetic instruction tuning dataset
13 sampling_params = {'max_new_tokens':256, 'top_p':0.95, 'temperature':0.5, 'num_return_sequences':1}
---> 14 synthetic_dataset = bonito.generate_tasks(
15 unannotated_text,
16 context_col="input",

18 frames
/usr/local/lib/python3.10/dist-packages/awq/modules/fused/norm.py in forward(self, x)
17
18 def forward(self, x):
---> 19 assert AWQ_INSTALLED, (
20 "AWQ kernels could not be loaded. "
21 "Please install them from https://github.com/casper-hansen/AutoAWQ_kernels"

AssertionError: AWQ kernels could not be loaded. Please install them from https://github.com/casper-hansen/AutoAWQ_kernels

TypeError: cannot pickle '_thread.RLock' object

TypeError: cannot pickle '_thread.RLock' object
iam getting this error

An error occurred when running 'bonito = QuantizedBonito("alexandreteles/bonito-v1-awq")'

I encountered this error while following the Quantized Bonito Tutorial and I would like to know what caused this error

/opt/anaconda3/envs/bonito/bin/python /home/easyai/Downloads/BONITO/bonito/1.py
/opt/anaconda3/envs/bonito/lib/python3.9/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
Replacing layers...: 100%|██████████| 32/32 [00:03<00:00, 8.10it/s]
Traceback (most recent call last):
File "/home/easyai/Downloads/BONITO/bonito/1.py", line 95, in
bonito = QuantizedBonito("alexandreteles/bonito-v1-awq")
File "/home/easyai/Downloads/BONITO/bonito/1.py", line 10, in init
self.model = AutoAWQForCausalLM.from_quantized(
File "/opt/anaconda3/envs/bonito/lib/python3.9/site-packages/awq/models/auto.py", line 101, in from_quantized
return AWQ_CAUSAL_LM_MODEL_MAP[model_type].from_quantized(
File "/opt/anaconda3/envs/bonito/lib/python3.9/site-packages/awq/models/base.py", line 454, in from_quantized
load_checkpoint_and_dispatch(
File "/opt/anaconda3/envs/bonito/lib/python3.9/site-packages/accelerate/big_modeling.py", line 607, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "/opt/anaconda3/envs/bonito/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 1699, in load_checkpoint_in_model
raise ValueError(
ValueError: /home/easyai/.cache/huggingface/hub/models--alexandreteles--bonito-v1-awq/snapshots/d0b7ae8fd6606d9f78038304dc75edb14d6a3757 is not a folder containing a .index.json file or a pytorch_model.bin or a model.safetensors file

Process finished with exit code 1

ImportError: cannot import name 'SamplingParams' from 'bonito' (/content/bonito/bonito/__init__.py)

from bonito import Bonito, SamplingParams
from datasets import load_dataset

# @title AutoBonito🐟

# @markdown ### Paramaters
bonito_model = "NousResearch/Genstruct-7B"  # @param {type:"string"}
dataset = "kaifahmad/indian-history-hindi-QA-3.4k" # @param {type:"string"}
unannotated_text = "unannotated_alhzheimer_corpus.txt" # @param {type:"string"}
split = "train" # @param {type:"string"}
number_of_samples = "100" # @param {type:"string"}
max_tokens = 256 # @param {type:"string"}

n = 1 # @param {type:"string"}
top_p = 0.95 # @param {type:"string"}
temperature = 0.5 # @param {type:"string"}

context_column = "Question"  # @param {type:"string"}
task_type = "qa"  # @param {type:"string"}

# Initialize the Bonito model
bonito = Bonito(bonito_model)

# load dataset with unannotated text
unannotated_text = load_dataset(
    dataset,
    unannotated_text
)[split].select(range(number_of_samples))

# Generate synthetic instruction tuning dataset
sampling_params = SamplingParams(max_tokens=256, top_p=0.95, temperature=0.5, n=1)
synthetic_dataset = bonito.generate_tasks(
    unannotated_text,
    context_col=context_column,
    task_type=task_type,
    sampling_params=sampling_params
)

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
[<ipython-input-2-c5929ed415ec>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from bonito import Bonito, SamplingParams
      2 from datasets import load_dataset
      3 
      4 # @title AutoBonito🐟
      5 

ImportError: cannot import name 'SamplingParams' from 'bonito' (/content/bonito/bonito/__init__.py)

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

Maximum sequence length error

Hey team, really excited to get started with your repo but I am running into the following error:

ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (16816). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
File <command-609584913430219>, line 5

Code I ran:

from bonito import Bonito, SamplingParams
from datasets import load_dataset

# Initialize the Bonito model
bonito = Bonito("BatsResearch/bonito-v1")

Hardware: I am running on a g5.8xlarge instance on AWS which corresponds to a single A10G with 128 GiB of memory, and 600 GB of local NVMe storage.

Any insights appreciated!

How can I set the gpu_memory_utilization?

I encountered this error when running the code:

ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine.

But I have already made these settings

bonito = Bonito("BatsResearch/bonito-v1",gpu_memory_utilization=0.9)

, it seems like they are not working, what should I do?

ModuleNotFoundError: No module named 'bonito'

Hello everyone,

I am trying to use Bonito on Colab (T4 GPU) and I am following these commands:

!pip install -e git+https://github.com/BatsResearch/bonito#egg=bonito

from bonito import Bonito, SamplingParams
from datasets import load_dataset

bonito = Bonito("BatsResearch/bonito-v1")

but it returns this error:

---

ModuleNotFoundError Traceback (most recent call last)
in <cell line: 1>()
----> 1 from bonito import Bonito, SamplingParams
2 from datasets import load_dataset
3
4 # Initialize the Bonito model
5 bonito = Bonito("BatsResearch/bonito-v1")

ModuleNotFoundError: No module named 'bonito'

How can I solve it? Can you help me?

Thank you so much in advance!

Does it support Chinese text

Can this run on the CPU as vLLM is being used

I want to create the dataset in the CPU and realized the code is using the vLLM, how can make it run in the CPU and generate the dataset?

bonito = Bonito("BatsResearch/bonito-v1")

File "/home/spr/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 109, in init
self.llm_engine = LLMEngine.from_engine_args(engine_args)
File "/home/spr/.local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 386, in from_engine_args
engine_configs = engine_args.create_engine_configs()
File "/home/spr/.local/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 286, in create_engine_configs
device_config = DeviceConfig(self.device)
File "/home/spr/.local/lib/python3.10/site-packages/vllm/config.py", line 496, in init
raise RuntimeError("No supported device detected.")
RuntimeError: No supported device detected.

Is there any smaller model available?

I have been working with limited resources, I was wondering if you have a smaller model available. Something which can be used in Google Colab (15GB T4 GPU)?

I checked your huggingface page, but it looked like you have only one available over there and that is Bonito-V1.

==