paperswithcode/galai

Model API for GALACTICA

galai's Issues

Running the huge model on CPU

From my understanding, using the transformers accelerate tool, running the HUGE model means it needs to load the entire thing into RAM. Is there any way for it to process as it loads into ram, or is it a necessity? I have 614GB of ram, I am also curious if there's a way to edit the program while the model is stored in memory. Is there any way to change how it processes on the CPU? I know that the GPU can choose between FP32,16, and INT 8 but I don't know how to find info on running on CPU beyond the huggingface.co example.

RuntimeError: CUDA error: invalid device ordinal

When I load model I have this error.

Traceback (most recent call last):
File "", line 1, in
File "test/env/lib/python3.9/site-packages/galai/init.py", line 39, in load_model
model._load_checkpoint(checkpoint_path=get_checkpoint_path(name))
File "test/env/lib/python3.9/site-packages/galai/model.py", line 63, in _load_checkpoint
load_checkpoint_and_dispatch(
File "test/env/lib/python3.9/site-packages/accelerate/big_modeling.py", line 366, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "test/env/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 701, in load_checkpoint_in_model
set_module_tensor_to_device(model, param_name, param_device, value=param)
File "test/env/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 124, in set_module_tensor_to_device
new_value = value.to(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Could not find a version that satisfies the requirement promptsource

Specifications:
Linux / pop!_os
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Python 3.10.6

Issue encountered when installing from repo
pip install git+https://github.com/paperswithcode/galai

ERROR: Could not find a version that satisfies the requirement promptsource (from galai) (from versions: none)
ERROR: No matching distribution found for promptsource

Might be associated with the following issue: bigscience-workshop/promptsource#728
Issue resolved when using Python 3.7

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]

Hi all,
I've tried to run galai on my mac (13.0.1 (22A400)) with Python 3.7 but unsuccessfully.
When running the below program:

import galai as gal

if __name__ == '__main__':
    model = gal.load_model("standard")
    model.generate("Some example text")

I get the following error:

tokenizer.json: 0.00B [00:00, ?B/s]Incomplete files for tokenizer; downloading
tokenizer.json: 0.00B [00:00, ?B/s]
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 850, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1108, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/maorgoaz/projects/galactica/main.py", line 4, in <module>
    model = gal.load_model("standard")
  File "/Users/maorgoaz/projects/galactica/venv/lib/python3.7/site-packages/galai/__init__.py", line 37, in load_model
    model._set_tokenizer(tokenizer_path=get_tokenizer_path())
  File "/Users/maorgoaz/projects/galactica/venv/lib/python3.7/site-packages/galai/utils.py", line 144, in get_tokenizer_path
    download_tokenizer(file_name)
  File "/Users/maorgoaz/projects/galactica/venv/lib/python3.7/site-packages/galai/utils.py", line 84, in download_tokenizer
    _download_file(TOKENIZER_URL, tokenizer_path)
  File "/Users/maorgoaz/projects/galactica/venv/lib/python3.7/site-packages/galai/utils.py", line 73, in _download_file
    urllib.request.urlretrieve(file_url, filename=file_loc, reporthook=t.update_to)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 247, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)>

Process finished with exit code 1

any ideas what's wrong?

Adding MLM head

Will future models be released with MLM heads?

Multi-threaded mode?

import torch, gc
from transformers import AutoTokenizer, OPTForCausalLM

tokenizer = AutoTokenizer.from_pretrained("galactica-30b")
tokenizer.pad_token_id = 1
tokenizer.padding_side = 'left'
tokenizer.model_max_length = 2020

model = OPTForCausalLM.from_pretrained("galactica-30b")

input_text = """# Scientific article.
title: Purpose of Humanity's continued existence alive.

# Introduction
"""
input_ids = tokenizer(input_text, return_tensors="pt", padding='max_length').input_ids

outputs = model.generate(input_ids,
                         max_new_tokens=1000,
                         do_sample=True,
                         temperature=0.7,
                         top_k=25,
                         top_p=0.9,
                         no_repeat_ngram_size=10,
                         early_stopping=True)

print(tokenizer.decode(outputs[0]).lstrip('<pad>'))

gc.collect()
torch.empty_cache()

When I run this, I can see it loads the model into ram; it seems only to be using one thread. The output is: a wall of various 'decoder.layers.xx.bias' and "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference."

Why is CUDA is required for pytorch?

Please remove the CUDA requirement or export the model in a device agnostic format like ONNX. Some people just want to try out your models but their systems may not have a compatible Nvidia GPU.

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

Running on Ubuntu 22.04 using conda with python 3.7 and a Titan X GPU with CUDA 11

conda create --name galactica python=3.7
conda activate galactica
pip install git+https://github.com/paperswithcode/galai
python3.7
>>> import galai as gal
>>> model=gal.load_model("mini", num_gpus=1)
>>> model.generate("Scaled dot product attention:\n\n\\[")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/model.py", line 140, in generate
    output_hidden_states=True
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/transformers/generation_utils.py", line 1499, in generate
    **model_kwargs,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/transformers/generation_utils.py", line 2237, in greedy_search
    output_hidden_states=output_hidden_states,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/architecture.py", line 974, in forward
    return_dict=return_dict,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/architecture.py", line 732, in forward
    use_cache=use_cache,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/architecture.py", line 333, in forward
    output_attentions=output_attentions,
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/galai/architecture.py", line 178, in forward
    query_states = self.q_proj(hidden_states) * self.scaling
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/anaconda3/envs/galactica/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

I encountered this error while installing requirements from requirements.txt file. Could you please tell me what is causing this issue?

I am on macOS. and I did try to update pip, but it didn't workout.

error: subprocess-exited-with-error

  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [51 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-11.0-arm64-cpython-38
      creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers
      copying py_src/tokenizers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers
      creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/models
      copying py_src/tokenizers/models/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/models
      creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/decoders
      copying py_src/tokenizers/decoders/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/decoders
      creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/normalizers
      copying py_src/tokenizers/normalizers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/normalizers
      creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/pre_tokenizers
      copying py_src/tokenizers/pre_tokenizers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/pre_tokenizers
      creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/processors
      copying py_src/tokenizers/processors/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/processors
      creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/trainers
      copying py_src/tokenizers/trainers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/trainers
      creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
      copying py_src/tokenizers/implementations/byte_level_bpe.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
      copying py_src/tokenizers/implementations/sentencepiece_unigram.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
      copying py_src/tokenizers/implementations/sentencepiece_bpe.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
      copying py_src/tokenizers/implementations/base_tokenizer.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
      copying py_src/tokenizers/implementations/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
      copying py_src/tokenizers/implementations/char_level_bpe.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
      copying py_src/tokenizers/implementations/bert_wordpiece.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
      creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/tools
      copying py_src/tokenizers/tools/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/tools
      copying py_src/tokenizers/tools/visualizer.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/tools
      copying py_src/tokenizers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers
      copying py_src/tokenizers/models/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/models
      copying py_src/tokenizers/decoders/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/decoders
      copying py_src/tokenizers/normalizers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/normalizers
      copying py_src/tokenizers/pre_tokenizers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/pre_tokenizers
      copying py_src/tokenizers/processors/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/processors
      copying py_src/tokenizers/trainers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/trainers
      copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/tools
      running build_ext
      running build_rust
      error: can't find Rust compiler

      If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

      To update pip, run:

          pip install --upgrade pip

      and then retry package installation.

      If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Please list python package dependencies on README

The galai package seems to have some dependencies (PyTorch? Tensorflow?). Please list those in your install instructions on the README.

Where is the model saved when downloaded?

How to set the correct view of bibliographic references?

How to display correct (full) references to literary sources in the text? Now it looks something like this: "...This has been mainly driven by the increase in the use of animal models for preclinical research [[START_REF] Animal models in translational medicine: Validation and prediction, Denayer[END_REF]]. ..."

Figure 3 explanation

Sorry, a few things are unclear to me about Fig 3:

1. What is generated by the model? Everything starting from <work>? Including Answer?
2. Is it zero-shot like in the image?
3. Is it a mistake from the model that it generates « & >?
4. Does the model literally predict the output of the program (0.0051) or did you run the program externally and then add it to the image?

Thanks 🤗

Which size model does the Demo Page use?

For example, is it the largest model?

I'm recording tests and so that was the reason why I'm wanting to know. Really I'll need the largest model.

torch.cuda.OutOfMemoryError: CUDA out of memory.

Hi,

I'm relatively new to all this. I used conda to install. Then I used this:

import galai as gal

model = gal.load_model(name = 'standard', num_gpus = 1)
model.generate("The Schwarzschild radius is defined as: \\[")

but I'm getting a:

torch.cuda.OutOfMemoryError: CUDA out of memory.

I had assumed 6B parameters would fit into 24GB VRAM.

I've also tried:

export 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128'

My questions are:

Is there any way to run the standard model on 24GB?

Is there any way to know ahead of time which models will fit into which GPUs without simply trying them out? I don't see any memory requirements listed for this (or most other projects, as well).

Which model will fit into 24GB?

Thanks!
Andrew

p.s.

(galai) andrewdo@ai2:/var/lib/myfrdcsa/sandbox/galai-20221116/galai-20221116$ python frdcsa.py
Traceback (most recent call last):
File "frdcsa.py", line 3, in
model = gal.load_model(name = 'standard', num_gpus = 1)
File "/media/andrewdo/s3/sandbox-new/galai-20221116/galai-20221116/galai/init.py", line 41, in load_model
model._load_checkpoint(checkpoint_path=get_checkpoint_path(name))
File "/media/andrewdo/s3/sandbox-new/galai-20221116/galai-20221116/galai/model.py", line 63, in _load_checkpoint
load_checkpoint_and_dispatch(
File "/home/andrewdo/miniconda3/envs/galai/lib/python3.8/site-packages/accelerate/big_modeling.py", line 366, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "/home/andrewdo/miniconda3/envs/galai/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 701, in load_checkpoint_in_model
set_module_tensor_to_device(model, param_name, param_device, value=param)
File "/home/andrewdo/miniconda3/envs/galai/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 124, in set_module_tensor_to_device
new_value = value.to(device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 23.70 GiB total capacity; 21.54 GiB already allocated; 281.31 MiB free; 21.55 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Why only part of the answer to the question is output？

model.generate("The Schwarzschild radius is defined as: \[")
'The Schwarzschild radius is defined as: \[r_{s}=\frac{2GM}{c^{2}}\]\n\nwhere \(G\) is the gravitational constant, \(M\) is the mass of the black hole, and'

Can't install python package

Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/private/var/folders/x8/l933jg8d5477w13l2z3s2nkc0000gn/T/pip-install-nbosimmo/galai_56ef70c550314d5a8e3a62c7d3b02310/setup.py", line 16, in
with open("requirements.txt", "r") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'

I've tried to create a requirements.txt file with pip freeze but nothing changed

AssertionError: Torch not compiled with CUDA enabled

if there isn't anything special, the normal quickstart install doesn't work.

Great project! Any plan on releasing the training data?

Galactica is amazing! I wonder if you guys have plans for releasing the pre-training dataset since it takes a huge effort to gather and organize all the data and it would really help the research community if such data were made public. Thanks!

Galai not cleaning the downloaded zip files

I just installed galai and started downloading the standard version on notebook.

Since each update of the progress bar creates a new line of log instead of refreshing the progress bar I stopped the download to start again on terminal.

The problem is when I try to download the model again it begins to download the second file... And if you interrupt again and try again to download the model it will crash with this error message: PytorchStreamReader failed reading zip archive: failed finding central directory

I am pretty sure the zip file it downloads are not cleared so it tries to unpack incomplete files and crashes because of that

Is there any way to delete the model files to download again?

Galai library doesn't seem to work

Hi,

I'm trying to use the galai quickstart prompt. This is what I'm running:

import galai as gal

model = gal.load_model("standard")
model.generate("Scaled dot product attention:\n\n\\[")
# Scaled dot product attention:\n\n\\[ \\displaystyle\\text{Attention}(Q,K,V)=\\text{softmax}(\\frac{QK^{T}}{\\sqrt{d_{k}}}%\n)V \\]

However, I'm getting TypeError: 'type' object is not subscriptable

Can't install galai via pip

Hi!

I tried installing galai in my conda environment (Python 3.8) via pip install galai but am seeing the following error:

Collecting galai
  Using cached galai-1.0.0.tar.gz (22 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-jk9o1509/galai_8a2029a3d367447e9978185cff95291d/setup.py", line 16, in <module>
          with open("requirements.txt", "r") as f:
      FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Thanks!

How to continue text generation?

After the first text generation, how would one continue to generate more?

Ablation: Pretrain first on OPT data _then_ on scientific texts?

First of all, great work!

Did you try pretraining Galactica using the original OPT checkpoint as a starting point? Since both models have similar architectures and Galactica's dataset is "only" 110B tokens, I imagine that starting from a model that was pretrained on more data would bring some gains.

Keyerror when load galactica-30b

Great work! When I load galactica-30b, it shows KeyError: 'decoder.layers.34.final_layer_norm.bias'

I have downloaded the model into my disk. The key error occurred when running the following code:

model_path = "local-model-path"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = OPTForCausalLM.from_pretrained(model_path, device_map="auto",  torch_dtype=torch.float16)

Thanks for any help!

ModuleNotFoundError: No module named 'transformers.models.opt'

On windows 10, python 3.9, I pip install galai without error but I got this error when importing galai:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_640\446023240.py in <module>
----> 1 import galai

~\anaconda3\lib\site-packages\galai\__init__.py in <module>
----> 1 from galai.model import Model
      2 from galai.utils import get_checkpoint_path, get_tokenizer_path
      3 
      4 
      5 def load_model(name: str, dtype: str=None, num_gpus: int=None):

~\anaconda3\lib\site-packages\galai\model.py in <module>
      6 from tokenizers import Tokenizer
      7 
----> 8 from galai.architecture import OPTForCausalLM, OPTConfig
      9 from galai.utils import escape_custom_split_sequence
     10 

~\anaconda3\lib\site-packages\galai\architecture.py in <module>
     11 
     12 from galai.config import OPTConfig
---> 13 from transformers.models.opt.modeling_opt import (
     14     ACT2FN,
     15     BaseModelOutputWithPast,

ModuleNotFoundError: No module named 'transformers.models.opt'

Fine-tuning

Any recommendations / advice on how we would go about fine-tuning this model?

Convert Model.Generate output to readable file (txt, pdf, doc, etc) ?

Hi I am not using a text editor, therefore I cannot visualize the output of the following code:

import galai as gal

#import gc
import torch
torch.cuda.empty_cache()
#gc.collect()

model = gal.load_model("mini", num_gpus = 1)
model.generate("Question: What is the notch signaling pathway?\n\nAnswer:") 

print(model.generate)

I would like to do something such as

model.generate("Question: What is the notch signaling pathway?\n\nAnswer:") > Output.txt

How can I do that?

30b Checkpoint pickle is published with half precision, no bias tensors and no final layers

The 30b model pickles seem to have no biases.

from tqdm import tqdm
import torch
from pathlib import Path
import pickle

blob_path = Path.home() / Path('.cache/huggingface/hub/models--facebook--galactica-30b/blobs')

keys2blob = {}
errors = {}
blobs = [blob for blob in blob_path.glob('./*') if blob.is_file()]

for blob in tqdm(blobs):
    try:
        keys2blob.update({k: blob for k in torch.load(blob).keys()})
    except pickle.UnpicklingError as e:
        errors[blob] = e

print(f"Num_weights: {len([i for i in keys2blob.keys() if 'weight' in i])}")
print(f"Num_biases: {len([i for i in keys2blob.keys() if 'bias' in i])}")

100%|██████████| 12/12 [00:50<00:00,  4.19s/it]
Num_weights: 290
Num_biases: 0

This is opposed to the 6.7b model which contains a lot of biases.

from tqdm import tqdm
import torch
from pathlib import Path
import pickle

blob_path = Path.home() / Path('.cache/huggingface/hub/models--facebook--galactica-6.7b/blobs')

keys2blob = {}
errors = {}
blobs = [blob for blob in blob_path.glob('./*') if blob.is_file()]

for blob in tqdm(blobs):
    try:
        keys2blob.update({k: blob for k in torch.load(blob).keys()})
    except pickle.UnpicklingError as e:
        errors[blob] = e

print(f"Num_weights: {len([i for i in keys2blob.keys() if 'weight' in i])}")
print(f"Num_biases: {len([i for i in keys2blob.keys() if 'bias' in i])}")

50%|█████     | 4/8 [00:14<00:14,  3.57s/it]
Num_weights: 260
Num_biases: 257

I do not believe I am missing any pickles because the disk usage of the cloned repository tallies with what is displayed by the huggingface site (note that du outputs gibibytes which is likely the cause of the slight discrepancy in raw numbers).

❯ du -csh ./models--facebook--galactica-30b/blobs/*
785M    ./models--facebook--galactica-30b/blobs/0379c39b5a0cb59453b14738ef1d4924e93599aba4e57f2599036e76f36532f6
9.2G    ./models--facebook--galactica-30b/blobs/05db345d4fcca580bed2c6e9d0fe8feead207c2c2fa8384c27c94cbd4ed0e0bf
4.0K    ./models--facebook--galactica-30b/blobs/0967ef424bce6791893e9a57bb952f80fd536e93
9.2G    ./models--facebook--galactica-30b/blobs/0d6ce164b560f4601d48f61c2a8d598106faa9f4b89c39334a712429649b75c8
4.0K    ./models--facebook--galactica-30b/blobs/28e11da7e191492f3f23d2aa35e9b60f8e9becf6
9.2G    ./models--facebook--galactica-30b/blobs/30a274571d49a30bb4d6872e69b96ad191fa22c92427d160c74ce225a566bd71
24K     ./models--facebook--galactica-30b/blobs/98d10d1a52ab2b70f1deff472512cbaa6065e317
9.2G    ./models--facebook--galactica-30b/blobs/aa79446f17da0f3b9f8815a3628c2b1935936ec819f09a5865ce4e3c4ee51aa7
9.2G    ./models--facebook--galactica-30b/blobs/b919005245e2b77d57bf3a73ac18415083aa32b6e2e4e89c96b8d988453a0e7f
4.0K    ./models--facebook--galactica-30b/blobs/bc97f8a9458a1fe096bec5d8ec938a02647bc4bb
9.2G    ./models--facebook--galactica-30b/blobs/c1cad10954e544c44aabd29f31e67292d1bc819d2e7b9842f14fdcef88d58f93
2.1M    ./models--facebook--galactica-30b/blobs/e18054f92dc016b43c940dd1c4a1c5da884539c0
56G     total

The output is so bad - total garbage what I am doing wrong? It is also super slow and requires huge amount of RAM

Here my entire command

from transformers import AutoTokenizer, OPTForCausalLM

tokenizer = AutoTokenizer.from_pretrained("facebook/galactica-1.3b")
model = OPTForCausalLM.from_pretrained("facebook/galactica-1.3b", device_map="auto")

 
input_text = "The benefits of deadlifting\n\n"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids,new_doc=False,top_p=0.7, max_length=1000)
print(tokenizer.decode(outputs[0]))

And the output is total repetition and garbage. I am trying to generate an article based on the topic sentence I provide

Also even 28 GB VRAM is not enough for 6.7b model. I am testing CPU runtime on IPU and it has been more than 2 hours with just 6.7b model.

the output as below

The benefits of deadlifting

The benefits of deadlifting are numerous. It is a simple, inexpensive, and effective method of reducing the risk of injury to the shoulder and elbow. It is also a simple and effective method of reducing the risk of injury to the hand.

Shoulder

The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper extremity. The shoulder is the most common site of injury to the upper

Using `huge` model with 80GB A100

Hi!

Thanks for releasing the models. The paper says that For training the largest 120B model, we use 128 NVIDIA A100 80GB nodes. For inference Galactica 120B requires a single A100 node. Does this mean 8 x 80 GB A100s? Or does this mean 11 80 GB A100 GPU?

If the latter, I haven't been able to load the model for inference, even with float16. :

In [1]: import galai as gal

In [2]: model = gal.load_model("huge", num_gpus=1, dtype='float16')
---------------------------------------------------------------------------
OutOfMemoryError                          Traceback (most recent call last)
Cell In [2], line 1
----> 1 model = gal.load_model("huge", num_gpus=1, dtype='float16')

File /juice/scr/nfliu/miniconda3/envs/galai/lib/python3.8/site-packages/galai/__init__.py:41, in load_model(name, dtype, num_gpus)
     39     model._load_checkpoint(checkpoint_path=get_checkpoint_path(name))
     40 else:
---> 41     model._load_checkpoint(checkpoint_path=get_checkpoint_path(name))
     43 return model

File /juice/scr/nfliu/miniconda3/envs/galai/lib/python3.8/site-packages/galai/model.py:63, in Model._load_checkpoint(self, checkpoint_path)
     60 if 'mini' in checkpoint_path or 'base' in checkpoint_path:
     61     checkpoint_path = checkpoint_path + '/pytorch_model.bin'
---> 63 load_checkpoint_and_dispatch(
     64     self.model.model,
     65     checkpoint_path,
     66     device_map=device_map,
     67     offload_folder=None,
     68     dtype=self.dtype,
     69     offload_state_dict=True
     70 )
     72 self.model.tie_weights()
     73 self.model.eval()

File /juice/scr/nfliu/miniconda3/envs/galai/lib/python3.8/site-packages/accelerate/big_modeling.py:366, in load_checkpoint_and_dispatch(model, checkpoint, device_map, max_memory, no_split_module_classes, offload_folder, offload_buffers, dtype, offload_state_dict, preload_module_classes)
    364 if offload_state_dict is None and "disk" in device_map.values():
    365     offload_state_dict = True
--> 366 load_checkpoint_in_model(
    367     model,
    368     checkpoint,
    369     device_map=device_map,
    370     offload_folder=offload_folder,
    371     dtype=dtype,
    372     offload_state_dict=offload_state_dict,
    373 )
    374 if device_map is None:
    375     return model

File /juice/scr/nfliu/miniconda3/envs/galai/lib/python3.8/site-packages/accelerate/utils/modeling.py:701, in load_checkpoint_in_model(model, checkpoint, device_map, offload_folder, dtype, offload_state_dict)
    699             offload_weight(param, param_name, state_dict_folder, index=state_dict_index)
    700         else:
--> 701             set_module_tensor_to_device(model, param_name, param_device, value=param)
    703 # Force Python to clean up.
    704 del checkpoint

File /juice/scr/nfliu/miniconda3/envs/galai/lib/python3.8/site-packages/accelerate/utils/modeling.py:124, in set_module_tensor_to_device(module, tensor_name, device, value)
    122     new_value = old_value.to(device)
    123 elif isinstance(value, torch.Tensor):
--> 124     new_value = value.to(device)
    125 else:
    126     new_value = torch.tensor(value, device=device)

OutOfMemoryError: CUDA out of memory. Tried to allocate 800.00 MiB (GPU 0; 79.20 GiB total capacity; 77.56 GiB already allocated; 735.50 MiB free; 77.56 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Could you confirm that 1xA100 should be fine? If so, what versions of torch / cuda did you use?

Thanks!

Launch requirements

Who can share what equipment specifications are needed for each of the model sizes?

Failed to load PyTorch C extensions:

raceback (most recent call last):
File "C:\Users\user\PycharmProjects\pythonProject19\HOF.py", line 1, in
import galai as gal
File "C:\Users\user\PycharmProjects\pythonProject19\venv\lib\site-packages\galai_init_.py", line 1, in
from galai.model import Model
File "C:\Users\user\PycharmProjects\pythonProject19\venv\lib\site-packages\galai\model.py", line 2, in
import torch
File "C:\Users\user\PycharmProjects\pythonProject19\venv\lib\site-packages\torch_init_.py", line 209, in
raise ImportError(textwrap.dedent('''
ImportError: Failed to load PyTorch C extensions:
It appears that PyTorch has loaded the torch/_C folder
of the PyTorch repository rather than the C extensions which
are expected in the torch._C namespace. This can occur when
using the install workflow. e.g.
$ python setup.py install && python -c "import torch"

This error can generally be solved using the `develop` workflow
    $ python setup.py develop && python -c "import torch"  # This should succeed
or by running Python from a different directory.

I get this problem eheh, what should I do? Sorry for my incompetence 

how to run the model on mps device?

how to run the model on mps device?

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

Hi, installed and imported galai successfully using Ubuntu 21.01. I installed galai as :

conda create -n galia python=3.9
conda activate galia
pip install git+https://github.com/paperswithcode/galai

and the code I tested is:

import galai as gal
model = gal.load_model(name = 'mini', num_gpus = 1)
model.generate("Lecture 1: The Ising Model\n\n", new_doc=True, top_p=0.7, max_length=200)

however, the mini obtains the following error:

Traceback (most recent call last):
File "mini.py", line 3, in
model = gal.load_model("standard")
File "/home/sebasmos/Desktop/AnpassenNN//galia/galai/galai/init.py", line 41, in load_model
model._load_checkpoint(checkpoint_path=get_checkpoint_path(name))
File "/home/sebasmos/Desktop/AnpassenNN//galia/galai/galai/model.py", line 69, in _load_checkpoint
offload_state_dict=True
File "/home/sebasmos/anaconda3/envs/yolo/lib/python3.7/site-packages/accelerate/big_modeling.py", line 372, in load_checkpoint_and_dispatch
offload_state_dict=offload_state_dict,
File "/home/sebasmos/anaconda3/envs/yolo/lib/python3.7/site-packages/accelerate/utils/modeling.py", line 679, in load_checkpoint_in_model
checkpoint = torch.load(checkpoint_file)
File "/home/sebasmos/anaconda3/envs/yolo/lib/python3.7/site-packages/torch/serialization.py", line 705, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "/home/sebasmos/anaconda3/envs/yolo/lib/python3.7/site-packages/torch/serialization.py", line 243, in init
super(_open_zipfile_reader, self).init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

What if I would run the "base" or "mini" versions on CPU?

Hi,

First of all thanks for the amazing work! I was wondering if it would be possible to run the smaller versions of the model on CPU, at the moment it seems not possible.

Preprocessing script for moleculenet / uniprot?

Hello,

I'd like to investigate / replicate the Galactica results on scientific domains. Is it possible to release the script used to preprocess the moleculenet/uniprot data? I'm unable to get Galactica to meaningfully answer queries about this data, likely due to my incorrect formatting of the datasets.

Thank you!

ValueError: model.decoder.embed_tokens.weight doesn't have any device set.

i have downloaded the huge model and i get the above error when i run the example below
import galai as gal
model = gal.load_model("huge", 'float16', 1)
model.generate("Scaled dot product attention:\n\n\[", device_map="auto")

CUDA error: CUBLAS_STATUS_INVALID_VALUE when trying to get query project

Hi!
First of all, thank you very much for this awesome project.
I'm facing some issues when trying to execute this code:

import galai as gal

model = gal.load_model(name="base", num_gpus=1)
model.generate("Scaled dot product attention:\n\n\\[")

This is the output error:

Traceback (most recent call last):
  File "/home/fmlopez/launch_galactica.py", line 4, in <module>
    model.generate("Scaled dot product attention:\n\n\\[")
  File "/home/fmlopez/venv/lib/python3.9/site-packages/galai/model.py", line 136, in generate
    out = self.model.generate(
  File "/home/fmlopez/venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/fmlopez/venv/lib/python3.9/site-packages/transformers/generation_utils.py", line 1490, in generate
    return self.greedy_search(
  File "/home/fmlopez/venv/lib/python3.9/site-packages/transformers/generation_utils.py", line 2233, in greedy_search
    outputs = self(
  File "/home/fmlopez/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/fmlopez/venv/lib/python3.9/site-packages/galai/architecture.py", line 965, in forward
    outputs = self.model.decoder(
  File "/home/fmlopez/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/fmlopez/venv/lib/python3.9/site-packages/galai/architecture.py", line 726, in forward
    layer_outputs = decoder_layer(
  File "/home/fmlopez/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/fmlopez/venv/lib/python3.9/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/fmlopez/venv/lib/python3.9/site-packages/galai/architecture.py", line 328, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/fmlopez/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/fmlopez/venv/lib/python3.9/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/fmlopez/venv/lib/python3.9/site-packages/galai/architecture.py", line 178, in forward
    query_states = self.q_proj(hidden_states) * self.scaling
  File "/home/fmlopez/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/fmlopez/venv/lib/python3.9/site-packages/accelerate/hooks.py", line 156, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/fmlopez/venv/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 116, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

It seems to have problems when trying to get query project (line 178 @ architecture.py)
Does anyone have the same problem?

Thanks for the help!

Clarification on lower precisions and questions about speed

Do the 16 and 8 bit precisions available on hugging face load the full precision model in with cut weights, or is the training done in low precision modes?

If the models trained with full precision are indeed loaded in with half or quarter precision, what kind of performance hit does the validation performance take? Is there a comparison of this alongside the graph in the paper that shows losses for different model sizes?

And finally, if we want to run the 120b or 30b version at a reasonable speed, is there any suggested method other than spinning up a p4dn instance on AWS?

Thank you.

hidden state of the eos token?

Do galactica model output the hidden state of the EOS ? Would it be possible to get it somehow using Huggingface's codebase or the original implementation? In a similar manner to OPT when doing sequence classification

Getting ZeroDivisionError when using `galai` module

I keep getting the ZeroDivisionError with the galai module

import galai
model = galai.load_model("mini", num_gpus = 1)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "███/myconda/condaGA/lib/python3.7/site-packages/galai/__init__.py", line 39, in load_model
    model._load_checkpoint(checkpoint_path=get_checkpoint_path(name))
  File "███/myconda/condaGA/lib/python3.7/site-packages/galai/model.py", line 69, in _load_checkpoint
    offload_state_dict=True
  File "███/myconda/condaGA/lib/python3.7/site-packages/accelerate/big_modeling.py", line 358, in load_checkpoint_and_dispatch
    low_zero=(device_map == "balanced_low_0"),
  File "███/myconda/condaGA/lib/python3.7/site-packages/accelerate/utils/modeling.py", line 370, in get_balanced_memory
    per_gpu = module_sizes[""] // (num_devices - 1 if low_zero else num_devices)
ZeroDivisionError: integer division or modulo by zero

But if I use the transformers module it works perfectly on GPU

The model is not open source

On the website it says:

We believe models want to be free and so we open source the model for those who want to extend it.

And in releases it says:

We believe models should be open

Just as open source code has accelerated technological progress, so will open source machine learning research.

We built Papers with Code four years ago on the principle that things are best done in the open.

We stick to this pledge by open sourcing our model, and we will look to work to improve accessibility in the coming months.

But your model is released under CC-BY-NC which is not free (according to FSF's definition) nor open source (according to OSI's definition).

Please don't call things which are not open source as "open source" (according to the most widely accepted definition of "open source").

Model Failing when using HuggingFace pipeline

Hi,

So I've been trying to use very basic text generation inference with this model using HuggingFace's pipeline API. However, it keeps on crashing when trying to generate sequences with max_tokens = 10000

from transformers import pipeline
generator = pipeline('text-generation', model = 'facebook/galactica-125m', device=0)
generator('covid-19', renormalize_logits=True, do_sample=True, max_new_tokens=10000)[0]['generated_text']

I updated my Transformers and Torch libraries. CUDA version = 11.7 | torch = 1.14.0 (nightly) [Stable also was not working] | transformers = 4.25.1

GPUs = NVIDIA A100

Error:
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul( ltHandle, computeDesc.descriptor(), &alpha_val, mat1_ptr, Adesc.descriptor(), mat2_ptr, Bdesc.descriptor(), &beta_val, result_ptr, Cdesc.descriptor(), result_ptr, Cdesc.descriptor(), &heuristicResult.algo, workspace.data_ptr(), workspaceSize, at::cuda::getCurrentCUDAStream())

Pip install failing with setup.py not able to find requirements.txt

I followed very simple install steps in my miniconda environment in my ubuntu 22.04

conda create --name galai
# Python version: 3.10.6
pip install galai

I am getting the following error

Defaulting to user installation because normal site-packages is not writeable
Collecting galai
  Using cached galai-1.0.1.tar.gz (22 kB)
  Preparing metadata (setup.py) ... done
Collecting accelerate
  Using cached accelerate-0.14.0-py3-none-any.whl (175 kB)
Collecting bert-score
  Using cached bert_score-0.3.12-py3-none-any.whl (60 kB)
Collecting datasets
  Using cached datasets-2.7.0-py3-none-any.whl (451 kB)
Requirement already satisfied: more_itertools in /usr/lib/python3/dist-packages (from galai) (8.10.0)
Collecting nltk
  Using cached nltk-3.7-py3-none-any.whl (1.5 MB)
Collecting openai
  Using cached openai-0.25.0.tar.gz (44 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting parallelformers
  Using cached parallelformers-1.2.7.tar.gz (48 kB)
  Preparing metadata (setup.py) ... done
Collecting prompt_toolkit
  Using cached prompt_toolkit-3.0.32-py3-none-any.whl (382 kB)
Collecting galai
  Using cached galai-1.0.0.tar.gz (22 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-k_x7j7ch/galai_f9ad4fa1673448a1aee54d93e0792767/setup.py", line 16, in <module>
          with open("requirements.txt", "r") as f:
      FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Dataset

Hi all,

Firstly, thank you for the awesome work.

I was wondering if there were any plans to release the full dataset?

I believe this would be immensely beneficial for research.

Thank you,

Enrico

Academic paper summary example

Could you please share an example of how to use this model to summarize academic papers given full text?

Files sizes of models

When running and fetching the models, what final file sizes can we expect from each of the available models?

Hugging face shows 9gb for the huge file, but running after PIP, the downloading hasn't stopped for an hour, it just keeps fetching ~9gb files

Can be used on CPU?

Hi,

amazing works and very curious to try

can be used on a CPU (or parallelized on different CPUs)?

thank you,

Salvatore

Galai import failing after 1.1.0 update

I recently updated to galai 1.1.0, and now I can't go past import:

>>> import galai as gal
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\Lucas\AppData\Local\Programs\Python\Python37\lib\site-packages\galai\__init__.py", line 3, in <module> from galai.model import Model File "C:\Users\Lucas\AppData\Local\Programs\Python\Python37\lib\site-packages\galai\model.py", line 11, in <module> from galai.utils import escape_custom_split_sequence File "C:\Users\Lucas\AppData\Local\Programs\Python\Python37\lib\site-packages\galai\utils.py", line 66, in <module> @dataclass File "C:\Users\Lucas\AppData\Local\Programs\Python\Python37\lib\site-packages\galai\utils.py", line 116, in ModelInfo def all() -> list["ModelInfo"]: TypeError: 'type' object is not subscriptable

After that, I have tried reinstalling transformers, accelerate, tokenizers, and huggingface-hub, but nothing worked so far.

Thank you for your time