ofa-sys/ofasys

OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models

ofasys's Issues

How to train the Generalist Model in your paper?

Thank you very much for sharing the awesome work!

Could you please share more information on how to train the Generalist Model in your paper? Like how to construct the datasets for different tasks.

BTW, when I run the inference code, it had the error: No module named 'datasets.utils'

Grounded Image Captioning doesn't work.

Feeding different boxes will always get the same outputs.

Upload pretrained checkpoints to alternative storage providers

Congrats on the awesome work in developing a solid framework that combines tasks and modalities. I'm impressed with the ease of combining a variety of types of input and output.

I'm experimenting with the code and pretrained checkpoints, but calling OFSys.from_pretrained with the multitasking checkpoint fails most of the time while it tries to download image generation checkpoints, e.g., ViT-B-16. I had to run it several times and it usually fails with a timeout error. Is it possible that you upload these models on HF Hub? I can also serve it on my Google Cloud Storage bucket if it's ok. We can also define a keyword argument in .from_pretrained factory method to set the preference for HF Hub or GCS as I guess this is only relevant outside China. If you agree, I can prepare a PR at the weekend.

unable to run 'usage in 15 minutes' example

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-3-8c9f93f73c07>](https://yucuju2ns5m-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230307-060233-RC01_514758488#) in <module>
      1 from ofasys import Task, Trainer, GeneralistModel
----> 2 task1 = Task(
      3     name='caption',
      4     instruction='[IMAGE:image_url] what does the image describe? -> [TEXT:caption]',
      5     micro_batch_size=4,

[/usr/local/lib/python3.9/dist-packages/ofasys/task/base.py](https://yucuju2ns5m-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230307-060233-RC01_514758488#) in __init__(self, cfg, **kwargs)
    205         self.cfg.update(**kwargs)
    206         self._generator = None
--> 207         self.diffuser_args = json.loads(cfg.diffuser_args)  # accessed by the diffusion criterion and generator
    208 
    209         self.datasets = {}

AttributeError: 'NoneType' object has no attribute 'diffuser_args'

AttributeError: type object 'OFASys' has no attribute 'from_pretrain'

Unable to load the checkpoint

Hi,

When I run the following code as shown in Readme:
model = OFASys.from_pretrained('multitask_10k.pt')

I had the following error:

about the running

the trainer cannot do the truncation by default

[/usr/local/lib/python3.9/dist-packages/ofasys/adaptor/general.py](https://rv8u4ubipi-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230307-060233-RC01_514758488#) in concat(self, modality_outputs)
    275                 end_pos = start_pos + seq_length
    276                 if modality_output.self_attn_bias[idx] is not None:
--> 277                     self_attn_bias[:, :, start_pos:end_pos, start_pos:end_pos] += modality_output.self_attn_bias[idx]
    278                 start_pos = end_pos
    279             assert start_pos == output.seq_length

RuntimeError: The size of tensor a (1025) must match the size of tensor b (1024) at non-singleton dimension 3

No Inference Stuff

AttributeError: 'GeneralistModel' object has no attribute 'active_executor'

Installation stuff

After installing using pip and run the example, the systems reports:

/usr/local/lib/python3.8/dist-packages/torch/cuda/__init__.py:497: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
100%|██████████| 1802818402/1802818402 [10:40<00:00, 2813923.19B/s]
download oss://ofasys/bpe/encoder.json: 100%|██████████| 0.99M/0.99M [00:02<00:00, 429kB/s]
download oss://ofasys/bpe/vocab.bpe: 100%|██████████| 446k/446k [00:03<00:00, 145kB/s]
download oss://ofasys/bpe/dict.txt: 100%|██████████| 589k/589k [00:05<00:00, 119kB/s]
/usr/local/lib/python3.8/dist-packages/torchvision/transforms/transforms.py:329: UserWarning: Argument 'interpolation' of type int is deprecated since 0.13 and will be removed in 0.15. Please use InterpolationMode enum.
  warnings.warn(
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-1-42de709d3bf6>](https://wjc8fl8bd8-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230306-060124-RC01_514432337#) in <module>
      1 from ofasys import OFASys
----> 2 model = OFASys.from_pretrained('http://ofasys.oss-cn-zhangjiakou.aliyuncs.com/model_hub/multitask_10k.pt')
      3 # model = model.cuda()  # Omit this line if you don't have a GPU

5 frames
[/usr/local/lib/python3.8/dist-packages/ofasys/preprocessor/default/image_code.py](https://wjc8fl8bd8-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230306-060124-RC01_514432337#) in __init__(self, global_dict, cfg)
     70         )
     71         if cfg.clip_model:
---> 72             import clip
     73 
     74             self.clip = clip

ModuleNotFoundError: No module named 'clip'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

oss://ofasys/bpe/encoder.json下载不下来

报错download oss://ofasys/bpe/encoder.json: 0.00B [00:00, ?B/s]

PermissionError Traceback (most recent call last)
Cell In[8], line 6
3 os.environ['CUDA_VISIBLE_DEVICES'] = '1'
4 from ofasys import OFASys
----> 6 model = OFASys.from_pretrained('https://ofasys.oss-cn-zhangjiakou.aliyuncs.com/model_hub/multitask_10k.pt')
7 model = model.cuda(0)

File d:\Anaconda3\lib\site-packages\ofasys\hub_interface.py:151, in OFASys.from_pretrained(cls, model_path, task_name, initialize_all_tasks)
148 tasks[cur_task].initialize(global_dict, is_train=False)
150 tasks['default'] = OFATask(task_cfg_default)
--> 151 tasks['default'].initialize(global_dict, is_train=False)
153 model_name, model_dict = list(state["configstore"]["model"].items())[0]
154 node = ConfigStore().get("ofasys.model", model_name)

File d:\Anaconda3\lib\site-packages\ofasys\task\base.py:222, in OFATask.initialize(self, global_dict, **kwargs)
220 if kwargs.get('is_train', True):
221 update_preprocess_config_by_template(self.cfg.preprocess, self.templates, self.name)
--> 222 self.general_preprocess = self.build_preprocess(self.cfg.preprocess, global_dict)
223 self.metrics = self.build_metrics(self.cfg.evaluation.metrics)
224 if kwargs.get('is_train', True):

File d:\Anaconda3\lib\site-packages\ofasys\task\base.py:338, in OFATask.build_preprocess(self, cfg, global_dict)
328 def build_preprocess(self, cfg: PreprocessConfig, global_dict):
329 """
...
905 result = self.get_object(key, byte_range=byte_range, headers=headers, progress_callback=progress_callback,
906 process=process, params=params)
908 if result.content_length is None:

PermissionError: [Errno 13] Permission denied: 'C:\Users\Aurora\.cache\ofa\tmpeyy32j3g'
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...