krypticmouse / double-bind-training Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 3.0 2 MB

double-bind-training's Introduction

Hi 👋, I'm Herumb Shandilya a.k.a. krypticmouse

🔭 I’m currently working on ColBERT and DSPy
🌱 I’m currently learning Rust, Distributed Systems and Compiler Optimizations
👯 I’m looking to collaborate on Research around LLM Agents
👨‍💻 All of my projects are available at krypticmouse.github.io/
📝 All of my blogs are available here
💬 Ask me about ML, DL, IoT, NLP, CV, Web
📫 How to reach me [email protected]
⚡ Fun fact I stalk PyTorch in my Free Time

Connect with me:

Languages and Tools:

double-bind-training's People

Contributors

Watchers

Forkers

joelthe1 manny-uncharted cleong110

double-bind-training's Issues

Feature request: add LM Adapter run name to the os.environ

If at some point in the LM Adapter training it saved the run name to the env, we could use it later in the notebook for, say, saving the folder, or tagging the downstream NER run.

NER training code showing as "crashed" on WandB even when it finished successfully.

All the NER runs on https://wandb.ai/double-bind-ner/masakhane-ner-test-run seem to show "crashed"

Feature: code for Masakhane News and Bloom-lm datasets

Create snippets that take in a language code and download from Masakhane News/Bloom-lm
Combine that data with train.txt, etc. from other datasets
Make sure it all gets tagged in Weights and Biases!
Incorporate into repo AND colab notebook for AfricaNLP

Feature request: tag the NER runs on wandb with the specific adapter model we used.

Feature request: add tag for datasets used.

A couple ways to go about this: we could ask the user to edit, or we could automatically infer from dataset downloads in the HuggingFace cache. The first one seems easier but less reliable? But would fit in nicely with our current paradigm of having people do this in Colab

RuntimeError in train_ner_adapter: expanded size of the tensor must match the existing size at non-singleton dimension 1.

Evaluating:   0% 0/38 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "train_ner_adapter.py", line 726, in <module>
    main()
  File "train_ner_adapter.py", line 685, in main
    result, _ = evaluate(args, model, tokenizer, labels, pad_token_label_id, mode="dev", prefix=global_step)
  File "train_ner_adapter.py", line 288, in evaluate
    logits = model(avg_emb)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/adapters/models/roberta/adapter_model.py", line 68, in forward
    outputs = self.roberta(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/adapters/context.py", line 108, in wrapper_func
    results = f(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/roberta/modeling_roberta.py", line 843, in forward
    buffered_token_type_ids_expanded = buffered_token_type_ids.expand(batch_size, seq_length)
RuntimeError: The expanded size of the tensor (768) must match the existing size (514) at non-singleton dimension 1.  Target sizes: [1312, 768].  Tensor sizes: [1, 514]

Update Readme with usage instructions

After #10 is complete, we should add instructions for how to do/how we did our phase 1 experiments to the readme. For a start can just be "run this Colab notebook which trains a language adapter and then finetunes on Masakhane NER".

Merge train-lm-adapter into Master

we've been using the https://github.com/krypticmouse/double-bind-training/tree/train-lm-adapter as our "master" branch. For future iterations we would like to merge this into Master so that it's the first thing people see.

merge https://github.com/krypticmouse/double-bind-training/tree/train-lm-adapter as the master branch.