pharringtonp19/llmft

LLMs for Causal Inference

Hi there 👋

llmft's Issues

Experiment # 2

• Create New Dataset (positive health didn't work)
• Cross Tabs to evaluate what the model learns

Experiment # 1 (Descriptive)

• Seed

Focus Loss Function

• Raises an error with the decoder model

Inference

• Conformal Prediction
• Closed Form

Encoder Model

• Add a k-fold split option

To Do

• Encoder Model

  • Class Imbalance

• Decoder Model

  • Add a callback that can record the models output to various prompts during training
  • Enable Early Stopping (Callback)

Tokenizers Parallelism

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Enable Sharing Compiled Models Across Executions of the notebook

Report Standard Validation Loss

Decoder Model

• Implement compute_recall
  • Currently this involves a separate forward pass of the model. We should be able to combine this with evaluation_loss
  • model.generate isn't working atm
• flash-attention implementation
• Improve logging format: log_predictions
  • Can we make this interactive via gradio?
• Plot learning rate
• Early Stopping
• two 'yes' tokens