krish1925/llamapy

Implementing LLAMA 3 on python using pytorch

LlamaPy

LlamaPy is a Python project for natural language processing tasks using a custom model called LLaMA3. This model is designed for tokenization and language generation tasks.

Architecture

• Transformer-based architecture
• Multi-head attention mechanism
• Multiple layers for hierarchical representation learning
• Long-range sequence modeling capabilities

Specifications

• Dependencies: LlamaPy relies on the following Python libraries:
  • torch for deep learning computations
  • nltk for natural language processing tasks such as tokenization
  • tqdm for progress tracking during training
• Model Parameters:
  • hidden_dim: Dimensionality of the model's hidden states
  • num_heads: Number of attention heads in the multi-head attention mechanism
  • num_layers: Number of layers in the transformer architecture
  • max_length: Maximum sequence length supported by the model
• Training Procedure:
  • The model is trained using an Adam optimizer with a cross-entropy loss function.
  • Training data is fed to the model in batches, with a specified batch size and sequence length.
  • Training progresses over multiple epochs, with progress tracked using the tqdm library.

Installation

To use LlamaPy, you need to install the required dependencies. You can install them using pip:

pip install tqdm nltk torch

You also need to download the NLTK corpora. Run the following commands after installing NLTK:

import nltk
nltk.download('punkt')
nltk.download('gutenberg')

Usage

Here's a brief overview of how to use LlamaPy:

• Import the necessary modules:

import subprocess
import nltk
from tqdm import tqdm
import torch

Next, run the code in a jupyter notebook, which would train the llama3 model on nltk's gutenberg corpus.

2. Model Initialization:

   • Initialize the LLaMA3 model with appropriate parameters such as vocabulary size, hidden dimension, number of heads, and number of layers.

3. Training:

   • Train the model using your dataset by feeding tokenized sequences to the model in batches.
   • Adjust training parameters such as batch size, sequence length, learning rate, and number of epochs as needed.

4. Inference:

   • Once trained, use the model for text generation or other natural language processing tasks.
   • Provide input sequences to the model and obtain predictions for the next tokens.

llamapy's Issues

Training Issue

Unable to train the llama3 model due to token index exceeding the vocab size.

Training Issue, Loss Stagnation

Generation just yields a slew of commas, needs fixing. Loss doesn't reduce, and stabilizes around 6.5