jopetty/growing-tokens

Train a tokenizer while training a language model

growing-tokens

Can you train a tokenizer while training a language model? Kind of! This project shows how you can train a language model starting with a character-level tokenizer and progressively merging tokens with high mutual information over the course of training. We provide a demonstration for how to train on the BabyLM corpus.

Installation Steps on Greene

1. Clone this repo to /scratch/NETID and cd into it.
2. Move to an interactive job node:

srun --pty /bin/bash

3. Copy the following singularity overlay:

cp -rp /scratch/work/public/overlay-fs-ext3/overlay-15GB-500K.ext3.gz .

4. Extract the gzipped overlay:

gunzip overlay-15GB-500K.ext3.gz

5. Download the Miniconda Installer

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

6. Launch the container in read/write mode:

singularity exec --overlay overlay-15GB-500K.ext3:rw /scratch/work/public/singularity/cuda11.6.124-cudnn8.4.0.27-devel-ubuntu20.04.4.sif /bin/bash

7. Install Miniconda

bash Miniconda3-latest-Linux-x86_64.sh -b -p /ext3/miniconda3 && rm Miniconda3-latest-Linux-x86_64.sh

8. Create the following script at /ext3/env.sh:

#!/bin/bash

source /ext3/miniconda3/etc/profile.d/conda.sh
export PATH=/ext3/miniconda3/bin:$PATH
export PYTHONPATH=/ext3/miniconda3/bin:$PATH

9. Activate the conda base environment

source /ext3/env.sh

10. Install packages we need.

conda env create

11. Download data

conda activate ccm && python src/data.py

growing-tokens's Issues

Cleaning babylm data

• clean data to remove title lines in wikipedia (they start with '=')
• consider any other possible things to remove

Training baselines

• how long does a single epoch take?
• do multiple epochs (w/ normal tokenizer) help?

Tokenization comparison metrics

Add methods to compare any two tokenizations for:

• Overlap
• AoA (if both tokenizations are incremental)
• Adherence to gold standard morphological segmentation?

Do tokens split across spaces?

Initial alphabet isn't normalized

Initial vocab is too large if we're going to lowercase + remove accents

https://github.com/jopetty/ccm-project/blob/543ae628b2f5696efc472968a7950ca38daca7cd/src/data.py#L158C1-L162C1

retokenize every k steps

• rn we tokenize once, which works for usual tasks but not for ours since we need to adjust the tokenizer during training
• should be a training param vocab_update_steps which controls this

Tokenizer baselines

• End-to-end BPE vocabulary
• BPE vocabulary with steps
• Multiple iterations of BPE on subsets of the data