danielsaggau/ldsi_project

Project on language models for legal text generation

LDSI_Project

This is my project on language models for legal text generation.

Loading data into Python as a Dataframe

You can create the pandas dataframe using the bulk data provided by 'courtlistener'. To generate your own dataset see the 'create_dataset.py' file. You can adjust the size by setting a different k value. I used 50.000 as mentioned in the paper.

Cleaning

Further, you can do the cleaning using the 'preprocessing.py' file.

Tokenization & Chuncks

To create the sentence boundary detection use the 'embedding.py' file.

Fine Tuning

See the 'fine_tuning.py' file to fine tune your model.

Evaluation

To reproduce the plots in the paper use the 'plots.py' document