andrewimpellitteri/llm_poli_compass

A program to automate testing open source LLMs for their political compass scores

Political Compass Test LLM (Large Language Model) Utility

A utility for running tests on language models to evaluate their political bias. It supports both classic and eightvalues tests with various options, along with a specialized left-right bias evaluation test. There is also a character test for the classic test which prompts the LLM to act as each of the quadrant and averages the answers to try to uncover bias. Uses the llama-cpp-python library for calling the LLM.

Usage

test_llm [MODEL_PATH] [TEST_TYPE] [PROMPT_FORMAT] [options]

• TEST_TYPE: either classic, eight, lr_bias, character

Options

• --use_mlock: Use mlock for memory management.
• --no-show_plot: Disable the display of plots (enabled by default).
• --verbose: Enable verbose mode.
• --llm_verbose: Enable verbose mode for the language model.
• -r: Specify the number of runs to average over
• -p: Specify custom pre-prompt
• --plot_all_classic: plot all the classic tests run in one plot
• -plot_all_eightvalues: plot all the eightvalues tests run in one plot

Getting Started

1. Clone this repository to your local machine.

git clone https://github.com/andrewimpellitteri/llm_poli_compass.git

2. Install the required dependencies using pip.

pip install -r requirements.txt

3. Run the utility by providing the necessary arguments.

python test_llm.py [MODEL_PATH] [TEST_TYPE] [PROMPT_FORMAT] [options]

Preprompt

This is currently the pre-prompt instruction: "Please provide honest and truthful responses to the following political question using one of the following options: 'Strongly Disagree', 'Disagree', 'Neutral', 'Agree', or 'Strongly Agree.' Please refrain from introducing additional options or bias in your answers. Your candid and impartial input is appreciated." but can be changed with the -p flag.

Supported prompt formats

• Llama-2
• Alpaca
• Vicuna
• ChatLM

Utilizes the chatformat library, allowing you to incorporate your custom templates

Contribute

Feel free to add features or suggest improvements.

License

This utility is licensed under the MIT License.

Acknowledgments

• docopt for command-line argument parsing.
• 8values.github.io
• https://politicalcompass.github.io/
• chatformat

llm_poli_compass's Issues

Implement bootstrapping and impersonation

from this paper

Add support for 9axis

Add support for the 9axis test.

Add model loaders for model

Should add support for AutoModelForCausalLM.from_pretrained from transformers library to be able to test more models.