This repository contains the data and code used in our research paper: "Navigating the Ocean of Biases: Political Bias Attribution in Language Models via Causal Structures". The study delves into the biases present in language models, particularly focusing on political biases, and employs causal structures to attribute and understand these biases. While this repository does not provide any readily usable tools, it allows using our dataset and reproducing or building on our results. Due to the size of the large dataset, we uploaded them to our GitLab.
- Python 3.x
- OpenAI API key for accessing GPT models
Clone the repository and install the required Python packages:
git clone [repository URL]
cd [repository directory]
pip install -r requirements.txt- To explore and analyze the dataset, refer to Jupyter notebooks:
create_llm_measurements_dataset.ipynbandstatistical_analysis.ipynb. - These notebooks include examples on how to load and work with the dataset.
- Set your
OPENAI_API_KEYin the environment. - Our caching system stores previous prompts and responses. You can use our cache files, which contains all prompts and their responses from our runs. They are too big for GitHub and you will have to access it via our GitLab.
- To generate new data, run
create_llm_measurements_dataset.ipynb. - To add observables, modify
datasets/llm_measurements/observables.py. For custom prompts, seedatasets/llm_measurements/prompt_builder.py.
- The raw CPD debate dataset and the scraper we used to create it can be found in
/datasets/cpd_debates/cpd_debate_scraper.py.
We welcome contributions to improve the dataset and tools. Please feel free to fork the repository and submit pull requests.
This project is licensed under the MIT License - see the LICENSE file for details.