fstiehle/llms4science

Replication package for LLMs for Science: Usage for Code Generation and Data Analysis

This is the replication package for the pre-print LLMs for Science: Usage for Code Generation and Data Analysis, which studies the use of LLM-tools for code generation in the scientific process.

Following open science principles, and to support replicability, we publish this replication package alongside recording all our interactions with the tools and our evaluation criteria.

Structure

The logs of our interactions with the tools are structured by use case. They contain two logs for each of our two authors conducting the experiments. The code generation folder also contains our code to conduct the performance benchmark.

• Code_Generation
• Data Analysis
• Data Visualization

We provide our assessment rubric, used to assess the above mentioned results. For convenience, they're provided in different formats.

• Rubric Assessment.csv
• Rubric Assessment.pdf
• Rubric Assessment.xlsx

We have compiled some examples of misleading results for the data analysis and visualization use cases.

• examples_misleading_results.pdf