tkhang1999/repair-prompts

Various prompt templates for automatic program repair using LLMs

repair-prompts

Various repair prompts for finding and fixing security vulnerabilities in JavaScript programs using Large Language Models (LLMs).

Prompt Design

The repair prompts are categorized into 3 types of prompt templates, ranging from no additional context to comprehensive detail.

• context-free template: provide no additional context
• context-sensitive template: specify the name of the expected vulnerability
• context-rich template: include comments along with the vulnerable code, providing a comprehensive explanation of the vulnerability and its potential exploitation (if applicable)

Vulnerability Selection

The vulnerabilities described in these prompts are adapted from the latest 2023 CWE Top 25 List. However, as not all vulnerabilities in this list are applicable to JavaScript, 20 out of the 25 vulnerabilities that are most relevant to JavaScript are selected.

Evaluation

Based on the 3 proposed prompt templates and the identified 20 vulnerabilities, there is a total number of 60 prompts used in testing with LLMs, namely ChatGPT and Bard.

The performance of these LLMs is as follows:

	ChatGPT			Bard
	context-free	content-sensitive	context-rich	context-free	content-sensitive	context-rich
CWE-20	✗	✓	✓	✗	✓	✓
CWE-22	✗	✓	✓	✗	✓	✓
CWE-77	✓	✓	✓	✓	✓	✓
CWE-78	✓	✓	✓	✗	✓	✓
CWE-79	✓	✓	✓	✓	✓	✓
CWE-89	✓	✓	✓	✓	✓	✓
CWE-94	✓	✓	✓	✓	✓	✓
CWE-125	✗	✓	✓	✗	✓	✓
CWE-190	✗	✗	✓	✗	✗	✗
CWE-269	✗	✗	✗	✗	✓	✓
CWE-276	✓	✓	✓	✓	✓	✓
CWE-287	✗	✗	✓	✗	✓	✓
CWE-306	✗	✗	✓	✗	✓	✓
CWE-434	✗	✓	✓	✗	✓	✓
CWE-476	✓	✓	✓	✓	✗	✗
CWE-502	✗	✓	✓	✓	✓	✓
CWE-787	✓	✓	✓	✓	✓	✓
CWE-798	✗	✓	✓	✗	✗	✗
CWE-862	✗	✓	✓	✗	✗	✓
CWE-863	✗	✓	✓	✗	✓	✓

Citation

If you find this repository helpful, feel free to cite our publication A Study of Vulnerability Repair in JavaScript Programs with Large Language Models:

@inproceedings{le2024study,
  title={A Study of Vulnerability Repair in JavaScript Programs with Large Language Models},
  author={Le, Tan Khang and Alimadadi, Saba and Ko, Steven Y},
  booktitle={Companion Proceedings of the ACM on Web Conference 2024},
  pages={666--669},
  year={2024}
}