martinezmatias/quixbugs

A multi-lingual program repair benchmark set based on the Quixey Challenge

QuixBugs Benchmark

The QuixBugs benchmark consists of 40 programs from the Quixey Challenge translated into both Python and Java. Each contains a one-line defect, along with passing (when possible) and failing testcases. Defects fall into one of 14 defect classes. Corrected Python programs are also supplied. Quixbugs is intended for investigating cross-language performance by multi-lingual program repair tools. For more details, see the paper.

Background: Quixey Challenge

From 2011 to 2013, mobile app search startup Quixey ran a challenge in which programmers were given an implementation of a classic algorithm with a bug on a single line, and had one minute to supply a fix. Success entailed $100 and a possible interview. These programs were developed as challenges for humans by people unaware of program repair.

Installation & Usage

Simply clone the repo.

git clone https://github.com/jkoppel/QuixBugs

The Java programs are already compiled (see *.class files in java_programs). Note the all java programs are in the same package called java_programs. The utility class JavaDeserialization.java requires you to download the external library Gson.

All Python is written in Python3.

To run both defective versions of a program against their tests, as well as the corrected Python version, use the test driver:

python3 tester.py program_name

Output is printed for visual comparison.

Structure & Details

The root folder holds the test driver. It deserializes the JSON testcases for a selected program, then runs them against the defective versions located in java_programs/ and python_programs/. The exception is graph-based programs, for which the testcases are located in the same folder as the corresponding program (they are still run with the test driver in the same manner).

For reference, corrected versions of the Python programs are in correct_python_programs/.

Programs include:

• bitcount
• breadth_first_search*
• bucketsort
• depth_first_search*
• detect_cycle*
• find_first_in_sorted
• find_in_sorted
• flatten
• gcd
• get_factors
• hanoi
• is_valid_parenthesization
• kheapsort
• knapsack
• kth
• lcs_length
• levenshtein
• lis
• longest_common_subsequence
• max_sublist_sum
• mergesort
• minimum_spanning_tree*
• next_palindrome
• next_permutation
• pascal
• possible_change
• powerset
• quicksort
• reverse_linked_list*
• rpn_eval
• shortest_path_length*
• shortest_path_lengths*
• shortest_paths*
• shunting_yard
• sieve
• sqrt
• subsequences
• to_base
• topological_ordering*
• wrap

* - graph-based algorithm

To generate Junit test cases

• Run TestsGenerator and check the generated Junit test cases in the folder of java_testcases/junit folder

Authors

Contact Derrick Lin @drrckln, Angela Chen @angchen, or James Koppel @jkoppel for questions.