GRAMMAR2FIX is an active oracle learning technique for programs processing string inputs. Given a single failing input of a bug, it learns a grammar describing the pattern of all the failing inputs of the bug, interacting with a bug oracle systematically. GRAMMAR2FIX returns this grammar as a collection of Deterministic Finite Automata(DFA), and the grammar can serve as an automated test oracle for the bug. GRAMMAR2FIX also produces a test suite in grammar learning, which can be used as a repair test suite in Automated Program Repair.

GRAMMAR2FIX is implemented in Python. The following benchmarks are used in the experiments.

• QuixBugs
• IntroClass
• Codeflaws

GenProg (Paper, Tool) is used as the automated program repair tool.

We conducted our experiments in Ubutu 18.04.6 LTS with 32 logical cores.

Getting Started

Step 1- Install Supporting Components

GRAMMAR2FIX needs Python 3.7.2 or greater. Also, "git", "numpy" and "wget" are necessary. If these are not available, use the following commands to install it in linux.

apt-get update
apt-get -y install git wget build-essential time zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev
#Install Python 3.7
pushd /tmp
wget https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tar.xz
tar -xf Python-3.7.2.tar.xz
cd Python-3.7.2
./configure --enable-optimizations
make -j4
make altinstall
ln -s $(which pip3.7) /usr/bin/pip
mv /usr/bin/python /usr/bin/python.old
ln -s $(which python3.7) /usr/bin/python
popd
# install numpy
pip install numpy

Step 2- Install benckmarks

Download and install benchmarks as follows.

# Install QuixBugs
git clone https://github.com/jkoppel/QuixBugs
# Install IntroClass
git clone https://github.com/ProgramRepair/IntroClass
cd IntroClass
make
cd ~
# Install Codeflaws
git clone https://github.com/codeflaws/codeflaws
cd codeflaws/all-script
wget http://www.comp.nus.edu.sg/~release/codeflaws/codeflaws.tar.gz
tar -zxf codeflaws.tar.gz
cd ~

Step 3- Install GRAMMAR2FIX

Clone GRAMMAR2FIX github repository

git clone https://github.com/charakageethal/grammar2fix.git

Under our setup, Getting Started step took approximately 35 minutes to complete.

Reproduce Experimental Results

Running the experiments to measure oracle accuracy and labelling effort of GRAMMAR2FIX.

To repat the experiments related to oracle accuracy and labelling effort, GRAMMAR2FIX contains "quixbugs_experiments.sh" for QuixBugs, "introclass_experiments.sh" for IntroClass and "codeflaws_experiments.sh" for Codeflaws.

Step 1- Run GRAMMAR2FIX on the benchmarks

cd grammar2fix
./quixbugs_experiments.sh <<path to QuixBugs>>
./introclass_experiments.sh <<path to IntroClass>>
./codeflaws_experiments.sh <<path to Codeflaws>>

The approximate time taken for each script under our setup is as follows.

The .csv files are copied to the grammar2fix/results folder

Step 2-Reproduce the results

We use R to generate the graphs. Follow these instructions to install R and Rstudio. After that, go to the results folder and run "overall_accuracy_benchmarks.R" and "grammar_gen_wise_plots.R". "overall_accuracy_benchmarks.R" will generate Figure 7 (a) and 7 (b), and "grammar_gen_wise_plots.R" will generate Figure 8 and 9 in the paper.

Running Program Repair experiments with codeflaws.

We use GenProg docker to run the program repair experiments.

Step 1- Setup GenProg docker

Setup a docker container for GenProg repair tool

docker pull squareslab/genprog
docker run -it squareslab/genprog /bin/bash

Step 2- Setup GRAMMAR2FIX and benchmarks

Repeat the steps in Getting Started to setup GRAMMAR2FIX and Codeflaws in the docker container. Download GRAMMAR2FIX repository to the "/root" directory of the docker container.

Step 3 - Run GRAMMAR2FIX on Codeflaws

Run the script "codeflaws_repair_experiments.sh" as follows.

cd grammar2fix
./codeflaws_repair_experiments.sh <<path to codeflaws>>

Step 4 - Reproduce program repair results

Go to grammar2fix/results/codeflaws_repair and run "codeflaws_repair.R". This script will generate Figure 10 (a) and 10 (b).

Running experiments through Dockerfile

We have provided a Dockerfile to run the experiments realated to oracle accuracy and labelling effort of GRAMMAR2FIX (See Above). Use the following command to build the docker container from the Dockerfile. Under our setup, it took approximately 45 minutes to build the docker container.

sudo docker build -t <<docker_container_name>> .

Use the following command to run the docker container.

sudo docker run -it <<docker_container_name>> /bin/bash

The Dockerfile downloads the benchmarks to the following locations.

• QuixBugs : /root/QuixBugs
• IntroClass : /root/IntroClass
• Codeflaws : /root/codeflaws/all-script/codelfaws

After that run the experiment scripts as previously (See Above)