This is an evaluation set for the problem of directed/targeted test input generation, especially targeting Large Language Models (LLMs). The goal of directed test input (a.k.a. targeted test input) generation is to automatically generate test inputs to reach a certain code location or produce a particular result.

For example:

// EXAMPLE 1:
// generate an input of `s` that can reach the target line:
int arr(char *s){
    int symvar = s[0] - 48;
    int ary[] ={1,2,3,4,5};
    if (ary[symvar%5] == 5){
        // target line
    }
}
// EXAMPLE 2:
// or produce the expected output
int execute(char* s) {
    int ret = system(s);
    if(ret == 0){
        return OK; // expected
    }
    return ERR;
}

Targeted input generation plays a crucial role in various software engineering and security tasks, including fuzzing, bug reproduction (where the target is the bug location), and test suite augmentation (where the target is the specific code to be covered). It has also been extensively integrated with conventional testing tools to enhance coverage and overall performance.

Constraint-based techniques, such as symbolic execution and concolic testing, have been well-explored in this problem while Large Language Models (LLMs) have demonstrated exceptionally good performance in code understanding and reasoning. We use PathEval to benchmark and evaluate the ability of LLMs to solve the problem of directed test input generation.

LLMs and constraint-based tools have distinct advantages and disadvantages towards this problem. For instance, it is easy for constrint-based tools to generate the target input for the above EXAMPLE 1 while being challenging for LLMs. Interestingly, it is the opposite for EXAMPLE 2.

Installation

Linux (Debian)

Install gcc, g++, python3 and openjdk by the following commands.

apt update
apt install -y -q build-essential gcc g++ python3 python3-pip openjdk-11-jdk-headless libssl-dev
python3 -m pip install tqdm
git clone https://github.com/CGCL-codes/PathEval.git
cd PathEval

Docker

git clone https://github.com/CGCL-codes/PathEval.git
cd PathEval
docker build -t patheval .
docker run -it --rm patheval /bin/bash
# cd /work in container

Usage

The evaluation will compile and execute untrusted model-generated data and code (see scripts/*_check.py). It is strongly encouraged to run this project in a sandbox (e.g., docker container).

Users can simply use this dataset through a couple of APIs.

Example

from patheval import set_dataset, read_problems, evaluate_one
# select the dataset that corresponds to the language
set_dataset("patheval_cpp") # or "patheval_java" or "patheval_py" or "logic_bombs_c"
# load the problems
problems = read_problems()
# give the completion from your LLMs and the evaluate result will return.
# completion = query(...)
evaluate_one(problems[0], completion)['pass'] # True / False

Or you can validate offline with the script patheval.py:

python3 patheval.py --dataset patheval_java --input sample.jsonl

For the optional dataset see the documentation for set_dataset below. The format of sample.jsonl is as follows, the order doesn't matter.

{"index": 486, "completion": "\"Mary had a little lamb\", 4"}
{"index": 532, "completion": "\"Hello,Hello,world !\" "}
{"index": 572, "completion": "Arrays.asList(1.0, 2., 3.)"}
...

We expect that each question can have one or more rows of answers (multiple rows of data with the same index). If a question does not have a corresponding answer, then it fails by default.

APIs

set_dataset(dataset_name)

• Purpose: Selects the dataset for evaluation.
• Parameters:
  • dataset_name (str): The name of the dataset to use. Valid values are:
    • "patheval_cpp": C++ dataset
    • "patheval_java": Java dataset
    • "patheval_py": Python dataset
    • "logic_bombs_c": logic bombs dataset (in C language)
• Returns: None

read_problems()

• Purpose: Loads the problems from the selected dataset.
• Parameters: None
• Returns: problems (list): A list of problems.

evaluate_one(problem, completion)

• Purpose: Evaluate the given completion for a single problem.
• Parameters:
  • problem (object): One problem from the read_problems() returned list.
  • completion (str): The completion generated by LLMs.
• Returns: the input problem dictionary, with an additional key:
  • "pass" (bool): Indicates whether the completion passes the problem

Sample

For each sample, the following information is provided:

In logic_bombs samples, humaneval_task_id is replaced by logic_bombs_task_id.

The samples are placed under the data folder in .jsonl files, the samples in the three different files are semantically equivalent, but are implemented in different programming languages.

An example C++ sample is shown as follow.