HumanEval: Functional Code Generation Evaluation with Pass@k

Article links

Make inline references clickable

How do we know if a language model can write code? The goal is code that runs correctly and passes test cases, not code that merely looks right. This question became urgent as models like GPT-3 demonstrated the ability to generate plausible-looking Python functions, yet often produced subtle bugs that rendered them useless in practice. The stakes are high. Incorrect code can crash systems, introduce security vulnerabilities, or silently produce wrong results. Traditional NLP evaluation metrics like BLEU or ROUGE, which we explored in Part LIII: Evaluation Fundamentals, measure lexical overlap between generated and reference text. These metrics were originally designed for machine translation and text summarization, where n-gram overlap correlates reasonably well with human judgments of quality. But code is fundamentally different from natural language: a function can be lexically distant from a reference implementation while being perfectly correct, or lexically similar while containing a critical flaw. A single misplaced semicolon or off-by-one error can make code completely non-functional, even if 99% of the tokens match the reference.

HumanEval, introduced by OpenAI in 2021 alongside the Codex model, addressed this evaluation gap by focusing exclusively on functional correctness. Rather than comparing strings, HumanEval executes generated code against unit tests. It consists of 164 hand-crafted Python programming problems, each with a function signature, docstring, several test cases, and a canonical solution. The benchmark quickly became the standard for evaluating code generation capabilities, establishing a methodology that subsequent benchmarks like MBPP would extend and refine. Before HumanEval, the field lacked a rigorous way to verify whether models could produce working software, leading to inflated claims based on superficial metrics.

The core innovation of HumanEval lies in its evaluation metric: pass@k. Instead of measuring similarity, pass@k estimates the probability that at least one out of independently generated samples passes all test cases for a given problem. This shifts the focus from single-shot perfection to the model's ability to generate correct solutions within a budget of attempts, much as a programmer tries multiple approaches until one succeeds. This probabilistic framing acknowledges the inherent stochasticity of large language models, where temperature sampling produces diverse outputs, and a failed first attempt does not indicate incapability.

To appreciate why HumanEval was designed the way it was, it helps to understand the context in which it appeared. In 2021, OpenAI released Codex, a model fine-tuned from GPT-3 on a large corpus of publicly available code from GitHub. Codex demonstrated remarkable ability to complete code from natural language descriptions, form the basis of GitHub Copilot, and generate entire functions from docstring prompts. This was a qualitative leap from earlier code generation approaches that relied on template-filling or grammar-based synthesis.

But how capable was Codex, really? The question proved surprisingly difficult to answer. Early informal evaluations showed that Codex could produce impressively correct-looking code for simple problems, yet fail silently on edge cases that a human would catch immediately. Researchers needed a systematic benchmark that could produce a single comparable number while still capturing the nuances of functional correctness. They could not simply ask human judges to rate code quality at scale; that would be too expensive and inconsistent. They could not use BLEU scores for the reasons described above. They needed execution.

The Codex paper introduced HumanEval alongside the model, serving the dual purpose of showing the model's capabilities and establishing a reproducible evaluation protocol. Codex achieved 28.8% pass@1 on HumanEval in its initial evaluation, meaning that roughly one in four randomly generated solutions passed all test cases on the first attempt. This number was simultaneously impressive, far beyond what previous code generation approaches could reach, and sobering: it meant that more than 70% of first-attempt solutions were incorrect. The pass@100 score of 72.3% told a different story: when the model could generate 100 samples per problem, the probability of finding at least one working solution was much higher, suggesting that Codex possessed the relevant knowledge but required multiple attempts to express it reliably.

The contrast between pass@1 and pass@100 immediately raised an important practical question: what is the right for your use case? A developer using a coding assistant interactively might tolerate examining 3 or 5 suggestions. An automated test repair system might be willing to generate 100 candidates and run them all through a test suite. The pass@k metric's ability to capture performance at different budgets made it immediately useful for a range of deployment scenarios.

HumanEval comprises 164 programming problems designed to test algorithmic reasoning, language understanding, and basic mathematics. Each problem follows a consistent structure that facilitates automated evaluation while testing real programming capability.

The four components of each problem are:

• Prompt: A function signature (name and parameters) followed by a docstring describing the problem, including examples of expected input-output behavior
• Canonical Solution: A reference implementation that correctly solves the problem, written by expert programmers
• Test Cases: A series of assertions that verify functional correctness across normal inputs, edge cases, and boundary conditions
• Entry Point: The function name to be tested, so the evaluation harness knows which function to call

The problems range from simple string manipulations, such as reversing words or checking palindromes, to more complex algorithmic challenges involving data structures, recursion, and combinatorics. The difficulty gradient ensures that benchmarks can distinguish between models with different capability levels, from basic syntax understanding to advanced algorithmic reasoning.

These problems were hand-written by humans rather than scraped from GitHub or Stack Overflow. This design choice intentionally limits the benchmark's size to reduce data contamination and the chance that models were inadvertently trained on the test set during pre-training on public code repositories. The authors specifically curated problems to be distinct from common programming exercises found online, reducing the risk that models reach high scores merely by memorizing solutions rather than reasoning about novel problems. This careful curation process explains why the dataset contains only 164 problems: distinct, contamination-resistant problems take precedence over a larger collection.

The selection of Python as the sole programming language shows practical considerations at the time. Python dominated scientific computing and machine learning workflows, Codex's primary use case. Python's readable syntax also makes docstrings more natural and expressive than in languages with more ceremonial type annotations. The choice came at a cost, however. Code generation capability in languages like Java, C++, Rust, or JavaScript may differ substantially from Python, and a benchmark restricted to one language cannot capture the full picture of multilingual code generation capability.

Each problem includes multiple test cases designed to cover edge cases and prevent trivial solutions. On average, problems contain 7.7 test cases, ranging from basic functionality checks to boundary condition verification. For instance, a sorting problem might test empty lists, single elements, already-sorted data, reverse-sorted data, and lists with duplicates. The tests use standard Python assert statements and are executed in a sandboxed environment for security, preventing generated code from accessing the filesystem, network, or system resources.

Test case design is subtle. If tests are too simple, models can pass them by implementing incorrect but coincidentally correct solutions for the tested inputs. If tests are too complete, they become tantamount to a formal specification, which defeats the purpose of testing generalization. The HumanEval authors aimed for the middle ground: enough tests to reject obviously wrong solutions and catch common edge cases, but not so many that the test suite effectively encodes the full problem specification.

One consequence of sparse test coverage is that some solutions that pass all tests are technically incorrect for inputs outside the tested domain. This phenomenon, known as overfitting to the test suite, can inflate pass@k scores when the model generates solutions that satisfy the observable tests while failing on more complete inputs. Later benchmarks like HumanEval+ addressed this by materially expanding the test suite, revealing that many solutions declared correct by the original tests were in fact subtly wrong.

The 164 problems span a wide range of difficulty levels, though they skew toward beginner and intermediate programming challenges rather than advanced algorithmic problems. This distribution shows the benchmark's intended use case: measuring whether a model understands programming concepts well enough to implement common patterns, not whether it can solve competitive programming olympiad problems.

Approximate difficulty categories include:

• Easy (roughly 40%): Single-operation transformations, basic string processing, simple arithmetic. A human programmer could solve these in under a minute.
• Medium (roughly 45%): Multi-step algorithms, list processing, basic data structure usage, simple recursion. These require understanding the problem structure but not advanced algorithms.
• Hard (roughly 15%): Dynamic programming, combinatorics, mathematical reasoning, or complex string parsing. These require algorithmic insight and careful implementation.

This distribution means that even modest language models can reach nonzero pass@1 scores by solving the easiest problems, while state-of-the-art models approach saturation on the medium difficulty tier and are distinguished primarily by their performance on hard problems. As models improved, the easy and medium problems became near-saturated, reducing the discriminative power of the benchmark and motivating more challenging successors.

Before HumanEval, code generation models were often evaluated using BLEU scores against reference implementations. BLEU (Bilingual Evaluation Understudy) counts matching n-grams between generated and reference text, awarding higher scores for longer matching sequences. However, code exhibits a many-to-one mapping between syntax and semantics: numerous syntactically different implementations can solve the same problem correctly.

Consider a function that computes the sum of a list. A reference implementation might use a for loop with an accumulator variable, while a generated solution uses Python's built-in sum() function, or perhaps numpy.sum(), or a recursive approach, or a list comprehension with sum(). Lexically, these share few n-grams. The loop-based solution contains tokens like for, in, range, and +=, while the built-in solution contains just return sum(nums). A BLEU score would penalize the correct built-in solution heavily, despite it being more Pythonic and efficient. Conversely, a buggy implementation might differ by only a single character (changing + to - or < to <=), achieving high lexical similarity while being completely wrong or failing on edge cases.

HumanEval discards lexical comparison entirely. A generated solution is correct if and only if it passes all test cases when executed. This binary judgment, pass or fail, eliminates the ambiguity of partial credit. It also aligns evaluation with the ultimate purpose of code generation: creating runnable, correct programs that satisfy specifications. This execution-based approach captures semantic equivalence regardless of syntactic form, recognizing that there are infinitely many ways to write correct code.

This approach requires executing untrusted code, necessitating security measures like sandboxing, timeout limits, and resource constraints. Generated code could theoretically contain infinite loops, attempts to delete files, or memory exhaustion attacks. The evaluation harness runs each generated solution in an isolated environment, catching exceptions, timeouts, and infinite loops as failures. This sandboxing adds computational overhead compared to string matching, but gives definitive evidence of correctness.

The sandboxing problem is non-trivial in practice. A naive execution environment allows generated code to import arbitrary libraries, make network requests, read environment variables containing API keys, or spawn subprocesses. Production evaluation frameworks like the human-eval package released by OpenAI use Unix process isolation, memory limits, and execution timeouts to constrain these risks. The timeout limit also handles infinite loops: if a solution does not terminate within a fixed time (typically a few seconds per test case), it is marked as failing. This prevents pathological cases where a model generates an exponential-time algorithm that technically produces the correct answer for small inputs but times out on larger test cases.

The difficulty of secure code execution has prevented some researchers from running HumanEval evaluations in-house, instead relying on cloud-based evaluation services. This creates a reproducibility challenge: different sandboxing implementations may produce slightly different results, and slight differences in Python version or library availability can cause edge-case test failures. The field has gradually converged on standardized evaluation containers to address this, but it remains a source of subtle inconsistency in reported numbers.

The primary metric for HumanEval is pass@k, which measures the probability that at least one of samples from the model passes the test suite for a given problem. This metric shows a practical reality: we rarely expect perfect code on the first attempt, and modern coding assistants often generate multiple suggestions for you to choose from. Because we cannot sample infinitely from the model, we estimate this probability using a finite number of samples (where ).

The metric addresses a subtle but important statistical issue. If we simply reported the fraction of correct samples ( ), we would underestimate a model's utility in scenarios where users can examine multiple options. For example, if a model generates 100 samples and 10 pass, the single-sample accuracy is 10%, but the probability of finding a working solution when allowed to see 10 samples is substantially higher than 10%. Pass@k quantifies this cumulative probability, capturing the model's capacity to generate diverse, correct solutions within a search budget.

For a given problem, suppose we generate samples and find that of them pass all tests. If we were to randomly select samples without replacement from these , the probability that at least one is correct is given by the hypergeometric distribution:

where:

• : total number of samples generated per problem (the sample pool size)
• : number of correct samples (passing all tests)
• : number of samples selected for evaluation (the budget of attempts)
• : the total number of ways to choose samples from
• : the number of ways to choose samples from the incorrect ones

The formula uses the complement rule: we calculate the probability that all selected samples are incorrect (the ratio of binomial coefficients) and subtract this from 1 to get the probability that at least one is correct. This approach assumes sampling without replacement, which is appropriate when selecting distinct samples from the generated candidates. Without replacement is the correct model here because in practice, we would present distinct solutions to you, not the same solution times.

In practice, researchers often generate samples per problem but report pass@k for various values (typically ). When is large relative to , we can approximate the binomial coefficient ratio. We can expand the ratio of binomial coefficients step-by-step:

When , each term , so the product becomes . This yields the approximation:

where:

• : the empirical probability that any single sample is correct (the maximum likelihood estimate of the true pass rate)
• : the probability that all samples are incorrect, assuming independence

This approximation treats the sampling as approximately independent (with replacement), which is valid when is large and is small relative to . The approximation is useful because it avoids computing large binomial coefficients and gives intuitive insight: the probability of failure decays exponentially with when the base success probability is fixed.

The final pass@k score is the mean pass@k across all problems in the benchmark, giving a single number that characterizes model performance:

where:

• : the number of problems in HumanEval ( )
• : the pass@k score for the -th problem

This macro-averaging approach gives equal weight to each problem, regardless of difficulty. An alternative would be to pool all samples across problems, but averaging per-problem scores prevents easy problems from dominating the metric and gives progress on hard problems equal weight. The choice of macro-averaging has an important implication: a model that perfectly solves 82 problems and fails all remaining 82 reaches the same pass@1 as a model that has a 50% success rate on every problem, even though these stand for qualitatively different capability profiles.

One of the most practically important aspects of the pass@k evaluation is the interaction between temperature and performance. Temperature controls the randomness of the language model's sampling distribution: at temperature 0, the model always produces its single most likely next token (greedy decoding), creating deterministic output. At higher temperatures, the distribution becomes more uniform, increasing diversity at the potential cost of individual sample quality.

For pass@1, lower temperatures typically perform better. Greedy decoding or low-temperature sampling concentrates probability mass on the most likely sequence, which for well-trained models tends to be the correct solution when one exists. For pass@k with larger , higher temperatures often improve performance. The intuition is that you want diverse samples: if the model generates the same wrong answer 100 times at temperature 0, pass@100 equals pass@1. Higher temperatures explore more of the probability space, increasing the chance that at least one sample is correct.

This tension creates a practical challenge: the optimal temperature for single-shot deployment differs from the optimal temperature for multi-sample evaluation. Research using the HumanEval benchmark found that models often benefit from separate temperature settings depending on the deployment context. Production coding assistants like GitHub Copilot typically operate at low temperatures to maximize pass@1 for immediate suggestions, while automated code repair systems can afford higher temperatures to explore more diverse solutions.

The original Codex paper used temperature 0.8 for sampling, which was found to give a good balance between diversity (needed for high estimates) and quality (needed for reasonable base rates). Subsequent work explored temperature annealing: starting with a high temperature for the first few samples to explore diverse approaches, then reducing temperature as the number of remaining samples decreases. This strategy is analogous to simulated annealing in optimization and can improve pass@k compared to fixed-temperature sampling.

Let us walk through a concrete example to solidify the calculation. Suppose for a specific problem, we generate samples and find that of them pass all tests.

To calculate pass@1:

This matches our intuition: with 2 correct samples out of 5, the probability of randomly selecting a correct one is .

To calculate pass@2:

There is a 70% chance that at least one of two randomly selected samples is correct. This is substantially higher than the 40% single-sample probability. This shows the benefit of multiple attempts.

For pass@3:

With only 3 incorrect samples available, selecting 3 samples without replacement has only a 10% chance of getting all incorrect ones. This leaves a 90% chance of getting at least one correct solution. As approaches , the probability approaches 1. This shows the fact that with enough attempts, we will eventually examine all correct solutions in the pool.

Notice what happens at the boundary. When , we must select all 5 samples, so we are guaranteed to include the 2 correct ones. Pass@5 equals exactly 1.0. This behavior is automatically handled by the formula: since , so the ratio is 0 and pass@5 = 1. When , all samples are incorrect, so no matter how many we select, none will be correct. Pass@k = 0 for all when . These boundary cases give useful sanity checks when implementing the metric.

Let us implement the pass@k calculation and examine the HumanEval dataset structure. We will use the math library for exact combinatorial calculations and implement the metric from scratch to ensure understanding.

Simulating a Multi-Problem Benchmark EvaluationLink Copied

Aggregating Across the BenchmarkLink Copied

Several design choices materially affect the numbers you report when evaluating against HumanEval. Understanding these choices is needed for correctly interpreting results and making fair comparisons across papers.

The key parameters are:

• n (samples per problem): Total number of samples generated per problem. This determines the precision of the estimate and must be greater than or equal to k. In research settings, n is typically set to 200 to ensure stable estimates while keeping computational costs manageable. Setting n too low results in high variance estimates, particularly for hard problems where c is small.
• c (correct count): Number of correct samples that pass all test cases. This is the observed count of successful solutions from the n samples, determined by executing each generated sample against the test harness and counting successes.
• k (evaluation budget): Number of samples selected for evaluation, representing the number of attempts allowed to solve the problem. Common values are 1 (single-shot accuracy), 10 (moderate search budget), and 100 (extensive search).
• Temperature: The sampling temperature used to generate each of the n samples. It directly affects the diversity and quality tradeoff described above.

Understanding the relationship between these parameters is important for interpreting results. A model might show modest pass@1 but excellent pass@100, showing high diversity in generation but inconsistent single-sample quality. Conversely, high pass@1 but low relative improvement at higher k suggests the model produces similar, high-quality outputs consistently. These profiles correspond to different practical use cases: the high-diversity model suits automated selection systems, while the high-precision model suits interactive assistants where users see only one suggestion.

A common source of confusion in the literature involves the distinction between pass@1 as measured with temperature sampling versus greedy decoding. Some papers report pass@1 using temperature 0 (greedy), which measures the model's most confident prediction. Others use temperature 0.8 and the formula above with , . These can differ substantially. Greedy decoding generally yields higher pass@1 for well-calibrated models, since the mode of the distribution is typically the most likely correct answer. The formula-based estimate from temperature sampling can be lower because high temperature introduces more noise. When comparing results across papers, you must check which method each paper uses.

The original HumanEval test cases were designed to be correct but not exhaustive. A natural question arose: how many solutions that pass HumanEval's tests are wrong on a more complete suite? In 2023, researchers at the University of Illinois developed HumanEval+, which augmented each HumanEval problem with up to 80 times more test cases generated using the EvoEval mutation-based approach. The results were sobering.

Across a range of state-of-the-art models, HumanEval+ scores consistently fell below the original HumanEval scores by a real margin. Models that appeared to solve 80% of problems correctly on the original benchmark turned out to solve only 70-75% correctly on the expanded test suite. This gap varied by model: some models exhibited highly reliable solutions that passed the expanded tests, while others relied on solutions that happened to satisfy the sparse original tests but failed on edge cases that were not directly tested.

This finding has two practical consequences. First, HumanEval scores from 2021 and 2022 should be interpreted with some skepticism: part of the apparent progress may reflect models learning patterns that satisfy sparse tests rather than generating correct solutions across edge cases. Second, for evaluation purposes, HumanEval+ is a more rigorous standard, and recent papers increasingly report results on both benchmarks.

The test augmentation methodology used in HumanEval+ also introduced a new evaluation paradigm: instead of hand-crafting every test case, generate them programmatically using mutation testing or property-based testing techniques. A mutation involves making small, systematic changes to the canonical solution (swapping < for <=, negating a condition, changing an off-by-one index) and checking whether the model's solution correctly handles the mutated inputs. This automated approach to test generation scales more cheaply than human curation and can cover edge cases that human test writers might overlook.

One of the most significant concerns about HumanEval performance claims is data contamination. The HumanEval problems were designed to be novel in 2021, but as the benchmark grew in prominence, solutions appeared in blog posts, tutorials, GitHub repositories, and Stack Overflow discussions. Models trained on internet data crawled after HumanEval's release may have been exposed to the exact problems and solutions during pre-training or fine-tuning. This exposure creates an optimistic bias: the model may perform well not because it can reason about novel code generation tasks, but because it has memorized the solutions.

Detecting contamination is a methodological challenge. You cannot simply check whether the training data contains HumanEval problems verbatim: paraphrasing, reformatting, or presenting the solution as part of a tutorial discussion can expose a model to the answer without creating an exact string match. Some researchers have examined contamination by checking whether model-generated solutions resemble canonical solutions more closely than independent implementations would, but this is imperfect because elegant solutions to simple problems naturally look similar regardless of whether the model saw the reference.

A practical response to contamination concerns has been the development of private evaluation sets held by organizations and not released publicly. The MultiPL-E benchmark, for instance, extends HumanEval to 19 programming languages: since it was created after HumanEval's release, fewer solutions exist publicly in each language. This gives somewhat more contamination-resistant measurements. The Mostly Basic Programming Problems (MBPP) benchmark was released before being made public as a test set. This keeps the test problems themselves had not circulated online.

The contamination problem has also motivated a shift toward measuring capability rather than just performance. Instead of asking only whether a model passes HumanEval tests, researchers now analyze the types of errors models make, the strategies models use when they fail, and whether models can explain their reasoning. A model that has memorized solutions would be expected to fail on simple problem variants that require adapting a known solution to a new constraint, whereas a model with real understanding should handle such variants gracefully.

As HumanEval became the standard benchmark for code generation, a progression of models demonstrated rapid improvement. Codex achieved 28.8% pass@1 in 2021. GPT-4 achieved over 67% pass@1 by 2023. Models specifically fine-tuned for code generation, such as Code Llama and DeepSeek Coder, pushed pass@1 above 80%. By 2024, models with larger context windows and instruction tuning were approaching 90% pass@1 on the original benchmark, representing near-saturation.

This rapid saturation illustrates both the progress of the field and the limitations of any single benchmark. When models approach ceiling performance, the benchmark loses discriminative power: differences between top models become statistically insignificant given HumanEval's 164-problem sample size. The standard error of a pass@1 estimate is roughly , which at equals about 2.3 percentage points. This means that two models with reported pass@1 scores of 87% and 90% are not statistically distinguishable on this benchmark alone without additional testing.

While HumanEval established the gold standard for code generation evaluation, it carries significant limitations that you must understand when using or interpreting results from this benchmark.

The most immediate limitation is the benchmark's small size. With only 164 problems, statistical noise is substantial. As shown above, the confidence intervals around pass@k estimates are wide, which makes it difficult to distinguish different capability levels from random variation. This problem grows more acute as models approach saturation, where small changes in the number of problems solved can produce large swings in the reported percentage. Researchers comparing closely ranked models should always account for this uncertainty rather than treating raw pass@k differences as real without statistical testing.

The benchmark's exclusive focus on Python and self-contained functions ignores the multilingual reality of software engineering and the importance of context. Real code generation tasks require understanding imports, class hierarchies, existing conventions, and surrounding code. A developer rarely writes functions from scratch in a vacuum: they fill in implementations that fit an existing codebase, adapt library examples to their context, or modify existing code to fix bugs. HumanEval captures none of this contextual complexity, which makes it a poor proxy for real-world coding assistant performance.

The docstring-to-code format assumes perfect, complete specifications. Production coding tasks often involve ambiguous requirements, underdefined edge cases, and requirements that evolve during implementation. A model evaluated on HumanEval always receives a complete, unambiguous docstring. The ability to ask clarifying questions, handle underspecified requirements, or iteratively refine solutions based on feedback is not captured. As coding assistants evolved toward chat-based interfaces, this limitation became more pronounced.

The functions in HumanEval are deliberately short, typically under 20 lines, and each is a self-contained algorithmic exercise. Real software engineering involves reasoning about hundreds or thousands of lines of code across multiple files, understanding architectural patterns, and making decisions that affect downstream components. The cognitive demands of real-world coding are qualitatively different from those of HumanEval's isolated algorithmic puzzles. Benchmarks that test longer, multi-file code generation stand for a materially harder and more realistic evaluation target.

Perhaps most critically, the public availability of HumanEval solutions on the internet creates a contamination risk: models trained on GitHub likely encountered these exact problems or similar solutions, potentially inflating scores without representing true reasoning capabilities. This concern motivated the development of private evaluation sets and techniques like MBPP that test broader programming knowledge, as well as research into benchmark contamination detection methods. Some researchers have responded by creating held-out test sets or using time-based splits to ensure models were trained only on code written before the benchmark's release.

Despite these limitations, HumanEval's impact on the field was large. It shifted the evaluation approach from syntactic similarity to functional correctness, establishing a template that subsequent benchmarks extended. The pass@k metric specifically acknowledges the stochastic nature of language models and the practical reality that developers generate multiple candidates before finding working solutions. This insight influenced evaluation and deployment strategies, popularizing techniques like generating multiple samples and using test cases to filter candidates. As the community develops more complete evaluations to address HumanEval's narrow scope, the benchmark remains the basic baseline against which new models are measured. This gives a shared point of reference that lets comparison across years of research.

The broader methodological contribution of HumanEval may be more important than the benchmark itself. It demonstrated that execution-based evaluation is feasible at scale, established the hypergeometric pass@k estimator as the standard for probabilistic code generation metrics, and shifted community expectations about what constitutes valid evidence of code generation capability. Every subsequent code generation benchmark, from MBPP and MultiPL-E to SWE-bench and HumanEval+, builds on the foundation that HumanEval established.

HumanEval changed code generation evaluation by measuring functional correctness rather than lexical similarity. The benchmark consists of 164 hand-crafted Python problems, each with test cases that unambiguously determine whether a generated solution works correctly. The pass@k metric captures the probability that at least one of samples passes all tests, calculated as:

where:

• : total samples generated per problem
• : count of correct samples (passing all tests)
• : number of samples selected for evaluation

This approach acknowledges that language models are probabilistic: even capable models may not generate correct code on the first attempt, but given multiple tries, their effective problem-solving rate increases substantially. The metric's formulation using the hypergeometric distribution properly accounts for sampling without replacement, while the approximation gives intuition about the exponential decay of failure probability with increased sampling budget.

Key points to carry forward:

• HumanEval evaluates functional correctness through code execution, not lexical similarity, representing a basic move from BLEU-based evaluation
• The pass@k formula uses the complement rule with the hypergeometric distribution; the approximation applies when
• Temperature affects the diversity-quality tradeoff: lower temperatures improve pass@1, higher temperatures improve pass@k at large k
• The benchmark's 164 problems give limited statistical power, especially in the saturation regime where top models cluster above 85%
• HumanEval+ revealed that many solutions declared correct by the original sparse tests fail on more complete test suites
• Data contamination from public solutions is a significant concern when comparing results across model generations

The benchmark's legacy lies in the specific numbers it produces, in establishing execution-based evaluation as the standard for code generation research, and in popularizing the pass@k framework that quantifies probabilistic correctness in stochastic generation systems.

QuizLink Copied