MBPP: Python Code Generation Benchmark

Article links

Make inline references clickable

When we evaluate whether a language model can write code, we need benchmarks that reflect real programming tasks. While HumanEval gives us carefully hand-crafted problems written by experts, it covers only a narrow range of programming tasks. We need larger, more diverse collections to understand how models generalize across different problem types, difficulty levels, and coding styles. Expert-crafted benchmarks reveal one dimension of capability; crowd-sourced benchmarks reveal another. Each illuminates failure modes the other obscures.

MBPP (Mostly Basic Python Programming) fills this gap. Created by researchers at Google and published in 2021, MBPP contains nearly 1,000 Python programming problems sourced from a crowd of developers rather than curated by a small group of experts. This move from expert craftsmanship to crowd-sourced diversity reveals different failure modes in code-generating models and gives a complementary view of programming capabilities. Where expert-written benchmarks might reflect idealized or algorithmically advanced scenarios, crowd-sourced data captures the messy, practical reality of day-to-day software development: string manipulations, date arithmetic, file parsing, and data structure transformations that comprise the bulk of what working programmers write.

The creation of MBPP came at a point where practice changed. By 2021, language models had shown impressive ability on HumanEval's structured algorithmic problems, yet practitioners remained skeptical about whether these capabilities would transfer to real programming assistance. If a developer types a comment describing a function they want and presses a completion shortcut, the model receives an informal, conversational specification, not a carefully structured docstring with type hints. MBPP was designed precisely to test this more realistic scenario, bridging the gap between laboratory benchmarks and deployed coding assistants.

In this chapter, we will explore the structure of the MBPP dataset, examine how it differs from HumanEval in both construction and evaluation philosophy, work through the mathematics of pass@k evaluation in detail, and confront the unique challenges of execution-based code evaluation. Understanding these distinctions matters because the choice of benchmark fundamentally shapes what we optimize for when training code-generating models. If we train exclusively on expert-curated algorithmic puzzles, we might produce models that excel at interview questions but struggle with simple utility scripts. Conversely, overfitting to basic crowd-sourced tasks might leave models unprepared for complex architectural decisions. The full picture of programming competence requires both kinds of evaluation.

MBPP emerged from a recognition that expert-written benchmarks, while high-quality, might not capture the full distribution of programming tasks that practitioners encounter daily. The researchers crowdsourced programming problems from a pool of Python developers at varying skill levels, creating a dataset that shows how everyday programmers describe and solve tasks. The collection process asked contributors to write programming problems they had personally encountered or imagined, then give reference solutions and test cases. This bottom-up approach produced a dataset with a qualitatively different character than HumanEval.

The philosophical move from expert curation to crowd-sourcing is a bet on ecological validity. By collecting problems the way they naturally occur in development environments, MBPP aims to measure whether models can handle the irregular, often ambiguous specifications that working developers give. This diversity comes with trade-offs. While expert benchmarks ensure consistent quality and complete edge-case coverage, crowd-sourced data introduces variability in problem difficulty, description clarity, and test thoroughness. MBPP embraces this variability as a feature rather than a bug, arguing that reliable code generation requires handling imperfect specifications and incomplete requirements.

Consider the perspective of a developer using a code generation tool. They are rarely in the mindset of constructing a carefully defined algorithmic problem. They need help with a specific, immediate task: "write a function to strip HTML tags from this string" or "create a function that groups these objects by date." These informal, task-oriented descriptions are exactly what MBPP captures. A model that performs well on MBPP has learned to interpret human intent even when the specification leaves details unstated, infer reasonable design choices when the prompt is ambiguous, and generate code that solves the practical problem rather than merely satisfying formal constraints.

The full MBPP dataset contains 974 programming problems divided into subsets based on quality and verification levels. Understanding these splits matters for interpreting benchmark results, as different papers report scores on different subsets and direct comparisons require awareness of which split was used.

The three primary divisions are:

• Sanitized subset (427 problems): Problems that human reviewers manually verified to have correct solutions and reasonably complete test cases. This is the most commonly reported split for benchmarking newer models. The smaller size shows the labor-intensive verification process.
• Crowd-sourced subset (547 problems): Problems as originally submitted, without manual verification. These may contain ambiguous specifications, subtly incorrect reference solutions, or test cases that do not fully capture the intended functionality.
• Augmented versions: Later releases, such as MBPP+ from the EvalPlus project, include additional test cases generated to improve coverage and reduce the fraction of "false positive" evaluations where models pass insufficient tests.

Reporting practices in the literature vary. Some papers report pass@1 on the full 974-problem set, while others use only the 427-problem sanitized split. When comparing scores across papers, you should always verify which split was used, as scores on the sanitized subset tend to be slightly lower due to more rigorous test cases.

Each problem in MBPP follows a consistent structure that makes automated evaluation straightforward:

• Task ID: A unique integer identifier for the problem, letting reproducible reporting of per-problem results.
• Text: The natural language description of the programming task, serving as the prompt given to the model. This is typically one to three sentences in a conversational, informal style.
• Code: A reference solution implementing the described functionality. This is ground truth but is not shown to the model during evaluation.
• Test Cases: A list of Python assertion statements that verify the correctness of any proposed solution. These are the executable unit tests that determine pass or fail.
• Test Setup: Any necessary imports or initialization code that must run before the test assertions. This allows the test cases to import the function being tested.

Keeping the prompt (what the model sees), the reference solution (what a correct answer looks like), and the test cases (how correctness is verified) separate makes MBPP easy to integrate into evaluation pipelines. The test cases are particularly important: they are the actual mechanism of evaluation, not the reference solution. A model's generated code is run against the test assertions, and the reference solution is used only for comparison purposes, not for grading.

Unlike HumanEval, which focuses on algorithmic interview-style problems, MBPP spans a broader range of programming tasks with generally lower algorithmic complexity. The "Mostly Basic" moniker is not false modesty: the dataset stresses basic programming competencies rather than competitive programming puzzles. This focus aligns with the reality that professional programming involves substantial amounts of routine data processing, format conversion, and utility function implementation alongside occasional complex algorithmic work.

The problems in MBPP exhibit several characteristics that distinguish them from other benchmarks:

• Conversational descriptions: Problems are often phrased informally, resembling how you might ask a programming assistant for help rather than how you would write a formal specification.
• Standard library focus: Solutions typically rely on Python's standard library, particularly string methods, list operations, the datetime module, and basic data structure operations. This keeps the problems self-contained without requiring knowledge of specialized packages.
• Single-function scope: Most problems ask for a single function implementation, though some require brief scripts. This narrowly defined scope makes evaluation clear but may underrepresent real-world tasks that span multiple functions.
• Self-contained nature: Each problem includes all necessary context without external dependencies, making batch evaluation straightforward.
• Breadth over depth: MBPP problems rarely require deep algorithmic insight, but they cover a wide range of Python programming scenarios, from string manipulation and list sorting to simple number theory and basic data structure operations.

The typical MBPP problem asks for something like: "Write a function to find the second largest element in a list," "Write a function to check if a string is a palindrome ignoring whitespace," or "Write a function to convert a decimal number to binary representation." These are the kinds of tasks that appear in tutorials, coding exercises for beginners, and everyday utility scripting. They are not trivial, but they do not demand the advanced reasoning that HumanEval's algorithmic problems require.

This scope makes MBPP particularly useful for evaluating instruction-tuned models designed to serve as coding assistants. When you ask an AI to "write a function to convert this CSV format," you are not giving formal specifications or function signatures. You are giving conversational, often incomplete descriptions. MBPP tests a model's ability to bridge the gap between informal human intent and executable machine instructions, a capability that matters more for deployed systems than the ability to solve algorithmic puzzles under pressure.

The MBPP problems cluster into several informal categories that reveal the breadth of everyday Python programming. While the dataset does not come with official category labels, researchers have analyzed the distribution and identified recurring patterns. The major categories include the following types.

String operations and text processing constitute a large portion of the dataset. Problems ask for functions to reverse words, count vowels, extract substrings matching certain patterns, format text according to rules, and manipulate strings in various ways. These problems test whether models understand Python's string API and can combine multiple operations to reach a desired transformation.

Mathematical and numerical functions form another substantial category. Problems involve computing properties of numbers (checking primality, finding factors, computing digit sums), performing geometric calculations (area, perimeter, distance), and implementing simple numerical algorithms (binary search, basic sorting variants). These problems require translating mathematical descriptions into executable code without necessarily requiring deep algorithmic insight.

List and collection operations appear frequently, covering sorting with custom keys, filtering based on predicates, aggregating values, finding extremes with various conditions, and transforming between different collection types. Python's rich built-in collection operations make many of these problems amenable to concise solutions, testing whether models know the standard library well.

Date and time operations stand for a practically important category that is often surprisingly tricky. Converting between date formats, computing differences between dates, finding the day of week for a given date, and performing calendar arithmetic require understanding Python's datetime module and its interface.

Basic algorithmic problems round out the set, including simple dynamic programming tasks, graph traversals on small inputs, and recursive computations. These overlap with HumanEval in spirit but tend to be simpler in scope and described in less formal terms.

To understand what MBPP contributes to code evaluation, we must compare it directly against HumanEval. These two benchmarks stand for different philosophies in dataset construction: expert curation versus crowd-sourced diversity.

HumanEval is careful, adversarial test design, with each problem crafted to probe specific reasoning capabilities and edge cases that trip up naive implementations. MBPP is the wisdom of the crowd, capturing the statistical distribution of tasks that working programmers encounter. The tension between these approaches mirrors broader debates in artificial intelligence evaluation. Should we test models against carefully designed adversarial examples that reveal the limits of their reasoning, or against naturalistic samples that reveal their utility in real-world deployment? The answer, of course, is both. HumanEval excels at identifying whether models understand algorithmic concepts deeply enough to handle edge cases and boundary conditions. MBPP excels at determining whether models can interpret ambiguous natural language and generate useful code for routine tasks.

HumanEval consists of 164 hand-written problems created by OpenAI researchers. Each problem includes a function signature, docstring, and solution. The test cases are written by experts to be complete and challenging. The authors of HumanEval specifically designed problems to be difficult for existing models at the time of publication, focusing on algorithmic reasoning, mathematical insight, and careful handling of data structures. This adversarial design philosophy means that passing HumanEval tests requires real correctness, not just creating superficially reasonable code.

MBPP, by contrast, crowdsourced problems from a larger pool of contributors. This difference in construction leads to distinct characteristics across multiple dimensions:

Prompt Structure DifferencesLink Copied

This format tells the model the exact function name, the parameter name and implied type, and through the docstring the precise behavioral specification including the tie-breaking rule. The model's only job is to implement the body.

MBPP gives more conversational, free-form descriptions without any structural hints:

Write a function to sort a list of non-negative integers based on the count of 1s in their binary representation.

The model must now decide what to name the function, what to name the parameter, whether to include type hints, how to handle ties, and what to return. This tests a different skill: understanding informal requirements and making reasonable design decisions without explicit guidance.

This difference has measurable implications for evaluation. Models must parse natural language intent from MBPP prompts without the structural hints provided by function signatures. In HumanEval, the model knows exactly what the function should be named, what arguments it takes, and what the expected return type should be from the Python signature. In MBPP, the model must infer these details from context, decide on appropriate variable names, and determine the function signature based on the description of inputs and outputs.

This distinction matters materially for practical applications. When you use code generation tools in integrated development environments, you rarely give formal function signatures with complete docstrings. Instead, you write comments or descriptions like "need a function to parse these log files and extract error counts." MBPP's format better simulates this real-world usage pattern, testing models' ability to act as real programming assistants rather than just algorithmic puzzle solvers.

In practice, when running MBPP evaluation, researchers often prefix the natural language description with an instruction like "You are an expert Python programmer, and here is your task: {text} Your code should pass these tests: {test_list}." This prompting approach grounds the model in the evaluation context without giving the function signature that HumanEval supplies. The choice of prompt template can materially affect scores, which is why standardized evaluation setups matter for reproducible benchmarking.

HumanEval test cases are designed to be adversarial, checking edge cases and boundary conditions that would catch subtle bugs. MBPP test cases, being crowd-sourced, often focus on basic functionality verification. This difference shows the contrasting goals of the two benchmarks.

Consider how a function to find the second largest element in a list might be tested in each benchmark's style. A HumanEval-style test suite might include an empty list (should it return None or raise an exception?), a list with a single element, a list where all elements are equal, a list where the largest appears multiple times, a very large list to test performance, and negative numbers mixed with positive values. An MBPP-style test suite typically includes a standard case like [1, 3, 2, 4] where the answer is clearly 3, a few variations with different sizes and values, and possibly one edge case like a two-element list.

This does not make MBPP inferior. It measures different capabilities. MBPP evaluates whether models can generate broadly correct code for everyday tasks, while HumanEval tests robustness against edge cases. The adversarial testing in HumanEval serves an important purpose: it distinguishes between code that works on typical inputs and code that is truly correct under all specified conditions.

However, in many practical applications, code that handles 95% of cases correctly gives significant value, even if it fails on exotic edge cases. A developer using a code generation tool will typically run the generated code and notice if it crashes on their specific input, then fix the issue. MBPP captures this "good enough for common cases" quality that characterizes much of real-world software development, where perfect correctness is the goal but practical usefulness is the immediate criterion.

The consequence of this difference shows up in model rankings. Some models perform much better on MBPP relative to HumanEval, suggesting they have learned to generate plausible-looking code that satisfies basic tests but fails to handle edge cases. Other models show more consistent performance across both benchmarks, suggesting they have learned deeper programming correctness rather than surface-level pattern matching.

Both MBPP and HumanEval rely on execution-based evaluation rather than textual similarity metrics like BLEU or ROUGE. Traditional NLP metrics fail for code generation because syntactically different solutions can be semantically identical, while similar-looking code can behave completely differently.

The basic issue is that code is not text to be compared, but instructions to be executed. Two solutions might look completely different, one using list comprehensions and another using explicit loops, yet produce identical outputs for all inputs. Conversely, two code snippets might differ by only a single character, perhaps a less-than versus less-than-or-equal sign, yet produce drastically different behaviors. Execution-based evaluation cuts through this surface variation to measure the relevant behavior: does the code do what it is supposed to do?

This principle sounds obvious but imposes concrete requirements on evaluation infrastructure. Running generated code requires a test harness that can execute Python in a controlled environment, catch exceptions, enforce time limits, and aggregate results across potentially hundreds of candidates per problem. Building this infrastructure correctly involves solving problems in software engineering, security, and systems programming that have nothing to do with natural language processing.

The standard evaluation metric for MBPP is pass@k, which measures the probability that at least one out of generated samples passes all test cases. This metric acknowledges the stochastic nature of language model generation: even capable models might produce incorrect code on any single attempt due to sampling randomness. By evaluating multiple samples and asking whether at least one succeeds, pass@k gives a more complete picture of model capability.

The intuition behind pass@k is straightforward. If you can only see one output from the model (the case ), pass@1 tells you the probability that this single output is correct. This is the most relevant metric for applications where you show users a single suggestion. If you can generate multiple candidates and use a reranker, an execution sandbox, or other selection mechanism to pick the best one (the case ), then pass@k for larger tells you the probability that at least one acceptable solution exists among the candidates.

To compute pass@k rigorously, we treat the selection of samples from total candidates as a hypergeometric sampling problem. Suppose we generate candidate solutions for a problem and find that of them are correct (pass all tests). We want the probability that a random selection of of these candidates contains at least one correct solution. The complement approach yields:

where:

• : the total number of samples generated per problem (typically 200 in the original paper), representing the complete pool of candidate solutions from which we draw
• : the number of correct samples, meaning those that pass all test assertions, representing the successful candidates within the pool
• : the number of samples we select for evaluation (typically 1, 10, or 100), representing how many attempts the model gets before we check for success
• : the number of ways to choose items exclusively from the incorrect candidates, which is all selections containing no correct solution
• : the total number of ways to choose any samples from all candidates
• : the expectation computed by averaging across all problems in the evaluation set

This formula works by computing the complement of the probability of failure. The fraction is the hypergeometric probability of drawing zero successes when selecting items from a population of items containing successes. Subtracting this from 1 yields the probability of obtaining at least one correct sample.

When is large relative to , the hypergeometric distribution approximates a binomial distribution. In this regime, each draw is approximately independent with success probability . The probability of failing all independent draws is then , giving:

where:

• : the empirical probability that any single randomly generated sample is correct, estimated from the pool of samples
• : the probability under independent Bernoulli sampling that all draws are incorrect

For the specific case of , this simplifies to:

which is simply the fraction of correct samples in the pool. This makes intuitive sense: if 20 out of 100 generated samples are correct, the probability that a randomly chosen single sample is correct is 20%.

For the case where the per-sample success probability is small, we can use the first-order Taylor expansion of the binomial to obtain an even simpler approximation. When :

This linear approximation holds when is small, giving the intuitive result that pass@k scales roughly linearly with both and the base success probability. However, this approximation breaks down as grows, since pass@k is bounded above by 1 but the linear approximation is not.

The key reason for using the exact combinatorial formula rather than the approximation is numerical stability and unbiasedness. The approximation introduces upward bias when is small relative to , precisely the regime where accurate estimation matters most for identifying the capability ceiling of weak models. The exact formula is unbiased regardless of the relationship between and .

Let us work through a concrete example to build intuition. Suppose we generate candidate solutions for a problem and find that of them pass all test cases. We want to compute pass@k for .

For :

With 4 correct solutions out of 20, we have a 20% chance that any single sample is correct. This equals exactly, confirming the simplified formula for .

For :

With 5 samples, we have a 71.8% chance of including at least one correct solution. The approximation gives , which is noticeably lower due to the finite population correction. Drawing without replacement makes it easier to get diverse samples, which increases the chance of hitting a correct one.

For :

With 10 samples drawn from our pool of 20 (which contains 4 correct solutions), we have a 95.7% chance of selecting at least one correct solution. The large improvement from 20% at to 95.7% at illustrates why the choice of matters so much for comparing models. A model that generates correct code 20% of the time at is not useless: if given 10 attempts and a way to identify the correct solution, it succeeds 96% of the time.

Calculating pass@k requires executing generated code against test cases in a controlled environment. This introduces several engineering challenges that do not exist in text-based evaluation.

Designing an MBPP evaluation run requires choosing values for several parameters that materially affect the results:

• (samples per problem): The total number of candidate solutions to generate. The original paper used , which gives stable estimates but requires substantial computation. Smaller values like or are often used when evaluating large models with expensive inference. Smaller increases variance in the estimates but is often acceptable for comparing models that differ by several percentage points.
• (correct count): This is determined empirically by running all samples against the test suite. It is not a hyperparameter but rather an observed quantity per problem.
• (samples to select): The published metric. Common choices are (single-shot performance), (performance with limited retries), and (upper bound on performance given many attempts). Most modern papers report pass@1 as the primary metric since it best shows real-world usage scenarios.
• Timeout: The maximum execution time per candidate. MBPP problems are computationally lightweight, so a 5-second timeout is usually sufficient. Longer timeouts increase evaluation time; shorter timeouts risk false negatives for correct solutions that happen to be computationally inefficient.
• Temperature: The sampling temperature used during generation. Higher temperatures produce more diverse samples, which can increase pass@k for larger but may decrease pass@1. Temperature 0 (greedy decoding) is often used for pass@1 evaluation.

Execution-based evaluation introduces complexities that purely textual benchmarks avoid. When we run generated code, we enter the domain of software engineering with all its attendant concerns about security, determinism, and resource management. Unlike evaluating text generation, where the worst-case scenario is creating nonsensical output, executing arbitrary code can actively harm the evaluation environment, consume excessive resources, or introduce security vulnerabilities. These concerns are not hypothetical: models trained on large code corpora have occasionally generated code that performs file system operations, initiates network connections, or triggers infinite recursion.

Executing arbitrary code generated by language models requires reliable isolation. Models might produce code that accesses the file system inappropriately, attempts network connections to exfiltrate data, consumes excessive memory through large data structure creation, contains infinite loops that never terminate, or exploits Python interpreter vulnerabilities to escape the sandbox.

MBPP evaluation requires sandboxed execution environments. Production-grade evaluation systems typically combine multiple layers of defense:

• Process isolation: Running untrusted code in a separate process that can be killed if it exceeds time limits. This gives basic protection against infinite loops and runaway computation.
• Containerization: Running the subprocess inside a Docker container with restricted capabilities, no network access, and a read-only file system except for a designated temporary directory. This prevents file system manipulation and network access.
• Resource limits: Enforcing memory caps (typically 256 MB to 1 GB), CPU time limits, and strict timeouts (typically 2 to 5 seconds per execution). These prevent denial-of-service scenarios where a single problem consumes all available resources.
• Restricted Python: Using environments like RestrictedPython that disable dangerous builtins, or running code in a custom Python interpreter that blocks imports of modules like os, subprocess, and socket.
• Virtual machine isolation: For the most stringent security requirements, running entire VMs per evaluation batch. This keeps even container escapes cannot affect the host system.

The sanitized subset of MBPP is particularly useful because it reduces (though does not eliminate) the risk of harmful patterns in the ground truth solutions. Even with sanitized reference solutions, generated model outputs remain potentially dangerous, requiring strict isolation between the evaluation harness and the execution environment.

Python code can behave non-deterministically due to random number generation, dictionary ordering in certain contexts, timing-dependent operations, and mutable global state. Non-determinism creates problems for reproducibility in research, since the same code might pass tests on one run but fail on another.

MBPP mitigates this by focusing on pure functions with deterministic outputs for given inputs. However, evaluating models that use randomization requires careful test design. Tests must either set random seeds explicitly before calling the function, allow for multiple valid outputs by testing properties rather than exact equality, or verify that the output belongs to a set of acceptable values.

Non-determinism presents a particular challenge for pass@k estimation. If a function's test result varies between runs, the estimate of (the number of correct samples) becomes unstable, introducing noise into the metric. The standard practice is to run each candidate exactly once against the test suite, accepting that any non-determinism in the test outcomes adds small amounts of noise to the final metric. For most MBPP problems, this is not a concern since the problems are defined to have deterministic correct answers.

One of the most contentious issues in code evaluation is how to handle partially correct solutions. A generated function might pass basic cases but fail edge cases, handle the main logic but miss error handling, return correct results but with incorrect types, implement the right algorithm with an off-by-one error, or solve a subtly different problem than the one specified.

MBPP, like HumanEval, uses a binary pass/fail criterion: either all assertions pass, or the solution is considered incorrect. This simplifies evaluation but loses granularity. A model that generates almost-correct code, failing only on one edge case out of ten tests, receives the same score as one that produces complete nonsense.

Alternative approaches have been proposed. Fractional scoring awards partial credit based on the percentage of test cases passed, so a function that passes 9 out of 10 tests receives a score of 0.9 instead of 0. Severity-based evaluation distinguishes between failure types: syntax errors (the code does not even parse), runtime errors (the code crashes), and logic errors (the code runs but produces wrong answers). Each type shows different levels of model capability. Property-based testing, rather than testing specific input-output pairs, verifies that the generated code satisfies high-level properties, such as "the output is always sorted" or "the output contains exactly the same elements as the input."

These alternatives add complexity and sometimes subjectivity. The binary approach remains standard because it correlates well with the practical utility of code: in production systems, a function that fails 10% of cases is often as unusable as one that fails 100% when those failures occur on real inputs. While a human developer might debug a nearly-correct solution more easily than starting from scratch, the benchmark setting does not model that iterative repair process.

The quality of MBPP evaluation depends entirely on the quality of the test cases. Crowd-sourced test cases vary materially in their coverage of the problem specification. Some problems have complete suites checking edge cases; others verify only the most basic functionality.

This variability affects model rankings in subtle ways. A model might reach a high MBPP score by generating "obvious" solutions that satisfy basic tests while failing edge cases that more thorough testing would catch. The sanitized subset partially addresses this by including only problems where reviewers confirmed that the test cases adequately cover the specification, but even expert-verified tests cannot prove absence of bugs for all possible inputs.

The coverage problem has motivated augmented versions of MBPP. The MBPP+ dataset from the EvalPlus project augmented the original test cases by generating additional inputs using a combination of LLM-generated tests and mutation testing, then having human experts verify them. When evaluated on MBPP+, models score lower than on the original MBPP, suggesting that many apparent successes on the original benchmark reflect weak test coverage rather than real code correctness. This gap between MBPP and MBPP+ performance is an important calibration tool: it quantifies how much a model's apparent capability depends on the leniency of the evaluation.

We examine a typical MBPP problem to understand the complete evaluation flow. Consider a problem asking for a function to calculate the number of days between two dates. This problem is representative of MBPP's practical, utility-function style: it asks for something you might need in a data processing script, uses a standard library module, and has a clear correct answer that is easy to verify.

The correct implementation uses strptime to parse date strings and computes the difference as a timedelta object, then takes the absolute value of its .days attribute. The buggy implementation only compares day numbers within the month.

Interestingly, the buggy solution passes all three tests in this example, because the tests only check cases within the same month where day subtraction happens to give the right answer. A case like days_between('2023-01-01', '2023-02-01') would expose the bug (returning 0 instead of 31), but that case is not in the test suite. This illustrates the coverage problem concretely: weak test suites allow incorrect solutions to appear correct, inflating evaluation scores.

We implement a complete pipeline for loading the MBPP dataset and running evaluation. We will use the HuggingFace datasets library, which hosts the sanitized MBPP split, and build a minimal evaluation harness that shows the key engineering patterns.

Safe Execution EnvironmentLink Copied

Computing pass@k Across a DatasetLink Copied

After evaluating all problems, we aggregate the per-problem metrics into dataset-level pass@k scores. The standard approach is to average pass@k across all problems:

Beyond the single pass@k number, understanding how evaluation metrics behave across different settings helps calibrate expectations and design better experiments.

While MBPP has become a standard benchmark for code generation, understanding its limitations is important for interpreting results correctly and improving future evaluation frameworks. No benchmark perfectly captures the full complexity of programming, and MBPP's specific design choices introduce biases that must be accounted for when assessing model capabilities.

MBPP evaluates only Python code. This shows the language's dominance in machine learning research. This creates a skewed picture of programming ability. Models that excel at MBPP might struggle with statically typed languages like TypeScript, Rust, or Go, memory management in C or C++, concurrent programming patterns with threads or async code, domain-specific languages for SQL or shell scripting, and low-level systems programming where undefined behavior and memory safety are central concerns.

Python's dynamic typing and garbage collection hide many classes of errors that plague systems programming languages. A model trained primarily on Python may develop intuitions about type safety and resource management that do not transfer to languages where these concerns are explicit. When we say a model "knows how to program," we really mean it knows how to write Python code that passes a specific set of tests, a narrower claim than the general statement implies.

Python's prominence in machine learning also creates a potential data contamination issue. Many models are pretrained on GitHub repositories, documentation sites, and code tutorials that may include the MBPP problems themselves or problems with very similar structure. When a model reaches high scores on MBPP, it is not always clear whether the performance shows real reasoning ability or pattern matching against training data. Decontamination analyses, which attempt to identify and remove training examples that overlap with evaluation sets, are standard practice in careful benchmark reporting but are computationally expensive and imperfect.

Like HumanEval, MBPP uses exact assertion matching for evaluation. This creates brittleness where semantically correct solutions fail due to type differences. If a test expects a list [1, 2, 3] but the function returns a tuple (1, 2, 3), the assertion == [1, 2, 3] fails despite the two collections containing identical elements. If a test expects a float 2.5 but the function returns an integer 2 due to integer division, the test fails despite representing the same mathematical quantity in many contexts.

More advanced evaluation might use structural or semantic equivalence checking, but these are computationally expensive and harder to standardize. Structural equivalence would require comparing abstract syntax trees or proving mathematical equivalence, while property-based testing would require specifying the properties that a correct solution must satisfy rather than specific input-output pairs. These approaches remain active research areas but have not yet been adopted as standard evaluation methodology.

Research has shown that models can reach surprisingly high scores on MBPP by exploiting weak test coverage rather than truly understanding the problem specification. The MBPP+ augmentation project quantified this gap rigorously. By generating hundreds of additional test cases per problem and having human experts verify them, the EvalPlus team found that model scores dropped materially when evaluated on the augmented test suite. A model achieving 70% pass@1 on the original MBPP might score only 55% on MBPP+, with the 15-percentage-point gap representing solutions that appeared correct on basic tests but contained bugs revealed by more thorough testing.

This finding changes the interpretation: benchmark scores should not be taken as absolute measures of capability, but as lower bounds subject to the quality of the test suite. The coverage problem also interacts with the partial correctness issue. Since MBPP uses binary pass/fail, a solution that passes 9 out of 10 tests receives the same score (0) as one that passes no tests. But when you compare models, a model that consistently generates nearly-correct solutions is qualitatively different from one that generates random code, even if their binary pass@1 scores are similar. Fractional test pass rates, reported alongside pass@k, give a more fine-grained view of this dimension.

Benchmarks have a shelf life. As models are trained specifically to perform well on established benchmarks, the benchmark scores stop which shows general capability and start which shows optimization toward the specific test distribution. This phenomenon is clearly visible in the history of MBPP scores.

When MBPP was introduced in 2021, the best models achieved pass@1 scores around 14-17% on the full dataset. By 2023-2024, state-of-the-art models were achieving 70-80%+ on the sanitized subset. Some of this improvement shows real capability gains from scaling and instruction tuning. But some shows the fact that models have been extensively trained on code, and the MBPP-style problems are now commonly seen in training data or in models fine-tuned specifically on MBPP-like distributions.

The field has responded by introducing harder benchmarks (LiveCodeBench, which uses problems from recent competitions to avoid contamination), problem-type diversification (BigCodeBench, which tests library usage patterns), and augmentation (MBPP+). These developments reflect the healthy cycle of benchmark-driven research: as models master one level of difficulty, the community raises the bar.

Despite these limitations, MBPP has materially influenced how we train and evaluate code-generating models. Its scale allowed researchers to study scaling behaviors in code generation, correlating model size with benchmark performance and identifying the dataset sizes needed for different performance levels. Its diversity of problem types encouraged models to develop broad Python competence rather than specializing narrowly. Its natural language prompt format pushed the field toward instruction-following models that can interpret informal requirements, aligning evaluation methodology with actual deployment scenarios.

MBPP's emphasis on crowd-sourced problems also democratized benchmark creation. Demonstrating that useful benchmarks could be constructed through relatively inexpensive crowd-sourcing rather than expensive expert curation opened the door to much larger evaluation sets. This insight influenced the construction of several subsequent benchmarks and established that dataset scale and variety could partially compensate for the reduced per-problem quality of crowd-sourced data.

The benchmark has also enabled the study of training data efficiency and few-shot learning in code generation. By giving a large, diverse set of problems with clear input-output format, MBPP facilitated research into how many examples a model needs to see during fine-tuning before achieving competence on the distribution, and how well few-shot prompting can substitute for fine-tuning when examples are limited.

Finally, MBPP's connection to reinforcement learning from execution feedback is significant. The pass/fail signal from test execution gives a natural reward signal for reinforcement learning: generate code, run the tests, use the binary outcome as the reward. This clean evaluation loop has powered several code generation improvements through execution-based reinforcement learning. Models trained with this approach learn to generate code that executes correctly rather than just code that looks correct to a language model evaluating textual similarity. The intersection of MBPP-style evaluation with reinforcement learning has become one of the most productive research directions in code generation, leading to systems that sharply outperform models trained solely on supervised next-token prediction.

MBPP complements HumanEval by giving a larger, more diverse set of programming problems drawn from crowd-sourced contributions rather than expert curation. This diversity reveals different aspects of code generation capability: handling informal natural language descriptions, managing varied problem types, and generating solutions for everyday programming tasks rather than algorithmic puzzles.

The benchmark introduced or popularized several important ideas in code evaluation. The pass@k metric, grounded in the exact hypergeometric formula, gives an unbiased estimator of model capability that accounts for the stochastic nature of language model generation. The emphasis on execution-based evaluation over textual similarity marked a shift toward measuring code behavior rather than appearance. The crowd-sourced construction approach demonstrated that useful benchmarks do not require expert-level curation for every problem, letting the creation of larger and more diverse evaluation sets.

Key takeaways from this chapter:

• Scale and diversity: With 974 problems total (427 sanitized), MBPP offers broader coverage than smaller benchmarks, though with variable test quality across the unsanitized portion.
• Evaluation methodology: pass@k uses the exact hypergeometric formula

to give an unbiased estimate of the probability that at least one of randomly selected candidates is correct. For large , the approximation is convenient but slightly biased.

• Prompt style: MBPP's conversational problem descriptions test different skills than HumanEval's structured function signatures. Models must infer function signatures, choose variable names, and resolve ambiguities from natural language alone.
• Execution challenges: Code evaluation requires sandboxed environments, timeout handling, and careful treatment of non-determinism, partial correctness, and test coverage variability. These engineering concerns are as important as the statistical methodology.
• Benchmark limitations: MBPP's Python monoculture, variable test coverage, and risk of data contamination limit the generalizability of its scores. MBPP+ addresses the coverage concern by augmenting test suites, and newer benchmarks address contamination by drawing from recent programming competitions.
• Impact on the field: MBPP's pass@k metric and execution-based evaluation approach became foundational for the broader code generation research community, letting reinforcement learning from execution feedback and systematic studies of scaling behavior in code models.

As code generation models continue to improve and MBPP scores approach saturation, the field will increasingly rely on harder, more diverse, and more contamination-resistant benchmarks. For now, MBPP remains a foundational tool for measuring programming competence, and understanding its construction, evaluation methodology, and limitations is needed background for anyone working with language models in software engineering.

QuizLink Copied

BIBTEXAcademic