← back

Code Execution Verifier for Programming Benchmarks

#324 · LLM · Medium

⊣ Solve on deep-ml.com

Problem

Implement a code execution verifier for programming benchmarks. Given a candidate code solution and a list of test cases (input/expected output pairs), execute the code safely and verify correctness.

Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
from typing import Dict, List, Tuple
import sys
import io

def code_execution_verifier(
    code: str,
    test_cases: List[Tuple[str, str]],
    function_name: str = "solution",
    timeout: float = 5.0
) -> Dict:
    results = []
    passed = 0

    try:
        namespace = {}
        exec(code, namespace)
    except Exception as e:
        return {
            "compile_error": str(e),
            "passed": 0,
            "total": len(test_cases),
            "results": []
        }

    func = namespace.get(function_name)
    if func is None:
        return {
            "compile_error": f"Function '{function_name}' not found",
            "passed": 0,
            "total": len(test_cases),
            "results": []
        }

    for inp, expected in test_cases:
        old_stdout = sys.stdout
        sys.stdout = io.StringIO()
        try:
            args = eval(inp)
            if not isinstance(args, tuple):
                args = (args,)
            result = func(*args)
            actual = repr(result)
            stdout_output = sys.stdout.getvalue()
        except Exception as e:
            actual = None
            stdout_output = ""
            results.append({
                "input": inp,
                "expected": expected,
                "actual": f"ERROR: {e}",
                "passed": False
            })
            continue
        finally:
            sys.stdout = old_stdout

        test_passed = str(result) == expected or actual == expected
        if test_passed:
            passed += 1
        results.append({
            "input": inp,
            "expected": expected,
            "actual": actual,
            "passed": test_passed
        })

    return {
        "passed": passed,
        "total": len(test_cases),
        "pass_rate": round(passed / len(test_cases), 4) if test_cases else 0,
        "results": results
    }

Explanation

  1. First compile and execute the code string to define the solution function in a namespace.
  2. If compilation fails or the function is not found, return an error immediately.
  3. For each test case, parse the input, call the function, capture the output.
  4. Compare the actual result against the expected output (string comparison).
  5. Return a summary with pass count, total, pass rate, and per-test results.

Complexity

  • Time: O(T * F) where T is the number of test cases and F is the runtime of each function call
  • Space: O(T) for storing results
← #323#325 →