Crash report
I've been using eBPF with the static markers in Python 3.10 and wanted to try out 3.11 now that it is released.
But it seems that the function__entry and line markers are broken in 3.11.
The function__entry probe simply does not fire into the eBPF program.
The line probe crashes the interpreter.
I'm using BCC to load the eBPF program like this:
#!/usr/bin/python3
import argparse
from bcc import BPF, USDT
parser = argparse.ArgumentParser()
parser.add_argument("pid", type=int)
args = parser.parse_args()
program = """
int trace_entry(struct pt_regs *ctx) {
bpf_trace_printk("Entry:");
return 0;
}
int trace_return(struct pt_regs *ctx) {
bpf_trace_printk("Return:");
return 0;
}
int trace_line(struct pt_regs *ctx) {
bpf_trace_printk("Line:");
return 0;
}
"""
usdt = USDT(pid=args.pid)
usdt.enable_probe_or_bail("python:function__entry", 'trace_entry')
usdt.enable_probe_or_bail("python:function__return", 'trace_return')
usdt.enable_probe_or_bail("python:line", 'trace_line')
bpf = BPF(text=program, usdt_contexts=[usdt] if usdt else [], debug=0)
try:
bpf.trace_print()
except KeyboardInterrupt:
exit()
If I don't enable the line probe this is the output:
❯ sudo ./ebpf-test.py 80715
b' python-80715 [001] d...1 6539.145626: bpf_trace_printk: Return:'
The entry message is never printed.
Error messages
With the line probe enabled, this is the crash error:
❯ ./python
Python 3.11.0+ (heads/3.11:57dd11038f, Oct 31 2022, 10:30:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello world!")
python: Python/ceval.c:5620: _PyEval_EvalFrameDefault: Assertion `cframe.use_tracing' failed.
[1] 80398 IOT instruction (core dumped) ./python
Expected result
Running the same test in 3.10.6 I get this result, as expected:
❯ python3
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print("Hello world!")
Hello world!
❯ sudo ./ebpf-test.py 81962
b' python3-81962 [003] d...1 6907.857690: bpf_trace_printk: Entry:'
b' python3-81962 [003] d...1 6907.857710: bpf_trace_printk: Line:'
b' python3-81962 [003] d...1 6907.857765: bpf_trace_printk: Return:'
Your environment
I used the head of the 3.11 branch with ./configure --with-dtrace --with-pydebug
On Ubuntu 22.04.
Linked PRs
- GH-98894: Fix
function__returnandfunction__entrydTrace probe missing afterGH-103083#125019 - gh-98894: Update DTrace Docs in instrumentation.rst #139334
- gh-98894: Restore function entry/exit DTrace probes #142397
- gh-98894: Fix DTrace test_check_probes for shared builds #151122
- [3.15] gh-98894: Fix DTrace test_check_probes for shared builds (GH-151122) #151235
- gh-98894: Skip dtrace tests when building without dtrace #152239
- [3.15] gh-98894: Skip test_dtrace when building without dtrace (GH-152239) #152300
- [3.14] gh-98894: Skip test_dtrace when building without dtrace (GH-152239) #152301
- [3.13] gh-98894: Skip test_dtrace when building without dtrace (GH-152239) #152302
- gh-98894: Check readelf failures in test_dtrace #152345
- [3.15] gh-98894: Check readelf failures in test_dtrace (GH-152345) #152891
- gh-98894: Check tracer exit status in test_dtrace #152893
- [3.15] gh-98894: Check tracer exit status in test_dtrace (GH-152893) #152900
- gh-98894: Quote test_dtrace tracer subcommands #152901
- gh-98894: Fix dtrace tests in shared builds #153372
- [3.15] gh-98894: Quote test_dtrace tracer subcommands (GH-152901) #153459
- gh-98894: fix
CheckDtraceProbestests on Solaris #154803 - [3.15] gh-98894: Fix dtrace tests in shared builds (GH-153372) #155803