Created on 2022-01-04 23:09 by pablogsal, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Seems that the FreeBSD buildbots cannot compile Python 3.11 anymore:
LD_LIBRARY_PATH=/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build CC='cc -pthread' LDSHARED='cc -pthread -shared ' OPT='-g -O0 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
Traceback (most recent call last):
File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/./setup.py", line 49, in <module>
from distutils.command.build_ext import build_ext
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/distutils/command/build_ext.py", line 13, in <module>
from distutils.sysconfig import customize_compiler, get_python_version
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/distutils/sysconfig.py", line 53, in <module>
_config_vars = get_config_vars()
^^^^^^^^^^^^^^^^^
File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/sysconfig.py", line 621, in get_config_vars
_init_posix(_CONFIG_VARS)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/sysconfig.py", line 482, in _init_posix
_temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata_d_freebsd14_x86_64-unknown-freebsd14'
*** [sharedmods] Error code 1
make: stopped in /usr/home/buildbot/python/3.x.koobs-freebsd-564d/build
make: stopped in /usr/home/buildbot/python/3.x.koobs-freebsd-564d/build
https://buildbot.python.org/all/#/builders/483/builds/1408
There is also this other builder: https://buildbot.python.org/all/#/builders/172/builds/1093 failing what what seems is a freeze problem: - 'use_frozen_modules': 1, + 'use_frozen_modules': -1, ? + 'use_hash_seed': 0, 'user_site_directory': 1, 'verbose': 0, 'warn_default_encoding': 0, 'warnoptions': [], 'write_bytecode': 1, 'xoptions': []} ---------------------------------------------------------------------- Ran 1 test in 0.152s FAILED (failures=1) test test_embed failed 1 test failed again: test_embed Adding Eric and Guido for this particular problem to the nosy list
Not to point out the obvious but this buildbot uses --enable-shared which is often a source of build / bootstrap issues, such as the compiler executable being built ends up dynamically loading a libpython from a previous installation or build.
On my FreeBSD 13 VM the sysconfig data module is build/lib.freebsd-13.0-RELEASE-amd64-3.11-pydebug/_sysconfigdata_d_freebsd13_.py It does not have any multiarch suffix. The multiarch suffix is detected by configure.ac. Did FreeBSD 14.0's CC get support for "cc --print-multiarch"? I found a related FreeBSD bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258377
I have tested my PR on a FreeBSD VM. Python fails to compile on main and builds correctly with my patch.
New changeset cae55542d23e606dde9819d5dadd7430085fcc77 by Christian Heimes in branch 'main': bpo-46263: Don't use MULTIARCH on FreeBSD (#30410) https://github.com/python/cpython/commit/cae55542d23e606dde9819d5dadd7430085fcc77
New changeset 7e951f356ec76a5a5fdb851d71df5d120014bf3f by Miss Islington (bot) in branch '3.10': bpo-46263: Don't use MULTIARCH on FreeBSD (GH-30410) https://github.com/python/cpython/commit/7e951f356ec76a5a5fdb851d71df5d120014bf3f
New changeset 64199e9235275a795097ee0c53b2c560e21c70d0 by Miss Islington (bot) in branch '3.9': bpo-46263: Don't use MULTIARCH on FreeBSD (GH-30410) https://github.com/python/cpython/commit/64199e9235275a795097ee0c53b2c560e21c70d0
I have backported the patch to all versions that receive regular fixes.
I closed the bug without intention to do so. Sorry for the noise.
FWIW: from test_embed.test_init_setpythonhome:
if not config['executable']:
config['use_frozen_modules'] = -1
From the buildbot test stdout (but not pythoninfo) I see that config["executable"] is set to "".
Marked as a release blocker. See python-committers post: https://mail.python.org/archives/list/python-committers@python.org/thread/7OWGAL4UL5HNKKQFUGH6N6L4JDVPEN7R/
Unfortunately the buildbots are still red: https://buildbot.python.org/all/#/builders/483/builds/1416/steps/5/logs/stdio https://buildbot.python.org/all/#/builders/172/builds/1102 One is the test_embed problem with frozen modules: ====================================================================== FAIL: test_init_setpythonhome (test.test_embed.InitConfigTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib/test/test_embed.py", line 1253, in test_init_setpythonhome self.check_all_configs("test_init_setpythonhome", config, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib/test/test_embed.py", line 736, in check_all_configs self.check_config(configs, expected_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib/test/test_embed.py", line 662, in check_config self.assertEqual(config, expected) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: {'_co[346 chars]'], 'xoptions': [], 'warnoptions': [], 'python[1945 chars]': 1} != {'_co[346 chars]'], 'orig_argv': [], 'xoptions': [], 'warnopti[1946 chars]: -1} {'_config_init': 1, '_init_main': 1, '_install_importlib': 1, '_isolated_interpreter': 0, 'argv': [''], 'base_exec_prefix': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build', 'base_executable': '', 'base_prefix': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build', 'buffered_stdio': 1, 'bytes_warning': 0, 'check_hash_pycs_mode': 'default', 'code_debug_ranges': 1, 'configure_c_stdio': 0, 'dev_mode': 0, 'dump_refs': 0, 'exec_prefix': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build', 'executable': '', 'faulthandler': 0, 'hash_seed': 0, 'home': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build', 'import_time': 0, 'inspect': 0, 'install_signal_handlers': 1, 'interactive': 0, 'isolated': 0, 'malloc_stats': 0, 'module_search_paths': ['/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/target/lib/python311.zip', '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib', '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/build/lib.freebsd-12.3-RELEASE-amd64-3.11', '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/lib/python311.zip', '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/lib/python3.11', '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/lib/python3.11/lib-dynload'], 'module_search_paths_set': 1, 'optimization_level': 0, 'orig_argv': [], 'parse_argv': 0, 'parser_debug': 0, 'pathconfig_warnings': 1, 'platlibdir': 'lib', 'prefix': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build', 'program_name': 'python3', 'pycache_prefix': None, 'pythonpath_env': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/target/lib/python311.zip:/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib:/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/build/lib.freebsd-12.3-RELEASE-amd64-3.11', 'quiet': 0, 'run_command': None, 'run_filename': None, 'run_module': None, 'show_ref_count': 0, 'site_import': 1, 'skip_source_first_line': 0, 'stdlib_dir': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/lib/python3.11', 'tracemalloc': 0, 'use_environment': 1, - 'use_frozen_modules': 1, + 'use_frozen_modules': -1, ? + 'use_hash_seed': 0, 'user_site_directory': 1, 'verbose': 0, 'warn_default_encoding': 0, 'warnoptions': [], 'write_bytecode': 1, 'xoptions': []} ---------------------------------------------------------------------- Ran 1 test in 0.131s FAILED (failures=1) test test_embed failed 0:27:39 load avg: 1.11 Re-running test_logging in verbose mode (matching: test_udp_reconnection) test_udp_reconnection (test.test_logging.IPv6SysLogHandlerTest) ... ok test_udp_reconnection (test.test_logging.SysLogHandlerTest) ... ok test_udp_reconnection (test.test_logging.UnixSysLogHandlerTest) ... ok ---------------------------------------------------------------------- The other is a capi problem: ====================================================================== FAIL: test_pyobject_freed_is_freed (test.test_capi.PyMemMallocDebugTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_capi.py", line 943, in test_pyobject_freed_is_freed self.check_pyobject_is_freed('check_pyobject_freed_is_freed') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_capi.py", line 931, in check_pyobject_is_freed assert_python_ok('-c', code, PYTHONMALLOC=self.PYTHONMALLOC) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok return _assert_python(True, *args, **env_vars) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python res.fail(cmd_line) ^^^^^^^^^^^^^^^^^^ File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail raise AssertionError("Process return code is %d\n" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Process return code is 1 command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-c', '\nimport gc, os, sys, _testcapi\n# Disable the GC to avoid crash on GC collection\ngc.disable()\ntry:\n _testcapi.check_pyobject_freed_is_freed()\n # Exit immediately to avoid a crash while deallocating\n # the invalid object\n os._exit(0)\nexcept _testcapi.error:\n os._exit(1)\n'] stdout: --- --- stderr: --- ---
Here's a strange observation from my FreeBSD 14.0 VM: With system supplied python3.8 (dependency of the git package), test_embed succeeds. If I explicitly install python3.9 (pkg install python3.9 installs Python 3.9.9), rebuild and test, test_embed fails.
FYI, the buildbot has Python 3.9.9 installed, as seen from the configure step. (nit: it's pkg install python39, not pkg install python3.9)
Wait a minute. It is not tied to the Python version installed; it seems to be an issue with in-tree vs. out-of-tree builds. The failed build was in-tree. The build that succeeded was out-of-tree. I did a new build with Python 3.9 installed, which succeeded out-of-tree. To clear things up: - out-of-tree build with system Python 3.8 => succeeds - out-of-tree build with system Python 3.9 => succeeds - in-tree build with system Python 3.9 => fails
FTR: - in-tree build with system Python 3.8 => fails
The test_capi check for check_pyobject_freed_is_freed() is failing because FreeBSD 14.0 uses jemalloc, which performs its own dead memory cleaning. jemalloc's free() fills freed memory with byte 0x5a, which overrides Python's 0xdd marker. $ PYTHONMALLOC=malloc_debug gdb ./python (gdb) b check_pyobject_freed_is_freed (gdb) b _PyMem_DebugRawFree (gdb) disable 2 (gdb) run -X faulthandler t.py ... Breakpoint 1, check_pyobject_freed_is_freed (self=0x800f16fe0, _unused_args=0x0) at Modules/_testcapimodule.c:5032 5032 PyObject *op = PyObject_CallNoArgs((PyObject *)&PyBaseObject_Type); (gdb) enable 2 (gdb) c Continuing. Breakpoint 2, _PyMem_DebugRawFree (ctx=0x5f8ad8 <_PyMem_Debug+96>, p=0x800e1a0a0) at Objects/obmalloc.c:2544 2544 if (p == NULL) { (gdb) n 2549 uint8_t *q = (uint8_t *)p - 2*SST; /* address returned from malloc */ (gdb) n 2552 _PyMem_DebugCheckAddress(__func__, api->api_id, p); (gdb) n 2553 nbytes = read_size_t(q); (gdb) n 2554 nbytes += PYMEM_DEBUG_EXTRA_BYTES; (gdb) n 2555 memset(q, PYMEM_DEADBYTE, nbytes); (gdb) n 2556 api->alloc.free(api->alloc.ctx, q); (gdb) p *q@20 $18 = '\335' <repeats 19 times>, <incomplete sequence \335> (gdb) n _PyMem_RawFree (ctx=0x0, ptr=0x800e1a090) at Objects/obmalloc.c:127 127 { (gdb) n 128 free(ptr); (gdb) p free $19 = {void (void *)} 0x8006002c0 <free> (gdb) s __free (ptr=0x800e1a090) at jemalloc_jemalloc.c:2848 2848 jemalloc_jemalloc.c: No such file or directory. (gdb) n 2851 in jemalloc_jemalloc.c (gdb) 2852 in jemalloc_jemalloc.c ... (gdb) check_pyobject_freed_is_freed (self=0x800f16fe0, _unused_args=0x0) at Modules/_testcapimodule.c:5038 5038 Py_SET_REFCNT(op, 1); (gdb) p *(char*)op@20 $23 = 'Z' <repeats 20 times>
$ PYTHONMALLOC=malloc_debug ./python t.py
Traceback (most recent call last):
File "/usr/home/heimes/dev/python/cpython/t.py", line 4, in <module>
_testcapi.check_pyobject_freed_is_freed()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_testcapi.error: check_pyobject_freed_is_freed: object is not seen as freed
$ echo $?
1
$ MALLOC_CONF="junk:false" PYTHONMALLOC=malloc_debug ./python t.py
$ echo $?
0
New changeset a4aa52dc2801d25b6343fe2ef8de7f40ea3bc883 by Christian Heimes in branch 'main': bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434) https://github.com/python/cpython/commit/a4aa52dc2801d25b6343fe2ef8de7f40ea3bc883
New changeset b951dec4418326c06d493020dadedf85a3b29a82 by Miss Islington (bot) in branch '3.10': bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434) https://github.com/python/cpython/commit/b951dec4418326c06d493020dadedf85a3b29a82
New changeset b259015c1079744dbe3d58bd4c27757a6df9d1c6 by Miss Islington (bot) in branch '3.9': [3.9] bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434) (GH-30437) https://github.com/python/cpython/commit/b259015c1079744dbe3d58bd4c27757a6df9d1c6
Is this under control, or do you still need help with some part? (Are there two separate issues or not?)-- --Guido (mobile)
test_capi is fixed now. test_embed is still an issue.
In summary, regarding the test_embed failure: > FWIW: from test_embed.test_init_setpythonhome: > > if not config['executable']: > config['use_frozen_modules'] = -1 > > From the buildbot test stdout (but not pythoninfo) I see that > config["executable"] is set to "". ... > ====================================================================== > FAIL: test_init_setpythonhome (test.test_embed.InitConfigTests) > ---------------------------------------------------------------------- > ... > - 'use_frozen_modules': 1, > + 'use_frozen_modules': -1, > ? + > ... ... > - out-of-tree build with system Python 3.8 => succeeds > - out-of-tree build with system Python 3.9 => succeeds > - in-tree build with system Python 3.9 => fails > - in-tree build with system Python 3.8 => fails Note that in-tree builds are working find on other platforms, AFAICS.
> Is this under control, or do you still need help with some part? As Christian mentions, we still have the test_embed issue
> Note that in-tree builds are working find on other platforms, AFAICS. Yes, almost all of the buildbots are using in-tree builds. IIRC, buildbots for out-of-tree builds were added just recently.
AFAICS, the test_embed failure first appeared after GH-29041 (bpo-45582) was merged. After GH-29041, there has been numerous fixes for test_embed: $ git log --oneline | grep bpo-45582 3f398a77d3 bpo-45582: Fix test_embed failure during a PGO build on Windows (GH-30014) b0b3086279 bpo-45582: Write empty pybuilddir.txt on Windows to allow relocatable build directories (GH-29979) 06c4ae8b13 bpo-45582: Fix framework path and bootstrap build (GH-29954) b7ef27bc08 bpo-45582: Ensure PYTHONHOME still overrides detected build prefixes (GH-29948) f16f93e527 bpo-45582: framework build: modPath must not be const (GH-29944) af1db4eb55 bpo-45582: Fix getpath_isxfile() and test_embed on Windows (GH-29930) 628abe4463 bpo-45582: Fix signature of _Py_Get_Getpath_CodeObject (GH-29921) 0ae4e0c959 bpo-45582 Fix prototype of _Py_Get_Getpath_CodeObject. (GH-29907) 7d7c91a8e8 bpo-45582: Add a NOT operator to the condition in getpath_isxfile (GH-29906) ccb73a0d50 bpo-45582: Fix out-of-tree build issues with new getpath (GH-29902) 99fcf15052 bpo-45582: Port getpath[p].c to Python (GH-29041)
> the test_embed failure first appeared after GH-29041 (bpo-45582) was merged. Make that "first appeared on FreeBSD". Also, numerous was perhaps an exaggeration. Make that a few :)
Just to be clear, the test_embed failure is on AMD64 FreeBSD Non-Debug 3.x (https://buildbot.python.org/all/#/builders/172). I agree that 99fcf15052 (bpo-45582: Port getpath[p].c to Python (GH-29041)) is likely to be the cause.
IIUC Steve is still on vacation, so we shouldn't wait for him to fix this.
A FreeBSD builder where it's working: AMD64 FreeBSD Shared 3.x (koobs-freebsd-564d) OS: FreeBSD 14.0-CURRENT main-n244626-cb7cc72c546: GENERIC-NODEBUG (amd64) System: 2-core, 8Gb, VirtualBox (6.x) Guest Features: - Compiler: FreeBSD clang version 11.0.1 (llvmorg-11.0.1-0-g43ff75f2c3fe) - Clang "is" cc (default system compiler) - OpenSSL 1.1.1i-freebsd 8 Dec 2020 Where it isn't: AMD64 FreeBSD Non-Debug 3.x (koobs-freebsd-9e36) Environment: * OS: FreeBSD 12.3-RELEASE r371126 GENERIC amd64 * System: 2-core, 2Gb, VirtualBox (6.x) Guest Features: * cc = FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) * DTRACE * ZFS * Posix Semaphores The main difference I see is the FreeBSD version.
Also, AMD64 FreeBSD Shared 3.x is configured --with-pydebug and --enable-shared. I'd be surprised if those were the root cause of this though. Both buildbots are in-tree builds. FWIW, my FreeBSD 14.0-CURRENT main-n250453-7ac82c96fe7 VM also fails, as noted in previous posts. It's built with just ./configure && make.
I just put up a PR that should fix the failure.
New changeset 68c76d9766cccb5fd992b0ac4b39645d9665dbe2 by Eric Snow in branch 'main': bpo-46263: Do not ever expect "use_frozen_modules" to be -1. (gh-30438) https://github.com/python/cpython/commit/68c76d9766cccb5fd992b0ac4b39645d9665dbe2
Great. I tried the same trick on my VM earlier today, but I was unsure if it was the correct fix :) Thanks!
test_pyobject_freed_is_freed is still failing on AMD64 FreeBSD Shared 3.x.
New changeset c9137d4b638c0699b904011cafe68895d28dd80b by Christian Heimes in branch 'main': bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440) https://github.com/python/cpython/commit/c9137d4b638c0699b904011cafe68895d28dd80b
New changeset 3cb64ede57bba260228864150123014fc682167e by Miss Islington (bot) in branch '3.10': bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440) https://github.com/python/cpython/commit/3cb64ede57bba260228864150123014fc682167e
New changeset b98730c5165467c6053e7274ae094d733aea804d by Miss Islington (bot) in branch '3.9': bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440) https://github.com/python/cpython/commit/b98730c5165467c6053e7274ae094d733aea804d
Both FreeBSD bots are green again. Thank you Christian and Eric!
priority: release blocker ->
messages: + msg409924
resolution: fixed
stage: patch review -> resolved
pull_requests: + pull_request28647
messages:
+ msg409900
stage: patch review -> (no value)
messages: + msg409897
messages: + msg409885
pull_requests: + pull_request28641
messages: + msg409824
nosy:
+ erlendaasland
messages:
+ msg409768
resolution: fixed -> (no value)
type: compile error
messages: + msg409765
resolution: fixed
stage: patch review -> resolved
pull_requests: + pull_request28619
components: + Build
versions: + Python 3.9, Python 3.10
stage: patch review
pull_requests: + pull_request28616
messages: + msg409731
messages: + msg409727
messages: + msg409722