diff --git a/Doc/howto/gdb_helpers.rst b/Doc/howto/gdb_helpers.rst new file mode 100644 index 00000000000000..53bbf7ddaa2ab9 --- /dev/null +++ b/Doc/howto/gdb_helpers.rst @@ -0,0 +1,449 @@ +.. _gdb: + +========================================================= +Debugging C API extensions and CPython Internals with GDB +========================================================= + +.. highlight:: none + +This document explains how the Python GDB extension, ``python-gdb.py``, can +be used with the GDB debugger to debug CPython extensions and the +CPython interpreter itself. + +When debugging low-level problems such as crashes or deadlocks, a low-level +debugger, such as GDB, is useful to diagnose and correct the issue. +By default, GDB (or any of its front-ends) doesn't support high-level +information specific to the CPython interpreter. + +The ``python-gdb.py`` extension adds CPython interpreter information to GDB. +The extension helps introspect the stack of currently executing Python functions. +Given a Python object represented by a :c:expr:`PyObject *` pointer, +the extension surfaces the type and value of the object. + +Developers who are working on CPython extensions or tinkering with parts +of CPython that are written in C can use this document to learn how to use the +``python-gdb.py`` extension with GDB. + +.. note:: + + This document assumes that you are familiar with the basics of GDB and the + CPython C API. It consolidates guidance from the + `devguide `_ and the + `Python wiki `_. + + +Prerequisites +============= + +You need to have: + +- GDB 7 or later. (For earlier versions of GDB, see ``Misc/gdbinit`` in the + sources of Python 3.11 or earlier.) +- GDB-compatible debugging information for Python and any extension you are + debugging. +- The ``python-gdb.py`` extension. + +The extension is built with Python, but might be distributed separately or +not at all. Below, we include tips for a few common systems as examples. +Note that even if the instructions match your system, they might be outdated. + + +Setup with Python built from source +----------------------------------- + +When you build CPython from source, debugging information should be available, +and the build should add a ``python-gdb.py`` file to the root directory of +your repository. + +To activate support, you must add the directory containing ``python-gdb.py`` +to GDB's "auto-load-safe-path". +If you haven't done this, recent versions of GDB will print out a warning +with instructions on how to do this. + +.. note:: + + If you do not see instructions for your version of GDB, put this in your + configuration file (``~/.gdbinit`` or ``~/.config/gdb/gdbinit``):: + + add-auto-load-safe-path /path/to/cpython + + You can also add multiple paths, separated by ``:``. + + +Setup for Python from a Linux distro +------------------------------------ + +Most Linux systems provide debug information for the system Python +in a package called ``python-debuginfo``, ``python-dbg`` or similar. +For example: + +- Fedora: + + .. code-block:: shell + + sudo dnf install gdb + sudo dnf debuginfo-install python3 + +- Ubuntu: + + .. code-block:: shell + + sudo apt install gdb python3-dbg + +On several recent Linux systems, GDB can download debugging symbols +automatically using *debuginfod*. +However, this will not install the ``python-gdb.py`` extension; +you generally do need to install the debug info package separately. + + +Using the Debug build and Development mode +========================================== + +For easier debugging, you might want to: + +- Use a :ref:`debug build ` of Python. (When building from source, + use ``configure --with-pydebug``. On Linux distros, install and run a package + like ``python-debug`` or ``python-dbg``, if available.) +- Use the runtime :ref:`development mode ` (``-X dev``). + +Both enable extra assertions and disable some optimizations. +Sometimes this hides the bug you are trying to find, but in most cases they +make the process easier. + + +Using the ``python-gdb`` extension +================================== + +When the extension is loaded, it provides two main features: +pretty printers for Python values, and additional commands. + +Pretty-printers +--------------- + +This is what a GDB backtrace looks like (truncated) when this extension is +enabled:: + + #0 0x000000000041a6b1 in PyObject_Malloc (nbytes=Cannot access memory at address 0x7fffff7fefe8 + ) at Objects/obmalloc.c:748 + #1 0x000000000041b7c0 in _PyObject_DebugMallocApi (id=111 'o', nbytes=24) at Objects/obmalloc.c:1445 + #2 0x000000000041b717 in _PyObject_DebugMalloc (nbytes=24) at Objects/obmalloc.c:1412 + #3 0x000000000044060a in _PyUnicode_New (length=11) at Objects/unicodeobject.c:346 + #4 0x00000000004466aa in PyUnicodeUCS2_DecodeUTF8Stateful (s=0x5c2b8d "__lltrace__", size=11, errors=0x0, consumed= + 0x0) at Objects/unicodeobject.c:2531 + #5 0x0000000000446647 in PyUnicodeUCS2_DecodeUTF8 (s=0x5c2b8d "__lltrace__", size=11, errors=0x0) + at Objects/unicodeobject.c:2495 + #6 0x0000000000440d1b in PyUnicodeUCS2_FromStringAndSize (u=0x5c2b8d "__lltrace__", size=11) + at Objects/unicodeobject.c:551 + #7 0x0000000000440d94 in PyUnicodeUCS2_FromString (u=0x5c2b8d "__lltrace__") at Objects/unicodeobject.c:569 + #8 0x0000000000584abd in PyDict_GetItemString (v= + {'Yuck': , '__builtins__': , '__file__': 'Lib/test/crashers/nasty_eq_vs_dict.py', '__package__': None, 'y': , 'dict': {0: 0, 1: 1, 2: 2, 3: 3}, '__cached__': None, '__name__': '__main__', 'z': , '__doc__': None}, key= + 0x5c2b8d "__lltrace__") at Objects/dictobject.c:2171 + +Notice how the dictionary argument to ``PyDict_GetItemString`` is displayed +as its ``repr()``, rather than an opaque ``PyObject *`` pointer. + +The extension works by supplying a custom printing routine for values of type +``PyObject *``. If you need to access lower-level details of an object, then +cast the value to a pointer of the appropriate type. For example:: + + (gdb) p globals + $1 = {'__builtins__': , '__name__': + '__main__', 'ctypes': , '__doc__': None, + '__package__': None} + + (gdb) p *(PyDictObject*)globals + $2 = {ob_refcnt = 3, ob_type = 0x3dbdf85820, ma_fill = 5, ma_used = 5, + ma_mask = 7, ma_table = 0x63d0f8, ma_lookup = 0x3dbdc7ea70 + , ma_smalltable = {{me_hash = 7065186196740147912, + me_key = '__builtins__', me_value = }, + {me_hash = -368181376027291943, me_key = '__name__', + me_value ='__main__'}, {me_hash = 0, me_key = 0x0, me_value = 0x0}, + {me_hash = 0, me_key = 0x0, me_value = 0x0}, + {me_hash = -9177857982131165996, me_key = 'ctypes', + me_value = }, + {me_hash = -8518757509529533123, me_key = '__doc__', me_value = None}, + {me_hash = 0, me_key = 0x0, me_value = 0x0}, { + me_hash = 6614918939584953775, me_key = '__package__', me_value = None}}} + +Note that the pretty-printers do not actually call ``repr()``. +For basic types, they try to match its result closely. + +An area that can be confusing is that the custom printer for some types look a +lot like GDB's built-in printer for standard types. For example, the +pretty-printer for a Python ``int`` (:c:expr:`PyLongObject *`) +gives a representation that is not distinguishable from one of a +regular machine-level integer:: + + (gdb) p some_machine_integer + $3 = 42 + + (gdb) p some_python_integer + $4 = 42 + +The internal structure can be revealed with a cast to :c:expr:`PyLongObject *`: + + (gdb) p *(PyLongObject*)some_python_integer + $5 = {ob_base = {ob_base = {ob_refcnt = 8, ob_type = 0x3dad39f5e0}, ob_size = 1}, + ob_digit = {42}} + +A similar confusion can arise with the ``str`` type, where the output looks a +lot like gdb's built-in printer for ``char *``:: + + (gdb) p ptr_to_python_str + $6 = '__builtins__' + +The pretty-printer for ``str`` instances defaults to using single-quotes (as +does Python's ``repr`` for strings) whereas the standard printer for ``char *`` +values uses double-quotes and contains a hexadecimal address:: + + (gdb) p ptr_to_char_star + $7 = 0x6d72c0 "hello world" + +Again, the implementation details can be revealed with a cast to +:c:expr:`PyUnicodeObject *`:: + + (gdb) p *(PyUnicodeObject*)$6 + $8 = {ob_base = {ob_refcnt = 33, ob_type = 0x3dad3a95a0}, length = 12, + str = 0x7ffff2128500, hash = 7065186196740147912, state = 1, defenc = 0x0} + +``py-list`` +----------- + + The extension adds a ``py-list`` command, which + lists the Python source code (if any) for the current frame in the selected + thread. The current line is marked with a ">":: + + (gdb) py-list + 901 if options.profile: + 902 options.profile = False + 903 profile_me() + 904 return + 905 + >906 u = UI() + 907 if not u.quit: + 908 try: + 909 gtk.main() + 910 except KeyboardInterrupt: + 911 # properly quit on a keyboard interrupt... + + Use ``py-list START`` to list at a different line number within the Python + source, and ``py-list START,END`` to list a specific range of lines within + the Python source. + +``py-up`` and ``py-down`` +------------------------- + + The ``py-up`` and ``py-down`` commands are analogous to GDB's regular ``up`` + and ``down`` commands, but try to move at the level of CPython frames, rather + than C frames. + + GDB is not always able to read the relevant frame information, depending on + the optimization level with which CPython was compiled. Internally, the + commands look for C frames that are executing the default frame evaluation + function (that is, the core bytecode interpreter loop within CPython) and + look up the value of the related ``PyFrameObject *``. + + They emit the frame number (at the C level) within the thread. + + For example:: + + (gdb) py-up + #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/ + gnome_sudoku/main.py, line 906, in start_game () + u = UI() + (gdb) py-up + #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/ + gnome_sudoku/gnome_sudoku.py, line 22, in start_game(main=) + main.start_game() + (gdb) py-up + Unable to find an older python frame + + so we're at the top of the Python stack. + + The frame numbers correspond to those displayed by GDB's standard + ``backtrace`` command. + The command skips C frames which are not executing Python code. + + Going back down:: + + (gdb) py-down + #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game () + u = UI() + (gdb) py-down + #34 (unable to read python frame information) + (gdb) py-down + #23 (unable to read python frame information) + (gdb) py-down + #19 (unable to read python frame information) + (gdb) py-down + #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': , 'gsd.hints': 0, 'timer.active_time': , 'timer.total_time': }], dialog=, saved_game_model=, sudoku_maker=, main_page=0) at remote 0x98fa6e4>, d=) + gtk.main() + (gdb) py-down + #8 (unable to read python frame information) + (gdb) py-down + Unable to find a newer python frame + + and we're at the bottom of the Python stack. + + Note that in Python 3.12 and newer, the same C stack frame can be used for + multiple Python stack frames. This means that ``py-up`` and ``py-down`` + may move multiple Python frames at once. For example:: + + (gdb) py-up + #6 Frame 0x7ffff7fb62b0, for file /tmp/rec.py, line 5, in recursive_function (n=0) + time.sleep(5) + #6 Frame 0x7ffff7fb6240, for file /tmp/rec.py, line 7, in recursive_function (n=1) + recursive_function(n-1) + #6 Frame 0x7ffff7fb61d0, for file /tmp/rec.py, line 7, in recursive_function (n=2) + recursive_function(n-1) + #6 Frame 0x7ffff7fb6160, for file /tmp/rec.py, line 7, in recursive_function (n=3) + recursive_function(n-1) + #6 Frame 0x7ffff7fb60f0, for file /tmp/rec.py, line 7, in recursive_function (n=4) + recursive_function(n-1) + #6 Frame 0x7ffff7fb6080, for file /tmp/rec.py, line 7, in recursive_function (n=5) + recursive_function(n-1) + #6 Frame 0x7ffff7fb6020, for file /tmp/rec.py, line 9, in () + recursive_function(5) + (gdb) py-up + Unable to find an older python frame + + +``py-bt`` +--------- + + The ``py-bt`` command attempts to display a Python-level backtrace of the + current thread. + + For example:: + + (gdb) py-bt + #8 (unable to read python frame information) + #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=, main_page=0) at remote 0x98fa6e4>, d=) + gtk.main() + #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': , 'gsd.hints': 0, 'timer.active_time': , 'timer.total_time': }], dialog=, saved_game_model=, sudoku_maker=) + main.start_game() + + The frame numbers correspond to those displayed by GDB's standard + ``backtrace`` command. + +``py-print`` +------------ + + The ``py-print`` command looks up a Python name and tries to print it. + It looks in locals within the current thread, then globals, then finally + builtins:: + + (gdb) py-print self + local 'self' = , + main_page=0) at remote 0x98fa6e4> + (gdb) py-print __name__ + global '__name__' = 'gnome_sudoku.dialog_swallower' + (gdb) py-print len + builtin 'len' = + (gdb) py-print scarlet_pimpernel + 'scarlet_pimpernel' not found + + If the current C frame corresponds to multiple Python frames, ``py-print`` + only considers the first one. + +``py-locals`` +------------- + + The ``py-locals`` command looks up all Python locals within the current + Python frame in the selected thread, and prints their representations:: + + (gdb) py-locals + self = , + main_page=0) at remote 0x98fa6e4> + d = + + If the current C frame corresponds to multiple Python frames, locals from + all of them will be shown:: + + (gdb) py-locals + Locals for recursive_function + n = 0 + Locals for recursive_function + n = 1 + Locals for recursive_function + n = 2 + Locals for recursive_function + n = 3 + Locals for recursive_function + n = 4 + Locals for recursive_function + n = 5 + Locals for + + +Use with GDB commands +===================== + +The extension commands complement GDB's built-in commands. +For example, you can use a frame numbers shown by ``py-bt`` with the ``frame`` +command to go a specific frame within the selected thread, like this:: + + (gdb) py-bt + (output snipped) + #68 Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in () + main() + (gdb) frame 68 + #68 0x00000000004cd1e6 in PyEval_EvalFrameEx (f=Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in (), throwflag=0) at Python/ceval.c:2665 + 2665 x = call_function(&sp, oparg); + (gdb) py-list + 1543 # Run the tests in a context manager that temporary changes the CWD to a + 1544 # temporary and writable directory. If it's not possible to create or + 1545 # change the CWD, the original CWD will be used. The original CWD is + 1546 # available from test_support.SAVEDCWD. + 1547 with test_support.temp_cwd(TESTCWD, quiet=True): + >1548 main() + +The ``info threads`` command will give you a list of the threads within the +process, and you can use the ``thread`` command to select a different one:: + + (gdb) info threads + 105 Thread 0x7fffefa18710 (LWP 10260) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86 + 104 Thread 0x7fffdf5fe710 (LWP 10259) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86 + * 1 Thread 0x7ffff7fe2700 (LWP 10145) 0x00000038e46d73e3 in select () at ../sysdeps/unix/syscall-template.S:82 + +You can use ``thread apply all COMMAND`` or (``t a a COMMAND`` for short) to run +a command on all threads. With ``py-bt``, this lets you see what every +thread is doing at the Python level:: + + (gdb) t a a py-bt + + Thread 105 (Thread 0x7fffefa18710 (LWP 10260)): + #5 Frame 0x7fffd00019d0, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140737213728528), count=1, owner=140737213728528) + self.__block.acquire() + #8 Frame 0x7fffac001640, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=, _RLock__count=1) at remote 0xd7ff40>, acquire=, _is_owned=, _release_save=, release=, _acquire_restore=, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=, saved_state=(1, 140737213728528)) + self._acquire_restore(saved_state) + #12 Frame 0x7fffb8001a10, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f () + cond.wait() + #16 Frame 0x7fffb8001c40, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140737213728528) + f() + + Thread 104 (Thread 0x7fffdf5fe710 (LWP 10259)): + #5 Frame 0x7fffe4001580, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140736940992272), count=1, owner=140736940992272) + self.__block.acquire() + #8 Frame 0x7fffc8002090, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=, _RLock__count=1) at remote 0xd7ff40>, acquire=, _is_owned=, _release_save=, release=, _acquire_restore=, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=, saved_state=(1, 140736940992272)) + self._acquire_restore(saved_state) + #12 Frame 0x7fffac001c90, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f () + cond.wait() + #16 Frame 0x7fffac0011c0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140736940992272) + f() + + Thread 1 (Thread 0x7ffff7fe2700 (LWP 10145)): + #5 Frame 0xcb5380, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 16, in _wait () + time.sleep(0.01) + #8 Frame 0x7fffd00024a0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 378, in _check_notify (self=, skipped=[], _mirrorOutput=False, testsRun=39, buffer=False, _original_stderr=, _stdout_buffer=, _stderr_buffer=, _moduleSetUpFailed=False, expectedFailures=[], errors=[], _previousTestClass=, unexpectedSuccesses=[], failures=[], shouldStop=False, failfast=False) at remote 0xc185a0>, _threads=(0,), _cleanups=[], _type_equality_funcs={: , : , : , : , `_ The NumPy package defines another array type. - diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index 73779547b35a1f..702955659a3bb9 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -200,7 +200,7 @@ calls). Python objects that can directly be used as parameters in these function calls. ``None`` is passed as a C ``NULL`` pointer, bytes objects and strings are passed as pointer to the memory block that contains their data (:c:expr:`char *` or -:c:expr:`wchar_t *`). Python integers are passed as the platforms default C +:c:expr:`wchar_t *`). Python integers are passed as the platform's default C :c:expr:`int` type, their value is masked to fit into the C type. Before we move on calling functions with other parameter types, we have to learn @@ -1445,7 +1445,7 @@ function exported by these libraries, and reacquired afterwards. All these classes can be instantiated by calling them with at least one argument, the pathname of the shared library. If you have an existing handle to an already loaded shared library, it can be passed as the ``handle`` named -parameter, otherwise the underlying platforms :c:func:`!dlopen` or +parameter, otherwise the underlying platform's :c:func:`!dlopen` or :c:func:`!LoadLibrary` function is used to load the library into the process, and to get a handle to it. @@ -1456,7 +1456,7 @@ configurable. The *use_errno* parameter, when set to true, enables a ctypes mechanism that allows accessing the system :data:`errno` error number in a safe way. -:mod:`ctypes` maintains a thread-local copy of the systems :data:`errno` +:mod:`ctypes` maintains a thread-local copy of the system's :data:`errno` variable; if you call foreign functions created with ``use_errno=True`` then the :data:`errno` value before the function call is swapped with the ctypes private copy, the same happens immediately after the function call. diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst index 935e872480efda..a6ae8fdaa4991c 100644 --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -214,7 +214,8 @@ XMLParser Objects :meth:`CharacterDataHandler` callback whenever possible. This can improve performance substantially since Expat normally breaks character data into chunks at every line ending. This attribute is false by default, and may be changed at - any time. + any time. Note that when it is false, data that does not contain newlines + may be chunked too. .. attribute:: xmlparser.buffer_used @@ -372,7 +373,10 @@ otherwise stated. marked content, and ignorable whitespace. Applications which must distinguish these cases can use the :attr:`StartCdataSectionHandler`, :attr:`EndCdataSectionHandler`, and :attr:`ElementDeclHandler` callbacks to - collect the required information. + collect the required information. Note that the character data may be + chunked even if it is short and so you may receive more than one call to + :meth:`CharacterDataHandler`. Set the :attr:`buffer_text` instance attribute + to ``True`` to avoid that. .. method:: xmlparser.UnparsedEntityDeclHandler(entityName, base, systemId, publicId, notationName) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index afeb6596fbb978..6438abbba10e37 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -34,7 +34,7 @@ represented by objects.) Every object has an identity, a type and a value. An object's *identity* never changes once it has been created; you may think of it as the object's address in -memory. The ':keyword:`is`' operator compares the identity of two objects; the +memory. The :keyword:`is` operator compares the identity of two objects; the :func:`id` function returns an integer representing its identity. .. impl-detail:: @@ -81,7 +81,7 @@ are still reachable. Note that the use of the implementation's tracing or debugging facilities may keep objects alive that would normally be collectable. Also note that catching -an exception with a ':keyword:`try`...\ :keyword:`except`' statement may keep +an exception with a :keyword:`try`...\ :keyword:`except` statement may keep objects alive. Some objects contain references to "external" resources such as open files or @@ -89,8 +89,8 @@ windows. It is understood that these resources are freed when the object is garbage-collected, but since garbage collection is not guaranteed to happen, such objects also provide an explicit way to release the external resource, usually a :meth:`!close` method. Programs are strongly recommended to explicitly -close such objects. The ':keyword:`try`...\ :keyword:`finally`' statement -and the ':keyword:`with`' statement provide convenient ways to do this. +close such objects. The :keyword:`try`...\ :keyword:`finally` statement +and the :keyword:`with` statement provide convenient ways to do this. .. index:: single: container diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index bfcf1c6882f29d..cc4db34b04d900 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -14,7 +14,7 @@ know about when using Python on Microsoft Windows. Unlike most Unix systems and services, Windows does not include a system supported installation of Python. To make Python available, the CPython team -has compiled Windows installers (MSI packages) with every `release +has compiled Windows installers with every `release `_ for many years. These installers are primarily intended to add a per-user installation of Python, with the core interpreter and library being used by a single user. The installer is also diff --git a/Doc/whatsnew/3.13.rst b/Doc/whatsnew/3.13.rst index 7c6a2af28758be..40823587fb9417 100644 --- a/Doc/whatsnew/3.13.rst +++ b/Doc/whatsnew/3.13.rst @@ -313,6 +313,14 @@ ipaddress * Add the :attr:`ipaddress.IPv4Address.ipv6_mapped` property, which returns the IPv4-mapped IPv6 address. (Contributed by Charles Machalow in :gh:`109466`.) +itertools +--------- + +* Added a ``strict`` option to :func:`itertools.batched`. + This raises a :exc:`ValueError` if the final batch is shorter + than the specified batch size. + (Contributed by Raymond Hettinger in :gh:`113202`.) + marshal ------- diff --git a/Lib/importlib/util.py b/Lib/importlib/util.py index 3ad71d31c2f438..ff4f12fb1af70c 100644 --- a/Lib/importlib/util.py +++ b/Lib/importlib/util.py @@ -13,6 +13,7 @@ import _imp import sys +import threading import types @@ -171,36 +172,54 @@ class _LazyModule(types.ModuleType): def __getattribute__(self, attr): """Trigger the load of the module and return the attribute.""" - # All module metadata must be garnered from __spec__ in order to avoid - # using mutated values. - # Stop triggering this method. - self.__class__ = types.ModuleType - # Get the original name to make sure no object substitution occurred - # in sys.modules. - original_name = self.__spec__.name - # Figure out exactly what attributes were mutated between the creation - # of the module and now. - attrs_then = self.__spec__.loader_state['__dict__'] - attrs_now = self.__dict__ - attrs_updated = {} - for key, value in attrs_now.items(): - # Code that set the attribute may have kept a reference to the - # assigned object, making identity more important than equality. - if key not in attrs_then: - attrs_updated[key] = value - elif id(attrs_now[key]) != id(attrs_then[key]): - attrs_updated[key] = value - self.__spec__.loader.exec_module(self) - # If exec_module() was used directly there is no guarantee the module - # object was put into sys.modules. - if original_name in sys.modules: - if id(self) != id(sys.modules[original_name]): - raise ValueError(f"module object for {original_name!r} " - "substituted in sys.modules during a lazy " - "load") - # Update after loading since that's what would happen in an eager - # loading situation. - self.__dict__.update(attrs_updated) + __spec__ = object.__getattribute__(self, '__spec__') + loader_state = __spec__.loader_state + with loader_state['lock']: + # Only the first thread to get the lock should trigger the load + # and reset the module's class. The rest can now getattr(). + if object.__getattribute__(self, '__class__') is _LazyModule: + # The first thread comes here multiple times as it descends the + # call stack. The first time, it sets is_loading and triggers + # exec_module(), which will access module.__dict__, module.__name__, + # and/or module.__spec__, reentering this method. These accesses + # need to be allowed to proceed without triggering the load again. + if loader_state['is_loading'] and attr.startswith('__') and attr.endswith('__'): + return object.__getattribute__(self, attr) + loader_state['is_loading'] = True + + __dict__ = object.__getattribute__(self, '__dict__') + + # All module metadata must be gathered from __spec__ in order to avoid + # using mutated values. + # Get the original name to make sure no object substitution occurred + # in sys.modules. + original_name = __spec__.name + # Figure out exactly what attributes were mutated between the creation + # of the module and now. + attrs_then = loader_state['__dict__'] + attrs_now = __dict__ + attrs_updated = {} + for key, value in attrs_now.items(): + # Code that set an attribute may have kept a reference to the + # assigned object, making identity more important than equality. + if key not in attrs_then: + attrs_updated[key] = value + elif id(attrs_now[key]) != id(attrs_then[key]): + attrs_updated[key] = value + __spec__.loader.exec_module(self) + # If exec_module() was used directly there is no guarantee the module + # object was put into sys.modules. + if original_name in sys.modules: + if id(self) != id(sys.modules[original_name]): + raise ValueError(f"module object for {original_name!r} " + "substituted in sys.modules during a lazy " + "load") + # Update after loading since that's what would happen in an eager + # loading situation. + __dict__.update(attrs_updated) + # Finally, stop triggering this method. + self.__class__ = types.ModuleType + return getattr(self, attr) def __delattr__(self, attr): @@ -244,5 +263,7 @@ def exec_module(self, module): loader_state = {} loader_state['__dict__'] = module.__dict__.copy() loader_state['__class__'] = module.__class__ + loader_state['lock'] = threading.RLock() + loader_state['is_loading'] = False module.__spec__.loader_state = loader_state module.__class__ = _LazyModule diff --git a/Lib/pathlib/_abc.py b/Lib/pathlib/_abc.py index 27c6b4e367a050..44fea525b6cac7 100644 --- a/Lib/pathlib/_abc.py +++ b/Lib/pathlib/_abc.py @@ -313,7 +313,14 @@ def with_name(self, name): def with_stem(self, stem): """Return a new path with the stem changed.""" - return self.with_name(stem + self.suffix) + suffix = self.suffix + if not suffix: + return self.with_name(stem) + elif not stem: + # If the suffix is non-empty, we can't make the stem empty. + raise ValueError(f"{self!r} has a non-empty suffix") + else: + return self.with_name(stem + suffix) def with_suffix(self, suffix): """Return a new path with the file suffix changed. If the path @@ -324,6 +331,7 @@ def with_suffix(self, suffix): if not suffix: return self.with_name(stem) elif not stem: + # If the stem is empty, we can't make the suffix non-empty. raise ValueError(f"{self!r} has an empty name") elif suffix.startswith('.') and len(suffix) > 1: return self.with_name(stem + suffix) diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py index a3804a945f2e3a..71bb1e75c6affd 100644 --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -1599,6 +1599,13 @@ def test_extend(self): a = bytearray(b'') a.extend([Indexable(ord('a'))]) self.assertEqual(a, b'a') + a = bytearray(b'abc') + self.assertRaisesRegex(TypeError, # Override for string. + "expected iterable of integers; got: 'str'", + a.extend, 'def') + self.assertRaisesRegex(TypeError, # But not for others. + "can't extend bytearray with float", + a.extend, 1.0) def test_remove(self): b = bytearray(b'hello') diff --git a/Lib/test/test_capi/test_opt.py b/Lib/test/test_capi/test_opt.py index 38c6fa4b47d0c9..25fc36dec93ddc 100644 --- a/Lib/test/test_capi/test_opt.py +++ b/Lib/test/test_capi/test_opt.py @@ -4,6 +4,7 @@ import textwrap import unittest import gc +import os import _testinternalcapi @@ -568,6 +569,8 @@ def testfunc(n): count = ops.count("_GUARD_IS_TRUE_POP") + ops.count("_GUARD_IS_FALSE_POP") self.assertLessEqual(count, 2) + +@unittest.skipIf(os.getenv("PYTHONUOPSOPTIMIZE", default=0) == 0, "Needs uop optimizer to run.") class TestUopsOptimization(unittest.TestCase): def _run_with_optimizer(self, testfunc, arg): diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index 5217f2a71ec0f9..d74ab7e016f78c 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -265,9 +265,11 @@ def test_write_lineterminator(self): writer = csv.writer(sio, lineterminator=lineterminator) writer.writerow(['a', 'b']) writer.writerow([1, 2]) + writer.writerow(['\r', '\n']) self.assertEqual(sio.getvalue(), f'a,b{lineterminator}' - f'1,2{lineterminator}') + f'1,2{lineterminator}' + f'"\r","\n"{lineterminator}') def test_write_iterable(self): self._write_test(iter(['a', 1, 'p,q']), 'a,1,"p,q"') @@ -507,22 +509,44 @@ def test_read_linenum(self): self.assertEqual(r.line_num, 3) def test_roundtrip_quoteed_newlines(self): - with TemporaryFile("w+", encoding="utf-8", newline='') as fileobj: - writer = csv.writer(fileobj) - rows = [['a\nb','b'],['c','x\r\nd']] - writer.writerows(rows) - fileobj.seek(0) - for i, row in enumerate(csv.reader(fileobj)): - self.assertEqual(row, rows[i]) + rows = [ + ['\na', 'b\nc', 'd\n'], + ['\re', 'f\rg', 'h\r'], + ['\r\ni', 'j\r\nk', 'l\r\n'], + ['\n\rm', 'n\n\ro', 'p\n\r'], + ['\r\rq', 'r\r\rs', 't\r\r'], + ['\n\nu', 'v\n\nw', 'x\n\n'], + ] + for lineterminator in '\r\n', '\n', '\r': + with self.subTest(lineterminator=lineterminator): + with TemporaryFile("w+", encoding="utf-8", newline='') as fileobj: + writer = csv.writer(fileobj, lineterminator=lineterminator) + writer.writerows(rows) + fileobj.seek(0) + for i, row in enumerate(csv.reader(fileobj)): + self.assertEqual(row, rows[i]) def test_roundtrip_escaped_unquoted_newlines(self): - with TemporaryFile("w+", encoding="utf-8", newline='') as fileobj: - writer = csv.writer(fileobj,quoting=csv.QUOTE_NONE,escapechar="\\") - rows = [['a\nb','b'],['c','x\r\nd']] - writer.writerows(rows) - fileobj.seek(0) - for i, row in enumerate(csv.reader(fileobj,quoting=csv.QUOTE_NONE,escapechar="\\")): - self.assertEqual(row,rows[i]) + rows = [ + ['\na', 'b\nc', 'd\n'], + ['\re', 'f\rg', 'h\r'], + ['\r\ni', 'j\r\nk', 'l\r\n'], + ['\n\rm', 'n\n\ro', 'p\n\r'], + ['\r\rq', 'r\r\rs', 't\r\r'], + ['\n\nu', 'v\n\nw', 'x\n\n'], + ] + for lineterminator in '\r\n', '\n', '\r': + with self.subTest(lineterminator=lineterminator): + with TemporaryFile("w+", encoding="utf-8", newline='') as fileobj: + writer = csv.writer(fileobj, lineterminator=lineterminator, + quoting=csv.QUOTE_NONE, escapechar="\\") + writer.writerows(rows) + fileobj.seek(0) + for i, row in enumerate(csv.reader(fileobj, + quoting=csv.QUOTE_NONE, + escapechar="\\")): + self.assertEqual(row, rows[i]) + class TestDialectRegistry(unittest.TestCase): def test_registry_badargs(self): diff --git a/Lib/test/test_importlib/test_lazy.py b/Lib/test/test_importlib/test_lazy.py index cc993f333e355a..38ab21907b58d9 100644 --- a/Lib/test/test_importlib/test_lazy.py +++ b/Lib/test/test_importlib/test_lazy.py @@ -2,9 +2,12 @@ from importlib import abc from importlib import util import sys +import time +import threading import types import unittest +from test.support import threading_helper from test.test_importlib import util as test_util @@ -40,6 +43,7 @@ class TestingImporter(abc.MetaPathFinder, abc.Loader): module_name = 'lazy_loader_test' mutated_name = 'changed' loaded = None + load_count = 0 source_code = 'attr = 42; __name__ = {!r}'.format(mutated_name) def find_spec(self, name, path, target=None): @@ -48,8 +52,10 @@ def find_spec(self, name, path, target=None): return util.spec_from_loader(name, util.LazyLoader(self)) def exec_module(self, module): + time.sleep(0.01) # Simulate a slow load. exec(self.source_code, module.__dict__) self.loaded = module + self.load_count += 1 class LazyLoaderTests(unittest.TestCase): @@ -59,8 +65,9 @@ def test_init(self): # Classes that don't define exec_module() trigger TypeError. util.LazyLoader(object) - def new_module(self, source_code=None): - loader = TestingImporter() + def new_module(self, source_code=None, loader=None): + if loader is None: + loader = TestingImporter() if source_code is not None: loader.source_code = source_code spec = util.spec_from_loader(TestingImporter.module_name, @@ -140,6 +147,37 @@ def test_module_already_in_sys(self): # Force the load; just care that no exception is raised. module.__name__ + @threading_helper.requires_working_threading() + def test_module_load_race(self): + with test_util.uncache(TestingImporter.module_name): + loader = TestingImporter() + module = self.new_module(loader=loader) + self.assertEqual(loader.load_count, 0) + + class RaisingThread(threading.Thread): + exc = None + def run(self): + try: + super().run() + except Exception as exc: + self.exc = exc + + def access_module(): + return module.attr + + threads = [] + for _ in range(2): + threads.append(thread := RaisingThread(target=access_module)) + thread.start() + + # Races could cause errors + for thread in threads: + thread.join() + self.assertIsNone(thread.exc) + + # Or multiple load attempts + self.assertEqual(loader.load_count, 1) + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_pathlib/test_pathlib_abc.py b/Lib/test/test_pathlib/test_pathlib_abc.py index 1d30deca8f7a1b..5bfb76f85c7909 100644 --- a/Lib/test/test_pathlib/test_pathlib_abc.py +++ b/Lib/test/test_pathlib/test_pathlib_abc.py @@ -957,6 +957,8 @@ def test_with_stem_empty(self): self.assertEqual(P('/').with_stem('d'), P('/d')) self.assertEqual(P('a/b').with_stem(''), P('a/')) self.assertEqual(P('a/b').with_stem('.'), P('a/.')) + self.assertRaises(ValueError, P('foo.gz').with_stem, '') + self.assertRaises(ValueError, P('/a/b/foo.gz').with_stem, '') def test_with_stem_seps(self): P = self.cls diff --git a/Misc/NEWS.d/next/Core and Builtins/2024-02-12-23-29-17.gh-issue-115323.3t6687.rst b/Misc/NEWS.d/next/Core and Builtins/2024-02-12-23-29-17.gh-issue-115323.3t6687.rst new file mode 100644 index 00000000000000..171855608fbc6a --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2024-02-12-23-29-17.gh-issue-115323.3t6687.rst @@ -0,0 +1,2 @@ +Make error message more meaningful for when :meth:`bytearray.extend` is +called with a :class:`str` object. diff --git a/Misc/NEWS.d/next/Core and Builtins/2024-02-22-11-33-20.gh-issue-115778.jksd1D.rst b/Misc/NEWS.d/next/Core and Builtins/2024-02-22-11-33-20.gh-issue-115778.jksd1D.rst new file mode 100644 index 00000000000000..023f6aa3026733 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2024-02-22-11-33-20.gh-issue-115778.jksd1D.rst @@ -0,0 +1 @@ +Add ``tierN`` annotation for instruction definition in interpreter DSL. diff --git a/Misc/NEWS.d/next/Library/2024-01-26-16-42-31.gh-issue-114610.S18Vuz.rst b/Misc/NEWS.d/next/Library/2024-01-26-16-42-31.gh-issue-114610.S18Vuz.rst new file mode 100644 index 00000000000000..519aede72aaf29 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-01-26-16-42-31.gh-issue-114610.S18Vuz.rst @@ -0,0 +1,4 @@ +Fix bug where :meth:`pathlib.PurePath.with_stem` converted a non-empty path +suffix to a stem when given an empty *stem* argument. It now raises +:exc:`ValueError`, just like :meth:`pathlib.PurePath.with_suffix` does when +called on a path with an empty stem, given a non-empty *suffix* argument. diff --git a/Misc/NEWS.d/next/Library/2024-01-30-23-28-29.gh-issue-114763.BRjKkg.rst b/Misc/NEWS.d/next/Library/2024-01-30-23-28-29.gh-issue-114763.BRjKkg.rst new file mode 100644 index 00000000000000..e8bdb83dde61fb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-01-30-23-28-29.gh-issue-114763.BRjKkg.rst @@ -0,0 +1,3 @@ +Protect modules loaded with :class:`importlib.util.LazyLoader` from race +conditions when multiple threads try to access attributes before the loading +is complete. diff --git a/Misc/NEWS.d/next/Library/2024-02-20-22-02-34.gh-issue-67044.QF9_Ru.rst b/Misc/NEWS.d/next/Library/2024-02-20-22-02-34.gh-issue-67044.QF9_Ru.rst new file mode 100644 index 00000000000000..095e69b6cadab6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-02-20-22-02-34.gh-issue-67044.QF9_Ru.rst @@ -0,0 +1,2 @@ +:func:`csv.writer` now always quotes or escapes ``'\r'`` and ``'\n'``, +regardless of *lineterminator* value. diff --git a/Modules/_csv.c b/Modules/_csv.c index 8d0472885afd96..660c5455af764e 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -1152,6 +1152,8 @@ join_append_data(WriterObj *self, int field_kind, const void *field_data, if (c == dialect->delimiter || c == dialect->escapechar || c == dialect->quotechar || + c == '\n' || + c == '\r' || PyUnicode_FindChar( dialect->lineterminator, c, 0, PyUnicode_GET_LENGTH(dialect->lineterminator), 1) >= 0) { diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c index acc59b926448ca..5e3b3affbc76c5 100644 --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -1729,6 +1729,10 @@ bytearray_extend(PyByteArrayObject *self, PyObject *iterable_of_ints) while ((item = PyIter_Next(it)) != NULL) { if (! _getbytevalue(item, &value)) { + if (PyErr_ExceptionMatches(PyExc_TypeError) && PyUnicode_Check(iterable_of_ints)) { + PyErr_Format(PyExc_TypeError, + "expected iterable of integers; got: 'str'"); + } Py_DECREF(item); Py_DECREF(it); Py_DECREF(bytearray_obj); diff --git a/Python/bytecodes.c b/Python/bytecodes.c index 7e2c9c4d6a6db4..e9e9425f826a2d 100644 --- a/Python/bytecodes.c +++ b/Python/bytecodes.c @@ -141,8 +141,7 @@ dummy_func( RESUME_CHECK, }; - inst(RESUME, (--)) { - TIER_ONE_ONLY + tier1 inst(RESUME, (--)) { assert(frame == tstate->current_frame); uintptr_t global_version = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker) & @@ -268,8 +267,7 @@ dummy_func( macro(END_FOR) = POP_TOP; - inst(INSTRUMENTED_END_FOR, (receiver, value -- receiver)) { - TIER_ONE_ONLY + tier1 inst(INSTRUMENTED_END_FOR, (receiver, value -- receiver)) { /* Need to create a fake StopIteration error here, * to conform to PEP 380 */ if (PyGen_Check(receiver)) { @@ -286,8 +284,7 @@ dummy_func( Py_DECREF(receiver); } - inst(INSTRUMENTED_END_SEND, (receiver, value -- value)) { - TIER_ONE_ONLY + tier1 inst(INSTRUMENTED_END_SEND, (receiver, value -- value)) { if (PyGen_Check(receiver) || PyCoro_CheckExact(receiver)) { PyErr_SetObject(PyExc_StopIteration, value); if (monitor_stop_iteration(tstate, frame, this_instr)) { @@ -319,7 +316,6 @@ dummy_func( }; specializing op(_SPECIALIZE_TO_BOOL, (counter/1, value -- value)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -506,8 +502,7 @@ dummy_func( // So the inputs are the same as for all BINARY_OP // specializations, but there is no output. // At the end we just skip over the STORE_FAST. - op(_BINARY_OP_INPLACE_ADD_UNICODE, (left, right --)) { - TIER_ONE_ONLY + tier1 op(_BINARY_OP_INPLACE_ADD_UNICODE, (left, right --)) { assert(next_instr->op.code == STORE_FAST); PyObject **target_local = &GETLOCAL(next_instr->op.arg); DEOPT_IF(*target_local != left); @@ -545,7 +540,6 @@ dummy_func( }; specializing op(_SPECIALIZE_BINARY_SUBSCR, (counter/1, container, sub -- container, sub)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -693,7 +687,6 @@ dummy_func( }; specializing op(_SPECIALIZE_STORE_SUBSCR, (counter/1, container, sub -- container, sub)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -762,8 +755,7 @@ dummy_func( ERROR_IF(res == NULL, error); } - inst(RAISE_VARARGS, (args[oparg] -- )) { - TIER_ONE_ONLY + tier1 inst(RAISE_VARARGS, (args[oparg] -- )) { PyObject *cause = NULL, *exc = NULL; switch (oparg) { case 2: @@ -787,8 +779,7 @@ dummy_func( ERROR_IF(true, error); } - inst(INTERPRETER_EXIT, (retval --)) { - TIER_ONE_ONLY + tier1 inst(INTERPRETER_EXIT, (retval --)) { assert(frame == &entry_frame); assert(_PyFrame_IsIncomplete(frame)); /* Restore previous frame and return. */ @@ -980,7 +971,6 @@ dummy_func( }; specializing op(_SPECIALIZE_SEND, (counter/1, receiver, unused -- receiver, unused)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -1075,8 +1065,7 @@ dummy_func( goto resume_frame; } - inst(YIELD_VALUE, (retval -- unused)) { - TIER_ONE_ONLY + tier1 inst(YIELD_VALUE, (retval -- unused)) { // NOTE: It's important that YIELD_VALUE never raises an exception! // The compiler treats any exception raised here as a failed close() // or throw() call. @@ -1105,8 +1094,7 @@ dummy_func( Py_XSETREF(exc_info->exc_value, exc_value == Py_None ? NULL : exc_value); } - inst(RERAISE, (values[oparg], exc -- values[oparg])) { - TIER_ONE_ONLY + tier1 inst(RERAISE, (values[oparg], exc -- values[oparg])) { assert(oparg >= 0 && oparg <= 2); if (oparg) { PyObject *lasti = values[0]; @@ -1127,8 +1115,7 @@ dummy_func( goto exception_unwind; } - inst(END_ASYNC_FOR, (awaitable, exc -- )) { - TIER_ONE_ONLY + tier1 inst(END_ASYNC_FOR, (awaitable, exc -- )) { assert(exc && PyExceptionInstance_Check(exc)); if (PyErr_GivenExceptionMatches(exc, PyExc_StopAsyncIteration)) { DECREF_INPUTS(); @@ -1141,8 +1128,7 @@ dummy_func( } } - inst(CLEANUP_THROW, (sub_iter, last_sent_val, exc_value -- none, value)) { - TIER_ONE_ONLY + tier1 inst(CLEANUP_THROW, (sub_iter, last_sent_val, exc_value -- none, value)) { assert(throwflag); assert(exc_value && PyExceptionInstance_Check(exc_value)); if (PyErr_GivenExceptionMatches(exc_value, PyExc_StopIteration)) { @@ -1214,7 +1200,6 @@ dummy_func( }; specializing op(_SPECIALIZE_UNPACK_SEQUENCE, (counter/1, seq -- seq)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -1284,7 +1269,6 @@ dummy_func( }; specializing op(_SPECIALIZE_STORE_ATTR, (counter/1, owner -- owner)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { PyObject *name = GETITEM(FRAME_CO_NAMES, oparg); @@ -1403,7 +1387,6 @@ dummy_func( }; specializing op(_SPECIALIZE_LOAD_GLOBAL, (counter/1 -- )) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1); @@ -1731,7 +1714,6 @@ dummy_func( }; specializing op(_SPECIALIZE_LOAD_SUPER_ATTR, (counter/1, global_super, class, unused -- global_super, class, unused)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION int load_method = oparg & 1; if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { @@ -1744,8 +1726,7 @@ dummy_func( #endif /* ENABLE_SPECIALIZATION */ } - op(_LOAD_SUPER_ATTR, (global_super, class, self -- attr, null if (oparg & 1))) { - TIER_ONE_ONLY + tier1 op(_LOAD_SUPER_ATTR, (global_super, class, self -- attr, null if (oparg & 1))) { if (opcode == INSTRUMENTED_LOAD_SUPER_ATTR) { PyObject *arg = oparg & 2 ? class : &_PyInstrumentation_MISSING; int err = _Py_call_instrumentation_2args( @@ -1847,7 +1828,6 @@ dummy_func( }; specializing op(_SPECIALIZE_LOAD_ATTR, (counter/1, owner -- owner)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1); @@ -2172,7 +2152,6 @@ dummy_func( }; specializing op(_SPECIALIZE_COMPARE_OP, (counter/1, left, right -- left, right)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -2297,28 +2276,24 @@ dummy_func( b = res ? Py_True : Py_False; } - inst(IMPORT_NAME, (level, fromlist -- res)) { - TIER_ONE_ONLY + tier1 inst(IMPORT_NAME, (level, fromlist -- res)) { PyObject *name = GETITEM(FRAME_CO_NAMES, oparg); res = import_name(tstate, frame, name, fromlist, level); DECREF_INPUTS(); ERROR_IF(res == NULL, error); } - inst(IMPORT_FROM, (from -- from, res)) { - TIER_ONE_ONLY + tier1 inst(IMPORT_FROM, (from -- from, res)) { PyObject *name = GETITEM(FRAME_CO_NAMES, oparg); res = import_from(tstate, from, name); ERROR_IF(res == NULL, error); } - inst(JUMP_FORWARD, (--)) { - TIER_ONE_ONLY + tier1 inst(JUMP_FORWARD, (--)) { JUMPBY(oparg); } - inst(JUMP_BACKWARD, (unused/1 --)) { - TIER_ONE_ONLY + tier1 inst(JUMP_BACKWARD, (unused/1 --)) { CHECK_EVAL_BREAKER(); assert(oparg <= INSTR_OFFSET()); JUMPBY(-oparg); @@ -2373,8 +2348,7 @@ dummy_func( JUMP_BACKWARD_NO_INTERRUPT, }; - inst(ENTER_EXECUTOR, (--)) { - TIER_ONE_ONLY + tier1 inst(ENTER_EXECUTOR, (--)) { CHECK_EVAL_BREAKER(); PyCodeObject *code = _PyFrame_GetCode(frame); _PyExecutorObject *executor = code->co_executors->executors[oparg & 255]; @@ -2423,8 +2397,7 @@ dummy_func( macro(POP_JUMP_IF_NOT_NONE) = unused/1 + _IS_NONE + _POP_JUMP_IF_FALSE; - inst(JUMP_BACKWARD_NO_INTERRUPT, (--)) { - TIER_ONE_ONLY + tier1 inst(JUMP_BACKWARD_NO_INTERRUPT, (--)) { /* This bytecode is used in the `yield from` or `await` loop. * If there is an interrupt, we want it handled in the innermost * generator or coroutine, so we deliberately do not check it here. @@ -2520,7 +2493,6 @@ dummy_func( }; specializing op(_SPECIALIZE_FOR_ITER, (counter/1, iter -- iter)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -3018,7 +2990,6 @@ dummy_func( }; specializing op(_SPECIALIZE_CALL, (counter/1, callable, self_or_null, args[oparg] -- callable, self_or_null, args[oparg])) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -3480,8 +3451,7 @@ dummy_func( } // This is secretly a super-instruction - inst(CALL_LIST_APPEND, (unused/1, unused/2, callable, self, args[oparg] -- unused)) { - TIER_ONE_ONLY + tier1 inst(CALL_LIST_APPEND, (unused/1, unused/2, callable, self, args[oparg] -- unused)) { assert(oparg == 1); PyInterpreterState *interp = tstate->interp; DEOPT_IF(callable != interp->callable_cache.list_append); @@ -3819,8 +3789,7 @@ dummy_func( } } - inst(RETURN_GENERATOR, (--)) { - TIER_ONE_ONLY + tier1 inst(RETURN_GENERATOR, (--)) { assert(PyFunction_Check(frame->f_funcobj)); PyFunctionObject *func = (PyFunctionObject *)frame->f_funcobj; PyGenObject *gen = (PyGenObject *)_Py_MakeCoro(func); @@ -3886,7 +3855,6 @@ dummy_func( } specializing op(_SPECIALIZE_BINARY_OP, (counter/1, lhs, rhs -- lhs, rhs)) { - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -3992,8 +3960,7 @@ dummy_func( INSTRUMENTED_JUMP(this_instr, next_instr + offset, PY_MONITORING_EVENT_BRANCH); } - inst(EXTENDED_ARG, ( -- )) { - TIER_ONE_ONLY + tier1 inst(EXTENDED_ARG, ( -- )) { assert(oparg); opcode = next_instr->op.code; oparg = oparg << 8 | next_instr->op.arg; @@ -4001,14 +3968,12 @@ dummy_func( DISPATCH_GOTO(); } - inst(CACHE, (--)) { - TIER_ONE_ONLY + tier1 inst(CACHE, (--)) { assert(0 && "Executing a cache."); Py_FatalError("Executing a cache."); } - inst(RESERVED, (--)) { - TIER_ONE_ONLY + tier1 inst(RESERVED, (--)) { assert(0 && "Executing RESERVED instruction."); Py_FatalError("Executing RESERVED instruction."); } @@ -4048,8 +4013,7 @@ dummy_func( CHECK_EVAL_BREAKER(); } - op(_SET_IP, (instr_ptr/4 --)) { - TIER_TWO_ONLY + tier2 op(_SET_IP, (instr_ptr/4 --)) { frame->instr_ptr = (_Py_CODEUNIT *)instr_ptr; } @@ -4062,45 +4026,37 @@ dummy_func( #endif } - op(_EXIT_TRACE, (--)) { - TIER_TWO_ONLY + tier2 op(_EXIT_TRACE, (--)) { EXIT_IF(1); } - op(_CHECK_VALIDITY, (--)) { - TIER_TWO_ONLY + tier2 op(_CHECK_VALIDITY, (--)) { DEOPT_IF(!current_executor->vm_data.valid); } - pure op(_LOAD_CONST_INLINE, (ptr/4 -- value)) { - TIER_TWO_ONLY + tier2 pure op(_LOAD_CONST_INLINE, (ptr/4 -- value)) { value = Py_NewRef(ptr); } - pure op(_LOAD_CONST_INLINE_BORROW, (ptr/4 -- value)) { - TIER_TWO_ONLY + tier2 pure op(_LOAD_CONST_INLINE_BORROW, (ptr/4 -- value)) { value = ptr; } - pure op(_LOAD_CONST_INLINE_WITH_NULL, (ptr/4 -- value, null)) { - TIER_TWO_ONLY + tier2 pure op(_LOAD_CONST_INLINE_WITH_NULL, (ptr/4 -- value, null)) { value = Py_NewRef(ptr); null = NULL; } - pure op(_LOAD_CONST_INLINE_BORROW_WITH_NULL, (ptr/4 -- value, null)) { - TIER_TWO_ONLY + tier2 pure op(_LOAD_CONST_INLINE_BORROW_WITH_NULL, (ptr/4 -- value, null)) { value = ptr; null = NULL; } - op(_CHECK_GLOBALS, (dict/4 -- )) { - TIER_TWO_ONLY + tier2 op(_CHECK_GLOBALS, (dict/4 -- )) { DEOPT_IF(GLOBALS() != dict); } - op(_CHECK_BUILTINS, (dict/4 -- )) { - TIER_TWO_ONLY + tier2 op(_CHECK_BUILTINS, (dict/4 -- )) { DEOPT_IF(BUILTINS() != dict); } @@ -4113,8 +4069,7 @@ dummy_func( /* Only used for handling cold side exits, should never appear in * a normal trace or as part of an instruction. */ - op(_COLD_EXIT, (--)) { - TIER_TWO_ONLY + tier2 op(_COLD_EXIT, (--)) { _PyExecutorObject *previous = (_PyExecutorObject *)tstate->previous_executor; _PyExitData *exit = &previous->exits[oparg]; exit->temperature++; @@ -4147,8 +4102,7 @@ dummy_func( GOTO_TIER_TWO(executor); } - op(_START_EXECUTOR, (executor/4 --)) { - TIER_TWO_ONLY + tier2 op(_START_EXECUTOR, (executor/4 --)) { Py_DECREF(tstate->previous_executor); tstate->previous_executor = NULL; #ifndef _Py_JIT @@ -4156,14 +4110,12 @@ dummy_func( #endif } - op(_FATAL_ERROR, (--)) { - TIER_TWO_ONLY + tier2 op(_FATAL_ERROR, (--)) { assert(0); Py_FatalError("Fatal error uop executed."); } - op(_CHECK_VALIDITY_AND_SET_IP, (instr_ptr/4 --)) { - TIER_TWO_ONLY + tier2 op(_CHECK_VALIDITY_AND_SET_IP, (instr_ptr/4 --)) { DEOPT_IF(!current_executor->vm_data.valid); frame->instr_ptr = (_Py_CODEUNIT *)instr_ptr; } diff --git a/Python/ceval_macros.h b/Python/ceval_macros.h index 01a9b32229d8a5..085199416c1523 100644 --- a/Python/ceval_macros.h +++ b/Python/ceval_macros.h @@ -372,12 +372,6 @@ static inline void _Py_LeaveRecursiveCallPy(PyThreadState *tstate) { tstate->py_recursion_remaining++; } -/* Marker to specify tier 1 only instructions */ -#define TIER_ONE_ONLY - -/* Marker to specify tier 2 only instructions */ -#define TIER_TWO_ONLY - /* Implementation of "macros" that modify the instruction pointer, * stack pointer, or frame pointer. * These need to treated differently by tier 1 and 2. diff --git a/Python/compile.c b/Python/compile.c index d857239690e7b5..6b17f3bcaf2264 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -921,11 +921,10 @@ dict_add_o(PyObject *dict, PyObject *o) PyObject *v; Py_ssize_t arg; - v = PyDict_GetItemWithError(dict, o); + if (PyDict_GetItemRef(dict, o, &v) < 0) { + return ERROR; + } if (!v) { - if (PyErr_Occurred()) { - return ERROR; - } arg = PyDict_GET_SIZE(dict); v = PyLong_FromSsize_t(arg); if (!v) { @@ -935,10 +934,10 @@ dict_add_o(PyObject *dict, PyObject *o) Py_DECREF(v); return ERROR; } - Py_DECREF(v); } else arg = PyLong_AsLong(v); + Py_DECREF(v); return arg; } diff --git a/Python/executor_cases.c.h b/Python/executor_cases.c.h index 3054058cf44d31..0b1d75985e1195 100644 --- a/Python/executor_cases.c.h +++ b/Python/executor_cases.c.h @@ -3716,7 +3716,6 @@ case _SET_IP: { PyObject *instr_ptr = (PyObject *)CURRENT_OPERAND(); - TIER_TWO_ONLY frame->instr_ptr = (_Py_CODEUNIT *)instr_ptr; break; } @@ -3733,13 +3732,11 @@ } case _EXIT_TRACE: { - TIER_TWO_ONLY if (1) goto side_exit; break; } case _CHECK_VALIDITY: { - TIER_TWO_ONLY if (!current_executor->vm_data.valid) goto deoptimize; break; } @@ -3747,7 +3744,6 @@ case _LOAD_CONST_INLINE: { PyObject *value; PyObject *ptr = (PyObject *)CURRENT_OPERAND(); - TIER_TWO_ONLY value = Py_NewRef(ptr); stack_pointer[0] = value; stack_pointer += 1; @@ -3757,7 +3753,6 @@ case _LOAD_CONST_INLINE_BORROW: { PyObject *value; PyObject *ptr = (PyObject *)CURRENT_OPERAND(); - TIER_TWO_ONLY value = ptr; stack_pointer[0] = value; stack_pointer += 1; @@ -3768,7 +3763,6 @@ PyObject *value; PyObject *null; PyObject *ptr = (PyObject *)CURRENT_OPERAND(); - TIER_TWO_ONLY value = Py_NewRef(ptr); null = NULL; stack_pointer[0] = value; @@ -3781,7 +3775,6 @@ PyObject *value; PyObject *null; PyObject *ptr = (PyObject *)CURRENT_OPERAND(); - TIER_TWO_ONLY value = ptr; null = NULL; stack_pointer[0] = value; @@ -3792,14 +3785,12 @@ case _CHECK_GLOBALS: { PyObject *dict = (PyObject *)CURRENT_OPERAND(); - TIER_TWO_ONLY if (GLOBALS() != dict) goto deoptimize; break; } case _CHECK_BUILTINS: { PyObject *dict = (PyObject *)CURRENT_OPERAND(); - TIER_TWO_ONLY if (BUILTINS() != dict) goto deoptimize; break; } @@ -3815,7 +3806,6 @@ case _COLD_EXIT: { oparg = CURRENT_OPARG(); - TIER_TWO_ONLY _PyExecutorObject *previous = (_PyExecutorObject *)tstate->previous_executor; _PyExitData *exit = &previous->exits[oparg]; exit->temperature++; @@ -3851,7 +3841,6 @@ case _START_EXECUTOR: { PyObject *executor = (PyObject *)CURRENT_OPERAND(); - TIER_TWO_ONLY Py_DECREF(tstate->previous_executor); tstate->previous_executor = NULL; #ifndef _Py_JIT @@ -3861,7 +3850,6 @@ } case _FATAL_ERROR: { - TIER_TWO_ONLY assert(0); Py_FatalError("Fatal error uop executed."); break; @@ -3869,7 +3857,6 @@ case _CHECK_VALIDITY_AND_SET_IP: { PyObject *instr_ptr = (PyObject *)CURRENT_OPERAND(); - TIER_TWO_ONLY if (!current_executor->vm_data.valid) goto deoptimize; frame->instr_ptr = (_Py_CODEUNIT *)instr_ptr; break; diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h index 87579134146c85..8b90e448a7b8fa 100644 --- a/Python/generated_cases.c.h +++ b/Python/generated_cases.c.h @@ -112,7 +112,6 @@ lhs = stack_pointer[-2]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -242,7 +241,6 @@ /* Skip 1 cache entry */ // _BINARY_OP_INPLACE_ADD_UNICODE { - TIER_ONE_ONLY assert(next_instr->op.code == STORE_FAST); PyObject **target_local = &GETLOCAL(next_instr->op.arg); DEOPT_IF(*target_local != left, BINARY_OP); @@ -429,7 +427,6 @@ container = stack_pointer[-2]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -739,7 +736,6 @@ frame->instr_ptr = next_instr; next_instr += 1; INSTRUCTION_STATS(CACHE); - TIER_ONE_ONLY assert(0 && "Executing a cache."); Py_FatalError("Executing a cache."); DISPATCH(); @@ -761,7 +757,6 @@ callable = stack_pointer[-2 - oparg]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -1477,7 +1472,6 @@ args = &stack_pointer[-oparg]; self = stack_pointer[-1 - oparg]; callable = stack_pointer[-2 - oparg]; - TIER_ONE_ONLY assert(oparg == 1); PyInterpreterState *interp = tstate->interp; DEOPT_IF(callable != interp->callable_cache.list_append, CALL); @@ -1953,7 +1947,6 @@ exc_value = stack_pointer[-1]; last_sent_val = stack_pointer[-2]; sub_iter = stack_pointer[-3]; - TIER_ONE_ONLY assert(throwflag); assert(exc_value && PyExceptionInstance_Check(exc_value)); if (PyErr_GivenExceptionMatches(exc_value, PyExc_StopIteration)) { @@ -1988,7 +1981,6 @@ left = stack_pointer[-2]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -2338,7 +2330,6 @@ PyObject *awaitable; exc = stack_pointer[-1]; awaitable = stack_pointer[-2]; - TIER_ONE_ONLY assert(exc && PyExceptionInstance_Check(exc)); if (PyErr_GivenExceptionMatches(exc, PyExc_StopAsyncIteration)) { Py_DECREF(awaitable); @@ -2383,7 +2374,6 @@ frame->instr_ptr = next_instr; next_instr += 1; INSTRUCTION_STATS(ENTER_EXECUTOR); - TIER_ONE_ONLY CHECK_EVAL_BREAKER(); PyCodeObject *code = _PyFrame_GetCode(frame); _PyExecutorObject *executor = code->co_executors->executors[oparg & 255]; @@ -2418,7 +2408,6 @@ frame->instr_ptr = next_instr; next_instr += 1; INSTRUCTION_STATS(EXTENDED_ARG); - TIER_ONE_ONLY assert(oparg); opcode = next_instr->op.code; oparg = oparg << 8 | next_instr->op.arg; @@ -2477,7 +2466,6 @@ iter = stack_pointer[-1]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -2867,7 +2855,6 @@ PyObject *from; PyObject *res; from = stack_pointer[-1]; - TIER_ONE_ONLY PyObject *name = GETITEM(FRAME_CO_NAMES, oparg); res = import_from(tstate, from, name); if (res == NULL) goto error; @@ -2885,7 +2872,6 @@ PyObject *res; fromlist = stack_pointer[-1]; level = stack_pointer[-2]; - TIER_ONE_ONLY PyObject *name = GETITEM(FRAME_CO_NAMES, oparg); res = import_name(tstate, frame, name, fromlist, level); Py_DECREF(level); @@ -2945,7 +2931,6 @@ PyObject *receiver; value = stack_pointer[-1]; receiver = stack_pointer[-2]; - TIER_ONE_ONLY /* Need to create a fake StopIteration error here, * to conform to PEP 380 */ if (PyGen_Check(receiver)) { @@ -2968,7 +2953,6 @@ PyObject *receiver; value = stack_pointer[-1]; receiver = stack_pointer[-2]; - TIER_ONE_ONLY if (PyGen_Check(receiver) || PyCoro_CheckExact(receiver)) { PyErr_SetObject(PyExc_StopIteration, value); if (monitor_stop_iteration(tstate, frame, this_instr)) { @@ -3248,7 +3232,6 @@ INSTRUCTION_STATS(INTERPRETER_EXIT); PyObject *retval; retval = stack_pointer[-1]; - TIER_ONE_ONLY assert(frame == &entry_frame); assert(_PyFrame_IsIncomplete(frame)); /* Restore previous frame and return. */ @@ -3281,7 +3264,6 @@ next_instr += 2; INSTRUCTION_STATS(JUMP_BACKWARD); /* Skip 1 cache entry */ - TIER_ONE_ONLY CHECK_EVAL_BREAKER(); assert(oparg <= INSTR_OFFSET()); JUMPBY(-oparg); @@ -3331,7 +3313,6 @@ frame->instr_ptr = next_instr; next_instr += 1; INSTRUCTION_STATS(JUMP_BACKWARD_NO_INTERRUPT); - TIER_ONE_ONLY /* This bytecode is used in the `yield from` or `await` loop. * If there is an interrupt, we want it handled in the innermost * generator or coroutine, so we deliberately do not check it here. @@ -3345,7 +3326,6 @@ frame->instr_ptr = next_instr; next_instr += 1; INSTRUCTION_STATS(JUMP_FORWARD); - TIER_ONE_ONLY JUMPBY(oparg); DISPATCH(); } @@ -3414,7 +3394,6 @@ owner = stack_pointer[-1]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1); @@ -4109,7 +4088,6 @@ // _SPECIALIZE_LOAD_GLOBAL { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { PyObject *name = GETITEM(FRAME_CO_NAMES, oparg>>1); @@ -4311,7 +4289,6 @@ global_super = stack_pointer[-3]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION int load_method = oparg & 1; if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { @@ -4326,7 +4303,6 @@ // _LOAD_SUPER_ATTR self = stack_pointer[-1]; { - TIER_ONE_ONLY if (opcode == INSTRUMENTED_LOAD_SUPER_ATTR) { PyObject *arg = oparg & 2 ? class : &_PyInstrumentation_MISSING; int err = _Py_call_instrumentation_2args( @@ -4737,7 +4713,6 @@ INSTRUCTION_STATS(RAISE_VARARGS); PyObject **args; args = &stack_pointer[-oparg]; - TIER_ONE_ONLY PyObject *cause = NULL, *exc = NULL; switch (oparg) { case 2: @@ -4769,7 +4744,6 @@ PyObject **values; exc = stack_pointer[-1]; values = &stack_pointer[-1 - oparg]; - TIER_ONE_ONLY assert(oparg >= 0 && oparg <= 2); if (oparg) { PyObject *lasti = values[0]; @@ -4794,7 +4768,6 @@ frame->instr_ptr = next_instr; next_instr += 1; INSTRUCTION_STATS(RESERVED); - TIER_ONE_ONLY assert(0 && "Executing RESERVED instruction."); Py_FatalError("Executing RESERVED instruction."); DISPATCH(); @@ -4806,7 +4779,6 @@ INSTRUCTION_STATS(RESUME); PREDICTED(RESUME); _Py_CODEUNIT *this_instr = next_instr - 1; - TIER_ONE_ONLY assert(frame == tstate->current_frame); uintptr_t global_version = _Py_atomic_load_uintptr_relaxed(&tstate->eval_breaker) & @@ -4884,7 +4856,6 @@ frame->instr_ptr = next_instr; next_instr += 1; INSTRUCTION_STATS(RETURN_GENERATOR); - TIER_ONE_ONLY assert(PyFunction_Check(frame->f_funcobj)); PyFunctionObject *func = (PyFunctionObject *)frame->f_funcobj; PyGenObject *gen = (PyGenObject *)_Py_MakeCoro(func); @@ -4951,7 +4922,6 @@ receiver = stack_pointer[-2]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -5138,7 +5108,6 @@ owner = stack_pointer[-1]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { PyObject *name = GETITEM(FRAME_CO_NAMES, oparg); @@ -5430,7 +5399,6 @@ container = stack_pointer[-2]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -5532,7 +5500,6 @@ value = stack_pointer[-1]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -5741,7 +5708,6 @@ seq = stack_pointer[-1]; { uint16_t counter = read_u16(&this_instr[1].cache); - TIER_ONE_ONLY #if ENABLE_SPECIALIZATION if (ADAPTIVE_COUNTER_IS_ZERO(counter)) { next_instr = this_instr; @@ -5876,7 +5842,6 @@ INSTRUCTION_STATS(YIELD_VALUE); PyObject *retval; retval = stack_pointer[-1]; - TIER_ONE_ONLY // NOTE: It's important that YIELD_VALUE never raises an exception! // The compiler treats any exception raised here as a failed close() // or throw() call. diff --git a/Python/initconfig.c b/Python/initconfig.c index a6d8c176156617..74f28f3b39175b 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -153,7 +153,8 @@ Options (and corresponding environment variables):\n\ .pyc extension; also PYTHONOPTIMIZE=x\n\ -OO : do -O changes and also discard docstrings; add .opt-2 before\n\ .pyc extension\n\ --P : don't prepend a potentially unsafe path to sys.path; also PYTHONSAFEPATH\n\ +-P : don't prepend a potentially unsafe path to sys.path; also\n\ + PYTHONSAFEPATH\n\ -q : don't print version and copyright messages on interactive startup\n\ -s : don't add user site directory to sys.path; also PYTHONNOUSERSITE\n\ -S : don't imply 'import site' on initialization\n\ @@ -169,9 +170,10 @@ Options (and corresponding environment variables):\n\ -X opt : set implementation-specific option\n\ --check-hash-based-pycs always|default|never:\n\ control how Python invalidates hash-based .pyc files\n\ ---help-env : print help about Python environment variables and exit\n\ ---help-xoptions : print help about implementation-specific -X options and exit\n\ ---help-all : print complete help information and exit\n\ +--help-env: print help about Python environment variables and exit\n\ +--help-xoptions: print help about implementation-specific -X options and exit\n\ +--help-all: print complete help information and exit\n\ +\n\ Arguments:\n\ file : program read from script file\n\ - : program read from stdin (default; interactive mode if a tty)\n\ @@ -180,73 +182,61 @@ arg ...: arguments passed to program in sys.argv[1:]\n\ static const char usage_xoptions[] = "\ The following implementation-specific options are available:\n\ -\n\ -X faulthandler: enable faulthandler\n\ -\n\ -X showrefcount: output the total reference count and number of used\n\ - memory blocks when the program finishes or after each statement in the\n\ - interactive interpreter. This only works on debug builds\n\ -\n\ + memory blocks when the program finishes or after each statement in\n\ + the interactive interpreter. This only works on debug builds\n\ -X tracemalloc: start tracing Python memory allocations using the\n\ - tracemalloc module. By default, only the most recent frame is stored in a\n\ - traceback of a trace. Use -X tracemalloc=NFRAME to start tracing with a\n\ - traceback limit of NFRAME frames\n\ -\n\ --X importtime: show how long each import takes. It shows module name,\n\ - cumulative time (including nested imports) and self time (excluding\n\ - nested imports). Note that its output may be broken in multi-threaded\n\ - application. Typical usage is python3 -X importtime -c 'import asyncio'\n\ -\n\ --X dev: enable CPython's \"development mode\", introducing additional runtime\n\ - checks which are too expensive to be enabled by default. Effect of the\n\ - developer mode:\n\ - * Add default warning filter, as -W default\n\ - * Install debug hooks on memory allocators: see the PyMem_SetupDebugHooks()\n\ - C function\n\ - * Enable the faulthandler module to dump the Python traceback on a crash\n\ - * Enable asyncio debug mode\n\ - * Set the dev_mode attribute of sys.flags to True\n\ - * io.IOBase destructor logs close() exceptions\n\ -\n\ --X utf8: enable UTF-8 mode for operating system interfaces, overriding the default\n\ - locale-aware mode. -X utf8=0 explicitly disables UTF-8 mode (even when it would\n\ - otherwise activate automatically)\n\ -\n\ --X pycache_prefix=PATH: enable writing .pyc files to a parallel tree rooted at the\n\ - given directory instead of to the code tree\n\ -\n\ + tracemalloc module. By default, only the most recent frame is stored\n\ + in a traceback of a trace. Use -X tracemalloc=NFRAME to start\n\ + tracing with a traceback limit of NFRAME frames\n\ +-X importtime: show how long each import takes. It shows module name,\n\ + cumulative time (including nested imports) and self time (excluding\n\ + nested imports). Note that its output may be broken in\n\ + multi-threaded application.\n\ + Typical usage is python3 -X importtime -c 'import asyncio'\n\ +-X dev : enable CPython's \"development mode\", introducing additional runtime\n\ + checks which are too expensive to be enabled by default. Effect of\n\ + the developer mode:\n\ + * Add default warning filter, as -W default\n\ + * Install debug hooks on memory allocators: see the\n\ + PyMem_SetupDebugHooks() C function\n\ + * Enable the faulthandler module to dump the Python traceback on\n\ + a crash\n\ + * Enable asyncio debug mode\n\ + * Set the dev_mode attribute of sys.flags to True\n\ + * io.IOBase destructor logs close() exceptions\n\ +-X utf8: enable UTF-8 mode for operating system interfaces, overriding the\n\ + default locale-aware mode. -X utf8=0 explicitly disables UTF-8 mode\n\ + (even when it would otherwise activate automatically)\n\ +-X pycache_prefix=PATH: enable writing .pyc files to a parallel tree rooted\n\ + at the given directory instead of to the code tree\n\ -X warn_default_encoding: enable opt-in EncodingWarning for 'encoding=None'\n\ -\n\ --X no_debug_ranges: disable the inclusion of the tables mapping extra location \n\ - information (end line, start column offset and end column offset) to every \n\ - instruction in code objects. This is useful when smaller code objects and pyc \n\ - files are desired as well as suppressing the extra visual location indicators \n\ - when the interpreter displays tracebacks.\n\ -\n\ --X perf: activate support for the Linux \"perf\" profiler by activating the \"perf\"\n\ - trampoline. When this option is activated, the Linux \"perf\" profiler will be \n\ - able to report Python calls. This option is only available on some platforms and will \n\ - do nothing if is not supported on the current system. The default value is \"off\".\n\ -\n\ +-X no_debug_ranges: disable the inclusion of the tables mapping extra location\n\ + information (end line, start column offset and end column offset) to\n\ + every instruction in code objects. This is useful when smaller code\n\ + objects and pyc files are desired as well as suppressing the extra\n\ + visual location indicators when the interpreter displays tracebacks.\n\ +-X perf: activate support for the Linux \"perf\" profiler by activating the\n\ + \"perf\" trampoline. When this option is activated, the Linux \"perf\"\n\ + profiler will be able to report Python calls. This option is only\n\ + available on some platforms and will do nothing if is not supported\n\ + on the current system. The default value is \"off\".\n\ -X frozen_modules=[on|off]: whether or not frozen modules should be used.\n\ - The default is \"on\" (or \"off\" if you are running a local build).\n\ -\n\ + The default is \"on\" (or \"off\" if you are running a local build).\n\ -X int_max_str_digits=number: limit the size of int<->str conversions.\n\ - This helps avoid denial of service attacks when parsing untrusted data.\n\ - The default is sys.int_info.default_max_str_digits. 0 disables.\n\ -\n\ + This helps avoid denial of service attacks when parsing untrusted\n\ + data. The default is sys.int_info.default_max_str_digits.\n\ + 0 disables.\n\ -X cpu_count=[n|default]: Override the return value of os.cpu_count(),\n\ - os.process_cpu_count(), and multiprocessing.cpu_count(). This can help users who need\n\ - to limit resources in a container." - + os.process_cpu_count(), and multiprocessing.cpu_count(). This can\n\ + help users who need to limit resources in a container." #ifdef Py_STATS "\n\ -\n\ -X pystats: Enable pystats collection at startup." #endif #ifdef Py_DEBUG "\n\ -\n\ -X presite=package.module: import this module before site.py is run." #endif ; @@ -254,65 +244,73 @@ The following implementation-specific options are available:\n\ /* Envvars that don't have equivalent command-line options are listed first */ static const char usage_envvars[] = "Environment variables that change behavior:\n" -"PYTHONSTARTUP: file executed on interactive startup (no default)\n" -"PYTHONPATH : '%lc'-separated list of directories prefixed to the\n" -" default module search path. The result is sys.path.\n" -"PYTHONHOME : alternate directory (or %lc).\n" -" The default module search path uses %s.\n" -"PYTHONPLATLIBDIR : override sys.platlibdir.\n" -"PYTHONCASEOK : ignore case in 'import' statements (Windows).\n" -"PYTHONUTF8: if set to 1, enable the UTF-8 mode.\n" +"PYTHONSTARTUP : file executed on interactive startup (no default)\n" +"PYTHONPATH : '%lc'-separated list of directories prefixed to the\n" +" default module search path. The result is sys.path.\n" +"PYTHONHOME : alternate directory (or %lc).\n" +" The default module search path uses %s.\n" +"PYTHONPLATLIBDIR: override sys.platlibdir.\n" +"PYTHONCASEOK : ignore case in 'import' statements (Windows).\n" +"PYTHONUTF8 : if set to 1, enable the UTF-8 mode.\n" "PYTHONIOENCODING: Encoding[:errors] used for stdin/stdout/stderr.\n" "PYTHONFAULTHANDLER: dump the Python traceback on fatal errors.\n" -"PYTHONHASHSEED: if this variable is set to 'random', a random value is used\n" -" to seed the hashes of str and bytes objects. It can also be set to an\n" -" integer in the range [0,4294967295] to get hash values with a\n" -" predictable seed.\n" +"PYTHONHASHSEED : if this variable is set to 'random', a random value is used\n" +" to seed the hashes of str and bytes objects. It can also be\n" +" set to an integer in the range [0,4294967295] to get hash\n" +" values with a predictable seed.\n" "PYTHONINTMAXSTRDIGITS: limits the maximum digit characters in an int value\n" -" when converting from a string and when converting an int back to a str.\n" -" A value of 0 disables the limit. Conversions to or from bases 2, 4, 8,\n" -" 16, and 32 are never limited.\n" -"PYTHONMALLOC: set the Python memory allocators and/or install debug hooks\n" -" on Python memory allocators. Use PYTHONMALLOC=debug to install debug\n" -" hooks.\n" +" when converting from a string and when converting an int\n" +" back to a str. A value of 0 disables the limit.\n" +" Conversions to or from bases 2, 4, 8, 16, and 32 are never\n" +" limited.\n" +"PYTHONMALLOC : set the Python memory allocators and/or install debug hooks\n" +" on Python memory allocators. Use PYTHONMALLOC=debug to\n" +" install debug hooks.\n" "PYTHONCOERCECLOCALE: if this variable is set to 0, it disables the locale\n" -" coercion behavior. Use PYTHONCOERCECLOCALE=warn to request display of\n" -" locale coercion and locale compatibility warnings on stderr.\n" +" coercion behavior. Use PYTHONCOERCECLOCALE=warn to request\n" +" display of locale coercion and locale compatibility warnings\n" +" on stderr.\n" "PYTHONBREAKPOINT: if this variable is set to 0, it disables the default\n" -" debugger. It can be set to the callable of your debugger of choice.\n" +" debugger. It can be set to the callable of your debugger of\n" +" choice.\n" "PYTHON_CPU_COUNT: Overrides the return value of os.process_cpu_count(),\n" -" os.cpu_count(), and multiprocessing.cpu_count() if set to a positive integer.\n" -"PYTHONDEVMODE: enable the development mode.\n" +" os.cpu_count(), and multiprocessing.cpu_count() if set to\n" +" a positive integer.\n" +"PYTHONDEVMODE : enable the development mode.\n" "PYTHONPYCACHEPREFIX: root directory for bytecode cache (pyc) files.\n" "PYTHONWARNDEFAULTENCODING: enable opt-in EncodingWarning for 'encoding=None'.\n" -"PYTHONNODEBUGRANGES: if this variable is set, it disables the inclusion of the \n" -" tables mapping extra location information (end line, start column offset \n" -" and end column offset) to every instruction in code objects. This is useful \n" -" when smaller code objects and pyc files are desired as well as suppressing the \n" -" extra visual location indicators when the interpreter displays tracebacks.\n" -"PYTHON_FROZEN_MODULES : if this variable is set, it determines whether or not \n" -" frozen modules should be used. The default is \"on\" (or \"off\" if you are \n" -" running a local build).\n" -"PYTHON_COLORS : If this variable is set to 1, the interpreter will" -" colorize various kinds of output. Setting it to 0 deactivates this behavior.\n" -"PYTHON_HISTORY : the location of a .python_history file.\n" -"These variables have equivalent command-line parameters (see --help for details):\n" -"PYTHONDEBUG : enable parser debug mode (-d)\n" -"PYTHONDONTWRITEBYTECODE : don't write .pyc files (-B)\n" -"PYTHONINSPECT : inspect interactively after running script (-i)\n" -"PYTHONINTMAXSTRDIGITS : limit max digit characters in an int value\n" -" (-X int_max_str_digits=number)\n" -"PYTHONNOUSERSITE : disable user site directory (-s)\n" -"PYTHONOPTIMIZE : enable level 1 optimizations (-O)\n" -"PYTHONSAFEPATH : don't prepend a potentially unsafe path to sys.path (-P)\n" -"PYTHONUNBUFFERED : disable stdout/stderr buffering (-u)\n" -"PYTHONVERBOSE : trace import statements (-v)\n" -"PYTHONWARNINGS=arg : warning control (-W arg)\n" +"PYTHONNODEBUGRANGES: if this variable is set, it disables the inclusion of\n" +" the tables mapping extra location information (end line,\n" +" start column offset and end column offset) to every\n" +" instruction in code objects. This is useful when smaller\n" +" code objects and pyc files are desired as well as\n" +" suppressing the extra visual location indicators when the\n" +" interpreter displays tracebacks.\n" +"PYTHON_FROZEN_MODULES: if this variable is set, it determines whether or not\n" +" frozen modules should be used. The default is \"on\" (or\n" +" \"off\" if you are running a local build).\n" +"PYTHON_COLORS : if this variable is set to 1, the interpreter will colorize\n" +" various kinds of output. Setting it to 0 deactivates\n" +" this behavior.\n" +"PYTHON_HISTORY : the location of a .python_history file.\n" +"\n" +"These variables have equivalent command-line options (see --help for details):\n" +"PYTHONDEBUG : enable parser debug mode (-d)\n" +"PYTHONDONTWRITEBYTECODE: don't write .pyc files (-B)\n" +"PYTHONINSPECT : inspect interactively after running script (-i)\n" +"PYTHONINTMAXSTRDIGITS: limit max digit characters in an int value\n" +" (-X int_max_str_digits=number)\n" +"PYTHONNOUSERSITE: disable user site directory (-s)\n" +"PYTHONOPTIMIZE : enable level 1 optimizations (-O)\n" +"PYTHONSAFEPATH : don't prepend a potentially unsafe path to sys.path.\n" +"PYTHONUNBUFFERED: disable stdout/stderr buffering (-u)\n" +"PYTHONVERBOSE : trace import statements (-v)\n" +"PYTHONWARNINGS=arg: warning control (-W arg)\n" #ifdef Py_STATS -"PYTHONSTATS : turns on statistics gathering\n" +"PYTHONSTATS : turns on statistics gathering\n" #endif #ifdef Py_DEBUG -"PYTHON_PRESITE=pkg.mod : import this module before site.py is run\n" +"PYTHON_PRESITE=pkg.mod: import this module before site.py is run\n" #endif ; @@ -2387,9 +2385,9 @@ static void config_complete_usage(const wchar_t* program) { config_usage(0, program); - puts("\n"); + putchar('\n'); config_envvars_usage(); - puts("\n"); + putchar('\n'); config_xoptions_usage(); } diff --git a/Python/optimizer_analysis.c b/Python/optimizer_analysis.c index 68ef8254b494a4..47bfc8cf1061d9 100644 --- a/Python/optimizer_analysis.c +++ b/Python/optimizer_analysis.c @@ -315,6 +315,7 @@ sym_new_known_notnull(_Py_UOpsAbstractInterpContext *ctx) if (res == NULL) { return NULL; } + sym_set_flag(res, KNOWN); sym_set_flag(res, NOT_NULL); return res; } @@ -809,9 +810,12 @@ _Py_uop_analyze_and_optimize( peephole_opt(frame, buffer, buffer_size); - err = uop_redundancy_eliminator( - (PyCodeObject *)frame->f_executable, buffer, - buffer_size, curr_stacklen, dependencies); + char *uop_optimize = Py_GETENV("PYTHONUOPSOPTIMIZE"); + if (uop_optimize != NULL && *uop_optimize > '0') { + err = uop_redundancy_eliminator( + (PyCodeObject *)frame->f_executable, buffer, + buffer_size, curr_stacklen, dependencies); + } if (err == 0) { goto not_ready; diff --git a/Python/tier2_redundancy_eliminator_bytecodes.c b/Python/tier2_redundancy_eliminator_bytecodes.c index ff2b9a42272863..b9afd3089e1077 100644 --- a/Python/tier2_redundancy_eliminator_bytecodes.c +++ b/Python/tier2_redundancy_eliminator_bytecodes.c @@ -295,6 +295,31 @@ dummy_func(void) { (void)owner; } + op(_LOAD_ATTR_METHOD_WITH_VALUES, (descr/4, owner -- attr, self if (1))) { + (void)descr; + OUT_OF_SPACE_IF_NULL(attr = sym_new_known_notnull(ctx)); + self = owner; + } + + op(_LOAD_ATTR_METHOD_NO_DICT, (descr/4, owner -- attr, self if (1))) { + (void)descr; + OUT_OF_SPACE_IF_NULL(attr = sym_new_known_notnull(ctx)); + self = owner; + } + + op(_LOAD_ATTR_METHOD_LAZY_DICT, (descr/4, owner -- attr, self if (1))) { + (void)descr; + OUT_OF_SPACE_IF_NULL(attr = sym_new_known_notnull(ctx)); + self = owner; + } + + op(_INIT_CALL_BOUND_METHOD_EXACT_ARGS, (callable, unused, unused[oparg] -- func, self, unused[oparg])) { + (void)callable; + OUT_OF_SPACE_IF_NULL(func = sym_new_known_notnull(ctx)); + OUT_OF_SPACE_IF_NULL(self = sym_new_known_notnull(ctx)); + } + + op(_CHECK_FUNCTION_EXACT_ARGS, (func_version/2, callable, self_or_null, unused[oparg] -- callable, self_or_null, unused[oparg])) { sym_set_type(callable, &PyFunction_Type); (void)self_or_null; diff --git a/Python/tier2_redundancy_eliminator_cases.c.h b/Python/tier2_redundancy_eliminator_cases.c.h index 58c11b7a1c87e0..ca341e4dde5d93 100644 --- a/Python/tier2_redundancy_eliminator_cases.c.h +++ b/Python/tier2_redundancy_eliminator_cases.c.h @@ -1300,12 +1300,14 @@ } case _LOAD_ATTR_METHOD_WITH_VALUES: { + _Py_UOpsSymType *owner; _Py_UOpsSymType *attr; _Py_UOpsSymType *self = NULL; - attr = sym_new_unknown(ctx); - if (attr == NULL) goto out_of_space; - self = sym_new_unknown(ctx); - if (self == NULL) goto out_of_space; + owner = stack_pointer[-1]; + PyObject *descr = (PyObject *)this_instr->operand; + (void)descr; + OUT_OF_SPACE_IF_NULL(attr = sym_new_known_notnull(ctx)); + self = owner; stack_pointer[-1] = attr; stack_pointer[0] = self; stack_pointer += 1; @@ -1313,12 +1315,14 @@ } case _LOAD_ATTR_METHOD_NO_DICT: { + _Py_UOpsSymType *owner; _Py_UOpsSymType *attr; _Py_UOpsSymType *self = NULL; - attr = sym_new_unknown(ctx); - if (attr == NULL) goto out_of_space; - self = sym_new_unknown(ctx); - if (self == NULL) goto out_of_space; + owner = stack_pointer[-1]; + PyObject *descr = (PyObject *)this_instr->operand; + (void)descr; + OUT_OF_SPACE_IF_NULL(attr = sym_new_known_notnull(ctx)); + self = owner; stack_pointer[-1] = attr; stack_pointer[0] = self; stack_pointer += 1; @@ -1346,12 +1350,14 @@ } case _LOAD_ATTR_METHOD_LAZY_DICT: { + _Py_UOpsSymType *owner; _Py_UOpsSymType *attr; _Py_UOpsSymType *self = NULL; - attr = sym_new_unknown(ctx); - if (attr == NULL) goto out_of_space; - self = sym_new_unknown(ctx); - if (self == NULL) goto out_of_space; + owner = stack_pointer[-1]; + PyObject *descr = (PyObject *)this_instr->operand; + (void)descr; + OUT_OF_SPACE_IF_NULL(attr = sym_new_known_notnull(ctx)); + self = owner; stack_pointer[-1] = attr; stack_pointer[0] = self; stack_pointer += 1; @@ -1373,12 +1379,13 @@ } case _INIT_CALL_BOUND_METHOD_EXACT_ARGS: { + _Py_UOpsSymType *callable; _Py_UOpsSymType *func; _Py_UOpsSymType *self; - func = sym_new_unknown(ctx); - if (func == NULL) goto out_of_space; - self = sym_new_unknown(ctx); - if (self == NULL) goto out_of_space; + callable = stack_pointer[-2 - oparg]; + (void)callable; + OUT_OF_SPACE_IF_NULL(func = sym_new_known_notnull(ctx)); + OUT_OF_SPACE_IF_NULL(self = sym_new_known_notnull(ctx)); stack_pointer[-2 - oparg] = func; stack_pointer[-1 - oparg] = self; break; diff --git a/Tools/cases_generator/analyzer.py b/Tools/cases_generator/analyzer.py index 49b8b426444d24..b0a15e6d87c2c6 100644 --- a/Tools/cases_generator/analyzer.py +++ b/Tools/cases_generator/analyzer.py @@ -17,7 +17,6 @@ class Properties: needs_this: bool always_exits: bool stores_sp: bool - tier_one_only: bool uses_co_consts: bool uses_co_names: bool uses_locals: bool @@ -25,6 +24,7 @@ class Properties: side_exit: bool pure: bool passthrough: bool + tier: int | None = None oparg_and_1: bool = False const_oparg: int = -1 @@ -46,7 +46,6 @@ def from_list(properties: list["Properties"]) -> "Properties": needs_this=any(p.needs_this for p in properties), always_exits=any(p.always_exits for p in properties), stores_sp=any(p.stores_sp for p in properties), - tier_one_only=any(p.tier_one_only for p in properties), uses_co_consts=any(p.uses_co_consts for p in properties), uses_co_names=any(p.uses_co_names for p in properties), uses_locals=any(p.uses_locals for p in properties), @@ -68,7 +67,6 @@ def from_list(properties: list["Properties"]) -> "Properties": needs_this=False, always_exits=False, stores_sp=False, - tier_one_only=False, uses_co_consts=False, uses_co_names=False, uses_locals=False, @@ -312,6 +310,15 @@ def variable_used(node: parser.InstDef, name: str) -> bool: token.kind == "IDENTIFIER" and token.text == name for token in node.tokens ) +def tier_variable(node: parser.InstDef) -> int | None: + """Determine whether a tier variable is used in a node.""" + for token in node.tokens: + if token.kind == "ANNOTATION": + if token.text == "specializing": + return 1 + if re.fullmatch(r"tier\d", token.text): + return int(token.text[-1]) + return None def is_infallible(op: parser.InstDef) -> bool: return not ( @@ -498,7 +505,6 @@ def compute_properties(op: parser.InstDef) -> Properties: needs_this=variable_used(op, "this_instr"), always_exits=always_exits(op), stores_sp=variable_used(op, "SYNC_SP"), - tier_one_only=variable_used(op, "TIER_ONE_ONLY"), uses_co_consts=variable_used(op, "FRAME_CO_CONSTS"), uses_co_names=variable_used(op, "FRAME_CO_NAMES"), uses_locals=(variable_used(op, "GETLOCAL") or variable_used(op, "SETLOCAL")) @@ -506,6 +512,7 @@ def compute_properties(op: parser.InstDef) -> Properties: has_free=has_free, pure="pure" in op.annotations, passthrough=passthrough, + tier=tier_variable(op), ) diff --git a/Tools/cases_generator/interpreter_definition.md b/Tools/cases_generator/interpreter_definition.md index 9b5733562f77b4..889f58fc3e1a75 100644 --- a/Tools/cases_generator/interpreter_definition.md +++ b/Tools/cases_generator/interpreter_definition.md @@ -168,6 +168,7 @@ list of annotations and their meanings are as follows: * `override`. For external use by other interpreter definitions to override the current instruction definition. * `pure`. This instruction has no side effects. +* 'tierN'. This instruction only used by tier N interpreter. ### Special functions/macros diff --git a/Tools/cases_generator/lexer.py b/Tools/cases_generator/lexer.py index 0077921e7d7fa1..13aee94f2b957c 100644 --- a/Tools/cases_generator/lexer.py +++ b/Tools/cases_generator/lexer.py @@ -224,6 +224,8 @@ def choice(*opts: str) -> str: "pure", "split", "replicate", + "tier1", + "tier2", } __all__ = [] diff --git a/Tools/cases_generator/opcode_metadata_generator.py b/Tools/cases_generator/opcode_metadata_generator.py index 24fbea6cfcb77a..ab597834a8892f 100644 --- a/Tools/cases_generator/opcode_metadata_generator.py +++ b/Tools/cases_generator/opcode_metadata_generator.py @@ -284,7 +284,7 @@ def is_viable_expansion(inst: Instruction) -> bool: continue if "replaced" in part.annotations: continue - if part.properties.tier_one_only or not part.is_viable(): + if part.properties.tier == 1 or not part.is_viable(): return False return True diff --git a/Tools/cases_generator/tier2_abstract_generator.py b/Tools/cases_generator/tier2_abstract_generator.py index 47a862643d987e..58c3110dc3facb 100644 --- a/Tools/cases_generator/tier2_abstract_generator.py +++ b/Tools/cases_generator/tier2_abstract_generator.py @@ -176,7 +176,7 @@ def generate_abstract_interpreter( if uop.name in abstract.uops: override = abstract.uops[uop.name] validate_uop(override, uop) - if uop.properties.tier_one_only: + if uop.properties.tier == 1: continue if uop.replicates: continue diff --git a/Tools/cases_generator/tier2_generator.py b/Tools/cases_generator/tier2_generator.py index 6fbe5c355c9083..d8eed1078b0914 100644 --- a/Tools/cases_generator/tier2_generator.py +++ b/Tools/cases_generator/tier2_generator.py @@ -194,7 +194,7 @@ def generate_tier2( out = CWriter(outfile, 2, lines) out.emit("\n") for name, uop in analysis.uops.items(): - if uop.properties.tier_one_only: + if uop.properties.tier == 1: continue if uop.properties.oparg_and_1: out.emit(f"/* {uop.name} is split on (oparg & 1) */\n\n") diff --git a/Tools/cases_generator/uop_id_generator.py b/Tools/cases_generator/uop_id_generator.py index 907158f2795959..eb5e3f4a324735 100644 --- a/Tools/cases_generator/uop_id_generator.py +++ b/Tools/cases_generator/uop_id_generator.py @@ -43,7 +43,7 @@ def generate_uop_ids( for name, uop in sorted(uops): if name in PRE_DEFINED: continue - if uop.properties.tier_one_only: + if uop.properties.tier == 1: continue if uop.implicitly_created and not distinct_namespace and not uop.replicated: out.emit(f"#define {name} {name[1:]}\n") diff --git a/Tools/cases_generator/uop_metadata_generator.py b/Tools/cases_generator/uop_metadata_generator.py index f85f1c6ce9c817..72eed3041c55c9 100644 --- a/Tools/cases_generator/uop_metadata_generator.py +++ b/Tools/cases_generator/uop_metadata_generator.py @@ -29,7 +29,7 @@ def generate_names_and_flags(analysis: Analysis, out: CWriter) -> None: out.emit("#ifdef NEED_OPCODE_METADATA\n") out.emit("const uint16_t _PyUop_Flags[MAX_UOP_ID+1] = {\n") for uop in analysis.uops.values(): - if uop.is_viable() and not uop.properties.tier_one_only: + if uop.is_viable() and uop.properties.tier != 1: out.emit(f"[{uop.name}] = {cflags(uop.properties)},\n") out.emit("};\n\n") @@ -41,7 +41,7 @@ def generate_names_and_flags(analysis: Analysis, out: CWriter) -> None: out.emit("};\n\n") out.emit("const char *const _PyOpcode_uop_name[MAX_UOP_ID+1] = {\n") for uop in sorted(analysis.uops.values(), key=lambda t: t.name): - if uop.is_viable() and not uop.properties.tier_one_only: + if uop.is_viable() and uop.properties.tier != 1: out.emit(f'[{uop.name}] = "{uop.name}",\n') out.emit("};\n") out.emit("#endif // NEED_OPCODE_METADATA\n\n") diff --git a/Tools/clinic/clinic.py b/Tools/clinic/clinic.py index 5d2617b3bd579f..7906e7c95d17ba 100755 --- a/Tools/clinic/clinic.py +++ b/Tools/clinic/clinic.py @@ -4069,8 +4069,6 @@ def parse_arg(self, argname: str, displayname: str, *, limited_capi: bool) -> st # mapping from arguments to format unit *and* registers the # legacy C converter for that format unit. # -ConverterKeywordDict = dict[str, TypeSet | bool] - def r(format_unit: str, *, accept: TypeSet, @@ -4086,7 +4084,7 @@ def r(format_unit: str, # # also don't add the converter for 's' because # the metaclass for CConverter adds it for us. - kwargs: ConverterKeywordDict = {} + kwargs: dict[str, Any] = {} if accept != {str}: kwargs['accept'] = accept if zeroes: diff --git a/configure b/configure index 73362c0c5b5f11..b565cdbfae1327 100755 --- a/configure +++ b/configure @@ -7462,11 +7462,15 @@ else # shared is disabled ;; esac fi +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $LDLIBRARY" >&5 +printf "%s\n" "$LDLIBRARY" >&6; } if test "$cross_compiling" = yes; then RUNSHARED= fi +{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking HOSTRUNNER" >&5 +printf %s "checking HOSTRUNNER... " >&6; } if test -z "$HOSTRUNNER" then @@ -7650,8 +7654,6 @@ fi esac fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking HOSTRUNNER" >&5 -printf %s "checking HOSTRUNNER... " >&6; } { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $HOSTRUNNER" >&5 printf "%s\n" "$HOSTRUNNER" >&6; } @@ -7659,9 +7661,6 @@ if test -n "$HOSTRUNNER"; then PYTHON_FOR_BUILD="_PYTHON_HOSTRUNNER='$HOSTRUNNER' $PYTHON_FOR_BUILD" fi -{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $LDLIBRARY" >&5 -printf "%s\n" "$LDLIBRARY" >&6; } - # LIBRARY_DEPS, LINK_PYTHON_OBJS and LINK_PYTHON_DEPS variable case $ac_sys_system/$ac_sys_emscripten_target in #( Emscripten/browser*) : @@ -16845,16 +16844,18 @@ printf "%s\n" "$ipv6type" >&6; } fi if test "$ipv6" = "yes" -a "$ipv6lib" != "none"; then + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: checking ipv6 library" >&5 +printf %s "checking ipv6 library... " >&6; } if test -d $ipv6libdir -a -f $ipv6libdir/lib$ipv6lib.a; then LIBS="-L$ipv6libdir -l$ipv6lib $LIBS" - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: using lib$ipv6lib" >&5 -printf "%s\n" "$as_me: using lib$ipv6lib" >&6;} + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: lib$ipv6lib" >&5 +printf "%s\n" "lib$ipv6lib" >&6; } else if test "x$ipv6trylibc" = xyes then : - { printf "%s\n" "$as_me:${as_lineno-$LINENO}: using libc" >&5 -printf "%s\n" "$as_me: using libc" >&6;} + { printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: libc" >&5 +printf "%s\n" "libc" >&6; } else $as_nop diff --git a/configure.ac b/configure.ac index 81f3f7753870f1..0694ef06364f82 100644 --- a/configure.ac +++ b/configure.ac @@ -1484,11 +1484,13 @@ else # shared is disabled ;; esac fi +AC_MSG_RESULT([$LDLIBRARY]) if test "$cross_compiling" = yes; then RUNSHARED= fi +AC_MSG_CHECKING([HOSTRUNNER]) AC_ARG_VAR([HOSTRUNNER], [Program to run CPython for the host platform]) if test -z "$HOSTRUNNER" then @@ -1534,7 +1536,6 @@ then ) fi AC_SUBST([HOSTRUNNER]) -AC_MSG_CHECKING([HOSTRUNNER]) AC_MSG_RESULT([$HOSTRUNNER]) if test -n "$HOSTRUNNER"; then @@ -1542,8 +1543,6 @@ if test -n "$HOSTRUNNER"; then PYTHON_FOR_BUILD="_PYTHON_HOSTRUNNER='$HOSTRUNNER' $PYTHON_FOR_BUILD" fi -AC_MSG_RESULT([$LDLIBRARY]) - # LIBRARY_DEPS, LINK_PYTHON_OBJS and LINK_PYTHON_DEPS variable AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], [Emscripten/browser*], [LIBRARY_DEPS='$(PY3LIBRARY) $(WASM_STDLIB) python.html python.worker.js'], @@ -4604,12 +4603,13 @@ yes fi if test "$ipv6" = "yes" -a "$ipv6lib" != "none"; then + AC_MSG_CHECKING([ipv6 library]) if test -d $ipv6libdir -a -f $ipv6libdir/lib$ipv6lib.a; then LIBS="-L$ipv6libdir -l$ipv6lib $LIBS" - AC_MSG_NOTICE([using lib$ipv6lib]) + AC_MSG_RESULT([lib$ipv6lib]) else AS_VAR_IF([ipv6trylibc], [yes], [ - AC_MSG_NOTICE([using libc]) + AC_MSG_RESULT([libc]) ], [ AC_MSG_ERROR([m4_normalize([ No $ipv6lib library found; cannot continue.