Pybind11

This is the first version to fully support embedding the newly released Python 3.11.

Changes:

- Allow `pybind11::capsule` constructor to take null destructor pointers. 4221
- `embed.h` was changed so that `PYTHONPATH` is used also with Python 3.11 (established behavior). 4119
- A `PYBIND11_SIMPLE_GIL_MANAGEMENT` option was added (cmake, C++ define), along with many additional tests in `test_gil_scoped.py`. The option may be useful to try when debugging GIL-related issues, to determine if the more complex default implementation is or is not to blame. See \4216 for background. WARNING: Please be careful to not create ODR violations when using the option: everything that is linked together with mutual symbol visibility needs to be rebuilt. 4216
- `PYBIND11_EXPORT_EXCEPTION` was made non-empty only under macOS. This makes Linux builds safer, and enables the removal of warning suppression pragmas for Windows. 4298

Bug fixes:

- Fixed a bug where `UnicodeDecodeError` was not propagated from various `py::str` ctors when decoding surrogate utf characters. 4294
- Revert perfect forwarding for `make_iterator`. This broke at least one valid use case. May revisit later. 4234
- Fix support for safe casts to `void*` (regression in 2.10.0). 4275
- Fix `char8_t` support (regression in 2.9). 4278
- Unicode surrogate character in Python exception message leads to process termination in `error_already_set::what()`. 4297
- Fix MSVC 2019 v.1924 & C++14 mode error for `overload_cast`. 4188
- Make augmented assignment operators non-const for the object-api. Behavior was previously broken for augmented assignment operators. 4065
- Add proper error checking to C++ bindings for Python list append and insert. 4208
- Work-around for Nvidia's CUDA nvcc compiler in versions 11.4.0 - 11.8.0. 4220
- A workaround for PyPy was added in the `py::error_already_set` implementation, related to PR 1895 released with v2.10.0. 4079
- Fixed compiler errors when C++23 `std::forward_like` is available. 4136
- Properly raise exceptions in contains methods (like when an object in unhashable). 4209
- Further improve another error in exception handling. 4232
- `get_local_internals()` was made compatible with `finalize_interpreter()`, fixing potential freezes during interpreter finalization. 4192

Performance and style:

- Reserve space in set and STL map casters if possible. This will prevent unnecessary rehashing / resizing by knowing the number of keys ahead of time for Python to C++ casting. This improvement will greatly speed up the casting of large unordered maps and sets. 4194
- GIL RAII scopes are non-copyable to avoid potential bugs. 4183
- Explicitly default all relevant ctors for pytypes in the `PYBIND11_OBJECT` macros and enforce the clang-tidy checks `modernize-use-equals-default` in macros as well. 4017
- Optimize iterator advancement in C++ bindings. 4237
- Use the modern `PyObject_GenericGetDict` and `PyObject_GenericSetDict` for handling dynamic attribute dictionaries. 4106
- Document that users should use `PYBIND11_NAMESPACE` instead of using `pybind11` when opening namespaces. Using namespace declarations and namespace qualification remain the same as `pybind11`. This is done to ensure consistent symbol visibility. 4098
- Mark `detail::forward_like` as constexpr. 4147
- Optimize unpacking_collector when processing `arg_v` arguments. 4219
- Optimize casting C++ object to `None`. 4269

Build system improvements:

- CMake: revert overwrite behavior, now opt-in with `PYBIND11_PYTHONLIBS_OVERRWRITE OFF`. 4195
- Include a pkg-config file when installing pybind11, such as in the Python package. 4077
- Avoid stripping debug symbols when `CMAKE_BUILD_TYPE` is set to `DEBUG` instead of `Debug`. 4078
- Followup to 3948, fixing vcpkg again. 4123

Removed support for Python 2.7, Python 3.5, and MSVC 2015. Support for MSVC 2017 is limited due to availability of CI runners; we highly recommend MSVC 2019 or 2022 be used. Initial support added for Python 3.11.

New features:

- `py::anyset` & `py::frozenset` were added, with copying (cast) to `std::set` (similar to `set`). [3901](https://github.com/pybind/pybind11/pull/3901)
- Support bytearray casting to string. [3707](https://github.com/pybind/pybind11/pull/3707)
- `type_caster<std::monostate>` was added. `std::monostate` is a tag type that allows `std::variant` to act as an optional, or allows default construction of a `std::variant` holding a non-default constructible type. [3818](https://github.com/pybind/pybind11/pull/3818)
- `pybind11::capsule::set_name` added to mutate the name of the capsule instance. [3866](https://github.com/pybind/pybind11/pull/3866)
- NumPy: dtype constructor from type number added, accessors corresponding to Python API `dtype.num`, `dtype.byteorder`, `dtype.flags` and `dtype.alignment` added. [3868](https://github.com/pybind/pybind11/pull/3868)

Changes:

- Python 3.6 is now the minimum supported version. [3688](https://github.com/pybind/pybind11/pull/3688) [#3719](https://github.com/pybind/pybind11/pull/3719)
- The minimum version for MSVC is now 2017. [3722](https://github.com/pybind/pybind11/pull/3722)
- Fix issues with CPython 3.11 betas and add to supported test matrix. [3923](https://github.com/pybind/pybind11/pull/3923)
- `error_already_set` is now safer and more performant, especially for exceptions with long tracebacks, by delaying computation. [1895](https://github.com/pybind/pybind11/pull/1895)
- Improve exception handling in python `str` bindings. [3826](https://github.com/pybind/pybind11/pull/3826)
- The bindings for capsules now have more consistent exception handling. [3825](https://github.com/pybind/pybind11/pull/3825)
- `PYBIND11_OBJECT_CVT` and `PYBIND11_OBJECT_CVT_DEFAULT` macro can now be used to define classes in namespaces other than pybind11. [3797](https://github.com/pybind/pybind11/pull/3797)
- Error printing code now uses `PYBIND11_DETAILED_ERROR_MESSAGES` instead of requiring `NDEBUG`, allowing use with release builds if desired. [3913](https://github.com/pybind/pybind11/pull/3913)
- Implicit conversion of the literal `0` to `pybind11::handle` is now disabled. [4008](https://github.com/pybind/pybind11/pull/4008)

Bug fixes:

- Fix exception handling when `pybind11::weakref()` fails. [3739](https://github.com/pybind/pybind11/pull/3739)
- `module_::def_submodule` was missing proper error handling. This is fixed now. [3973](https://github.com/pybind/pybind11/pull/3973)
- The behavior or `error_already_set` was made safer and the highly opaque "Unknown internal error occurred" message was replaced with a more helpful message. [3982](https://github.com/pybind/pybind11/pull/3982)
- `error_already_set::what()` now handles non-normalized exceptions correctly. [3971](https://github.com/pybind/pybind11/pull/3971)
- Support older C++ compilers where filesystem is not yet part of the standard library and is instead included in `std::experimental::filesystem`. [3840](https://github.com/pybind/pybind11/pull/3840)
- Fix `-Wfree-nonheap-object` warnings produced by GCC by avoiding returning pointers to static objects with `return_value_policy::take_ownership`. [3946](https://github.com/pybind/pybind11/pull/3946)
- Fix cast from pytype rvalue to another pytype. [3949](https://github.com/pybind/pybind11/pull/3949)
- Ensure proper behavior when garbage collecting classes with dynamic attributes in Python \>=3.9. [4051](https://github.com/pybind/pybind11/pull/4051)
- A couple long-standing `PYBIND11_NAMESPACE` `__attribute__((visibility("hidden")))` inconsistencies are now fixed (affects only unusual environments). [4043](https://github.com/pybind/pybind11/pull/4043)
- `pybind11::detail::get_internals()` is now resilient to in-flight Python exceptions. [3981](https://github.com/pybind/pybind11/pull/3981)
- Arrays with a dimension of size 0 are now properly converted to dynamic Eigen matrices (more common in NumPy 1.23). [4038](https://github.com/pybind/pybind11/pull/4038)
- Avoid catching unrelated errors when importing NumPy. [3974](https://github.com/pybind/pybind11/pull/3974)

Performance and style:

- Added an accessor overload of `(object &&key)` to reference steal the object when using python types as keys. This prevents unnecessary reference count overhead for attr, dictionary, tuple, and sequence look ups. Added additional regression tests. Fixed a performance bug the caused accessor assignments to potentially perform unnecessary copies. [3970](https://github.com/pybind/pybind11/pull/3970)
- Perfect forward all args of `make_iterator`. [3980](https://github.com/pybind/pybind11/pull/3980)
- Avoid potential bug in pycapsule destructor by adding an `error_guard` to one of the dtors. [3958](https://github.com/pybind/pybind11/pull/3958)
- Optimize dictionary access in `strip_padding` for numpy. [3994](https://github.com/pybind/pybind11/pull/3994)
- `stl_bind.h` bindings now take slice args as a const-ref. [3852](https://github.com/pybind/pybind11/pull/3852)
- Made slice constructor more consistent, and improve performance of some casters by allowing reference stealing. [3845](https://github.com/pybind/pybind11/pull/3845)
- Change numpy dtype from_args method to use const ref. [3878](https://github.com/pybind/pybind11/pull/3878)
- Follow rule of three to ensure `PyErr_Restore` is called only once. [3872](https://github.com/pybind/pybind11/pull/3872)
- Added missing perfect forwarding for `make_iterator` functions. [3860](https://github.com/pybind/pybind11/pull/3860)
- Optimize c++ to python function casting by using the rvalue caster. [3966](https://github.com/pybind/pybind11/pull/3966)
- Optimize Eigen sparse matrix casting by removing unnecessary temporary. [4064](https://github.com/pybind/pybind11/pull/4064)
- Avoid potential implicit copy/assignment constructors causing double free in `strdup_gaurd`. [3905](https://github.com/pybind/pybind11/pull/3905)
- Enable clang-tidy checks `misc-definitions-in-headers`, `modernize-loop-convert`, and `modernize-use-nullptr`. [3881](https://github.com/pybind/pybind11/pull/3881) [#3988](https://github.com/pybind/pybind11/pull/3988)

Build system improvements:

- CMake: Fix file extension on Windows with cp36 and cp37 using FindPython. [3919](https://github.com/pybind/pybind11/pull/3919)
- CMake: Support multiple Python targets (such as on vcpkg). [3948](https://github.com/pybind/pybind11/pull/3948)
- CMake: Fix issue with NVCC on Windows. [3947](https://github.com/pybind/pybind11/pull/3947)
- CMake: Drop the bitness check on cross compiles (like targeting WebAssembly via Emscripten). [3959](https://github.com/pybind/pybind11/pull/3959)
- Add MSVC builds in debug mode to CI. [3784](https://github.com/pybind/pybind11/pull/3784)
- MSVC 2022 C++20 coverage was added to GitHub Actions, including Eigen. [3732](https://github.com/pybind/pybind11/pull/3732), [#3741](https://github.com/pybind/pybind11/pull/3741)

Backend and tidying up:

- New theme for the documentation. [3109](https://github.com/pybind/pybind11/pull/3109)
- Remove idioms in code comments. Use more inclusive language. [3809](https://github.com/pybind/pybind11/pull/3809)
- `include <iostream>` was removed from the `pybind11/stl.h` header. Your project may break if it has a transitive dependency on this include. The fix is to "Include What You Use". [3928](https://github.com/pybind/pybind11/pull/3928)
- Avoid `setup.py <command>` usage in internal tests. [3734](https://github.com/pybind/pybind11/pull/3734)

Changes:

- Enum now has an `__index__` method on Python \<3.8 too. [\3700](https://github.com/pybind/pybind11/pull/3700)
- Local internals are now cleared after finalizing the interpreter. [\3744](https://github.com/pybind/pybind11/pull/3744)

Bug fixes:

- Better support for Python 3.11 alphas. [\3694](https://github.com/pybind/pybind11/pull/3694)
- `PYBIND11_TYPE_CASTER` now uses fully qualified symbols, so it can be used outside of `pybind11::detail`. [\3758](https://github.com/pybind/pybind11/pull/3758)
- Some fixes for PyPy 3.9. [\3768](https://github.com/pybind/pybind11/pull/3768)
- Fixed a potential memleak in PyPy in `get_type_override`. [\3774](https://github.com/pybind/pybind11/pull/3774)
- Fix usage of `VISIBILITY_INLINES_HIDDEN`. [\3721](https://github.com/pybind/pybind11/pull/3721)

Build system improvements:

- Uses `sysconfig` module to determine installation locations on Python \>= 3.10, instead of `distutils` which has been deprecated. [\3764](https://github.com/pybind/pybind11/pull/3764)
- Support Catch 2.13.5+ (supporting GLIBC 2.34+). [\3679](https://github.com/pybind/pybind11/pull/3679)
- Fix test failures with numpy 1.22 by ignoring whitespace when comparing `str()` of dtypes. [\3682](https://github.com/pybind/pybind11/pull/3682)

Backend and tidying up:

- clang-tidy: added `readability-qualified-auto`, `readability-braces-around-statements`, `cppcoreguidelines-prefer-member-initializer`, `clang-analyzer-optin.performance.Padding`, `cppcoreguidelines-pro-type-static-cast-downcast`, and `readability-inconsistent-declaration-parameter-name`. [\3702](https://github.com/pybind/pybind11/pull/3702), [\#3699](https://github.com/pybind/pybind11/pull/3699), [\#3716](https://github.com/pybind/pybind11/pull/3716), [\#3709](https://github.com/pybind/pybind11/pull/3709)
- clang-format was added to the pre-commit actions, and the entire code base automatically reformatted (after several iterations preparing for this leap). [\3713](https://github.com/pybind/pybind11/pull/3713)

Changes:

- If possible, attach Python exception with `py::raise_from` to `TypeError` when casting from C++ to Python. This will give additional info if Python exceptions occur in the caster. Adds a test case of trying to convert a set from C++ to Python when the hash function is not defined in Python. [\3605](https://github.com/pybind/pybind11/pull/3605)
- Add a mapping of C++11 nested exceptions to their Python exception equivalent using `py::raise_from`. This attaches the nested exceptions in Python using the `__cause__` field. [\3608](https://github.com/pybind/pybind11/pull/3608)
- Propagate Python exception traceback using `raise_from` if a pybind11 function runs out of overloads. [\3671](https://github.com/pybind/pybind11/pull/3671)
- `py::multiple_inheritance` is now only needed when C++ bases are hidden from pybind11. [\3650](https://github.com/pybind/pybind11/pull/3650) and [\#3659](https://github.com/pybind/pybind11/pull/3659)

Bug fixes:

- Remove a boolean cast in `numpy.h` that causes MSVC C4800 warnings when compiling against Python 3.10 or newer. [\3669](https://github.com/pybind/pybind11/pull/3669)
- Render `py::bool_` and `py::float_` as `bool` and `float` respectively. [\3622](https://github.com/pybind/pybind11/pull/3622)

Build system improvements:

- Fix CMake extension suffix computation on Python 3.10+. [\3663](https://github.com/pybind/pybind11/pull/3663)
- Allow `CMAKE_ARGS` to override CMake args in pybind11's own `setup.py`. [\3577](https://github.com/pybind/pybind11/pull/3577)
- Remove a few deprecated c-headers. [\3610](https://github.com/pybind/pybind11/pull/3610)
- More uniform handling of test targets. [\3590](https://github.com/pybind/pybind11/pull/3590)
- Add clang-tidy readability check to catch potentially swapped function args. [\3611](https://github.com/pybind/pybind11/pull/3611)

This is the last version to support Python 2.7 and 3.5.

New Features:

- Allow `py::args` to be followed by other arguments; the remaining arguments are implicitly keyword-only, as if a `py::kw_only{}` annotation had been used. [3402](https://github.com/pybind/pybind11/pull/3402)

Changes:

- Make str/bytes/memoryview more interoperable with `std::string_view`. [3521](https://github.com/pybind/pybind11/pull/3521)
- Replace `_` with `const_name` in internals, avoid defining `pybind::_` if `_` defined as macro (common gettext usage) [3423](https://github.com/pybind/pybind11/pull/3423)

Bug fixes:

- Fix a rare warning about extra copy in an Eigen constructor. [3486](https://github.com/pybind/pybind11/pull/3486)
- Fix caching of the C++ overrides. [3465](https://github.com/pybind/pybind11/pull/3465)
- Add missing `std::forward` calls to some `cpp_function` overloads. [3443](https://github.com/pybind/pybind11/pull/3443)
- Support PyPy 7.3.7 and the PyPy3.8 beta. Test python-3.11 on PRs with the `python dev` label. [3419](https://github.com/pybind/pybind11/pull/3419)
- Replace usage of deprecated `Eigen::MappedSparseMatrix` with `Eigen::Map<Eigen::SparseMatrix<...>>` for Eigen 3.3+. [3499](https://github.com/pybind/pybind11/pull/3499)
- Tweaks to support Microsoft Visual Studio 2022. [3497](https://github.com/pybind/pybind11/pull/3497)

Build system improvements:

- Nicer CMake printout and IDE organisation for pybind11's own tests. [3479](https://github.com/pybind/pybind11/pull/3479)
- CMake: report version type as part of the version string to avoid a spurious space in the package status message. [3472](https://github.com/pybind/pybind11/pull/3472)
- Flags starting with `-g` in `$CFLAGS` and `$CPPFLAGS` are no longer overridden by `.Pybind11Extension`. [3436](https://github.com/pybind/pybind11/pull/3436)
- Ensure ThreadPool is closed in `setup_helpers`. [3548](https://github.com/pybind/pybind11/pull/3548)
- Avoid LTS on `mips64` and `ppc64le` (reported broken). [3557](https://github.com/pybind/pybind11/pull/3557)

Changes and additions:
- The simple namespace creation shortcut added in 2.8.0 was deprecated due to usage of CPython internal API, and will be removed soon. Use `py::module_::import("types").attr("SimpleNamespace")`. [\3374](https://github.com/pybinyyd/pybind11/pull/3374)
- Add C++ Exception type to throw and catch `AttributeError`. Useful for defining custom `__setattr__` and `__getattr__` methods. [\3387](https://github.com/pybind/pybind11/pull/3387)

Fixes:
- Fixed the potential for dangling references when using properties with `std::optional` types. [\3376](https://github.com/pybind/pybind11/pull/3376)
- Modernize usage of `PyCodeObject` on Python 3.9+ (moving toward support for Python 3.11a1) [\3368](https://github.com/pybind/pybind11/pull/3368)
- A long-standing bug in `eigen.h` was fixed (originally PR \3343). The bug was unmasked by newly added `static_assert`'s in the Eigen 3.4.0 release. [\3352](https://github.com/pybind/pybind11/pull/3352)
- Support multiple raw inclusion of CMake helper files (Conan.io does this for multi-config generators). [\3420](https://github.com/pybind/pybind11/pull/3420)
- Fix harmless warning on upcoming CMake 3.22. [\3368](https://github.com/pybind/pybind11/pull/3368)
- Fix 2.8.0 regression with MSVC 2017 + C++17 mode + Python 3. [\3407](https://github.com/pybind/pybind11/pull/3407)
- Fix 2.8.0 regression that caused undefined behavior (typically segfaults) in `make_key_iterator`/`make_value_iterator` if dereferencing the iterator returned a temporary value instead of a reference. [\3348](https://github.com/pybind/pybind11/pull/3348)