pybind11 python modes aren't supported as expected

[BetsyMcPhail]

What happened?

pybind11 supports two modes of finding python: the classic (deprecated) mode and the new (preferred) mode. See https://pybind11.readthedocs.io/en/stable/cmake/index.html#modes.

The logic that determines which mode to use is in https://github.com/pybind/pybind11/blob/v2.12/tools/pybind11Common.cmake. For CMake 3.27 and newer, the new mode is the default.

Within drake, pybind11-config.cmake effectively replaces pybind11Common.cmake.

Previously, pybind11-config.cmake always used the classic method. It first found Python using the classic mode (not part of pybind11Common.cmake) and then included pybind11Tools.cmake (same as pybind11Common.cmake).

In #21804, pybind11-config.cmake was updated to find python using the new mode but still included pybind11Tools.cmake.

This eventually caused issues in Mac drake external examples, and in #22012 the file was update to use pybind11NewTools.cmake (same as the new mode in pybind11Common.cmake).

Based on errors from drake-ros (RobotLocomotion/drake-ros#368), it was discovered that the method of finding python was incomplete, instead of find_package (Python3), find_package (Python3 COMPONENTS Interpreter Development) is needed. A workaround for that repo was to call find_package (Python3 COMPONENTS Interpreter Development) prior to finding pybind11.

Currently, pybind11Tools.cmake does not support the same options for finding Python as pybind11Common.cmake. Additionally, it is not even consistent within itself.

Version

1.34.0

What operating system are you using?

No response

What installation option are you using?

No response

Relevant log output

No response