`PYBIND11_PLATFORM_ABI_ID` Modernization Continued (WIP) #5439

rwgk · 2024-11-10T23:34:48Z

Description

WORK IN PROGRESS

PR #4953 already modernized the PYBIND11_BUILD_ABI for MSVC.

To be addressed in this PR:

Fix MSVC MT/MD incompatibility in PYBIND11_BUILD_ABI #4953 (comment)
Ideas from Build platform ABI tag improvements #5437
Maybe: Fix MSVC MT/MD incompatibility in PYBIND11_BUILD_ABI #4953 (comment)
Maybe: Also simplify the code setting the PYBIND11_INTERNALS_VERSION

Suggested changelog entry:

…bind#4953 and PR pybind#5437

rwgk · 2024-11-11T00:43:16Z

With this PR at commit 14f8425 and logs_30720380481.zip downloaded from here:

https://github.com/pybind/pybind11/actions/runs/11769449932/job/32780487991?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c

     26 __pybind11_internals_vX_clang_libcpp_libcpp1__
     10 __pybind11_internals_vX_clang_libstdcpp_usecxx111__
     40 __pybind11_internals_vX_gcc_libstdcpp_usecxx111__
      2 __pybind11_internals_vX_icc_libstdcpp_usecxx111__
      6 __pybind11_internals_vX_mingw_libstdcpp_usecxx111__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1941__
      1 __pybind11_internals_vX_pgi_libstdcpp__

This doesn't look right, _LIBCPP_ABI_VERSION and _GLIBCXX_USE_CXX11_ABI always appear to be 1.

@robertmaynard I was trying to follow your #4953 (comment) but maybe I'm misunderstanding?

(I want to leave NVHPC for later.)

@wjakob @henryiii @isuruf @cryos for awareness.

robertmaynard · 2024-11-12T17:57:52Z

With this PR at commit 14f8425 and logs_30720380481.zip downloaded from here:

https://github.com/pybind/pybind11/actions/runs/11769449932/job/32780487991?pr=5439
$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c
     26 __pybind11_internals_vX_clang_libcpp_libcpp1__
     10 __pybind11_internals_vX_clang_libstdcpp_usecxx111__
     40 __pybind11_internals_vX_gcc_libstdcpp_usecxx111__
      2 __pybind11_internals_vX_icc_libstdcpp_usecxx111__
      6 __pybind11_internals_vX_mingw_libstdcpp_usecxx111__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1941__
      1 __pybind11_internals_vX_pgi_libstdcpp__
This doesn't look right, _LIBCPP_ABI_VERSION and _GLIBCXX_USE_CXX11_ABI always appear to be 1.

@robertmaynard I was trying to follow your #4953 (comment) but maybe I'm misunderstanding?

(I want to leave NVHPC for later.)

@wjakob @henryiii @isuruf @cryos for awareness.

_GLIBCXX_USE_CXX11_ABI is a boolean which represents which ABI for a set of the C++ std containers you are using ( std::string, list? ). It is an independent axis compared to __GXX_ABI_VERSION.

__GXX_ABI_VERSION	_GLIBCXX_USE_CXX11_ABI	Notes
102	0	Only compatible with 102. Predates _GLIBCXX_USE_CXX11_ABI
1002	0	Compatible with >= 1002 && < 2000 + CXX11 abi of 0
1002	1	Compatible with >= 1002 && < 2000 + CXX11 abi of 1
1013	0	Compatible with >= 1002 && < 2000 + CXX11 abi of 0
1013	1	Compatible with >= 1002 && < 2000 + CXX11 abi of 1

This presumes that GCC will rev the major value of __GXX_ABI_VERSION when it next breaks ABI compat

isuruf · 2024-11-12T18:05:00Z

One more thing to fix would be to remove the compiler id _gcc or _clang as they are compatible with each other. I think the current checks are supposed to protect against the C standard library, but is too broad.

wjakob · 2024-11-15T05:24:49Z

I am surprised by __pybind11_internals_vX_msvc_mt_mscver1941__ -- how does the current code manage to generate the number 1941? (I thought 19 was hardcoded).

rwgk · 2024-11-15T05:34:30Z

I am surprised by __pybind11_internals_vX_msvc_mt_mscver1941__ -- how does the current code manage to generate the number 1941? (I thought 19 was hardcoded).

It's generated in this line:

pybind11/include/pybind11/conduit/pybind11_platform_abi_id.h

Line 67 in 08095d9

    
           #            define PYBIND11_BUILD_ABI "_mt_mscver" PYBIND11_PLATFORM_ABI_ID_TOSTRING(_MSC_VER)

wjakob · 2024-11-17T15:36:05Z

Ah, interesting. So for /MT, the whole _MSCVER should be included? @isuruf, can you confirm? I am not doing this in nanobind currently and will have to make a change if so.

isuruf · 2024-11-18T16:05:04Z

Yes. /MT means a static linkage of the C runtime, and it's better to have the full version.

Further constrain to GXX_ABI 1002 or greater and less than 2000, hopefully future proof by summarizing that as `1` along with CXX11 on or off.

cryos · 2024-11-20T21:52:45Z

I just pushed a new commit that produces something like

__pybind11_internals_v5_gcc_libstdcpp_gxx_abi_1_usecxx11_1__

It also ensures __GXX_ABI_VERSION is less than 2000, so labeling gxx_abi_1, maybe gxx_abi_1xxx would be better. The thought process was to include that as well should it be bumped in the future.

rwgk · 2024-11-20T22:54:37Z

Here are the results extracted from the latest CI logs:

https://github.com/pybind/pybind11/actions/runs/11942204141

(Click on any job, then the gear that appears in the top-right corner of the window with the log for that job, then "Download log archive".)

$ mkdir junk && cd junk
$ unzip ../logs_31139436091.zip
$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c

     26 __pybind11_internals_vX_clang_libcpp_libcpp1__
     10 __pybind11_internals_vX_clang_libstdcpp_gxx_abi_1_usecxx11_1__
     40 __pybind11_internals_vX_gcc_libstdcpp_gxx_abi_1_usecxx11_1__
      2 __pybind11_internals_vX_icc_libstdcpp_gxx_abi_1_usecxx11_1__
      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1_usecxx11_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
      1 __pybind11_internals_vX_pgi_libstdcpp__

On chat @cryos and I decided to go with gxx_abi_1xxx_ instead of gxx_abi_1_, for clarity.

Looking at these:

__pybind11_internals_vX_clang_libstdcpp_gxx_abi_1_usecxx11_1__
__pybind11_internals_vX_gcc_libstdcpp_gxx_abi_1_usecxx11_1__

I think we want to omit clang and gcc here, based on my understanding of suggestions by @robertmaynard and @isuruf.

But what about
__pybind11_internals_vX_icc_libstdcpp_gxx_abi_1_usecxx11_1__
?

Does anyone know if icc-compiled code is ABI-compatible with clang and gcc?

And what should we do about this one? __pybind11_internals_vX_clang_libcpp_libcpp1__

Could/should we just omit these lines

#    elif defined(_LIBCPP_ABI_VERSION) // https://libcxx.llvm.org/DesignDocs/ABIVersioning.html
#        define PYBIND11_BUILD_ABI "_libcpp" PYBIND11_PLATFORM_ABI_ID_TOSTRING(_LIBCPP_ABI_VERSION)

and rely on the new __GXX_ABI_VERSION, _GLIBCXX_USE_CXX11_ABI logic instead?

After discussions with Ralf Grosse-Kunstleve we think these would make good identifiers that are concise and clear.

Within the `__GXX_ABI_VERSION` block this should always be defined, guard against unexpected defines and make the error obvious.

rwgk · 2024-11-21T02:27:55Z

Here are the results extracted from the latest CI logs:

https://github.com/pybind/pybind11/actions/runs/11944671063/job/33296033078?pr=5439 (attempt 1, 1 flake)
https://github.com/pybind/pybind11/actions/runs/11944671063?pr=5439 (deflaked)

     26 __pybind11_internals_vX_clang_libcpp_abi1__
     10 __pybind11_internals_vX_clang_libstdcpp_gxx_abi_1xxx_usecxx11_1__
     42 __pybind11_internals_vX_gcc_libstdcpp_gxx_abi_1xxx_usecxx11_1__
      2 __pybind11_internals_vX_icc_libstdcpp_gxx_abi_1xxx_usecxx11_1__
      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_usecxx11_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
      1 __pybind11_internals_vX_pgi_libstdcpp__

rwgk · 2024-11-24T20:39:26Z

The current changes affect icc, therefore it'll be best if we settle the PYBIND11_PLATFORM_ABI_ID also for that compiler in this PR. According to

https://www.intel.com/content/www/us/en/docs/dpcpp-cpp-compiler/developer-guide-reference/2023-2/gcc-compatibility-and-interoperability.html

"C language object files created with the compiler are binary compatible with the GCC and C/C++ language library."

AFAICT, the compiler also links against the systemlibstdcpp, therefore I figure it must be binary compatible for C++ as well.

Upshot: I think we should also omit icc in the PYBIND11_PLATFORM_ABI_ID.

…X_USE_CXX11_ABI` (similarly to `gxx_abi` for `__GXX_ABI_VERSION`).

…`_gcc` with `_system`

rwgk · 2024-11-24T21:16:20Z

I just pushed up the experimental commit fe2dbcb which changes PYBIND11_COMPILER_TYPE.

Rationale:

Test for special cases first, system compiler last.
Replace "unknown" fall-through with #error, similar to the elimination of fall-throughs for PYBIND11_BUILD_ABI. (Possibly we may want to enable fall-throughs for releases, but always #error in the pybind11 CI jobs.)
Assume __MINGW32__ and __CYGWIN__ are not binary compatible with MSVC.
Use _system for gcc, clang, icc.
Leave __PGI untouched for now. (Possibly after we're sure about gcc, clang, icc, we can also settle the case for PGI in this PR.)

I'll post a new PYBIND11_PLATFORM_ABI_ID overview when all jobs are done and green.

rwgk · 2024-11-25T04:01:34Z

Latest PYBIND11_INTERNALS_ID overview (this PR at commit fe2dbcb):

logs_31272968106.zip — https://github.com/pybind/pybind11/actions/runs/11999580827/job/33447666543?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | c
ut -d' ' -f1 | sort | uniq -c

      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
      1 __pybind11_internals_vX_pgi_libstdcpp__
     26 __pybind11_internals_vX_system_libcpp_abi1__
     52 __pybind11_internals_vX_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__

rwgk · 2024-11-25T04:20:21Z

@robertmaynard do you have a moment to look at this PR? (The changes are small and all in just one file.)

Is the PYBIND11_COMPILER_TYPE and PYBIND11_BUILD_ABI logic good as-is for gcc and clang (Linux, macOS)?
Do we have to worry about _DEBUG under Linux, macOS?
Should we leave NVHPC (PGI) for later, or try to handle that in this PR as well?

robertmaynard · 2024-11-25T19:16:25Z

@robertmaynard do you have a moment to look at this PR? (The changes are small and all in just one file.)

Is the PYBIND11_COMPILER_TYPE and PYBIND11_BUILD_ABI logic good as-is for gcc and clang (Linux, macOS)?

Looks good to me

Do we have to worry about _DEBUG under Linux, macOS?

_DEBUG is a MSVCism. Linux and macOS have NDEBUG but that doesn't change ABI.

Should we leave NVHPC (PGI) for later, or try to handle that in this PR as well?

NVHPC should be ABI compatible with gcc. I am fine either way in how you want to manage that integration from a PR perspective.

@robertmaynard

…piler. See comment by @robertmaynard: pybind#5439 (comment)

rwgk · 2024-11-25T19:40:40Z

Thanks a lot @robertmaynard for the review!

NVHPC should be ABI compatible with gcc.

Based on your comment, I added commit 9fc9515:

Add NVHPC (__PGI) to the list of compilers (assumed to be) compatible with system compiler.

…block. Also add comment pointing to this PR (pybind#5439).

…ILD_ABI block." This reverts commit d412303.

…stem compiler." This reverts commit 9fc9515.

…CXX_USE_CXX11_ABI

rwgk · 2024-11-26T08:49:44Z

Add NVHPC (__PGI) to the list of compilers (assumed to be) compatible with system compiler.

I had to backtrack (b6ccce3, a584398), after spending more time on this than I'd like to admit.

I determined conclusively that nvc++ does not define __GXX_ABI_VERSION, even though it links against

libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6

Without a way to obtain __GXX_ABI_VERSION, my idea that we can simply use the same approach as for gcc, clang, icc falls flat. :-(

For now I settled on 23a5f2b, which is probably (?) what @robertmaynard suggested previously.

rwgk · 2024-11-26T08:55:27Z

Latest PYBIND11_INTERNALS_ID overview (this PR at commit 23a5f2b):

logs_31336371389.zip — https://github.com/pybind/pybind11/actions/runs/12026563895/job/33525815426?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | c
ut -d' ' -f1 | sort | uniq -c

      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
      1 __pybind11_internals_vX_pgi_libstdcpp_gnuc_9_4_use_cxx11_abi_1__
     26 __pybind11_internals_vX_system_libcpp_abi1__
     52 __pybind11_internals_vX_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__

…C is always in the 1xxx ABI family.

rwgk · 2024-11-26T22:03:44Z

Latest PYBIND11_INTERNALS_ID overview (this PR at commit 8fa10bf):

logs_31368853117.zip — https://github.com/pybind/pybind11/actions/runs/12039062120/job/33566108736?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | c
ut -d' ' -f1 | sort | uniq -c

      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
     26 __pybind11_internals_vX_system_libcpp_abi1__
     53 __pybind11_internals_vX_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__

rwgk · 2024-11-26T22:56:49Z

@dkolsen-pgi wrote on chat (thanks a lot!):

NVC++ is ABI-compatible with GCC. If you encounter any ABI differences between the two in compiler-generated code (as opposed to library things), that is considered a bug in NVC++.
NVC++ always uses libstdc++ as its standard library. You should be able to check the libstdc++ related macros to figure out its ABI.
The macro __PGI is obsolete. Don't use it anymore. It has been replaced by __NVCOMPILER.

I used this command to produce a list of all pre-defined preprocessor macros (NVHPC 23.5):

nvc++ -stdpar -cuda -acc -mp -target=multicore -dM -E /dev/null

__GXX_ABI_VERSION is not in the list. I looked very carefully but couldn't figure out how to derive the ABI version from the available macros.

I settled on this approach (@ commit 02daf15):

#    elif defined(_GLIBCXX_USE_CXX11_ABI) // See PR #5439.                      
#        if defined(__NVCOMPILER)                                               
//           // Assume that NVHPC is in the 1xxx ABI family.                    
//           // THIS ASSUMPTION IS NOT FUTURE PROOF but apparently the best we can do.
//           // Please let us know if there is a way to validate the assumption here.
#        elif !defined(__GXX_ABI_VERSION)                                       
#            error                                                                                \
                "Unknown platform or compiler (_GLIBCXX_USE_CXX11_ABI): PLEASE REVISE THIS CODE."
#        endif                                                                  
#        if defined(__GXX_ABI_VERSION) && __GXX_ABI_VERSION < 1002 || __GXX_ABI_VERSION >= 2000
#            error "Unknown platform or compiler (__GXX_ABI_VERSION): PLEASE REVISE THIS CODE."
#        endif                                                                  
#        define PYBIND11_BUILD_ABI                                                                \
            "_gxx_abi_1xxx_use_cxx11_abi_" PYBIND11_PLATFORM_ABI_ID_TOSTRING(                     \
                _GLIBCXX_USE_CXX11_ABI)                                         
#    else                                                                       
#        error "Unknown platform or compiler: PLEASE REVISE THIS CODE."         
#    endif

rwgk · 2024-11-26T23:28:10Z

Hi @isuruf, could you please help with a review of the latest state? Do you think this will work for Conda?

The latest PYBIND11_INTERNALS_ID overview (this PR at commit 02daf15) is still identical to what's shown under #5439 (comment).

wjakob · 2024-11-27T12:33:14Z

Two comments about this PR: Can we remove PYBIND11_INTERNALS_KIND? It is completely unclear what it does, undocumented, and I cannot find any place that actually uses it.

Could the PYBIND11_PLATFORM_ABI_ID be defined so that it does not start with an underscore? (By moving that underscore to the PYBIND11_INTERNALS_ID).

rwgk · 2024-11-29T20:21:05Z

Two comments about this PR: Can we remove PYBIND11_INTERNALS_KIND? It is completely unclear what it does, undocumented, and I cannot find any place that actually uses it.

Could the PYBIND11_PLATFORM_ABI_ID be defined so that it does not start with an underscore? (By moving that underscore to the PYBIND11_INTERNALS_ID).

I want get rid of it, too, but I felt hesitant because

there is a (very small) risk of that being a breaking change for someone,
it's orthogonal to what this PR is about, and
ultimately it's a tiny and inconsequential (to pybind11 development efforts) cleanup.

Regarding the very small risk, I used github search code to look for PYBIND11_INTERNALS_KIND. The vast majority of the 494 Code matches are simply clones of internals.h, but there is also this

https://github.com/pytorch/pytorch/blob/7dd9b5fc4343d101294dbbab4b4172f2859460bc/torch/utils/cpp_extension.py#L1376-L1378

and many clones of that. But then again, that only mentions PYBIND11_INTERNALS_KIND and doesn't actually manipulate it. I didn't find anything else, so removing PYBIND11_INTERNALS_KIND should be fine.

I believe the best approach is to remove it in a follow-on PR. I'll do right after this PR is merged. That way it'll be easier to refer to and discuss the removal, in case there are breakages.

wjakob · 2024-12-01T12:09:20Z

I believe the best approach is to remove it in a follow-on PR. I'll do right after this PR is merged. That way it'll be easier to refer to and discuss the removal, in case there are breakages.

Great, that sounds like a good plan.

Regarding this message:

Could the PYBIND11_PLATFORM_ABI_ID be defined so that it does not start with an underscore? (By moving that underscore to the PYBIND11_INTERNALS_ID).

The leading underscore is an artifact of string concatenation in PYBIND11_INTERNALS_ID, where the ABI and pybind11 data structure version are combined. But it's weird to have the leading underscore in PYBIND11_INTERNALS_ID, especially if it is to be used as a marker to signal binary compatibility with other frameworks in the future.

rwgk · 2024-12-01T17:58:16Z

The leading underscore is an artifact of string concatenation in PYBIND11_INTERNALS_ID, where the ABI and pybind11 data structure version are combined. But it's weird to have the leading underscore in PYBIND11_INTERNALS_ID, especially if it is to be used as a marker to signal binary compatibility with other frameworks in the future.

I totally agree. Sorry this slipped my mind before. I'll try to fold that into this PR.

rwgk · 2024-12-01T19:10:40Z

Getting rid of the leading underscore isn't so easy, unfortunately, mainly because I found many manipulations of PYBIND11_COMPILER_TYPE (and PYBIND11_STDLIB, PYBIND11_BUILD_ABI) in the wild.

I'm guessing people found themselves forced to introduce those manipulations exactly because of the problems fixed by this PR. Here is one randomly picked example (out of nearly 600 build.ninja file paths produced by github search code):

https://github.com/GANWANSHUI/GaussianOcc/blob/db0b37e341bef841ea7550a6a9c0c8e5f80b98cc/lib/dvr/build.ninja#L5-L7

cflags = -DTORCH_EXTENSION_NAME=dvr -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isstem /uge_mnt/home/wsgan/.raiden/aip-pytorch-2105-gl1mesa-1/lib/python3.8/site-packages/torch/include -isystem /uge_mnt/home/wsgan/.raiden/aip-pytorch-2105-gl1mesa-1/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /uge_mnt/home/wsgan/.raiden/aip-pytorch-2105-gl1mesa-1/lib/python3.8/site-packages/torch/include/TH -isystem /uge_mnt/home/wsgan/.raiden/aip-pytorch-2105-gl1mesa-1/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14
post_cflags = 
cuda_cflags = -DTORCH_EXTENSION_NAME=dvr -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /uge_mnt/home/wsgan/.raiden/aip-pytorch-2105-gl1mesa-1/lib/python3.8/site-packages/torch/include -isystem /uge_mnt/home/wsgan/.raiden/aip-pytorch-2105-gl1mesa-1/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /uge_mnt/home/wsgan/.raiden/aip-pytorch-2105-gl1mesa-1/lib/python3.8/site-packages/torch/include/TH -isystem /uge_mnt/home/wsgan/.raiden/aip-pytorch-2105-gl1mesa-1/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -allow-unsupported-compiler -std=c++14

Now, assuming PYBIND11_INTERNALS_KIND is gone, and we do something like this ...

diff --git a/include/pybind11/conduit/pybind11_platform_abi_id.h b/include/pybind11/conduit/pybind11_platform_abi_id.h
index af9a68ff..a44bd7f4 100644
--- a/include/pybind11/conduit/pybind11_platform_abi_id.h
+++ b/include/pybind11/conduit/pybind11_platform_abi_id.h
@@ -24,13 +24,13 @@
 // compiler is compatible.
 #ifndef PYBIND11_COMPILER_TYPE
 #    if defined(__MINGW32__)
-#        define PYBIND11_COMPILER_TYPE "_mingw"
+#        define PYBIND11_COMPILER_TYPE "mingw"
 #    elif defined(__CYGWIN__)
-#        define PYBIND11_COMPILER_TYPE "_gcc_cygwin"
+#        define PYBIND11_COMPILER_TYPE "gcc_cygwin"
 #    elif defined(_MSC_VER)
-#        define PYBIND11_COMPILER_TYPE "_msvc"
+#        define PYBIND11_COMPILER_TYPE "msvc"
 #    elif defined(__INTEL_COMPILER) || defined(__clang__) || defined(__GNUC__)
-#        define PYBIND11_COMPILER_TYPE "_system" // Assumed compatible with system compiler.
+#        define PYBIND11_COMPILER_TYPE "system" // Assumed compatible with system compiler.
 #    else
 #        error "Unknown PYBIND11_COMPILER_TYPE: PLEASE REVISE THIS CODE."
 #    endif
diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h
index 278f35bb..037f59e0 100644
--- a/include/pybind11/detail/internals.h
+++ b/include/pybind11/detail/internals.h
@@ -272,11 +272,11 @@ struct type_info {
 
 #define PYBIND11_INTERNALS_ID                                                                     \
     "__pybind11_internals_v" PYBIND11_TOSTRING(PYBIND11_INTERNALS_VERSION)                        \
-        PYBIND11_PLATFORM_ABI_ID "__"
+        "_" PYBIND11_PLATFORM_ABI_ID "__"
 
 #define PYBIND11_MODULE_LOCAL_ID                                                                  \
     "__pybind11_module_local_v" PYBIND11_TOSTRING(PYBIND11_INTERNALS_VERSION)                     \
-        PYBIND11_PLATFORM_ABI_ID "__"
+        "_" PYBIND11_PLATFORM_ABI_ID "__"
 
 /// Each module locally stores a pointer to the `internals` data. The data
 /// itself is shared among modules with the same `PYBIND11_INTERNALS_ID`.

... we'll essentially break manipulations like above, because the PYBIND11_INTERNALS_ID will have an extra underscore.

On balance I'd say, the leading underscore is annoying, but removing it isn't worth such troubles. Unless we find some other compromise solution?

…d compatibility.

…ro definitions is the same as in the list defining `PYBIND11_PLATFORM_ABI_ID`

rwgk · 2024-12-01T20:52:56Z

What I learned from the work on #5439 (comment) got me to add

commit 3f90808, and while I was at it, also
commit b47be2d.

I don't expect any extra fallout from those two changes. The PYBIND11_STDLIB change consolidates the logic related to libc++ and libstdc++. The logic is now easier to understand and certain to be self-consistent.

rwgk · 2024-12-01T21:01:27Z

Just to double-check, the latest PYBIND11_INTERNALS_ID overview (this PR at commit b47be2d) is still identical to that posted under #5439 (comment):

logs_31527439795.zip — https://github.com/pybind/pybind11/actions/runs/12108518322/job/33756607834?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | c
ut -d' ' -f1 | sort | uniq -c

      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
     26 __pybind11_internals_vX_system_libcpp_abi1__
     53 __pybind11_internals_vX_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__

THIS IS JUST A START: First attempt to combine information from PR py…

14f8425

…bind#4953 and PR pybind#5437

rwgk mentioned this pull request Nov 10, 2024

Build platform ABI tag improvements #5437

Open

rwgk marked this pull request as ready for review November 11, 2024 00:43

rwgk and others added 3 commits November 18, 2024 09:06

Merge branch 'master' into pybind11_platform_abi_id_cont

e46982c

Include GXX_ABI and USE_CXX in the identifier

b72c42d

Further constrain to GXX_ABI 1002 or greater and less than 2000, hopefully future proof by summarizing that as `1` along with CXX11 on or off.

style: pre-commit fixes

8369fdc

cryos added 2 commits November 20, 2024 19:47

Use gxx_abi_1xxx and simplify the Clang string

476c322

After discussions with Ralf Grosse-Kunstleve we think these would make good identifiers that are concise and clear.

Error if _GLIBCXX_USE_CXX11_ABI is not defined

271720f

Within the `__GXX_ABI_VERSION` block this should always be defined, guard against unexpected defines and make the error obvious.

rwgk added 3 commits November 24, 2024 12:39

Merge branch 'master' into pybind11_platform_abi_id_cont

970a7eb

Change usecxx11 to use_cxx11_abi for correspondence with `_GLIBCX…

28081fc

…X_USE_CXX11_ABI` (similarly to `gxx_abi` for `__GXX_ABI_VERSION`).

PYBIND11_COMPILER_TYPE overhaul, mainly: replace _icc, _clang, …

fe2dbcb

…`_gcc` with `_system`

robertmaynard approved these changes Nov 25, 2024

View reviewed changes

rwgk added 2 commits November 25, 2024 11:30

Merge branch 'master' into pybind11_platform_abi_id_cont

9acf764

Add NVHPC (__PGI) to the list of compilers compatible with system com…

9fc9515

…piler. See comment by @robertmaynard: pybind#5439 (comment)

rwgk added 4 commits November 25, 2024 13:24

Fix oversight: remove __NVCOMPILER elif branch in PYBIND11_BUILD_ABI …

d412303

…block. Also add comment pointing to this PR (pybind#5439).

Revert "Fix oversight: remove __NVCOMPILER elif branch in PYBIND11_BU…

b6ccce3

…ILD_ABI block." This reverts commit d412303.

Revert "Add NVHPC (__PGI) to the list of compilers compatible with sy…

a584398

…stem compiler." This reverts commit 9fc9515.

Define NVHPC PYBIND11_BUILD_ABI using __GNUC__, __GNUC_MINOR__, _GLIB…

23a5f2b

…CXX_USE_CXX11_ABI

Use _GLIBCXX_USE_CXX11_ABI to detect libstdc++, then assume that NVHP…

8fa10bf

…C is always in the 1xxx ABI family.

Enhance NVHPC comment and limited future proofing.

02daf15

rwgk added 2 commits December 1, 2024 11:27

The PYBIND11_STDLIB is obsolete but kept around to maintain backwar…

3f90808

…d compatibility.

Move PYBIND11_BUILD_TYPE down in the file, so that the order of mac…

b47be2d

…ro definitions is the same as in the list defining `PYBIND11_PLATFORM_ABI_ID`

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`PYBIND11_PLATFORM_ABI_ID` Modernization Continued (WIP) #5439

`PYBIND11_PLATFORM_ABI_ID` Modernization Continued (WIP) #5439

rwgk commented Nov 10, 2024 •

edited

Loading

rwgk commented Nov 11, 2024

robertmaynard commented Nov 12, 2024

isuruf commented Nov 12, 2024

wjakob commented Nov 15, 2024

rwgk commented Nov 15, 2024

wjakob commented Nov 17, 2024

isuruf commented Nov 18, 2024

cryos commented Nov 20, 2024

rwgk commented Nov 20, 2024 •

edited

Loading

rwgk commented Nov 21, 2024

rwgk commented Nov 24, 2024

rwgk commented Nov 24, 2024

rwgk commented Nov 25, 2024

rwgk commented Nov 25, 2024

robertmaynard commented Nov 25, 2024

rwgk commented Nov 25, 2024

rwgk commented Nov 26, 2024

rwgk commented Nov 26, 2024

rwgk commented Nov 26, 2024

rwgk commented Nov 26, 2024

rwgk commented Nov 26, 2024

wjakob commented Nov 27, 2024

rwgk commented Nov 29, 2024

wjakob commented Dec 1, 2024

rwgk commented Dec 1, 2024

rwgk commented Dec 1, 2024

rwgk commented Dec 1, 2024

rwgk commented Dec 1, 2024

PYBIND11_PLATFORM_ABI_ID Modernization Continued (WIP) #5439

Are you sure you want to change the base?

PYBIND11_PLATFORM_ABI_ID Modernization Continued (WIP) #5439

Conversation

rwgk commented Nov 10, 2024 • edited Loading

Description

Suggested changelog entry:

rwgk commented Nov 11, 2024

robertmaynard commented Nov 12, 2024

isuruf commented Nov 12, 2024

wjakob commented Nov 15, 2024

rwgk commented Nov 15, 2024

wjakob commented Nov 17, 2024

isuruf commented Nov 18, 2024

cryos commented Nov 20, 2024

rwgk commented Nov 20, 2024 • edited Loading

rwgk commented Nov 21, 2024

rwgk commented Nov 24, 2024

rwgk commented Nov 24, 2024

rwgk commented Nov 25, 2024

rwgk commented Nov 25, 2024

robertmaynard commented Nov 25, 2024

rwgk commented Nov 25, 2024

rwgk commented Nov 26, 2024

rwgk commented Nov 26, 2024

rwgk commented Nov 26, 2024

rwgk commented Nov 26, 2024

rwgk commented Nov 26, 2024

wjakob commented Nov 27, 2024

rwgk commented Nov 29, 2024

wjakob commented Dec 1, 2024

rwgk commented Dec 1, 2024

rwgk commented Dec 1, 2024

rwgk commented Dec 1, 2024

rwgk commented Dec 1, 2024

`PYBIND11_PLATFORM_ABI_ID` Modernization Continued (WIP) #5439

`PYBIND11_PLATFORM_ABI_ID` Modernization Continued (WIP) #5439

rwgk commented Nov 10, 2024 •

edited

Loading

rwgk commented Nov 20, 2024 •

edited

Loading