Consider adding PyType_FromMetaclass

[erlend-aasland]

As suggested in python/cpython#93012 (comment), we should consider adding PyType_FromMetaclass.

@wjakob already has an implementation, see python/cpython#93012 (comment).

[vstinner]

What's the use case? Convert a static type to a heap type and be compatible with Python 3.11 and older, or use PyType_FromMetaclass() on Python 3.12 and newer?

[vstinner]

cc @encukou

[encukou]

The first. PyType_FromMetaclass() is already merged in main :)

The function is apparently very useful for language/framework bindings, several of which (PySide, JPype) now use ugly hacks to reimplement its functionality. Or they would need to use hacks if they want to switch to limited API (nanobind).

Before including it here, please wait for a few more fixes & tests to get sorted out & merged: python/cpython#28748

[vstinner]

I'm not comfortable with adding such function since the implementation is quite big :-( Can projects which need it copy it from CPython?

[vstinner]

When I tried to convert static types to heap types and port C extension to multiphase init API (PEP 489), I saw that storing the module in a type is also a key feature, and it's not supported by Python 3.8 and older, whereas pythoncapi-compat supports Python 3.5 to 3.13.

Python 3.9 added PyType_FromModuleAndSpec(), PyType_GetModule(), PyType_GetModuleState(), and PyHeapTypeObject.ht_module member.

PyType_GetModuleByDef() was only added to Python 3.11, whereas it's required to port many C extensions to multiphase init API. PyType_GetModuleByDef() implementation is around 34 lines.

Backporting these functions to Python 3.8 would require a hash table associating a type to a module (using a weak reference), since there is no API in old Python versions to store arbitrary data in PyTypeObject when creating a new type :-(

I don't think that pythoncapi_compat.h design is a good fit to implement these features. I would be more comfortable with a library, rather than a header file.

[vstinner]

I don't think that pythoncapi_compat.h design is a good fit to implement these features. I would be more comfortable with a library, rather than a header file.

I created the experimental https://github.com/vstinner/deadparrot/ project which is a shared library. Maybe this project is a better home for PyType_FromMetaclass().

[vstinner]

@wjakob already has an implementation, see python/cpython#93012 (comment).

I tried to implement this function in the deadparrot project: draft PR vstinner/deadparrot#2 which does crash...

I tried this implementation on Python 3.11. I had a hard time to make it work :-( It doesn't set tp_base properly. I had issues with tp_name (not set properly). I hacked the code to set tp_base and tp_name, but then I got negative reference counts and memory corruptions :-( I'm not sure that it's sustainable to maintain such code :-(

I also tried to backport PyType_FromMetaclass() from Python 3.13: I gave up after 500 lines of C code, it pulls way too many internals which would be too expensive to maintain, and it requires access to internals which are simply not accessible from the output, especially slots offsets.

I close the pythoncapi-compat issue. If someone wants to write a PR for the deadparrot project, please go ahead :-)