[BUG]: Having trouble creating immutable type with `tp_flags |= Py_TPFLAGS_IMMUTABLETYPE`

[XuehaiPan]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.3

Problem description

I'm trying to bind a C++ class to Python using py::class_<CustomClass>. I have some internal knowledge about my custom class, which should be immutable. So I want to turn on Py_TPFLAGS_IMMUTABLETYPE (since Python 3.10) in my custom Python type object's tp_flags.

I tried to set the tp_flags using py::custom_type_setup:

auto ImmutableCustomTypeObject =
    py::class_<ImmutableCustom>(mod,
                                "ImmutableCustom",
                                "Instances of this class are immutable.",
                                py::custom_type_setup([](PyHeapTypeObject* heap_type) {
                                    auto* type = &heap_type->ht_type;
#ifdef Py_TPFLAGS_IMMUTABLETYPE
                                    type->tp_flags |= Py_TPFLAGS_IMMUTABLETYPE;
#endif
                                }));

However, the typeobject becomes frozen immediately in PyType_Ready after calling the custom_type_setup_callback:

pybind11/include/pybind11/detail/class.h

Lines 716 to 735 in 3cc7e42

	if (rec.custom_type_setup_callback) {
	rec.custom_type_setup_callback(heap_type);
	}
	if (PyType_Ready(type) < 0) {
	pybind11_fail(std::string(rec.name) + ": PyType_Ready failed: " + error_string());
	}
	assert(!rec.dynamic_attr || PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC));
	/* Register type with the parent scope */
	if (rec.scope) {
	setattr(rec.scope, rec.name, (PyObject *) type);
	} else {
	Py_INCREF(type); // Keep it alive forever (reference leak)
	}
	if (module_) { // Needed by pydoc
	setattr((PyObject *) type, "__module__", module_);
	}

then a TypeError will raise at line 734 when importing the built extension:

pybind11/include/pybind11/detail/class.h

Lines 733 to 735 in 3cc7e42

	if (module_) { // Needed by pydoc
	setattr((PyObject *) type, "__module__", module_);
	}

which is equivalent to:

ImmutableCustom.__module__: str = mod.__name__

it raises:

>>> import _C
TypeError: cannot set '__module__' attribute of immutable type '_C.ImmutableCustom'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: initialization failed

Also, after PyType_Ready, I cannot do cls.def_property_readonly(...).def(...); since it is immutable.

---

I tried to modify the tp_flags after I had fully initialized the class definition:

auto ImmutableCustomTypeObject =
    py::class_<ImmutableCustom>(mod,
                                "ImmutableCustom",
                                "Instances of this class are immutable.");

reinterpret_cast<PyTypeObject*>(ImmutableCustomTypeObject.ptr())->tp_name = "mymod.ImmutableCustomTypeObject";
py::setattr(ImmutableCustomTypeObject, "__module__", py::str("mymod"));

ImmutableCustomTypeObject
    .def_property_readonly(...)
    .def_property_readonly(...)
    .def(...)
    .def(...);

#ifdef Py_TPFLAGS_IMMUTABLETYPE
reinterpret_cast<PyTypeObject*>(ImmutableCustomTypeObject.ptr())->tp_flags |=
    Py_TPFLAGS_IMMUTABLETYPE;
#endif

I'm not sure if this is allowed to flip tp_flags if the type is already called with PyType_Ready.

Reproducible example code

Initialize a bound class with tp_flags |= Py_TPFLAGS_IMMUTABLETYPE (Python 3.10+):

auto ImmutableCustomTypeObject =
    py::class_<ImmutableCustom>(mod,
                                "ImmutableCustom",
                                "Instances of this class are immutable.",
                                py::custom_type_setup([](PyHeapTypeObject* heap_type) {
                                    auto* type = &heap_type->ht_type;
#ifdef Py_TPFLAGS_IMMUTABLETYPE
                                    type->tp_flags |= Py_TPFLAGS_IMMUTABLETYPE;
#endif
                                }));

Then import the C++ extension in Python:

>>> import _C
TypeError: cannot set '__module__' attribute of immutable type '_C.ImmutableCustom'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: initialization failed

Is this a regression? Put the last known working version here if it is.

Not a regression

[Skylion007]

@XuehaiPan Would you mind opening a PR adding this as a test case? This is a really an API issue. The proper thing to do would be not to set the module and just use the 'module' appropriately generated from the tp_name: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_name

[XuehaiPan]

This is a really an API issue. The proper thing to do would be not to set the module and just use the 'module' appropriately generated from the tp_name: docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_name

The __module__ attribute is not the only thing that is broken here. Py_TPFLAGS_IMMUTABLETYPE also breaks py::class_<CustomClass>(...).def(...).

In the standard C API type creation, users need to provide PyMethodDef array before calling PyType_Ready.

static PyMethodDef custom_methods[] = {
    {"method1", (PyCFunction)method1, METH_O, method1_doc},
    {"method2", (PyCFunction)method2, METH_O, method2_doc},
    ...,
    {NULL,      NULL}   /* sentinel */
}

static PyTypeObject custom_type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "mymod.MyClass",  /* tp_name */
    ...,
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,  /* tp_flags */
    ...,
    custom_methods,  /* tp_methods */
    custom_members,  /* tp_members */
    ...
}

PyMODINIT_FUNC
PyInit_mymod(void)
{
    if (PyType_Ready(&custom_type) < 0)
        return NULL;
    ...
}

In pybind11, we use cls.def(...).def(...); to create methods, which uses cls.attr("name") = function. We are calling PyType_Ready with empty tp_methods and then assign the methods with setattr. That will also raise TypeError if PyType_HasFeature(cls.ptr(), Py_TPFLAGS_IMMUTABLETYPE).

auto cls = py::class_<MyClass>(mod, "MyClass", "MyClass docs");  // PyType_Ready is called during initialization
cls.def("method1", &MyClass::method1)   // calls `setattr` and raises TypeError
   .def("method2", &MyClass::method2);  // calls `setattr` and raises TypeError

[XuehaiPan]

This is a really an API issue.

I wonder if should we forbid Py_TPFLAGS_IMMUTABLETYPE and raise a TypeError when PyType_HasFeature(type, Py_TPFLAGS_IMMUTABLETYPE). Because it will break the user-friendly API like cls.def(...).

---

Since most members and methods are defined after the type initialization (after PyType_Ready), could we add a new method like custom_type_setup? Like:

namespace pybind11 {

struct custom_type_finalize {
    using callback = std::function<void(PyHeapTypeObject *heap_type)>;

    explicit custom_type_finalize(callback value) : value(std::move(value)) {}

    callback value;
};

}

py::class_& py::class_::finalize(py::custom_type_finalize callback) {
    callback.callback(m_ptr);
    if (PyType_Ready(m_ptr) < 0) {
        pybind11_fail("class_::finalize(): failure in PyType_Ready()!");
    }
}

For example:

auto cls = py::class_<MyClass>(mod, "MyClass", "MyClass docs");
cls.def("method1", &MyClass::method1)
   .def("method2", &MyClass::method2)
   .finalize(
    py::custom_type_finalize([](PyTypeObject* type) {
#ifdef Py_TPFLAGS_IMMUTABLETYPE
        type->tp_flags |= Py_TPFLAGS_IMMUTABLETYPE;
#endif
    })
)

Set type->tp_flags |= Py_TPFLAGS_IMMUTABLETYPE and recall PyType_Ready after all members and methods are properly added.

[Skylion007]

@XuehaiPan Could this be related to how we structure enums too? IE: #3865

[Skylion007]

@XuehaiPan Are we sure there isn't a way to enable this behavior without an API change? Even if that is special handling for the ImmutableFlag?

[XuehaiPan]

@XuehaiPan Are we sure there isn't a way to enable this behavior without an API change? Even if that is special handling for the ImmutableFlag?

@Skylion007 Py_TPFLAGS_IMMUTABLETYPE tp_flags is not compatible with custom_type_setup. The type object becomes immutable immediately after PyType_Ready.

There may be two ways to enable that flag.

1. Use standard type creation with PyMemberDef and PyMethodDef like CPython. But this is not user-friendly.
2. Set Py_TPFLAGS_IMMUTABLETYPE tp_flags manually after all members and methods are assigned to the type object. As I posted in [BUG]: Having trouble creating immutable type with tp_flags |= Py_TPFLAGS_IMMUTABLETYPE #4548 (comment). I had this in https://github.com/metaopt/optree/blob/9e150ce0cf85924184b630ef5a079fcbd031e1b8/src/optree.cpp#L214-L220 and no unittest is failed and no bug has been reported yet.

I tried to modify the tp_flags after I had fully initialized the class definition:

auto ImmutableCustomTypeObject =
    py::class_<ImmutableCustom>(mod,
                                "ImmutableCustom",
                                "Instances of this class are immutable.");

reinterpret_cast<PyTypeObject*>(ImmutableCustomTypeObject.ptr())->tp_name = "mymod.ImmutableCustomTypeObject";
py::setattr(ImmutableCustomTypeObject, "__module__", py::str("mymod"));

ImmutableCustomTypeObject
    .def_property_readonly(...)
    .def_property_readonly(...)
    .def(...)
    .def(...);

#ifdef Py_TPFLAGS_IMMUTABLETYPE
reinterpret_cast<PyTypeObject*>(ImmutableCustomTypeObject.ptr())->tp_flags |=
    Py_TPFLAGS_IMMUTABLETYPE;
#endif

I'm not sure if this is allowed to flip tp_flags if the type is already called with PyType_Ready.