Factory class doesn't allow derived classes to be created

[hughed2]

I'm trying to bind a factory class which takes in a constructor function and can later use that function to create an instance of the desired object. This should be working with inherited classes as well. In C++ it works fine, but in Python it looks like I'm losing the derived type. The derived class is created, deleted, and then an instance of the base class is returned.

Here are example bindings, keeping it as close to our original code as possible.
example.cpp

And then some sample python showing where it doesn't work as I expect
import example

def create_dog():
  class Dog(example.Animal):
    def _init__(self):
      example.Animal.__init__(self)
    def speak(self):
      print "Woof!"
    def __del__(self):
      print "The puppy is moving to a farm upstate"
  return Dog

in_python = create_dog()()
in_python.speak()
# Expected output: Woof!
# Actual output: Woof!

example.set_factory(create_dog())
in_cpp = example.get_animal()
# Expected output: Making new animal from factory
# Actual output: Making new animal from factory
#                The puppy is moving to a farm upstate
in_cpp.speak()
# Expected output: Woof!
# Actual output: I am just a generic animal, I have no voice

[uwydoc]

i think the problem is the cast in line 17:

#16    std::shared_ptr<Animal> operator()() { py::object animal = obj();
#17                                           return animal.cast< std::shared_ptr<Animal> >();}

what you are actually doing is converting an object created in python-side to some type in C++ side, which i do not think works in this simple way. yes, pybind11 supports converting between python object and c++ object for function arguments and return values, but with some book keeping codes, especially with creating proxy python object and properly handles reference counting.

i think the cast operator does not necessarily takes care of reference counting of the original python object, nor does it handles inheritance smartly enough. thus in your use case, what the cast operator did is just make a base class instance and return it, and after the return statement, the python object animal just got destructed.

so, for your use case, i think the proper way is do not perform the cast and just return the py::object instance created by your factory method:

    py::object operator()() { return obj(); }

however, this way is not very type-safe though.

[hughed2]

Thanks, you got me on the right track. It would be nice to have the same kind of boost::python functionality that is able to deal with inheritance, but I can subclass the factory easily enough and bind that instead of the original one.

[hughed2]

I'm reopening this because I need to get that factory working on the C++ side. As it stands, I have two factories. If I am working in C++, I use an Animal factory which can be used in C++, or, through my bindings, in python. If I am working in python, I use a pybind11::object factory which can ONLY be used in python, because of the casting issue seen above.

So, to go back to my original question: can we get this working, preferably without a workaround like above?

[jagerman]

I may be missing something, but I think you can solve this by always creating the new instance through Python via its __class__ attribute, by doing the equivalent of newobj = obj.__class__() in C++: py::object newobj = obj.attr("__class__")();. If object was an unextended C++ object, that just creates another one; if it was extended, that creates the other one through the extending Python class.

Here's an example:

myclone.cpp:

#include <iostream>
#include <pybind11/pybind11.h>

namespace py = pybind11;

struct Foobar {
    Foobar() {}
    virtual ~Foobar() {}

    virtual void speak() { std::cout << "I'm mute :(\n"; }
};

struct FoobarTrampoline : Foobar {
    using Foobar::Foobar;
    void speak() override { PYBIND11_OVERLOAD(void, Foobar, speak, ); }
};

PYBIND11_MODULE(myclone, m) {

    py::class_<Foobar, FoobarTrampoline>(m, "Foobar")
        .def(py::init<>())
        .def("speak", &Foobar::speak)
        ;

    m.def("clone_foo", [](py::object foobar) {
        return foobar.attr("__class__")();
    });

    m.def("make_it_speak", [](Foobar &f) { f.speak(); });
}

pyclone.py:

from myclone import Foobar, clone_foo, make_it_speak

class MyFoobar(Foobar):
    def __init__(self, *args, **kwargs):
        Foobar.__init__(self, *args, **kwargs)

    def speak(self):
        print("Woof!")

a = MyFoobar()
b = Foobar()
c = clone_foo(a)
d = clone_foo(b)

for i in [a, b, c, d]:
    print(type(i))
    i.speak()
    make_it_speak(i)

Output:

c++ -std=c++14 -Os -I/usr/include/python3.6m -I. myclone.cpp -shared -fPIC -o myclone.cpython-36m-x86_64-linux-gnu.so
jagerman@betwixt:~/src/pybind11/scratch⦗upstream✻$⦘$ python3 pyclone.py 
<class '__main__.MyFoobar'>
Woof!
Woof!
<class 'myclone.Foobar'>
I'm mute :(
I'm mute :(
<class '__main__.MyFoobar'>
Woof!
Woof!
<class 'myclone.Foobar'>
I'm mute :(
I'm mute :(

[hughed2]

I had that working on python previously, but the issue is doing the same in C++. I've tried dealing with a corresponding factory method in C++ itself rather than just in python, and that hasn't been working. I can write up a better example later you can actually run, but in the interest of time here's a quick example

//Create a MyFoobar object foobar_from_python
Foobar foobar_in_cpp = foobar_from_python.cast<Foobar>();
foobar_from_python.attr("speak")(); // Outputs "Woof!"
foobar_in_cpp.speak(); // Outputs "I'm mute :(" even though it's copied from a MyFoobar

A very similar problem, but one that can't be solved quite so easily. If I try going a bit fancier (like your suggestion, calling the python __class__ attribute), I still end up with a pybind11::object on the C++ side, which similarly becomes an unextended Foobar once it gets cast.

To complicate things further, these factories contain functions that return the desired class, and those functions don't have to be the default constructor. Those functions could mess around with member parameters before returning the object, for example. So it'd be best if I stick with the original object output by the factory (cast into the appropriate type) rather than falling back on trying to use the default constructor.

[jagerman]

I still end up with a pybind11::object on the C++ side, which similarly becomes an unextended Foobar once it gets cast.

Try casting it to a reference or pointer rather than a value:

auto &foobar_in_cpp = foobar_from_python.cast<Foobar &>();

and if you need to combine it with a factory, do the factory call via Python side (via invoking __class__, as in the examples above) then cast that resulting new py::object to a pointer or lvalue (rather than value): you'll get a reference to the actual C++ object stored inside the new Python object, which will actually be a FoobarTrampoline instance with the overridden virtual methods that forward into Python calls.

[hughed2]

Those were some of my first thoughts, and what I'm seeing is

auto foobar_in_cpp = foobar_from_python.cast< Foobar *>();
foobar_in_cpp->speak();

leads to a segfault. foobar_in_cpp is non-negative, and no pybind11::cast_error is thrown, but it looks like it's not properly casting.

auto foobar_in_cpp = foobar_from_python.cast< std::shared_ptr< Foobar > >();
foobar_in_cpp->speak();

leads to "I'm mute"

auto &foobar_in_cpp = foobar_from_python.cast<Foobar &>();
foobar_in_cpp.speak();

similarly segfaults.

[jagerman]

I think the segfault is coming from something else; I don't see any segfaults if I amend my example to m.def("make_it_speak", [](py::object o) { o.cast<Foobar &>().speak(); });

[hughed2]

The issue isn't when I'm trying to run something in the pybind11 module, it's when I'm trying to send it back to C++.

For example

m.def("clone_foo", [](py::object o) {
          Foobar *foo_ptr = o.cast<Foobar *>();
          foo_ptr->speak();
          return foo_ptr;
}

works, it gives the expected Woof! output. However, if I later (in C++, outside of any pybind11 modules) call

//object foo_from_python is a MyFoobar
Foobar *foo_ptr = foo_from_python.attr("clone_foo").cast<Foobar *>();
foo_ptr->speak();

It'll first produce the Woof! from the clone_foo call, but then I get a segfault (or, with some changes to the code, it won't segfault but will give me an I'm mute message).

[jagerman]

Foobar *foo_ptr = foo_from_python.attr("clone_foo").cast<Foobar *>();

That needs to be:

/*...*/ .attr("clone_foo")().cast<Foobar *>();

otherwise you're trying to cast the "clone_foo" attribute itself (rather than the result of calling it) to a Foobar pointer.

[hughed2]

It was correct in my actual code, there was just an editing error in my previous comment.

[EricCousineau-TRI]

I have not had time to fully process the entirety of this issue, but it looks like this is an aliasing / truncation issue due to the Python interpreter losing references on Pythonic portion of the instance, and leaving the C++ portion out in the cold all alone.

Can you look at this PR and let me know if that actually might solve your issue? (The caveat is having to be beholden to a specific type of holder)
#1146
Specifically, this portion of the documentation highlights this edge case that you may be running into:
https://github.com/EricCousineau-TRI/pybind11/blob/feature/py_move_shared-wip/docs/advanced/classes.rst#virtual-inheritance-and-lifetime

I ask because ultimately our project may want something similar; a factory method that can be called on the C++ side, generating Python-derived objects that were registered with the factory.

All that being said, have y'all already discussed the workaround of having a Python-specific base class / proxy, which stores its own py::object reference to the effective base class? (This was suggested by someone else.)
This gets hairy with (a) opaque reference cycles between pybind11 holders and Python references, and (b) becomes crappy when you want to reference state in the base class (if your base is not a pure virtual class); however, this could ultimately be a first pass to more C++-friendly keep alive semantics.