Boost.Python: return_internal_reference<> bug

[Christoff Heinrich Kok <christoff.kok@…>]

I am having an issue with Boost.Python with a very simple use case.

I am returning a reference to an object, and it seems that my python object looses its C++ object's reference at a stage for some reason.

Please see my example below reproducing this issue.

C++ Code:

#include <vector>
#include <string>

#include <cmath>
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>

class Car {
public:
    Car(std::string name) : m_name(name) {}

    bool operator==(const Car &other) const {
        return m_name == other.m_name;
    }

    std::string GetName() { return m_name; }
private:
    std::string m_name;
};

class Factory {
public:
    Factory(std::string name) : m_name(name) {}

    bool operator==(const Factory &other) const {
        return m_name == other.m_name
            && m_car_list == other.m_car_list;
    }

    Car& create_car(std::string name)
    {
        m_car_list.emplace_back(Car(name));
        return m_car_list.back();
    }

    std::string GetName() { return m_name; }
    std::vector<Car>& GetCarList() { return m_car_list;}
private:
    std::string m_name;
    std::vector<Car> m_car_list;
};

class Manufacturer {
public:
    Manufacturer(std::string name) : m_name(name) {}

    bool operator==(const Manufacturer &other) const {
        return m_name == other.m_name
            && m_factory_list == other.m_factory_list;
    }

    Factory& create_factory(std::string name)
    {
        m_factory_list.emplace_back(Factory(name));
        return m_factory_list.back();
    }

    std::string GetName() { return m_name; }
    std::vector<Factory>& GetFactoryList() { return m_factory_list;}
private:
    std::string m_name;
    std::vector<Factory> m_factory_list;
};

BOOST_PYTHON_MODULE(carManufacturer)
{
    using namespace boost::python;
    class_<Manufacturer>("Manufacturer", init<std::string>())
        .add_property("factory_list", make_function(&Manufacturer::GetFactoryList, return_internal_reference<>()))
        .add_property("name", &Manufacturer::GetName)
        .def("create_factory", &Manufacturer::create_factory, return_internal_reference<>());
    class_<Factory>("Factory", init<std::string>())
        .add_property("car_list", make_function(&Factory::GetCarList, return_internal_reference<>()))
        .add_property("name", &Factory::GetName)
        .def("create_car", &Factory::create_car, return_internal_reference<>());
    class_<Car>("Car", init<std::string>())
        .add_property("name", &Car::GetName);

    class_<std::vector<Factory> >("FactoryList")
        .def(vector_indexing_suite<std::vector<Factory> >());
    class_<std::vector<Car> >("Car")
        .def(vector_indexing_suite<std::vector<Car> >());
}

Python Code:

from carManufacturer import *

vw = Manufacturer("VW")
vw_bra_factory = vw.create_factory("Brazil Factory")
beetle = vw_bra_factory.create_car("Beetle69")

if vw_bra_factory is vw.factory_list[0]:
    print("equal.")
else:
    print("NOT EQUAL")
print("## I expected them to be the same reference..?")


print("vw_bra_factory Car List size : " + str(len(vw_bra_factory.car_list)))
print("Actual Car List size         : " + str(len(vw.factory_list[0].car_list)))
print("## This still works. Maybe the python objects differ, but refer to the same C++ object. I can live with that.")

vw_sa_factory = vw.create_factory("South Africa Factory")
print("vw_bra_factory Car List size : " + str(len(vw_bra_factory.car_list)))
print("Actual Car List size         : " + str(len(vw.factory_list[0].car_list)))
print("## .. what? why? brazil py object has no cars now? I don't get it. I can't have any of that.")

print("## What will happen if I create another car in the brazil factory?")
combi = vw_bra_factory.create_car("Hippie van")
print("vw_bra_factory Car List size : " + str(len(vw_bra_factory.car_list)))
print("Actual Car List size         : " + str(len(vw.factory_list[0].car_list)))

print("## And another.")
citi_golf = vw_bra_factory.create_car("Citi golf")
print("vw_bra_factory Car List size : " + str(len(vw_bra_factory.car_list)))
print("Actual Car List size         : " + str(len(vw.factory_list[0].car_list)))
print("## 'vw_bra_factory' must have lost its C++ reference it had to 'vw.factory_list[0]' when I created a new factory. Why?")

Output:

NOT EQUAL
## I expected them to be the same reference..?
vw_bra_factory Car List size : 1
Actual Car List size         : 1
## This still works. Maybe the python objects differ, but refer to the same C++ object. I can live with that.
vw_bra_factory Car List size : 0
Actual Car List size         : 1
## .. what? why? brazil py object has no cars now? I don't get it. I can't have any of that.
## What will happen if I create another car in the brazil factory?
vw_bra_factory Car List size : 1
Actual Car List size         : 1
## And another.
vw_bra_factory Car List size : 2
Actual Car List size         : 1
## 'vw_bra_factory' must have lost its C++ reference it had to 'vw.factory_list[0]' when I created a new factory. Why?

This is just an example made to reproduce my real work's problem in a presentable way.

Is there any workaround to this? I could not find any in bug reports, the mailing list or anywhere else.

Regards, Christoff

[anonymous]

Workarounds:

Thanks to someone on the mailing list for figuring out what the problem is:

It works when I reserve the vectors sizes to e.g. 32. (Not that it fixes the bug, it only deters the bug to occur until the reserve size is exceeded.)

It also works if I allocate the objects stored in the vectors on the heap.