ext/pybind11/docs/advanced/misc.rst - public/gem5 - Git at Google

 Miscellaneous
 #############

 .. _macro_notes:

 General notes regarding convenience macros
 ==========================================

 pybind11 provides a few convenience macros such as
 :func:`PYBIND11_DECLARE_HOLDER_TYPE` and ``PYBIND11_OVERRIDE_*``. Since these
 are "just" macros that are evaluated in the preprocessor (which has no concept
 of types), they *will* get confused by commas in a template argument; for
 example, consider:

 .. code-block:: cpp

     PYBIND11_OVERRIDE(MyReturnType<T1, T2>, Class<T3, T4>, func)

 The limitation of the C preprocessor interprets this as five arguments (with new
 arguments beginning after each comma) rather than three.  To get around this,
 there are two alternatives: you can use a type alias, or you can wrap the type
 using the ``PYBIND11_TYPE`` macro:

 .. code-block:: cpp

     // Version 1: using a type alias
     using ReturnType = MyReturnType<T1, T2>;
     using ClassType = Class<T3, T4>;
     PYBIND11_OVERRIDE(ReturnType, ClassType, func);

     // Version 2: using the PYBIND11_TYPE macro:
     PYBIND11_OVERRIDE(PYBIND11_TYPE(MyReturnType<T1, T2>),
                       PYBIND11_TYPE(Class<T3, T4>), func)

 The ``PYBIND11_MAKE_OPAQUE`` macro does *not* require the above workarounds.

 .. _gil:

 Global Interpreter Lock (GIL)
 =============================

 When calling a C++ function from Python, the GIL is always held.
 The classes :class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be
 used to acquire and release the global interpreter lock in the body of a C++
 function call. In this way, long-running C++ code can be parallelized using
 multiple Python threads. Taking :ref:`overriding_virtuals` as an example, this
 could be realized as follows (important changes highlighted):

 .. code-block:: cpp
     :emphasize-lines: 8,9,31,32

     class PyAnimal : public Animal {
     public:
         /* Inherit the constructors */
         using Animal::Animal;

         /* Trampoline (need one for each virtual function) */
         std::string go(int n_times) {
             /* Acquire GIL before calling Python code */
             py::gil_scoped_acquire acquire;

             PYBIND11_OVERRIDE_PURE(
                 std::string, /* Return type */
                 Animal,      /* Parent class */
                 go,          /* Name of function */
                 n_times      /* Argument(s) */
             );
         }
     };

     PYBIND11_MODULE(example, m) {
         py::class_<Animal, PyAnimal> animal(m, "Animal");
         animal
             .def(py::init<>())
             .def("go", &Animal::go);

         py::class_<Dog>(m, "Dog", animal)
             .def(py::init<>());

         m.def("call_go", [](Animal *animal) -> std::string {
             /* Release GIL before calling into (potentially long-running) C++ code */
             py::gil_scoped_release release;
             return call_go(animal);
         });
     }

 The ``call_go`` wrapper can also be simplified using the ``call_guard`` policy
 (see :ref:`call_policies`) which yields the same result:

 .. code-block:: cpp

     m.def("call_go", &call_go, py::call_guard<py::gil_scoped_release>());


 Binding sequence data types, iterators, the slicing protocol, etc.
 ==================================================================

 Please refer to the supplemental example for details.

 .. seealso::

     The file :file:`tests/test_sequences_and_iterators.cpp` contains a
     complete example that shows how to bind a sequence data type, including
     length queries (``__len__``), iterators (``__iter__``), the slicing
     protocol and other kinds of useful operations.


 Partitioning code over multiple extension modules
 =================================================

 It's straightforward to split binding code over multiple extension modules,
 while referencing types that are declared elsewhere. Everything "just" works
 without any special precautions. One exception to this rule occurs when
 extending a type declared in another extension module. Recall the basic example
 from Section :ref:`inheritance`.

 .. code-block:: cpp

     py::class_<Pet> pet(m, "Pet");
     pet.def(py::init<const std::string &>())
        .def_readwrite("name", &Pet::name);

     py::class_<Dog>(m, "Dog", pet /* <- specify parent */)
         .def(py::init<const std::string &>())
         .def("bark", &Dog::bark);

 Suppose now that ``Pet`` bindings are defined in a module named ``basic``,
 whereas the ``Dog`` bindings are defined somewhere else. The challenge is of
 course that the variable ``pet`` is not available anymore though it is needed
 to indicate the inheritance relationship to the constructor of ``class_<Dog>``.
 However, it can be acquired as follows:

 .. code-block:: cpp

     py::object pet = (py::object) py::module_::import("basic").attr("Pet");

     py::class_<Dog>(m, "Dog", pet)
         .def(py::init<const std::string &>())
         .def("bark", &Dog::bark);

 Alternatively, you can specify the base class as a template parameter option to
 ``class_``, which performs an automated lookup of the corresponding Python
 type. Like the above code, however, this also requires invoking the ``import``
 function once to ensure that the pybind11 binding code of the module ``basic``
 has been executed:

 .. code-block:: cpp

     py::module_::import("basic");

     py::class_<Dog, Pet>(m, "Dog")
         .def(py::init<const std::string &>())
         .def("bark", &Dog::bark);

 Naturally, both methods will fail when there are cyclic dependencies.

 Note that pybind11 code compiled with hidden-by-default symbol visibility (e.g.
 via the command line flag ``-fvisibility=hidden`` on GCC/Clang), which is
 required for proper pybind11 functionality, can interfere with the ability to
 access types defined in another extension module.  Working around this requires
 manually exporting types that are accessed by multiple extension modules;
 pybind11 provides a macro to do just this:

 .. code-block:: cpp

     class PYBIND11_EXPORT Dog : public Animal {
         ...
     };

 Note also that it is possible (although would rarely be required) to share arbitrary
 C++ objects between extension modules at runtime. Internal library data is shared
 between modules using capsule machinery [#f6]_ which can be also utilized for
 storing, modifying and accessing user-defined data. Note that an extension module
 will "see" other extensions' data if and only if they were built with the same
 pybind11 version. Consider the following example:

 .. code-block:: cpp

     auto data = reinterpret_cast<MyData *>(py::get_shared_data("mydata"));
     if (!data)
         data = static_cast<MyData *>(py::set_shared_data("mydata", new MyData(42)));

 If the above snippet was used in several separately compiled extension modules,
 the first one to be imported would create a ``MyData`` instance and associate
 a ``"mydata"`` key with a pointer to it. Extensions that are imported later
 would be then able to access the data behind the same pointer.

 .. [#f6] https://docs.python.org/3/extending/extending.html#using-capsules

 Module Destructors
 ==================

 pybind11 does not provide an explicit mechanism to invoke cleanup code at
 module destruction time. In rare cases where such functionality is required, it
 is possible to emulate it using Python capsules or weak references with a
 destruction callback.

 .. code-block:: cpp

     auto cleanup_callback = []() {
         // perform cleanup here -- this function is called with the GIL held
     };

     m.add_object("_cleanup", py::capsule(cleanup_callback));

 This approach has the potential downside that instances of classes exposed
 within the module may still be alive when the cleanup callback is invoked
 (whether this is acceptable will generally depend on the application).

 Alternatively, the capsule may also be stashed within a type object, which
 ensures that it not called before all instances of that type have been
 collected:

 .. code-block:: cpp

     auto cleanup_callback = []() { /* ... */ };
     m.attr("BaseClass").attr("_cleanup") = py::capsule(cleanup_callback);

 Both approaches also expose a potentially dangerous ``_cleanup`` attribute in
 Python, which may be undesirable from an API standpoint (a premature explicit
 call from Python might lead to undefined behavior). Yet another approach that
 avoids this issue involves weak reference with a cleanup callback:

 .. code-block:: cpp

     // Register a callback function that is invoked when the BaseClass object is collected
     py::cpp_function cleanup_callback(
         [](py::handle weakref) {
             // perform cleanup here -- this function is called with the GIL held

             weakref.dec_ref(); // release weak reference
         }
     );

     // Create a weak reference with a cleanup callback and initially leak it
     (void) py::weakref(m.attr("BaseClass"), cleanup_callback).release();

 .. note::

     PyPy does not garbage collect objects when the interpreter exits. An alternative
     approach (which also works on CPython) is to use the :py:mod:`atexit` module [#f7]_,
     for example:

     .. code-block:: cpp

         auto atexit = py::module_::import("atexit");
         atexit.attr("register")(py::cpp_function([]() {
             // perform cleanup here -- this function is called with the GIL held
         }));

     .. [#f7] https://docs.python.org/3/library/atexit.html


 Generating documentation using Sphinx
 =====================================

 Sphinx [#f4]_ has the ability to inspect the signatures and documentation
 strings in pybind11-based extension modules to automatically generate beautiful
 documentation in a variety formats. The python_example repository [#f5]_ contains a
 simple example repository which uses this approach.

 There are two potential gotchas when using this approach: first, make sure that
 the resulting strings do not contain any :kbd:`TAB` characters, which break the
 docstring parsing routines. You may want to use C++11 raw string literals,
 which are convenient for multi-line comments. Conveniently, any excess
 indentation will be automatically be removed by Sphinx. However, for this to
 work, it is important that all lines are indented consistently, i.e.:

 .. code-block:: cpp

     // ok
     m.def("foo", &foo, R"mydelimiter(
         The foo function

         Parameters
         ----------
     )mydelimiter");

     // *not ok*
     m.def("foo", &foo, R"mydelimiter(The foo function

         Parameters
         ----------
     )mydelimiter");

 By default, pybind11 automatically generates and prepends a signature to the docstring of a function
 registered with ``module_::def()`` and ``class_::def()``. Sometimes this
 behavior is not desirable, because you want to provide your own signature or remove
 the docstring completely to exclude the function from the Sphinx documentation.
 The class ``options`` allows you to selectively suppress auto-generated signatures:

 .. code-block:: cpp

     PYBIND11_MODULE(example, m) {
         py::options options;
         options.disable_function_signatures();

         m.def("add", [](int a, int b) { return a + b; }, "A function which adds two numbers");
     }

 Note that changes to the settings affect only function bindings created during the
 lifetime of the ``options`` instance. When it goes out of scope at the end of the module's init function,
 the default settings are restored to prevent unwanted side effects.

 .. [#f4] http://www.sphinx-doc.org
 .. [#f5] http://github.com/pybind/python_example

 .. _avoiding-cpp-types-in-docstrings:

 Avoiding C++ types in docstrings
 ================================

 Docstrings are generated at the time of the declaration, e.g. when ``.def(...)`` is called.
 At this point parameter and return types should be known to pybind11.
 If a custom type is not exposed yet through a ``py::class_`` constructor or a custom type caster,
 its C++ type name will be used instead to generate the signature in the docstring:

 .. code-block:: text

      |  __init__(...)
      |      __init__(self: example.Foo, arg0: ns::Bar) -> None
                                               ^^^^^^^


 This limitation can be circumvented by ensuring that C++ classes are registered with pybind11
 before they are used as a parameter or return type of a function:

 .. code-block:: cpp

     PYBIND11_MODULE(example, m) {

         auto pyFoo = py::class_<ns::Foo>(m, "Foo");
         auto pyBar = py::class_<ns::Bar>(m, "Bar");

         pyFoo.def(py::init<const ns::Bar&>());
         pyBar.def(py::init<const ns::Foo&>());
     }
	Miscellaneous
	#############

	.. _macro_notes:

	General notes regarding convenience macros
	==========================================

	pybind11 provides a few convenience macros such as
	:func:`PYBIND11_DECLARE_HOLDER_TYPE` and ``PYBIND11_OVERRIDE_*``. Since these
	are "just" macros that are evaluated in the preprocessor (which has no concept
	of types), they will get confused by commas in a template argument; for
	example, consider:

	.. code-block:: cpp

	PYBIND11_OVERRIDE(MyReturnType<T1, T2>, Class<T3, T4>, func)

	The limitation of the C preprocessor interprets this as five arguments (with new
	arguments beginning after each comma) rather than three. To get around this,
	there are two alternatives: you can use a type alias, or you can wrap the type
	using the ``PYBIND11_TYPE`` macro:

	.. code-block:: cpp

	// Version 1: using a type alias
	using ReturnType = MyReturnType<T1, T2>;
	using ClassType = Class<T3, T4>;
	PYBIND11_OVERRIDE(ReturnType, ClassType, func);

	// Version 2: using the PYBIND11_TYPE macro:
	PYBIND11_OVERRIDE(PYBIND11_TYPE(MyReturnType<T1, T2>),
	PYBIND11_TYPE(Class<T3, T4>), func)

	The ``PYBIND11_MAKE_OPAQUE`` macro does not require the above workarounds.

	.. _gil:

	Global Interpreter Lock (GIL)
	=============================

	When calling a C++ function from Python, the GIL is always held.
	The classes :class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be
	used to acquire and release the global interpreter lock in the body of a C++
	function call. In this way, long-running C++ code can be parallelized using
	multiple Python threads. Taking :ref:`overriding_virtuals` as an example, this
	could be realized as follows (important changes highlighted):

	.. code-block:: cpp
	:emphasize-lines: 8,9,31,32

	class PyAnimal : public Animal {
	public:
	/* Inherit the constructors */
	using Animal::Animal;

	/* Trampoline (need one for each virtual function) */
	std::string go(int n_times) {
	/* Acquire GIL before calling Python code */
	py::gil_scoped_acquire acquire;

	PYBIND11_OVERRIDE_PURE(
	std::string, /* Return type */
	Animal, /* Parent class */
	go, /* Name of function */
	n_times /* Argument(s) */
	);
	}
	};

	PYBIND11_MODULE(example, m) {
	py::class_<Animal, PyAnimal> animal(m, "Animal");
	animal
	.def(py::init<>())
	.def("go", &Animal::go);

	py::class_<Dog>(m, "Dog", animal)
	.def(py::init<>());

	m.def("call_go", [](Animal *animal) -> std::string {
	/* Release GIL before calling into (potentially long-running) C++ code */
	py::gil_scoped_release release;
	return call_go(animal);
	});
	}

	The ``call_go`` wrapper can also be simplified using the ``call_guard`` policy
	(see :ref:`call_policies`) which yields the same result:

	.. code-block:: cpp

	m.def("call_go", &call_go, py::call_guard<py::gil_scoped_release>());


	Binding sequence data types, iterators, the slicing protocol, etc.
	==================================================================

	Please refer to the supplemental example for details.

	.. seealso::

	The file :file:`tests/test_sequences_and_iterators.cpp` contains a
	complete example that shows how to bind a sequence data type, including
	length queries (``__len__``), iterators (``__iter__``), the slicing
	protocol and other kinds of useful operations.


	Partitioning code over multiple extension modules
	=================================================

	It's straightforward to split binding code over multiple extension modules,
	while referencing types that are declared elsewhere. Everything "just" works
	without any special precautions. One exception to this rule occurs when
	extending a type declared in another extension module. Recall the basic example
	from Section :ref:`inheritance`.

	.. code-block:: cpp

	py::class_<Pet> pet(m, "Pet");
	pet.def(py::init<const std::string &>())
	.def_readwrite("name", &Pet::name);

	py::class_<Dog>(m, "Dog", pet /* <- specify parent */)
	.def(py::init<const std::string &>())
	.def("bark", &Dog::bark);

	Suppose now that ``Pet`` bindings are defined in a module named ``basic``,
	whereas the ``Dog`` bindings are defined somewhere else. The challenge is of
	course that the variable ``pet`` is not available anymore though it is needed
	to indicate the inheritance relationship to the constructor of ``class_<Dog>``.
	However, it can be acquired as follows:

	.. code-block:: cpp

	py::object pet = (py::object) py::module_::import("basic").attr("Pet");

	py::class_<Dog>(m, "Dog", pet)
	.def(py::init<const std::string &>())
	.def("bark", &Dog::bark);

	Alternatively, you can specify the base class as a template parameter option to
	``class_``, which performs an automated lookup of the corresponding Python
	type. Like the above code, however, this also requires invoking the ``import``
	function once to ensure that the pybind11 binding code of the module ``basic``
	has been executed:

	.. code-block:: cpp

	py::module_::import("basic");

	py::class_<Dog, Pet>(m, "Dog")
	.def(py::init<const std::string &>())
	.def("bark", &Dog::bark);

	Naturally, both methods will fail when there are cyclic dependencies.

	Note that pybind11 code compiled with hidden-by-default symbol visibility (e.g.
	via the command line flag ``-fvisibility=hidden`` on GCC/Clang), which is
	required for proper pybind11 functionality, can interfere with the ability to
	access types defined in another extension module. Working around this requires
	manually exporting types that are accessed by multiple extension modules;
	pybind11 provides a macro to do just this:

	.. code-block:: cpp

	class PYBIND11_EXPORT Dog : public Animal {
	...
	};

	Note also that it is possible (although would rarely be required) to share arbitrary
	C++ objects between extension modules at runtime. Internal library data is shared
	between modules using capsule machinery [#f6]_ which can be also utilized for
	storing, modifying and accessing user-defined data. Note that an extension module
	will "see" other extensions' data if and only if they were built with the same
	pybind11 version. Consider the following example:

	.. code-block:: cpp

	auto data = reinterpret_cast<MyData *>(py::get_shared_data("mydata"));
	if (!data)
	data = static_cast<MyData *>(py::set_shared_data("mydata", new MyData(42)));

	If the above snippet was used in several separately compiled extension modules,
	the first one to be imported would create a ``MyData`` instance and associate
	a ``"mydata"`` key with a pointer to it. Extensions that are imported later
	would be then able to access the data behind the same pointer.

	.. [#f6] https://docs.python.org/3/extending/extending.html#using-capsules

	Module Destructors
	==================

	pybind11 does not provide an explicit mechanism to invoke cleanup code at
	module destruction time. In rare cases where such functionality is required, it
	is possible to emulate it using Python capsules or weak references with a
	destruction callback.

	.. code-block:: cpp

	auto cleanup_callback = []() {
	// perform cleanup here -- this function is called with the GIL held
	};

	m.add_object("_cleanup", py::capsule(cleanup_callback));

	This approach has the potential downside that instances of classes exposed
	within the module may still be alive when the cleanup callback is invoked
	(whether this is acceptable will generally depend on the application).

	Alternatively, the capsule may also be stashed within a type object, which
	ensures that it not called before all instances of that type have been
	collected:

	.. code-block:: cpp

	auto cleanup_callback = []() { /* ... */ };
	m.attr("BaseClass").attr("_cleanup") = py::capsule(cleanup_callback);

	Both approaches also expose a potentially dangerous ``_cleanup`` attribute in
	Python, which may be undesirable from an API standpoint (a premature explicit
	call from Python might lead to undefined behavior). Yet another approach that
	avoids this issue involves weak reference with a cleanup callback:

	.. code-block:: cpp

	// Register a callback function that is invoked when the BaseClass object is collected
	py::cpp_function cleanup_callback(
	[](py::handle weakref) {
	// perform cleanup here -- this function is called with the GIL held

	weakref.dec_ref(); // release weak reference
	}
	);

	// Create a weak reference with a cleanup callback and initially leak it
	(void) py::weakref(m.attr("BaseClass"), cleanup_callback).release();

	.. note::

	PyPy does not garbage collect objects when the interpreter exits. An alternative
	approach (which also works on CPython) is to use the :py:mod:`atexit` module [#f7]_,
	for example:

	.. code-block:: cpp

	auto atexit = py::module_::import("atexit");
	atexit.attr("register")(py::cpp_function([]() {
	// perform cleanup here -- this function is called with the GIL held
	}));

	.. [#f7] https://docs.python.org/3/library/atexit.html


	Generating documentation using Sphinx
	=====================================

	Sphinx [#f4]_ has the ability to inspect the signatures and documentation
	strings in pybind11-based extension modules to automatically generate beautiful
	documentation in a variety formats. The python_example repository [#f5]_ contains a
	simple example repository which uses this approach.

	There are two potential gotchas when using this approach: first, make sure that
	the resulting strings do not contain any :kbd:`TAB` characters, which break the
	docstring parsing routines. You may want to use C++11 raw string literals,
	which are convenient for multi-line comments. Conveniently, any excess
	indentation will be automatically be removed by Sphinx. However, for this to
	work, it is important that all lines are indented consistently, i.e.:

	.. code-block:: cpp

	// ok
	m.def("foo", &foo, R"mydelimiter(
	The foo function

	Parameters
	----------
	)mydelimiter");

	// not ok
	m.def("foo", &foo, R"mydelimiter(The foo function

	Parameters
	----------
	)mydelimiter");

	By default, pybind11 automatically generates and prepends a signature to the docstring of a function
	registered with ``module_::def()`` and ``class_::def()``. Sometimes this
	behavior is not desirable, because you want to provide your own signature or remove
	the docstring completely to exclude the function from the Sphinx documentation.
	The class ``options`` allows you to selectively suppress auto-generated signatures:

	.. code-block:: cpp

	PYBIND11_MODULE(example, m) {
	py::options options;
	options.disable_function_signatures();

	m.def("add", [](int a, int b) { return a + b; }, "A function which adds two numbers");
	}

	Note that changes to the settings affect only function bindings created during the
	lifetime of the ``options`` instance. When it goes out of scope at the end of the module's init function,
	the default settings are restored to prevent unwanted side effects.

	.. [#f4] http://www.sphinx-doc.org
	.. [#f5] http://github.com/pybind/python_example

	.. _avoiding-cpp-types-in-docstrings:

	Avoiding C++ types in docstrings
	================================

	Docstrings are generated at the time of the declaration, e.g. when ``.def(...)`` is called.
	At this point parameter and return types should be known to pybind11.
	If a custom type is not exposed yet through a ``py::class_`` constructor or a custom type caster,
	its C++ type name will be used instead to generate the signature in the docstring:

	.. code-block:: text

	\| __init__(...)
	\| __init__(self: example.Foo, arg0: ns::Bar) -> None
	^^^^^^^


	This limitation can be circumvented by ensuring that C++ classes are registered with pybind11
	before they are used as a parameter or return type of a function:

	.. code-block:: cpp

	PYBIND11_MODULE(example, m) {

	auto pyFoo = py::class_<ns::Foo>(m, "Foo");
	auto pyBar = py::class_<ns::Bar>(m, "Bar");

	pyFoo.def(py::init<const ns::Bar&>());
	pyBar.def(py::init<const ns::Foo&>());
	}