| Frequently asked questions |
| ########################## |
| |
| "ImportError: dynamic module does not define init function" |
| =========================================================== |
| |
| 1. Make sure that the name specified in PYBIND11_MODULE is identical to the |
| filename of the extension library (without suffixes such as .so) |
| |
| 2. If the above did not fix the issue, you are likely using an incompatible |
| version of Python (for instance, the extension library was compiled against |
| Python 2, while the interpreter is running on top of some version of Python |
| 3, or vice versa). |
| |
| "Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``" |
| ======================================================================== |
| |
| See the first answer. |
| |
| "SystemError: dynamic module not initialized properly" |
| ====================================================== |
| |
| See the first answer. |
| |
| The Python interpreter immediately crashes when importing my module |
| =================================================================== |
| |
| See the first answer. |
| |
| .. _faq_reference_arguments: |
| |
| Limitations involving reference arguments |
| ========================================= |
| |
| In C++, it's fairly common to pass arguments using mutable references or |
| mutable pointers, which allows both read and write access to the value |
| supplied by the caller. This is sometimes done for efficiency reasons, or to |
| realize functions that have multiple return values. Here are two very basic |
| examples: |
| |
| .. code-block:: cpp |
| |
| void increment(int &i) { i++; } |
| void increment_ptr(int *i) { (*i)++; } |
| |
| In Python, all arguments are passed by reference, so there is no general |
| issue in binding such code from Python. |
| |
| However, certain basic Python types (like ``str``, ``int``, ``bool``, |
| ``float``, etc.) are **immutable**. This means that the following attempt |
| to port the function to Python doesn't have the same effect on the value |
| provided by the caller -- in fact, it does nothing at all. |
| |
| .. code-block:: python |
| |
| def increment(i): |
| i += 1 # nope.. |
| |
| pybind11 is also affected by such language-level conventions, which means that |
| binding ``increment`` or ``increment_ptr`` will also create Python functions |
| that don't modify their arguments. |
| |
| Although inconvenient, one workaround is to encapsulate the immutable types in |
| a custom type that does allow modifications. |
| |
| An other alternative involves binding a small wrapper lambda function that |
| returns a tuple with all output arguments (see the remainder of the |
| documentation for examples on binding lambda functions). An example: |
| |
| .. code-block:: cpp |
| |
| int foo(int &i) { i++; return 123; } |
| |
| and the binding code |
| |
| .. code-block:: cpp |
| |
| m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); }); |
| |
| |
| How can I reduce the build time? |
| ================================ |
| |
| It's good practice to split binding code over multiple files, as in the |
| following example: |
| |
| :file:`example.cpp`: |
| |
| .. code-block:: cpp |
| |
| void init_ex1(py::module_ &); |
| void init_ex2(py::module_ &); |
| /* ... */ |
| |
| PYBIND11_MODULE(example, m) { |
| init_ex1(m); |
| init_ex2(m); |
| /* ... */ |
| } |
| |
| :file:`ex1.cpp`: |
| |
| .. code-block:: cpp |
| |
| void init_ex1(py::module_ &m) { |
| m.def("add", [](int a, int b) { return a + b; }); |
| } |
| |
| :file:`ex2.cpp`: |
| |
| .. code-block:: cpp |
| |
| void init_ex2(py::module_ &m) { |
| m.def("sub", [](int a, int b) { return a - b; }); |
| } |
| |
| :command:`python`: |
| |
| .. code-block:: pycon |
| |
| >>> import example |
| >>> example.add(1, 2) |
| 3 |
| >>> example.sub(1, 1) |
| 0 |
| |
| As shown above, the various ``init_ex`` functions should be contained in |
| separate files that can be compiled independently from one another, and then |
| linked together into the same final shared object. Following this approach |
| will: |
| |
| 1. reduce memory requirements per compilation unit. |
| |
| 2. enable parallel builds (if desired). |
| |
| 3. allow for faster incremental builds. For instance, when a single class |
| definition is changed, only a subset of the binding code will generally need |
| to be recompiled. |
| |
| "recursive template instantiation exceeded maximum depth of 256" |
| ================================================================ |
| |
| If you receive an error about excessive recursive template evaluation, try |
| specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The |
| culprit is generally the generation of function signatures at compile time |
| using C++14 template metaprogramming. |
| |
| .. _`faq:hidden_visibility`: |
| |
| "‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]" |
| ============================================================================================================ |
| |
| This error typically indicates that you are compiling without the required |
| ``-fvisibility`` flag. pybind11 code internally forces hidden visibility on |
| all internal code, but if non-hidden (and thus *exported*) code attempts to |
| include a pybind type (for example, ``py::object`` or ``py::list``) you can run |
| into this warning. |
| |
| To avoid it, make sure you are specifying ``-fvisibility=hidden`` when |
| compiling pybind code. |
| |
| As to why ``-fvisibility=hidden`` is necessary, because pybind modules could |
| have been compiled under different versions of pybind itself, it is also |
| important that the symbols defined in one module do not clash with the |
| potentially-incompatible symbols defined in another. While Python extension |
| modules are usually loaded with localized symbols (under POSIX systems |
| typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default |
| can be changed, but even if it isn't it is not always enough to guarantee |
| complete independence of the symbols involved when not using |
| ``-fvisibility=hidden``. |
| |
| Additionally, ``-fvisiblity=hidden`` can deliver considerably binary size |
| savings. (See the following section for more details). |
| |
| |
| .. _`faq:symhidden`: |
| |
| How can I create smaller binaries? |
| ================================== |
| |
| To do its job, pybind11 extensively relies on a programming technique known as |
| *template metaprogramming*, which is a way of performing computation at compile |
| time using type information. Template metaprogamming usually instantiates code |
| involving significant numbers of deeply nested types that are either completely |
| removed or reduced to just a few instructions during the compiler's optimization |
| phase. However, due to the nested nature of these types, the resulting symbol |
| names in the compiled extension library can be extremely long. For instance, |
| the included test suite contains the following symbol: |
| |
| .. only:: html |
| |
| .. code-block:: none |
| |
| __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ |
| |
| .. only:: not html |
| |
| .. code-block:: cpp |
| |
| __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_ |
| |
| which is the mangled form of the following function type: |
| |
| .. code-block:: cpp |
| |
| pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28]) |
| |
| The memory needed to store just the mangled name of this function (196 bytes) |
| is larger than the actual piece of code (111 bytes) it represents! On the other |
| hand, it's silly to even give this function a name -- after all, it's just a |
| tiny cog in a bigger piece of machinery that is not exposed to the outside |
| world. So we'll generally only want to export symbols for those functions which |
| are actually called from the outside. |
| |
| This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC |
| and Clang, which sets the default symbol visibility to *hidden*, which has a |
| tremendous impact on the final binary size of the resulting extension library. |
| (On Visual Studio, symbols are already hidden by default, so nothing needs to |
| be done there.) |
| |
| In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids |
| potential serious issues when loading multiple modules and is required for |
| proper pybind operation. See the previous FAQ entry for more details. |
| |
| Working with ancient Visual Studio 2008 builds on Windows |
| ========================================================= |
| |
| The official Windows distributions of Python are compiled using truly |
| ancient versions of Visual Studio that lack good C++11 support. Some users |
| implicitly assume that it would be impossible to load a plugin built with |
| Visual Studio 2015 into a Python distribution that was compiled using Visual |
| Studio 2008. However, no such issue exists: it's perfectly legitimate to |
| interface DLLs that are built with different compilers and/or C libraries. |
| Common gotchas to watch out for involve not ``free()``-ing memory region |
| that that were ``malloc()``-ed in another shared library, using data |
| structures with incompatible ABIs, and so on. pybind11 is very careful not |
| to make these types of mistakes. |
| |
| How can I properly handle Ctrl-C in long-running functions? |
| =========================================================== |
| |
| Ctrl-C is received by the Python interpreter, and holds it until the GIL |
| is released, so a long-running function won't be interrupted. |
| |
| To interrupt from inside your function, you can use the ``PyErr_CheckSignals()`` |
| function, that will tell if a signal has been raised on the Python side. This |
| function merely checks a flag, so its impact is negligible. When a signal has |
| been received, you must either explicitly interrupt execution by throwing |
| ``py::error_already_set`` (which will propagate the existing |
| ``KeyboardInterrupt``), or clear the error (which you usually will not want): |
| |
| .. code-block:: cpp |
| |
| PYBIND11_MODULE(example, m) |
| { |
| m.def("long running_func", []() |
| { |
| for (;;) { |
| if (PyErr_CheckSignals() != 0) |
| throw py::error_already_set(); |
| // Long running iteration |
| } |
| }); |
| } |
| |
| CMake doesn't detect the right Python version |
| ============================================= |
| |
| The CMake-based build system will try to automatically detect the installed |
| version of Python and link against that. When this fails, or when there are |
| multiple versions of Python and it finds the wrong one, delete |
| ``CMakeCache.txt`` and then add ``-DPYTHON_EXECUTABLE=$(which python)`` to your |
| CMake configure line. (Replace ``$(which python)`` with a path to python if |
| your prefer.) |
| |
| You can alternatively try ``-DPYBIND11_FINDPYTHON=ON``, which will activate the |
| new CMake FindPython support instead of pybind11's custom search. Requires |
| CMake 3.12+, and 3.15+ or 3.18.2+ are even better. You can set this in your |
| ``CMakeLists.txt`` before adding or finding pybind11, as well. |
| |
| Inconsistent detection of Python version in CMake and pybind11 |
| ============================================================== |
| |
| The functions ``find_package(PythonInterp)`` and ``find_package(PythonLibs)`` |
| provided by CMake for Python version detection are modified by pybind11 due to |
| unreliability and limitations that make them unsuitable for pybind11's needs. |
| Instead pybind11 provides its own, more reliable Python detection CMake code. |
| Conflicts can arise, however, when using pybind11 in a project that *also* uses |
| the CMake Python detection in a system with several Python versions installed. |
| |
| This difference may cause inconsistencies and errors if *both* mechanisms are |
| used in the same project. Consider the following CMake code executed in a |
| system with Python 2.7 and 3.x installed: |
| |
| .. code-block:: cmake |
| |
| find_package(PythonInterp) |
| find_package(PythonLibs) |
| find_package(pybind11) |
| |
| It will detect Python 2.7 and pybind11 will pick it as well. |
| |
| In contrast this code: |
| |
| .. code-block:: cmake |
| |
| find_package(pybind11) |
| find_package(PythonInterp) |
| find_package(PythonLibs) |
| |
| will detect Python 3.x for pybind11 and may crash on |
| ``find_package(PythonLibs)`` afterwards. |
| |
| There are three possible solutions: |
| |
| 1. Avoid using ``find_package(PythonInterp)`` and ``find_package(PythonLibs)`` |
| from CMake and rely on pybind11 in detecting Python version. If this is not |
| possible, the CMake machinery should be called *before* including pybind11. |
| 2. Set ``PYBIND11_FINDPYTHON`` to ``True`` or use ``find_package(Python |
| COMPONENTS Interpreter Development)`` on modern CMake (3.12+, 3.15+ better, |
| 3.18.2+ best). Pybind11 in these cases uses the new CMake FindPython instead |
| of the old, deprecated search tools, and these modules are much better at |
| finding the correct Python. |
| 3. Set ``PYBIND11_NOPYTHON`` to ``TRUE``. Pybind11 will not search for Python. |
| However, you will have to use the target-based system, and do more setup |
| yourself, because it does not know about or include things that depend on |
| Python, like ``pybind11_add_module``. This might be ideal for integrating |
| into an existing system, like scikit-build's Python helpers. |
| |
| How to cite this project? |
| ========================= |
| |
| We suggest the following BibTeX template to cite pybind11 in scientific |
| discourse: |
| |
| .. code-block:: bash |
| |
| @misc{pybind11, |
| author = {Wenzel Jakob and Jason Rhinelander and Dean Moldovan}, |
| year = {2017}, |
| note = {https://github.com/pybind/pybind11}, |
| title = {pybind11 -- Seamless operability between C++11 and Python} |
| } |