pybind / pybind11_json Goto Github PK
View Code? Open in Web Editor NEWUsing nlohmann::json with pybind11
License: BSD 3-Clause "New" or "Revised" License
Using nlohmann::json with pybind11
License: BSD 3-Clause "New" or "Revised" License
This is the same issue that #15 attempted to solve. In CMake, we don't want to search for the dependencies (pybind11 and nlohmann_json) using find_package
if the user is adding them using add_directories
. In this case, the find_package
command would fail because it would force CMake to look for these dependencies in the wrong places. So #15 added the following guards: only call find_package(nlohmann_json)
if the target nlohmann_json
doesn't already exist, and only call find_package(pybind11)
if the target pybind11
doesn't already exist:
if (NOT TARGET pybind11)
find_package(pybind11 ${pybind11_REQUIRED_VERSION} REQUIRED)
endif()
if (NOT TARGET nlohmann_json)
find_package(nlohmann_json ${nlohmann_json_REQUIRED_VERSION} REQUIRED)
endif()
Unfortunately, the first guard does not work. The problem is that the target pybind11
does not exist even if pybind11 has been properly added. I was using add_directories
to add pybind11 to my project and getting the following error from CMake:
CMake Error at build/_deps/pybind11_json-src/CMakeLists.txt:26 (find_package):
By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "pybind11",
but CMake did not find one.
Could not find a package configuration file provided by "pybind11"
(requested version 2.2.4) with any of the following names:
pybind11Config.cmake
pybind11-config.cmake
Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
"pybind11_DIR" to a directory containing one of the above files. If
"pybind11" provides a separate development package or SDK, be sure it has
been installed.
-- Configuring incomplete, errors occurred!
This is because pybind11_json/CMakeLists.txt:26 is calling find_package(pybind11)
when it shouldn't be, which happens because the target pybind11
does not exist.
Is possible to "register" support for serialize other kind of non trivial python object like pathlib.Path
--> serialized as str
?
@wjakob or @SylvainCorlay would you mind activating AppVeyor CI for this repo?
NLohmann JSON version 3.10.4 has binary arrays: https://json.nlohmann.me/api/basic_json/binary/
If such array is encountered in JSON object, pyjson::from_json() crashes with stack overflow because it falls to "else // Object" clause and then the for() loop tries to iterate over non-existing values, since binary array is not an object.
Proposed fix:
Add include file:
#include "pybind11/numpy.h"
In pyjson::from_json() add:
...
else if (j.is_binary())
{
const auto &bin = j.get_binary();
return py::array( bin.size(), bin.data() );
}
else // Object
{
...
@wjakob last time I ping you, I promise!
The appveyor badge url is wrong on the README, but I don't seem to have the authorization for getting access to the pybind/pybind11_json
project on AppVeyor (because I'm not part of the organization). So I am not able to find this badge url.
Would you mind sharing the url here or fixing the README?
I wrote a Pybind11 module and just one function is not working only when it is called in the Python environment (C++ side works great).
When I call it in the C++ environment, this function takes 30 seconds to return the result and works great.
This function uses 2 external libraries:
When I call it in the Python side, I got a error inside the pugi
library.
python3: /src/pugixml.cpp:12202: const pugi::xpath_node& pugi::xpath_node_set::operator[](size_t) const: Assertion `index < size()' failed.
Aborted (core dumped)
How can I debug (line by line) my C++ code in the Python environment?
I have a class function that returns a nlohmann::json object that I would like to call directly in python.
Is there any way to use this function without having to explicitly do the conversion myself?
What I am currently doing is:
py::class_<myClass> (m, "myClass")
.def (py::init<const char*> ())
.def ("getJSON", [] (const myClass& my_class) -> py::object {
nl::json json_obj = my_class.getJSON ();
py::object py_obj = json_obj; // convert to type already bound
return py_obj;
})
;
I am trying to achieve something like:
py::class_<myClass> (m, "myClass")
.def (py::init<const char*> ())
.def ("getJSON", &myClass::getJSON)
;
I tried to register a py::implicitly_convertible<py::object, nl::json> (); but it just works from python to C++, not the other way around.
I also tried defining a json binding class with:
py::class_<nl::json> (m, "json")
.def (py::init<> ())
.def (py::init<py::object> ())
;
py::implicitly_convertible<py::object, nl::json> ();
I also know that what would probably work is to return a py::object from myClass::getJSON as mentioned in #18. However, I would like to avoid myClass having to depend on a pybind11 implementation.
This error is caused by using the option ARCH_INDEPENDENT
, which was only added in CMake version 3.14 (see the release notes for that version here). Building the library fails with older versions of CMake. For example, with CMake 3.11 I get:
CMake Error at /usr/share/cmake/Modules/WriteBasicConfigVersionFile.cmake:30 (message):
Unknown keywords given to WRITE_BASIC_CONFIG_VERSION_FILE():
"ARCH_INDEPENDENT"
Call Stack (most recent call first):
/usr/share/cmake/Modules/CMakePackageConfigHelpers.cmake:209 (write_basic_config_version_file)
CMakeLists.txt:116 (write_basic_package_version_file)
This recent commit introduced the use of ARCH_INDEPENDENT
, but did not change the minimum required CMake version.
Hi,
sorry for being lame but I can't put together a complete working example with pybind11_json.
Here is what I need to do:
1.) I have a native C++ class like
class Foo {
public:
int member1;
int member2;
};
2.) I have the corresponding pybind11 class_ mapping which defines a "Foo" class and does ".def_readwrite" on "member1" and "member2"
3.) Now I need to have to create something which automatically makes it possible that from Python I could:
foo = Foo()
foo.member1 = 1
foo.member2 = 2
jsonFoo = json(foo)
print(jsonFoo) // This would print '{ member1 : 1, member2 : 2 }'
// ???
foo2 = json('{ member1: 4, member2 : 5}')
print(foo2.member1) // this would print 4
print(foo2.member2) // this would print 5
Is it possible at all? Or do I have to create Nlohmann boilerplate code to be able to "Jsonify" my "Foo" class?
What is the full workflow here to be able to serialize back and forther PyBind11 classes in Python?
Thanks,
p.s. I have tried using the "implicitly_convertible" trick
pythonbind::class_nl::json(mod, "Json")
.def(py::init<>())
.def(py::initpy::object());
py::implicitly_convertible<py::object, nl::json>();
and then from Python:
foo = Foo()
jsonFoo = Json(foo)
json.dumps(jsonFoo)
but that gives me "to_json not implemented for this type of object: xxxx_pybind11.Foo object at XXXXX"
(which I suppose is not a surprise, as the C++ PyBind classes are not iterable objects, hence you need to write
for each and every Pybind class its own Json converter, I suppose?)
We should be able to serialize sets and frozensets
As you can see here #18 I understood that I can bind nlohmann::json
as a py::object
type.
I would like to know if is possible to bind the nlohmann::json::array_t
type to some native Python type.
I mean, which Python type should I use to be able to return a nlohmann
array of objects from C++ to Python side.
On newer versions of cmake I get this warning:
CMake Deprecation Warning at CMakeLists.txt:9 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
I'm building a C++ Pybind11 module to use my C++ functions inside some Python scripts.
I would like to know if is possible to write a C++ function that returns a nlohmann::json
type and get this output result in the Python side as JSON...or dict...??
Example: C++ side
nlohmann::json my_func()
{
nlohmann::json j;
j["key_a"] = 0;
j["key_b"] = 1;
j["tree"]["first"] = "node 1";
return j;
}
Python side:
import my_module
data = my_module.my_func()
How can I do that?
There seem to be an error when trying to serialize bytes, bytearray, memoryview
objects
Adding support for the ordered_json
type in addition to currently used json
would be beneficial for users intending to preserve order of elements in the JSON structures.
For further discussion, see:
https://github.com/nlohmann/json#order-of-object-keys
nlohmann/json#2179
nlohmann/json#2258
P.S. Thank you for a great piece of software (we use it in https://github.com/open-atmos/PyPartMC/)
In some paths within py::object from_json(const nl::json& j)
the returned object is returned via std::move(obj);
. This triggers a redundant-move
warning in GCC 12.1, as the move prevents copy elision.
I'm having a tough time using this lib. I've copied pybind_json.hpp into my include dir as I don't have access to conda.
Then my main file is:
#include <pybind_json.hpp>
int main()
{
nlohmann::json j;
j["a"] = 1;
j["b"] = 2;
py::dict d;
d["a"] = 1;
d["b"] = 2;
py::dict d2 = j;
nlohmann::json j2 = d;
}
The last two lines give compilation errors of:
"no suitable user-defined conversion from "nlohmann::json" to "pybind11::dict" exists"
"no suitable user-defined conversion from "pybind11::dict" to "nlohmann::json" exists"
respectively. So I am unable to do any conversion. I must be missing something simple. I am able to compile the code with the last 2 lines commented out so I know the nlohmann and pybind11 libs are properly being used.
Hi,
I must say I really like the idea of this utility to ease my parameters structures usage in Python. I encountered an issue with bound enumerations. I use enumerations quite a lot in C++ mainly in factories. I can easily have a nlohmann converters for them, I can easily bind them, but I'm lost when trying to have the two functionalities working together.
It's quite logic since the nlohmann json bindings expect classic types and I guess enumerations fall into a special structures. I tried to get how I could handle those errors but I struggle finding how I could do that.
#include
#include "nlohmann/json.hpp"
// This is where I copied pybind11_json.hpp
#include "converters/json.h"
#define MODULE_NAME footest
namespace testing_pybind11_json {
enum class Foo : uint32_t { ZERO, ONE, TWO, THREE };
NLOHMANN_JSON_SERIALIZE_ENUM(Foo,
{{Foo::ZERO, "ZERO"},
{Foo::ONE, "ONE"},
{Foo::TWO, "TWO"},
{Foo::THREE, "THREE"}})
void print_json(const nlohmann::json &j) {
std::cout << std::endl << std::setw(4) << j.dump() << std::endl;
}
void print_dict(const py::dict &d) {
nlohmann::json j;
j = d;
std::cout << std::endl << std::setw(4) << j.dump() << std::endl;
}
} // namespace testing_pybind11_json
PYBIND11_MODULE(MODULE_NAME, m) {
Py_Initialize();
PyEval_InitThreads();
py::enum_<testing_pybind11_json::Foo>(m, "Foo")
.value("ZERO", testing_pybind11_json::Foo::ZERO)
.value("ONE", testing_pybind11_json::Foo::ONE)
.value("TWO", testing_pybind11_json::Foo::TWO)
.value("THREE", testing_pybind11_json::Foo::THREE);
m.def("print_json", &testing_pybind11_json::print_json);
m.def("print_dict", &testing_pybind11_json::print_dict);
}
footest_pytest.cpp
import footest
import pytest
@pytest.mark.module("binding.utils")
def pytestcase_testing_pybind11_print_json():
footest.print_json({'a': 1, 'b': 2})
footest.print_json({'a': 1, 'b': 2, 'c': footest.Foo.ONE})
@pytest.mark.module("binding.utils")
def pytestcase_testing_pybind11_print_dict():
footest.print_dict({'a': 1, 'b': 2})
footest.print_dict({'a': 1, 'b': 2, 'c': footest.Foo.ONE})
It returns me different type of error depending on the function I'm using.
With print_json
: it tries to use directly type_caster
but seems to fall in a strange case.
@pytest.mark.module("binding.utils")
def pytestcase_testing_pybind11_print_json():
mci.print_json({'a': 1, 'b': 2})
> mci.print_json({'a': 1, 'b': 2, 'c': mci.Foo.ONE})
E TypeError: print_json(): incompatible function arguments. The following argument types are supported:
E 1. (arg0: json) -> None
E
E Invoked with: {'a': 1, 'b': 2, 'c': <Foo.ONE: 1>}
sdk/modules/computational_imaging/python/tests/bindings/calibration_pytest.py:148: TypeError
With print_dict
: I avoid using type_caster
but calls directly the to_json
function which goes well until the incompatible enum.
@pytest.mark.module("binding.utils")
def pytestcase_testing_pybind11_print_dict():
mci.print_dict({'a': 1, 'b': 2})
> mci.print_dict({'a': 1, 'b': 2, 'c': mci.Foo.ONE})
E RuntimeError: to_json not implemented for this type of object: <Foo.ONE: 1>
sdk/modules/computational_imaging/python/tests/bindings/calibration_pytest.py:154: RuntimeError
I don't know if it's even possible to catch the enumeration case and call the to_json
/ from_json
of the underlying enumeration.
Error: nlohmann::json as input argument
As you can see in this topic #18, I can return a nlohmann::json
from a Python function as a pybind11::object
type.
Now, I'm trying to send a JSON as a input argument to my_func()
, modify it and return it back in my Python script.
Example: C++ side
pybind11::object my_func(nlohmann::json& a)
{
a["new"] = "new string";
return a;
}
It compiles Ok, but when I tried to call this function inside my Python module, I got this error below:
Error: Python side
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: test(): incompatible function arguments. The following argument types are supported:
1. (self: my_lib.MusicXML, arg0: nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer>) -> object
What I have to do to fix it?
Thank you,
Rename cmake project
We should rename the cmake project to pybind11_json
pybind11_json with c++ class
I am trying to use the pybind11_json with a c++ class.
When I use the example provided in this git: void take_json(const nlohmann::json& json)
, it works fine.
But I am trying to use it as a method of my class:
py::class_<myClass>(module, "myClass", py::buffer_protocol())
.def("fromJson", &myClass::fromJson);
I am getting the following error when I call it from python:
obj.fromJson({"a": 1234})
TypeError: fromJson(): incompatible function arguments. The following argument types are supported:
1. (self: myModule.myClass, arg0: json) -> None
Invoked with: {'a': 1234}
Do I need to do anything else to make it work with methods of a c++ class?
Infinity mishandled in nl::json => py::object conversion
When converting a nl::json object that contains the infinity to Python, infinity gets interpreted as an integer instead of a float. If I bind the following function
#include <cmath>
nl::json test() {
return nl::json(INFINITY);
}
it returns -9223372036854775808
on the Python side instead of math.inf
.
No 'test' target when BUILD_TESTS=ON
When BUILD_TESTS=ON
it builds the executable test_pybind11_json
but 'make test' doesn't run it.
Move to pybind organization?
Hi Martin,
several projects that integrate pybind11 with other dependencies (abseil, basel, protobuf) are slated to move into the pybind organization (https://github.com/pybind/). I thought that this would also make sense for this project, which is both useful and popular. It's completely up to you, of course. If you would like to move, please let me know and I'll invite you to the organization.
Best,
Wenzel
local variable 'obj' will be copied despite being returned by name
During my app compilation, I got a multiple repetition of 2 warnings:
/Users/studio/maialib/core/external/pybind11_json/pybind11_json.hpp:57:20: note:
call 'std::move' explicitly to avoid copying
return obj;
^~~
std::move(obj)
/Users/studio/maialib/core/external/pybind11_json/pybind11_json.hpp:66:20: warning:
local variable 'obj' will be copied despite being returned by name
[-Wreturn-std-move]
return obj;
^~~
Please, could you fix it to clean the terminal output compilation?
My System:
- Apple OS X Catalina 10.15.7
- Xcode Command Tools 11.5
- Pybind11 2.9.0
- Using CMake with
C++17
flags enable
Obs.: This 2 warnings appear on OS X system only. I got no warnings on Windows and Linux.
Reactivate TravisCI
@wjakob would you mind activating TravisCI for this project?
Recommend Projects
-
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
-
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
-