Comments (10)
I guess that it is already possible somehow, as you do communicate with native binary from python.
Could you point me to the right place to look for code that does this communication?
If it is possible to save CGT-generated C/C++ sources this probably could help me with understanding how to run graph from external application.
Thanks.
from cgt.
We're working on this, stay tuned.
from cgt.
@joschu What do you think about cross-language execution graph serialization, with JSON or protobuf? Right now execution graphs are given to the C++ interpreter via Cython reading Python objects, but for this case we want a standalone C++ library to be able to read a serialized execution graph. This will introduce a dependency into the C++ side (for reading the serialized representation with a JSON library, for example), and might also introduce it to the Python side (if we use protobuf, which is not installed by default).
from cgt.
I'm sorry to bother you...
Here is my two cents:
- JSON -- using it involves base64 encoding/decoding which takes time and makes data about 30% bigger.
- protobuf looks like a quite fat dependency. In my experience protobuf is rather inconvenient when using from python.
MessagePack (http://msgpack.org/) also looks quite prominent for binary serialization, though I don't have any experience with it.
from cgt.
Jonathan, I like the idea of having cross-language serialization using some standard format.
One constraint is that we'll need to serialize numerical array data. This would be kludgy in JSON.
Protobuf would be an OK choice, though it might be a bit too heavyweight.
If I remember correctly, @pcmoritz also recommended msgpack
At first glance, it looks like a good choice to me.
from cgt.
HDF5 is a good option and is already used in many Python projects.
On Tue, Sep 22, 2015 at 1:19 PM, John Schulman [email protected]
wrote:
Jonathan, I like the idea of having cross-language serialization using
some standard format.One constraint is that we'll need to serialize numerical array data. This
would be kludgy in JSON.
Protobuf would be an OK choice, though it might be a bit too heavyweight.If I remember correctly, @pcmoritz https://github.com/pcmoritz also
recommended msgpack
At first glance, it looks like a good choice to me.—
Reply to this email directly or view it on GitHub
#31 (comment).
from cgt.
I was just referring to serialization of instructions/execution graph. This shouldn't involve serializing binary data, right?
from cgt.
The ReturnByRef and ReturnByVal instructions have to store the closure data for the Ops they're associated with. And a Constant Op has a value associated with it, so it'll be necessary to serialize the data.
from cgt.
I have started implementing serialization for execution graphs here: https://github.com/hojonathanho/cgt/tree/serialization
It's incomplete for now, but cgtArrays can be serialized.
I chose a C++-only serialization framework because I think that with the current way things are set up, it's best for serialization/deserialization to happen in C++/Cython only -- we should never be constructing execution graphs in Python anyway. It's also a lot faster to not have to worry about cross-language compatibility, since we can effectively directly serialize the bits in structs. Let me know if you think this is the right way to go.
from cgt.
Agreed that that it makes sense to serialize through in C++. But the closure data for the Ops is created in python. How do you plan to serialize each piece of closure data?
from cgt.
Related Issues (20)
- Product of diagonal elements may be `nan` HOT 1
- cgt.stack only supports stacking scalars HOT 1
- Error on tutorial
- Bug in graph simplification HOT 2
- OSX: Build fails with OpenBLAS dependency HOT 1
- Questions about Installation : cmake and windows HOT 1
- Memory consumption of native backend on Linux
- Random number generation and graph simplification HOT 2
- Negative reshaping check not enforced sometimes HOT 1
- Parameter indexing leads to an assertion error in module definition HOT 1
- Issues with shared scalar HOT 2
- Random number generation and modules (and simplification again)
- CIFAR Demo Error when using GPU
- Installation with anaconda
- Support for square root operation on Node HOT 2
- Simple operation breaks CGT
- Berkely BidMach and CGT
- How to build with python 3?
- Any reference to read? HOT 1
- Links in read-me are dead
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cgt.