Node.js bindings to C API

[lastmjs]

Node.js bindings to C API

[nkreeger]

As mentioned - here is our node binding https://github.com/tensorflow/tfjs-node

The project is still under heavy development and we haven't published anything to NPM to date. We only support new-ish Linux and Mac OS builds. We'll have details for how to support GPU and other optimized builds this week.

[dsmilkov]

We are planning to make the node bindings public very soon, just need to work out a few more details. Stay tuned!

[bileschi]

Thanks for your interest! There is not a separate location for this, as of yet, other than what is mentioned in the FAQ. We can use this issue to track developments.

[nsthorat]

We are considering WebAssembly, in fact there is a prototype backend. A naive WebAssembly implementation is much much slower than our WebGL implementation, which is why we haven't pushed it forward. We were only seeing ~3x speed improvement over vanilla JS. WebGL is about ~60 times faster (on mid-2015 MBP).

That being said, we are following the chromium bug about SIMD support which may make it competitive: https://bugs.chromium.org/p/v8/issues/detail?id=6020&desc=2

[nsthorat]

@erikwilson that is likely the API we'll be going for with node! WebGL for node is going to be similar speeds as the browser (headless-gl is a good project), but in general we'll likely focus on TF C API for node. When we release the node repository, this will also be a template for other backends.

@jart we have done a few investigations into WASM, and we'll likely open up the repo that has this as a backend. We see ~60x difference in performance of WebGL vs naive WASM implementation (only 4x speed improvement over JS CPU which is also naive and could be improved dramatically). WASM does not have SIMD yet, but it's actively being worked on. Likely still won't be a big win over WebGL. At the moment, it's not worth the energy.

[zandaqo]

Please take into consideration that Node was not designed for CPU bound operations. Last time I checked, there's no threads in Node, so there's also no C++ GIL unlock like Python. Yes, Node can serve static content to 10k HTTP sockets simultaneously. Now if just one of those requests, spends a few seconds doing linear algebra, hoo boy.

That's V8's limitations, not Node.js in general. Node's C++ addons can use multithredding and most everything else C++ provides. The only overhead here is shuffling data between native addon and JavaScript.

[nsthorat]

JS isn't be the bottleneck for our WebGL calls, the fragment shaders themselves are, so I'm not sure what WASM talking to WebAssembly would give us.

Doing things in JS also gives us a lot flexibility with debugging that WASM doesn't.

A fun slightly relevant post: https://mrale.ph/blog/2018/02/03/maybe-you-dont-need-rust-to-speed-up-your-js.html

[jart]

I did some analysis on how to build NodeJS bindings for TensorFlow in this tensorflow/tensorflow#37 (comment).

Please take into consideration that Node was not designed for CPU bound operations. Last time I checked, there's no threads in Node, so there's also no C++ GIL unlock like Python. Yes, Node can serve static content to 10k HTTP sockets simultaneously. Now if just one of those requests, spends a few seconds doing linear algebra, hoo boy.

It'd probably be more beneficial to Node folks to write middleware libraries that communicate with TensorFlow over sockets.

[rchipka]

Glad to see that there's official progress on this. I'd love to have fast, parallel GPU compute power at my fingertips with the ease and composability of JS.

I started working on a NodeJS binding for TensorFlow a while ago, but a haven't had much free time to devote to it lately.

I had three goals in mind for the project:

1. Don't require building or installing tensorflow

Instead, it should download and use the pre-built, multi-platform python binaries and download any needed source files on the fly.

2. Don't require a complete C++ or JS reproduction/abstraction of the API

Instead, it should provide a complete 1-to-1 interface with the C API, providing convenient JS abstractions as much as possible.

3. Don't maintain the C API bindings by hand

Instead, it should use a swig script to map the core data structures between Tensorflow/stdc++/V8/node and the rest will follow.

---

I got pretty far along with this, but last I remember a was having issues with TF_Session related segfaults.

I still feel that this approach is the most maintainable as it instantly provides the complete C API for any tensorflow version, which would quickly allow JS developers to start abstracting things.

We'd get a lot of things for free, one of which is graph export since we'd have access to libprotobuf via the python bindings.

@nkreeger @dsmilkov what are your thoughts on this approach? Is this something your team would consider or is this something I should jump back into at some point?

[zandaqo]

I've done some benchmarks comparing WebAssembly to C++ addons and JavaScript in Node. WebAssembly seem to suffer less penalties than C++/N-API addons for copying/converting data from JavaScript, however, addons still beat it significantly in tasks with low data transfer rate and heavy computational load. I imagine for model training N-API addon is far more preferable than the alternatives, while performance on inference may wary depending on the model.

[caisq]

cc @nkreeger

[zandaqo]

PropelML has bindings for the C API. I am not very cognizant of the APIs in question, but I assume there is a significant overlap since PropelML depends on deeplearn.js. Maybe some cooperation will help to speed this up?

[jart]

Node is actually capable of capable of computing anything that's computable. Sometimes it's a good idea to have the C++ code talk to Node using stdio.h or Berkeley Sockets.

[nsthorat]

If you want to have discussions about the merits of Node.js bindings itself we can move that to the community discussion. Let's keep this issue as technical / implementation details of the bindings as these points are not relevant to this issue.

[lastmjs]

I just want to bring up WebAssembly. I hope it's being considered as a potential way to elegantly solve problems between environments, most notably the browser and Node.js evironments. WebAssembly compiled through Emscripten has support for WebGL. It would be really nice to have a portable bytecode version of TensorFlow that could run in any wasm environment, even outside of Node.js and the browser.

[lastmjs]

@nsthorat That's great to hear. What about WebAssembly with WebGL? Emscripten should convert OpenGL to compatible WebGL code, if I'm not mistaken.

[erikwilson]

Just wanted to make a quick mention that, as far as I know and have tried the tfjs CPU backend does work in Node.js, so for people wanting to experiment or use small tensors the lack of an accelerated binding shouldn't be an issue.

Related discussions:
#78
#76

It would be nice if we could do something like:

const tf = require('@tensorflow/tfjs');
const tfNodeBackend = require('@tensorflow/tfjs-node-backend');

tf.setBackend(tfNodeBackend);

[jart]

Have we examined the webassembly implementation? Is it able to generate SSE, AVX, etc. CPU opcodes? Because let's assume: a) that hasn't been done; and b) webassembly is capable of choosing aligned non-overlapping (i.e. __restrict) memory addresses. In that case, I'd love to bring it to the attention of browser authors as soon as possible. It might be nice to have the option in the future to get a 10x improvement on webasm (for the scientific computing use case) even though GPUs are probably better for our purposes.

[erikwilson]

I agree with @nsthorat in that I think webassembly is a non-starter. There is already a Typescript CPU backend that works great, and I have a feeling that using WASM to create yet another CPU backend will yield little improvement in comparison to getting GPU acceleration working.

The bigger issue I think that has been hinted at, is WebGL fast enough for Node (in comparison to Python Tensorflow)? Personally I think the deeplearn-gl project (modified headless-gl) is a good start and I would like to see those changes in a form that can be pull requested back to the main project (stackgl). Supporting WebGL/tfjs in node would also be an easy benchmark against the browser and should be a baseline for measuring other methods of GPU acceleration.

[zandaqo]

in general we'll likely focus on TF C API for node

Is there any ETA/schedule for this? Can we follow this work, meaning, is there a separate project/branch for it?

Since PropelML is dropping its implementation in favor of tfjs, this becomes the only place where we can expect this feature. WebGL/WebAssembly is all good and fun, but to use Node.js for serious training we need to use CUDA and that means bindings for TF C API.

[camsjams]

Thanks @nkreeger looking forward to it!

[josiahbryan]

@erikwilson - I was just trying to use tfjs in Node - nonstarter for me, since requestAnimationFrame didn't exist (obviously), even when trying to use the raf polyfill from npm. (I was able to at least load tfjs using the esm module from NPM, so at least import * as tf from '@tensorflow/tfjs'; executed as expected.)

Can you share any info on how you got the CPU backend to work in Node? That would be useful to play with, at least for learning purposes.

Also, even with the CPU backend, how did you save models trained? AFAIK tfjs can't save models - yet (ref #13 - coming soon from what I read.)

[erikwilson]

@josiahbryan Are you calling requestAnimationFrame or tf.nextFrame yourself? Since there is no UI to render I imagine that part could be taken out?

Model & data i/o are pretty important, that is a whole other issue I was trying to tackle too and hope will be solved here soon.

[josiahbryan]

I agree, the saving of the data really would be a showstopper anyway. Regarding calling those frame functions, I wasn't calling it either of those, somewhere internally it was doing it itself and I couldn't figure out how to stop it.

…

On Tue, May 15, 2018, 9:48 AM Erik Wilson ***@***.***> wrote: @josiahbryan <https://github.com/josiahbryan> Are you calling requestAnimationFrame or tf.nextFrame yourself? Since there is no UI to render I imagine that part could be taken out? Model & data i/o are pretty important, that is a whole other issue I was trying to tackle too and hope will be solved here soon. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEmSLAcfxY5fNVM0g7cteCSpM1OUN5NJks5tytycgaJpZM4TB67O> .

[nsthorat]

Model exporting is very close to being done. We will have a release soon for browser exporting, and the node work will be next.

[erikwilson]

Are you able to provide a stacktrace without the polyfill @josiahbryan? It would be easy to shim requestAnimationFrame, but it shouldn't be needed and we don't want to pollute the global namespace.

[josiahbryan]

Sure, I can do that later this evening and I'll reply accordingly.

…

On Tue, May 15, 2018, 7:43 PM Erik Wilson ***@***.***> wrote: Are you able to provide a stacktrace without the polyfill @josiahbryan <https://github.com/josiahbryan>? It would be easy to shim requestAnimationFrame, but it shouldn't be needed and we don't want to pollute the global namespace. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEmSLHDUDK7y30bRp4OOoGDXDUW9wDwJks5ty2g0gaJpZM4TB67O> .

[alvinhui]

@nsthorat Is already allow export the model in the browser, what's the plan for node.js?

[caisq]

We are working on model importing and exporting in tensorflow.js-node.js. It will be based on the same IOHandler interface ( https://github.com/tensorflow/tfjs-core/blob/master/src/io/types.ts#L207) and file format as the browser-based importing and exporting. The work is being tracked by this GitHub issue: #343

…

On Wed, May 30, 2018 at 5:04 AM 梧忌 ***@***.***> wrote: @nsthorat <https://github.com/nsthorat> I see that is already allow export the model in the browser, what's the plan for node.js? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQC5fo9s7Cp6mUHZPlFm71M8i1lsxtX_ks5t3mCogaJpZM4TB67O> .

-- --- Shanqing Cai Software Engineer Google [email protected]

[nsthorat]

Closing this since we launched tfjs-node. Also, @nkreeger, it would be good to respond to @rchipka since I think you are already thinking along these lines.