nkzawa / socket.io-stream Goto Github PK
View Code? Open in Web Editor NEWStream for Socket.IO
License: MIT License
Stream for Socket.IO
License: MIT License
Is this possible? I've looked everywhere to try to find an example of it and can't seem to find one.
Hi!
I want to know how can you send custom data from a browser through one of your streams, how can you "write" on this stream anything that's not a string?
I want to pass an array (or bufferArray, or one of those) do I need to use a Buffer object?
Thanks!
You have an awesome implementation here, but the frontend (browser) library is a massive 147.975 kb minified, which for me makes it a too heavy implementation. I have no experience with browserify, can it compile a smaller lib?
Supporting Duplex streams that are both readable and writable would be really nice. Nothing should prevent it technically.
I'm currently going through your code to see how we could implement it.
Let me know what your status is on this and if you have any advice.
Thanks
Hello,
I'm having issues pausing and resuming the stream upload using BlobStream.pause() and and BlobStream.resume(). I read BlobStream returned a Readable Stream and that these methods were implemented. Pause does nothing, and resume emits 1 data event and pauses.
Have I missed something ? What would be the best way to pause and resume a stream upload?
Thanks a lot! :)
Hello, this module is very good, however we need more documentation, specially about:
Please, I'm looking forward to this. Let me know if I can help on this task.
Regards.
I need to forward socket.io-streams over a server (from client to another client), so I needed a way to handle any event. I already found tricks to do this with socket.io, see:
socketio/socket.io#1715
But since socket.io-stream works differently, it for example relies on overriding the event handler the users supplies (to replace the StreamID with a actual IOStream), I had to find a different solution.
The basics of my solution:
function createAnyEvents(socket) {
if(socket.sio && socket.$emit) {
// listen for stream events on original socket.io socket
socket.sio.on("stream",function() {
var args = Array.prototype.slice.call(arguments);
// Chanding original event to any event,
// adding original event type as argument
// from: eventType, pos, streamID, data, callback
// to: any, pos, eventType, streamID, data, callback
// Adding original eventType after pos:
args.splice(2,0,args[0]);
// Changing event type to any:
args[0] = 'any';
// Increment pos (streamID positions)
// (because we added eventType in front of them)
for(var i in args[1]) args[1][i]++;
socket.$emit.apply(socket,args);
});
} else {
debug("Error: Can't create 'any' event");
}
}
createAnyEvents(ss(socket));
ss(socket).on('any', function (eventType,incomingStream,data,callback) {
var outgoingStream = ss.createStream();
ss(consumerSocket).emit(eventType,outgoingStream,data,callback);
incomingStream.pipe(outgoingStream);
});
I'm posting this here so others can use it, maybe we can find a better solution in the future.
I do it as your introduce on the README to browserify. But When I reference it with src = '/js/socket.io-stream.js' or 'socket.io-stream/socket.io-stream.js', My browser complain that can't get the resource.
note: I start my server with node app.js.
var app = require("express")();
var http = require("http").Server(app);
var io = require("socket.io")(http);
var fs = require("fs");
//var ss = require("socket.io-steam");
var sockets = {};
app.get("/", function (req, resp) {
resp.sendFile(__dirname + "/view/message.html");
});
http.listen(3000, function () {
console.log("listening on 3000");
});
socket.io-stream uses FileReader
which is not supported in Firefox workers. Instead, FileReaderSync
can be used. Is there any way to work around this?
It appears that you are faking binary streams here. All you do is to convert these back to base64 so that it's easy to send in a string. See https://github.com/nkzawa/socket.io-stream/blob/master/lib/socket.js#L98
Can you explain?
npm install socket.io-stream
npm WARN package.json [email protected] No description
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No README data
npm WARN package.json [email protected] No description
npm http GET https://registry.npmjs.org/socket.io-stream
npm http 304 https://registry.npmjs.org/socket.io-stream
npm http GET https://registry.npmjs.org/socket.io-stream/-/socket.io-stream-0.5.2.tgz
npm http 200 https://registry.npmjs.org/socket.io-stream/-/socket.io-stream-0.5.2.tgz
> [email protected] prepublish z:\temp\Users\Farzher\AppData\Local\Temp\npm-501412-vqhKrfgb\socket.io-stream-0.5.21393148684527-0.14867000188678503\package
> make test-all build
make: *** No rule to make target `test-all'. Stop.
npm ERR! addLocalDirectory Could not pack "z:\\temp\\Users\\Farzher\\AppData\\Local\\Temp\\npm-501412-vqhKrfgb\\socket.io-stream-0.5.21393148684527-0.14867000188678503\\package" to "C:\\Users\\Farzher
\\AppData\\Roaming\\npm-cache\\socket.io-stream\\0.5.2\\package.tgz"
npm ERR! [email protected] prepublish: `make test-all build`
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the [email protected] prepublish script.
I'm noticing that when a client disconnects, an error event gets emitted on my server's socket.io-stream even though the server stream has already received an 'end' event. I would have expected that the 'end' event would exclude any further events from happening on my server read stream (except maybe 'close'), and that the stream would be cleaned up. Specifically, I would expect Socket.prototype._onend
to do something like stream.destroy()
or socket.cleanup(id)
to the readable end of my stream.
I might be overlooking something, but isn't this a memory leak? How should I close down my streams, if receiving the 'end' event on them doesn't do the job? Should I manually be calling destroy()
on an 'end' handler?
The symptom of this problem is that my server-side readable socket.io-stream receives the commands in Socket.prototype._ondisconnect
even after it has "ended", particularly the stream.emit('error', new Error('Connection aborted'));
. I wouldn't expect stream that I've already consumed to give me some error event a some future time, just because the client eventually disconnects.
Am I missing something? Thanks.
Hi there!
There is a stream error 'Invalid non-string/buffer chunk' when i'm trying to send file from node webkit:
stream = ss.createStream()
require("fs").createReadStream(file.path, highWaterMark: 16 * 1024).pipe stream
I feel like i need to convert file's chunks into suitable format but I didn't realize what it should be and how.
An error:
https://github.com/joyent/node/blob/master/lib/_stream_writable.js#L166
Hi
I try use socket.io-stream for streaming socket.io requests!
I have two nodejs servers connected by socket.io (v 1.0) and socket.io-client.
All requests from first server by Express redirect by Socket.io chennel and anotwher side by Request (module) transit to first server by socket.io-stream. (request().pull(soi-stream))
f i use get or post queryes all ok
but if i use xhr i have timeout on request().pull(soi-stream)
I can`t correct you code (all packed)
Thank you !
Right now since we toggle readable/writable everytime to synchronize reads/writes, I suspect that's damaging the performance of socket-io.stream.
Would be great if we could do something smarter rather than simply toggling. I think it would dramatically improve performance.
I am trying to implement socket.io-stream into my project, and I have noticed that when I stream objects to my client, I keep getting an invalid stream id error. Now it appears that all my data has been streamed successfully and the stream has ended successfully too, but it comes after the stream end event is sent and acknowledged.
Here is an image to show what I mean
Any suggestions or ideas would be appreciated.
Thanks!
May be to indicate in docs the browsers which support FileReader?
Hello, I'm trying to use socket.io-stream in a browser to upload a file. However, I keep getting this error, in the file change event.
"Uncaught ReferenceError: ss is not defined "
Am I missing something obvious? Please help.
Thanks in advance.
Here is the html source:
<script type="text/javascript" src="/socket.io/socket.io.js"></script> <script type="text/javascript" src="/js/socket.io-stream.js"></script> <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script> <script> $(function() { $('#file').change(function(e) { var file = e.target.files[0]; var stream = ss.createStream(); // upload a file to the server.
ss(socket).emit('file', stream, {size: file.size});
ss.createBlobReadStream(file).pipe(stream);
});
});
var socket = io.connect('http://lb1:8080');
</script>
Hi, Im trying to recieve a file from a server, so far I can not find the way to do it.
any help is appreciated!
Is there a function how to abort (cancel) all open streams asap?
It would be nice to have the option to create the read stream from a dataURL instead of a file. Something like this:
ss.createDataURLReadStream(dataURL)
Let me explain the scenario. I'm doing some image manipulation on a canvas in the browser (crop, resize, etc...) and I want to upload the processed image, not the image from disk.
Hello there
I experience performance issues with the following piece of code:
var stream = ss.createStream(),
self = this,
framesCount = self.framesCount;
this.canvas.toBlob(function(blob) {
// this is too slow!
ss(self.socket).emit(
'newFrame',
stream,
{
framesCount: framesCount
}
);
ss.createBlobReadStream(blob).pipe(stream);
}, 'image/jpeg', this.settings.video.quality);
My goal is to send images to the server in binary. I made measurements and found out that ss.createBlobReadStream(blob).pipe(stream);
is the slowest part.
Firefox alone needs about 50ms - 60ms to pipe that blob to the server. In other words the main thread is blocked by 50ms - 60ms!
Any clues?
Hello! Would you put a non-compressed version of the file 'socket.io-stream.js' in your repository? The other files are not minified, just this one. Thanks.
In the same manner as sending binary data on socket.io v1.0 like the following:
ss(socket).emit('myevent', [stream1, stream2]);
ss(socket).emit('myevent', {data: stream});
Hello everybody,
I have to build a web app in order to send the voice of my boss to all users who are connected to the society's website.
I am using NodeJS for my server, and Node-Core-Audio in order to get the sound of my soundcard (a microphone is connected to the computer where Node-Core-Audio is installed. So i'm able to send a JavaScript Array with 1024 audio samples to my NodeJS server
Node-Core-Audio (with NodeJs) --------------> NodeJS Server |---------------------> Users (AngularJS)
I'm using socket.io to transport the sound, but it's not perfect, i've got some latency periods.
I'm reading about socket.io-stream but it seems that it is only used to send files ?
How can I build a socket.io-stream which send all my arrays filled with 1024 audio samples ?
Is there a way to broadcast an image or another type of file from the browser to all nodes in browser as well?. All is good until the moment when I have to receive the image on all the sockets connected. Is there a seamless way to accomplish this?
Hi
I'm trying to use the latest socket.io-stream with socket.io v.1.3.5. I've set up the most basic example (as per the github page). Even though the event is handled correctly, I always get the error that stream has no method 'pipe' and that data is undefined. Socket.io itself runs without any issues. Tested on Chrome/Safari/Firefox.
ss(socket).on('file', function(stream, data) {
var filename = path.basename(data.name);
stream.pipe(fs.createWriteStream(filename));
});
Could there be any issues with the latest version of socket.io? (I'm also using the express framework on my nodejs server). I 'd love to use the library in my project, any help is apreciated.
We need to make sure that all these operations work without crashing on destroyed sockets.
I don't really have a snippet to reproduce this yet, that'll have to wait a few days since I'm very busy on other projects (for my Startup).
In one of the applications that streams terminals over socket.io-stream, I seem to have an issue when trying to open multiple streams over the same socket.
Do you have an idea where this is coming from ?
We will follow the implementation of net.Socket
, since IOStream
has similar nature.
allowHalfOpen
to false by default.destroy
method.destroy
method on finish
or end
when allowHalfOpen
was false.destroy
method after both finish
and end
were emitted when allowHalfOpen
was true.Hi nkzawa,
Thanks for creating socket.io-stream!
I'm wondering if it's possible to use callbacks with socket.io-stream. This would be very convenient.
Best wishes,
Rick
server.js
var fs = require('fs');
var app = require('express')();
var http = require('http').Server(app);
var io = require('socket.io')(http);
var ss = require('socket.io-stream');
io.of('/').on('connection', function(socket) {
ss(socket).on("getImage", function(cb) {
var outstream = ss.createStream();
cb(outstream); // 'announcing' a stream in a callback. Is this possible?
fs.createReadStream("image.jpg").pipe(outstream);
});
});
http.listen(3000, function(){
console.log('listening on *:3000');
});
client.js
var io = require('socket.io-client');
var ss = require('socket.io-stream');
var socket = io.connect('http://localhost:3000');
socket.on("connect", function() {
ss(socket).emit("getImage",function(stream) {
//how to decorate a stream received as argument from the server
//would something like socketIOStream(stream) be possible?
socketIOStream(stream).on("data", function(data) {
console.log("data",data);
});
});
});
I'm developing a server that needs to forward relatively big files (average of around 10MB). Streaming them from client to client using Socket.io-stream sounds ideal, because the server wouldn't need to buffer/store them.
I created a simple test where my server receives file through a socket.io-stream, creates a new stream and streams it to another client. But the problem was that the cpu was skyrocketing, with 10 clients, sending 10MB files every 10 seconds. So I simplified the server to just receiving socket.io-streams and writing them to disk (on RAM) (using pipe), but the problem persists. It's a problems because new clients can't connect anymore, probably because it's to busy.
When post files to a node.js server using Express, handling posts and piping incoming req to the same RAM disk it stays between 0% and 50%, when I have 100 clients, sending 10MB files every 10 seconds. (Using 50 to 190 MB memory)
Is there anyway I can tweak socket.io-stream. I tried increasing the highWaterMark but that seemed to create connection problems even earlier (new clients not being able to connect).
Currently when you call removeAllListeners on the socket connection that's used by socket.io-stream you break the socket.io-stream insance.
Should we overrule the removeAllListeners method and make sure we re-add the socket.io-stream listeners?
Something like:
var originalRemoveAllListeners = sio.removeAllListeners;
sio.removeAllListeners = function() {
originalRemoveAllListeners.apply(sio, arguments);
// re-add socket.io listeners
}
When I used socket.io-stream for a bigger project where I did things like pipe multiple streams and sometimes streams where not handled on the end I ran into lots of little memory leak issues. I really had to dive into how socket.io-stream works to figure out what was going wrong. So I'd like to come up with ways to make this easier for other people.
One of the "invisible" problem areas is that each socket.io-stream socket stores references to socket.io-stream streams it uses. When streams are not handled these streams keep piling up. This is especially problematic when you put streams in flowing mode and they already have a chunk of data in their buffer.
end()
and destroy()
that stream.Wouldn't it be nice to have the close/end events propagated to the child streams if either happens to their parent socket.
Is there anyway to broadcast the stream to all clients in a namespace or room?
I'd like to do something like this:
var nsp = io.of('/user');
nsp.on('connection', function(socket) {
// receive a image by a broadcasting client
ss(socket).on('image', function(incomingStream, data) {
debug("on image ",data);
// send the image
var outgoingStream = ss.createStream();
// emit stream to only one socket
//ss(consumerSocket).emit('image', outgoingStream, {name: data.name});
// broadcast stream to all other clients in namespace
ss(socket).broadcast('image', outgoingStream, {name: data.name});
incomingStream.pipe(outgoingStream);
});
});
Currently it's hard to know whether a incoming stream is being read. One of my workarounds has been to listen to the ss.Socket.event+"-read"
event on the socket instance.
I'd like some more properties and maybe events on the IOStream instance so it's easier to know the status of the stream.
Maybe we can add a status
propertie to the IOStream instance, that is one of the following:
idle
(The stream might have been emitted, but it's not being consumed yet)streaming
(The stream is being read)finished
(The stream has finished streaming)failed
(Some error has occored and the stream is aborted)Thoughts on this? I'm open to contribute on this, but I'd like some input first.
Hello
I have following questions
Is it correct that following code on client side is blocking
var socket = io.connect('http://localhost');
var stream = ss.createStream();
ss(socket).emit('foo', stream);
fs.createReadStream('foo.txt').pipe(stream);
If yes , do you see some problem to wrap the sending part above in promise based code :
Var d=Q.defer();
ss(socket).emit('foo', stream);
fs.createReadStream('foo.txt').pipe(stream);
....
return d.promise;
Additional question : what is recommended way to handle sending errors?
Thanks in advance
I want to cancel the process the which is streaming to server
It would be nice to be able to toggle debug, because it's quite verbose.
Which makes it hard to see debugging coming from other parts of an application.
Hi,
I have set up an upload system which reports progress etc, all taken from the main github page. Code:
Client
var stream = ss.createStream();
ss(Socket).emit('beginUpload', stream, $scope.video);
var blobStream = ss.createBlobReadStream($scope.videoFile);
var size = 0;
$scope.uploading = true;
blobStream.on('data', function(chunk) {
size += chunk.length;
var progress = (size / $scope.videoFile.size * 100).toFixed(2);
//console.log('progress', progress);
if (progress != 100) {
updateProgress(progress);
}
});
blobStream.on('error', function(err) {
console.log('ERROR', err);
});
blobStream.on('end', function() {
Socket.emit('uploadComplete', $scope.video);
});
blobStream.pipe(stream);
Server
ss(socket).on('beginUpload', function(stream, data) {
// Save stream to file
var filename = config.upload.tmpFolder+'/'+data.filename;
stream.pipe(fs.createWriteStream(filename)).on('error', function(err) {
console.log('crashed at begin upload', err);
});
});
This works perfectly on Chrome/Firefox & Internet Explorer from my tests, but for reason using Safari causes the following error on the server:
buffer.js:377
throw new TypeError('Unknown encoding: ' + encoding);
^
TypeError: Unknown encoding: buffer
at Buffer.write (buffer.js:377:13)
at new Buffer (buffer.js:215:28)
at IOStream.Readable.push (_stream_readable.js:122:15)
at push (/home/user/www/node_modules/socket.io-stream/lib/iostream.js:147:20)
at IOStream._onwrite (/home/user/www/node_modules/socket.io-stream/lib/iostream.js:153:5)
at Socket._onwrite (/home/user/www/node_modules/socket.io-stream/lib/socket.js:231:10)
at Socket.<anonymous> (/home/user/www/node_modules/socket.io-stream/node_modules/component-bind/index.js:21:15)
at Socket.EventEmitter.emit (events.js:106:17)
at Socket.onevent (/home/user/www/node_modules/socket.io/lib/socket.js:327:8)
at Socket.onpacket (/home/user/www/node_modules/socket.io/lib/socket.js:287:12)
Am I doing something wrong or is this a problem?
I'm trying to forward a file over socket.io-stream to multiple clients. But when the same stream is piped to two socket.io-stream clients, I notices that if one client doesn't handle the event (or use the stream) the other client can't fully retrieve the stream either.
I've tried to create an as simple as possible example.
How should I handle this? Am I trying to do something impossible?
I can't just add another read stream because in my actual use case that stream is another socket.io-stream that is send to the server.
To clarify, when both clients listen (handle the event and the stream) it works fine.
var socketClient = require('socket.io-client');
var debug = require('debug')('streams');
var ss = require("socket.io-stream");
var fs = require('fs');
var PORT = 5000;
var URL = "http://localhost:"+PORT;
server();
consumer("a",true);
consumer("b",false);
// SERVER
function server() {
var http = require('http').Server();
var io = require('socket.io')(http);
http.listen(PORT, function(){
debug('server listening on *:' + PORT);
});
var nsp = io.of('/');
var consumers = [];
nsp.on('connection', function(socket) {
var query = socket.handshake.query;
debug("server: new connection: ",query.name);
consumers.push(ss(socket));
});
setTimeout(function() {
var filename = "image.jpg";
var readStream = fs.createReadStream("assets/"+filename);
for(var i in consumers) {
var outgoingStream = ss.createStream();
consumers[i].emit('file',outgoingStream,{name:filename});
readStream.pipe(outgoingStream);
}
},500);
}
// CLIENTS
function consumer(name,listen) {
var serverSocket = socketClient(URL+"?name="+name,{forceNew:true});
var serverStreamSocket = ss(serverSocket);
serverSocket.once('connect', function(){
debug("consumer "+name+" connected");
if(listen) {
serverStreamSocket.on("file",function(stream,data) {
debug("consumer on file");
stream.pipe(fs.createWriteStream(name+"-"+data.name));
});
}
});
}
If I use the module via
require('socket.io-stream');
and then I compile my app.js with browserify, I receive the following error in web console:
Uncaught Error: Cannot find module './_stream_duplex'
I've build a service that would receive a socket.io-stream from one socket, and then tried to forward it to other socket's. This has lot's of challenges like, sharing a stream over multiple node instances and piping a stream to multiple sockets (in the same namespace). It was build as API, which meant that I couldn't control how it was used. I couldn't rely on client's listening to new stream events or then piping the stream.
I had to write some code to manage this, that I would like to share. Maybe something can be integrated into socket.io-stream or it can help others. Some of this might constitute a nice separate module, but since the project was aborted, I can't.
I used two basic solutions:
1. Limit the number of streams to one per namespace. Override an idle stream when a new stream is send, but block a new stream when a stream is already streaming.
I've written a StreamManager that stores streams and allows sharing streams over multiple node.js instances.
When forwarding a stream withing a namespace I would check whether a stream was already stored under an id (in my case the namespace name), and check using a isStreaming method whether it was already streaming. When it wasn't streaming I could remove the old stream and add a new one. When it was streaming I would respond with an error.
To check whether a stream was streaming I had to override the stream's push method, which is a ugly hack. I would love something like a streaming boolean in the native Stream API.
2. Add timeouts.
Because streams weren't always handled I had to build in timeouts. When adding a stream to this StreamManager you can also provide a timeout. This would auto destroy a stream automatically.
I'm trying to write data to the stream as it comes in, but nothing seems to be getting sent from the client to the server. When I attach an 'on data' event listener on the client everything breaks. When I attach an 'on data' event listener on the server it is never triggered.
1. @bStream = ss.createStream()
2.
3. # if i add this i see an error
4. @bStream.on 'data', (data) ->
5. console.log data.length
6.
7. ss(@socket).emit 'talk', @bStream, meta
8. @bStream.write myInt16ArrayBuffer
Error triggered on line 4 above.
Uncaught TypeError: Cannot read property '_read' of null
IOStream._read @ socket.io-stream.js:223
Readable.read @ socket.io-stream.js:3193
Readable.resume @ socket.io-stream.js:3562
Readable.on @ socket.io-stream.js:3528
I had everything working with Binaryjs, I wonder am I allowed to write to the stream like that with socket.io-stream?
Client/Browser upload file with file event.
ss(socket).emit('file', stream, {size: file.size, name: file.name})
Then I can pipe back to the client from server.
ss(socket).on('file', function (stream, data) {
var outStream = ss.createStream()
ss(socket).emit('file', outStream, data)
//outStream.pipe(stream) // this is wrong
stream.pipe(outStream)
})
But I want to sent it to the room. For example, send msg to the room with socketio / sio
sio.sockets.in('roomName').emit('msg', JSON.stringify(msgObj))
How can I sent stream to the room?
Regards,
Mike
Ref: http://stackoverflow.com/questions/10058226/send-response-to-all-clients-except-sender-socket-io
I get that in the browser (using browserify). What am I missing here?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.