Scalability

[DomVinyard]

Scalability

[DomVinyard]

🙂 ooh, do let me know if you do! and happy christmas!

[tunnckoCore]

Okey, I'll ping you if I can do something. Happy to you too.

[tunnckoCore]

Mm not sure. It just does single call of fs.readdir with given dir. I don't tested it yet, but I believe recursion can be done easily through plugins - no matter it is "regular/smart" or "pipeline" plugin. Actually these "smart plugins" is just a step before pushing the paths to the stream, so then these "pipeline" plugins (think "gulp plugins" which are just [transform] streams) will be used only on these paths.

The cool and lightweight thing is that it doesn't do any more I/O than fs.readdir, so you can add plugin that calls fs.(l)stat, then check if filepath is directory and etc.

But in anyway, I won't recommend it for such case. It exists for another reason.

In your case you, it may be better to use glob-stream package or directly glob. I have absolutely times faster and better library (than mentioned) for globbing and recursion and such stuff. But it's not stable and public currently.

[DomVinyard]

Thank you for your help, unfortunately glob also appears to use fs.readdir rather than the underlying POSIX version (unlike the module I linked above) and so the problem still exists.

[tunnckoCore]

I don't know why you want to use natives, when nodejs tries to do its best. And globbing is largely used everywhere. But okey, you may want to try https://github.com/jonschlinkert/bash-glob then, which uses native bash and globbing. Just not sure how good it works in Windows.

[DomVinyard]

with ten million files? Node does try its best, but as memory shoots into the gigabytes the best is sadly not good enough. I will read about bash-glob, thank you for your advice, it is very very much appriciated

[tunnckoCore]

with ten million files?

Not sure. In anyway it seems like an edge case. Try first, before searching for native things.

It's interesting to me, what's that case? What are these files? What will do them?

edit: i'll go close this, cuz I can't do anything and because it's out of the scope and purpose of this package.

[DomVinyard]

Yes, it should be closed.

It is a piece of medical analysis equipment which writes this many files into a single folder. This is obviously insane (!), but we are not in control of the hardware and the output is out of our hands. Perhaps node is not the correct technology for this project but I think with C++ bindings it should be possible!

Thank you for all your help and good luck with this project.

[tunnckoCore]

Huh, yea, strange case.

Maybe I can think for wrapper or something in a future.