silexlabs / unifile Goto Github PK
View Code? Open in Web Editor NEWUnified access to cloud storage services through a simple web API.
Home Page: https://silexlabs.github.io/unifile/
License: MIT License
Unified access to cloud storage services through a simple web API.
Home Page: https://silexlabs.github.io/unifile/
License: MIT License
We're very close to the 1.0.0 release but for now the only available connectors are:
I'd really love if someone would port the WebDAV (looking at you @Sergio-Muriel 😉) to the new version so I would have a valuable feedback from this person.
Hello
Here is what happens with the github service online (silex preprod)
http://preprod.silex.me/api/v1.0/github/connect/
Follow the auth url and you will see that the callback is hard coded 🌳
rclone looks like an open source competitor of unifile. It is interesting to me ang the supported services can give us ideas. Maybe the code too?
I tryed to rename a file from my github repo github page into unifile, it didn't succeed
The current www-auth
template has problems rendering nicely on different screen sizes. The below screenshot presents the problem (when loading it on the fly inside a modal popup in bootstrap).
The login.css
has few hardcoded styles (such as width, margin etc.) that I modified as below and it looks now decent enough with different screen sizes. Unable to attach the modified css file here, hence pasting the modified css content below.
.logo{
width: 32px;
height: 32px;
background-repeat: no-repeat;
background-size: cover;
}
.logo-ftp{
background-image: url('/unifile-assets/services/ftp.png');
}
.logo-www{
background-image: url('/unifile-assets/services/www.png');
}
.main-container {
display: inline-block;
padding: 25px;
border: 1px solid #D9D9D9;
-webkit-border-radius: 12px;
-moz-border-radius: 12px;
-ms-border-radius: 12px;
-o-border-radius: 12px;
border-radius: 12px;
}
.inner-container{
font-weight: normal;
display: inline-block;
text-align: left;
}
.title{
font-size: 18px;
font-weight: normal;
}
.all-text{
color: rgb(61, 70, 77);
font-family: 'Open Sans', 'lucida grande', 'Segoe UI', arial, verdana, 'lucida sans unicode', tahoma, sans-serif;
font-size: 14px;
text-align: center;
}
With the modifications now, the bootstrap model looks as below (width/height automatically adjusted itself):
On a narrow screen it looks like below (no more scrollbars):
May be related to #92 : I do a unifile.readFile then fs.writeFile and the new image is changed... Or it may be a problem of encoding?
I do something like this:
unifile.readFile(session, 'fs', 'preview1.jpeg').then(result => {
fs.writeFileSync('/home/lexoyo/preview2.jpeg', result, 'binary');
});
And the images differ:
preview1.jpeg starts with ÿØÿà^@^PJFIF^@^A^A^A^@H^@H^@^@ÿÛ^@C^@^C^B^B^C^B^B^C^C^C^C^D^C^C^D^E^H^E^E^D^D^E
whereas preview2.jpeg starts with ýýýý^@^PJFIF^@^A^A^A^@H^@H^@^@ýý^@C^@^C^B^B^C^B^B^C^C^C^C^D^C^C^D^E^H^E^E^D^D^E
I got this error when I click on "Pick" button. The loading popup screen shows up and nothings happens. This is from the console:
Error loading service services [TypeError: Cannot read property 'dropbox_access _token' of undefined] TypeError: Cannot read property 'dropbox_access_token' of undefined
How can we change the dropbox access token?
When entering wrong credentials on connect to a ftp, the server stop working.
Error received:
tfe@work:~/unifile$ node server.js
auth error for tfe { [Error: Login incorrect.] code: 530 }
auth error for tfe { [Error: OOPS: vsf_sysutil_recv_peek: no data] code: 500 }
auth error for tfe { [Error: OOPS: child died] code: 500 }
http.js:691
throw new Error('Can\'t set headers after they are sent.');
^
Error: Can't set headers after they are sent.
at ServerResponse.OutgoingMessage.setHeader (http.js:691:11)
at ServerResponse.header (/home/tfe/unifile/node_modules/express/lib/response.js:717:10)
at ServerResponse.send (/home/tfe/unifile/node_modules/express/lib/response.js:146:12)
at /home/tfe/unifile/lib/core/utils.js:25:37
at fs.js:268:14
at Object.oncomplete (fs.js:107:15)
Hello
When I clone the repo and do npm i
and then npm test
, I have failing unit tests, some of them because .should.be.fulfilled
is spelled .should.be.fulfilled
(ll instead of l) and these other ones:
1) FtpConnector createWriteStream() creates a writable stream:
AssertionError: expected { Object (_readableState, readable, ...) } to be an instance of Writable
at Context.<anonymous> (test/unifile-ftp.js:342:38)
2) FtpConnector createWriteStream() "after all" hook: Cleaning:
Error: ENOENT: no such file or directory, unlink 'tmp.test'
at Error (native)
at Object.fs.unlinkSync (fs.js:1089:18)
at Context.<anonymous> (test/unifile-ftp.js:353:7)
3) FtpConnector readFile() returns the content of a file:
AssertionError: expected '' to deeply equal 'lorem ipsum'
+ expected - actual
+lorem ipsum
at Proxy.assertEqual (node_modules/chai/lib/chai/core/assertions.js:1020:19)
at Proxy.methodWrapper (node_modules/chai/lib/chai/utils/addMethod.js:57:25)
at getBasePromise.then.then.newArgs (node_modules/chai-as-promised/lib/chai-as-promised.js:302:22)
at tryCatcher (node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (node_modules/bluebird/js/release/promise.js:509:31)
at Promise._settlePromise (node_modules/bluebird/js/release/promise.js:566:18)
at Promise._settlePromise0 (node_modules/bluebird/js/release/promise.js:611:10)
at Promise._settlePromises (node_modules/bluebird/js/release/promise.js:690:18)
at Async._drainQueue (node_modules/bluebird/js/release/async.js:138:16)
at Async._drainQueues (node_modules/bluebird/js/release/async.js:148:10)
at Immediate.Async.drainQueues (node_modules/bluebird/js/release/async.js:17:14)
4) SFTPConnector mkdir() creates a new directory:
Error: Unable to create remote dir. Does it already exist?
at sftp.mkdir.catch.catch (lib/unifile-sftp.js:147:29)
at process._tickCallback (internal/process/next_tick.js:103:7)
5) SFTPConnector writeFile() writes into a file:
Uncaught Error: write EPIPE
at exports._errnoException (util.js:1026:11)
at WriteWrap.afterWrite (net.js:795:14)
6) SFTPConnector "after each" hook: Tear down for "writes into a file":
Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
Are you aware of this? Any suggestions? I'll keep looking into it
Hello
I don't see any pagination mechanism in unifile, is it on purpose or on the roadmap?
Do you plan the use of an open protocol like RemoteStorage?
https://remotestorage.io/
Yesterday Silex was down all day long because the folder /lamp0/tmp/
was full of temp files
This must be caused by unifile somehow?
The user should be able to set a different name/description for a given connector (for locale purpose or customization) without redefining the connector completely.
GET /api/v1.0/dropbox/exec/rm/path/to/folder-or-file/ remove a file or directory
Is this okay?
@JbIPS great work on the 1.0.0 version of unifile
https://github.com/JbIPS/unifile/tree/1.0.0
When I do this I have errors like these
$ npm i && npm test
1) Unifile class constructor create a new instance with config:
AssertionError: expected undefined to deeply equal { key: 'value' }
at Assertion.assertEqual (node_modules/chai/lib/chai/core/assertions.js:485:19)
at Assertion.ctx.(anonymous function) [as equal] (node_modules/chai/lib/chai/utils/addMethod.js:41:
25)
at Context.<anonymous> (test/index.js:26:38)
2) Unifile class getAuthorizeURL() throws an error if connectorName is undefined:
AssertionError: expected [Function] to throw error matching /Unknown connector/ but got 'Cannot read
property \'toLowerCase\' of undefined'
at Context.<anonymous> (test/index.js:61:26)
Are the tests up to date, did I miss something? I have node v6.2.2...
Also I tried
$ cd samples
$ npm i
$ node simple-api-server.js
Listening on 6805
Then I open http://localhost:6805/
and click on "Login", but it opens a popup on http://localhost:6805/github/oauth-callback?code=a25b7f4b4e268321dcad&state=aaathub
and says "Cannot GET /github/oauth-callback?code=a25b7f4b4e268321dcad&state=aaathub"
Finally, how can I play with unifile and the command line? 🌱
When I upload a file I get this error
TypeError: Cannot read property 'then' of undefined
at /home/lexoyo/Projects/CloudExplorer2/simple-api-server.js:89:3
at Layer.handle [as handle_request] (/home/lexoyo/Projects/CloudExplorer2/node_modules/express/lib/router/layer.js:95:5)
at next (/home/lexoyo/Projects/CloudExplorer2/node_modules/express/lib/router/route.js:131:13)
at Form.<anonymous> (/home/lexoyo/Projects/CloudExplorer2/node_modules/connect-multiparty/index.js:103:9)
Is this code using unifile api correctly? Because this is what I use...
And here is the request I make
curl 'http://localhost:6805/GitHub/put/test2/gh-pages/linux-cifs-client-guide.pdf' -X PUT -H 'Pragma: no-cache' -H 'Origin: http://localhost:6805' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2609.0 Safari/537.36' -H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryifOPjjZJGmpQS9jV' -H 'Accept: */*' -H 'Cache-Control: no-cache' -H 'X-Requested-With: XMLHttpRequest' -H 'Cookie: ...' --data-binary '[my file data]' --compressed
A github and a dropbox apps are used to authorize access to github and dropbox. They have secret and public keys hard coded in default-config.js.
We should probably not put this config data there, and force the developer who hosts an instance of unifile to provide his own keys, either in a config.js file, or in env vars.
Is it possible to remove several files at once? Is it interesting to have such a mechanism?
I have an error when I try to list "/":
Error in call to API function "files/list_folder": request body: path: Specify the root folder as an empty string rather than as "/".
It comes from dropbox js sdk and it is discussed here:
This is expected. With Dropbox API v2, the root is expressed as empty string "", and non-root paths are expressed with a leading slash, e.g. "/test.txt".
Do you think that Unifile should include a workaround? I think so... I can add this myself, just tell me if what you think?
As a user I should not have to provide a state
to unifile connectors as it is more secure that you generate it
As you may know Cloud Explorer is a web client to use unifile backend
Cloud Explorer has a bug when using the new unifile service "github" by @JbIPS, user has to refresh window after login to access his github content.
I believe the bug occures because here in the source code of cloud explorer, result
is undefined... But I did not investigate further for now
We shouldn't try to guess the encoding of a file. For consistency, we should always return a Buffer, letting the user choose how to deal with it.
We should use CozyCloud as a connector.
They have an OAuth flow described here.
@KrishnaPG got me thinking about the next version of unifile
Here is what I think we should do
And here is the details, feel free to help me on this definition of unifile v1
In node, for use with slenium or on the server side when you already have a token (the user has authorized the app)
var unifile = require('unifile')
var dropbox = new unifile.Dropbox({
root : 'dropbox',
app_key : 'ngk5t8312eo4agh',
app_secret : '7uxqr625pct7zbl'
})
dropbox.connect(function (authorizeUrl) {
// open authorizeUrl, login and authorise the app
// then call this
dropbox.setSession({
access_token: 'the access_token from dropbox'
})
dropbox.ls('/', function(files) {
console.log('files and folder at the root of your dropbox:', files);
});
});
Use with CLI
$ unifile dropbox ls /
waiting for you to authorize the app (a browser window has opened)
app authorized
file 81 Nov 20 23:22 admin.html
file 302 Nov 20 21:24 login.html
folder 136 Nov 20 23:34 node_modules
Use the middleware to activate a service and expose the HTTP route
var unifile = require('unifile')
var dropbox = new unifile.Dropbox({
root : 'dropbox',
app_key : 'ngk5t8312eo4agh',
app_secret : '7uxqr625pct7zbl'
})
var router = new unifile.Router(dropbox);
var connect = require('connect')
var app = connect()
app.use('/api/dropbox', router)
use GET, POST, UPDATE, DELETE...
See these interesting links
*Google Cloud Storage API Reference
Everything should be cleaned up and ported to ES2015, whith promise, the API should mimic Node.js File System api
See these links for the API methods specs
FTP service returns "." and ".." folders...
This comes from this issue of Cloud Explorer
silexlabs/cloud-explorer#30
cc @zabojad
What is the best way to implement multi-tenancy with this package?
For example, for the www
, I would like to have user x
, mapped to data/x
as the root folder and user y
mapped to, say data/y
, as the root www folder.
This should happen upon successful login (automatically), so that ls /
for user x gives contents of folder data/x
and for user y gives contents of data/y
.
Is it possible in the current version?
Use this lib which looks cool
https://github.com/visionmedia/debug
The goal is to offer a new service in unifile, to use github to store your files. The advantages are
I have tested the github API, which is quite simple to use, without any SDK or library, just the HTTP API, see my little nodejs test:
https://gist.github.com/lexoyo/932550e8ddbae2ec1d2d
But there is the question of the UI and how we handle repos and branches in unifile. First I thought that
/xxx
would be the repo xxx/xxx/yyy/
would be the branch yyy of the repo xxx/xxx/yyy/zzz
would be the folder or file zzz on the branch yyy of the repo xxxBut then I thought that the user will try to store files at the repo or branch level, and will not understand what they are doing. So I see several solutions
info
field in the json response which tells the user that "at this level, you can not store files, you need to have only folders" or do not even allow to create folders for repos and branches but say "you can not create files or folders here, you need to create a repository in your github acount"The next question would then be how to make people understand how to publish on github pages, they need to publish their site on a branch called gh-pages...
the rm function on ftp directories doesn't work: result is a 500 error.
When there is an authentication error, the FTP service logs something like this in the console ERROR { Error: Login authentication failed
and returns this in the response:
1- Do you think we should display an error page, or is the JSON on purpose?
2- Maybe also more data about why it failed?
3- There is the login and password in the URL of the callback in GET, do you think we could make it more secure? (POST at least or in the session)
When I receive an error object, his content
property is of type buffer instead of string
Do you think it is linked to the previous PR I did?
Hello
I guess many services offer to filter by mime type or file extensions
Do you think we should add that to readdir function? Or filter after getting the result is better?
I use github service
I saved a website many times and published many times
It made a repo with so many commits github refuses to save
I squashed these commits and now I am able to save again :'(
What do you think about have a SSH connector? A lot of Node.js already implemented SSH client and this would mean a lot of newly available servers for Unifile users 😃
I think we should provide app keys by default for Dropbox and Github, but I am not sure that this is possible because sometimes the app will have the port and callbacks...
Otherwise a doc about where to get it and how to set it up with unifile would be helpful.
It will be awesome to be able to have a multi-login for the dropbox/ftp/... services, without having to disconnect/reconnect.
The main problem is that the /exec/ls/path like commands does not specifiy the account they are asking, so we cannot know which connected account use.
Maybe with a query parameter?
Hi,
I think I have a bug with unifile (1.0.0) and the github connector
I make this request to rename a file and it fails (I have cleanedup the request a bit):
curl 'http://localhost:6805/GitHub/mv/test2/gh-pages/2016-01-02-your-filename.md' -X PATCH 'Content-Type: text/plain;charset=UTF-8' -H 'Accept: */*' --data-binary '{"destination":"test2/gh-pages/new_name.md"}' --compressed
The problem seems to be that getPathTokens
receives an undefined path here
https://github.com/JbIPS/unifile/blob/1.0.0/lib/unifile-github.js#L17
PS: the fork of @JbIPS seems different from the one of silexlabs
When logging on a service with ftp, and an error occured, the message
Invalid access codes
is displayed and the form is displayed again, but the data previously entered by the user are lost, and he has to reenter all the form data.
For the www
service, if any directory entry has permission issue, ls
is failing completely, throwing below style of error, instead of gracefully handling it and moving on.
express:router anonymous /api : /api/v1.0/www/exec/ls/ +1ms
{ [Error: EPERM, operation not permitted '\ui\index.html~RFbbf65e4e.TMP']
errno: -4048,
code: 'EPERM',
path: '\ui\\index.html~RFbbf65e4e.TMP',
syscall: 'stat' }
error code 500 { success: false,
message:
{ [Error: EPERM, operation not permitted '\ui\index.html~RFbbf65e4e.TMP']
errno: -4048,
code: 'EPERM',
path: '\ui\\index.html~RFbbf65e4e.TMP',
syscall: 'stat' } }
In many scenarios the permission issues are very common and a simple ls
should be capable of ignoring such errors and move ahead with rest of the list (instead of bailing out completely, causing 500 error in the server)
This can be easily fixed by moving the try .. catch block inside the for loop (closer to fs.statSync()
)
I can implement and do a PR for this, if ok with you.
Also, few other things:
fstat
details are being returned to the caller. I would say, returning the full fstat
object (+ the usual additional extra derived fields) so that caller can take what they want and ignore the rest, is much better option. For example, user cannot derive mode, creation_time etc. if the results are already pre-filtered at the unifile level.ls
should return exactly those fields. But I would leave it to some future versions, may be v2.0 or v3.0. For now, a full fstat object with compression enabled on the results (at the nodejs express layer) should be fine enough.fstat
would be supported by all other services (such as dropbox). But I guess that would have to be taken care by the caller (being at the app level), rather than a middle-ware library like unifile. Besides, caller can choose not to use all fields, if they want to.fs
. By switching to the fs-extra
, more options can be implemented in the future (such as recursive file copies, directory deletions etc.), without extra code. Also fs-extra
is fully promise
enabled (just in case in v2.0/v3.0 some one needs it).For a while now, automatic tests fail
https://travis-ci.org/silexlabs/unifile
And we should have a precommit hook to avoid pushing code which fails the tests
I think I've spotted some issue with logging out from services:
Here is a network call snapshot from my web browser:
logout
/api/1.0/dropbox
Status Code:200 OK
{"success":true,"message":"Now logged out."}
logout
/api/1.0/www
Status Code:200 OK
{"success":true,"message":"Now logged out."}
ls/
/api/1.0/www/exec
Status Code:401 Unauthorized
{
"success": false,
"message": "User not connected yet. You need to call the 'login' service first.",
"code": 401
}
list
/api/1.0/services
Status Code:200 OK
[{"name":"dropbox","display_name":"Dropbox","image_small":"unifile-assets/services/dropbox.png","description":"Edit html files from your Dropbox.","visible":true,"isLoggedIn":true,"isConnected":true,"isOAuth":true,"user":{"display_name":"Thomas Fétiveau","quota_info":{"available":2952790016,"used":2402940491}}},{"name":"www","display_name":"Web server","image_small":"unifile-assets/services/www.png","description":"Edit files on the server where Silex is installed.","visible":true,"isLoggedIn":false,"isConnected":false,"isOAuth":false},{"name":"ftp","display_name":"FTP","image_small":"unifile-assets/services/ftp.png","description":"Edit files on a web server.","visible":true,"isLoggedIn":false,"isConnected":false,"isOAuth":false,"user":{"host":"blablablabla","port":"21","username":"blabla","password":"blablabla"}}]
ls/
/api/1.0/dropbox/exec
Status Code:200 OK
[{"name":"anim3d-zabo-nico","bytes":0,"modified":"Wed, 21 May 2014 19:22:33 +0000","is_dir":true},{"name":"apps","bytes":0,"modified":"Tue, 10 Jun 2014 09:04:47 +0000","is_dir":true},{"name":"EuropaApps","bytes":0,"modified":"Mon, 16 Dec 2013 08:19:23 +0000","is_dir":true},{"name":"Launcher","bytes":0,"modified":"Mon, 16 Dec 2013 08:19:23 +0000","is_dir":true},{"name":"Player_v60x","bytes":0,"modified":"Mon, 16 Dec 2013 08:19:23 +0000","is_dir":true},{"name":"pol","bytes":0,"modified":"Fri, 27 Sep 2013 08:25:13 +0000","is_dir":true},{"name":"test1.htm","bytes":22,"modified":"Thu, 17 Apr 2014 18:48:14 +0000","is_dir":false},{"name":"test2.htm","bytes":22,"modified":"Thu, 17 Apr 2014 18:48:30 +0000","is_dir":false},{"name":"test3.htm","bytes":22,"modified":"Sun, 18 May 2014 15:06:35 +0000","is_dir":false},{"name":"test4.htm","bytes":22,"modified":"Sun, 18 May 2014 15:07:05 +0000","is_dir":false}]
The problem here is that we expect the call to services/list to return the services with LoggedIn set to false. And it appears to be true for Dropbox... The last ls call should return 401 forbidden. It seems logging out from DB did not succeed whereas it returned true...
Note that the calls after the logout calls (ls, list, ...) happened when logout calls already returned to my browser.
As pointed by @lexoyo, we should add a .stat()
function that return an object description (same as .ls()
)
hello
thank you so much for the awsome library
when i do this, what is the best way to set the mimetype header? is there a way to access the mimetype header as provided by the third party service?
app.get(/\/(.*)\/get\/(.*)/, (req, res) => {
this.unifile.readFile(req.session.unifile, req.params[0], req.params[1])
.then((result) => {
res..send(result);
})
.catch((err) => {
console.error(err);
res.status(400).send(err);
});
});
Could you please reconsider the license?
MIT, BSD or Apahce 2 would be more apt for this. I see Cloud Explorer (which is using this) is already using MIT.
Not really major problem, but came across this and thought of notifying it:
In the lib/default-config.js
file
__dirname +
(since __dirname does not end with /
it is causing the paths such as /lib../
).path: __dirname + '../../unifile-assets/'
path: __dirname + '/../../unifile-assets/'
ROOT : __dirname + '../../www',
ROOT : __dirname + '/../../www',
Other lines seem to be fine.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.