fabe/gatsby-plugin-drive

🚗 Downloads a Google Drive folder that you can query with gatsby-source-filesystem.

gatsby-plugin-drive's Issues

Version 0.2.3 not working

When i installed the package using "yarn add @fs/gatsby-plugin-drive", in the node_modules folder, @fa/gatsby-plugin-drive folder I don't find the gatsby-node.js file. so "gatsby develop" not invoking this plugin. This issue is there in 0.2.3 version.

Note that the 0.2.1 version works fine.

I analyzed the npm package file https://registry.npmjs.org/@fs/gatsby-plugin-drive/-/gatsby-plugin-drive-0.2.3.tgz which is not having a gatsby-node.js file. Looks like it is a packing problem.

Unable to download files

Hello,

I've currently set up my config like so:

{
      resolve: '@fs/gatsby-plugin-drive',
      options: {
        folderId: '1EHtphzknN_vt9QtVjl1AziHu2_p0VEYL',
        keyFile: path.resolve(__dirname, 'client_secret.json'),
        destination: path.join(__dirname, 'src/content'),
        exportGDocs: true,
        exportMimeType: 'application/pdf',
      }
    },

I've got two documents in that folder that I'd like to use. And I have shared those docs with the specified the email but the download seems to be failing. Here's the error:

🚗  Started downloading content
⠸ onPreBootstrap

 ERROR 

UNHANDLED REJECTION filesToDownload is not iterable



  TypeError: filesToDownload is not iterable
  
  - gatsby-node.js:42 
    [karanr.dev]/[@fs]/gatsby-plugin-drive/gatsby-node.js:42:22
  
  - new Promise
  
  - gatsby-node.js:38 recursiveFolders
    [karanr.dev]/[@fs]/gatsby-plugin-drive/gatsby-node.js:38:10
  
  - gatsby-node.js:30 
    [karanr.dev]/[@fs]/gatsby-plugin-drive/gatsby-node.js:30:5
  
  - task_queues.js:93 processTicksAndRejections
    internal/process/task_queues.js:93:5

Any idea what's going wrong?

silently truncates folders at around file number 100

The Google Drive Files API v3 requires you to track page tokens to know whether there are more files to download. Currently the queries are truncated to the first "page" which seems to be a little over 100 files. It needs to track nextPageToken to handle this correctly. But it's a pain to rewrite because promises etc.

Deleting files/rate limiting

More of a question really, but I've just installed the plugin and am using it quite successfully, but then got stung by the rate limiting.

My use case is I'm using Google Drive to store images for a couple of areas of the site. I have two folders, one with (currently) six images in it, the other has around 30 images.

I was hoping that I'd be able to effectively "sync" the folders to my Gatsby site, so that when a file is deleted, a Gatsby rebuild would also remove that file. Obviously that's much more tricky from the plugin point of view, so I figured I could just clean those directories before a build (as the docs suggested) and do a fresh pull of the Google Drive folder every time.

It feels like the rate limiting is going to get in the way of that though, but I'm not sure how it's doing the calculations. Given that I should be allowed 1000 requests every 100 seconds, it surprised me that I got blocked after downloading less than 40 images. The developer console isn't much help either!

Just wondering if you know any more about how the requests are used?

folder tree / images files

I'm new using Gatsby, I'd like to know if there's any way of query a folder tree and .png, .jpg files. And, if so, could you please provide some query examples?
Thank you in advance

Two files with same name results in silent overwrite

Kind of an edge case, but I'm syncing photos to a drive folder from my phone, and the phone names them with the date (e.g., 'January 08, 2019 at 0634PM.png'). If I take two pictures in the same minute, they get the same name. This (oddly) isn't a problem on my phone or on drive, but when the files are downloaded, only the second one would remain in this case (the first one with the same name being overwritten).

Not sure of the best way to handle this case, though.

Thanks for starting this project 👍

Passing in credentials from environment variables

Is there a way to use environment variables instead of a keyFile to pass in the private key? It seems node-gtoken supports it via options.key.

Rate limiting

when downloading many files from google drive, I quickly hit an API rate limit. Instead of the actual file the following file is created:

403: User Rate Limit Exceeded

The per-user limit has been reached. This may be the limit from the Developer Console or a limit from the Drive backend.

{ "error": { "errors": [ { "domain": "usageLimits", "reason": "userRateLimitExceeded", "message": "User Rate Limit Exceeded" } ], "code": 403, "message": "User Rate Limit Exceeded" } }

Suggested actions:

Raise the per-user quota in the Developer Console project.
If one user is making a lot of requests on behalf of many users of a G Suite domain, consider a Service Account with authority delegation (setting the quotaUser parameter).
Use exponential backoff.

This can be resolved by deleting affected files and retrying. However this is quite cumbersome for automation.

I solved it with a dirty fix: a sleep of ~1000ms right before the getFile() call.

log(`sleep for ${file.id}`);
sleep(1100);
const buffer = file.mimeType === GOOGLE_DOC ? await middleware(googleapi.getGDoc(file.id, token, exportMime)) : await googleapi.getFile(file.id, token);
[...]
const sleep = (ms) => {
  var start = new Date().getTime(), expire = start + ms;
  while (new Date().getTime() < expire) { }
  return;
}

Thank you for your great work on this plugin 👍

Final promise not resolved on Gitlab?

We deploy our Gatsby app on Gitlab and while the plugin works beautifully locally with both gatsby develop and gatsby build, the build simply hangs on the final promise when deploying to Gitlab. I've tried a whole lot of options but I'm stuck.