parse-community/parse-server-gcs-adapter

parse-server adapter for Google Cloud Storage

Parse Server GCS Adapter

---

The Parse Server Google Cloud Storage Adapter.

---

• Installation
• Usage with Parse Server
  • Config File
  • Environment Variables
  • Instance Parameters
  • Options
  • Obtaining Credentials File

Installation

npm install --save @parse/gcs-files-adapter

Usage with Parse Server

Config File

{
  // Parse server options
  appId: 'my_app_id',
  masterKey: 'my_master_key',
  // other options
  filesAdapter: {
    module: '@parse/gcs-files-adapter',
    options: {
      projectId: 'projectId',
      keyFilename: '/path/to/keyfile',
      bucket: 'my_bucket',
      // optional:
      bucketPrefix: '', // default value
      directAccess: false // default value
    } 
  }
}

Environment Variables

Set your environment variables:

GCP_PROJECT_ID=projectId
GCP_KEYFILE_PATH=/path/to/keyfile
GCS_BUCKET=bucketName

And update your config / options

{
  // Parse server options
  appId: 'my_app_id',
  masterKey: 'my_master_key',
  // other options
  filesAdapter: '@parse/gcs-files-adapter'
}

Alternatively, you can use

GCLOUD_PROJECT and GOOGLE_APPLICATION_CREDENTIALS environment variables.

Instance Parameters

var GCSAdapter = require('@parse/gcs-files-adapter');

var gcsAdapter = new GCSAdapter(
  'project', 
  'keyFilePath', 
  'bucket' , {
    bucketPrefix: '',
    directAccess: false
  }
);

var api = new ParseServer({
  appId: 'my_app',
  masterKey: 'master_key',
  filesAdapter: gcsAdapter
})

or with an options hash

var GCSAdapter = require('@parse/gcs-files-adapter');

var gcsOptions = {
  projectId: 'projectId',
  keyFilename: '/path/to/keyfile',
  bucket: 'my_bucket',
  bucketPrefix: '',
  directAccess: false
}

var gcsAdapter = new GCSAdapter(gcsOptions);

var api = new ParseServer({
  appId: 'my_app',
  masterKey: 'master_key',
  filesAdapter: gcsAdapter
});

Options

• directAccess: if set to true, uploaded files will be set as public and files will be served directly by Google Cloud Storage. Default is false and files are proxied by Parse Server.

Obtaining Credentials File

See the Google Cloud documentation for how to generate a key file with credentials.

parse-server-gcs-adapter's Issues

Security issues

There are multiple security issues being reported by snyk https://snyk.io/test/npm/@parse/gcs-files-adapter/1.1.2
Can you please update your dependencies to latest versions or versions without security issues?

direct access url zertificate issues

Hi

there seems to be an issue with file URLs returned by parse server with GCS when direct access = true is set.
I get (net::ERR_CERT_COMMON_NAME_INVALID)
If line 133 is changed to it should work:
return https://storage.googleapis.com/${this._bucket}/${this._bucketPrefix + filename};

This is also the format the google cloud console returns if you use a public link.

Please remove deprecated 'gcloud' module and replace with 'google-cloud'

gcloud module has now been moved to google-cloud

Unable to delete a file stored in Google Cloud Storage using Parse.Cloud.httpRequest

Issue Description

Unable to delete a file stored in Google Cloud Storage using Parse.Cloud.httpRequest.

Steps to reproduce

Parse.Cloud.httpRequest({
	method: 'DELETE',
	url: photoUrl,
	headers: {
		"X-Parse-Application-Id": process.env.APP_ID,
		"X-Parse-Master-Key" : process.env.MASTER_KEY,
		"X-Parse-Session-Token": requester.getSessionToken(),
	}
})
.then((httpResponse) => {
	console.log(httpResponse.text);
})
.catch((error) => {
	console.error('Request failed with error:' + error.text);
})

Expected Results

File is deleted

Actual Outcome

Google Cloud Storage returns Access denied error.

Environment Setup

• Server

  • parse-server version : 2.3.8
  • parse-server-gcs-adapter version : 1.1.2
  • Operating System: MacOS Sierra 10.12.4
  • Hardware: Apple iMac
  • Localhost or remote server? Localhost

• Google Cloud Platform

  • Service account permissions set to ["Editor", "Storage Admin", "Storage Object Admin"]

Logs/Trace

Parse-server console output:

verbose: REQUEST for [DELETE] /parse/classes/Photo/HDLbppIQpB: {} method=DELETE, url=/parse/classes/Photo/HDLbppIQpB, host=192.168.0.190:1337, connection=keep-alive, content-length=195, origin=http://localhost:3003, user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36, content-type=text/plain, accept=*/*, referer=http://localhost:3003/, accept-encoding=gzip, deflate, accept-language=en-US,en;q=0.8, 
Request failed with error:<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous users does not have storage.objects.delete access to bucket <my bucket>.</Details></Error>

Google Cloud Storage log:

cs_method: DELETE
cs_uri: /<bucket name>/76cf0e7588857ca92cdcc1bfb8540c6b_Room.jpg
sc_status: 403
cs_bytes: 0
sc_bytes: 208
cs_host: storage.googleapis.com
cs_user_agent: gzip(gfe)

I've also successfully tested DELETE using Google OAuth 2.0 Playground using my personal gmail account set with same permissions as the service account used by Parse Server. So, I'm reasonably confident that my permissions are set correctly.

Upgrade to typescript

I have written a TS version for myself, feel free to use the same code although it would need changes to be compiled

import { Storage, StorageOptions } from '@google-cloud/storage';


interface GCSAdapterOptions {
    projectId?: string;
    keyFilename?: string;
    bucket: string;
    bucketPrefix?: string;
    directAccess?: boolean;
    baseUrl?: string;
}

type GCSAdapterConstructorArgs = [GCSAdapterOptions];
function optionsFromArguments(args: GCSAdapterConstructorArgs): GCSAdapterOptions {
    const defaultOptions: Partial<GCSAdapterOptions> = {
        projectId: process.env['GCP_PROJECT_ID'],
        keyFilename: process.env['GCP_KEYFILE_PATH'],
        bucket: process.env['GCS_BUCKET'],
        bucketPrefix: process.env['GCS_BUCKET_PREFIX'] || '',
        directAccess: process.env['GCS_DIRECT_ACCESS'] === 'true' ? true : false
    };

    return {
        ...defaultOptions,
        ...args[0]
    };
}

class GCSAdapter {
    private _bucket: string;
    private _bucketPrefix: string;
    private _directAccess: boolean;
    private _gcsClient: Storage;
    private _baseUrl: string;

    constructor(...args: GCSAdapterConstructorArgs) {
        let options = optionsFromArguments(args);

        this._bucket = options.bucket;
        this._bucketPrefix = options.bucketPrefix || '';
        this._directAccess = options.directAccess || false;
        this._baseUrl = options.baseUrl || 'https://storage.googleapis.com/';
        this._gcsClient = new Storage(options as StorageOptions);
    }

    async createFile(filename: string, data: Buffer | string, contentType?: string): Promise<void> {
        const params = {
            metadata: {
                contentType: contentType || 'application/octet-stream',
            },
        };

        const file = this._gcsClient.bucket(this._bucket).file(this._bucketPrefix + filename);
        await new Promise((resolve, reject) => {
            const uploadStream = file.createWriteStream(params);
            uploadStream.on('error', reject);
            uploadStream.on('finish', resolve);
            uploadStream.end(data);
        });

        if (this._directAccess) {
            await file.makePublic().catch(err => {
                throw new Error(`Error making file public: ${err}`);
            });
        }
    }

    async deleteFile(filename: string): Promise<void> {
        try {
            const file = this._gcsClient.bucket(this._bucket).file(this._bucketPrefix + filename);
            await file.delete();
        } catch (err) {
            throw new Error(`Error deleting file ${filename}: ${err}`);
        }
    }

    async getFileData(filename: string): Promise<Buffer> {
        try {
            const file = this._gcsClient.bucket(this._bucket).file(this._bucketPrefix + filename);
            const [exists] = await file.exists();
            if (!exists) {
                throw new Error(`File ${filename} does not exist.`);
            }
            const [data] = await file.download();
            return data;
        } catch (err) {
            throw new Error(`Error downloading file ${filename}: ${err}`);
        }
    }


    getFileLocation(config: { mount: string; applicationId: string; }, filename: string): string {
        if (this._directAccess) {
            return `${this._baseUrl}${this._bucket}/${this._bucketPrefix + filename}`;
        }
        return `${config.mount}/files/${config.applicationId}/${encodeURIComponent(filename)}`;
    }
}


export default GCSAdapter;

Narrowing down gc dependencies

I’m new to this and I might be missing something, but what about narrowing down google dependencies?

We have a dependency on google-cloud which is huge, see /packages.
Looks like we only need @google-cloud/storage.

Seek for audio / video doesn't work

@flovilmart
Confirming this issue using the GCS Adapter. My files seek/ buffer fine on googles side but not as PFfiles. Is there any recommended fix for GCS Adapter ??

Direct access works fine. But we want to continue using the indirect access dues to authentication restrictions.

File is saved multiple times, and error is returned

I am trying to save a png image in a ParseFile from the android SDK. I do what I have always successfully done with parse.com. Just for reference:

ByteArrayOutputStream stream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, stream);
ParseFile imageParseFile = new ParseFile(filename, stream.toByteArray(), "image/png");
imageParseFile.saveInBackground();

The save operation fails after a while, with a ParseRequestException (code 100) caused by a SocketTimeoutException. At the same time, here’s what happens to parse-server:

• no error logs (running in verbose mode)
• 5 (not one) successful POST requests to parse/files/file.png
• file is, in fact, saved 5 times in my bucket

Any idea? I am 100% sure that save() is called just once. Somehow that is looped and repeated. Should I post this @ Parse-SDK-Android ?

Parse push not working on Android Marshmallow (6.0) after shifting Parse to my own server

Parse push not working on Android 6.0 after moving Parse server to my own server. It is working fine for all other versions of Android.

Update README

Update documentation the need of projectId & keyFilename

Related #13

Bitnami Google Cloud - adapter not working

Hi, i'm using a Bitnami Google Cloud Launcher with Parse Server, then i installed parse-server-gcs-adapter, like this:

// ...
var api = new ParseServer({
    databaseURI: "xxxxxxxxxxxxx",
    cloud: "./node_modules/parse-server/lib/cloud-code/Parse.Cloud.js",
    appId: "xxxxxxxxxx",
    masterKey: "xxxxxxxxx",
    fileKey: "xxxxxxxxxxxxxxx",
    serverURL: 'http://xxxxxxxxxx:80/parse',
    filesAdapter: {
        module: "parse-server-gcs-adapter",
        options: {
            projectId: "xxxxxxxxxxxx",
            keyFilename: "./xxxxxxxxx.json",
            bucket: "xxxxxxxxx"
        }
    }
});
// Serve the Parse API on the /parse URL prefix
app.use('/parse', api);

// ...

The parse-server-gcs-adapter is at node_modules folder.
I dropped the .json file key at the same folder. This server is HTTP, not using SSL actually, i have an other that is SSL and the gcs adapter works perfectly, but no in this server.
I tried as intance too, but nothing happens.

parse-server-gcs-adapter content type is blank

I configure google cloud storage in my parse-server. When i try to upload a new file i see this file in GCS bucket but the file content-type is blank.
after debugging it a bit i found that the file content-type is sent by the parse-server-gcs-adapter to google cloud storage nodejs module but the gcloud storage module ignore it.

  options = options || {};

  var self = this;

  var gzip = options.gzip;

  var metadata = options.metadata || {};
  if (gzip) {
    metadata.contentEncoding = 'gzip';
  }

After investigate it a bit i found parse-server-gcs-adapter should wrap the contentType attribute inside a metadata object
so instead of doing:

let params = { contentType: contentType || 'application/octet-stream' };

you should do:

let params = { metadata: { contentType: contentType || 'application/octet-stream' } };

the metadata can contain more info about the file like: content-encoding but the content-type is very important because the GCS bucket download the file instead of presenting it.

Can you please fix it?

Thanks,
Ran.

Upgrade GCS library

New Issue Checklist

• I am not disclosing a vulnerability.
• I am not just asking a question.
• I have searched through existing issues.

Issue Description

Current version of @google-cloud/storage is 1.7.0 in the lock file, which is an outdated version from 4 years ago, which is a bit concerning. Latest version is 6.2.3.
We found out about this the hard way, as adding gcs-adapter downgraded google-auth-library and broke typescript compilation.

Expected Outcome

Can we bring the library up to date?

On the fly Resizing

Hello I would like to know if there is a known solution for on the fly image resizing that integrated well with parse server gcs adapter workflow. Thanks !
PS : I know that is not exactly an issue with the module but seeking for help !!