parse-community / parse-server-gcs-adapter Goto Github PK
View Code? Open in Web Editor NEWparse-server adapter for Google Cloud Storage
Home Page: https://parseplatform.org
parse-server adapter for Google Cloud Storage
Home Page: https://parseplatform.org
I am trying to save a png image in a ParseFile from the android SDK. I do what I have always successfully done with parse.com. Just for reference:
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, stream);
ParseFile imageParseFile = new ParseFile(filename, stream.toByteArray(), "image/png");
imageParseFile.saveInBackground();
The save operation fails after a while, with a ParseRequestException
(code 100) caused by a SocketTimeoutException
. At the same time, here’s what happens to parse-server:
Any idea? I am 100% sure that save() is called just once. Somehow that is looped and repeated. Should I post this @ Parse-SDK-Android ?
Hello I would like to know if there is a known solution for on the fly image resizing that integrated well with parse server gcs adapter workflow. Thanks !
PS : I know that is not exactly an issue with the module but seeking for help !!
I’m new to this and I might be missing something, but what about narrowing down google dependencies?
We have a dependency on google-cloud which is huge, see /packages.
Looks like we only need @google-cloud/storage.
I configure google cloud storage in my parse-server. When i try to upload a new file i see this file in GCS bucket but the file content-type is blank.
after debugging it a bit i found that the file content-type is sent by the parse-server-gcs-adapter to google cloud storage nodejs module but the gcloud storage module ignore it.
options = options || {};
var self = this;
var gzip = options.gzip;
var metadata = options.metadata || {};
if (gzip) {
metadata.contentEncoding = 'gzip';
}
After investigate it a bit i found parse-server-gcs-adapter should wrap the contentType attribute inside a metadata object
so instead of doing:
let params = { contentType: contentType || 'application/octet-stream' };
you should do:
let params = { metadata: { contentType: contentType || 'application/octet-stream' } };
the metadata can contain more info about the file like: content-encoding but the content-type is very important because the GCS bucket download the file instead of presenting it.
Can you please fix it?
Thanks,
Ran.
gcloud module has now been moved to google-cloud
Current version of @google-cloud/storage is 1.7.0 in the lock file, which is an outdated version from 4 years ago, which is a bit concerning. Latest version is 6.2.3.
We found out about this the hard way, as adding gcs-adapter downgraded google-auth-library and broke typescript compilation.
Can we bring the library up to date?
@flovilmart
Confirming this issue using the GCS Adapter. My files seek/ buffer fine on googles side but not as PFfiles. Is there any recommended fix for GCS Adapter ??
Direct access works fine. But we want to continue using the indirect access dues to authentication restrictions.
I have written a TS version for myself, feel free to use the same code although it would need changes to be compiled
import { Storage, StorageOptions } from '@google-cloud/storage';
interface GCSAdapterOptions {
projectId?: string;
keyFilename?: string;
bucket: string;
bucketPrefix?: string;
directAccess?: boolean;
baseUrl?: string;
}
type GCSAdapterConstructorArgs = [GCSAdapterOptions];
function optionsFromArguments(args: GCSAdapterConstructorArgs): GCSAdapterOptions {
const defaultOptions: Partial<GCSAdapterOptions> = {
projectId: process.env['GCP_PROJECT_ID'],
keyFilename: process.env['GCP_KEYFILE_PATH'],
bucket: process.env['GCS_BUCKET'],
bucketPrefix: process.env['GCS_BUCKET_PREFIX'] || '',
directAccess: process.env['GCS_DIRECT_ACCESS'] === 'true' ? true : false
};
return {
...defaultOptions,
...args[0]
};
}
class GCSAdapter {
private _bucket: string;
private _bucketPrefix: string;
private _directAccess: boolean;
private _gcsClient: Storage;
private _baseUrl: string;
constructor(...args: GCSAdapterConstructorArgs) {
let options = optionsFromArguments(args);
this._bucket = options.bucket;
this._bucketPrefix = options.bucketPrefix || '';
this._directAccess = options.directAccess || false;
this._baseUrl = options.baseUrl || 'https://storage.googleapis.com/';
this._gcsClient = new Storage(options as StorageOptions);
}
async createFile(filename: string, data: Buffer | string, contentType?: string): Promise<void> {
const params = {
metadata: {
contentType: contentType || 'application/octet-stream',
},
};
const file = this._gcsClient.bucket(this._bucket).file(this._bucketPrefix + filename);
await new Promise((resolve, reject) => {
const uploadStream = file.createWriteStream(params);
uploadStream.on('error', reject);
uploadStream.on('finish', resolve);
uploadStream.end(data);
});
if (this._directAccess) {
await file.makePublic().catch(err => {
throw new Error(`Error making file public: ${err}`);
});
}
}
async deleteFile(filename: string): Promise<void> {
try {
const file = this._gcsClient.bucket(this._bucket).file(this._bucketPrefix + filename);
await file.delete();
} catch (err) {
throw new Error(`Error deleting file ${filename}: ${err}`);
}
}
async getFileData(filename: string): Promise<Buffer> {
try {
const file = this._gcsClient.bucket(this._bucket).file(this._bucketPrefix + filename);
const [exists] = await file.exists();
if (!exists) {
throw new Error(`File ${filename} does not exist.`);
}
const [data] = await file.download();
return data;
} catch (err) {
throw new Error(`Error downloading file ${filename}: ${err}`);
}
}
getFileLocation(config: { mount: string; applicationId: string; }, filename: string): string {
if (this._directAccess) {
return `${this._baseUrl}${this._bucket}/${this._bucketPrefix + filename}`;
}
return `${config.mount}/files/${config.applicationId}/${encodeURIComponent(filename)}`;
}
}
export default GCSAdapter;
Update documentation the need of projectId & keyFilename
Related #13
Parse push not working on Android 6.0 after moving Parse server to my own server. It is working fine for all other versions of Android.
Hi, i'm using a Bitnami Google Cloud Launcher with Parse Server, then i installed parse-server-gcs-adapter
, like this:
// ...
var api = new ParseServer({
databaseURI: "xxxxxxxxxxxxx",
cloud: "./node_modules/parse-server/lib/cloud-code/Parse.Cloud.js",
appId: "xxxxxxxxxx",
masterKey: "xxxxxxxxx",
fileKey: "xxxxxxxxxxxxxxx",
serverURL: 'http://xxxxxxxxxx:80/parse',
filesAdapter: {
module: "parse-server-gcs-adapter",
options: {
projectId: "xxxxxxxxxxxx",
keyFilename: "./xxxxxxxxx.json",
bucket: "xxxxxxxxx"
}
}
});
// Serve the Parse API on the /parse URL prefix
app.use('/parse', api);
// ...
The parse-server-gcs-adapter
is at node_modules
folder.
I dropped the .json
file key at the same folder. This server is HTTP, not using SSL actually, i have an other that is SSL and the gcs adapter works perfectly, but no in this server.
I tried as intance too, but nothing happens.
Hi
there seems to be an issue with file URLs returned by parse server with GCS when direct access = true is set.
I get (net::ERR_CERT_COMMON_NAME_INVALID)
If line 133 is changed to it should work:
return
https://storage.googleapis.com/${this._bucket}/${this._bucketPrefix + filename};
This is also the format the google cloud console returns if you use a public link.
Unable to delete a file stored in Google Cloud Storage using Parse.Cloud.httpRequest.
Parse.Cloud.httpRequest({
method: 'DELETE',
url: photoUrl,
headers: {
"X-Parse-Application-Id": process.env.APP_ID,
"X-Parse-Master-Key" : process.env.MASTER_KEY,
"X-Parse-Session-Token": requester.getSessionToken(),
}
})
.then((httpResponse) => {
console.log(httpResponse.text);
})
.catch((error) => {
console.error('Request failed with error:' + error.text);
})
File is deleted
Google Cloud Storage returns Access denied error.
Server
Google Cloud Platform
Parse-server console output:
verbose: REQUEST for [DELETE] /parse/classes/Photo/HDLbppIQpB: {} method=DELETE, url=/parse/classes/Photo/HDLbppIQpB, host=192.168.0.190:1337, connection=keep-alive, content-length=195, origin=http://localhost:3003, user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36, content-type=text/plain, accept=*/*, referer=http://localhost:3003/, accept-encoding=gzip, deflate, accept-language=en-US,en;q=0.8,
Request failed with error:<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous users does not have storage.objects.delete access to bucket <my bucket>.</Details></Error>
Google Cloud Storage log:
cs_method: DELETE
cs_uri: /<bucket name>/76cf0e7588857ca92cdcc1bfb8540c6b_Room.jpg
sc_status: 403
cs_bytes: 0
sc_bytes: 208
cs_host: storage.googleapis.com
cs_user_agent: gzip(gfe)
I've also successfully tested DELETE
using Google OAuth 2.0 Playground
using my personal gmail account set with same permissions as the service account used by Parse Server. So, I'm reasonably confident that my permissions are set correctly.
There are multiple security issues being reported by snyk https://snyk.io/test/npm/@parse/gcs-files-adapter/1.1.2
Can you please update your dependencies to latest versions or versions without security issues?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.