Giter Site home page Giter Site logo

googlearchive / appengine-nodejs Goto Github PK

View Code? Open in Web Editor NEW
82.0 66.0 16.0 272 KB

Utility library for accessing App Engine services using Node.js

Home Page: https://developers.google.com/appengine/docs/managed-vms/

License: Apache License 2.0

JavaScript 100.00%

appengine-nodejs's Introduction

status: inactive

This project is no longer actively developed or maintained.

To use Google Cloud APIs from Google App Engine in Node.js, use gcloud-node.


appengine-nodejs

An experimental set of node.js libraries for interacting with App Engine specific services from within Google App Engine Managed VMs using Custom Runtimes.

To get started, clone the quickstart project and follow the instructions there.

Usage

This library will only work inside App Engine Managed VMs, or with the Managed VM SDK.

Declare this library as a dependency in package.json:

  "dependencies": {
    "appengine": "git://github.com/GoogleCloudPlatform/appengine-nodejs.git",
  }

Add a Dockerfile to the application with the following contents:

  FROM google/nodejs-runtime

Then, before calling functions in this library, http handlers need to make sure to call the appengine.middleware.base function with three arguments: the node.js request, the node.js response and a callback.

Frameworks such as express.js do this automatically if you register that function as a middleware. Here's an example:

  var appengine = require('appengine');
  var express = require('express');

  var app = express();
  app.use(appengine.middleware.base);
  // ... register any handlers...
  app.listen(8080, '0.0.0.0');

APIs

This library currently supports a subset of the App Engine APIs available to the official runtimes such as Java or Python.

An application may use any other Google APIs, such as Google Cloud Storage or Google Calendar, e.g. via the googleapis or gcloud-node libraries:

  "dependencies": {
    "googleapis" : "latest"
    "gcloud" : "latest"
  }

All API functions provided by this library take as their last argument a callback, i.e. a function of the form function(err, arg1, arg2, ...). The first argument is always an optional Error object (or null if there is no error). The subsequent arguments depend on the API function itself and are described below.

Logging

appengine.logOneLine
  appengine.logOneLine(request, message, level, function(err) { ... });

Logs a message (a string) at the specified log level.

Log records are buffered by default. They are flushed when request.end() is called or when the buffer grows too big or after a certain amount of time (currently, after 60 seconds).

Two caveats:

  • log flushes are never asynchronous, so for a flush to happen after 60 seconds your application must regularly call appengine.logOneLine
  • the request.end() method is never called on background requests

The appengine.flushLogs method can be used to flush the buffered logs at any time.

Valid log levels are:

  appengine.LogLevel.DEBUG
  appengine.LogLevel.INFO
  appengine.LogLevel.WARNING
  appengine.LogLevel.ERROR
  appengine.LogLevel.CRITICAL

The callback is optional. Note that while most calls to logOneLine will return immediately, some may result in the buffer being flushed.

The following code snippet logs at the INFO level and ignores any logging errors:

  appengine.logOneLine(request, 'this is a test', appengine.LogLevel.INFO);

For backward compatibility, the same function can be invoked without a log level, with or without a callback:

  appengine.logOneLine(request, message, function(err) { ... });

  appengine.logOneLine(request, message);

In this case, it logs the given message at the DEBUG log level.

appengine.flushLogs
  appengine.flushLogs(request, function(err) { ... });

Flushes any buffered log records.

Memcache

appengine.memcache.get
  appengine.memcache.get(request, key, function(err, value) { ... });

Reads from memcache the value (a string) for a given key (also a string).

appengine.memcache.set
  appengine.memcache.set(request, key, value, function(err) { ... });

Writes to memcache a value (a string) for a given key (also a string).

Task Queues

appengine.taskqueue.add
  appengine.taskqueue.add(request, taskOptions, function(err) { ... });

Add a task to a queue. taskOptions must be an object with the following properties:

  • url: a string, url to dispatch to, required
  • queueName: a string, queue name, optional
  • taskName: a string, task name, optional
  • etaUsec: a string, ETA in microseconds, e.g. "1000", optional
  • method: http method to use, one of "get", "post", "put", or "delete", optional, defaults to "post"
  • body: a string, the body of the request, optional, for put and post methods only
  • headers: optional, an object containing http headers as property/value pairs

Modules

appengine.modules.getHostname
  appengine.modules.getHostname(request, module, version, instance, function(err, hostname) { ... });

Returns the hostname (a string) to use to talk to the given module/version/instance (all strings, all required).

System

appengine.system.getBackgroundRequest
  appengine.system.getBackgroundRequest(request, function(err, backgroundRequest) { ... });

Returns a new request-like object that can be used to make App Engine API calls in the background, i.e. outside a request handler.

A common pattern is to call this API during a start request and store the returned request object for later use.

Application code should not assume any specific properties to be defined on the returned background request object.

Auth

appengine.auth.getServiceAccountToken
  appengine.auth.getServiceAccountToken(request, function(err, token) { ... });

Returns an OAuth token for the service account for the VM as provided by the metadata server. token is an object with three properties:

  • accessToken: a string, the access token itself
  • tokenType: a string, the type of token returned, e.g. "Bearer"
  • expiresIn: a number, the time to expiration in seconds

Metadata

appengine.metadata.getAttribute
  appengine.metadata.getAttribute(request, name, function(err, value) { ... });

Returns the value (a string) of the metadata attribute with the specified name (a string). For example, to retrieve the foo instance attribute, the name to use is "instance/attributes/foo".

Middleware

appengine.middleware.base
  appengine.middleware.base(request, response, function() { ... });

A middleware function that extracts App Engine-specific information from request headers and environment properties and sets them into the request under request.appengine. This function must be called by all http handlers that intend to make any App Engine API calls.

appengine-nodejs's People

Contributors

ajessup avatar beriberikix avatar elibixby avatar rrch avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

appengine-nodejs's Issues

Logging

Hello,

In the docs it states that I can use appengine.logOneLine(request, message, function(err) { ... }); to log a line in app engine. I found some problems with it.

When I implement this I need to wait for the callback of the logOneLine function. Example:

router.get('/status', function(req, res) {
  appengine.logOneLine(req, 'test log message', function(err) {
    console.log('logged a message in appengine', err);
    res.send('OK');
  });
});

This will delay the response making a production use case slower. But I hope the logging in a production environment is really fast and thus it not being a really big problem.
But doing this will fail when running the app locally. The logOneLine function will timeout with an ETIMEDOUT after a long time and the request will fail because the logging failed.

Also I do not see an option to specify a log level even though the dashboard where I view the logs supports this. Is this undocumented or not possible?

Thanks for the help and let me know if i need to be more specific.

Unable to see logs in console

Apologies in advance for the newb question:

Given the following route, I was expecting to see a log in my local terminal window

app.get('/run', function(req, res) {
  appengine.logOneLine(req, 'Just a test message.', appengine.LogLevel.INFO);
  res.send('Running...');
});

I've tried running the app with both gcloud preview app run . and just npm start but neither produces a log in the local terminal. Am I going about this all wrong?

Socket module

Hi!

I'm writting an application and this is based on socket(using the net/tls package).
The AppEngine only work on HTTP protocol, but, some applications run using sockets and the modules have libraries to help the developers when they need work with this application types.

My question is why the nodejs livrary don't have this module to work with sockets too?

Error when logging (locally)

Hello,

When I use the logOneLine I get this error:

TypeError: Cannot read property 'constructor' of undefined
   at goog.proto2.Message.checkFieldType_ (C:\Projects\hc-client-eastwood\node_modules\appengine\lib\closure\goog\proto2\message.js:736:30)
   at goog.proto2.Message.set$Value (C:\Projects\hc-client-eastwood\node_modules\appengine\lib\closure\goog\proto2\message.js:686:10)
   at apphosting.ext.remote_api.Request.setRequestId (C:\Projects\hc-client-eastwood\node_modules\appengine\lib\apphosting\ext\remote_api\remote_api.pb.js:224:8)
   at AppEngine.makeRemoteApiRequestBuffer_ (C:\Projects\hc-client-eastwood\node_modules\appengine\lib\index.js:247:14)
   at AppEngine.callApi_ (C:\Projects\hc-client-eastwood\node_modules\appengine\lib\index.js:136:19)
   at AppEngine.logOneLine (C:\Projects\hc-client-eastwood\node_modules\appengine\lib\index.js:325:8)
   at C:\Projects\hc-client-eastwood\src\app.js:60:13
   at Layer.handle_error (C:\Projects\hc-client-eastwood\node_modules\express\lib\router\layer.js:58:5)
   at trim_prefix (C:\Projects\hc-client-eastwood\node_modules\express\lib\router\index.js:268:13)
   at C:\Projects\hc-client-eastwood\node_modules\express\lib\router\index.js:237:9

This is when running my app locally using npm start. Not in the managed VM docker container or deployed to app engine.
I can understand that this can not log successfully. But this error looks like a "mistake". And it is thrown synchronously instead of through the callback's error argument. As you can see this is bad ;-)

My code:

appengine.logOneLine(req, 'Just a test message.', appengine.LogLevel.INFO);

Either with or without a callback as the last argument.

Logging throws error when request comes from outside appengine

Hello,

I just noticed that if I access my managed VM from outside appengine it throws this error when logging to appengine:

172.17.42.1 - GET /status HTTP/1.1 500 22 - 0.715 ms
TypeError: Cannot read property 'constructor' of undefined
    at goog.proto2.Message.checkFieldType_ (/app/node_modules/appengine/lib/closure/goog/proto2/message.js:736:30)
    at goog.proto2.Message.set$Value (/app/node_modules/appengine/lib/closure/goog/proto2/message.js:686:10)
    at apphosting.ext.remote_api.Request.setRequestId (/app/node_modules/appengine/lib/apphosting/ext/remote_api/remote_api.pb.js
:224:8)
    at AppEngine.makeRemoteApiRequestBuffer_ (/app/node_modules/appengine/lib/index.js:222:14)
    at AppEngine.callApi_ (/app/node_modules/appengine/lib/index.js:129:19)
    at AppEngine.logOneLine (/app/node_modules/appengine/lib/index.js:274:8)
    at ip (/app/src/routes/index.js:25:13)
    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:76:5)
    at next (/app/node_modules/express/lib/router/route.js:100:13)
    at Route.dispatch (/app/node_modules/express/lib/router/route.js:81:3)
    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:76:5)

It's not handed as an error to the callback but looks to be thrown outside it.
Furthermore I can understand the logging can not succeed because i'm missing some appengine load information in the request but this could fail more gentle.

Edit: to clarify; i'm using the external IP of the VM or doing a curl command when i'm SSHed into it.

Possible performance gain by using maxSockets

Hello,

We are looking to handle many Request Per Second on our app's but until now the logOneLine() looked to be a bottleneck. Say that for every request we log one line to app engine. That number could run into the hundreds of simultaneous app engine logging API requests at the same time, all delaying the original response to the client (because logging needs to be done on an active request).

We ran into the same problem for our app because we are doing an external call for every request. What I did not know is that on Node.js 0.10 and lower the default is a max of 5 simultaneous sockets. Any more and it becomes a big queue and all the request get an extremely high latency.
FYI: we are using https://github.com/request/request

Then it dawned on me that the logOneLine() probably has the same problem/bottleneck.
Changing the global value http.globalAgent.maxSockets in my app does not seem like a elegant solution; every module should configure it for its own scope.

Does it make sense for this module to configure a higher maxSockets? I expect much better performance under load that way.

Let me know what you think!

Versioning

Hello,

When will this module have active versioning? Right now I have to force a new download (using npm) to get the new features; because the version number did not change. Also for real world usage we would like to lock down the version when rolling out a release.

Exposing original IP

The original IP of the user is not exposed because of the appengine loadbalancer in between the client and server. Other loadbalancers most of the time add a x-forwarded-for header that things like Express can understand as to expose the original IP.
Looking at the headers appengine adds I see one that looks promising: x-appengine-user-ip.
But this is not exposed in request.appengine as the middleware from this package should do. Looking at the code (https://github.com/GoogleCloudPlatform/appengine-nodejs/blob/master/lib/index.js#L522) i see that this is only done when an user email is know.

For now I can look at the x-appengine-user-ip manually as a workaround.

Are there any plans to maintain this project in long term?

I would like to know whether this project will be supported in long term. It says experimental, so just wondering. I'd like to use node + socket io on managed VM and use other services(like memcache etc) through node as well. If this is experimental and won't be maintained moving forward I wouldn't probably use this library.

Error when not waiting for logOneLine callback

Hello,

A few days ago you showed me that you implemented a callbackless logOneLine. This works, but it's dangerous. It only works if your request is still open when appengine-nodejs is logging to appengine in the background.
In my case i finish the request after calling logOneLine and not waiting for it's callback. This gives an error that there is no active request. So this means you need to delay your response to the client and wait for the logOneLine to finish.

This is safe in the sense that you won't have a race condition, but also dangerous because if there is a problem with logging (slow) the whole request get's delayed.

I'm curious what your thoughts are on this.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.