googleapis/google-cloud-datastore

Low-level, Protobuf-based Java and Python client libraries for Cloud Datastore. Check out google-cloud-java and google-cloud-python first!

Google Cloud Datastore

Note: This repository contains low-level Java and Python client libraries for Google Cloud Datastore. For more idiomatic and usable client libraries in these languages, please visit the Google Cloud Datastore Client for Java and Google Cloud Datastore Client for Python repositories. You can also find the full list of supported client libraries in a variety of languages on the Client Libraries page of Cloud Datastore.

Cloud Datastore is a highly-scalable NoSQL database for your applications. Cloud Datastore automatically handles sharding and replication, providing you with a highly available and durable database that scales automatically to handle your applications' load. Cloud Datastore provides a myriad of capabilities such as ACID transactions, SQL-like queries, indexes and much more. For more information, see the Cloud Datastore documentation.

This repository contains clients that are deliberately low-level and map directly to the underlying Datastore RPC model. They're designed to provide more flexibility to developers and higher level library implementers.

• Python Proto Client Library and Samples
• RPC API Reference

Samples

Proto

• Python
• Java

Client Libraries

You can learn more about client libraries for Cloud Datastore here.

• Java

  The datastore-v1-proto-client client code has been moved to java-datastore

• Python:

pip install googledatastore

Documentation

For more information, see the Cloud Datastore documentation.

Filing Issues

1. For production issues and support options, see Cloud Datastore support.
2. For bugs or feature requests, please first look at existing issues.
3. When applicable, create a new report. Note that this repo exclusively covers the low-level, Protobuf-based clients. If you're using com.google.cloud.google-cloud-datastore (Java) or google-cloud-datastore (Python), please file your issue in the appropriate repo, [google-cloud-java][26] or [google-cloud-python][27]. If you file an issue with either of those client libraries here, we will (gently) redirect you to the right repo and close the issue in this one.
4. For bugs, detail the steps to reproduce the problem and the affected version number.
5. For feature requests, articulate the use case you are trying solve and describe any current workaround(s).

Contributing changes

• See CONTRIB.md

Licensing

• See LICENSE

google-cloud-datastore's Issues

Migrate Node samples to gcloud

We've just released a pre-stable Datastore client with user friendly interfaces as a part of gcloud npm module. Additionally, [googleapis](the current client Node samples are using) is going to make a major release that will deprecate the current discovery model. We need to migrate to existing samples to gcloud.

current version of Java demos

The Java (maven) demos do not compile because the classes in com.google.api.services.datastore.DatastoreV1.* are not available.

Does adding extra levels in my Ancestor path cost more index writes/reads/size for the children I add?

Suppose this structure:

[Person:Grandpa, Person:Dad, Person:Me]

And my dad has multiple children so I also have:

[Person:Grandpa, Person:Dad, Person:Sis]
[Person:Grandpa, Person:Dad, Person:Bro]

Then does adding a GreatGrandPa to my datamodel cost me more writes than in the previous situation? (If we neglect the cost of inserting a GreatGrandPa)

LocalDevelopmentDatastore cannot start gcd - "Server did not start normally"

I'm trying to use LocalDevelopmentDatastore according to its documentation. However, it seems to be using an unsupported option --allow_remote_shutdown of gcd.sh.

How to reproduce:

• Download gcd tool from https://cloud.google.com/datastore/docs/downloads , I downloaded version 1.2.1.0.
• Create the following example:

import com.google.api.services.datastore.client.DatastoreOptions;
import com.google.api.services.datastore.client.LocalDevelopmentDatastore;
import com.google.api.services.datastore.client.LocalDevelopmentDatastoreFactory;

class Example {
  public static void main(String[] args) throws Exception {
    LocalDevelopmentDatastore datastore;
    DatastoreOptions opts = new DatastoreOptions.Builder()
        .host("http://localhost:8080")
        .dataset("myapp")
        .build();
    datastore = LocalDevelopmentDatastoreFactory.get().create(opts);
    datastore.start("......../path/to/google-cloud-datastore-sdk/gcd", "myapp");
  }
}

• Download dependencies
• Compile:

javac -cp google-api-services-datastore-protobuf-v1beta2-rev1-2.1.0.jar Example.java 

• Run:

java -cp google-api-services-datastore-protobuf-v1beta2-rev1-2.1.0.jar:google-http-client-1.19.0.jar:protobuf-java-2.5.0.jar:. Example

I get an exception:

Nov 24, 2014 10:54:58 AM com.google.api.services.datastore.client.DatastoreFactory makeClient
WARNING: Not using any credentials
Unexpected option(s): myapp --allow_remote_shutdown.
Exiting.
Exception in thread "main" com.google.api.services.datastore.client.LocalDevelopmentDatastoreException: Server did not start normally
    at com.google.api.services.datastore.client.LocalDevelopmentDatastore.startDatastoreInternal(LocalDevelopmentDatastore.java:164)
    at com.google.api.services.datastore.client.LocalDevelopmentDatastore.start(LocalDevelopmentDatastore.java:132)
    at Example.main(Example.java:13)

How do I do a multilevel insert?

Level1 -> Level2 -> Level3

req = datastore.BlindWriteRequest()

level1Ent = req.mutation.upsert.add()
path_element = level1Ent.key.path_element.add()
path_element.kind = 'Level1'
path_element.name = 'Level1'

level2Ent = req.mutation.insert_auto_id.add()
level2Ent.key.path_element.extend(level1Ent.key.path_element)
path_element = level2Ent.key.path_element.add()
path_element.kind = 'Level2'

level3Ent = req.mutation.insert_auto_id.add()
#??? level3Ent.key.path_element.extend(level1Ent.key.path_element)
level3Ent.key.path_element.extend(level2Ent.key.path_element)
path_element = level3Ent.key.path_element.add()
path_element.kind = 'Level3'

set_property(level3Ent.property.add(), "Foo", "Bar")

print datastore.blind_write(req)

The parent pom in the Maven Central Repo seems to be missing

Trying to use the Java lib with a dependency tool like SBT or Maven fails. There seems to be a parent pom missing or something like that.
See also:
http://stackoverflow.com/questions/18385456/how-to-use-google-api-services-datastore-protobuf-with-sbt-or-maven?noredirect=1

High latency

b/8334662

I'm consistently seeing hight latency numbers when using the GCD - about 1.5 seconds on average for queries (10 items or less) and single item writes ( < 1kb).

My ping time to the API endpoint is about 50ms, so discounting a 100ms round trip, that still leaves more than a second to GCD latency. This simply won't work for a server environment, certainly not one that scales. This is very surprising because I was expecting latencies much closer to GAE: https://code.google.com/status/appengine/detail/hr-datastore/2013/06/05#ae-trust-detail-hr-datastore-query-latency

Can we get a dashboard like the GAE HR Datastore status board? Possibly measuring latencies from Google Compute Engine instances and a few different AWS Regions?

Document how to use GCD development server in combination with local app engine environment

b/11970066

Can I somehow point App Engine Launcher to the GCD development server?
Or if that is not possible. Is there any place where I can put the existing data files so that App Engine can use the existing enities?

Better error message for incomplete key

b/11253114

The current exception is:

com.google.api.services.datastore.client.DatastoreException: Key path is complete

It should say something like:

com.google.api.services.datastore.client.DatastoreException: Key path should not be complete when auto allocating ids.

Python client should check environment variables before using GCE credentials

b/11474814

gcd.sh vacumindexes is broken with gcd-v1beta2-rev1-2.1.0.zip

This is due to an underlying bug in the App Engine SDK 1.8.5

Updating the internal SDK copy workarounds this issue:

wget http://commondatastorage.googleapis.com/gcd/tools/gcd-v1beta2-rev1-2.1.0.zip
unzip gcd-v1beta2-rev1-2.1.0.zip
rm -fR gcd-v1beta2-rev1-2.0.0/.appengine/
wget http://googleappengine.googlecode.com/files/appengine-java-sdk-1.8.9.zip
unzip appengine-java-sdk-1.8.9.zip
mv appengine-java-sdk-1.8.9 gcd-v1beta2-rev1-2.0.0/.appengine

b/12911096

Broken link in documentation

Following page:
https://developers.google.com/datastore/docs/concepts/entities
Contains this link:
https://developers.google.com/appengine/docs/python/datastore/config/appconfig#Datastore_app_yaml_Auto_ID_policy

async support for Java and Python client library

b/11970323

using https://github.com/AsyncHttpClient/async-http-client or another popular async http client for Java.
and facilitating integration with python event loop like twisted or tulip.

service account authorization doesn't work with domain-restricted App Engine application

b/10865628

My app with projection ID "appscale-staging" gives errors when trying to access data. I did the steps mentioned in
https://developers.google.com/datastore/docs/activate#google_cloud_datastore_from_other_platforms

And when I run a sample script I get the following error (with httplib.debuglevel set to 3):

root@appscale-image0:~# python test.py appscale-staging
connect: (accounts.google.com, 443)
send: 'POST /o/oauth2/token HTTP/1.1\r\nHost: accounts.google.com\r\nContent-Length: 643\r\ncontent-type: application/x-www-form-urlencoded\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.8 (gzip)\r\n\r\n'
send: 'grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIzNjQxODc0OTA2NDctaHZwMDNiNjE0bGttMmhqb2JscXZuZGI5bDQzNmE2dHNAZGV2ZWxvcGVyLmdzZXJ2aWNlYWNjb3VudC5jb20iLCJzY29wZSI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2F1dGgvZGF0YXN0b3JlIGh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2F1dGgvdXNlcmluZm8uZW1haWwiLCJhdWQiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20vby9vYXV0aDIvdG9rZW4iLCJleHAiOjEzNzkxMTEwODksImlhdCI6MTM3OTEwNzQ4OX0.P_ePlrb6rDMZiEbgG5k3pU6lxx-5VryaYMK500Ma2-aZseYg-t4Uv6y6MeIQEbv45COv18-K-WOYuTZ0tScy_NXdW7dhfMB75GYeJ_kv8FmScg6bou_3Z6nEayvwy7aUkbig5YCqYkeqtHjB9WCOrNaskfoNAg27-C1TR2wcRFM'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Fri, 01 Jan 1990 00:00:00 GMT
header: Date: Fri, 13 Sep 2013 21:24:47 GMT
header: Content-Type: application/json
header: Content-Encoding: gzip
header: Transfer-Encoding: chunked
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Alternate-Protocol: 443:quic
connect: (www.googleapis.com, 443)
send: 'POST /datastore/v1beta1/datasets/appscale-staging/beginTransaction HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-length: 0\r\ncontent-type: application/x-protobuf\r\naccept-encoding: gzip, deflate\r\nauthorization: Bearer ya29.AHES6ZSBVZToloupANihvFflfnNvj1wGExzr8CX-ez7JU9I\r\nuser-agent: Python-httplib2/0.8 (gzip)\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Content-Type: text/html; charset=UTF-8
header: Content-Encoding: gzip
header: Date: Fri, 13 Sep 2013 21:24:48 GMT
header: Expires: Fri, 13 Sep 2013 21:24:48 GMT
header: Cache-Control: private, max-age=0
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Content-Length: 33
header: Server: GSE
header: Alternate-Protocol: 443:quic
ERROR:root:Error while doing datastore operation
ERROR:root:RPCError: beginTransaction Unauthorized.
ERROR:root:HTTPError: 403 Forbidden

In the App Engine app I can go to the "ah-builtin-datastoreservice" version and see I'm getting 200's for BeginTransaction, so it looks like its doing something. Please advise on how to correct this. I have no problem when doing it with a newly created project.

run_query fails if dataset in set_options is a unicode string

Not sure whether this is is an issue that should be fixed but I think it's worth knowing its' existance.

failing code:

import googledatastore as datastore
from googledatastore import set_options
set_options(dataset=u'myProjectName') # note: unicode
req = datastore.RunQueryRequest()
req.query.projection.add().property.name = '__key__'
req.query.kind.add().name = 'Person'
propertyFilter = req.query.filter.property_filter

propertyFilter.operator = datastore.PropertyFilter.EQUAL
propertyFilter.property.name = 'name'
propertyFilter.value.string_value = 'bla'

stacktrace:

  File "/usr/local/lib/python2.7/dist-packages/googledatastore/__init__.py", line 80, in run_query
    return get_default_connection().run_query(request)
  File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 103, in run_query
    datastore_v1_pb2.RunQueryResponse)
  File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 193, in _call_method
    self._url + method, method='POST', body=payload, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 490, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1593, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1335, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1258, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 962, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 996, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 816, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8a in position 41: ordinal not in range(128)

msg is unicode
and message_body is str

version:

import pkg_resources
print pkg_resources.get_distribution('googledatastore').version
>>> 'v1beta2-rev1-2.1.0'

This can easily be fixed by switching to a str in stead of unicode. I noticed because my config file just happened to be unicode.

Simple query filter not working

import googledatastore as datastore

"""
INSERT
"""
from core.stateMessageHub import createPropertyOnEntity
from googledatastore.helper import set_property
req = datastore.BlindWriteRequest()
host = req.mutation.upsert.add()
path_element = host.key.path_element.add()
path_element.kind = 'Host'
path_element.name = "myServer"
set_property(host.property.add(), "ip", "1.2.3.4")
resp = datastore.blind_write(req)
print resp


"""
SELECT
"""
req = datastore.RunQueryRequest()
query = req.query

query.kind.add().name = 'Host'
queryFilter = query.filter.property_filter
queryFilter.property.name = 'ip'
queryFilter.operator = datastore.PropertyFilter.EQUAL
queryFilter.value.string_value = "1.2.3.4"
resp = datastore.run_query(query)
import googledatastore as datastore

Example directly taken from: https://developers.google.com/datastore/docs/concepts/queries#Datastore_query_interface

# Get the people who are too short.
req = datastore.RunQueryRequest()
query = req.query
query.kind.add().name = 'Person'

too_short_filter = query.filter.property_filter
too_short_filter.property.name = 'height'
too_short_filter.operator = datastore.PropertyFilter.LESS_THAN
too_short_filter.value.integer_value = min_height

Error:

    resp = datastore.run_query(query)
  File "C:\Python27\lib\site-packages\googledatastore\__init__.py", line 76, in run_query
    return get_default_connection().run_query(request)
  File "C:\Python27\lib\site-packages\googledatastore\connection.py", line 118, in run_query
    datastore_v1_pb2.RunQueryResponse)
  File "C:\Python27\lib\site-packages\googledatastore\connection.py", line 210, in _call_method
    raise RPCError(method, response, content)
googledatastore.connection.RPCError: runQuery RPC server failure with HTTP(500) Could not initialize class com.google.apphosting.client.datastoreservice.api.DatastoreServiceRpcProto: <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 500 Could not initialize class com.google.apphosting.client.datastoreservice.api.DatastoreServiceRpcProto</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /datastore/v1beta1/datasets/data-segment-326/runQuery. Reason:
<pre>    Could not initialize class com.google.apphosting.client.datastoreservice.api.DatastoreServiceRpcProto</pre></p><h3>Caused by:</h3><pre>java.lang.NoClassDefFoundError: Could not initialize class com.google.apphosting.client.datastoreservice.api.DatastoreServiceRpcProto   
    at com.google.apphosting.client.datastoreservice.api.DatastoreServiceRpcProto$RunQueryRequest.internalGetFieldAccessorTable(DatastoreServiceRpcProto.java:20151)    
    at com.google.appengine.repackaged.com.google.protobuf.GeneratedMessage.getDescriptorForType(GeneratedMessage.java:71)  
    at com.google.appengine.repackaged.com.google.protobuf.MessageReflection.findMissingFields(MessageReflection.java:156)  
    at com.google.appengine.repackaged.com.google.protobuf.MessageReflection.findMissingFields(MessageReflection.java:193)  
    at com.google.appengine.repackaged.com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:311)   
    at com.google.appengine.repackaged.com.google.protobuf.AbstractMessage.newUninitializedMessageException(AbstractMessage.java:177)   
    at com.google.appengine.repackaged.com.google.protobuf.AbstractParser.newUninitializedMessageException(AbstractParser.java:29)  
    at com.google.appengine.repackaged.com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:47)   
    at com.google.appengine.repackaged.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)    
    at com.google.appengine.repackaged.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:199)    
    at com.google.appengine.repackaged.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:21) 
    at com.google.apphosting.client.datastoreservice.app.ApiServlet.readRequestProto(ApiServlet.java:277)   
    at com.google.apphosting.client.datastoreservice.app.ApiServlet.call(ApiServlet.java:216)   
    at com.google.apphosting.client.datastoreservice.app.ApiServlet.doPost(ApiServlet.java:145) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) 
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)   
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)  
    at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)    
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)  
    at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123)  
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)  
    at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)   
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)  
    at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63) 
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)  
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)  
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)  
    at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)  
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)  
    at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:368) 
    at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:351)   
    at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)    
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)  
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) 
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)  
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) 
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)    
    at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:97)   
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
    at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:485)  
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 
    at org.mortbay.jetty.Server.handle(Server.java:326) 
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)  
    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938) 
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)  
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) 
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) 
    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) 
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

</pre>
<hr /><i><small>Powered by Jetty://</small></i><br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                

</body>
</html>

Support the IN, OR, != operators in queries

Is Composite Filter OR Supported

From the Java documents it appears OR is supported for composite filters. However, my experience with Node.js is it is not supported. Trying to use it leads to a misleading error message: "Message missing required fields: query.filter.composite_filter.operator" However, it is not documented as such.

Deep in the Go language documentation I found this: ""operator": {
"type": "string",
"description": "The operator for combining multiple filters. Only "and" is currently supported."
}
.

datastore.beginTransaction returns a string under nodejs

Using googleapis.discover('datastore', 'v1beta2'), and then calling client.datastore.datasets.beginTransaction returns a string containing binary data in the result callback parameter, not an object with a base64 .transaction property, as referenced in the Adams example.

Unsure how to proceed.

Attached code:

var googleapis = require('googleapis'),
    fs = require('fs');

var C = JSON.parse(fs.readFileSync("access.json").toString());

var jwt = new googleapis.auth.JWT(
          C.googleApiEmailAddress,
          C.googleApiPrivateKey,
          C.googleApiScopes
        );

var datastore, credentials;

function connectDataStore() {
     googleapis.discover('datastore', 'v1beta2')
     .withAuthClient(jwt)
         .execute(function(err, client) {
           if (err) {
             console.error(err);
             return;
           }
           datastore = client.datastore.datasets;
           datastore.beginTransaction({
             datasetId: C.dataSet
           }).execute(function(err, result) {
               transaction = result.transaction; // undefined, since result is a string.
               console.log(result); // prints unintelligible binary string, typeof result == "string"
                                    // Did I do it wrong, or has some unchecked error cascaded into
                                    // my code? Either way I expect more explosion and something
                                    // meaningful in err at this point rather than gibberish in result, 
                                    // unless I've missed something and there's another way to interpret
                                    // the result of beginTransaction that I've not understood, but accidentally implied.
             if(!err) {
                 var entity = {
                    key: { path: [{kind: 'User', name: 'someuser'}]},
                    properties: {
                        displayName: {stringValue: "myUserDisplayName"},
                        joinedDate: {stringValue: new Date().toString() }
                    }
                 }
                 mutation = { insert: [entity] };
                 datastore.commit({ transaction: transaction, mutation: mutation}).execute(function(err, result) {
                     if(err)
                         console.error(err); // { errors: [ { domain: 'global', reason: 'backendError', message: 'Backend Error' } ], code: 503, message: 'Backend Error' }, because I just sent the transaction as undefined... what happened?
                     else {
                         console.log("win");
                         console.log(result);
                     }
                 });
             } else {
                 console.error(err);
             }
           });
         });
};

jwt.authorize(function(err, result) {
    if(!err) {
        credentials = result;
        connectDataStore();
    } else {
        console.error(err);
    }
});

Contents of access.json:

{
    "googleApiEmailAddress": "[email protected]",
    "googleApiPrivateKey": "myPEM.pem",
    "googleApiScopes": ["https://www.googleapis.com/auth/userinfo.email", "https://www.googleapis.com/auth/datastore"],
    "dataSet": "myDataSet"
}

Data in result after beginTransaction (typeof == "string")

  "\nT\u0011#Ô‚7R�\u0017�\u001aI\u0000\u0005\u0019�1mU\u001fA�2f�M�V鵖�t�9����\u0016�\u0012����\u0004\u0003\u0019ß?eu\u0016\">��P��B0@\r:�S_̒�\u0004�`,E\u0011k����6�"

Command used to convert p12 to pem:

openssl pkcs12 -in myP12.p12 -out myPEM.pem -nodes

"Activate" documentation should calls out that the Datastore API doesn't work with projects using the old APIs Console

While the activate documentation specifically says that you should create project from cloud.google.com/console, it should also calls out that legacy or new project created using the "old' Google APIs console will not work with the Datastore API.

More context on gcd-discuss.

Return empty `entityResults` array when no results are available in a query

On the following request, not having the entityResults property throws exceptions into any code that attempts use it. It makes it very ugly to keep checking to see if the property exists before trying to iterate over the result set. It would be much better to just send an empty entityResults array.

  HTTP POST (1691.91ms)   https://www.googleapis.com:443/datastore/v1beta1/datasets/active-datastore-test/runQuery
  Request body   {"query":{"kinds":[{"name":"Todo"}],"filter":{"propertyFilter":{"property":{"name":"__key__"},"operator":"hasAncestor","value":{"keyValue":{"path":[{"kind":"TodoList","name":"default"}]}}}}}}
  Response status   Net::HTTPOK (200)
  Response body   {
 "kind": "datastore#runQueryResponse",
 "batch": {
  "entityResultType": "full",
  "endCursor": "CgA=",
  "moreResults": "moreResultsAfterLimit",
  "skippedResults": 0
 }
}

Document error responses

To allow writing better client libraries, it would be helpful if the API docs laid out the HTTP response codes and payload formats for errors returned by each API.

E.g., see googleapis/google-cloud-python#626 (comment), where we made an invalid assumption that the payload would be JSON.

MORE_RESULTS_AFTER_LIMIT Erroneously Returned

When setting a LIMIT in the query, the JSON API always returns 'MORE_RESULTS_AFTER_LIMIT' for the the moreResults property in the response even though there are no more results.

Document JWTclient usage with node.js

b/10860172

Moved issue here from googleapis/google-api-nodejs-client#5 @proppy referrer.

Dataset ID aliases don't behave consistently.

Using the "human-readable" version foo of the prefixed ID s~foo (or e~foo) is allowed in a URI but not in a request payload (and will never be returned in a response since foo gets mapped to s~foo in the response).

This inconsistency can catch user's off-guard and requires custom behavior in to_protobuf()-type methods which essentially always ignore the dataset_id on a PartitionId message and only use the dataset ID in the URI.

/cc @silvolu @tseaver

---

Some examples

1. Sending a query to

https://www.googleapis.com/datastore/v1beta2/datasets/foo/runQuery

which has RunQueryRequest.partition_id.dataset_id == 'foo' in the payload results in

app s~foo cannot access app foo's data

2. Sending a mutation (or a lookup) to one of

https://www.googleapis.com/datastore/v1beta2/datasets/foo/allocateIds
https://www.googleapis.com/datastore/v1beta2/datasets/foo/commit
https://www.googleapis.com/datastore/v1beta2/datasets/foo/lookup

which has Key.partition_id.dataset_id == 'foo' (either directly in a key
or in an Entity) in the payload results in

app s~foo cannot access app foo's data

3. Sending a filtered or ancestor query which does not have
   RunQueryRequest.partition_id.dataset_id set in the payload but does have
   the dataset ID set in either the ancestor key

The query app is 'foo' but ancestor.key app is 'foo'.

or in the __key__ filter

__key__ filter app is foo but query app is s~foo

unknown transaction handle

https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/v1beta2-rev1-2.1.1/ruby/demos/trivial/adams.rb#L62

puts resp.inspect

Gives me:

{\n  \"errors\": [\n   {\n    \"domain\": \"global\",\n    \"reason\": \"INVALID_ARGUMENT\",\n    \"message\": \"unknown transaction handle\"\n   }\n  ],\n  \"code\": 400,\n  \"message\": \"unknown transaction handle\"\n }

ruby 2.1.2p95 (2014-05-08) [x86_64-linux-gnu]
google-api-client (0.8.2)

Is there something im missing?

unauthorized errors when using certificate to access Cloud Datastore

Reported on Stack Overflow:
http://stackoverflow.com/questions/17885347/all-requests-return-403-unauthorized

To work around this issue:

• Visit the Google Cloud Console
• Click on your existing Cloud project.
• Click on "Teams" under the settings menu.
• Find the email address corresponding to your certificate (it will be a ~45 character string followed by @developer.gserviceaccount.com).
• Remove and then re-add this email address as a member of your project.

[python] Add objects to wrap protobuffers and encapsulate helper methods

@proppy

Thought from working with this for the gcloud-python libraries; I'm wondering wondering if it makes sense to wrap each protobuf in an object to encapsulate methods in helper

I'll add them to my fork and you can tell me what you think.

Does node.js query interface actually work?

I have a simple query that always returns no results. The query is successful in the Datastore admin console, with the exception it does not use named args there. The query will not work at all from code unless named args are used, i.e. if I use SELECT * FROM Person WHERE lastName='Blackwell' I get the message: 'Disallowed literal: 'Blackwell'.'.

'cloudstore.runQuery({
auth: credentials,
datasetId: datasetId,
resource: {
gqlQuery: {
queryString: "SELECT * FROM Person WHERE lastName=@lastname",
nameArgs: [{name: "lastName", value: { stringValue:"Blackwell"}}]
},
}
}'

'{ batch:
{ entityResultType: 'FULL',
entityResults: [],
endCursor: '',
moreResults: 'MORE_RESULTS_AFTER_LIMIT',
skippedResults: null } }'

Publish consistent (correct) scopes

Seems the source on github reports the correct two https://github.com/GoogleCloudPlatform/google-cloud-datastore

• https://www.googleapis.com/auth/datastore
• https://www.googleapis.com/auth/userinfo.email

Looking at the discovery doc for datastore: https://www.googleapis.com/discovery/v1/apis/datastore/v1beta1/rest

Missing one

• https://www.googleapis.com/auth/userinfo.email

Looking at the online docs: https://developers.google.com/datastore/docs/apis/v1beta1/datasets/allocateIds

One too many

• https://www.googleapis.com/auth/userinfo#email
• https://www.googleapis.com/auth/userinfo.email
• https://www.googleapis.com/auth/datastore

See discussion here: https://groups.google.com/d/topic/gcd-discuss/IobUNprRooc/discussion

Extend the Datastore API to add more control of http.

It would be good to extend the Java Datastore API to be able to set more specific properties on the underlying HttpRequest, such as retries, timeouts etc.

null values not supported in JSON API

The following request:

{
 "mode": "NON_TRANSACTIONAL",
 "mutation": {
  "insertAutoId": [
   {
    "key": {
     "path": [
      {
       "kind": "test"
      }
     ]
    },
    "properties": {
     "propertyName": {
     }
    }
   }
  ]
 }
}

results in this response:

400 Bad Request

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "INVALID_ARGUMENT",
    "message": "Property \"propertyName\" has no value.",
   }
  ],
  "code": 400,
  "message": "Property \"propertyName\" has no value."
 }
}

Null values work correctly in the proto API but not JSON.

http://stackoverflow.com/questions/24141010/json-structure-to-insert-a-null-value-for-a-property-into-google-cloud-datastore/24174662

Appengine project will not serve my code, but serves a GCD service page

Current page: https://detect-analyze-notify-01a.appspot.com/
Page it should serve: https://over-sight.appspot.com/

Both projects are serving the same code. I suppose I have to add some kind of url handler somewhere? Or change the url handling in my app.yaml?
Currently the app.yaml is similar to this example: https://developers.google.com/appengine/docs/python/gettingstartedpython27/helloworld

I've tried uploading multiple versions but that isn't working. I might be remembering incorrectly but I thought that App Engine gave you the choice which version it should serve. But as far as I can see it's only serving the GCD service.

python client: gce error on dev server

I connect to the local version of my gcd using

datastore.set_options(dataset="datasetId", host="http://localhost:8888")
    try:
        # Create a RPC request to begin a new transaction.
        req = datastore.BeginTransactionRequest()
        # Execute the RPC synchronously.
        resp = datastore.begin_transaction(req)

On the last line it throws the error

File "/app/googledatastore/helper.py", line 70, in get_credentials_from_env
    credentials.refresh(http)
File "/app/oauth2client/client.py", line 516, in refresh
    self._refresh(http.request)
File "/app/oauth2client/gce.py", line 82, in _refresh
    response, content = http_request(uri)
File "/app/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
File "/app/oauth2client/client.py", line 475, in new_request
    self._refresh(request_orig)
File "/app/oauth2client/gce.py", line 82, in _refresh
    response, content = http_request(uri)
File "/app/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/app/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/app/httplib2/__init__.py", line 1286, in _conn_request
    response = conn.getresponse()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/dist27/gae_override/httplib.py", line 510, in getresponse
    'An error occured while connecting to the server: %s' % e)
error: An error occured while connecting to the server: Unable to fetch URL: http://metadata.google.internal/0.1/meta-data/service-accounts/default/acquire?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdatastore%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email Error: [Errno 8] nodename nor servname provided, or not known

Looks like the request to refresh the gce credentials throws a unhandled exception that breaks the code.

I comment out line 70 in helper.py and raised an exception there to temp fix it and get it to work on my dev environment.

certificate error when not running as root

Perhaps not an issue but I'm tought that my application shouldn't run as root.

In python interpreter:

import googledatastore as datastore
from googledatastore.helper import set_property
datastore.set_options(dataset="data-segment-326")
req = datastore.BlindWriteRequest()
host = req.mutation.upsert.add()
path_element = host.key.path_element.add()
path_element.kind = 'Foo'
path_element.name = "Bar"
set_property(host.property.add(), "aa", "bbb")
resp = datastore.blind_write(req)

Error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-x86_64/egg/googledatastore/__init__.py", line 71, in blind_write
  File "build/bdist.linux-x86_64/egg/googledatastore/connection.py", line 102, in blind_write
  File "build/bdist.linux-x86_64/egg/googledatastore/connection.py", line 208, in _call_method
  File "build/bdist.linux-x86_64/egg/oauth2client/util.py", line 132, in positional_wrapper
  File "build/bdist.linux-x86_64/egg/oauth2client/client.py", line 490, in new_request
  File "/usr/local/lib/python2.7/dist-packages/httplib2-0.8-py2.7.egg/httplib2/__init__.py", line 1570, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2-0.8-py2.7.egg/httplib2/__init__.py", line 1317, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2-0.8-py2.7.egg/httplib2/__init__.py", line 1252, in _conn_request
    conn.connect()
  File "/usr/local/lib/python2.7/dist-packages/httplib2-0.8-py2.7.egg/httplib2/__init__.py", line 1021, in connect
    self.disable_ssl_certificate_validation, self.ca_certs)
  File "/usr/local/lib/python2.7/dist-packages/httplib2-0.8-py2.7.egg/httplib2/__init__.py", line 80, in _ssl_wrap_socket
    cert_reqs=cert_reqs, ca_certs=ca_certs)
  File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 141, in __init__
    ciphers)
ssl.SSLError: [Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib

Better error message for Backend Error

b/11969636

Currently some non transactional write can failed with:

Backend Error

We should provide a more comprehensive error about what happens, and hint the user that he should retry the operation.

Error in node.js example

Running nodejs example adams.js I get the following error:

polo@gcloud-test:~/google-cloud-datastore-master/nodejs/demos/trivial$ node adams.js nirror-cloud

/home/polo/google-cloud-datastore-master/nodejs/demos/trivial/adams.js:56
  this.credentials.authorize((function(computeErr) {
                   ^
TypeError: Object #<Compute> has no method 'authorize'
    at Adams.authorize (/home/polo/google-cloud-datastore-master/nodejs/demos/trivial/adams.js:56:20)
    at new Adams (/home/polo/google-cloud-datastore-master/nodejs/demos/trivial/adams.js:45:8)
    at Object.<anonymous> (/home/polo/google-cloud-datastore-master/nodejs/demos/trivial/adams.js:208:12)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:906:3

Is this related to issue #43 ?

Error admin console

When I open http://localhost:8080/_ah/admin, I receive this error

HTTP ERROR 500

Problem accessing /_ah/admin. Reason:

For input string: "10,00"

Caused by:

java.lang.NumberFormatException: For input string: "10,00"
at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source)
at java.lang.Float.valueOf(Unknown Source)
at com.google.appengine.api.datastore.dev.DefaultHighRepJobPolicy.getUnappliedJobPctPropValue(DefaultHighRepJobPolicy.java:100)
at com.google.appengine.api.datastore.dev.DefaultHighRepJobPolicy.(DefaultHighRepJobPolicy.java:91)
at com.google.appengine.api.datastore.dev.LocalDatastoreService.initHighRepJobPolicy(LocalDatastoreService.java:507)
at com.google.appengine.api.datastore.dev.LocalDatastoreService.init(LocalDatastoreService.java:480)
at com.google.appengine.tools.development.ApiProxyLocalImpl.startServices(ApiProxyLocalImpl.java:598)
at com.google.appengine.tools.development.ApiProxyLocalImpl.access$700(ApiProxyLocalImpl.java:46)
at com.google.appengine.tools.development.ApiProxyLocalImpl$2.run(ApiProxyLocalImpl.java:584)
at com.google.appengine.tools.development.ApiProxyLocalImpl$2.run(ApiProxyLocalImpl.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.appengine.tools.development.ApiProxyLocalImpl.getService(ApiProxyLocalImpl.java:580)
at com.google.apphosting.utils.servlet.DatastoreViewerServlet.init(DatastoreViewerServlet.java:126)
at javax.servlet.GenericServlet.init(GenericServlet.java:212)
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
at org.mortbay.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:339)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:368)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:351)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:97)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:485)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

and in the shell i read this

GRAVE: javax.servlet.ServletContext log: unavailable
java.lang.NumberFormatException: For input string: "10,00"
at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source)
at java.lang.Float.valueOf(Unknown Source)
at com.google.appengine.api.datastore.dev.DefaultHighRepJobPolicy.getUna
ppliedJobPctPropValue(DefaultHighRepJobPolicy.java:100)
at com.google.appengine.api.datastore.dev.DefaultHighRepJobPolicy.
(DefaultHighRepJobPolicy.java:91)
at com.google.appengine.api.datastore.dev.LocalDatastoreService.initHigh
RepJobPolicy(LocalDatastoreService.java:507)
at com.google.appengine.api.datastore.dev.LocalDatastoreService.init(Loc
alDatastoreService.java:480)
at com.google.appengine.tools.development.ApiProxyLocalImpl.startService
s(ApiProxyLocalImpl.java:598)
at com.google.appengine.tools.development.ApiProxyLocalImpl.access$700(A
piProxyLocalImpl.java:46)
at com.google.appengine.tools.development.ApiProxyLocalImpl$2.run(ApiPro
xyLocalImpl.java:584)
at com.google.appengine.tools.development.ApiProxyLocalImpl$2.run(ApiPro
xyLocalImpl.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.appengine.tools.development.ApiProxyLocalImpl.getService(A
piProxyLocalImpl.java:580)
at com.google.apphosting.utils.servlet.DatastoreViewerServlet.init(Datas
toreViewerServlet.java:126)
at javax.servlet.GenericServlet.init(GenericServlet.java:212)
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.jav
a:440)
at org.mortbay.jetty.servlet.ServletHolder.getServlet(ServletHolder.java
:339)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487
)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1166)
at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocke
tFilter.java:74)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.tools.development.ResponseRewriterFilter.doFilte
r(ResponseRewriterFilter.java:123)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.tools.development.HeaderVerificationFilter.doFil
ter(HeaderVerificationFilter.java:34)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(Serve
BlobFilter.java:63)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter
(TransactionCleanupFilter.java:43)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.tools.development.StaticFileFilter.doFilter(Stat
icFileFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doDi
rectRequest(DevAppServerModulesFilter.java:368)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doDi
rectModuleRequest(DevAppServerModulesFilter.java:351)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doFi
lter(DevAppServerModulesFilter.java:116)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:3
88)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.jav
a:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:1
82)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:7
65)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

    at com.google.appengine.tools.development.DevAppEngineWebAppContext.hand

le(DevAppEngineWebAppContext.java:97)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:1
52)
at com.google.appengine.tools.development.JettyContainerService$ApiProxy
Handler.handle(JettyContainerService.java:485)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:1
52)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:54
2)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCo
nnection.java:923)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.ja
va:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.j
ava:582)

20-ago-2013 10.41.59 com.google.apphosting.utils.jetty.JettyLogger warn
AVVERTENZA: /_ah/admin
java.lang.NumberFormatException: For input string: "10,00"
at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source)
at java.lang.Float.valueOf(Unknown Source)
at com.google.appengine.api.datastore.dev.DefaultHighRepJobPolicy.getUna
ppliedJobPctPropValue(DefaultHighRepJobPolicy.java:100)
at com.google.appengine.api.datastore.dev.DefaultHighRepJobPolicy.
(DefaultHighRepJobPolicy.java:91)
at com.google.appengine.api.datastore.dev.LocalDatastoreService.initHigh
RepJobPolicy(LocalDatastoreService.java:507)
at com.google.appengine.api.datastore.dev.LocalDatastoreService.init(Loc
alDatastoreService.java:480)
at com.google.appengine.tools.development.ApiProxyLocalImpl.startService
s(ApiProxyLocalImpl.java:598)
at com.google.appengine.tools.development.ApiProxyLocalImpl.access$700(A
piProxyLocalImpl.java:46)
at com.google.appengine.tools.development.ApiProxyLocalImpl$2.run(ApiPro
xyLocalImpl.java:584)
at com.google.appengine.tools.development.ApiProxyLocalImpl$2.run(ApiPro
xyLocalImpl.java:581)
at java.security.AccessController.doPrivileged(Native Method)
at com.google.appengine.tools.development.ApiProxyLocalImpl.getService(A
piProxyLocalImpl.java:580)
at com.google.apphosting.utils.servlet.DatastoreViewerServlet.init(Datas
toreViewerServlet.java:126)
at javax.servlet.GenericServlet.init(GenericServlet.java:212)
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.jav
a:440)
at org.mortbay.jetty.servlet.ServletHolder.getServlet(ServletHolder.java
:339)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487
)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1166)
at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocke
tFilter.java:74)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.tools.development.ResponseRewriterFilter.doFilte
r(ResponseRewriterFilter.java:123)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.tools.development.HeaderVerificationFilter.doFil
ter(HeaderVerificationFilter.java:34)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(Serve
BlobFilter.java:63)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter
(TransactionCleanupFilter.java:43)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.tools.development.StaticFileFilter.doFilter(Stat
icFileFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doDi
rectRequest(DevAppServerModulesFilter.java:368)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doDi
rectModuleRequest(DevAppServerModulesFilter.java:351)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doFi
lter(DevAppServerModulesFilter.java:116)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:3
88)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.jav
a:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:1
82)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:7
65)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

    at com.google.appengine.tools.development.DevAppEngineWebAppContext.hand

le(DevAppEngineWebAppContext.java:97)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:1
52)
at com.google.appengine.tools.development.JettyContainerService$ApiProxy
Handler.handle(JettyContainerService.java:485)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:1
52)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:54
2)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpCo
nnection.java:923)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.ja
va:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.j
ava:582)
20-ago-2013 10.41.59 com.google.appengine.tools.development.LocalResourceFileSer
vlet doGet
AVVERTENZA: No file found for: /favicon.ico
20-ago-2013 10.41.59 com.google.appengine.tools.development.LocalResourceFileSer
vlet doGet
AVVERTENZA: No file found for: /favicon.ico

Feature request: add support for urlsafe keys as generated from NDB

Curently this library uses the protobuff serialize and serialize keys protobuff style. Ndb's Key serialization scheme is different.

I would like to share ndb.keys between the two libraries but I have no clue how to convert them.

Does anyone have a conversion example to constuct a googledatastore.Key from a ndb.key.urlsafe() string? An example to build a urlsafe string from a googledatastore.Key would be nice as well

googledatastore python default connection is not thread safe

b/11969608

googledatastore.get_default_connection is not threadsafe because httplib2.Http isn't.

---

Any clue what I could be doing wrong?

013-11-27 16:51:15+0000 [HTTPChannel (TLSMemoryBIOProtocol),40,89.20.92.161] Unhandled Error
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 525, in __bootstrap
        self.__bootstrap_inner()
      File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 505, in run
        self.__target(*self.__args, **self.__kwargs)
    --- <exception caught here> ---
      File "/usr/local/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
        result = context.call(ctx, function, *args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/usr/local/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
        return func(*args,**kw)
      File "/home/sjuul/workspace/oversightBackend/messageHub/core/messageHubServerProtocol.py", line 92, in sendConfigSync
        "config": State.getCoreConfig(coreUuid),
      File "/home/sjuul/workspace/oversightBackend/messageHub/core/stateMessageHub.py", line 543, in getCoreConfig
        hosts = custKey.getDescendants(kind='Host', parentCore=coreUuid)
      File "/home/sjuul/workspace/oversightBackend/messageHub/core/dbWrapper.py", line 377, in getDescendants
        return req.execute(keysOnly)
      File "/home/sjuul/workspace/oversightBackend/messageHub/core/dbWrapper.py", line 140, in requestExecute
        resp = datastore.run_query(req)
      File "/usr/local/lib/python2.7/dist-packages/googledatastore/__init__.py", line 71, in run_query
        return get_default_connection().run_query(request)
      File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 103, in run_query
        datastore_v1_pb2.RunQueryResponse)
      File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 197, in _call_method
        resp.ParseFromString(content)
      File "/usr/local/lib/python2.7/dist-packages/google/protobuf/message.py", line 182, in ParseFromString
        self.MergeFromString(serialized)
      File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/python_message.py", line 795, in MergeFromString
        if self._InternalParse(serialized, 0, length) != length:
      File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/python_message.py", line 819, in InternalParse
        new_pos = local_SkipField(buffer, new_pos, end, tag_bytes)
      File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/decoder.py", line 716, in SkipField
        return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end)
      File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/decoder.py", line 685, in _RaiseInvalidWireType
        raise _DecodeError('Tag had invalid wire type.')
    google.protobuf.message.DecodeError: Tag had invalid wire type.

2013-11-27 16:51:15+0000 [-] loop failed
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
        self.mainLoop()
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
        self.runUntilCurrent()
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/task.py", line 218, in __call__
        d = defer.maybeDeferred(self.f, *self.a, **self.kw)
    --- <exception caught here> ---
      File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 139, in maybeDeferred
        result = f(*args, **kw)
      File "/home/sjuul/workspace/oversightBackend/messageHub/core/stateMessageHub.py", line 75, in emptyBatchQueue
        printReqResponse(datastore.commit(req))
      File "/usr/local/lib/python2.7/dist-packages/googledatastore/__init__.py", line 81, in commit
        return get_default_connection().commit(request)
      File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 135, in commit
        datastore_v1_pb2.CommitResponse)
      File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 193, in _call_method
        self._url + method, method='POST', body=payload, headers=headers)
      File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
        return wrapped(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 490, in new_request
        redirections, connection_type)
      File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1570, in request
        (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
      File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1317, in _request
        (response, content) = self._conn_request(conn, request_uri, method, body, headers)
      File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1286, in _conn_request
        response = conn.getresponse()
      File "/usr/lib/python2.7/httplib.py", line 1034, in getresponse
        response.begin()
      File "/usr/lib/python2.7/httplib.py", line 407, in begin
        version, status, reason = self._read_status()
      File "/usr/lib/python2.7/httplib.py", line 371, in _read_status
        raise BadStatusLine(line)
    httplib.BadStatusLine: ''

New Code For Node.js Sample Needed

The existing code for the node.js example is both complex and broken. It won't work with the most recent googleapis version. Below is code that is simple and will connect and authenticate. Querying is another issue, see open bug on this:

'var util = require('util');
var google = require("googleapis");
var SCOPES = ['https://www.googleapis.com/auth/userinfo.email',
'https://www.googleapis.com/auth/datastore'];
var datasetId = "";
credentials = new google.auth.JWT(
process.env['DATASTORE_SERVICE_ACCOUNT'],
process.env['DATASTORE_PRIVATE_KEY_FILE'],
null,
SCOPES);
credentials.authorize((function(jwtErr) {
if (jwtErr) {
errors = {'jwt auth error':jwtErr};
console.log(errors);
return;
}
var cloudstore = google.datastore('v1beta2').datasets;

'

Add NDB Python library

b/11319745

We already use Google App Engine, it would be nice if the NDB library was available. Do you guys have a timeline on that?

What is the difference between datastore.Query() and datastore.RunQueryRequest() ?

What does this repo do? What should it do?

There's been a bunch of discussion (recently mostly me just rambling to myself) around what to do with the stale nodejs samples in this repo now that googleapis and gcloud have both been updated/created. I've been wondering what the point of this repo is nowadays... Three paragraphs into the README it says "This repository contains the source code of samples and developer resources related to Google Cloud Datastore". I think if developers want to learn about Datastore in general, the https://cloud.google.com/datastore website is best for that. Samples should be found in their respective api client repos in an examples folder or available in their own repo altogether e.g. like gcloud-node-todos.

So can someone shine some light on what this repo is meant to accomplish that is not already being accomplished in other repos or websites? This repo hasn't been getting much love recently and I'm curious if it's really useful anymore.

Long integer values not supported in the Python API's set_value / set_property / add_property helpers

Using long values with the set & add helper functions in the Python API results in an error like:
TypeError: value type: 1015221965056L not supported

The proto and Java API's support 64 bit integers, and long values up to 64 bit integer size work in Python if you set them directly, just not when using the helper functions.

Python 3.3 support

Hi,

any chances to get python 3.3 support? Maybe you accept patches for that?

Thanks

Add support for App Engine Java client library

Add support for App Engine Java client library (as well as objectify), as requested on http://stackoverflow.com/questions/21502396/communication-with-datastore-and-cloud-storage-from-google-compute-engine-in-jav

List of nested entities not working anymore (or I should do it differently)

I've written a unittest which fails on my side:

import unittest
import googledatastore as datastore
from googledatastore.datastore_v1_pb2 import CommitRequest
from googledatastore.helper import set_property


class  TestState(unittest.TestCase):

    def setUp(self):
        datastore.set_options(dataset='detect-analyze-notify-01a')

    def test_01NormalEntity(self):
        """
        Normal entity; works fine
        """
        req = CommitRequest()
        req.mode = CommitRequest.NON_TRANSACTIONAL

        key = datastore.Key()
        path_element = key.path_element.add()
        path_element.kind = 'SomeKind'
        path_element.name = 'SomeName'

        entity = req.mutation.upsert.add()
        set_property(entity.property.add(), "SomeProperty", "SomeValue", indexed=False)
        entity.key.CopyFrom(key)

        print datastore.commit(req)

    def test_02NestedEntity(self):
        """
        Nested entity example; works fine
        """
        req = CommitRequest()
        req.mode = CommitRequest.NON_TRANSACTIONAL

        key = datastore.Key()
        path_element = key.path_element.add()
        path_element.kind = 'SomeKind'
        path_element.name = 'SomeName'

        entity = req.mutation.upsert.add()
        set_property(entity.property.add(), "SomeProperty", "SomeValue", indexed=False)

        subEnt = datastore.Entity()
        set_property(entity.property.add(), "SomeSubEnt", subEnt, indexed=False)

        entity.key.CopyFrom(key)

        print datastore.commit(req)

    def test_03NestedEntityList_INDEXED_FALSE(self):
        """
        Nested entity list, INDEXED = False, this is how I specified it in the previous version. This used to work
        """
        req = CommitRequest()
        req.mode = CommitRequest.NON_TRANSACTIONAL

        key = datastore.Key()
        path_element = key.path_element.add()
        path_element.kind = 'SomeKind'
        path_element.name = 'SomeName'

        entity = req.mutation.upsert.add()
        set_property(entity.property.add(), "SomeProperty", "SomeValue", indexed=False)

        subEntList = [datastore.Entity(), datastore.Entity()]

        set_property(entity.property.add(), "SomeSubEntList", subEntList, indexed=False)

        entity.key.CopyFrom(key)

        print datastore.commit(req)
        """
        Traceback (most recent call last):
          File "tests\nestedEntities.py", line 69, in test_03NestedEntityList_INDEXED_FALSE
            print datastore.commit(req)
          File "C:\Python27\lib\site-packages\googledatastore\__init__.py", line 81, in commit
            return get_default_connection().commit(request)
          File "C:\Python27\lib\site-packages\googledatastore\connection.py", line 135, in commit
            datastore_v1_pb2.CommitResponse)
          File "C:\Python27\lib\site-packages\googledatastore\connection.py", line 195, in _call_method
            raise RPCError(method, response, content)
        RPCError: commit RPC client failure with HTTP(400) Bad Request: {
          "error": {
            "errors": [
             {
               "domain": "global",
               "reason": "INVALID_ARGUMENT",
               "message": "A Value containing a list_value cannot specify indexed."
             }
            ],
            "code": 400,
            "message": "A Value containing a list_value cannot specify indexed."
          }
        }
        """

    def test_04NestedEntityList_INDEXED_TRUE(self):
        """
        Nested entity list INDEXED; this one should fail
        """
        req = CommitRequest()
        req.mode = CommitRequest.NON_TRANSACTIONAL

        key = datastore.Key()
        path_element = key.path_element.add()
        path_element.kind = 'SomeKind'
        path_element.name = 'SomeName'

        entity = req.mutation.upsert.add()
        set_property(entity.property.add(), "SomeProperty", "SomeValue", indexed=False)

        subEntList = [datastore.Entity(), datastore.Entity()]

        set_property(entity.property.add(), "SomeSubEntList", subEntList, indexed=True)

        entity.key.CopyFrom(key)
        print datastore.commit(req)
        """
        Traceback (most recent call last):
          File "tests\nestedEntities.py", line 90, in test_04NestedEntityList_INDEXED_TRUE
            print datastore.commit(req)
          File "C:\Python27\lib\site-packages\googledatastore\__init__.py", line 81, in commit
            return get_default_connection().commit(request)
          File "C:\Python27\lib\site-packages\googledatastore\connection.py", line 135, in commit
            datastore_v1_pb2.CommitResponse)
          File "C:\Python27\lib\site-packages\googledatastore\connection.py", line 195, in _call_method
            raise RPCError(method, response, content)
        RPCError: commit RPC client failure with HTTP(400) Bad Request: {
          "error": {
            "errors": [
             {
               "domain": "global",
               "reason": "INVALID_ARGUMENT",
               "message": "Entity value is indexed."
             }
            ],
            "code": 400,
            "message": "Entity value is indexed."
          }
        }"""

    def test_05NestedEntityList_INDEXED_NONE(self):
        """
        Set indexed to None, I expected this to work
        """
        req = CommitRequest()
        req.mode = CommitRequest.NON_TRANSACTIONAL

        key = datastore.Key()
        path_element = key.path_element.add()
        path_element.kind = 'SomeKind'
        path_element.name = 'SomeName'

        entity = req.mutation.upsert.add()
        set_property(entity.property.add(), "SomeProperty", "SomeValue", indexed=False)

        subEntList = [datastore.Entity(), datastore.Entity()]

        set_property(entity.property.add(), "SomeSubEntList", subEntList, indexed=None)

        entity.key.CopyFrom(key)

        print datastore.commit(req)
        """
        Traceback (most recent call last):
          File "tests\nestedEntities.py", line 111, in test_05NestedEntityList_INDEXED_NONE
            print datastore.commit(req)
          File "C:\Python27\lib\site-packages\googledatastore\__init__.py", line 81, in commit
            return get_default_connection().commit(request)
          File "C:\Python27\lib\site-packages\googledatastore\connection.py", line 135, in commit
            datastore_v1_pb2.CommitResponse)
          File "C:\Python27\lib\site-packages\googledatastore\connection.py", line 195, in _call_method
            raise RPCError(method, response, content)
        RPCError: commit RPC client failure with HTTP(400) Bad Request: {
          "error": {
            "errors": [
             {
               "domain": "global",
               "reason": "INVALID_ARGUMENT",
               "message": "Entity value is indexed."
             }
            ],
            "code": 400,
            "message": "Entity value is indexed."
          }
        }"""

if __name__ == '__main__':
    unittest.main()

Dashboard for GCD latency

b/11970400

It would be great to have a dashboard like the one here https://code.google.com/status/appengine/detail/hr-datastore/2013/06/13#ae-trust-detail-hr-datastore-get-latency for GCD. Of particular interest would be latencies of these operations from the Google Compute Engine and all AWS regions worldwide.

A dashboard like this would be a great way to showcase GCD's capabilities and for people to evaluate where to deploy their now distributed applications.

App Identity based authentication always return quota errors

b/10097704

App Identity based authentication always return the following error:

 (403) Daily Limit for Unauthenticated Use Exceeded

Since there is no project associated with the underlying App Engine service account.