Comments (10)
@chetanmeda I tested it many times but still not able to reproduce that error,
does this happening with every partition table or specific table ? if yes, could you please provide response pb ?
from java-bigquery.
@pmakani A table created with the API google-api-services-bigquery does not define partition type as an Enum. And the TimePartitioning Type need not be "DAY" and can be null as well.
Try creating a table with the below code.
Table tablePb = new Table();
`Table tablePb = new Table();
tablePb.setType("TABLE");
TableSchema tableSchemaPb = new TableSchema();
TableFieldSchema tableFieldSchema1 = new TableFieldSchema();
tableFieldSchema1.setName("stringField");
tableFieldSchema1.setType("STRING");
TableFieldSchema tableFieldSchema2 = new TableFieldSchema();
tableFieldSchema2.setName("booleanField");
tableFieldSchema2.setType("BOOL");
TableFieldSchema tableFieldSchema3 = new TableFieldSchema();
tableFieldSchema3.setName("timestampField");
tableFieldSchema3.setType("TIMESTAMP");
List<TableFieldSchema> tableFieldSchemaList= new ArrayList<TableFieldSchema>();
tableFieldSchemaList.add(tableFieldSchema1);
tableFieldSchemaList.add(tableFieldSchema2);
tableFieldSchemaList.add(tableFieldSchema3);
tableSchemaPb.setFields(tableFieldSchemaList);
tablePb.setSchema(tableSchemaPb);
TimePartitioning timePartitioning = new TimePartitioning();
timePartitioning.setField("timestampField");
timePartitioning.setType(null);
tablePb.setTimePartitioning(timePartitioning);
TableReference tableReference = new TableReference();
tableReference.setProjectId("bigquerty-poc");
tableReference.setDatasetId("some_data_set");
tableReference.setTableId("try_1_part");
tablePb.setTableReference(tableReference);
try {
bigquery
.tables()
.insert(tableReference.getProjectId(), tableReference.getDatasetId(), tablePb)
.execute();
}
catch (Exception e){
e.printStackTrace();
}
}`
bigquery above is an instance of com.google.api.services.bigquery.Bigquery.
Table created using code like above ( timePartitioning.setType(null) ) is a valid partitioned table and works perfectly. But the new API has Timepartitioning.Type as Enum which causes the exception.
As per the below Type is a enum
So we have to change the below line
to
newBuilder(Type.valueOf(firstNonNull(partitioningPb.getType(),Type.DAY))
from java-bigquery.
So, this is an odd behavior, and the backend is allowing nulls but probably should not. I've filed an internal issue (155198865) to see if this can be corrected or documented in the backend.
The important thing to note here is that the behavior (setting a null partitioning type) is undefined, and shouldn't be used. We can consider allowing it through the manual library, but I'd like to confirm what the intent is internally (document or restrict) for unset types before we address this in this library.
from java-bigquery.
@shollyman The backend should allow nulls because setting TimePartitioning type as "DAY" is not mandatory requirement. So In the above table creation code, omitting
timePartitioning.setType(null);
will cause the same issue. So it was more of a non-mandatory internally assigned parameter being made mandatory on the new library. It is always a "DAY" internally. So it need not be set explicitly.
The new library can expect it to be mandatorily assigned, but cannot always expect a value to be returned for TimePartitioning.Type because most of the tables created using java library did not need it. More of what the client java library is expecting as mandatory behaviour should become non-mandatory behaviour. This issue is causing the new client library to be unusable for us and we have to revert to using google-api-services-bigquery directly.
Can this change for TimePartitioning.Type to be able to accept null Types while reading a Table be done so that we can continue using google-cloud-bigquery.
from java-bigquery.
@pmakani - Thanks for the PR , Let me know it will get merged and when will we get the latest jar out if it
from java-bigquery.
@stephaniewang526 - Any idea when we can expect the latest jar ? https://repo.maven.apache.org/maven2/com/google/cloud/google-cloud-bigquery/1.111.1/
from java-bigquery.
It should be release on Monday, 5/4.
from java-bigquery.
@stephaniewang526 - Thanks for the update
from java-bigquery.
@stephaniewang526 - I could see latest jars https://repo.maven.apache.org/maven2/com/google/cloud/google-cloud-bigquery/1.111.3/
from java-bigquery.
https://repo1.maven.org/maven2/com/google/cloud/google-cloud-bigquery/ please find the latest jar, 1.111.3 here.
from java-bigquery.
Related Issues (20)
- auto-value is included in the dependency HOT 3
- bigquery.it.ITBigQueryTest: testGeographyParameter failed HOT 1
- Add Manifest FileSetSpecType to External Table Configuration HOT 3
- What is the relationship between the Protobuf models in bigquery.storage.v1 and JSON models in bigquery.model ?. HOT 2
- Don't throw exception in BigQueryResultSet.close()
- bigquery.it.ITBigQueryTest: testCreateAndGetJobWithSelectedFields failed HOT 2
- BigQuery: google-cloud-bigquery 2.31.0 breaks compatibility with Kotlin 1.6 via dependency on auto-value 1.10.2 HOT 4
- bigquery.it.ITBigQueryTest: testFastDMLQuery failed HOT 1
- bigquery.it.ITBigQueryTest: testQueryJobWithSearchReturnsSearchStatisticsUnused failed HOT 1
- Samples snippet InsertingDataTypes.java is writing byte array to `ARRAY<BYTES>` instead of `BYTES`
- Unchecked NPE in ConnectionImpl:dryRun
- `com.google.cloud.bigquery.it.ITBigQueryTest.testFastQuerySlowDDL` is flaky HOT 4
- bigquery.it.ITBigQueryTest: testFastDMLQuery failed HOT 4
- BigQueryResultImpl.getDouble throws ClassCastException if read API is used HOT 1
- Add BigLakeConfiguration as a property of StandardTableDefinition
- bigquery.it.ITBigQueryTest: testExtractJobWithModel failed HOT 3
- bigquery.it.ITBigQueryTest: testModelLifecycle failed HOT 3
- bigquery.it.ITBigQueryTest: testReferenceFileSchemaUriForAvro failed HOT 2
- bigquery.it.ITBigQueryTest: testLocationFastSQLQueryWithJobId failed HOT 2
- Is java google cloud bigquery is threadsafe or not ? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from java-bigquery.