Comments (4)
Hi,
I'm encountering the same issue.
If I use a "return null" statement when I try to skip the document row I obtain the "schema can not be null" error.
Did anyone manage to resolve the issue?
Many thanks!
from dataflowtemplates.
Hi,
I'm encountering the same issue. If I use a "return null" statement when I try to skip the document row I obtain the "schema can not be null" error. Did anyone manage to resolve the issue? Many thanks!
Hi @britz89 . I have not found a fix but I found an alternative way to skip it.
I pull all the data then use a saved query to run and create a new Table from the import. I have the filter applied in that saved query
from dataflowtemplates.
So if I understood correctly you are pulling the full collection, storing in a temp table and then in a subsequent step filtering the rows. Correct?
My requirement is to avoid a full copy of the collection, so I hope that this issue will be fixed otherwise I will have to find another way.
Thanks for your suggestion, btw!
from dataflowtemplates.
So if I understood correctly you are pulling the full collection, storing in a temp table and then in a subsequent step filtering the rows. Correct? My requirement is to avoid a full copy of the collection, so I hope that this issue will be fixed otherwise I will have to find another way. Thanks for your suggestion, btw!
Yes that is what I am currently doing until it is fixed because I need a solution up. The other alternative I thought about is using a custom batch template and fixing the issue.
from dataflowtemplates.
Related Issues (20)
- [Feature Request]: Avoid duplicating every field in PubsubToElasticsearch
- [Bug]: mvn clean package v1 FAILED due to splunk.jfrog.io repo removal HOT 1
- Google_cloud_to_neo4j v2024-03-27-00_rc00 causes unexpected increase of shuffle data processed and worker OOMs HOT 3
- [Bug]: Investigate YamlTemplateIT fail on upgrading Beam 2.55.1 HOT 1
- [Feature Request]: Add new template to publish ordered spanner change records to pub/sub HOT 1
- [Bug]: <Kafka to BigQuery template is confusing and there are no mandatory parameters. Clean up is required. See 4 items mentioned in the What happened section>
- [Feature Request]: <Productize DataFlow engine so that automation could be obtained using natural key meta information for each topic/row/eventdata>
- [Feature Request]: <Provide ML features that augment automated data processing into BigQuery/BigTable>
- [Bug]: Oracle to BigQuery - "bigQuerySchemaPath" parameter issue HOT 1
- [Feature Request]: Better Developer Docs/Automation
- [Bug]: mongodb cdc to Bigquery is not reading data from mongoDB and publish to pubsub
- [Feature Request]: More informative error message for Dataplex_File_Format_Conversion template's NullPointerException error message when missing schema.
- MongoDB Batch to BigQuery not running on the specified machine type HOT 1
- [Feature Request]: Add param for database_id in Cloud Firestore (Datastore Mode) to Cloud Storage text
- [Bug]: mvn clean package v2 FAILED due to splunk.jfrog.io repo HOT 2
- [Bug]: Revert YamlTemplate once Beam 2.57.0 is released
- [Bug]: Cannot use SASL_PLAIN for connection to Kafka HOT 6
- [Feature Request]: text tranform reload interval for pub/sub to elastic template HOT 1
- Handle Severe errors: How to process severe errors in Spanner to BigQuery Change stream dataflow
- [Bug]: Kafka to BigQuery template not working for JSON HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataflowtemplates.