[Bug]: Dataflow - MongoDB-to-BigQuery batch mode failing with filter on data

[robbycarter]

[Bug]: Dataflow - MongoDB-to-BigQuery batch mode failing with filter on data

[britz89]

Hi,

I'm encountering the same issue.
If I use a "return null" statement when I try to skip the document row I obtain the "schema can not be null" error.
Did anyone manage to resolve the issue?
Many thanks!

[robbycarter]

Hi,

I'm encountering the same issue. If I use a "return null" statement when I try to skip the document row I obtain the "schema can not be null" error. Did anyone manage to resolve the issue? Many thanks!

Hi @britz89 . I have not found a fix but I found an alternative way to skip it.
I pull all the data then use a saved query to run and create a new Table from the import. I have the filter applied in that saved query

[britz89]

So if I understood correctly you are pulling the full collection, storing in a temp table and then in a subsequent step filtering the rows. Correct?
My requirement is to avoid a full copy of the collection, so I hope that this issue will be fixed otherwise I will have to find another way.
Thanks for your suggestion, btw!

[robbycarter]

So if I understood correctly you are pulling the full collection, storing in a temp table and then in a subsequent step filtering the rows. Correct? My requirement is to avoid a full copy of the collection, so I hope that this issue will be fixed otherwise I will have to find another way. Thanks for your suggestion, btw!

Yes that is what I am currently doing until it is fixed because I need a solution up. The other alternative I thought about is using a custom batch template and fixing the issue.