Comments (7)
We probably need to see the ibis expression. Column names that look like that should effectively never make it into bigquery.
Can you give a reproducible example that only uses Ibis? We definitely need to see what Ibis APIs you're using to produce this error.
from ibis.
Rewrite the problem description with an easy example
from ibis.
Updated the error message*******
400 POST https://bigquery.googleapis.com/upload/bigquery/v2/projects/voltrondata-demo/jobs?uploadType=multipart: Invalid field name "Cast(y, int64)_ef9a46". Fields must contain the allowed characters, and be at most 300 characters long. For allowed characters, please refer to https://cloud.google.com/bigquery/docs/schemas#column_names
seems like Cast(y, int64)_ef9a46
is not a valid name for bigquery, because have letters after number.
from ibis.
are you running against main
? That looks like the bigquery issues I fixed yesterday in #9141 and #9149
from ibis.
oh, no. Let me pull it from main
are you running against main? That looks like the bigquery issues I fixed yesterday in #9141 and #9149
from ibis.
we convert an intermediate table into a memtable
in bis-ml, see link
The intermediate information in the expression tree got lost, unfortunately, the new name "Cast(y, float64)"
is not a valid bigquery column name.
I think it is not an ibis issue, we could fix this in ibisml.
Here is the simplified code to reproduce the error:
import ibis
ibis.options.interactive = True
con = ibis.bigquery.connect(project_id="xx", dataset_id="xx")
t = ibis.memtable({
"x": ["a", "b"],
"y": ["0", "1"]
})
con.create_table(
"t", t.to_pyarrow(), overwrite=True
)
t = con.table("t")
base_table = t.drop("y")
target_table = t.y.cast(dt.float64)
y_m = ibis.memtable(target_table.as_table().to_pyarrow())
y_m = y_m.mutate(x=ibis.literal("a"))
base_table.join(y_m, "x")
And the new name "Cast(y, float64)"
is not a valid name in bigquery.
ibis.to_sql(y_m)
sql
SELECT
"t0"."Cast(y, float64)",
'a' AS "x"
FROM "ibis_pyarrow_memtable_cimh6vdzlvhyxlddji3g6jvugq" AS "t0"
from ibis.
I think that between the whitespace fixes in #9160 and the improved to_sql
for memtables in #9228 that this may not be an issue anymore. I'm going to close it out, but ping if this shows up again!
from ibis.
Related Issues (20)
- feat: add a method for table existence check HOT 5
- bug: `create_table(temp=True)` timing out due to slow table existence check HOT 2
- add support for TIMESTAMPTZ HOT 4
- feat: add a table_exists(table_name) api HOT 1
- bug: `to_sql` always shows DuckDB SQL for a memtable even if there's a default backend set HOT 1
- Polars backend can read only 1 csv HOT 1
- docs: add ops docstrings
- ci: more granular cloud run labels
- feat: allow to connect to a duckdb named in-memory database
- bug: memtable(..."1970-01-01"..., schema="date") errors HOT 2
- feat: better distinguish ibis scalars from evaluated results in interactive repr
- feat: add databricks support HOT 2
- bug: .insert() uses column position, not names, to align HOT 1
- feat(settings): generalize and extend settings passthrough dictionary
- feat: Leverage SQLGlot AST for query parsing HOT 2
- bug: EOFError when running ibis tutorial code with ClickHouse backend HOT 4
- bug: The function `parse_sql` fails when `COUNT(*)` is in the query HOT 4
- feat: Add the ability to pass regex flags HOT 2
- Config Value spark.sql.mapKeyDedupPolicy not Supported by Databricks SQL Warehouse HOT 1
- feat: profiling? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ibis.